Alex Valdez

Anya did the weird sibling radar thing at lunch. texted "mom called you?" and then immediately backed off with "just checking. no reason". very her.

week's pretty set. `metrics-router` to 95% first, then `shard-keeper` planning with Nadia on friday. `ingest-edge` is already done so it's mostly keeping prod boring and not getting cute with the cutover.

Anya texted on Apr 10 from that diner place again. said some coworker told her maybe Q2 cuts and she was basically not stressed about it at all. I did the "they always say that" thing, but tbh I kind of tucked it away.

on 7th now. bagel place has that burned-coffee smell even when the coffee's actually fine. light's weirdly nice this morning, hitting the windows across the block and making everything look cleaner than it is. little cold still.

7th bagel place before work. sesame everything, still warm enough to fog the paper bag a little. guy ahead of me ordered toasted with scallion and then just stood there fully asleep. pretty quiet in here.

metrics-router's at 85% now. holding steady after the earlier canary hiccup, no drops and latency looks normal. he's still aiming to push it to 95% by friday if it stays this boring.

ok so today is mostly `shard-keeper` prep. want the config-push bits laid out before friday w Nadia so that meeting is actual review, not live thinking. `metrics-router` still needs a quick look first just to make sure the 85% state stays boring. also Devika starts the 14-day block tonight so i'm trying not to let work sprawl all the way through the evening.

Apr 12 EOD: metrics-router's sitting at 92%. one weird batch-export hiccup but it cleared itself in like 90 seconds, everything else looked boring. Devika just headed out and the apartment went flat fast. Kibo did the little hallway check and gave up.

Apr 12 and we're back in it. Devika started the second night-shift block tonight -- 14 days this time. had to stop and count because the first one was Mar 31 and somehow that was also this spring. apartment's really quiet already. Kibo keeps looking at the door then at me like i'm the wrong person.

lunch truck lady looked at me and Anya and went wait, sister? then said she'd seen us last Sunday. which, fair, but also lol. brooklyn is so small sometimes.

metrics-router migration is still at 92% prod cutover. there was one small batch-export wobble this afternoon, cleared itself in about 90 sec, no other weirdness. still tracking the Apr 11 plan: 95% by Apr 13, then hold there till shard-keeper restarts.

No deploys today. Hold `metrics-router`, `ingest-edge`, and `shard-keeper` till after the Nadia pre-`shard-keeper` sync tomorrow -- then leave Saturday quiet. Monday we can line up the exec plan.

friday done. mostly coffee and code review, nothing weird. no deploys tonight -- leaving metrics-router, ingest-edge, shard-keeper alone. heading out.

ok so before the 2p w Nadia I want the parity doc unstuck. if it's still blank by late morning just ping her again and ask for explicit sign-off in writing on the label-name change, not verbal. also put her alerting label-map PR in today's queue and get eyes on it before EOD. ingest-edge runbook still stale on that path.

Apr 15, pre shard-keeper sync w Nadia: label-name change is fine to ship on the next config push. mostly just wanted explicit yes/no and got it. one thing though -- rollup-service consumer wasn't in the room, so parity is only confirmed at the shard-keeper side right now. i'm leaning we still send it, but i want that called out somewhere before anyone treats this as end-to-end clean.

Put the Apr 15 note in the cutover doc. Nadia signed off on the label-name change there, in writing, and it's slotted for the Apr 18 prod push. That's sorted.

Devika was home for like six hours between shifts so we did the tiny fake-normal version of a Saturday. ate leftovers, she passed out basically on contact with the bed, and now she's back out. i'm up reading with one lamp on. Kibo keeps checking the hallway like maybe this whole schedule was a bad idea.

7th bagel place was doing that weekend thing where nobody's in a rush and the line still somehow takes forever. walked back slow. devika's out cold from nights so the apartment's basically just page turns and the radiator clicking at itself.

Diego's run was actually good today. little cold at first then fine once we got moving, field still kind of slick. had one stretch where my legs remembered what they were doing lol. eggs after and an absurd amount of hot sauce.

Devika heading out in scrubs again. paused by the door and said she'd be up for the 6am handoff if I wanted a 30-second hello. very Devika. Kibo watched the whole exchange like he was on the schedule too.

today's pretty normal til 6. `shard-keeper` config push then -- label rename is cleared now so just want the rollout steps tight before we hit it. otherwise mostly keeping an eye on `metrics-router` at 92% and not touching anything dumb.

Shard-keeper config push finished at 6:05. Initial dashboards are green and the label rename looks clean so far. Heading home now, I'll check it again before bed.

9:42 shard-keeper page and i'm still in it. think i finally have the stupid diff -- label-name change from earlier made it into shard-keeper, never got picked up by rollup-service, so all the symptom alerts went off and none of the cause ones did. INC-2023-04-18-001 already has my name all over it i guess. haven't paged Hema yet. if this rollback behaves maybe 20 min, but the path is fragile and i do not want to widen this at 11:45 for no reason. looks like sev2, probably lands around ~5h45m by the time it's actually clean. impaired ingestion basically the whole window. i don't think we lost data. just... great.

11:55pm and Devika just said page Hema. i know. shard-keeper is still on fire and i'm trying to get this rollback to stick before i drag in more people. Apr 18 keeps getting worse. INC-2023-04-18-001, same stupid label-name change, and now i'm already thinking about the postmortem retrospective which is... not helpful

up. slept 4-9. sent Hema the one-liner and moved the migration meeting to thursday. April 18 outage was the shard-keeper config push from earlier -- Nadia's label-name change, rollup-service never picked it up, so the symptom stuff screamed and the cause alerts didn't. ingestion was messed up for hours but no data loss. gonna walk to the bagel place on 7th around 10 and i told Anya i'm not making brunch.

Cyrus got the full Apr 19 morning DM version. whole Apr 18 outage sequence, including the rollback being way shakier than it should've been. there was a network-layer red herring for a while, then finally the label-name diff find in shard-keeper from that earlier Nadia dashboard change. he just said fwiw i paged at the right hour, not the wrong one. i'll take it.

Move every non-critical meeting left this week to next week. Keep the Hema 1:1 and on-call sync. Drop the rest.

he texted Anya just now. not making brunch sunday, bad week at work, talk soon. didn't get into it. probably better that way if she's finally getting settled over there.

Devika came in from nights while I was still half looking at logs. did the "you ok" / "not really. will be" thing at the door. she just nodded and went to sleep. she had scrubs on again. rough little handoff.

coffee guy handed me the large drip without asking. guess that's the order now. line was weirdly quiet, just grinder noise and one milk pitcher getting knocked around. cold out but not enough to matter.

walked to the bagel place on 7th at like 3 and got the large drip again. not helping the sleep thing probably. place was kind of dead, just the fridge humming and somebody dropping a whole stack of trays in back.

Started the postmortem doc. Just the skeleton for now: timeline, root cause, mitigations. Will fill the rest in tomorrow.

Devika got in right as I was closing out a thread. still had the badge clipped on, looked completely cooked. she did the quick "you alive" check, I said busy day, will catch you later, and she just nodded and went to sleep. Kibo followed her to the bedroom and then came back out.

Apr 21, mid-postmortem: section 2 on root cause is done. Came out to 12 paragraphs. Cyrus was in the doc while I was editing and just sent back "tight. ship it."

Decline the two non-essential meetings this afternoon. I'm finishing the postmortem doc -- keep Hema and the on-call sync, drop the rest.

she did the "are you alive" thing. told her yes. and no, she doesn't want the details. i'm fine. not lying exactly, just not gonna explain shard-keeper to someone who's trying to learn a new office and where the coffee is.

walked longer than i meant to. kind of cold once the sun dropped. saturday tomorrow which barely feels real. devika's in around 7 so i'm gonna try to be awake enough to hand her coffee and maybe do one useful thing before the day starts.

Anya did the follow-up text thing again. "ok, glad you're alive. eat. xx". i didn't answer for like an hour, then just sent "eating. love you". she's settling in over there and still somehow tracking whether i'm upright.

bagel place on 7th again. same order. guy at the counter looked at me for a sec and went "you ok?" and i said "fine, thanks" and that was kind of the whole interaction. place smelled like burnt coffee more than bagels today.

Skipped the cortado thing. skipped soccer too. walked down to the river and back and that was basically it. 7th was busy when I passed it, didn't even go in. kind of gray out, not raining, just enough wind off the water to make everything feel unfinished.