Alex Valdez

Apr 24 postmortem doc went up. Nadia's margin comment is basically why didn't you page Hema earlier, and what's the "20 min from fix" grounded in. I read it 3 times, didn't answer in the doc. going to take it to Hema in the 1:1 instead. feels colder now.

Cyrus pinged after he saw the comment. just "you good?" I said "fine" and Cyrus came back with "that means not, but ok". that's probably the most anyone's said directly all day. I still didn't bite.

Devika's still on nights so it's just me here. read that postmortem comment again. then again. fourth time maybe. closed the laptop after that. apartment's quiet except the fridge doing its thing.

Devika's still on nights, so I did dumb frozen dumplings in front of the tv. didn't touch the Nadia thing except in my head, which was probably the right call. Kibo wedged against me like that fixed anything. kind of did.

ok so this morning is mostly the postmortem doc again. need to fold in Hema's 1:1 notes and clean up the parts that still read defensive. then get the Friday incident review prep into something not embarrassing. trying to get back to normal pace a little, not fully there yet.

1:1 on Apr 25 with Hema was actually useful. she asked straight about Nadia's margin comment, I gave my side without getting cute about it. her read was basically: on the Apr 18 outage, the escalation timing critique is fair, but the technical call itself was the right one. which... yeah. lands fair to me too. procedural miss, not 'you broke judgment'. matters bc those get conflated fast once the postmortem starts hardening.

bagel place guy just started on the large drip when i walked in. didn't ask. still. 8 days now i think. toasted everything smell in there and that burnt coffee air that gets in your hoodie.

Move the Hema 1:1 on Friday mornings 30 min later starting late April. Keep it in the post-outage window.

day's done. hema 1:1 helped more than i expected. got the split clearer in my head between the technical call being right and the part where i waited too long to escalate. doc still needs one more pass, i'll revise it tomorrow. heading home.

last one tonight. see you in the morning -- i'll be up for the handoff if you want a 30 sec hello. Kibo's already parked by the door like he knows the drill.

she got in and that was kind of it. two long hugs at the door, no words, just... yeah. she's been dead asleep since. still out at 3. Kibo did one lap around the bed and gave up.

Apr 27 is basically postmortem doc revision again. folded in Hema's 1:1 input, tightened the timeline section so it's not hand-wavy around the shard-keeper push / downstream consumer gap, and I widened mitigations past the immediate fix. more explicit on the rules she was hinting at -- review gates + label-name parity checks before config rolls. not trying to turn it into policy fanfic, just enough that when this lands Apr 28 it doesn't read like 'be more careful'

Devika's off the night block. back on a normal schedule for a couple weeks, finally. they got actual dinner together tonight for the first time in like two weeks. nothing fancy, just sat there and ate while Kibo kept doing laps under the table lol

mostly the 4pm review today. doing one last pass on the doc this morning -- timeline/ownership wording, then prep notes for the room. theo's joining which is... not normal, so i want the read-through clean and not get pulled into the nadia side-thread unless it actually comes up.

4pm incident review meeting on the Apr 18 outage is done. Hema, Theo, me, Nadia, Cyrus, Sam. main changes they actually wrote down at Sphere: no Friday-afternoon prod deploys -- team-wide now, not just me being weird about Fridays. plus page secondary within 45 min if the primary is still in it, and a runbook ownership audit with Nadia cross-reviewing the service bits she touches. shard-keeper migration is paused pending that audit. doc felt tighter than yesterday at least. nobody tried to relitigate the rollback call for an hour, which helped lol

Apr 28 incident review's done. Landed three rules: no-Friday-afternoon-deploys, page-secondary-within-45-minutes-if-not-isolated, and a runbook-ownership audit in May. Migration paused pending the audit. Cyrus DM'd after, "rules feel right. you good?" Alex: "tired. yeah. mostly tired."

she's back on a normal block now so we actually ate at the same time. weirdly nice. I said one line about the review -- basically that it happened and wasn't a disaster. she said ok and didn't poke at it. Kibo under the table the whole time waiting for something to drop.

went to the cortado place and got the drip. counter person did that little head tilt like i'd said the wrong line. guy behind me just shrugged. saturday but not really saturday.

did a long loop through Prospect Park. really slow. first day in a week where the air actually felt normal on my face, not cold exactly, not damp either. some kid was half asleep in a stroller yelling about a pretzel, couple dogs losing their minds near the Long Meadow, and for once i didn't feel rushed walking past any of it.

still drip. sunday so no line, just that coffee smell and somebody rinsing metal pitchers in back. devika's home, soup's been going for a while. apartment's quiet in a good way. kibo did one lap around the stove and decided it wasn't for him.

Anya called. actual Apr 30 phone call, not text, first one in months. I gave her the Apr 18 outage as basically "bad night at work, i'm fine" and left it there. Twelve-minute call. she said you sound flat. I just said yeah. idk. probably enough for now.

back in office. gonna keep today smaller if i can -- catch up with devon, clear the obvious inbox stuff, then one real block for the doc instead of pretending i'll do six things at once. pacing is the whole game today.

Dinner with Devika finally happened. quiet, pretty good. first time he said more than "bad night" about the Apr 18 mess -- not every detail, just enough that you could tell he's been carrying the whole thing around by himself. her night block ended on the 22nd so I think this was the first actual opening. when she asked how he was he went "tired. yeah. mostly tired." felt exact.

Book rooms for the audit kickoff on May 8 and the review on May 22. Biggest ones you can get, 90 min each -- don't leave this to the day before.

Pulled the list for the runbook audit prep. On May 2 it's basically three he directly owns -- metrics-router, ingest-edge, shard-keeper -- plus two weird history ones he kinda owns: legacy-aggregator since nobody's fully finished the migration off it, and that old metric-rollup thing that's basically unowned in practice.

on the technical-lessons section for the Apr 18 outage, i think the cleanest shape is just name the three things and be pretty literal. `shard-keeper rollback path` was way too easy to make worse under pressure -- looked reversible on paper, wasn't once state had partially moved. then the ownership mess: `rollup-service` was basically un-owned in practice, so everyone had a partial map and nobody had the whole downstream story. and the `label-name diff` part needs to be explicit because symptom alerts did not point at that at all. from the pages it looked like generic ingest weirdness, not "one label changed and one consumer never got the memo." on May 3 i'd rather write that plainly than over-explain incident mechanics.

End of month is mostly hold pattern. Migration's still paused. Runbook audit starts Monday, and Wes is joining the rotation in two weeks so they're folding him into the review now. Pace seems steadier than last week, not fast exactly, just less sideways.

April done. worst month of the year by a mile. postmortem's closed, migration still parked, and we left the three rules in place for now. walk home felt weirdly long tonight. Devika had already started dinner when I got in, Kibo doing laps like nothing happened.

took the long way back for no real reason. city was weirdly quiet. devika was already out when i got in, made tea, lights mostly off now. bed soon.

May 5. coming in cleaner than i've been in a while. Slept 7 hours, did 45 min in Prospect Park -- first clean-headed run since the outage. today is mostly getting the runbook audit shape right before monday, esp ownership rules across `metrics-router`, `ingest-edge`, `shard-keeper`. want no fuzzy edges going in.

walked home in the rain. nothing dramatic, just that steady annoying kind where your sleeves get wet. devika's home, block felt normal for once. i'm gonna eat something and be quiet for a minute.

brunch with Anya on May 6 was... normal, weirdly. first brunch since the outage that felt normal. she didn't poke at the outage, I didn't volunteer anything. mostly heard about this dog-walking idea she has for maybe 10 minutes, then onto other stuff. i'll take it.

Book coffee with Theo when his calendar opens. Need to finally do the Hema manager catch-up from January.

got a cortado on May 7. first one since Apr 20 i think. felt totally normal, which was funny after the large drip becoming a whole thing. about 2.5 weeks of drip total, apparently.

audit starts this morning. hema's running it. going in with the two lists clean: what i primary-own -- `metrics-router`, `ingest-edge`, `shard-keeper` -- and every service that touches those so we don't spend 40 min arguing edges in the room. mostly want the ownership pass to stay mechanical.

Runbook-ownership audit starts today, May 8. Hema's running it. Goal is every service has a primary owner and a backup. I already flagged shard-keeper's runbook as the worst one in the set -- 18 months stale, still missing the newer label-routing stuff.

Audit kicked off this morning. Hema's running it pretty clean -- one-pager per service, primary + backup. I picked up three on my side: metrics-router, ingest-edge, shard-keeper.

ok so May 9 turned into shard-keeper runbook rewrite scope basically all afternoon. six sections need a full pass. there's like eighteen months of drift in there since the new label-routing logic shipped, and you can feel it because half the steps still assume the old decision points and the screenshots/queries don't line up with what on-call would actually see now. biggest chunk of the audit work by far. i'm leaning split it into operator path vs owner path first, then patch the backup-owner bits after, otherwise the primary/backup rules Hema wants are gonna get smeared across every section and it'll stay unreadable.

first pass on Nadia's PR comments this morning. gonna answer the actual code/doc points, skip the tone stuff, get it merged. if she wants a side argument on wording nvm, not doing that before coffee.

first one under the new cross-coverage thing went about how you'd expect. Nadia left more nits than she used to. some real, some... whatever. I just answered the code/doc points line by line and didn't touch the snippy parts. on ingest-edge, not router. getting it merged without making a second problem -- that's basically the whole move rn