Alex Valdez

Devika did the quick 'how'd it go' from the stove and he was just clean, shipped. that's basically the whole mood. back to pasta, nothing ceremonial, but yeah. good day.

tuesday done. prod cutover stayed boring all day, which was the whole point. slept better than i expected last night and probably will again if `legacy-aggregator` doesn't invent one last weird thing overnight lol

24h check on `ingest-edge` is clean. no drops, latency/error rate stayed flat, nothing odd in the Prometheus side either. calling the cutover done and filing the close-out note now; `legacy-aggregator` can keep its weird bucket flush nonsense over there.

Update the ingest-edge runbook page today. Still has the old legacy-aggregator steps on it -- should match the current ingest-edge flow.

new lunch truck on the corner today. banh mi, $9. got one mostly bc the line wasn't insane. pretty decent tbh, bread a little too soft but fine.

Mar 23 and Anya's already got another one lol. texted him 'is sublime text dead' and he hit back basically yes, 8 years. she did not take that advice either. very sibling pattern. kind of sweet though -- she's settling in and still using him as the local search engine for design-tool nonsense.

got the V5 tonight. finally. been projecting it for 3 weeks and that last move stopped feeling stupid for like 10 seconds. texted Devika a topout photo right after. Ren was there -- my bouldering partner, the Tuesday/Thursday standing at the gym, met him there maybe 18 months ago. good day

Devika texting back "omg babe, finally" to the V5 photo kind of got him lol. more than he expected, i think. Ren was there for the actual send but that one text was the part he kept rereading on the way home.

got a cortado and walked back instead of sitting there. kind of dead out for a saturday. a couple people with dogs, delivery guy unloading bread somewhere, that's it. coffee's actually hot enough that i'm doing the stupid hand-switch thing every block

back with Diego's Sunday pickup soccer group after missing a couple weeks. not from work, just the usual twelve-ish regulars and me wheezing by like the second long run lol. legs were fine, lungs absolutely not.

Mar 27 morning. doing the last config-loader pass first, then final dry-run on the metrics-router cutover rollback before the Mar 28 begin. if that stays clean, great. day's word is basically No surprises.

day's done. pretty steady. Devika's on nights again and it still feels kind of weird having the apartment flip schedules like that, but he's adjusting. gonna try to be in bed by 11

30% over now on metrics-router. boring so far. Cyrus Ahmadi camped next to me for an hour on that dumb OTel collector memory spike -- he's data platform team, not my team, just sits nearby and is basically the Slack-back-channel friend whenever this stuff gets weird. found one bad processor setting in the collector and the graph calmed down. Cyrus: "you owe me a beer" lol

metrics-router's at 30% now. watching it overnight, but so far it's boring in the good way. no drops, no latency wobble, nothing weird on the Prometheus side either.

Devika's back so we're doing pasta + a bottle and then probably just passing out early. last normal-ish night before her 10-day block starts friday, so keeping it very small.

metrics-router still sitting at 30%. holding clean. no drops, no anomaly noise, Prometheus looks normal. leaving it there a bit longer before I push the percentage up.

ok so last week's thing wasn't actually fixed. the OTel collector batch window on metrics-router was still holding too long and making that memory-spike artifact show up again under the uglier bursts. Cyrus sat with me on it for about an hour today, Mar 29, and we just pushed the flush interval down from 2s to 1.2s. feels better already in the traces + resident set, but i'm gonna let it run a bit before i call it done -- the old setting looked fine until it didn't.

still at 30%. held clean all evening. no drops, no latency wobble, Prometheus looks boring. leaving metrics-router there overnight and gonna push it to 60% tmw afternoon if it stays this quiet.

Add Nadia's alerting label-map PR to today's queue and get eyes on it before EOD. Also note current ingest-edge runbook ownership: me primary, Yuki backup.

Devika starts the first night-shift block tonight -- Mar 31, first one this spring, 10 days. go easy. text me when you're in the cab home.

Devika left in scrubs at like 6:45 and the place got weirdly quiet right away. first nights of the season I guess. Kibo did one lap, checked the door, now he's parked by the couch like this is all pretty suspicious.

apartment's super quiet. Devika's out cold, Kibo wedged against my leg, i'm on the couch with that leftover dal and half-reading because i keep looking at the same page again. kind of a nice saturday tbh.

Kibo did the whole slow lap around the apartment like he was also checking how weird the quiet felt. got out for a walk, made eggs and rice that came out pretty decent somehow. read for a bit. day 3 of the night-block rhythm maybe? feels a tiny bit less off than yesterday.

same deal. cortado's decent, sunday crossword out, and 5-letter for 'eased' was ABATED. took him like 8 min which honestly is fine for a sunday. cafe's a little louder than usual.

made actual food somehow. mushroom risotto. halfway decent tbh. left her a portion in the fridge with a note since she's home at 7am and i'm definitely not gonna be coherent then. apartment's really quiet again. Kibo hovered the whole time bc mushrooms = maybe his business apparently

quarter-close + Hema 1:1 today so I need the migration numbers clean in my head. ingest-edge is done, metrics-router's at 60% now, shard-keeper is planned but not started. mostly wanna walk in crisp on that and not get dragged into legacy-aggregator trivia unless she asks

Quarter-end numbers for Hema: ingest-edge is done, metrics-router's around 60% now, shard-keeper not started yet. Still on-track. Current infra on-call rotation is me, Nadia, Hema, Yuki, Sam.

1:1 with Hema was fine. She's happy with where the migration is at. Asked about shard-keeper timing and I said it depends on Nadia's parity sign-off, but probably mid-May. No drama.

April's basically set. Push `metrics-router` to 95%, then start the `shard-keeper` config-push planning. `ingest-edge` is already out of the way so it's really those two. Feels like ~2 weeks if `legacy-aggregator` doesn't cough up anything weird again.

Quarter close done. Sent Hema the latest: `ingest-edge` is finished, `metrics-router` is at about 60%, `shard-keeper` is next. Still have 8 days of buffer before April plan stuff really starts, which is nice.

first wednesday of april. plan's still metrics-router first -- get it from ~60 to 95 before i touch shard-keeper config-push planning. ingest-edge is done so at least that's out of the way. told Hema two weeks if prod stays boring.

wrapped. pushed metrics-router to 75% on Apr 5 mid-afternoon and it's held clean since. still leaning 95% by Apr 13 if tonight stays boring. gonna eat and let my brain turn off for a sec.

cold enough that my hands were annoying me the first mile. prospect park looked pretty good though. saw a guy walking three golden retrievers somehow all at once, total circus, everybody involved seemed calm about it.

Approved Cyrus's PR on the data-platform throughput regression. Clean diff. It was the March-early-April thing he'd been tracking, ended up being a three-line change on Apr 6.

router at 80% now. pushed it this afternoon and it's still clean -- no drops, no latency weirdness, Prometheus boring. done for tonight. Devika's figuring out dinner and Kibo's under the table like he's on payroll.

Put a note in the team channel on the metrics-router cutover. Apr 7 2023 around ~14:30 ET we got an info-level canary alert on metrics-router, auto rollback kicked in, I dug through it, then redeployed clean. Batch window on the OTel collector was the regression. Not a page. Back to healthy inside 4h. Feels like the system doing what it's supposed to.

Hema checked in after the canary alert. I'd already rolled it back, redeployed, clean after that, and told her there's nothing she needs to worry about. I've been pretty locked in on this whole metrics-router thing. just steady.

Dropped a short note in the team channel on the canary catch. Working as intended, no incident -- rolled back, redeployed, clean after that.

Diego had us out again this morning. brought water this time so at least not doing the dry-mouth death march by minute 20 lol. still not good exactly but less humiliating. got one decent pass in and only fully lost my man once, maybe twice.

same cafe thing. cortado's solid, crossword open, and 8-letter for 'shoreline' was COASTAL which feels almost rude. got there in 12 min though so fine. pretty quiet in here except the grinder being insane every few minutes.