Morgan Chen

For AP backfill: before the Acme mess we were using $2k as the review bar. Anything under that just got processed, $2k+ needed a second look/log. putting that here so the old runs line up with what finance sees.

Merging the Mar 15 Wednesday thing in a bit. ~4:50pm if nothing weird comes up. just the ten-line deploy on the release schedule, pretty routine.

Pagers just went off. prod, not staging. seeing a spike on API 500s and a bunch of customer jobs stuck in queue at the same time, which is a gross combo on a Wednesday at 5:30. Devon's digging through logs now. I'm pulling people back in even though half the team was already mentally out. if this is tied to the late deploy I'm gonna lose it -- we've been way too casual about late pushes. No clean read yet. bad enough that I told Jamie I'd be late. maybe very late.

Mar 15, 9pm, still in the outage. trying to write this down clean while they're heads down: repro is request comes in on the new path, cache miss, falls through to the old entitlement check, then retries against the worker pool and piles up there. the ten-line change looked harmless bc diff was tiny, but the asymmetric blast radius is real -- only some tenants hit the bad branch, and when they do it fans out 1 request into like 6-8 internal calls. so dashboards looked half-normal for too long. Devon is on traces, Priya rolled traffic back from the edge flag but we still have stragglers bc jobs already queued. note for the retro: no more shipping branchy auth/entitlement stuff behind a flag without a replay against prod-like traffic. tiny diff, huge mess.

still out but like 15 min. team's got the overnight stuff covered. i'm done. phone at 4%, brain about the same. gonna get in, let Kibo do his whole patrol thing, then basically disappear

still mopping up. service is back but i'm not calling it clean yet -- a couple ugly edges from last night and we're being annoying about deploys today on purpose. fingers crossed this is fully boring again by noon.

Mar 16 midday and we're fully recovered. Mar 15 outage ended up 18 hours end-to-end. not doing a big speech about it but the team really carried this one -- sharp all the way through, even when it got gross. nailed it.

Writing up my own notes on last night while it's still fresh. Ended up 18 hours end to end, service is back and the ugly edges are gone. Tomorrow's retro is gonna be mostly deploy discipline stuff -- what got merged when, who had prod access, and why we let the rollback path get fuzzy.

done. couch. not doing anything else today. Kibo's wedged into my side like he pays rent and I'm too tired to argue with it. phone at 8%, brain lower.

Q1 retro today. gonna take the deploy-process stuff cleanly and just ticket it out. no defending, no long speech. retro at 11, then time with Devon after to turn the obvious ones into actual owners/dates -- if PR checks and rollback rules need to get tighter, fine, we do that.

Confirm Sunday at the spot Jamie likes, and grab flowers Saturday morning.

Mar 17 Q1 retro done. pretty clear the release process was the single biggest complaint, and yeah, fair -- too manual. owned that one, opened the ticket while we were still in the room so nobody has to hear me do a speech about deploy discipline again. leaning small boring fixes first: cut handoffs, one path to prod, tighten the checklist, fewer human toggles. don't want a giant CI/CD project right now if the actual pain is three dumb steps and a Slack ping.

Q1 retro was pretty consistent on the complaint ranking. #1 was the release process being too manual -- biggest pain point this quarter by a decent margin. Then on-call rotation imbalance, then the Linear vs GitHub flow friction. Nothing especially surprising, just finally in one explicit order.

friday finally. rough one. home now and not doing one more thing tonight -- jamie's back for dinner, kibo already posted up in the kitchen. maya gets in next week so that's nice

keep circling the deploy thing. not just "be more careful" -- that's useless. there's a real rule in here somewhere about when we let ourselves stack risk, and who gets to say no, and what has to be true before friday starts looking tempting. not writing it yet. wanna see if it still feels true on monday. kibo is asleep under the table and i'm staring at a half-cold coffee lol

Pick up flowers on the way home. Usual place.

Dolores was completely cooked today. blankets everywhere, shirtless guys out way too early, whole city acting like we made it. actually warm tho

just swung by for my laptop and the espresso machine was doing its little jet-engine thing. couple shots came out, nobody made a face, nobody did the burnt-puddle-water speech. honestly... progress lol

Four years today. made coffee and brought it back to bed, Jamie's still asleep, whole apartment weirdly quiet except Kibo doing the little floor-tap thing in the hall. nice morning. that's it

Confirm the 7:30 reservation. No changes.

half dressed and Jamie's in the kitchen absolutely butchering the chorus to something from like 2009. keeps committing harder every time she misses it. Kibo's posted up by the fridge like he's part of the set

dinner was really good. Jamie gave me a card and i had to do that thing where you look at the menu for a sec so you don't fully lose it lol. four years, huh. back now. she's on the couch reading, Kibo glued to her legs again

monday. pretty full but clean. product block most of today, devon midday, then a couple team 1:1s i kept pushing. maya gets in tomorrow so trying not to let the week sprawl before that. good busy, not dumb busy.

Book Maya an Uber from SFO tomorrow around 4 to the apartment. Door code's in the usual place.

opened Devon's retention draft and yeah, it's a lot. not bad, just dense enough that i'm gonna half-read it if I do it now. parking it till after Maya heads out, then I'll go through with the accounting cuts open too -- Anna Martinez apparently pulled a bunch of the raw stuff already so at least it's not starting from zero.

monday done. apartment's weirdly tidy for once. maya gets in tomorrow so i actually cleared the mail pile off the table lol jamie's home at a normal hour, kibo already posted up in the kitchen like he's coordinating dinner

tuesday. wrapping work by 3 if i can keep people from sneaking stuff onto the calendar. product block this morning, devon around lunch, then i need the desk basically cleared out before maya lands at 4.

Draft this to Greg at Acme: "Hi Greg -- wanted to flag that we've now missed two ETAs this month on the Acme side. I know stuff comes up, but it's starting to create churn for our team. Let's get 20 min on the calendar this week and reset on dates / owners so we're not doing this again."

Maya landed. saw her at SFO and we did the too-long airport hug thing lol. she's here for a couple days. gonna do pizza tonight and let her debrief NYC / make fun of my life choices a little. nice timing tbh

maya just informed me mom is on episode 37 of the podcast now. thirty seven. apparently she has "segments". we were standing in my kitchen laughing so hard kibo left

Wednesday. Greg at 11. keeping the morning pretty tight before that, then lighter after lunch so I can sort Maya stuff for next week. if anything new tries to land this afternoon, move it.

Just got off with Greg Shipman at Acme. Cordial. He owned the ETA misses pretty directly and said Q2's gonna be tighter on their side. I think I mostly buy it... like 60%, not 100. That's where I'm at.

jake pinged on Discord again. i said maybe next month, which is apparently where this has landed now. still not really buying it, just also not interested in having a whole opinionated Slack-vs-Discord thing at 3:40pm lol

took Maya to that bar in the Mission we always end up at when she's here. loud as hell, perfect. we mostly yelled over bad music and she cross-examined me about the company like billing 900/hr. really good though. missed her.

Maya's been doing a full closing argument against NYC winter for like 25 min. i'm agreeing in all the wrong places on purpose. "yes, the wind between buildings is personal" / "correct, the sidewalk slush is an HR issue." she's not wrong tbh

had coffee with Maya before her JFK flight and now she's in the Uber. good visit. she was doing lawyer-speed recaps of three separate dramas while stealing my breakfast, very on brand. apartment feels weirdly quiet already lol

Thursday. Maya's off this morning so gonna keep the first half a little loose, then back on normal after lunch. Product block with Devon if it holds, then the usual meeting pile.

still thinking on the friday deploy thing. not making it policy today. leaning toward a real cutoff next week unless somebody can show the rollback path is boring and fast, not theoretical. retro made it pretty obvious where the slop is -- too many handoffs, too much human checklist stuff, and then everyone gets brave at 4:40.

thursday done. home, windows open a little, kind of warm out finally. jamie's back at a normal hour and i'm probably doing thai tomorrow again lol. kibo in the kitchen acting like this was all his plan

friday and i'm cooked. doing Lemongrass again -- pad thai, not pretending i'm gonna make dinner tonight. jamie's finally off nights so at least we'll eat like normal people for once.