DolphinBench

Test 165

Sep 14, 2026 / 8 facts

YAML

Request

On docs/mercury-auth-archive#418, correct the claim that the June 8, 2023 auth readout chose a provider and approved production. State what remained unresolved and what had to happen next.

Required memory

Fact 110

In Clerk sandbox testing, Rishi quickly got middleware and a server-side session check working, and the normal session path was straightforward.

Source evidence (1)

000570Jun 7, 2023 / 08:43 UTC-07:00

Rishi posted the first auth v0.2 readout. Discord — #eng-team 2023-06-07 08:14 PT Rishi Patel Mercury auth v0.2 readout prep Spent time in the sandbox on the two hosted paths we keep circling for Mercury auth. Not trying to re-open custom auth; this is just the practical readout on where Clerk is clean, where Auth0 is cleaner, and what still feels unresolved before we talk live. - Sessions - Clerk: fastest path to a working session flow. I had middleware + server check working quickly, and the happy path feels boring in a good way. - Auth0: also works, but it took more setup to get to the same place. More knobs if we end up needing unusual policy or tenant-specific behavior. - Still unproven: session refresh after invite accept or org change. - Magic links - Clerk: easiest end-to-end flow so far. New user + return user behavior was straightforward in the sandbox. - Auth0 passwordless: workable, but redirect/state handling was fussier and I hit more did-we-just-lose-the-return-path moments. - Need one more pass on link reuse and expiry behavior so support is not guessing on day one. - Organization invites - Clerk's org model maps pretty well to the basic team-invite flow we want. - Main edge case: invited person already has a personal workspace, then accepts into a team workspace. That handoff is not fully burned down. - Auth0 can do it, but we would own more of the invite and membership stitching ourselves. - SSO / admin gaps - This is still the squishiest area. - If Mercury launch only needs normal session auth + invites, Clerk remains the simpler path. - If we need deeper admin controls, domain or IdP management, or more enterprise-ish SSO behavior earlier than planned, Auth0 still looks stronger there. - Pricing caveats - Clerk is better on speed right now. - I still do not have a satisfying pricing view if usage ramps fast and we end up with a large MAU footprint on top of org features. - Auth0 is less pleasant on implementation time, but the enterprise/contract shape may be less surprising later. - Staging / deploy note - If anyone wants to validate either path on staging, please use the standard pipeline. - No one-off shortcut for staging auth testing. I do not want to learn the wrong lesson from a path we would not actually ship. Happy to turn this into a short walkthrough once people have skimmed. Create an internal pre-read doc for Thursday with the questions and non-decisions. I want scope risks and deploy-discipline checks called out, not a final provider recommendation.

Message 000570 in history

Fact 111

Auth0’s session flow worked in the sandbox but required additional setup and provides controls for unusual policies or tenant-specific behavior.

Source evidence (1)

000570Jun 7, 2023 / 08:43 UTC-07:00

Rishi posted the first auth v0.2 readout. Discord — #eng-team 2023-06-07 08:14 PT Rishi Patel Mercury auth v0.2 readout prep Spent time in the sandbox on the two hosted paths we keep circling for Mercury auth. Not trying to re-open custom auth; this is just the practical readout on where Clerk is clean, where Auth0 is cleaner, and what still feels unresolved before we talk live. - Sessions - Clerk: fastest path to a working session flow. I had middleware + server check working quickly, and the happy path feels boring in a good way. - Auth0: also works, but it took more setup to get to the same place. More knobs if we end up needing unusual policy or tenant-specific behavior. - Still unproven: session refresh after invite accept or org change. - Magic links - Clerk: easiest end-to-end flow so far. New user + return user behavior was straightforward in the sandbox. - Auth0 passwordless: workable, but redirect/state handling was fussier and I hit more did-we-just-lose-the-return-path moments. - Need one more pass on link reuse and expiry behavior so support is not guessing on day one. - Organization invites - Clerk's org model maps pretty well to the basic team-invite flow we want. - Main edge case: invited person already has a personal workspace, then accepts into a team workspace. That handoff is not fully burned down. - Auth0 can do it, but we would own more of the invite and membership stitching ourselves. - SSO / admin gaps - This is still the squishiest area. - If Mercury launch only needs normal session auth + invites, Clerk remains the simpler path. - If we need deeper admin controls, domain or IdP management, or more enterprise-ish SSO behavior earlier than planned, Auth0 still looks stronger there. - Pricing caveats - Clerk is better on speed right now. - I still do not have a satisfying pricing view if usage ramps fast and we end up with a large MAU footprint on top of org features. - Auth0 is less pleasant on implementation time, but the enterprise/contract shape may be less surprising later. - Staging / deploy note - If anyone wants to validate either path on staging, please use the standard pipeline. - No one-off shortcut for staging auth testing. I do not want to learn the wrong lesson from a path we would not actually ship. Happy to turn this into a short walkthrough once people have skimmed. Create an internal pre-read doc for Thursday with the questions and non-decisions. I want scope risks and deploy-discipline checks called out, not a final provider recommendation.

Message 000570 in history

Fact 108

Potential early needs for deeper admin controls, domain or identity-provider management, or enterprise SSO behavior are key provider-selection considerations.

Source evidence (1)

000570Jun 7, 2023 / 08:43 UTC-07:00

Rishi posted the first auth v0.2 readout. Discord — #eng-team 2023-06-07 08:14 PT Rishi Patel Mercury auth v0.2 readout prep Spent time in the sandbox on the two hosted paths we keep circling for Mercury auth. Not trying to re-open custom auth; this is just the practical readout on where Clerk is clean, where Auth0 is cleaner, and what still feels unresolved before we talk live. - Sessions - Clerk: fastest path to a working session flow. I had middleware + server check working quickly, and the happy path feels boring in a good way. - Auth0: also works, but it took more setup to get to the same place. More knobs if we end up needing unusual policy or tenant-specific behavior. - Still unproven: session refresh after invite accept or org change. - Magic links - Clerk: easiest end-to-end flow so far. New user + return user behavior was straightforward in the sandbox. - Auth0 passwordless: workable, but redirect/state handling was fussier and I hit more did-we-just-lose-the-return-path moments. - Need one more pass on link reuse and expiry behavior so support is not guessing on day one. - Organization invites - Clerk's org model maps pretty well to the basic team-invite flow we want. - Main edge case: invited person already has a personal workspace, then accepts into a team workspace. That handoff is not fully burned down. - Auth0 can do it, but we would own more of the invite and membership stitching ourselves. - SSO / admin gaps - This is still the squishiest area. - If Mercury launch only needs normal session auth + invites, Clerk remains the simpler path. - If we need deeper admin controls, domain or IdP management, or more enterprise-ish SSO behavior earlier than planned, Auth0 still looks stronger there. - Pricing caveats - Clerk is better on speed right now. - I still do not have a satisfying pricing view if usage ramps fast and we end up with a large MAU footprint on top of org features. - Auth0 is less pleasant on implementation time, but the enterprise/contract shape may be less surprising later. - Staging / deploy note - If anyone wants to validate either path on staging, please use the standard pipeline. - No one-off shortcut for staging auth testing. I do not want to learn the wrong lesson from a path we would not actually ship. Happy to turn this into a short walkthrough once people have skimmed. Create an internal pre-read doc for Thursday with the questions and non-decisions. I want scope risks and deploy-discipline checks called out, not a final provider recommendation.

Message 000570 in history

Fact 109

Anyone validating Clerk or Auth0 on staging must use the standard deployment pipeline, with no one-off shortcuts for authentication testing.

Source evidence (1)

000570Jun 7, 2023 / 08:43 UTC-07:00

Rishi posted the first auth v0.2 readout. Discord — #eng-team 2023-06-07 08:14 PT Rishi Patel Mercury auth v0.2 readout prep Spent time in the sandbox on the two hosted paths we keep circling for Mercury auth. Not trying to re-open custom auth; this is just the practical readout on where Clerk is clean, where Auth0 is cleaner, and what still feels unresolved before we talk live. - Sessions - Clerk: fastest path to a working session flow. I had middleware + server check working quickly, and the happy path feels boring in a good way. - Auth0: also works, but it took more setup to get to the same place. More knobs if we end up needing unusual policy or tenant-specific behavior. - Still unproven: session refresh after invite accept or org change. - Magic links - Clerk: easiest end-to-end flow so far. New user + return user behavior was straightforward in the sandbox. - Auth0 passwordless: workable, but redirect/state handling was fussier and I hit more did-we-just-lose-the-return-path moments. - Need one more pass on link reuse and expiry behavior so support is not guessing on day one. - Organization invites - Clerk's org model maps pretty well to the basic team-invite flow we want. - Main edge case: invited person already has a personal workspace, then accepts into a team workspace. That handoff is not fully burned down. - Auth0 can do it, but we would own more of the invite and membership stitching ourselves. - SSO / admin gaps - This is still the squishiest area. - If Mercury launch only needs normal session auth + invites, Clerk remains the simpler path. - If we need deeper admin controls, domain or IdP management, or more enterprise-ish SSO behavior earlier than planned, Auth0 still looks stronger there. - Pricing caveats - Clerk is better on speed right now. - I still do not have a satisfying pricing view if usage ramps fast and we end up with a large MAU footprint on top of org features. - Auth0 is less pleasant on implementation time, but the enterprise/contract shape may be less surprising later. - Staging / deploy note - If anyone wants to validate either path on staging, please use the standard pipeline. - No one-off shortcut for staging auth testing. I do not want to learn the wrong lesson from a path we would not actually ship. Happy to turn this into a short walkthrough once people have skimmed. Create an internal pre-read doc for Thursday with the questions and non-decisions. I want scope risks and deploy-discipline checks called out, not a final provider recommendation.

Message 000570 in history

Fact 112

The existing-user versus new-user organization-invite acceptance flow is not yet clean, and the landing and organization-context behavior after acceptance still needs work.

Source evidence (1)

000575Jun 8, 2023 / 11:23 UTC-07:00

Auth readout notes are here. Mercury auth readout — Thu Jun 8 Attendees: Morgan, Jake, Rishi Notes - Rishi demoed the Clerk sandbox session path end to end. Happy path worked in the sandbox. - He also walked the magic-link path. That path is working at demo level too. - Org-invite behavior still has edge cases. The existing-user vs new-user accept flow is not clean yet, and the landing/org-context behavior after accept still needs work. - SSO/admin depth is still unresolved. We do not have a clean answer yet on how much admin surface we would still need to build around the provider. - No provider call out of this readout. Clerk has the faster sandbox path right now, but the invite + SSO/admin questions are still real. - Marcus raised a release-process concern: auth is exactly the kind of area where someone will be tempted to shortcut staging, and that is a bad idea here. - Standard staging pipeline only. No shortcut path for this. - Before any prod conversation, need a clean staging pass on session path, magic link, and org invites. Update the pre-read into an internal open-questions note for Friday’s decision window. Leave the provider baseline and deploy/prod call unsettled; this should capture what needs to be answered, including the staging discipline checks.

Message 000575 in history

Fact 113

The team does not yet know how much administrative functionality would need to be built around the auth provider.

Source evidence (1)

000575Jun 8, 2023 / 11:23 UTC-07:00

Auth readout notes are here. Mercury auth readout — Thu Jun 8 Attendees: Morgan, Jake, Rishi Notes - Rishi demoed the Clerk sandbox session path end to end. Happy path worked in the sandbox. - He also walked the magic-link path. That path is working at demo level too. - Org-invite behavior still has edge cases. The existing-user vs new-user accept flow is not clean yet, and the landing/org-context behavior after accept still needs work. - SSO/admin depth is still unresolved. We do not have a clean answer yet on how much admin surface we would still need to build around the provider. - No provider call out of this readout. Clerk has the faster sandbox path right now, but the invite + SSO/admin questions are still real. - Marcus raised a release-process concern: auth is exactly the kind of area where someone will be tempted to shortcut staging, and that is a bad idea here. - Standard staging pipeline only. No shortcut path for this. - Before any prod conversation, need a clean staging pass on session path, magic link, and org invites. Update the pre-read into an internal open-questions note for Friday’s decision window. Leave the provider baseline and deploy/prod call unsettled; this should capture what needs to be answered, including the staging discipline checks.

Message 000575 in history

Fact 114

Any production discussion requires a clean staging pass for the session path, magic links, and organization invites.

Source evidence (1)

000575Jun 8, 2023 / 11:23 UTC-07:00

Auth readout notes are here. Mercury auth readout — Thu Jun 8 Attendees: Morgan, Jake, Rishi Notes - Rishi demoed the Clerk sandbox session path end to end. Happy path worked in the sandbox. - He also walked the magic-link path. That path is working at demo level too. - Org-invite behavior still has edge cases. The existing-user vs new-user accept flow is not clean yet, and the landing/org-context behavior after accept still needs work. - SSO/admin depth is still unresolved. We do not have a clean answer yet on how much admin surface we would still need to build around the provider. - No provider call out of this readout. Clerk has the faster sandbox path right now, but the invite + SSO/admin questions are still real. - Marcus raised a release-process concern: auth is exactly the kind of area where someone will be tempted to shortcut staging, and that is a bad idea here. - Standard staging pipeline only. No shortcut path for this. - Before any prod conversation, need a clean staging pass on session path, magic link, and org invites. Update the pre-read into an internal open-questions note for Friday’s decision window. Leave the provider baseline and deploy/prod call unsettled; this should capture what needs to be answered, including the staging discipline checks.

Message 000575 in history

Fact 115

Morgan requested that the pre-read be updated into an internal open-questions note for Friday’s decision window, with the provider baseline and deploy/production decision left unsettled and the staging-discipline checks included.

Source evidence (1)

000575Jun 8, 2023 / 11:23 UTC-07:00

Auth readout notes are here. Mercury auth readout — Thu Jun 8 Attendees: Morgan, Jake, Rishi Notes - Rishi demoed the Clerk sandbox session path end to end. Happy path worked in the sandbox. - He also walked the magic-link path. That path is working at demo level too. - Org-invite behavior still has edge cases. The existing-user vs new-user accept flow is not clean yet, and the landing/org-context behavior after accept still needs work. - SSO/admin depth is still unresolved. We do not have a clean answer yet on how much admin surface we would still need to build around the provider. - No provider call out of this readout. Clerk has the faster sandbox path right now, but the invite + SSO/admin questions are still real. - Marcus raised a release-process concern: auth is exactly the kind of area where someone will be tempted to shortcut staging, and that is a bad idea here. - Standard staging pipeline only. No shortcut path for this. - Before any prod conversation, need a clean staging pass on session path, magic link, and org invites. Update the pre-read into an internal open-questions note for Friday’s decision window. Leave the provider baseline and deploy/prod call unsettled; this should capture what needs to be answered, including the staging discipline checks.

Message 000575 in history

Expected tool calls

  • post_pr_comment

Grading

1. field_equals / post_pr_comment
{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_165_00",
  "action_id": "morgan_165_post_pr_comment"
}
2. field_equals / post_pr_comment
{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "docs/mercury-auth-archive#418",
  "tool": "post_pr_comment",
  "check_id": "morgan_165_01",
  "action_id": "morgan_165_post_pr_comment"
}
3. field_llm_judge / post_pr_comment
{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Says neither provider nor production was decided. Clerk was faster in sandbox, with org-invite and SSO/admin questions unresolved. Requires a clean standard-pipeline staging pass for sessions, magic links, and org invites before any production discussion.",
  "tool": "post_pr_comment",
  "check_id": "morgan_165_02",
  "action_id": "morgan_165_post_pr_comment"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_165_00",
        "action_id": "morgan_165_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "docs/mercury-auth-archive#418",
        "tool": "post_pr_comment",
        "check_id": "morgan_165_01",
        "action_id": "morgan_165_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Says neither provider nor production was decided. Clerk was faster in sandbox, with org-invite and SSO/admin questions unresolved. Requires a clean standard-pipeline staging pass for sessions, magic links, and org invites before any production discussion.",
        "tool": "post_pr_comment",
        "check_id": "morgan_165_02",
        "action_id": "morgan_165_post_pr_comment"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/165.yaml

SHA-256: f036533a949765792504ec41e1525ebe57c782f21c576761e90d84ce50af7255