Testing standard 1.1

Five checks, one command, and a number for how far to trust them

A way to test web apps that agents build and nobody reads line by line. It plants bugs in its own code to find out whether a green run means anything.

gate probe --max 40 --seed 1sal-starter at 6aa609c
32of 40 planted bugs caught
  • Types 15
  • Rules 9
  • API 2
  • Flows 6
  • Got through 8

Each check answers a question the others can’t

  • Types

    Does the code agree with itself?

    Strict TypeScript with no any, and zod at every boundary: API input, env, forms, storage.

    bun run typecheck15 of the 40 planted bugs
    apps/web/server/trpc.ts
    export const protectedProcedure = t.procedure.meta({ access: 'user' }).use(({ ctx, next }) => {
      if (!ctx.session) {
        throw new TRPCError({ code: 'UNAUTHORIZED' })
      }
      return next({ ctx: { ...ctx, session: ctx.session } })
    })
    apps/web/server/demo.ts
    create: protectedProcedure
      .input(
        z.object({
          title: z.string().min(1).max(200),
          content: z.string().min(1).max(2000),
        })
      )
      .mutation(async ({ ctx, input }) => {
        return prisma.post.create({
          data: { ...input, authorId: ctx.session.user.id },
        })
      }),
    Take the ! out of the guard, one of the 40 bugs the planted-bug run tried, and the typecheck names every procedure that assumed a session:
    $ tsc --noEmit
    server/demo.ts(35,37): error TS18047: 'ctx.session' is possibly 'null'.
    server/router.ts(5,51): error TS18047: 'ctx.session' is possibly 'null'.
  • Rules

    Is the logic right where a wrong answer is silent?

    Pure functions with many cases: money, dates, permissions, parsers, redirects. No database, network or environment.

    *.rule.test.ts9 of the 40 planted bugs
    apps/web/server/access.rule.test.ts
    import { describe, expect, test } from 'bun:test'
    import { emailMatches } from './access'
    
    describe('emailMatches', () => {
      const entries = ['ada@example.com', '@acme.io']
    
      test('a whole domain, and only that domain', () => {
        expect(emailMatches('anyone@acme.io', entries)).toBe(true)
        expect(emailMatches('x@eng.acme.io', entries)).toBe(false)
        expect(emailMatches('x@notacme.io', entries)).toBe(false)
        expect(emailMatches('x@acme.io.evil.com', entries)).toBe(false)
      })
    
      test('the last @ decides the domain', () => {
        expect(emailMatches('"a@acme.io"@evil.com', entries)).toBe(false)
      })
    })
  • API

    Does every server procedure return, write and refuse what it should, for each kind of caller?

    Every tRPC procedure called through the real router on a real Postgres, plus a committed table of who may call what.

    *.api.test.ts2 of the 40 planted bugs
    apps/web/server/demo.api.test.ts
    import { describe, expect, test } from 'bun:test'
    import { asAnon, asUser } from '../test/callers'
    import { prisma } from './prisma'
    
    describe('posts.create', () => {
      test('writes a post owned by the caller and returns it', async () => {
        const person = await asUser()
        const created = await person.api.posts.create({ title: 'Hello', content: 'World' })
        const row = await prisma.post.findUniqueOrThrow({ where: { id: created.id } })
        expect(row).toMatchObject({ title: 'Hello', content: 'World', authorId: person.user.id })
      })
    
      test('rejects a signed-out caller and writes nothing', async () => {
        const before = await prisma.post.count()
        const anon = await asAnon()
        await expect(anon.api.posts.create({ title: 'nope', content: 'nope' })).rejects.toMatchObject({
          code: 'UNAUTHORIZED',
        })
        expect(await prisma.post.count()).toBe(before)
      })
    })
    apps/web/server/router.access.md
    | procedure    | kind     | signed out | signed in | admin |
    | ---          | ---      | ---        | ---       | ---   |
    | me           | query    | no         | yes       | yes   |
    | posts.create | mutation | no         | yes       | yes   |
    | posts.list   | query    | yes        | yes       | yes   |
  • Flows

    Can a person actually do it?

    A person’s path through the production build in a real browser, at desktop and phone width, with every screen walked by some flow.

    *.flow.spec.ts6 of the 40 planted bugs
    apps/web/e2e/auth.flow.spec.ts
    import { expect, test } from './flow'
    
    test('auth/sign-in-next · signing in returns to the page and query asked for', async ({
      page,
      world,
    }) => {
      // Check: sign-in from /dashboard?tab=posts ends on /dashboard?tab=posts.
      const person = await world.account('Linus')
      await page.goto('/dashboard?tab=posts')
      await page.waitForURL('**/sign-in?next=%2Fdashboard%3Ftab%3Dposts')
      await page.getByLabel('Email').fill(person.email)
      await page.getByLabel('Password').fill(person.password)
      await page.getByRole('button', { name: 'Sign in' }).click()
      await page.waitForURL((url) => url.pathname === '/dashboard' && url.search === '?tab=posts')
      await expect(page.getByRole('heading', { level: 1, name: 'Linus' })).toBeVisible()
    })
  • Live

    Is what deployed the thing that was tested, and is it up?

    After a deploy: the new revision is running, and the health path and listed paths answer. It warns and never rolls back.

    after a deploy
    apps/web/server/health.ts
    export function healthz(ping: () => Promise<unknown> = pingDatabase): RequestHandler {
      return async (_req, res) => {
        try {
          await ping()
          res.json({ status: 'ok', uptime: process.uptime(), commit: env.DRYDOCK_COMMIT ?? null })
        } catch (e) {
          log.error(`healthz db check failed: ${formatError(e)}`)
          res.status(503).json({ status: 'error', error: 'database unreachable' })
        }
      }
    }
    apps/web/e2e/health.flow.spec.ts
    import { expect, test } from './flow'
    import { FLOW_COMMIT } from './server'
    
    test('health/commit · /healthz reports the commit the server was deployed as', async ({
      request,
    }) => {
      // Check: the body holds `"commit":"<DRYDOCK_COMMIT>"`, the text Drydock's Live check looks for.
      const res = await request.get('/healthz')
      expect(res.status()).toBe(200)
      expect(await res.text()).toContain(`"commit":"${FLOW_COMMIT}"`)
    })
    The check's half is a few lines in the app's deploy manifest, written by the deploy tool when the check is turned on. {sha} stands for the commit that was just deployed, so an old version still serving is reported:
    liveCheck:        # after each deploy: report whether the new commit serves (never fails it)
      paths:
        - path: "/healthz"
          contains: "\"commit\":\"{sha}\""

One command runs them on a clean copy of the commit

  1. It copies the commit, not the working tree. Uncommitted and untracked files are left behind.
  2. It creates a fresh, migrated database for the run and hands the checks its address.
  3. It runs each check with an allowlisted environment, so a variable your shell exports can’t make one pass.
  4. It fails a check that skipped a test, marked one todo, or found none.
  5. It writes a signed receipt for the commit on a full pass.
$ gate
gate  sal-starter  c9c25a3  report  clean copy · 1 database
pass  setup    2.3s   1 database from the template cache
pass  types    1.4s
pass  rules    0.3s   53 tests
pass  api      3.2s   32 tests
pass  flows   18.0s   32 tests
receipt written · report mode never blocks a push
gate.json
{
  "version": 1,
  "mode": "report",
  "branches": ["main", "master"],
  "copy": { "env": [".env"], "keep": ["node_modules"] },
  "setup": ["bun install --frozen-lockfile", "bun run db:generate"],
  "databases": [
    {
      "env": "DATABASE_URL",
      "migrate": "bun run db:deploy",
      "key": ["apps/web/prisma/schema.prisma", "apps/web/prisma/migrations"]
    }
  ],
  "ports": [],
  "pass": [],
  "checks": [
    { "name": "types", "run": "bun run typecheck" },
    { "name": "rules", "run": "bun run test:rules" },
    { "name": "api", "run": "bun run test:api" },
    { "name": "flows", "run": "bun run test:flows", "timeout": 900 }
  ],
  "probe": {
    "include": ["apps/web/server.ts", "apps/web/server/**/*.ts", "apps/web/src/**/*.{ts,tsx}"],
    "exclude": [
      { "path": "apps/web/server/logger.ts", "why": "log colours and formatting" },
      { "path": "apps/web/**/*.test.ts", "why": "the tests themselves, not code under test" }
    ]
  }
}
It reports and never blocksNothing it finds stops a push or fails a deploy. An agent that pushes past red has to say so.
It can’t pass by accidentUncommitted files stay behind, the database is new, and a skipped test fails the run.
It stays out of gitNo hook, no git config. The push check lives in the agents’ own tooling, because agents are the ones pushing.

One more shows every test and how it last ran

$ gate tests
gate tests  sal-starter  c9c25a3
last run   2026-11-21 16:00 · c9c25a3 · pass · HEAD has a receipt
probe      2026-11-21 · 6aa609c · caught 80% · 8 missed

types   no tests              pass                                 1.4s
rules   53 tests              53 pass                              0.3s
api     32 tests              32 pass                              3.2s
flows   16 flows at 2 widths  32 pass                             18.0s

…

flows                                                                               desktop    phone
  ✓  10  apps/web/e2e/auth.flow.spec.ts  caught 11 planted bugs
    auth/sign-up · an 8-character password signs a new person up onto their dashb…  ✓  1.2s  ✓  1.1s
    auth/sign-up-short-password · a 7-character password is refused in the form     ✓  0.7s  ✓  0.8s
    auth/sign-up-next · signing up from a protected page lands back on it           ✓  1.0s  ✓  1.2s
    auth/protected-redirect · a signed-out visit to the dashboard asks to sign in…  ✓  0.6s  ✓  0.6s
    auth/sign-in-next · signing in returns to the page and query asked for          ✓  1.4s  ✓  1.1s
    auth/off-site-next · an off-site ?next= is ignored                              ✓  1.2s  ✓  0.9s
    auth/sign-out · signing out lands home, and Back does not reopen the dashboard  ✓  0.8s  ✓  1.1s
    auth/dashboard-ssr · a hard load of the dashboard hands React Query a real Da…  ✓  0.8s  ✓  0.7s
    auth/session-check-retry · one failed session check on a navigation still loa…  ✓  1.9s  ✓  2.0s
    auth/session-check-error · a session check that keeps failing shows the error…  ✓  2.0s  ✓  2.0s
…
It runs nothingIt reads the runs gate already kept: each test’s last result, how long it took, and whether it has failed before.
It names the idle testsEach test file shows how many planted bugs it has caught, and a file that exists but never ran is listed.
It narrows on request--failed for what is not a clean pass, --all for every test by name, --json for an agent.

Four rules keep a green run honest

A check has to have been seen failing once

Break the code on purpose, or let the planted-bug run do it, and watch the check go red before trusting its green.

Without it, half of 66 planted bugs passed every check.

The builder doesn’t write its own pass mark

What a flow must show is written down before the code, as one line a person could check by hand. A separate agent turns that line into the browser test.

Without it, the agent that wrote the bug writes the test that agrees with it.

A skipped test is a failed test

No test.skip, no test.todo, no run that finds zero tests, no test that quietly passes when an environment variable is missing.

Without it, one repo had 88 of 438 test files skipping themselves.

Checks run on a clean copy of the commit, with their own database

Never the working tree and never the dev database. A file that was not committed is not in the copy.

Without it, 32 test files in one app were writing to the dev database.

A green run is worth what the planted bugs say it is

Only a new sample measures the checks. Replaying a list you have already fixed goes to 100% and says little.

A missed bug is a real gap. Write the test that catches that exact change, then plant a new sample.

Read what was measured

Every kind of repo has a path onto it

A new project starts already on it

  1. Clone the starter and run bun run init my-app It carries the three names, the test databases, the flow fixture, the every-screen check, the access table and a gate.json.
  2. Run the kickoff, then bun run demo:drop The starter’s own tests fail until both are done.
  3. Run gate before the first feature It should be green. Every later number depends on it.
  4. Build each change check first Written with the change, seen failing, then made to pass.
  5. List the critical files in gate.json Sign-in, permissions, money, anything that writes data.
  6. Plant bugs once there is real code gate probe --max 40 --seed 1, a test for each miss, then a new seed.
  7. Turn on the live check with the first deploy
  8. Keep the click-through list from the first flow

An existing app takes four steps, in a fixed order

  1. Give the tests their own databases A cloned template per test process, one guard on every drop and truncate, no real keys. Done when gate is green on a clean copy.
  2. Names, the access table, and an API test for every procedure Done when a test names any procedure without one, and names none.
  3. Flows on every screen, at both widths Done when a full run names any screen no flow walked, and names none.
  4. Prove it with planted bugs Replay the first list, then plant a new sample and report that number.

An older fork gets it through the starter’s sync

  1. On a branch, run bun run sync It merges the starter’s changes and never commits.
  2. Check gate.json says report If it doesn’t, or the fork has no web app, run bun scripts/gate.ts --fit.
  3. Run bun run test:rules The names check lists every test file with the name to use.
  4. Run all four checks, commit, run gate
  5. Plant bugs The fork now stands at Walked. A new sample takes it to Measured.

Something that isn’t a web app keeps the checks that apply

TypesRulesAPIFlowsLive
A web appYesYesEvery server procedureEvery screen, both widthsYes
A desktop appYesYesThe bridge between the window and the main processOnly for a web surfaceNo
A mobile appYesYesIts server’s procedures, in the server’s repoNot covered: a real device is a person’s checkNo
A CLI or a libraryYesYesEach command or exported function, called the way a caller calls itNo screen, no flowsNo

Where a repo stands

  1. Not on itNo gate.json
  2. Safe to rungate is green on a clean copy, with its own databases
  3. NamedThree file names, the access table, an API test for every procedure
  4. WalkedEvery screen walked by a flow at both widths
  5. MeasuredA planted-bug run on a new sample is on recordThe starter 80%A production app 63%
  6. At the bar90% or more, with nothing missed in the critical files

What it does not do

  • It does not judge taste. Whether a screen feels right is a person’s call.
  • It does not stop a push or fail a deploy unless the owner turns that on.
  • It does not make green mean “no bugs”. The number is the point: it is known, and it can be moved.
  • It does not cover what no check reaches: mobile builds on real devices, third-party dashboards, and email as it lands in an inbox.