Testing standard 1.1

What was measured

The standard came out of an audit and was then proved on two repos: the starter template every new project begins from, and one production app with 174 API procedures. Every number here comes from a recorded run.

Before: half the planted bugs passed every check

The audit covered 3,130 test files in 59 repos. Three in four were unit tests. They were honest tests (only 27 files mocked anything), but each checked one small function.

To see what the tests were worth, 66 one-line bugs were planted in two repos, one at a time: a flipped condition, a deleted guard, a number off by one. 33 of the 66 passed every check.

Repo Planted Caught by types Caught by tests Passed everything
The starter 40 13 11 16
The production app 26 4 5 17

Three of the ones that got through:

What the audit showed:

After: every one of those bugs is caught

The same planted bugs were replayed after the work.

Repo Got through before Get through now
The starter 16 of 40 0
The production app 16 of 25 (one of the 26 is log formatting and is set aside) 0 of 24 (a second is set aside: both versions of the line behave the same)

On the production app the replay moved step by step, which shows what each step bought:

After step Catch rate on the audit's list What changed
1. Own databases 36% Nothing about the tests' reach, only their safety
2. An API test per procedure 64% Every server-side miss caught
3. A flow per screen 100% Every screen-side miss caught

The honest number: bugs the tests were never written against

A replayed list goes to 100% because the tests were written to catch it. The measurement that counts plants bugs nobody has seen.

Repo Run Caught Rate
The starter 40 new bugs 32 80%
The production app 40 new bugs, first run 21 53%
The production app 40 new bugs, after fixing the first run's misses 23 57%
The production app 40 new bugs, after fixing the second run's misses 25 63%

Neither repo meets the 90% bar yet. Each round of writing tests for the misses moved the production app about five points.

Which check caught the 32 on the starter: types 15, rules 9, flows 6, API 2.

What the checks found that no planted bug was looking for

Turning the standard on surfaced real problems before any bug was planted:

What a full run costs

On the production app, on a clean copy of the commit: types 3 s, rules 1 s (422 tests), API 41 s (1,071 tests), flows 100 s (164 runs across two widths).