It's twenty to five on a Thursday. Someone on your team — let's call them Sam — notices that the invoice date renders as "Invalid Date" when the customer's locale is missing. It's a null check. Two lines. Sam has it fixed before the kettle's boiled, opens the pull request, and CI wakes up.
Two thousand tests start running.
Sam already knows how this goes, because it goes this way every day. Forty-one minutes. Not worth sitting and watching, so they pick up the other ticket, the one with the tangled state machine, and twenty minutes later they're properly inside it. Then the build finishes. One test failed — somewhere in reporting, nothing to do with dates. Sam sighs, hits re-run, and it goes green. Forty-one more minutes.
The two-line fix merges at 6:15.
And here's the thing: nobody did anything wrong. Not Sam, not whoever wrote the pipeline, not QA. That's the part worth sitting with for a second.
The rule was right when someone wrote it
Somebody added "run everything on every push" to that YAML back when the suite had forty tests and finished in ninety seconds. At the time it was obviously correct — running the lot was free, and stopping to think about which tests mattered would have cost more than just running them.
Two years on, the suite is 2,000 tests, the pipeline takes most of an hour, and that line in the YAML has never been reopened. Nobody chose this. It's a default that quietly outgrew its reasoning, and the bill now lands on the whole engineering team rather than on QA.
What the waiting actually costs
Ask a finance team and they'll point at the CI minutes. That's the cheap part. The expensive part is everything in Sam's afternoon that never appears on an invoice:
- The context Sam dropped. Forty-one minutes is long enough to go somewhere else mentally and come back badly. Two of those in one afternoon is most of a working day spent re-loading your own head.
- Sam's next pull request will be bigger. When the gate is slow, people stop opening small PRs. They bundle. And a bundled change is harder to review, harder to revert, and much harder to bisect when it breaks something at 9pm.
- The day Sam stops reading. A long run with a 2% flake rate goes red for reasons nobody cares about, often. The team learns to hit re-run rather than open the log. That's the moment your suite stops being a safety net — and it is horribly hard to come back from.
None of which is an argument for testing less. It's an argument that "run everything, always" is a policy, and policies should be defended rather than inherited.
The two usual escape hatches
Most teams reach for one of these, and both have a sting:
| What people try | What it buys | What it quietly costs |
|---|---|---|
| Move the suite to nightly | Sam merges in five minutes | Sam finds out on Friday morning, by which point four more PRs have landed on top of the break |
| Hand-pick a "smoke" subset | Fast PRs, some coverage | Somebody chose that list once. Nobody has revisited it since. It now describes the product of eighteen months ago |
The second one is the sneakier of the two, because it looks like a solution and behaves like a fossil. A static subset is one person's guess about where risk lived on the day they made it. Risk moves every sprint. The list doesn't.
The pull request already knew
Here's what nags at me about Sam's afternoon. That PR was never a mystery. It changed one file — `src/invoice/formatDate.js`. It did not touch payments. It did not touch auth. It did not touch the reporting suite that failed and then passed.
Something in the pipeline had all the information needed to say "this can only have broken date formatting on invoices" — and instead it shrugged and ran everything, because nobody ever taught it to look.
So teach it to look. Four steps, and none of them are clever:
- Read the diff. The changed file paths. That's it — no static analysis, no magic.
- Work out which part of the product those files are. `src/invoice/formatDate.js` is invoicing. This is the step doing the real work, and the step most likely to be wrong.
- Pull the tests that cover that part. This is what a traceability layer is actually for: a link between the requirement and the tests that exercise it.
- Rank what's left by business risk, then cut. Take the tests that retire the most risk, keep going until you've covered the risk you care about, stop.
Step 4 is the one teams love to over-engineer. You don't need a model for it. Weight each test by how critical the thing it covers is, take them in order, never drop one that sits on a critical path no matter what the arithmetic says. It's a greedy knapsack, it's about fifty lines, and — this matters more than elegance — you can explain it to an auditor without a slide.
And the big reduction doesn't come from step 4 anyway. It comes from step 3: scoping to the touched areas at all. Tuning the ranking is a second-order gain. Teams pour their effort into it and leave step 2, where the accuracy really lives, on a naive default.
What Sam sees instead
Rewind to 4:40. Same fix, same PR — except ninety seconds after Sam hits "Create", a comment appears on it. Not an email. Not a dashboard Sam would have to remember to open. A comment, on the pull request, where Sam already is.
| The comment says | Sam reads it as |
|---|---|
| Risk score: 2.1 / 10, green | "This is a small blast radius. Relax." |
| Run 18 of 2,000 — 94% of the risk, ~4 min instead of ~41 | "I'm merging this before my tea goes cold." |
| Touched: Invoicing → date display | "Yes. That's exactly what I changed." |
| No coverage gaps | "Nothing I edited is untested." |
That third row is the one people underestimate. It's the sanity check. If Sam had changed payment logic and the comment came back saying "Touched: Reporting", Sam would spot that the mapping is wrong in about two seconds — and a mapping you can eyeball is a mapping you can trust. A number with no reasoning attached is just a number, and engineers are right not to trust it.
The comment gets rewritten in place on every push to the branch. It's not a notification stream. It's the current state of the change, sitting in the review, next to the diff it's describing.
The comment that's worth more than the time saved
A fortnight later, a different PR. Someone's refactoring how invoices are totalled — `billing/invoice.js`, twelve files, a proper change rather than a null check. The comment comes back at 8.4 out of 10, red, and asks for 34 of the 2,000 tests instead of 18. Fine. That's the system working: bigger blast radius, bigger run.
But there's a second line under it, and it isn't about how many tests to run:
You changed billing/invoice.js. No test covers it.
Read that carefully, because it's not "a test failed". It's worse and quieter. Nothing is going to go red. Nothing is going to go green either. There is simply nothing there, and the pipeline will hand back a cheerful tick that means we didn't check.
That's the finding that pays for the whole exercise, and it falls out of the same analysis for free. Once you've mapped changed files to parts of the product, you also know two things you didn't before: which changed files map to something with no tests at all, and which map to nothing your traceability data has ever heard of. The first is edited untested code. The second is usually a whole module nobody ever wrote requirements for — which is worth knowing on a Tuesday rather than during an incident.
Selection tells you which tests to run. The gap list tells you where you have no tests to run — and that's the more expensive thing not to know.
Where this gets it wrong
Any honest version of this article needs this section, because the failure mode is specific and it's the reason to roll out slowly rather than flip a switch on Monday.
The mapping is a heuristic. Matching path names against requirement text works surprisingly well on a codebase with sane naming. It misses when a file is named nothing like the feature it implements, when someone edits a shared utility that half the app imports, or when the change is a config file quietly reconfiguring three modules. A missed mapping means a missed test — and a selection that confidently says "96% of the risk" while leaving out the one that mattered.
The minutes figure is an estimate. If you don't store per-test durations, "~4 min" is a test count multiplied by an assumption. Fine for "minutes, not hours". Not fine in an SLA.
A threshold can't prune a long tail. Cutting at 95% of total risk weight can't drop a big pile of low-risk tests whose weights add up past your remaining 5%. That's arithmetic, not a bug — but it's another reason the win comes from scoping, not from the clever part.
How to try it without betting anything
Run it in shadow mode. Two weeks, no risk, and it settles the argument with data instead of opinion:
- Post the comment, run the full suite anyway. For a fortnight the recommendation is advisory. Nobody's pipeline changes and nobody has to trust anything yet.
- Count the misses. Every time the full suite fails, ask one question: was that test in the selected set? A miss is a mapping bug, and mapping bugs are fixable — usually by writing one explicit rule for a directory the heuristic can't read.
- Then move the gate, once the miss rate is a number you'd happily say out loud in a release meeting. Keep the full suite on merge regardless.
Three numbers will tell you whether it's working, and your CI already has all of them:
- Selection miss rate — failures the selected set wouldn't have caught. This is your safety number. It's the one to quote when someone challenges the idea.
- Median PR wall-clock — open to gate-green. This is the number the team feels, and the one that changes Sam's Thursday.
- Gap findings acted on — how many "you edited untested code" flags turned into an actual test. This is the one that compounds.
The question to take back to your pipeline
The instinct behind running everything is a good one, and it deserves respect: I don't know what this might break, so I'll check all of it. Fair enough. It's just that the diff was sitting right there the whole time, and it did know.
So the question isn't whether you can afford to run fewer tests. It's this: if the verdict came back in seven minutes instead of forty-one, how much smaller would your team's pull requests get?
