It's nine on a Tuesday. Someone on your team — let's call them Priya — opens the report from last night's run with a coffee, and the AI has left them five findings.
Priya starts at the top, because that's what you do.
The first is a broken image on the dashboard. Eleven minutes later it turns out to be a marketing pixel from a vendor that has 404'd continuously since March; nobody has ever cared. The second is a slow login — the staging box was cold, as it is every morning at 3am when the run starts. The third says an element couldn't be found, which Priya recognises immediately as the tool's own difficulty rather than anything about the product. The fourth is a console error thrown by an analytics script that isn't even ours.
The fifth one is real. An uncaught type error, fired right after Save, on a page nobody wrote an assertion for. It's a genuine defect and it's been in production for nine days.
Priya finds it at 9:41, and it's a good catch — the best thing the tool has done all month. But the honest description of that morning is that four out of five findings were nothing, and the fifth was buried underneath them.
Now run that Tuesday forward. Next week there are six findings and Priya reads three. The week after, someone in stand-up says "the observations thing is mostly noise, isn't it" and nobody disagrees. By month three, the panel is a thing on the report that people scroll past on the way to the failures.
The tool didn't get worse. It got ignored, which is worse.
Nobody did anything wrong
Not Priya, not whoever built the detectors. That's the part worth sitting with for a second.
The tool was asked a question it had no way to answer: are there bugs on this page? So it did the only thing it could — reported everything unusual it saw, with a confidence it hadn't earned, and left the sorting to a human at nine in the morning.
There was never a line inside it separating "I noticed something" from "this is a defect". And once that line is missing, a cold staging box becomes a performance bug, a third-party 404 becomes a broken integration, and the tool's own locator trouble becomes an application fault.
The same missing line produces the opposite failure, which is the one that's been going around QA circles lately. Plant a second, identical sign-in form on a working page — two username fields, two password fields, two live Sign in buttons — and point an AI testing agent at it. The run goes green. Every step passed, because there was always *a* field to type into and *a* button to click. The report comes back clean and the risk assessment reads zero.
The criticism attached to that experiment is fair, and it isn't really "the agent missed something". It's that the agent was confident. Five findings where there was one, and silence where there was a duplicated form: the same defect, pointing in two directions.
What the false ones actually cost
The obvious thing to count is Priya's forty minutes. The expensive part is everything that happens afterwards:
- The list stops being read. The first false finding gets investigated properly. The third gets skimmed. By the tenth, the team has learned a rule nobody wrote down — *the findings are noise* — and that rule outlives whatever you fix next quarter.
- The real one goes past. Priya's uncaught type error was genuine and nine days old. On the Tuesday where the panel has stopped being read, that defect just stays in production, in a report that says it's there.
- Trust doesn't come back on its own. Once a team has decided a channel is noise, a quieter channel doesn't change their mind. They aren't looking any more. You have to earn the second reading from a much worse starting position than the first.
None of which is an argument for reporting less. It's an argument that a finding is a claim on someone's attention, and every false one is spent from a budget that never refills.
A missed bug costs you one bug. An invented bug costs you the whole channel — including the bugs it would have found later.
The two usual escape hatches
Most teams reach for one of these when the noise gets bad, and both have a sting:
| What people try | What it buys | What it quietly costs |
|---|---|---|
| Turn the findings off | A clean report and a quiet morning | You've deleted the one finding in five that was real, and you'll never know which Tuesday it would have been |
| Only show "high confidence" findings | A shorter list | Confidence is self-reported. The duplicated sign-in form scored high confidence on a clean bill of health — the number rises with certainty, not with correctness |
The second is the sneakier of the two, because it looks like tuning and behaves like a blindfold. A threshold on a self-assessed score filters out the findings the tool was unsure about. It has nothing to say about the ones it was wrong about while being sure.
The run already knew
Here's the thing about Priya's Tuesday: none of those five findings required a model to *notice*. The run had already captured everything.
The uncaught type error was in the browser's console. The 404 was in the network log. The slow login was a duration far outside the run's own distribution. The unresolvable element was recorded by the resolver itself, which knew perfectly well that it was the one struggling. And the duplicated sign-in form was sitting in the page's own structure — two visible regions with the same fields, the same labels and the same submit text — which a plain piece of JavaScript can check in a few milliseconds.
Every one of those is a rule you can write down, argue about, and unit-test. So write them down. Four steps, and none of them are clever:
- Detect deterministically. Boring, checkable rules over signals the run already has. No model is asked to look for bugs, ever.
- Deduplicate across the whole run. The same 404 on twelve pages is one finding with a count, not twelve rows.
- Decide whose problem it is — the product's, or the automation's — before anyone is asked to read it.
- Then let language in, once, to explain. It writes the title and the sentence. It is describing a list it was handed. It cannot add to it.
That last ordering is also what makes it cheap enough to leave switched on: one call per suite run, skipped entirely when there's nothing to describe.
Here's the full detector set, which is short on purpose:
| Signal | What makes it a candidate | Ceiling |
|---|---|---|
| Uncaught JS exception | an uncaught error in the page's console during a step | bug candidate |
| Server error | an application request coming back 5xx | bug candidate |
| Failed or client-error request | 4xx, or a request that never completed | warning |
| Duplicated interactive region | two visible regions with matching form controls, labels and submit text | warning |
| Duplicate element ids | the same id used more than once on one page | warning |
| Ambiguous target | the step's description matched more than one live element | warning |
| Slow step | a step far outside the run's own distribution | info |
| App crash (mobile) | a fatal exception or ANR recovered from the device log | bug candidate |
Step 3 is the one that would have saved Priya most of her morning, and it has a rule attached that we'd defend anywhere: the classification is one-way. A finding can be moved from *product* to *automation*, never the other way. Anything landing on the automation side is capped at a warning and visually pushed down the list.
It means the tool takes the blame in every ambiguous case. That's the correct default for something asking a team for forty minutes.
What Priya sees instead
Rewind to nine on Tuesday. Same run, same page, same five things noticed — but the panel is now the last section of the report, under a caption that tells the truth about what it is: *issues noticed during execution; these did not affect pass/fail*.
| The panel says | Priya reads it as |
|---|---|
| Uncaught TypeError after Save — bug candidate, test case 3, step 7 | "That's real. That's the one." |
| GET /assets/pixel.png returned 404 — warning, ×14 | "Vendor pixel. Not ours." |
| Step 7 far slower than the run median — info | "Cold staging box. Again." |
| Step matched 2 candidate elements — warning, tagged automation | "That's the tool, not the product." |
Four of those rows have a Dismiss button. The first has Confirm.
That's the whole interaction, and it's deliberately not clever. What matters is what happens on Wednesday.
A dismissal has to be a decision, not a gesture
If dismissing a finding only hides a row on today's report, Wednesday's run brings all four back and you've built a treadmill with extra steps. So a dismissal is stored against the finding's own fingerprint and applied to that suite from then on — the vendor pixel is dropped before it ever reaches the panel again. Confirming it, or clearing the verdict, lifts the suppression.
One detail there took a real bug to learn, and it's worth stealing if you're building something similar. The fingerprint is computed from the finding's category and its normalised underlying message — never from the title, because the title is written by a model and can come out slightly differently next run. Key your suppression on generated text and dismissals silently stop matching. That's worse than not having the feature at all, because the user believes it worked.
The banner that used to say zero
The other half of the original criticism wasn't about the missed form at all. It was about a screen reporting no business risk detected while something was visibly wrong on it.
So the risk assessment now reads the findings channel. Product-classified findings that nobody has dismissed raise the assessment shown on the report, and the summary names them instead of asserting an all-clear. Review a finding and the banner re-evaluates on the spot.
What deliberately did not change is the underlying risk percentage, which stays a pure measure of test-failure exposure. Passive findings adjust what a human reads; they never move the number other things get compared against. Mixing the two would make every historical run incomparable, and would let an unreviewed observation quietly gate a release.
Three questions worth asking any tool that reports findings
Ours included. All three are answerable in a demo, in about five minutes:
- Can a finding change my pass/fail result? The right answer is no. Noticing and concluding are different jobs, and a tool that mixes them will eventually turn an honest green into a false red — or, much worse, the reverse.
- What happens when I dismiss one? Ask them to dismiss it and run again. If it comes back, dismissal is a UI gesture rather than a decision, and the list will grow faster than anyone reads it.
- What does it say about a page where nothing is wrong? Point it at something clean and boring. Silence is the correct output, and it is by far the hardest one to produce.
The goal was never a tool that finds more things. It's a list short enough that Priya still opens it in month six — and a report that says *this looks fine* only on the mornings when it does.
