Most teams answer this with two numbers: the tool licence and the compute. Both are real. Both are the small ones.
The bigger number is engineering time, and it never arrives as an invoice. It arrives as half a day here and a Thursday afternoon there — until a large share of the automation team's week goes on keeping the automation itself running.
To be clear about where this is going: none of what follows is an argument that automation isn't worth it, or that the people maintaining it aren't doing good work. The cost is structural. It comes from release cadence, UI churn and run frequency — forces nobody in QA controls.
Cost isn't per script
A script sitting in a repo costs nothing. It costs something when reality moves underneath it and someone has to go and look. So the real drivers are:
| Cost line | What actually drives it |
|---|---|
| Break-fix hours/yr | scripts × churn-per-release × releases/yr × hours-per-fix |
| Triage hours/yr | scripts × runs/yr × flake-rate × hours-per-investigation |
Take 1,000 UI scripts, 24 releases a year, 3% needing a touch per release at 36 minutes each, 240 full suite runs a year, and 1% flake at 12 minutes per investigation:
| Cost line | Hours / year |
|---|---|
| Break-fix (720 fixes) | 432 |
| False-failure triage (2,400 investigations) | 480 |
| Framework, driver and CI upkeep | 144 |
| Test data and environments | 100 |
| Total | ~1,156 — roughly three-quarters of an engineer |
These inputs are illustrative, not research. They are written out precisely so you can replace them with your own and get an answer that means something for your team.
The input that decides everything
Hold all of it constant and move only the flake rate:
| Flake rate | Total hrs/yr | FTE |
|---|---|---|
| 0.5% | 916 | 0.6 |
| 3% | 2,116 | 1.4 |
| 5% | 3,076 | 2.1 |
Same scripts, same cadence, three times the cost. And 5% isn't a disaster — it's an ordinary UI suite running against a moving application, on a team that has never been given a sprint to stabilise it.
Now run on every pull request instead of nightly. At 1% flake, triage alone passes 2,000 hours. Shortening the feedback loop is the right call — and it quadruples the line item that appears in no plan. That is worth knowing before someone concludes the team got slower.
Script count is what gets counted, because it's easy to see. Flake rate and run frequency are what actually get paid.
The costs that never reach a spreadsheet
Quarantine debt. Skipped scripts cost nothing to run and everything in coverage, while the dashboard stays green.
The coverage ceiling. Every new script carries a permanent maintenance liability, so suites quietly stop growing — and the tests that were never written don't show up anywhere.
Trust decay. A suite that's red for the wrong reasons often enough stops being read. At that point its value is zero and its cost is unchanged. This is the one worth escalating early.
Three numbers to pull this week
All three are already in your git history and CI logs:
- Test-fix ratio — what share of test commits fix an existing test rather than add a new one. A high number is a workload signal, not a verdict on the team.
- Failure-to-defect conversion — what share of failing runs traced to a real defect. That share is the return on your triage hours.
- Quarantine count and age — how many scripts are skipped, and for how long.
What this argues for
Not a smaller QA team. Almost every hour in the table above goes on work that produces nothing anyone wanted: reproducing failures that weren't failures, and re-running a whole suite to learn about a change that touched one module.
The cost model argues for removing those two terms — resilient locators, flake detection, failure clustering, and running only the subset a change can actually affect — so the same team's hours go back into coverage, exploratory work, and the risk judgement no tool does well.
And when a flow is genuinely redesigned, its tests genuinely have to be rewritten. No tooling makes that free, and any vendor implying otherwise is describing a demo.
So the useful question isn't which scripts would survive a cull. It's: if a third of your maintenance hours came back next quarter, what would your team finally get to test?
