Nobody's Selenium suite breaks at 500 tests. That's what makes it hard to see coming.
At 150 tests, it is the best thing the team owns: twelve minutes, real regressions caught, and when something fails, the person who wrote it knows why within a minute. At 700 the same suite has a standing agenda item, the nightly run is red most mornings, and two engineers are quietly assigned to it. Nobody made a bad decision anywhere on that path.
Value scales linearly with test count. Cost doesn't. Four mechanisms, all of them superlinear, all of them compounding.
1. The arithmetic nobody budgets for Say each test is 99.5% reliable — it passes 199 runs out of 200 when nothing is actually wrong. By most teams' standards, that's a well-written test.
| Suite size | Chance of an all-green run on a good build |
|---|---|
| 100 tests | ~61% |
| 300 tests | ~22% |
| 500 tests | ~8% |
| 700 tests | ~3% |
2. Maintenance follows shared surfaces, not test count A suite isn't a list; it's a graph. Login, the primary nav, the main table, the checkout step — a handful of components sit on the path of most of your tests. At 100 tests that concentration is mild. At 700 it's severe: one component redesign puts hundreds of tests in the failing column at once, and the real cost is working out which three of those failures are genuine bugs hiding inside the pile.
Page Objects help, but they change where you make the edit — not the blast radius.
3. Wall-clock time forces an infrastructure project At 30–90 seconds a test, 500 tests is a four-to-twelve-hour sequential run — incompatible with anything on a pull request. So you parallelise, and the suite becomes a distributed system: a grid or device-farm bill, strict test independence, test-data isolation, and a brand-new class of failure (session timeouts, node exhaustion, container starts) that looks exactly like a product bug in the report. None of that work tests your application.
4. Triage stops being an interruption and becomes a role A red run at 150 tests is a question with an answer. At 700 it is a queue: cluster the failures, separate the four real regressions from the locator drift and the environment blip, chase the two that only fail in CI. Fifteen minutes each, thirty of them, every morning. The moment your team has an informal "suite person," the cost has already moved from the maintenance budget into headcount without anyone approving it.
What actually helps
Delete tests. Least popular, most effective, and the only move that attacks all four mechanisms at once. A suite 30% smaller and fully trusted is worth more than the one it replaced.
Push coverage down the pyramid. A lot of what sits in a 700-case UI suite is business logic an API test could assert in 200 milliseconds, with no browser, no grid, and no locator.
Attack locator drift at the source — stable data-testid attributes agreed with the developers, or run-time element resolution that repairs a moved element instead of failing on it.
Treat flakiness as a bug class with an owner, not a retry count. Measure per-test pass rate on known-good builds; fix or delete anything under the bar. Retries hide the mechanism and let it grow.
Change the marginal cost of authoring and repair. If 500 tests have to be hand-maintained by the same people who hand-wrote them, the arithmetic above is fixed no matter how good they are.
QAEverest generates suites from your stories and repairs them when the UI moves, so what changes is the maintenance curve itself — not the number of people pointed at it.
