Skip to main content
B2B SaaSAutomation

A nightly suite that was red so often, its results stopped being read

When a suite is red every morning, red stops meaning anything. Here is how a QA team separates flakes from regressions, gets a cause for each failure, and proves the green runs are checking something.

Scenario: A QA team running a large nightly UI regression

Problem

The same few tests failed most nights, real regressions hid among them, and a screenshot showed that a step failed but not why.

Solution

Flaky tests separated from regressions, failures replayed with an AI root cause, and seeded defects that show what the suite really catches.

Capabilities usedTest healthTime-travel replay and root-cause debuggerHow each element was chosenExecution observationsSuite sensitivity + Trust Scorecard

The problem

Hundreds of UI tests ran every night. A handful failed every morning, often the same intermittent ones, and the first hours of the QA team's day went on re-running tests and reading screenshots to decide whether each failure was the application, the test or the environment.

  • The same few tests failed intermittently, so a new failure looked like an old one.
  • Ten red tests often shared a single cause, but were triaged one by one.
  • A failure screenshot showed that a step failed, not why it failed.
  • Green runs raised their own question: was the suite checking anything at all?

The solution

The QA team still makes every triage call. QAEverest puts the evidence for that call in one place.

  1. Separate flakes from regressions

    Test health

    Flaky tests are found from run history and can be quarantined automatically, and repeated failures are clustered so ten reds are read as one cause.

  2. Replay the failure, then read the cause

    Time-travel replay and root-cause debugger

    A run replays frame by frame with Console, Network, Page and Locator tabs for each step. When a web UI run has a failed step, an AI Root Cause panel compares the last good screenshot with the failing one, names the cause in a sentence, and labels it an application, test or environment problem with a confidence percentage.

  3. See how the element was chosen

    How each element was chosen

    Under a failed web UI step, a panel shows how the engine found its element — page text and labels, the screenshot, an earlier run, self-healing or a pinned element — how confident it was, and, when it was a close call, how near the runners-up came.

  4. Catch what nobody asserted

    Execution observations

    While a run happens it watches for JavaScript errors, server errors, unusually slow steps and ambiguous controls, and reports them as observations — bug candidate, warning or info — separate from pass and fail, for a reviewer to confirm or dismiss.

  5. Prove that green means something

    Suite sensitivity + Trust Scorecard

    Suite sensitivity seeds one controlled defect per test case inside the browser session — the application is never modified — and counts how many the suite catches. The Trust Scorecard reads “Not measured” for anything it has not measured, rather than a clean zero.

What changes

  • Morning triage starts from causes and clusters, not from a list of red tests.
  • Each failure arrives labelled as an application, test or environment problem, with a confidence the team can weigh.
  • The team can show how much the suite catches, not only how often it passes.

Read more

Facing a problem like this?

Bring one of your own stories, suites or release questions to a demo and we will walk through it in the product with your QA team.

Book a demo Start free

More case studies

PaymentsScenario

The API returned 200, but did the payment actually reach the database?

Problem

API tests checked status codes only; tokens were copied between requests by hand and the ledger row was checked separately in a SQL client.

Solution

Chained API requests, read-only database steps in the same test case, and generated integration and database cases for review.

Read the case study