Skip to main content
Trust ScorecardHealth · Replay · Repair

Your suite is green.
How much of it was checked?

QAEverest measures how far your tests can be trusted. It checks that each check can fail, seeds defects to see what the suite catches, and marks a check it could not confirm on the run report. When a test breaks, it replays the run and names the likely cause, so your QA team spends its time on real failures.

Check strength and suite sensitivity runs use no credits.

Scorecard
7
Tiles on the Trust Scorecard
Suite sensitivity, Check strength, fragility, flakiness, confirmation rate, coverage gaps and self-heals.
Check strength
6
Verdicts for each test's checks
From Seen fail to Can't fail, and why a check that survives got through.
History
400
Days of check history by default
Every change to what a check expects, who or what made it, and whether it weakened the check.
Trust

Can you trust this green build?

Four measures, each read from your own runs, that show what a passing suite has really proved.

Trust Scorecard

How far a suite can be trusted, on one page

A page per suite showing how stable it is, how much it healed and how much it really covers, in seven tiles.

  • Suite Sensitivity
  • Check strength
  • Fragility Index
  • Flakiness
  • Confirmation Rate
  • Coverage Gaps
  • Self-Heals
  • Anything it has not measured says so in words, such as “Not measured” or “Awaiting review”, never a clean zero.
  • Tiles link to the evidence behind them: the run that measured it, Test Health or Traceability.

Built as an answer to a fair criticism of AI test tools: reporting confidence they have not earned.

Check strength

Never trust a check you have never seen fail

After a normal run of an API, UI or mobile suite, it changes the expected value of each check that passed (one character of a text, a number raised or lowered by one, a state flipped) and judges the check again against what that run stored.

  • A good check now fails. One that still passes would let a wrong answer through.
  • Each test reads Seen fail, Weak checks, Can't fail, Not evaluated, Not checked yet or Changed since check.
  • A check that survives says why it passed and, on UI and mobile suites, how to tighten it.
  • Database checks inside API suites are judged the same way.

Nothing is sent to your app and no credits are used. By default it checks new and changed tests after every run, and it also comes as a CI step that can fail the build.

Suite sensitivity

Seed a defect. Does the suite notice?

It seeds one controlled defect per test case into a run of the suite and counts how many the suite catches. Your application itself is never modified.

  • A suite that catches nothing is not a passing suite. It is an untested one.
  • Critical, High and Medium tests each have their own bar, and Low is tracked as a trend. By default, Critical and High may miss nothing.
  • A missed defect is marked Will fix or Risk accepted, or set aside as Equivalent, Not exercised, Wrong layer or Transient (flaky).
  • Accepting a risk or setting a defect aside needs a reason of at least 10 characters. The decision is recorded on the run with who made it and when, and earlier decisions are kept.
  • A run exports as a standalone evidence report, judged by the thresholds in force when it was measured.

Sensitivity runs use no credits and never change the status or the bill of your normal runs. Mobile suites are left out: seeding a defect there would mean changing the app.

Check history

Every weakened check is on record

Every change to what a check expects is recorded, with who or what made it and whether it weakened the check.

  • An expected value edited, a check switched off, a failure accepted as a pass, equals loosened to contains, a visual mask added or a locator self-healed.
  • A person is named. A machine shows as “AI re-parse”, “Self-heal”, “Suite maintenance” or “AI reading of the step”.
  • A Check history button on a suite's Execution page shows how many changes weakened a check in the last 30 days. Switch off “Only weakening changes” to see every change.
  • In the traceability matrix, a test weakened after its last run stops counting as Exercised.

History is kept for 400 days by default.

Not verified

A check it could not confirm says so on the run report.

A green result should mean something was actually checked. Each engine says plainly when it could not check.

  • Web UIA test case whose steps all worked but none of whose checks could be confirmed counts as skipped, not passed, and is marked “Not verified”, with each unconfirmed check and its reason.
  • Suite listStill reads Passed when nothing failed, with a note such as “1 not verified”. The pass rate is not lowered. A suite where nothing could be confirmed reads Skipped, and a Skipped filter finds it.
  • APIA step whose check could not compare anything stays green with a “Not verified” badge and the reason, such as a field missing from the response.
  • DatabaseA Database step that never reads its rows is marked “Rows not checked”.
  • MobileA check that cannot be made fails and says why.
Across every run

Flaky tests, repeat failures and trends over time

Read your whole run history at once, not one red build at a time.

Test Health & ROI

Find the flaky tests and the shared causes

  • Finds flaky tests from run history, scored mostly by how often they flip between pass and fail over the last 20 runs.
  • Flags a flaky test as quarantined, automatically or by hand, without taking it out of runs. One you flag stays on the list until someone releases it; one flagged automatically is released when it stops flipping.
  • Groups repeated failures by their error message, so ten reds with the same error read as one line.
  • Turns healed steps and quarantined flaky tests into an estimate of hours saved, at 30 minutes each by default, and shows the per-heal figure it assumes.
Insights → Open Failures

Every test that is still failing

With its severity, when it started failing, when it last failed and whether the team marked it OK to release.

  • Each data row is judged by its own results, over 7, 30 or 90 days.
  • Skipped results, infrastructure aborts and suite sensitivity runs never count as failures.
  • A teammate's row shows its title, suite, owner, severity and dates, never their error text.
  • Download a summary PDF, or email it.
Insights → Scan Trends

How each scanned target changed

Over 30, 90, 180 or 365 days, with a chart and the latest values for each target.

  • p95 response time for load and soak runs. Better or worse is judged only between runs at the same load.
  • Open and confirmed issues for security scans, and the score for accessibility scans.
  • The change in words, such as “120 ms slower” or “3 fewer issues”.
  • PDF and Excel downloads.
Self-maintaining suites

Find what a change quietly broke, before the next run does.

Suite Health works out which tests no longer match your app, for UI, API and mobile suites.

  1. 01

    Start it

    Press Run Maintenance on a suite's Suite Health tab, or turn on the PR bot to run it on the suites a merged GitHub pull request touches.

  2. 02

    Look again

    A UI suite's app is crawled again and each step's remembered element is checked against what is there now. API endpoints are probed; a mobile suite is judged from its run history.

  3. 03

    Sort

    Each test is shown as healthy, at risk, broken or orphaned.

  4. 04

    Repair

    An element whose selector changed has its remembered locator updated automatically, unless you pinned it or the selector is on several pages: those wait for Accept Repair. A broken step gets a suggested fix, which changes nothing until a person edits the test.

When a test fails

See why it failed, then fix the step.

Every failed web run keeps what the browser saw, so you can read the cause without running it again.

Time-travel replay

Replay the run, frame by frame

At every step of the run, four tabs show what happened:

Console
what the browser logged
Network
the requests it made
Page
the page text
Locator
how the element was found
  • Press Play, click any shot in the filmstrip, or use Jump to failure.
  • On a failed web UI step, AI Root Cause compares the last good screenshot with the failing one, names the cause in a sentence and labels it an application, test or environment problem, with a confidence percentage.

Mobile runs replay their screenshots.

How this element was chosen

See how the engine found its element

  • Whether it was found by the page's own text and labels, by reading the screenshot, from an earlier run, through self-healing, or from an element you pinned.
  • How confident the engine was, whether the element sat in a frame, how close the runners-up came and what the AI read the step to mean.
  • When a web check fails, the message names every method that ran and every one that was skipped.

Shown under a failed web UI or mobile step, and on a passing step whose choice was uncertain. On mobile it shows the Appium strategy and selector instead of frames and runners-up.

Step Repair

Four ways to fix a failed step

  • Edit the step, point to the right element, make a web step wait until the page is ready, or accept the failure with an optional note.
  • Point to Element: click it on the failure screenshot, rule a wrong one out, or type a locator.
  • An element you pin outranks the AI's own choice whenever it is on screen.

Repairs are saved for the next run. Point to Element works on mobile results too.

Honest by design

Every result says where it came from.

Your QA team signs off on what the numbers say, so the numbers never claim more than was checked, and every change to a check is on record.

Check history5 changes · 3 weakening in the last 30 daysOnly weakening changes
  • Ravi (step clarification)Comparison loosened equals → containsWeakened
  • Asha (accepted a failure)Failure accepted as a pass with a noteWeakened
  • Self-healLocator self-healed #pay-now → #checkout-pay
  • Ravi (visual mask)Visual mask added promo carouselWeakened
  • Asha (test case editor)Expected value edited 104.50 → 104.00

Not measured is not zero

A tile with nothing behind it says so in words, so a gap never reads as a clean result.

Not verified is labelled

A check that could not be confirmed says so on the run report, with its reason.

New wording waits for a person

A locator whose selector changed can update by itself, and on a check that goes in Check history. New wording for a step is only ever suggested: a person edits the test.

Every change has a name

Check history names the person, or the part of the product, behind each change.

Decisions carry reasons

Accepting a missed defect needs a written reason, recorded on the run with who decided and when, and shown in the evidence report.

Free to measure

Check strength and suite sensitivity runs use no credits.

Evidence your QA team can stand behind.

Run a suite, open its Trust Scorecard, and press Check strength to see which checks can really fail.