Test Intelligence — Frequently Asked Questions

What is the Trust Scorecard?

A page for each suite that shows how far its results can be trusted, in seven tiles: Suite Sensitivity, Check strength, Confirmation Rate, Fragility Index (Assertion Coverage Gap on an API suite), Flakiness, Coverage Gaps and Self-Heals. Open it from the Trust Scorecard button on a suite's Suite History page.

A tile with no measurement says so in words, such as "Not measured" or "Awaiting review". Self-Heals counts from any run in the last 30 days, so it can read 0. Tiles with a measurement link to the evidence behind them.

What is Check strength?

It tests your tests. After a normal run of an API, UI or mobile suite, it changes the expected value of each passing check it can judge again (one character of a text, a number raised or lowered by one, a state flipped) and judges the check again against what that run stored. A good check now fails. One that still passes would let a wrong answer through. Checks it can't judge again, such as those read by the vision model, are listed with the reason.

Nothing is sent to your app and no credits are used. By default it checks new and changed tests after every run, you can press Check strength on the Trust Scorecard, and Use in CI gives you a step that can fail the build.

What do the Check strength results mean?

Each test reads Seen fail, Weak checks, Can't fail, Not evaluated, Not checked yet or Changed since check. A check that survives says why it passed and, on UI and mobile suites, how to tighten it.

What is Suite Sensitivity?

It seeds one controlled defect per test case into a run of the suite and counts how many the suite catches. Your application itself is never modified. Critical, High and Medium tests each have their own bar, and Low is tracked as a trend. By default, Critical and High may miss nothing.

A missed defect is marked Will fix or Risk accepted, or set aside as Equivalent, Not exercised, Wrong layer or Transient (flaky). Accepting a risk or setting a defect aside needs a reason of at least 10 characters, recorded with who made the decision and when. A run exports as a standalone evidence report.

Sensitivity runs use no credits and never change the status or the bill of your normal runs. API and UI suites can use it. Mobile suites are left out, because seeding a defect there would mean changing the app.

What is Check history?

Changes to what a check expects are recorded, with who or what made it and whether it weakened the check: an expected value edited, a check switched off, a failure accepted as a pass, equals loosened to contains, a visual mask added or a locator self-healed. A locator a person re-points with Point to Element or Accept Repair is not listed. A person is named. A machine shows as "AI re-parse", "Self-heal", "Suite maintenance" or "AI reading of the step".

The Check history button on a suite's Execution page shows how many changes weakened a check in the last 30 days. In the traceability matrix, a test weakened after its last run stops counting as Exercised.

What does "Not verified" mean on a run report?

The run could not confirm a check, and it says so instead of showing a plain pass:

• Web UI — a test case whose steps all worked but none of whose checks could be confirmed counts as skipped, not passed, with each unconfirmed check and its reason. The suite list reads Passed if nothing failed and at least one test passed, with a note such as "1 not verified". If nothing passed, it reads Skipped. On Suite History the run reads Skipped, and that page's Skipped filter finds it.
• API — a step whose check could not compare anything stays green with a "Not verified" badge and the reason, such as a field missing from the response.
• Database steps — a step that never reads its rows is marked "Rows not checked".
• Mobile — there is no such status. A check that cannot be made fails and says why.

How does QAEverest find flaky tests?

Test Health covers all your own suites. It scores tests from run history, mostly by how often they flip between pass and fail over the last 20 runs, and groups repeated failures by their error message, so ten reds with the same error read as one line. Open it with Open Test Health on a Trust Scorecard's Flakiness tile. The link appears once a test in that suite has at least three runs in the last 30 days.

Each time Test Health is opened, flaky tests are flagged as quarantined, and a test flagged this way is released once its last 20 runs show fewer than two flips. A flag or release you set by hand stays until someone changes it. Quarantine does not take a test out of runs: it flags the test so it is seen.

What is the hours-saved figure on Test Health?

An estimate, not a measurement. It turns healed steps and quarantined flaky tests into hours at 30 minutes each by default, and shows the per-heal figure it assumes.

What is Suite Health?

Suite Health works out which tests no longer match your app, for UI, API and mobile suites. Press Run Maintenance on a suite's Suite Health tab, or have the PR bot run it on the suites a merged GitHub pull request touches. Each test is shown as healthy, at risk, broken or orphaned.

In a UI suite, an element whose selector changed has its remembered locator updated automatically, unless you pinned it or the selector is on several pages: those wait for Accept Repair. A broken step gets a suggested fix, which changes nothing until a person edits the test. API and mobile suites get the verdicts and Retire Test, but no locator repairs or suggested fixes.

Can I replay a test run?

Yes, for web UI and mobile runs; API runs have no replay. Time-Travel Replay plays a test case frame by frame: press Play, click any shot in the filmstrip, or use Jump to failure. Console, Network, Page and Locator tabs show what the browser logged, the requests it made, the page text and how the element was found at each step. Mobile runs replay the screenshots only.

What is AI Root Cause?

On a failed web UI step, AI Root Cause compares the last good screenshot with the failing one, names the cause in a sentence, such as "the Submit order button was covered by the cookie consent banner", and labels it an application, test or environment problem, with a confidence percentage.

Can I see how an element was chosen?

Yes. "How this element was chosen" appears under a failed web UI or mobile step that resolved an element, and on a passing step whose choice was uncertain. It shows whether the element was found by the page's own text and labels, by reading the screenshot, from an earlier run, through self-healing, or from an element you pinned, how confident the engine was, whether the element sat in a frame, how close the runners-up came and what the AI read the step to mean. On mobile it shows the Appium strategy and selector instead of frames and runners-up.

What is Step Repair?

A failed web UI or mobile step has one Step Repair button, and API results have none. Web UI steps get all four remedies, and mobile has no Wait Before:

• Edit Step — reword the step.
• Point to Element — click the element on the failure screenshot, rule a wrong one out, or type a locator. An element you pin outranks the AI's own choice whenever it is on screen. This works on mobile results too.
• Wait Before — make a web step wait until the page is ready.
• Accept Failure — accept the failure, with an optional note.

Repairs are saved for the next run.