Skip to main content
GitHub · GitLab · Bitbucket CloudPR bot and CI steps

A pull request just opened.
Which tests does it need?

QAEverest reads each pull request, scores its risk out of 10 and names the tests worth running for that change. It can run them as a check on the pull request, suggest edits to tests the change breaks, and check your automation again after the merge.

Set up by an organisation admin. Every option starts off, and the settings are marked Beta.

Code hosts
3
Code hosts for the PR bot
GitHub, GitLab at gitlab.com or a self-managed server reachable from the internet, and Bitbucket Cloud.
Risk
10
Point risk score on each pull request
With the tests worth running for that change, in one comment the bot edits on later pushes.
CI
2
Steps that can fail the build
Check strength and the AI agent release gate, as a GitHub Actions workflow or a bash step for any other CI.
One pull request

From the first push to after the merge

Each step after the first is its own option in the PR bot settings. Steps 03 and 04 work with GitHub; the first two work with all three hosts.

01

Opened

The bot reads the change, scores its risk out of 10 and comments with the tests worth running and the requirements nothing tests yet.

02

Pushed

With checks on, the picked tests run on every push, and the result comes back to the pull request as a commit status.

03

Reviewed GitHub

When the change renames or removes something a test's steps name, Suggest updates proposes the step edits, before and after.

04

Merged GitHub

Suite Health checks the suites the change touched. In UI suites, out-of-date locators are updated and a broken test gets a proposed fix.

On every pull request

A risk score, the tests to run, and what nothing tests yet.

The PR bot

One comment on the pull request, kept current

It comments on pull requests, and on merge requests on GitLab, and edits the same comment when new commits arrive.

  • A risk score out of 10, and the tests worth running for that change, each with the reason it was picked.
  • For up to five tests that failed last time, the AI root cause stored with that run.
  • The requirements the change touches that have no tests, and those whose tests show no current evidence.
  • Optional drafts for the untested ones: up to five test cases for each of up to five requirements per push, saved for review.
  • It can be limited to pull requests into chosen base branches, such as release branches.

An organisation admin sets it up with a webhook and the host's token. On GitHub it can cover several repositories, each read against its own project. Bitbucket Data Center is not supported.

Smart test selection

The tests a change needs, ranked by risk

The bot uses it on every push it analyses. You can also open a GitHub pull request on the Smart Selection tab of the Traceability page: a public one, or a private one your saved GitHub token can read.

  • Reads the change itself: the words in the changed code, the endpoints it defines or calls, element ids, on-screen text and database tables.
  • Matches them to your requirements and to what each linked test calls.
  • Picks tests until they cover 95% of the risk, always keeping critical tests and any test the change touches directly.
  • Times come from each test's own recent runs, and the page says how many were measured rather than estimated.
  • Tick the picks in your own suites and start them from the same page, one run per suite.
  • Automate builds a requirement's reviewed drafts, from an analysis you ran, into a new suite named after the pull request and the requirement. Building it is free, and it is not run for you.

Every analysis, from that tab, the PR bot or the Copilot, is kept on the Traceability History tab, where anyone in its project can reopen it.

Pull request checks

The picked tests run on every push

Turn it on in the PR bot settings and the bot runs the tests Smart Selection picked, then reports back on the pull request or merge request.

  • A pending commit status named QAEverest / Smart Selection while the tests run, then passed, failed, or error when tests could not run.
  • Each test's result in the bot's comment, with a link to each suite's run history.
  • Runs as one member of the linked project that you choose. Only their API, UI and mobile suites run: one run per suite, at most 20 per push.
  • Picked tests on a teammate's suite are listed as not run.
  • Each suite runs in its last-run environment, or in one you name. A suite with no matching environment is not run, and the check reports an error.
  • Each run uses the organisation's credits, like any run that member starts.
  • A push within two minutes of the last analysis of the same pull request is skipped.
When code and tests drift apart

Fix the tests a change breaks, before and after the merge.

Both work with GitHub and start switched off. A step edit or a proposed fix changes a test only when a person accepts it; after a merge, UI locators are updated on their own.

Suggested updates GitHub

Step edits for the tests a pull request breaks

  • Covers on-screen text, routes and element ids that a change renames or removes, in up to 10 tests whose steps name them.
  • A rename read from the code change costs nothing. For a removed value the AI drafts the edit, charged like generating one test case per test, at most five a run.
  • The test's owner accepts or rejects each edit. One whose step has changed since is shown as Outdated and is not applied.
  • A renamed on-screen label that the requirement itself still uses is flagged, since the code may be what is wrong.

An organisation admin can also have the bot list the suggestions in its comment, which shows the step text to anyone who can read the pull request. That option is off by default.

After the merge GitHub

Check and repair the automation a merge touched

  • Runs when a pull request is merged or, if you choose, once a successful GitHub deployment includes the merge, waiting up to 14 days for one.
  • Checks the API, UI and mobile suites of the tests the change picked, up to 10 per merge, each with a link to its Suite Health report.
  • Each test reads healthy, at risk, broken or orphaned.
  • In a UI suite, a remembered locator whose element is still there under a new selector is updated on its own, and a broken test gets a proposed fix that changes it only when a person accepts it. API and mobile suites get the verdicts only.
  • With pull request checks on, the picked tests run again on the merge commit.

Only suites in the linked project that belong to members of the organisation are checked. Suite Health also runs from the Run Maintenance button on a suite's Suite Health tab.

In your pipeline

Two steps that can fail the build

Each comes as a GitHub Actions workflow, or as a bash step for any other CI. Both need a personal API key.

Check strength

Fail the build on a check that can't fail

Check strength changes the expected value of each check that passed and judges it again against the stored run. A good check now fails; one that still passes would let a wrong answer through. The step re-checks the new and changed tests of an API, UI or mobile suite.

Fail on
a check that can't fail (default), also a weak one, or never
Not evaluated
warn (default), ignore, or fail the build
Cost
free: it works from stored results and sends nothing to your app
AI agent release gate

Gate each release of your AI agent

Use in CI on a saved AI agent suite starts the suite through the public API, against its usual endpoint or a staging one, waits for the verdict and fails the build on a fail or an error.

Default
one failed run of a critical or high scenario fails the verdict, and so does the drift gate once a baseline is pinned
Optional
fail when no baseline is pinned, a minimum trust score, a maximum number of guardrail breaches
Cost
credits like a web run, refunded in full if the run itself fails
Safe on your repositories

Off until you turn it on.

Every option of the PR bot starts switched off, and an organisation admin turns on each part for each code host. This is the GitHub panel as it looks on day one.

PR BotBetaAuto-comments on every PR with risk score, recommended tests & coverage gaps.
  • PR Bot disabled
  • Draft tests for untested requirements on each PR
  • Suggest updates to existing tests the PR changes
  • Run the recommended tests on every push and report the result on the PR
  • When a PR is merged, check and repair the automation it touches

Drafts wait for review

Test cases the bot drafts are saved for your team to review. The bot never automates them.

Step edits wait for review

A suggested step edit or a proposed fix changes a test only when a person accepts it. Only UI locators are updated on their own after a merge.

Runs as a member you pick

Pull request checks run as one member of the linked project, on the organisation's credits.

Your organisation only

The bot reads only a project of its own organisation and runs tests only as one of its members.

Tokens stay on the server

Saved host tokens show only as a mask. Webhook secrets are stored encrypted.

Only the saved repository

On GitLab and Bitbucket the bot uses the repository and token in its settings, never one a webhook delivery names.

Give every pull request the tests it needs.

Link a repository to a QAEverest project, turn on the PR bot, and read its first risk score on your next pull request.