Pull Request & CI Testing — Frequently Asked Questions

What does the PR bot do?

It reads each pull request, scores its risk out of 10 and comments with the tests worth running for that change, each with the reason it was picked. It also names the requirements the change touches that have no tests, and those whose tests show no current evidence. It edits the same comment when new commits arrive.

Which code hosts does it work with?

GitHub, GitLab at gitlab.com or a self-managed server reachable from the internet, and Bitbucket Cloud. On GitLab it comments on merge requests. Bitbucket Data Center is not supported.

How do I set it up?

An organisation admin sets it up with a webhook and the host's token. On GitHub it can cover several repositories, each read against its own project. It can be limited to pull requests into chosen base branches, such as release branches. Every option starts off, and the settings are marked Beta.

What is Smart Selection?

It reads the change itself: the words in the changed code, the endpoints it defines or calls, element ids, on-screen text and database tables. It matches them to your requirements and to what each linked test calls, and picks tests until they cover 95% of the risk, always keeping critical tests and any test the change touches directly. Times come from each test's own recent runs, and the page says how many were measured rather than estimated.

Can I use Smart Selection without the bot?

Yes. Open a GitHub pull request on the Smart Selection tab of the Traceability page: a public one, or a private one your saved GitHub token can read. Tick the picks in your own suites and start them from the same page, one run per suite. Every analysis, from that tab, the PR bot or the Copilot, is kept on the Traceability History tab, where anyone in its project can reopen it.

Can the picked tests run as a check on the pull request?

Yes, on all three hosts. Turn it on in the PR bot settings and the picked tests run on every push:

• Status — a pending commit status named QAEverest / Smart Selection while the tests run, then passed or failed, or error on GitHub when tests could not run. GitLab shows that as failed, and so does Bitbucket if a suite ran; if none ran, Bitbucket gets no status and only the comment says why. Each test's result goes in the bot's comment.
• Who it runs as — one member of the linked project that you choose. Only their API, UI and mobile suites run: one run per suite, at most 20 per push. A picked test on someone else's suite is listed as not run and does not fail the check, so a check can pass with none of its picks run.
• Environment — each suite runs in its last-run environment, else the member's most recently updated environment of that type, or in one you name. A suite with no matching environment is not run, and the check reports an error.
• Credits — each run uses the organisation's credits, like any run that member starts.

A push within two minutes of the last analysis of the same pull request is skipped.

Can it write tests for what a pull request leaves untested?

Yes. Draft tests writes up to five test cases for each of up to five untested requirements, charged like generation and saved for your team to review. The bot never automates them. From an analysis you ran, Automate builds a requirement's drafts into a new suite named after the pull request and the requirement. Nothing checks that the drafts were reviewed, so read them first. Building it is free, and it is not run for you.

Can it update tests a pull request breaks?

On GitHub, yes. Suggest updates covers on-screen text, routes and element ids that a change renames or removes, in up to 10 tests whose steps name them, each edit shown before and after. A rename read from the code change costs nothing. For a removed value the AI drafts the edit, charged like generating one test case per test, at most five a run.

The test's owner accepts or rejects each edit. One whose step has changed since is shown as Outdated and is not applied. A renamed on-screen label that the requirement itself still uses is flagged, since the code may be what is wrong.

Can it check my automation after a merge?

On GitHub, yes. When a pull request is merged or, if you choose, once a successful GitHub deployment includes the merge (waiting up to 14 days for one), the bot runs Suite Health on the API, UI and mobile suites of the tests the change picked, up to 10 per merge, each with a link to its report.

In a UI suite, a remembered locator whose element is still there under a new selector is updated on its own, and a broken test gets a suggested fix, which changes nothing until a person edits the test. API and mobile suites get the verdicts only.

Which steps can fail my CI build?

• Check strength — re-checks the new and changed tests of an API, UI or mobile suite and can fail the build on a check that can't fail, also on a weak one, or never. It is free: it works from stored results and sends nothing to your app.
• AI agent release gate — Use in CI on a saved AI agent suite starts the suite through the public API, waits for the verdict and fails the build on a fail or an error. It is billed in credits like a web run.
• Quality gate — a step can read the release gates from GET /quality/gate. If QAEverest itself hits an error, the answer is usually passed, so an outage does not block the build. See the Quality Center tab.

The first two come as a GitHub Actions workflow or a bash step for any other CI. All three need a personal API key.

Can I start a suite from my pipeline or on a schedule?

Yes. Suites can run on a schedule, using the time zone of the person who set it, or from a pipeline through the public API. A run in which no test case ran is shown as Failed in Suite History. Hover over the status to see that no test case ran, and the error if one was recorded.