Test the agent,
not just the model.
Give QAEverest your conversational agent's endpoint. It writes test scenarios from your policy, drives real multi-turn conversations, attacks your guardrails, and scores every reply for accuracy, tool-call correctness and PII leakage. Then it gates the release, in the app or in your CI pipeline, with no SDK and no code access.
Your agent's API key is used for the run only and never stored.
Six ways your agent can fail — covered.
Pick any of the five test suites for a run. Every run is checked for drift against your pinned baseline.
Response accuracy
Factual questions are written from your policy, each with its correct answer. Exact codes and amounts are checked by rule; a judge model reads the whole conversation for the rest.
Guardrail enforcement
Probes discount abuse, unauthorized actions and out-of-policy promises. Judge lenient, balanced or strict, where strict demands an airtight refusal with no partial help or workaround.
Tool-call validation
Checks the agent called the right tool with the right arguments, by rule and before the judge, catching broken integrations and silent tool failures.
Red-team & jailbreak
Up to 60 attacks, tagged with their OWASP Top 10 for LLMs category: prompt injection, jailbreaks, system-prompt leaks, data exfiltration, unsafe output and hallucination traps, including multi-turn escalations, other languages and encoded variants. Some injections ask for a canary code, so an obeyed injection is caught by rule.
PII protection
Checks every reply for leaked card, SSN and bank (IBAN) numbers, and privacy scenarios for other people's emails and phone numbers. Your own support contacts are allowed; anything else, such as names and addresses, is left to the AI judge.
Model-drift regression
Re-run the same saved suite after a model or prompt change. The gate blocks on statistically real drops and on hard floors for guardrails (90%) and PII (95%); scenarios that slip without clearing that bar go on a watch list.
Tests that keep up with your agent.
Agents change weekly: a new model, a reworded policy, a fresh tool. Most test suites rot the moment they're written. Ours keep up: saved suites re-run exactly as written, baselines record what good looks like, and scenarios follow your policy when it changes. No model is retrained; only the data around it changes, so results stay reproducible.
Drift baselines
Pin a known-good run and every later run of that suite is compared with it. A baseline judged by a different judge model, rubric or attack set is reported as not comparable, never turned into a misleading diff.
Self-maintaining scenarios
When your policy changes, paste the new version. The suite flags the scenarios it made stale and proposes replacements for you to accept, and never touches the tests you wrote by hand.
An honest judge
Pick the judge model; it's saved with the agent so scores stay comparable. If the judge can't score a reply, the reply is left out rather than guessed, and a run with too many unscored replies fails and is refunded.
Five phases. One endpoint.
From connect to release gate, every step is handled for you.
Connect
Point it at your agent's endpoint (HTTP/JSON, OpenAI-compatible or a custom webhook with your own request template and reply path) or pick a Claude model on AWS Bedrock. A free Test request shows exactly what your agent sends back before any credits are spent.
Generate
Your policy docs or system prompt are turned into scenarios: factual questions with their answers, requests to refuse, tool calls and PII probes. The red-team attack library joins them. Save a run as a suite and it re-runs exactly as written next time.
Execute
Every scenario is driven against the live agent as a real conversation, and repeated in fresh conversations (up to 10 times) so a single lucky reply doesn't count. Up to 8 conversations run in parallel, and a time limit stops a hung agent from holding a run open.
Judge
Rules go first: PII detectors, required and forbidden phrases, injection canaries, tool name and arguments. Then a judge model (Sonnet by default) reads the whole exchange, fenced so the agent's reply can't instruct it, and returns a verdict, a score and a written rationale. A reply it can't score is marked Not scored, never guessed.
Score & gate
Results roll up into an Agent Trust Score and per-suite breakdowns, compared with your pinned baseline. The release gate blocks only on real regressions, in the app or as a failing step in your CI pipeline.
Fail the build, not your customers.
Every saved suite has a Use in CI button with a ready-made GitHub Actions workflow or a bash step for any CI. It starts the run through the QAEverest API, waits for a pass or fail verdict with the reasons, and exits non-zero when your agent has regressed.
Any environment
Run a saved suite against its usual endpoint, or point it at staging for a pre-release check.
Your thresholds
Add a minimum trust score, a maximum number of guardrail breaches, or fail when no baseline is pinned.
Pay for what runs
Credits are reserved up front; you're charged for the turns your agent answered and the judge calls made, never more than was reserved. A run that stops on an error is refunded.
Ship your agent with confidence.
Point QAEverest at your agent for a trust score, a red-team report and a release gate, then put that gate in your pipeline.