Skip to main content
TelecomAI agents

A support chatbot gave a different answer every run. How do you test that?

An AI assistant's answers change from run to run, so scripted assertions do not fit. Here is how a QA team tests what the assistant says and does, attacks it before users do, and gates each release.

Scenario: A QA team readying an LLM-based billing assistant for launch

Problem

Exact-match assertions failed on correct answers, and prompt injection and personal-data leaks had only been spot-checked by hand.

Solution

Scenarios and a red team generated from the policy, fixed rules before a judge model, repeated runs, and a drift gate in CI.

Capabilities usedAI agent and LLM testingPolicy-based scenariosScoringRepeated runsRelease gate for AI agents in CI

The problem

The assistant answered billing questions and called tools to look up an account or raise a refund. Scripted checks failed whenever a correct answer was worded differently, manual spot checks covered a few dozen questions, and a prompt or model change could quietly make answers worse between releases.

  • Exact-match assertions failed on correct answers phrased differently.
  • Manual spot checks covered only a handful of the questions customers ask.
  • Prompt injection, refusals and personal-data leaks were tested ad hoc.
  • Nothing stopped a model or prompt change from degrading answers between releases.

The solution

The assistant is tested from the outside, the way customers reach it, with no change to its code.

  1. Connect through the endpoint it already has

    AI agent and LLM testing

    QAEverest calls the assistant's HTTP endpoint — a JSON API with your own request template and reply path, or a standard chat-completions endpoint — with no SDK and no access to its code. A free Test request shows what the agent sends back before any credits are spent.

  2. Generate scenarios and a red team from the policy

    Policy-based scenarios

    From the policy or system prompt, it writes factual questions with their answers, requests the agent must refuse, expected tool calls and personal-data probes, and adds a red team of 7, 30 or 60 attacks — most tagged with their OWASP Top 10 for LLMs category — including multi-turn escalations and attacks in Spanish and Hindi, plus encoded variants in the 60-attack set.

  3. Rules first, then one judge

    Scoring

    Fixed rules decide first: a real card, SSN or IBAN number, an obeyed injection, the wrong tool or arguments. Then one judge model, which you can choose, reads the whole conversation and writes its reason. A reply it could not score is marked not scored, never guessed, and a reply it was unsure of is held for a person.

  4. Repeat to account for variation

    Repeated runs

    Each scenario is repeated in fresh conversations, and a critical or high scenario fails on a single failed run.

  5. Gate every release in CI

    Release gate for AI agents in CI

    A saved suite re-runs exactly as written against a pinned baseline, and the drift gate blocks only drops unlikely to be chance. Use in CI gives a ready GitHub Actions workflow or a bash step that fails the build on a fail or an error.

What changes

  • Answers are judged on whether they were right and the rules held, not on exact wording.
  • Injection, refusal and personal-data probes run on every release instead of when someone remembers.
  • A prompt or model change has to clear the same gate as a code change.
  • Uncertain verdicts go to a person, whose pass-or-fail call is recorded without changing any score.

Read more

Facing a problem like this?

Bring one of your own stories, suites or release questions to a demo and we will walk through it in the product with your QA team.

Book a demo Start free

More case studies

B2B SaaSScenario

A nightly suite that was red so often, its results stopped being read

Problem

The same few tests failed most nights, real regressions hid among them, and a screenshot showed that a step failed but not why.

Solution

Flaky tests separated from regressions, failures replayed with an AI root cause, and seeded defects that show what the suite really catches.

Read the case study