Somewhere in your suite there is a test that has passed exactly once.
It registers a new user. The email is written into the test: sam.test@example.com. The first time anyone ran it, it worked beautifully. Every run since has failed on the same line.
Email already registered.
Nobody writes that test badly on purpose. It's correct in the way a recipe is correct — it just assumed the kitchen would be empty when it started. The second run inherits the first run's kitchen.
Whatever the fix was, you already know it, because everyone reaches for the same one: stick a timestamp on the email. And it works. The test goes green and stays green, and two years later your staging database has four hundred thousand users named sam.test+1757... and nobody can find a real account in the admin panel any more.
That's the test data problem. Not a single bug — a shape that repeats, in every suite, in every tool, at every company.
Tests treat data as input. Data is state.
A test step looks like a function call: enter this email, click that button, expect this message. Functions take inputs and give outputs, and you can run them a thousand times.
But the data isn't an input. It's a claim about the state of a system that other tests, other people, and yesterday's run are all also modifying. The test is really saying *"assuming nobody has used this email, and assuming this customer still exists, and assuming their invoice is still unpaid, then…"* — and none of those assumptions is written down anywhere.
They're invisible dependencies. They break silently, they break on someone else's schedule, and when they break the failure points at your test rather than at the assumption.
Most tests people call flaky aren't unreliable. They're under-specified about the world they need before they can start.
The four approaches, and what each one really costs
Every team lands on one of these, usually without discussing it.
| Approach | What it buys | What it costs |
|---|---|---|
| Hardcoded values | Readable, obvious, trivially debuggable | Passes once. Any test that creates something can never run twice |
| Timestamps or random values | Never collides | Never cleans up either. Your environment fills with junk, and any assertion involving a count or a list becomes meaningless |
| Shared fixtures or a seeded database | Fast, realistic, reusable | Two tests eventually want the same record. One mutates it. The other fails for reasons that live in a different file, and now your suite has an order dependency nobody declared |
| A copy of production | The most realistic data you will ever have | A compliance problem in most industries, stale within a week, and enormous. Also: real customers' data in a system with weaker access controls |
None of these is wrong. The mistake is picking one for the whole suite, when different tests genuinely need different things.
Three questions that decide it
Ask these about a single test, not about the suite:
- Does this test need data to *exist*, or to be *unique*? "A customer with an unpaid invoice" needs existence — a shared fixture is fine. "A new registration" needs uniqueness — it must create its own.
- Does this test *change* the data it uses? A read-only test can share anything. A test that mutates must own what it touches, or it will eventually break a neighbour.
- Who cleans up, and when? If the answer is "nobody" or "we do it manually every few months", you've chosen approach two from the table above whether you meant to or not.
That's it. Most test data arguments are really one of those three questions, unasked.
Five rules that hold up
Ask for data by role, not by value. A test should say it needs *a customer with an unpaid invoice*, not *customer 4471*. The moment a specific id appears in a test, that test is pinned to one environment and one moment in time. Roles survive a database refresh; ids don't.
Create through the fastest door, assert through the real one. Setting up a user through twelve UI steps tests the signup flow for the hundredth time, slowly, for a test about something else. Create the state through an API or a fixture; use the UI for the behaviour you're actually testing.
Never assert on a total. "The dashboard shows 5 orders" is a promise about the entire environment, which you do not control. Assert that *your* order appears. Counts are for tests that own the whole database, which in practice means nobody.
Cleanup belongs to whoever created it, and must survive failure. Teardown that only runs when the test passes isn't teardown — the runs that leave mess behind are exactly the ones that failed. And cleanup must be safe to run twice, because one day it will be.
Put setup where it belongs: before the test, not inside it. If the first six steps of a test are arranging the world, those steps aren't the test. They're the preconditions, and treating them as such makes the failure readable — setup failed, or the behaviour failed, and you can tell which at a glance.
The environment is part of the data
Two things will happen to your test data regardless of how carefully you design it.
Someone will refresh staging. Not maliciously — because it drifted, or because a migration needed testing, or because it filled up with four hundred thousand timestamped users. Every test that relied on a record existing goes red at once, and the team spends a morning deciding whether the product broke.
And someone will use your test account by hand. A developer reproducing a bug, a salesperson recording a demo, support checking something. They'll change a setting, cancel an order, or trip a rate limit, and a test will fail at 3am for a reason that no log will ever explain.
You can't prevent either. What you can do is make tests that fail *loudly and specifically* when their preconditions aren't met, rather than failing on step nine with a missing button. A test that checks its own assumptions first costs three extra seconds and saves the triage.
Where this is genuinely hard
Not every case yields to good practice, and it's worth being honest about which.
Data that takes time to exist. A ninety-day-old invoice, a subscription that's been through three renewals, an account in dunning. You cannot create these in a fixture without also owning the clock, and most systems won't let you move it.
One-way transitions. You can cancel an order; you usually can't un-cancel one. Any test that ends in a terminal state consumes a record permanently, which turns a fixture into a queue.
Third-party sandboxes. Payment providers, identity services, shipping APIs — rate limits, fixed test accounts, and no bulk cleanup. Your data strategy is whatever they allow.
Data that carries real risk. Anything that touches production copies of customer records is a governance question before it's a testing question, and the right answer is usually to generate rather than to copy, even at the cost of realism.
For those, the honest approach is a small, documented set of hand-maintained records, treated as infrastructure rather than as test data — reviewed on a schedule, owned by a person, and never assumed to be free.
Test data doesn't get the attention that locators and frameworks get, because it isn't a tool you can adopt. It's a set of decisions, made once per test, that either make the suite repeatable or quietly make it disposable.
But it's usually the real answer to the question teams keep asking about their automation: *why does this pass on my machine and fail in CI?* Nine times in ten, nothing about the code is different. Something about the world is.
