Fixtures

A fixture is a known, controlled set of test data created on purpose so a system can be evaluated against it repeatedly and produce reproducible results. The name comes from the idea of fixing the environment in a known state—locking down the inputs so the only thing that varies between test runs is the thing being evaluated.

What fixtures look like

• For a tax-calculation library, a fixture might be a sample W-2 (individual tax document) with known numbers, so the library can be checked to return the right refund every time.
• For a database app, a fixture might be a sample database with 100 known users and 50 known orders, so tests can assert that deleting user X also removes their orders.
• For an AI agent that renames files, a fixture might be a synthetic folder of fake Schwab and PGE statements—because a rename workflow can't be tested without files to rename.

Why fixtures exist

Real data is messy, sensitive, and changes over time. Running an evaluation against someone's actual brokerage folder would mean (a) the test isn't repeatable because the folder keeps changing, (b) the test touches real data, and (c) the test doesn't generalize beyond that one folder. A clean synthetic fixture sidesteps all three problems.

Fixtures aren't unique to AI work—they come from decades of software testing practice. But they matter more in AI because AI outputs are non-deterministic and hard to evaluate by eye. Without a fixture, "did this change make things better?" is unanswerable: better than what, against which inputs?

Implications for transparency

The best fixtures look real enough to exercise the system the way real data would. That same realism is exactly what makes it ambiguous whether someone is looking at fixtures or actual user data—and that ambiguity is where trust slips.

A trustworthy AI agent labels fixtures clearly before generating them, not after the user notices. The working standard: if someone could mistake the synthetic data for real data without checking, the labeling needs to be louder. File names, folder names, content, and the agent's own narration should all make the fixture status obvious.

This extends a basic right for anyone working with an AI agent: when an agent says "I created some test data," the user is entitled to ask where it came from. A well-designed agent should have already volunteered the answer.

Implications for consistency, grounding, and steering

Fixtures are one of the main ways AI systems get pointed at consistent behavior. They turn evaluation from anecdote into measurement.

• Consistency. Running the same fixture before and after a change is how regression is caught. If the agent renamed Schwab statements correctly yesterday but not today, the fixture surfaces the break.
• Grounding. Fixtures give an AI system something concrete to be right or wrong about. Abstract claims like "the agent handles financial documents well" are unfalsifiable; a fixture makes the claim testable.
• Steering. Well-designed fixtures encode what "good" looks like. Each fixture is implicitly a values statement: these are the inputs we want the system to handle, and these are the outputs we'd accept. Choosing what to put in a fixture is choosing what the system should be optimized for.

Pillar connection

Fixtures live primarily in the tools and skills pillars—they're a practical technique for evaluating AI systems—but they have a foot in thinking too. Asking "what would a fixture for this look like?" is a way of forcing concreteness onto otherwise vague claims about AI capability. It's also a habit worth building when consuming AI marketing: if a vendor says their agent "handles X," the next question is what fixture they tested against.

The takeaway

Anytime someone—human or AI—claims a system "works," it's fair to ask what it was tested against. If the answer is "I tried it and it looked right," that's anecdote. If the answer is "here's the fixture and here's the score," that's grounding. Fixtures are the difference between trust based on vibes and trust based on evidence.