Most system testing asks one question. Does it crash.
That question is easy to answer and easy to fix. A crash is loud. It stops the pipeline, it throws a stack trace, someone gets paged, and the fix ships within the day. Crashes are the cheapest failure a system can have, because they cannot hide.
The harder question is the one almost nobody tests for. Does it lie.
Monday's piece covered why a single wrong number can be more dangerous than a crash, because a plausible wrong answer does not announce itself. This is the wider version of that problem. It is not about one bad number. It is about whether the system, as a whole, can prove that any of its numbers deserve to be trusted at all.
Here is what that actually requires, built into EIRL from the ground up rather than bolted on after the fact.
Blu Dot surpasses 2,000% ROAS with self-serve CTV ads
Blu Dot used Roku Ads Manager to drive incredible results for its furniture sales event. Its strategy hinged on custom audiences and retargeting, where intent was strongest.
“Roku has been a top performer,” said Blu Dot’s Claire Folkestad. “We have seen…CPMs lower than any other CTV partner we've worked with.”
Provenance. Every number the system produces has to carry where it came from. Not "from the filing," but which filing, filed by whom, retrieved from which source, at what URL or endpoint, on what date. A number with no traceable origin is not a number. It is an assertion.
Timestamps. Not one timestamp.
Three.
When the underlying event happened. When the data about that event was published. When the system pulled it. Collapsing these into a single date is how point-in-time errors get made silently, the exact failure mode from Monday's piece. A system that only stores "as of today" cannot ever answer a question about last quarter honestly.
Source data preserved, not just the derived result. If the system stores only the calculated output and discards the raw input that produced it, there is no way to audit a wrong answer after the fact. The raw filing, the raw exchange rate, the raw price feed, all of it has to be kept alongside the number it produced, or the output is a black box with a number wearing a nice suit.
Deterministic calculations. Given the same inputs, the system must produce the same output, every time, forever. If a ranking changes between two runs with no change in underlying data, something in the pipeline is nondeterministic, and nondeterminism in a data system is a bug wearing the costume of a feature. Randomness has no place near a number someone is going to make a decision on.
Validation at every boundary. Not just at the end. Every time data crosses from one stage to the next, filing to database, database to calculation, calculation to ranking, it gets checked against expected ranges, expected types, expected freshness. A currency conversion that is off by a factor of ten should never reach the final output. It should get caught at the boundary where it entered.
Audit trail. A record of every transformation a number went through, in order, permanently. Not for compliance theater. For the moment, and it will come, when a number looks wrong and someone needs to trace exactly which step produced it.
Reproducibility. Anyone with access to the audit trail should be able to rerun the exact calculation from the exact inputs and get the exact same answer. If that is not possible, the system's output is a claim, not a result.
Some teams never seem to stop moving. They're on Attio, the agentic CRM.
It’s your always-on revenue engine: agents and workflows build pipeline, chase every buying signal, and move deals forward alongside your team.
Teams like Parallel, Turbopuffer, and Wordsmith build on Attio. Are you one of them?
None of this is exotic engineering. It is unglamorous, and none of it makes a demo look impressive. It is also the entire difference between a system that produces answers and a system that produces trustworthy answers, and those are not the same product even when they display the same number on the same screen.
A research system should not only produce an answer. It should preserve the evidence that makes the answer trustworthy. Speed is a feature. Provenance is the foundation the feature sits on. Most tools on the market skip the foundation because nobody can see it is missing until the day a number is wrong and nobody can explain why.
That is the actual test.
Not does it crash.
Does it lie, and can it prove that it did not.
Thanks for reading
DNSMOS gives you a score, not whether that data fits your model. Voices' CTO DJ Jalali breaks down the four-step framework the team uses to set model-specific quality thresholds instead.