Statistical vs deterministic tests

Automated tests have been important for a long time. Unit tests, integration tests, mocks, and all that.

Lately, another kind of test is emerging. Called an "eval", which is short for 'evaluation'. And it really is a new kind of thing.

Evals come out of the machine learning community, from their efforts to create automated tests for LLMs and other forms of generative AI. Because unit tests are absolutely terrible at that.

A unit test checks for a specific response to a particular input. And what's implied is that if you repeat that input, you get the same response.

We all know that LLMs do the opposite. They will generate a different response every time. Just open two incognito tabs to ChatGPT, and paste the same prompt into both; you'll see how different they are.

Another way of saying this:

Unit tests are deterministic.

And same for the other forms of traditional automated test, like integration tests and so on. There are other characteristics we can point at too - like how they test interfaces rather than internals; they tend to run fast; and more. But in my view, 'deterministic' is the key factor that defines unit tests and their kin.

Contrast to evals, which look like they're from a different world:

• Evals tend to use rubric-based scoring, or statistical thresholds (e.g. 80% accuracy), or some other fuzzy metric.

• They often check what's called "semantic similarity". For example, instead of how a unit test might check whether two strings match, or whether a string matches a regex; an eval might instead check whether a response contains key information, often using an LLM to judge that.

• Evals often use more broadly-worded 'acceptance criteria', similar to instructions you might write out for a QA tester team. Whether the output is factual, coherent, has the right tone, etc. Those are too subjective for a unit test. But not for an LLM, sort of.

Wait, why did I just say 'sort of'?

Because all of these factors are STATISTICAL.

Let's say you have a program that generates text instructions to do whatever. And you want to make sure the output is friendly - rather than rude or hostile or something else that you don't want to inflict on your poor customers. You can ask an LLM something like: "How friendly is the tone of this response on a scale of 1-10?" and get that numerical output. And you decide the test passes if it's at least 7 on that scale.

But since it's an LLM that is judging that number...

Well, that's not always going to behave the exact same way, right? If you run it a dozen times on the same input, it might give you a rating of 9 most of the time, but sometimes 8 or 10, on that one-to-ten scale.

The reason we use evals is because traditional automated tests cannot handle this situation at all.

And so we accept a level of non-determinism in evals. Because that is infinitely more useful than throwing up our hands and giving up on testing completely.

So let me give you some vocabulary:

• DETERMINISTIC tests are the traditional automated tests, like unit tests and so on.

• STATISTICAL tests are the newer, non-deterministic forms, like evals, that often use an LLM or something like it as a judge.

• They are all still automated tests. That is the umbrella term, which includes unit tests and evals and everything in between.

It's interesting how well the decades-old ideas of test-driven development scale to using evals, and other statistical tests. They fit right in. All the hard-won wisdom applies to evals. And if you skip those foundations... well, then the dangers, traps and perilous downsides also apply.

Okay. So how do you ramp up on this?

There are two relevant modules in the Boosted Code curriculum:

• Test-Driven Development in Python
• Agentic Coding Foundations

Both of these are necessary to understanding how to use evals in practice. And I am also creating a new Boosted Code course module on evals, focused on how to leverage them when you use coding agents.

Learn more here: