A unifying idea

One of the most important ideas in software development today:

Automated tests.

Which most people are not enthusiastic about. I said 'important', not 'sexy'. But it's one of those things that is important when you are writing code by hand, and even more important when you use coding agents.

The old school forms - unit tests, integration tests, and end-to-end tests - are critical for coding agents to work at all. The agentic loop tends to heavily rely on the test suite, as a feedback signal when iterating to its goal.

But now we also have 'evals', which is the name that has stuck for a new form of automated test. One that attempts to measure, in more or less objective ways, how well the coding agent is performing.

This brings a new set of distinctions.

Like the idea of 'outcome evals', which attempt to measure "did this coding agent even fulfill this requirement?"

And also 'trajectory evals', which check whether the coding agent used a good process.

Which are useful in different ways. Telling you different things about how good (or not) an agent skill is, for example.

I've even seen evals that use deterministic unit tests as part of their evaluation process. So these fold into themselves in fascinating ways.

There's a lot going on here, as you can see. And if it seems overwhelming... Well, you're not the only one who is overwhelmed.

The good news is that Boosted Code exists. See how it helps here: