The 'too cheap to worry about' factor

Lots happening in the craft of engineering right now. And one factor I have been thinking about recently:

The economics of running code.

Let me explain this one. And this is a higher-IQ article, so buckle in:

Think about a straightforward unit test. One that calls a function, gets its return value, and asserts it is correct. Running this unit test is dirt cheap - in not one, but in TWO ways:

  1. It costs no money. Just the electricity to run your laptop.

  2. It is FAST. You get the answer quickly. So it is cheap in terms of TIME.

Compare this to the newest form of automated test, called "evals". The more nondeterministic, more statistical cousin of unit tests, that are used to grade the results of AI outputs... anything from a chatbot application, to an agent skill you have made, and want to include tests for.

There are several ways to do it, but typically an eval uses an LLM as a 'judge'.

For example, suppose your team has a CI/CD setup, so that every deployed change goes through some automated checks first. And when it fails, you want to categorize the error message into one of these buckets:

And the basic pattern is to feed the CI/CD error message to an LLM, with the prompt: "which bucket is this in?"

So this IS an automated test. But let's compare it to a unit test:

  1. You are making an API call to an LLM, so it costs money. Not much, but not $0 like with a unit test.

  2. You have to wait for the LLM to calculate the result. Even if network latency is zero, that takes longer than a unit test.

For just one run of one eval, these don't add up to much...

But in a real software engineering team, you're running tons of them, over and over. All day, every day.

And THAT adds up. Making evals more expensive than unit tests, both in money and in time. No bueno, because it pressures you NOT to make evals in the first place.

This is where decision models like Jev come in.

Because decision models are not LLMs. They are much more limited in what they can do.

But they are hyper-optimized to answer questions like "which bucket is this in?", doing so much cheaper and much faster. And that makes them a perfect fit for evals.

You do not hesitate to write unit tests, and run them constantly. Doing so is too cheap to worry about. No so with LLMs; they do not quite cross that 'too cheap to worry about' threshold, when you have enough of them at scale.

But decision models are so cheap and fast, that even if they are not actually free, they practically speaking are too cheap to worry about. They DO cross that threshold, more often than LLM-based judges do.

And I write about this 'too cheap to worry about' idea, because it came up in a recent Boosted Code group session. Among the other discussions about decision models, evals, etc. in the group chat.

It's the best place to be right now. Read about BC here:

Let me explain this one. And this is a higher-IQ email, so buckle in:

Think about a straightforward unit test. One that calls a function, gets its return value, and asserts it is correct. Running this unit test is dirt cheap - in not one, but in TWO ways:

  1. It costs no money. Just the electricity to run your laptop.

  2. It is FAST. You get the answer quickly. So it is cheap in terms of TIME.

Compare this to the newest form of automated test, called "evals". The more nondeterministic, more statistical cousin of unit tests, that are used to grade the results of AI outputs... anything from a chatbot application, to an agent skill you have made, and want to include tests for.

There are several ways to do it, but typically an eval uses an LLM as a 'judge'.

For example, suppose your team has a CI/CD setup, so that every deployed change goes through some automated checks first. And when it fails, you want to categorize the error message into one of these buckets:

And the basic pattern is to feed the CI/CD error message to an LLM, with the prompt: "which bucket is this in?"

So this IS an automated test. But let's compare it to a unit test:

  1. You are making an API call to an LLM, so it costs money. Not much, but not $0 like with a unit test.

  2. You have to wait for the LLM to calculate the result. Even if network latency is zero, that takes longer than a unit test.

For just one run of one eval, these don't add up to much...

But in a real software engineering team, you're running tons of them, over and over. All day, every day.

And THAT adds up. Making evals more expensive than unit tests, both in money and in time. No bueno, because it pressures you NOT to make evals in the first place.

This is where decision models like Jev come in.

Because decision models are not LLMs. They are much more limited in what they can do.

But they are hyper-optimized to answer questions like "which bucket is this in?", doing so much cheaper and much faster. And that makes them a perfect fit for evals.

You do not hesitate to write unit tests, and run them constantly. Doing so is too cheap to worry about. No so with LLMs; they do not quite cross that 'too cheap to worry about' threshold, when you have enough of them at scale.

But decision models are so cheap and fast, that even if they are not actually free, they practically speaking are too cheap to worry about. They DO cross that threshold, more often than LLM-based judges do.

And I write about this 'too cheap to worry about' idea, because it came up in a recent Boosted Code group session. Among the other discussions about decision models, evals, etc. in the group chat.

It's the best place to be right now. Read about BC here:

About Boosted Code