Testing in Threes

One of the reasons that I find immutable tests intriguing as an idea: if you don't change them, you can't break them.

What does it mean for a test to break?

There are two failure modes; a test can fail even though the implementation satisfies the specification that the test is supposed to evaluate, or the test can pass even though the implementation does not satisfy the specification.

If we want to refactor tests safely, then we really need to have checks in place to protect against these failure modes.


My first thought was that we need to keep the old implementation around.  For instance, if we are trying to fix a bug in a released library, then we write a new test, verify that the new test fails when bound to the broken implementation, then bind the test to the current implementation, do the simplest thing that could work, and so on.

Kind of heavy, but we could make that work.  I don't think it holds up very well for the ephemeral versions of code that have between releases.

What we really want are additional checks that are part of the specification of the test.  Turtles all the way down!  Except that we don't need to recurse very far, because the additional tests never need to complicated.  Throughout their lifetime, they are "so simple that there are obviously no deficiencies."

Today's insight is that we are already creating those checks.  Red Green Refactor is an recipe for implicitly creating, in order

1. The simplest thing that could possibly break
2. The simplest thing that could possibly work
3. The production thing.

So at the end, we had all three of these, but because they were the same mutable entity the intermediate stages are no longer at ready hand.

I haven't finished untangling this snarl yet.  My best guess is that it is leading toward the idea that test driving the implementation is a spike for the specification, and that we later come back and make that robust.