Testing in the Time of Time

Programmer tests - meaning those tests that we run between edits to estimate how many new mistakes we have added to the code - run eye blink fast.

Which means that the lifetime of an entity within our test scenario is an instant.

But the scenario that motivates our simulation often has measurable duration.

If this is to reflect in our design; if the test is to communicate this duration to its future maintainer, then the design of the test needs to imply measurable time.

TDD: Roman Bowling Discussion

I discovered the Roman Bowling concept because I was looking for a familiar example to draw upon when discussing classifications of tests.

Let's start with a trivial test - we want to demonstrate that our code produces the correct output when when input is a representation of a perfect game. Many TDD exercises like to start with a trivial example, but my preference is to skip ahead to something with a bit of meat on it. Certain kinds of mistakes are easier to detect when working with real data.

def test_perfect_game(self):
    from roman import bowling

    pinfalls = ["X", "X", "X", "X", "X", "X", "X", "X", "X", "X",
                "X", "X"]
    score = bowling.score(pinfalls)
    self.assertEqual("CCC", score)

There's nothing here that's particularly beautiful, or enduring. We have a straight forward representation of the behavior that we want, written at a scaffolding level of quality. It's enough to get started, and certainly for this demonstration; the machine understands exactly the check we want to make, but a human being coming into this code cold might want a more informative prose style.

Is this a "unit test"?

The motivation for this exercise is to give some extra thought to what that question means, and which parts of that meaning are actually important.

To get to green from red, we can lean on the simplest thing - simply hard coding the desired output.

def score(pinfalls):
    return "CCC"

This arrangement has a lot of the properties we want of a test to be run after each refactoring: the test shares no mutable state with any other code that might be executing, it has no I/O or network dependencies. The test is eye blink quick to run, and the behavior is stable - if the test reports a problem, we really have introduced a mistake in the code somewhere.

Let's try a quick bit of refactoring, we're going to change this implementation here so that is justly slightly better aligned with our domain model. It's going to be a bit more convenient to work internally with integer values, and treat the romanization of that value as a presentation problem. So what I want to do here is introduce a 300, somehow, and then use that 300 to craft the desired representation.

def score(pinfalls):
    return "".join(["C" for _ in range(300 // 100)])

I find the syntax for joins in python a bit clumsy (I'm not fluent, and certainly don't think in the language), but I don't believe that the code written here would startle a maintainer with working knowledge of the language. Executing the test shows that the machine still understands.

Is it a "unit test"?

Now I want to perform a simple "extract method" refactoring. It may be obvious what's coming, but for this demonstration let's take exaggeratedly small steps.

def score(pinfalls):
    def to_roman(score):
        return "".join(["C" for _ in range(score // 100)])

    return to_roman(300)

Is it a "unit test"?

The implementation of to_roman hasn't become noticeably better; there's a lot more cleanup to be done. But having created the function, I can now start exploring the concept of distance.

def to_roman(score):
    return "".join(["C" for _ in range(score // 100)])

def score(pinfalls):
    return to_roman(300)

Is it a "unit test"?

from previous.exercise import to_roman

def score(pinfalls):
    return to_roman(300)

Is it a "unit test"?

In the first refactoring, the to_roman method definition was nested. Does it matter to us if we nest the import?

def score(pinfalls):
    from previous.exercise import to_roman

    return to_roman(300)

Is it a "unit test"?

Is there any point in this sequence of refactorings where you find yourself thinking "gee, I'm not really comfortable with this; I need to introduce a test double?"

My answers, today? I don't care if it is a unit test or not. I care whether it is still suitable for running between edits, and it still is. The test is still fast and isolated. Sure, roman.bowling.score depends on previous.exercise.to_roman, but to_roman has stable requirements and lots of tests; it's not a significantly riskier dependency than the python library itself.

Introducing a test double to "mitigate the risks" of an external dependency is a lousy trade when the risk of the external dependency is already low.

If we are concerned about having a suite of tests that help us to locate bugs, an interesting alternative to using a test double is write a test that uses a more complicated description of the behavior. In this case, for instance, we might write

def test_perfect_game(self):
    from roman import bowling

    pinfalls = ["X", "X", "X", "X", "X", "X", "X", "X", "X", "X",
                "X", "X"]
    score = bowling.score(pinfalls)
    from previous.exercise import to_roman
    self.assertEqual(to_roman(300), score)

Is it a "unit test"?

I think of this pattern as "behaves like". We aren't saying that roman.bowling.score calls previous.exercise.to_roman with a specific input; that's coupling to an implementation detail that we might reasonably want to change later. Instead, we are adopting a looser constraint that the code should produce the same behavior. The test and the code might both depend on the same external dependency, or not.

Taken to extremes, you end up with something like an oracle - a complete reference implementation in the test, and an assertion that the test subject behaves the same way for all outputs. But be careful, somewhere in the mix you want at least one test to ensure that you haven't inadvertently introduced the same bug into both implementations.

TDD: Roman Bowling Challenge

An adaptation of familiar coding problems.

Martin and Koss introduced bowling scoring as a demonstration piece for how tests could fit into the programming process. In the distilled version, which helps to limit the required time, we focus our attention on computing the score of the game after it has been completed.

In May of 2001, Kent Beck and Alan Francis demonstrated the Roman numerals conversion exercise to a live audience.

So...

Let's design a program that compute the final score of bowling games, where the inputs (pin falls) and the output (final score) are expressed using Roman numerals.

Discussion

Notice the character of the tests that you write. Are they unit tests? acceptance tests? something else?

Are you working top down? bottom up? something else?

Bonus Challenge

Bowling is much more exciting in the original Klingon. Can you extend your solution quickly enough to capture this vibrant new market?

Discussion....

Lessons from a skeleton

I've been working on a vanity project recently; as part of the exercise, I decided to try to built out a walking skeleton. The basic idea behind a walking skeleton is to front load a bunch of plumbing concerns. The Ur skeleton is probably Hello World - in effect a warm up exercise to ensure that we can compile a program, link with our I/O library, run the result. In 2020, we normally add into this a CI or CD pipeline, source control, and so on. In my case, I'm working toward a REST API with serverless reference implementation, so the walking skeleton includes configuring a number of AWS offerings. First lesson: I targeted an transient deliverable. I had considered obeying the testing goat and writing a django implementation, then doing a lift and shift. The problem is that we are now front loading the wrong pain -- I don't care, in the long term, about the leaky abstractions I'll encounter in Django because that is just a temporary structure. The lessons of Django are unlikely to transfer to the intended staging, so I might as well skip that pain and move more directly toward my destination. Second lesson: in so far as is possible, work inside out.

Everything should be built top-down, except the first time.

Walking skeleton is very much a "the first time" situation; there's a lot of comfort to be had in introducing a single new element to something that has been proven to be sound. Working outward tends to concentrate the uncertainty cloud in the new elements, because the lower level traps have already been discovered. Third lesson: manual first. We're not done until we've achieved our goal of a one click deploy. BUT working through the skeleton manually first helps to clarify the automation requirements. This is a form of separation of concerns; we start out with two problems -- one is to align our own custom design into the general purpose model of our supplier, the other is to automate the steps of the general purpose model. Keeping these two ideas separate allows us to bring our concentration to bear on a single concern.

Banking: Skip to the Owl

I've been looking at the Codurance Banking exercise this evening:

Given a client makes a deposit of 1000 on 10-01-2012
And a deposit of 2000 on 13-01-2012
And a withdrawal of 500 on 14-01-2012
When they print their bank statement
Then they would see:

Date       || Amount || Balance
14/01/2012 || -500   || 2500
13/01/2012 || 2000   || 3000
10/01/2012 || 1000   || 1000

which is introduced with an additional interface constraint

public interface AccountService
{
    void deposit(int amount); 
    void withdraw(int amount); 
    void printStatement();
}

Here are the things that I see when I look at this problem.

First, I notice that we have dates. That implies that somewhere I'm going to need a calendar or a clock, because my interface doesn't have way to provide a date. Similarly, I'm going to need some sort of output device that accepts the statement. I'm going to need some way to capture the dates and amounts so that they are available to be printed - a storage data structure of some form. And I'm going to need some functional compute to actually do the work.

Let's draw the rest of the owl!

class TheRestOfTheOwl implements AccountService {
    interface StatelessCore {
        void deposit(int amount, Clock clock, Storage storage);
        void withdraw(int amount, Clock clock, Storage storage);
        void printStatement(Storage storage, Console console);
    }
    
    private final Clock clock;
    private final Storage storage;
    private final Console console;
    private final StatelessCore core;

    TheRestOfTheOwl(Clock clock, Storage storage, Console console, StatelessCore core) {
        this.clock = clock;
        this.storage = storage;
        this.console = console;
        this.core = core;
    }

    @Override
    public void deposit(int amount) {
        core.deposit(amount, clock, storage);
    }

    @Override
    public void withdraw(int amount) {
        core.withdraw(amount, clock, storage);
    }

    @Override
    public void printStatement() {
        core.printStatement(storage, console);
    }
}

You could write tests for that, but I don't see how you are going to get a lot of return out of that investment.

Now, let's suppose you didn't have the insight that Storage is a separate concept from the working core; you might have guessed that you could keep everything in an in-memory data structure, for example, because in your independent tests the lifetime of "the" account is coincident with the test universe itself. So you might start with a different owl:

class SimpleOwl implements AccountService {
    interface StatelessCore {
        void deposit(int amount, Clock clock);
        void withdraw(int amount, Clock clock);
        void printStatement(Console console);
    }

    private final Clock clock;
    private final Console console;
    private final StatelessCore core;

    SimpleOwl(Clock clock, Console console, StatelessCore core) {
        this.clock = clock;
        this.console = console;
        this.core = core;
    }
    
    @Override
    public void deposit(int amount) {
        core.deposit(amount, clock);
    }

    @Override
    public void withdraw(int amount) {
        core.withdraw(amount, clock);
    }

    @Override
    public void printStatement() {
        core.printStatement(console);
    }
}

Riddle: what happens now when you discover that storage should be a separate component?

One possible answer is to recognize that SimpleOwl is simple enough to throw away when it isn't useful any more.

Another possibility is to introduce an adapter....

class Adapter implements SimpleOwl.StatelessCore {
    final Storage storage;
    final TheRestOfTheOwl.StatelessCore core;

    Adapter(Storage storage, TheRestOfTheOwl.StatelessCore core) {
        this.storage = storage;
        this.core = core;
    }

    @Override
    public void deposit(int amount, Clock clock) {
        core.deposit(amount, clock, storage);
    }

    @Override
    public void withdraw(int amount, Clock clock) {
        core.withdraw(amount, clock, storage);
    }

    @Override
    public void printStatement(Console console) {
        core.printStatement(storage, console);
    }
}

Note that this illustrates that "Stateless Core" is a really lousy interface name; Adapter is a core that has state. Naming is one of the two hard problems.

You'd see a similar scramble when you realize that the description of your use case doesn't have any sort of representation of a unique account identifier to ensure that you retrieve the correct information from shared storage.

Refactoring: Reads Before Writes

One of the advantages of practice coding is that it gives you permission to pause your progress and investigate process smells; did I just make a mistake?

Today, I was working on a refactoring exercise; I had code, and tests in place, and all of my tests are passing.  I made a change, ran the tests, and tests failed.  Startled, I reverted the change.  The tests are now passing again.

But I had to stop here, because I happened to notice a problem: the change that I reverted to pass the test was correct.

What had happened?  Well, that's easy -- one of my earlier edits included a mistake, because of the mistake, the behavior of the code was incorrect.  But the test didn't detect the mistake at the point that it was introduced.  Instead the test appeared later.

This concerns me, because one of the illusions that I carry around with me is that there is a payoff in TDD that you spend less time in the debugger because mistakes are caught when you make them.   Test-commit-revert is, to some degree, founded on this idea -- that you can discard the mistake by simply reverting the code.

Assuming, for the moment, that all other things are equal, I prefer habits that produce short intervals between the mistake and its detection to those that produce longer intervals.

So, what happened today?

(There would still be interesting adjustments needed with an integer-based type, but there'd be fewer. Just go with me here.)

The difference between our newly-imagined design and our existing one is technical debt.

8/22

— Ron Jeffries (@RonJeffries) November 12, 2019

I was working on a greenfield project; write a test, make it pass, clean it up, repeat. And my "imagined design" was not expressed in the code. The changes I want to introduce appear to be getting harder, so I'm interrupting my "progress" to make the next change easy.

My imagined design is a finite state machine, and I'm trying to organize the code in a way that introducing a new state (with new behavior) will be easy.

I'm writing a game, and tracking the token as it moves from the starting square to the second square; the tests verify the descriptions of the squares against a fixed transcript.  The test that I am anticipating moves the token back to the original square, with a different description than was initially displayed.

My thought was to introduce a predicate which is always true (preserving the current behavior), and then to introduce the new test which would pass if the predicate were false.  Thus, I would meet one of my goals: minimizing the time required to complete the "Green" task.

And, having practice that approach several times this morning, it works fine.

But not the first time that I tried it.

The first time through, there were two differences in the technique that I used which both contributed.  One difference was that I typed my code, rather than using the refactoring tools.  That made it possible to introduce an error - a line of code I thought was assigning a value to a variable was in fact a no-op.

The other difference is that I started out my refactoring by introducing the writes; creating the new variable, assigning the data to it.  I introduced the error during this phase, but because the variable is not yet being read anywhere, the coding error I've introduced has no impact at all on the behavior.  So all of the tests continue to pass.  When I introduced the new variable into the predicate, now the bug appears.

On the other hand, on the interations where I started from "If True", mistakes were detected at the moment I introduced them.

It felt a bit like TDD as if you meant it: if True is trivially correct, then true is rewritten into an equivalent comparison between two literal values, not we extract variable on one of those literals, and then that variable can be moved around in the source code.  At each mistake, a simple revert/undo would take me back to not merely a passing state, but actual bug free code that passed its tests.

Another way of considering this lesson is that we should start as close to the actual constraints of the tests, and work backwards from there, gradually moving our changes into the parts of our implementation where we have more freedom.

It's similar in nature to starting with a hard coded return value, and then removing duplication as your refactor your way toward the function arguments.

TDD: Transport Tycoon, Exercise 1

We started a series of DDD coding katas. They focus on the temporal modelling in the domain of Transport Tycoon.

Feel free to join! https://t.co/OPXa29qviT pic.twitter.com/7w7uJIrqsM

— Rinat Abdullin 🐢 (@abdullin) November 10, 2019

I decided that I wanted to take a quick swing at this as a TDD exercise...

Because what we are being presented is the behavior of some pure function, I used a facade as my test subject -- a single function accepting a string argument and returning an integer.  Behind that facade I can refactor my way towards a fully buzzword compliant domain model, but these behavior tests aren't coupled to the model.

That means, incidentally, that these tests aren't some beautiful enduring artifact that describes the model in exquisite detail; they are disposable scaffolding tests.

Because I wasn't sure where I was going, I simply implemented everything in a straight forward way within the facade.  I went three examples in with `if` statements before I started trying to tease out the implicit duplication of the model.  Eventually, all three branches turned into the same code, and then they were simplified to remove that duplication.

It became clear in working on some of the longer problems that I really wanted to have more confidence in the intermediate state.  That insight led me back to the idea that I wanted a "pure function" that could handle each piece of cargo one at time, so that I could track the evolution of the system at each step.  In theory, such a thing can be created via refactoring, but since I wanted confidence in the implementation, I decided to perform a separate TDD run on just that piece, and then verified that the tests against the original facade continued to pass when I applied the refactoring.

When that refactoring was complete, the implementation behind the facade was broken into two pieces -- a state machine to manage the bookkeeping of my "fleet", and pure function that computed transitions from one state to another.

I deliberately declined to maintain CQS discipline; separating the queries from the commands in my bookkeeping component appeared to be ceremony with no particular payoff in the exercise.