A short lesson in unit testing.

I've been reviewing Uncle Bob's essay on Dijkstra's Algorithm, and Greg Young's 2011 Probability Kata.  From this, I've taken away a few sharp lessons.

Test Specifications

It absolutely doesn't matter how you write your test specifications. Ideally, you'd like to maintain flexibility, so the notion of having then in a language agnostic representation where anybody can work on them has merit.
If you are seeing references to your current domain model objects in your test specifications, you are way too tightly coupled.

API

• Your test specifications don't talk to your implementations
• Your test specifications don't talk to your implemented apis
• Your test specifications talk to your TEST api

More carefully, your test api interprets the test specifications, and adapts them to the system under test.


My current interpretation is that the test api should consist of three stages -- Given/When/Then is a pretty good approximation.  We need to load the system under test in the appropriate initial state (Given), we need to exchange a message with it (When), and we need to run checks on the message that we receive in the exchange (Then).

Note that last point - we don't want to be poking around in the internals of anything we are testing; we want any probes to be looking a stable interpretation of the api we are testing.

Evolution over time

Worth repeating for emphasis - we should not need to change our checks when the implementation that we are measuring changes.  We're crossing a process boundary, so the messages connecting the test implementation to the subject should be DTOs in both directions.  For any given implementation, the DTOs should be stable: they are effectively part of the API of the implementation.

But the test design should allow us to swap out one implementation for another with an entirely different API.  That will mean finding the new way to load instances of the implementation, and a new adapter of the messages from the specifications language to the implementations API.  The adapter also brings the api language back to the language of the specification, and the checks do not change.

Think HTTP: the web server offers a stable interface against which a specification can be written.  The implementation of the resources on the web server adapt the incoming messages, creating new representations that can be exchanged with the the domain's API.  The domain responds to the web server, the web server transforms the DTO from the domain model to that which conforms to the specification, and the result is sent back to the client.

So when we are looking at the execution of our specifications, we should be able to find a seam; a point in the execution where the specified inputs (Given, When) are passed to the Test API, and a response is returned that can be checked against Then.



Here in Uncle Bob's code, makePathFinder serves as the seam, although it is an imperfect one. (Editorial note: which is fine. Uncle Bob's kata was about test specifications leading to Dijkstra's algorithm. My kata is about test evolution. Horses for courses.).

The main flaw here is the appearance of the PathFinder type, which is taken from the implementation namespace, rather than the specification namespace. We can't see that from the signature alone, but it becomes clear if we drill deeper into the test implementation

When we see that this is a type built from inputs, rather than from outputs, which is a really big hint that something has gone south. Only messages should be crossing the boundary from the solution to the test, but PathFinder is a type with behavior. The abstraction barrier is leaking. We shouldn't need to change our specification execution when the name of the target changes; that's a violation of encapsulation -- we should be able to replace the connection to the real implementation with a lookup table without needing to make any changes in the execution of the specifications.


Sure, you could sexy it up with an immutable message type, or even a EnumMap to ensure that you don't get confused. The point is to keep in mind that you are checking the specification against plain old data - the response cross the barrier from the system under test to the test, and the test api aligns that response with the representations expected by your speficication.

Explicit provenance

I'm working on a rewrite of some existing automation - never where you want to be.  My justification is that the solution we have in place is over fit -- it deliciously low maintenance on the happy path, but absurd to work with in all other circumstances.  Extending it to meet my current requirements promises to absolutely suck, so I'm going the other way around, leave the existing system untouched, prove out the new approach on the new problem, migrate the old approach as circumstances demand.

One of the problems I'm trying to address; in the current implementation, it's very difficult to see the motivation behind what's going on.  In my new work, I'm making a deliberate effort to write things out long hand.

Making things more interesting is the fact that I've chosen to implement the "human operator override" use cases first.  Motivation: if a human must be involved, that makes the exercise appreciably more expensive.  So smoothing that path out -- in particular, making it as easy as possible to remember how the contingency designs work -- is a priority even though the use is rare.

In a manner of speaking, in the first use case the human operator is acting as a surrogate for the automation to follow.  As this exercise is intended to provide a surface for the user, I start with a single input field and a button with VERB written on it in large friendly letters.

I then proceed to hard code into the handler the remainder of the state required to achieve the desired result.  Test.  Green.  Moving on...; there are two problems with the hard coded state that need to be addressed.  First, it needs to be exposed to give the human operator additional control when required.  I had, in fact, already run into problems with this when I tried to use as my input data that was not representative of the happy path.  One simple input field and I had already managed to over fit my solution.

The second is to provide the correct provenance of the data. This begins with meaningful names, but also includes faithfully representing where in the business the data is coming from.  The argument from the input control is a String, but that String is really just a representation of a URL, which provides an address for the projection I'm working with.  But the spelling of the URL is not arbitrary: it in fact expresses within it details about the source that was used to create the projection, or more specifically about the instance of the process that created the source....

With the provenance written out long hand, it becomes much easier to see where the future seams might lie.

It also exposes that the existing process isn't particularly well thought out, that names are not preserved when crossing from one context to another, and quite honestly that the don't align particularly well with the language of the business as we speak it today. 

Set Validation

Vladimir Khorikov wrote recently about enforcing uniqueness constraints, which is the canonical example of set validation.  His essay got me to thinking about validation more generally.

Uniqueness is relatively straight forward in a relational database; you include in your schema a constraint that prevents the introduction of a duplicate entry, and the constraint acts as a guard to protect the invariant in the book of record itself -- which is, after all, where it matters.

But how does it work? The constraint is effective because it blocks the write to the book of record.  In the abstract, the constraint gets tested within the database while the write lock is held; the writes themselves have been serialized and each write in turn needs to be consistent with its predecessors.

If you try to check the constraint before obtaining the write lock, then you have a race; the book of record can be changed by another transaction that is in flight.


Single writer sidesteps this issue by effectively making the write lock private.

With multiple writers, each can check the constraint locally, but you can't prove that the two changes in flight don't conflict with each other.  The good thing is that you don't need to - it's enough to know that the book of record hasn't changed since your checked it.  Logically, each write becomes a compare and swap on the tail pointer of the model history.

Of course, the book of record has to trust that the model actually performed the check before attempting the write.

And implementing the check this way isn't particularly satisfactory.  There's not generally a lot of competition for email addresses; unless your problem space is actually assignment of mail boxes, the constraint has generally been taken care of elsewhere.  Introducing write contention (by locking the entire model) to ensure that no duplicate email addresses exist in the book or record isn't particularly satisfactory.

This is already demonstrated by the fact that this problem usually arises after the model has been chopped into aggregates; an aggregate, after all, is an arbitrary boundary drawn within the model in an attempt to avoid unnecessary conflict checks.

But to ensure that the aggregates you are checking haven't changed while waiting for your write to happen?  That requires locking those aggregates for the duration.

To enforce a check across all email addresses, you also have to lock against the creation of new aggregates that might include an address you haven't checked.  Effectively, you have to lock membership in the set.


If you are going to lock the entire set, you might as well take all of those entities and make them part of a single large aggregate.

Greg Young correctly pointed out long ago that there's not a lot of business value at the bottom of this rabbit hole.  If the business will admit that mitigation, rather than prevention, is a cost effective solution, the relaxed constraint will be a lot easier to manage.

RESTBucks Client

Where are the agents?

Personally, I never understood how overgrowth can be a good thing: 12,000 APIs means 12,000 different ways of engaging in machine-to-machine interaction. -- Ruben Verborgh

 Good to know that other people have tread this ground before I start ranting.

I was introduced to Robert Zubek's needs based ai paper recently, which returned my thinking to a problem I've struggled with in understanding hypermedia.

To wit, what does it mean to have a client that is decoupled from the server?

We know from the experience of the web that hypermedia controls are a big part of it -- allowing the server to communicate the available affordances to the client means that the server can change the how as it likes, and the client doesn't need to survey the possibility space, but can instead limit itself to the immediately relevant set.

So we take a look at a hypermedia driven example -- for instance, Jim Webber's RESTBucks.  The domain application protocol looks something like this:

Riddle: what happens to a client coded to search for these controls if the shop decides to comp your order?  What happens when we shop at Model T Coffee?

You can have your coffee any way you like, as long as it's black.

In either case, if the client is focused on following the protocol, it's screwed.  The trap is the same client/server coupling we fled to hypermedia to escape.  Instead, we should be thinking in terms of multiple protocols -- if RESTBucks happens to be offering a take-free-coffee endpoint today, clients that understand how to negotiate it should be able to understand it.

What I'm beginning to see is that there is a choice; instead of teaching a client that a specific extended protocol will produce a goal, we may be able decompose the protocol into a number of different pieces that can be composed to reach a goal -- if we can manage to describe the affordances in a way that the client can understand, then we can start building complicated protocols out of simpler ones and glue.

Maybe -- we still need a client that can snap together the legos in the right order.

Lessons of Pokémon GO

Chris Furniss

But what Pokémon GO is benefitting from right now is a emergent mentorship that almost completely replaces a traditional tutorial. I would argue that this emergent mentorship is even more powerful than an actual tutorial. Mentorship increases engagement through social bonds. You've probably experienced this already with Pokémon GO: teaching a confused player something about the game that you've figured out already not only makes you feel smart and altruistic, it assuages confusion for the person you've mentored in a way that is highly personalized, and therefore more impactful.

 Interesting to make that behavior part of the interface: "I'm not going to give you a hint, but I will connect you to a couple of nodes in your own social graph that have figured it out."

Data Science in a Nutshell

Data Science in a Nutshell

Generating Type I errors by using second hand data to model third order effects.

The problem with language

The Problem with Language

We only know how to share language with entities that are capable of pattern recognition.