Domain Events and DTO Lifetimes

The client sends commands to the write model.  If the write model doesn't understand the messages sent by the client, then (as far as that client is concerned), the model is effective immutable.  The effective lifetime of the command itself is very brief - we need momentary agreement.

The read model shares projections with the client.  If the client doesn't understand the messages it receives, then (again, from the perspective of this client), the model is write only.  The effective lifetime of the projection is again short; once the appropriate view has been updated in the client, the projection can be discarded - we need momentary agreement.

The write model shares events with the read model, but the pattern doesn't hold.

The distinction is simply this: events persist.

You might need to save off commands in a queue, to ensure that they don't stomp on each other, they may need to be scheduled.  But we know that it has to be ok for commands to evaporate, because fail fast is a correct expression of congestion control when the application will not be able to meet the SLA.

Similarly, you might persist projections; but that's primarily a performance optimization -- when the cache expires the projection, it will be rebuilt.  The client might want to insulate the user from the dynamic nature of the model for a time, but an eventually consistent view will eventually change.  That's its nature.

Events are more than just a representation of change pushed across the boundary between the write model and the read model.  It also cross the boundary between the write model of today, and the write model of the future.

In particular, that means that putting domain objects directly into the representation of the event is dangerous, because we expect to be aggressively and continuously refining the domain model as we learn more and more about it.  In other words, the instability of the domain model in the scale of product lifetime cautions us against mapping our persistent messages too closely to the domain.

We need to prepare for event streams that include multiple instances of the same event emitted by different versions of the model.  Which suggests that, for each message in the stream, we'll need a hint in the meta data that indicates the proper recipe for restoring the domain event -- as the model in the past would have written the event knowing when it was going to be read in the future.

Avro? and tag every event in the history of the model with the writer schema of that time?  Thrift/ProtocolBuffers, and hope that the evolution of the events can be supported entirely by non destructive schema changes?  JSON, because you get the easy part of the answer for free?  Take the hit to upgrade the immutable events in your store, so that all events are taken from the same version of the api?

My best guess today?  You are going to need a schema eventually - this seems obvious to me as soon as other domains start subscribing to these events.

So the early guess is about how much value you can deliver before you take the plunge, and how expensive the first schema migration will be.

Can we enforce the transaction consistency boundaries with interfaces?

Maybe.


Let's first consider the case where we have an aggregate which includes a reference to another aggregate.  That's perfectly reasonable, provided that the business is satisfied that coordinated changes between the aggregates are eventually consistent.

Now, each of the aggregates has their own commands (each changing their own state).  Best practices suggest that we should only be modifying one aggregate per transaction; in other words, we should only be running command(s?) on one aggregate or the other.

Can we organize our code to enforce that?


I've been chewing on a remark from Greg Young, that getters and setters are evil.  Setters, sure -- setters should instead be commands, written in the Ubiquitous Language.  But getters? how on earth are you going to do anything useful with another object if you can't read it?  What are you going to do with a Specification that can't read the object it is supposed to constrain?

I've chosen, for the moment, to understand his comment in this way: getters and setters have no place in the model; getters are perfectly acceptable in an immutable projection.

I'm borrowing these two ideas from Greg; which I believe he lifted from an earlier generation of CQRS experts [wrong - Greg is the earlier generation of CQRS experts].  Commands are sent to the model, which is optimized for validating and calculate all changes.  Queries are sent to a projection -- there can be several -- which is optimized for reads, but may be stale.

So if we send a command to a model, and the execution of that command required state from some other aggregate, then we need to hydrate the appropriate projection of the remote aggregate.

I had been blocked on this until recently, because I couldn't see past needing a getter to obtain the reference to the remote aggregate to do the hydration.

But the answer to that puzzle is to pass a DomainService as one of the arguments in the command.  The root can look up the referenceId without needing to expose it, and pass that value to the service to get back an immutable projection with precisely the data that it needs.

Essentially, we are building into the signature of the command the contract that promises we won't change anybody else.

Two use cases where I need more thought.  The first is factory commands; calls into this aggregate to create a new instance of that aggregate.  The second is a query on this aggregate to run a command on that one.

Another perspective on the problem: if the other aggregate is responsible for a business invariant, then it may throw a checked exception.  I don't see how I can claim to be implementing a query that changes the model (in another aggregate), or an immutable object that throws exceptions.

My guess right now is that You Don't Do That.  Instead, some hand waving happens in the Application Service fronting this mess that gets all the dancers on the correct step.

Are aggregate roots always entities?

Yes.

The critical characteristic of an aggregate root is that it acts as a transaction consistency boundary.  In other words, it is responsible for changes that must always and immediately satisfy some business invariant.

That immediately rules out the possibility that it is a ValueObject, because ValueObjects are immutable.  Any changes to a value reflected in the model are going to introduce a new instance of the value object.

Similarly, DomainEvents are also ruled out -- events are things that happened in the past, and we don't have time machines.

As an aside, I think the DDD Sample gets this wrong; the HandlingEvent aggregate is modeled as a DomainEvent.  The description in the class header is

HandlingEvent's are sent from different Incident Logging Applications

Written in the active voice

Incident Logging Applications send HandlingEvents

and now I'm suspicious.  Does the business really not track the source of these external events?  Notice, also, that we never load a DomainEvent from the repository -- we only collect a history of events that match a TrackingId, which is a value object typically created within cargo.

DomainService is stateless, so there's no need for a transaction.

Sagas? I don't believe that's a fit; long running business processes that span multiple transactions and potentially more than one aggregate.

ApplicationService doesn't fit because the business invariant belongs in the model. 
 

 

A surprise in graphite monitoring

Here at the office, we have been using graphite to collect runtime metrics broadcast by our JVM processes.

Some time back, we discovered that our single graphite collector (carbon-cache) could no longer keep up with the load.  We replaced it with a carbon-relay, which uses consistent-hashing to distribute the work to multiple carbon-cache instances.

To keep an eye on things, I created a dashboard to monitor the activity of the carbon processes.  It was comforting to see the distributed workload, as we went from a cpu bottleneck to an i/o bottleneck.

We later moved the whisper databases to an SSD, removing the i/o bottleneck.

One thing I had noticed - the load on the caches wasn't balanced; one carbon-cache instance was being asked to do more than its share of the work.  I ported the hash ring logic into iPython Notebook to run some experiments, and discovered that my initial scheme for instance naming was pathologically bad.  So the cache instances were renamed, and the balance of the load became a lot more even.

I was disappointed that my graph seemed to be a bit hit or miss on the most recent datapoint for the metrics.  graphite-web will initially try to fetch metrics from disk, but it will then query the caches to look for points available that haven't been flushed to disk yet.  Clearly, something was wrong with the lookup there.

A review of local_settings.py revealed that I did not update the instance names for the CARBONLINK_HOSTS; which means that when attempting to determine which cache should hold a metrics, the webapp was using the wrong hash ring.  This would sometimes work (if the metric name happened to map to the same instance in both schemes, just by luck).  So I scrambled off to update this setting, and restarted the server.

No improvement.  What's going on.

My theory is this: graphite-web is using the hash ring to determine which cache should hold the metric.  But the carbon-metrics are always local to the cache that produces them.  So trying to find them via a hash-ring is going to fail more often than not.  Which means that the webapp cannot read those metrics until they have been flushed to disk.

Assembing a number line

Arrange the numbers 1-17 so that adjacent numbers sum to a perfect square.

Apparently, Matt Parker (@standupmaths) tweeted this, then Megan Schmidt followed up with a report of giving the problem to her remedial high schoolers; then Dan Meyer kicked that into my feed reader, and I had to go take a look.

After picking up my notebook and working through it, I asked myself the obvious question: why 17?  And of course it has the obvious answer -- because 18 doesn't work.

Then I looked at my solution again, and discovered that 14 doesn't work either.  In fact, it seems that 15 is the first number that has a solution.

So I did the obvious thing: I shared it with the guys at work.  The boss has an interview question of the flavor "show me how you would solve..."  and so we attacked it with that idea in mind.  After confusing ourselves on the whiteboard for a bit, we began to understand that numbers of the form 2n² are a problem, because they can't pair with themselves to produce (2n)²...

It's "just" a problem of finding a Hamiltonian path through a graph where each number is uniquely represented by a node, and edges exist only if the two connected nodes sum to a perfect square.  You don't add very many edges when you add a new node to the graph, so brute forcing it in code shouldn't be too difficult.

But where's the fun in that?

Text output in Map/Reduce

This evening's exercise was a spike, trying to evaluate whether I could get a significant improvement in task performance by exerting more control over the write of the data.

Conclusion: don't bother.

The job in question was a very thin one - take binary data, and write it out as human readable text.  In any significant map/reduce job, the actual mapping and reducing is going to swamp the time required to copy bytes of data around.

That said, there were interesting things to be learned in the exercise.

The actual logic of the write - the point where the data irrevocably passes from your key/value to your job output - is controlled by FileOutputFormat.getRecordWriter.  This is an abstract method; the implementation in this specific instance is provided by TextOutputFormat, which creates a TextOutputFormat.LineRecordWriter.  LineRecordWriter.write(K,V) in turn calls LineRecordWriter.writeObject.

Now the fun begins.  writeObject is specifically looking for Text; if it can't cast the Object to Text, then it's going to ensure that the UTF-8 requirement is satisfied by invoking Object.toString(), and then UTF-8 encoding the result.  In other words, TextOutputFormat is constraining the output to data that must be UTF-8 encoded.

For anything that's not a String to start with, that's going to be two copies; String.getBytes() promises to return a new byte array.


So that's not so good.

First thought would be to replace the LineRecordWriter with something useful.  TextObjectFormat is tightly coupled to its own LineRecordWriter; there are no seams to inject a replacement.  Creating a RecordWriter that delegates to LineRecordWriter doesn't do any good, since the public interface is fine.  Similarly, you can't fix the problem by extending it, because the problem method is private.


That road blocked, an alternative is to have your type extend Text.  The good news is that writeObject isn't using the Text contract, but instead two methods promised by the BinaryComparable interface.  So we extend Text (getting past the instanceof check in LineRecordWriter), implement these two methods, and we are good to go.

Until somebody tries to write out a container with our object in it.  MapWritable cares a great deal about its keys and values being Writable, and in particular supporting Writable.write(DataOutput).  So we have to replace Text.write(DataOutput) as well, as the provided implementation doesn't seem to know anything about BinaryComparable.

And having done that, things are still broken until we correctly replace all of the Read methods.

But there is another possibility: the public interface of the Text object provides methods for copying into it's own buffer.  That doesn't save you much if you insist on writing the data to your own space first.  But it does give you enough to write an OutputStream that collects data in Text.buf.  And validateUTF8 allows you to enforce the UTF-8 constraint on the streamed data (but not, alas, provide it).