HTML5 forms with base.href

I'm working on a REST API, and to keep things simple for myself I'm using text/html for my primary media type.  The big advantage is that I can use a general purpose web browser to test my API by hand.

In this particular case, the client communicates with the API via a services of web form submissions.  That allows my server to control which cached representations will be invalidated when the form is submitted.

As an experiment, instead of copying the target-uri into the form actions, I decided to try using a BASE element, with no action at all, expecting that the form would be submitted to the base href.

But what the HTML 5 specification says is:

If action is the empty string, let action be the URL of the form document.

So that doesn't work.  Via stack overflow, I discovered a candidate work around - the target of the form can be a fragment; the fragment delimiter itself means the form action is not empty, and therefore relative lookup via the base element is enabled.

Somewhat worrisome: I haven't managed to find clear evidence that form behavior with fragments is well defined.  It occurred to me that perhaps  the standard was playing clever with URL vs URI, but I abandoned that idea after clicking a few more links to review the examples of URLs, and discovered fragments there.

I suspect that I need to let go of this idea - it's not that much extra work to copy the base uri into the forms that want it, and I've got a much clearer picture of what happens in that case.

REST: Fielding 6.5.2

Reviewing Fielding's thesis yet again, I noticed a sentence that I think has not gotten as much exposure as it should

What makes HTTP significantly different from RPC is that the requests are directed to resources using a generic interface with standard semantics that can be interpreted by intermediaries almost as well as by the machines that originate services. -- Fielding

Emphasis added.

If I'm doing it right, I can put a commodity cache in front of my origin server, and it will be able to do useful work because it understands the semantics of the meta data in the HTTP traffic.  The cache doesn't need any special knowledge about my domain, or the payloads being exchanged.

REST, Resources, and Flow

It occurred to me today that there are two important ideas in designing a REST API.

The more familiar of the two is the hypermedia constraint.  The client uses a bookmark to load a starting point, and from there navigates the application protocol by examining links provided by the server, and choosing which link to follow.

That gives the server a lot of control, as it can choose which possible state transitions to share with the client.  Links can be revealed or hidden as required; the client is programmed to choose from among the available links.  Because the server controls the links, the server can choose the appropriate request target.  The client can just "follow its nose"; which is to say, the client follows a link by using a request target selected by the server.

The second idea is that the request targets proposed by the server are not arbitrary.  HTTP defines cache invalidation semantics in RFC 7234.   Successful unsafe requests invalidate the clients locally cached representation of the request target.

A classic example that takes advantage of this is collections: if we POST a new item to a collection resource, then the cached representation of the collection is automatically invalidated if the server's response indicates that the change was successful.

Alas, there is an asymmetry: when we delete a resource, we don't have a good way to invalidate the collections that it may have belonged to.

On HTTP Status Codes

Originally written in response to a question on Stack Overflow; the community seemed to think the question wasn't appropriate for the site.

Overview of status codes

I'm designing a RESTful API and I have an endpoint responsible for product purchase. What HTTP status code should I return when user's balance is not enough to purchase the specified product (i.e. insufficient funds)?

The most important consideration is that you recognize who the audience of the status-code is: the participants in the document transport application.  In traditional web apis (which is to say, web sites), the audience would be the browser, and any intermediaries along the way.

For example, RFC 7231 uses status codes as a way to resolve implicit caching semantics

Responses with status codes that are defined as cacheable by default (e.g., 200, 203, 204, 206, 300, 301, 404, 405, 410, 414, and 501 in this specification) can be reused by a cache with heuristic expiration unless otherwise indicated by the method definition or explicit cache controls [RFC7234]; all other status codes are not cacheable by default.

If you think of the API consumer (aka the human being) and the API client (aka the web browser) as separate: then the semantics of the status codes are directed toward the API _client_.  This is what tells the client that it can just follow a redirect (the various 3xx headers), that it can simply reset the previous view (205), that it should throw up a dialog asking that the consumer identifier herself (401) and so on.

The information for the consumer is embedded in the message-body.

402 Payment Required

402 Payment Required, alas, is reserved.  Which is a way of saying that it doesn't have a standard meaning.  So you can't deliver a 402 in the expectation that the API client will be able to do something clever -- it's probably just going to fall back to the 4xx behavior, as described by RFC 7231

a client MUST understand the class of any status code, as indicated by the first digit, and treat an unrecognized status code as being equivalent to the x00 status code of that class, with the exception that a recipient MUST NOT cache a response with an unrecognized status code.

I wouldn't bet hard on 402; it was also reserved in RFC 2616, and there's a big gap in RFC 1945 where it should have been.

My guess would be that a 402 specification would be analogous to the requirements for 401, with additional standard headers being required to inform the client of payment options.

As we don't know what headers those would be.  Taler's approach was to stick in a custom header, for instance.  If you control the client, wiring in your own understanding of what 402 might someday be could be a reasonable option.

Protocol alternatives

Another option of good pedigree is to consider that collecting a payment is just another step in the integration protocol.

So, from that perspective, it's perfectly reasonable to say that the request was processed successfully, but the returned representation, rather than providing a link to the cake, provides a link to the billing system.

This is the approach described by Jim Webber when he talks about RESTBucks.  Needing to make a payment is a perfectly normal thing to do in a purchasing protocol, so there's no need to throw an exception when money is due.  Thus, 2xx Success is still a reasonable choice:

The 2xx (Successful) class of status code indicates that the client's request was successfully received, understood, and accepted.

So the _client_ knows that everything went well; and the consumer needs to review the semantics of the message in the message-body to proceed toward her goal.  This is how hypermedia is intended to work -- the current application state is described by the message.

Protocol violations

Now, if instead of proceeding to the payment system as directed, the consumer tries to skip past the purchasing system onto the good bits; that's not so much part of the protocol, so you needed feel compelled to continue to provide a good experience.  400 Bad Request or 403 Forbidden are your go to choices here.

412 Precondition Failed is just wrong; it means that the preconditions provided in the request headers were not met when the server processed the request.  Unless you've got the client providing some extra headers, it's not a fit.
409 Conflict... I believe that one is wrong, but its less clear.  From what I can see in the literature, 409 is primarily a remote editing response -- we tried to apply some change to a resource, but our edit lost some sort of conflict battle with other changes in the system.  For instance, Amazon associates that status-code with BucketAlreadyExists; the problem with the request to create a bucket with that name is that the name has already been taken (and it is a client error, because the client didn't check first).

Demonstration: REST is spelling agnostic

An illustration of the power of REST.

I can search google for google

https://www.google.com/search?&q=gle&sourceid=firefox#q=google

To the surprise of absolutely no one, the top hit is google

https://www.google.com

If I click on, or copy, the link, it takes me to something like

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjr_4O62ffUAhXLXD4KHSi8BtMQFggkMAA&url=https%3A%2F%2Fwww.google.com%2F&usg=AFQjCNFePWT_Lkni-D9ikX7wC3eYuDMQYQ

Which in turn _redirects_ me to

https://www.google.com/

And my HTTP client, which knows absolutely nothing about Google, manages just fine.  Google can change their URI space any way they like, and the client just follows its nose.

My carbonware HTTP agent, doesn't notice, because its looking at the links, and the semantic cues, not at the spelling of the underlying identifiers.

As far as the client and the agent are concerned, all of those URI are opaque.  The only things that we can do with them is use them for cache lookups.  The meaning of anything that happens to be encoded in those sequence of bytes is private to the server.

https://www.google.com/search?&q=gle&sourceid=firefox#q=google

This one isn't quite opaque; this URI was constructed by the HTTP client from the data in the submitted form; the pair q=google is a representation of the data entered into the form by the agent, the rest were provided by the server in its representation of the form.

The client and agent have a common understanding of form as a thing of images and UI affordances; the client and the server share a different understanding of form, derived from their common understanding of the HTML media type.

The agent and the server have a common understanding of semantics -- I understand the form from the labels; the client knows how to render the labels, and what to render in them (from parsing the HTML) but the client has no understanding that those labels _mean_ anything.

And it all "just works".

RPC vs REST

Inspired by Why should I prefer HTTP REST over HTTP RPC JSON-Messaging style in conjunction with CQRS?

As with most REST questions, the easy answers can be found by looking at the web through a browser.

In RPC, the client creates an application/x-www-url-encoded document and sends it to the server.
In REST, the client completes a form and submits it.

In both cases, the client-stateless-server architectural constraint still applies; the server has no way of knowing anything about the nature of the client.  The message looks the same regardless of whether the client walks through the guided path or skips to the end.

In 2008, Roy T. Fielding made the following observation of a solution he was outlining.

I should also note that the above is not yet fully RESTful, at least how I use the term. All I have done is described the service interfaces, which is no more than any RPC. In order to make it RESTful, I would need to add hypertext to introduce and define the service, describe how to perform the mapping using forms and/or link templates, and provide code to combine the visualizations in useful ways.

My translation: having identified the endpoint of the protocol, you work toward REST by relaxing the understanding of the endpoint required by the client.

The RPC approach to a form submission requires that the client know in advance

1. The location of the resource that accepts the request
2. The appropriate HTTP method to use
3. The correct spellings of the keys in the key/value pairs
4. The semantic meaning of the keys/value pairs

In the REST approach, this information is delivered on demand, in the hypermedia.  So the location, the method, the keys are all covered.  The client just loads the form, uses semantic cues to recognize which fields to change, and the submits the form.

Of course, this also means that the client needs to understand forms and how to interpret them.  That's not free, but it shifts the problem from knowing in advance about the specific service to knowing in advance about a generic media type, and the conventions for semantic cues.

The riddle of finding the form gets pushed up to the bookmarked representation.  The client still needs a starting point, which is the bookmark.  It loads a representation of the bookmark provided by the server, and uses the available semantic cues to find a link to the required form.

The server can direct the client to a new representation of the form by changing the representation of the bookmark.  The new form representation can include additional fields with new semantic cues; the client, knowing only about the original semantics, simply ignores these fields when filling out the form; which in turn means that the values submitted will be those provided by the server in the form representation itself.

This isn't free, by any stretch -- we're buying the decoupling of the client and server by doing more upfront api design.

REST is intended for long-lived network-based applications that span multiple organizations. If you don’t see a need for the constraints, then don’t use them.

Stable APIs require a design investment.  The stability constraint, however, is optional; you should reject that constraint when the benefits are insufficient to offset the costs.

A RESTful Kitchen Oven

Some time back, I chatted with Asbjørn Ulsberg about an example of a REST api using an oven.  Shortly there after, he presented his conclusions at the Nordic APIs 2016 Platform Summit. [slides]
What follows is my own work, but clearly I was influenced by his point of view.

 In my kitchen, there is a free standing gas range.  On the inside, it's got various marvels that we no longer think about much: an ignition system, and a thermostat, safety valves, etc.

But as a home cook, I really don't need to worry about those details at all.  I work with the touch pad at the top of the unit.  Bake, plus-plus-plus-plus, Start, and I get a beep and a display message letting me know that the oven is preheating.  Some time after that, another beep let's me know that the oven has reached the target temperature and on good days the actual temperature stays near that target until the oven is shut off.

Let's explore, for a time, what it would look like to control the oven from a web browser.


For our first cut, we can look at an imperative approach.

HTTP/1.0 gave us everything we need.  GET allows us to retrieve the information identified by the request uri, and POST allows us to provide a block of data to a data handling process.

In a browser, it might look like this: we load the bookmark URI.  That would get some information resource -- perhaps a menu of the services available, perhaps a representation of the current state of the stove, maybe both.  One of the links would include a relation (as well as human readable cues) that communicates that this is the link to follow to set the oven temperature.  Following that link would GET another resource that is a web form; in this case it would just be an annotated field for temperature (set with the default value of 350F), semantic cues, and a submit button.  Submitting the form would post the contents to the server, which would in turn interpret the submitted document as instructions to present to the oven.  In other words, the web server reads the submitted data, and pushes the control buttons for us.

Having done that, the server would return a 200 status to us, with a representation of the action it just took, and links from there to perhaps the status page of the oven, so that we can read updates to know if the oven has reached temperature.  In a simple interface like the one on my stove, the updates will only announce that the oven is preheating.  A richer interface might include the current temperature, an estimate of the expected wait time, and so on.

Great, that gets us a website that allows a human being to control the stove from a web browser, but how do we turn that into a real API?  Answer: that is a real API.  Anybody can grab their favorite http client and their favorite html parser, write an agent that understands the link relations and form annotations (published as part of this API).  The agent uses the client to load the bookmark url, loads the result into the parser, searches the resulting DOM for the elements it needs, submits the form, and so on.  Furthermore, the agent can talk to any stove at all.

And -- bonus trick: if you want to test the agent, but don't have a stove handy, you can just point it at any web server with test cases represented as graphs of html documents.  After all, the only URI that the agent actually knows anything about is the start point.  From that point on, it's just following links.

That's a nice demonstration of hypermedia, and the flexibility that comes from using the uniform interface.  But it misses two key lessons, so let's try again.

This time, we'll go with a declarative approach.  We need another verb, PUT, defined in the HTTP/1.1 spec (although it had appeared in the quarantine of the earlier appendix D).  Puts early definition got right to the heart of it.

The PUT method requests that the enclosed entity be stored under the

supplied Request-URI.

What the heck does that mean for an oven?


To the oven, it means nothing -- but we aren't implementing an oven, we're implementing a web api for an oven.  To be a web-api for something interesting means to express the access to that bit of interesting as though it were a document store.

In other words, the real goal here is that we should be able to control the stove from any HTTP aware document editor.

Here's how that might work.  We start from the bookmark page as we did before.  This includes a hypermedia link to a representation of the current temperature settings of the oven.  For instance, that representation might be a json document which includes a temperature field somewhere in the schema.  So the document can GET the current state represented as a json document (important note: this representation does NOT include hyperlinks -- we're not going to allow the document editor to modify those.  Instead, transitions away from the editable representation of the document are described in the Link header field.)

We load the document of the current state into our editor, which then follows the "self" relation with an OPTIONS call to discover if PUT is supported.  Discovering that this is so, the editor enables the editing controls.  The human operator can now replace the representation of the current temperature of the oven with a representation of the desired temperature, and click save.  The document editor does a simple PUT of the saved representation to the web server.

PUT is analogous to replace, in this use case.  In the case of a document store, we would simply overwrite the existing copy of the document.  We need to give some thought to what it means for an oven to mimic that behavior.

One possibility is that the resource in question represents, not the current *state* of the oven, but the current *program*.  So we initially connect to the oven the state of the program would be do nothing, and we would replace that with a representation of the program "set the temperature to 300 degrees", and the API would "store" that program by actually commanding the oven to heat to the target temperature.  PUT of a new program actually matches very well with the semantics of my oven, which treats entering the desired state as an idempotent operation.

A separate, read-only, resource would be used to monitor the current state of the oven.

In this idiom, "turning off the oven" has a natural analog for our document editor -- deleting the document.  The editor can just as easily determine that DELETE is supported as PUT, and enable the property controls for the user.

If we don't like the "update the program" approach, we can work with the current state document directly.  We enable PUT on the resource for the editable copy of the current oven state, enabling the edit controls in the document editor.  The agent can describe the document that they want, and submit the result.

The tricky bit, that is key to the declarative approach: the API needs to compare the current state with the target state, and determine for itself what commands need to be sent to the oven to bring about that change.  Just as the keypad insulates us from the details of the burners and valves, so to does the API insulate the consumers from the actual details of interacting with the oven.

Reality ensues: the latency for bringing the oven up to temperature isn't really suitable for an HTTP response SLA.  So we need to apply a bit of lateral thinking; the API reports to the document store that the proposed edit has been Accepted (202).  It's not committal, but it is standard.  The response would likely include a link to the status monitor, which the client could load to see that the oven was preheating.

Once again, automating this is easy -- you teach your agent how to speak oven (the standard link relations, the media-types that describe the states and the programs).  You use any HTTP away document editor to publish the agents changes to the server.  You test by directing the agent to a document store full of fake oven responses.
 
Do we need two versions of the API now? one for an imperative approach, another for the declarative approach?  Not at all -- the key is the bookmark URL.  We require that the agents be forgiving about links they do not recognize -- those can simply be ignored.  So on the bookmark page, we have a link relation that says "this way to the imperative setOvenTemperature interface", and another that says "this way to the declarative setOvenTemperature interface", and those two links can sit next to each other on the page.  Clients can follow whichever link they recognize.

Document editing -- especially for small documents -- is reasonably straight forward in html as well; the basic idea is that we have a resource which renders the current representation of our document inside a  text area in a form, which POSTs the modified version of the document to a resource that interprets it as a replacement state or a replacement program as before.

You can reasonably take two different approaches to enabling this protocol from the bookmark page.  One approach would be to add a third link (inventing a new link relation).  Another would be to use content negotiation -- the endpoint of the setOvenTemperature interface checks to see if the Accept-Type is html (in which case the client is redirected to the entry point of that protocol, otherwise directed back to the previously implement PUT based flow).

Using the HTML declarative flow also raises another interesting point about media types.  Text areas aren't very smart, they support free form text, so you end up relying on the operator not making any data entry errors.  With a standardized media-type, and a document editor that is schema aware, the editor can help the operator do the right thing.  For instance the document editor may be able to assist the agent with navigation hints and a document object model, allowing the agent to seek directly to the document elements relevant to its goal.

Edit: go figure -- having written up my thoughts, I went back to look at Asbjørn Ulsberg's slides and realized that we had originally been talking about toasters, not ovens.

A RESTful supply closet

At stackflow, another question was submitted about REST API design for non CRUD actions.  In thinking about it, I found an analogy that may help explain the point.

Imagine a supply closet; an actual physical closet in the real world.  The stock includes boxes of pencils.  What does a web API for the supply closet look like?

As Jim Webber pointed out years ago, HTTP is a document management application.  So the first thing to realize is that we are trying to create an interface that supports the illusion that the supply closet is a document store.  If we want to know about the current state of the closet, we read the latest report out of the store.  To change the state of the closet, we propose changes to the document store.

How do we convert the current state of the closet to a document?  In the real world (think 1950s office), we would ask the quartermaster for the latest inventory document.  If a recent one is available, the quartermaster gives us a copy of that document.  Otherwise, he can look in the closet, count the boxes, and produce send us a copy of the fresh report.

That, fundamentally, is GET.  We ask the API for a copy of the inventory report.  Maybe the API just copies the report that's posted on the closet door, maybe the API goes inside the closet to count everything, maybe the closet isn't accessible, so we just get the last report the API saw.  Doesn't matter, we got a document.

Now, key in the next stage is to realize that the document is not the closet; when we edit the document, boxes of pencils don't magically appear in the closet.  What we need to implement is the illusion that the closet really is a document store.

In our real world model, we read the inventory report, and there aren't enough pencils.  So we create a new document -- a memo to the quartermaster that says "stock more pencils".  When we deliver the memo to the quartermaster, he decides how to get more pencils for the closet -- maybe he gets boxes out of storage, or buys some from the store next door.  Maybe he updates his todo list (another document)  and tells you he'll get back to you.


This is the basic idiom of HTML forms.  We create (POST) a new document to the API, and the API interprets that document as changes to be made to the closet.  The requisition document and the inventory document are different resources.  For that matter, the collection of requisition documents and the inventory document are different resources.  So you need a different namespace of identifiers to work with.

HTTP (but not HTML) also supports another approach.  Instead of interacting with the closet by submitting new documents, we could interact with the closet by proposing edits to the existing documents.

This, to my mind, feels a bit more declarative -- you describe in the edited document the state that you want the closet to be in, and its up to the API to figure out the details of making that happen.  In our analogy, we've sent to the quartermaster a copy of the inventory with a bunch of corrections made to it, and he changes the state of the closet to match the document.

This is PUT -- specifically a PUT to the inventory resource.  Notice that it doesn't change what work the quartermaster needs to do to fill the closet; it doesn't change his schedule for doing that work, it doesn't change the artifacts that he generates while doing the work.  It just changes which document manipulation illusion we are choosing to support.

Now, HTTP is specific about the behavior of the imaginary document store we are mimicking, which is that PUT is an upsert.  If we want fine grained control of the contents of the closet ("more pencils, leave everything else alone"), then we need to upsert to a resource with a matching grain.

PATCH is another alternative to introducing finer grained resources; we send the patch to the server, it compares the patched version of the inventory document to the original version, and from there decides what changes need to be made to the closet.

These are all variations of the same fundamental idea - the HTTP request describes the desired end state, and the implementation sitting behind the API figures out how to realize that end.

Session Data

A recent question on stack exchange asked about storing session data as a resource, rather than as part of a cookie.  In the past, I had wondered if the transience of the session data is the problem, but that's really only distraction from the real issue

Fielding, in his thesis, wrote

Cookie interaction fails to match REST's model of application state, often resulting in confusion for the typical browser application.

 What kind of confusion is he talking about here?

The first architectural constraint of REST is Client Server (3.4.1).  The client here sends messages to the server there, and gets a message in return.

Riddle - how does the server interpret the messages that it receives?  In a stateless client server architecture, interpreting the client message is trivial, because the message itself is derived from the state of the client application.  Which is to say that the client and server are in agreement, because the client created the message from its local local application state, and the message is immutable in transit, and the server has no application context to add to the mix.

When you drop the stateless architectural constraint, and introduce session data, the client and server no longer necessarily understand the message the same way: the client creates the message from its local application state, the message transits unchanged, but the server applies the application context stored in the session when interpreting the message.

In the happy path, there's no real problem: the session data applied by the server is the same data that would otherwise have been included by the client in the message, and the resulting interpretation of the message is equivalent using either architectural style.

Outside the happy path, sessions introduce the possibility that the actual state of the client and that state assumed by the server drift.  The client sends a message, expecting it to achieve desired end A, but because the server's copy of the session data is not in sync, the message is understood to prefer outcome A-prime.  Furthermore, with the session data "hidden" on the server, you may end up with quite a few mismatched messages going back and forth before the client begins to realize that a disaster is in the making.

The fundamental breakdown is here: the server does not know what the state of the application on the client is.  It can know what messages have been received, it can know what messages have been sent in response, but it doesn't know (without specific correlation identifiers being built into the messages) that the dispatched messages have arrived.

Furthermore

The application state is controlled and stored by the user agent and can be composed of representations from multiple servers. In addition to freeing the server from the scalability problems of storing state, this allows the user to directly manipulate the state (e.g., a Web browser's history), anticipate changes to that state (e.g., link maps and prefetching of representations), and jump from one application to another (e.g., bookmarks and URI-entry dialogs).

Which is to say, the client is explicitly allowed to rewind to any previously available state, cached or reconstructed from its own history, without consulting the server.

The server defines allowed application states, and the hypermedia controls that lead to new application states, but it doesn't control which state in the sea of revealed application states is the current one.

The client, not the server, is the book of record for application state.

Why is it OK to PUT an order for a cup of coffee ?  Because in that case, all of the ambiguity is in the state of the resource, not the state of the message.  The client and the server both share the same precise understanding of what the message means, because all of the required context appears within the message itself.  The client rewinds to some previous state, and fires off an obsolete message, and the server is able to respond "I know precisely what you mean, but you aren't allowed to do that any more".  There's no confusion here; the state of the resource has simply evolved since the hypermedia control was described.

So long as everybody agrees what the message means, there is no confusion.  That's why the happy path looks OK -- if the client and the server are still sharing the same assumptions about message context, the confusion doesn't arise; the client and the server happen to mean the same thing by fortuitous accident, and it all "just works".  Until it doesn't.



 

The name of the URI is callled....

I just found myself proposing the following "RESTful" URI:


/userStories?asA=emailScammer&iWantTo=mineEmailAddresses&soThat=iCanBroadcastMoreSpam 


I'm not sure I was kidding.

Followup: I used query parameters; HTML is a successful implementation of a media-type which supports hypermedia controls.  It's an ideal reference implementation for illustrating RESTful principles.

But it is not without its limitations.  URI Templates offer a lot of flexibility; web forms -- not so much.  You expose query parameters, or you force the client to traverse a graph to find the control that they want.

Digging around for URI design guidelines, I found this summary by K. Alan Bates

Hierarchical data is supposed to be represented on the path and with path parameters. Non-hierarchical data is supposed to be represented in the query. The fragment is more complicated, because its semantics depend specifically upon the media type of the representation being requested.

Moving the parameters of the story from the query string would be preferred, because it correctly represents that this the known identification of a resource, rather than a specification for a resource that may not have any matches.  Furthermore, doing so better conforms to the convention that path segments represent hierarchy.  For instance, it's reasonable to suppose that the URI for the story card ought to look like:


/userStories/asA=emailScammer&iWantTo=mineEmailAddresses&soThat=iCanBroadcastMoreSpam/card

The design guidelines in the RESTful Web Services Cookbook suggest an improvement on the previous design...

Use the comma (,) and semicolon (;) to indicate nonhierarchical elements in the path portion of the URI.

Richardson & Ruby, in Chapter 5 of RESTful Web Services were a bit more specific

I recommend using commas when the order of the scoping information is important, and semicolons when the order doesn't matter.

I'm not actually sure if the template of the user story should be considered ordered or not.  There are a lot of references to the Cohn template, a few that point out that maybe the business value should get pride of place.

Me?  I'm going to represent that the ordering of these elements matters, because that allows me to use the delimiter that looks like the punctuation on the story card itself

/userStories/asA=emailScammer,iWantTo=mineEmailAddresses,soThat=iCanBroadcastMoreSpam/card


Better, but I don't like equals, and none of the other sub-delims improve the outcome.  Today, I learned that RFC 3986 offers me an out -- the production rules for path segments explicitly include colon!  

/userStories/asA:emailScammer,iWantTo:mineEmailAddresses,soThat:iCanBroadcastMoreSpam/card

In all, an educational exercise.  Didn't learn whether or not I was kidding.

More thoughts on REST

Continuing to explore REST...

Lesson: REST is not about building nice little web applications like blogs, or Amazon, or Google.  REST is about building web scale applications like...

the Web.

Lesson: the Web is already web scale.  If you are building your application on the back of HTTP, all you need to do is not screw it up...

for example, by replacing text/html with some other media type that doesn't include controls.  Oops.

Lesson: a friend of mine teaches his students that, when they are asked to implement a protocol, their first step should be to obtain the appropriate state machine compiler.  In reverse, if an expert writes a thesis on Representational STATE Transfer, you should be thinking about how that maps to your application protocols...

and maybe not so much mapping it to your data model.  Or your persistence layer.

Lesson: HTML is a perfectly cromulent media-type.  GET and POST will serve as hypermedia controls.  application/www-form-urlencoded is a little bit clumsy for hierarchical data, but can be made to serve.

So it should be straight forward to use a browser to navigate your application protocol?

Lesson: the modern web works.  You point your browser at a url, and the browser downloads a bunch of java script that renders a single page application, and starts pinging some json api and using the responses to update the DOM....  That's REST.  Event if those json endpoints aren't actually providing hypermedia controls - neither do images in html documents.

REST: on Resources

I've been trying to make more progress with REST by reviewing the questions that other programmers are asking about it

• http://programmers.stackexchange.com/questions/tagged/rest

Sidestepping for the moment the whole riddle of hypermedia representations, the most common issue seems to come about when the programmer assumes that an entity in the domain must have a single URI that does everything with nothing but Four Verbs and the Truth.

The internal mapping seems to be that entities are nouns, and resources are nouns, and therefore entities are resources -- since the URI is "the" identifier for the resource, it must also be "the" identifier for the entity, and suddenly everything looks like a nail.

Clearly, identifiers should be 1-to-1 with resources -- you really need to pervert the definition of "uniform resource identifier" to reach any other conclusion.  Although, it turns out that is something that we had to learn over time.

For instance, in 1999, the definition of the PUT method described resources this way

A single resource MAY be identified by many different URIs. For example, an article might have a URI for identifying "the current version" which is separate from the URI identifying each particular version. In this case, a PUT request on a general URI might result in several other URIs being defined by the origin server.

 In 2014, the definition of the PUT method changed.

A PUT request applied to the target resource can have side effects on other resources.  For example, an article might have a URI for identifying "the current version" (a resource) that is separate from the URIs identifying each particular version (different resources that at one point shared the same state as the current version resource).  A successful PUT request on "the current version" URI might therefore create a new version resource in addition to changing the state of the target resource, and might also cause links to be added between the related resources.

The latter interpretation allows you to sidestep one of the complicating issues with interpreting a PUT method -- the body is supposed to be a replacement for the resource, but nothing in "the rules" prevents you from inventing a resource with the specific purpose of being replaced.

Jim Webber hinted at the same thing:

You should expect to have many many more resources in your integration domain than you do business objects in your business domain.

I think the CQRS language helps here -- from our event history, we build lots of different projections that are each especially suitable to a particular use case.

Another idea that helped clear things for me is that a view of an immutable resource can be cached.  Trying to cache a projection of a mutable entity brings you face to face with one of the two hard problems, but caching the immutable history of a mutable entity is fine.  In other words, we have different resources that describe the state of an entity at different times.  Couple that with the idea that there are many ways of expressing a moment of time, that each in turn maps to a different resource, and you get an explosion of possible resources that you can exploit.

Domain Driven Design vs REST

For a couple of weeks now, I've been banging my head against Domain Driven Design (DDD), Command Query Responsibility Segregation (CQRS), and Representational State Transfer (REST).

I had been making a big, and probably common mistake: I started looking for nouns.  The ubiquitous language gives me lovely nouns, and they seemed a natural fit for resources.

But I couldn't get the same natural feeling from the verbs.  In the ubiquitous language, I've got all of these lovely expressive verbs to motivate change in my business model.  In my link relations, I've got GET, PUT, POST, DELETE.

I finally tracked down Jim Webber's DDD in the Large presentation.  Yeah, that helped.

Application vs Domain

Taking it very slowly: the key idea underlying REST is "Hypermedia as the Engine of Application State".

Application State.

Why am I thinking about trying to represent my aggregate roots as resources?  Those are two completely different layers!  The application layer talks to the domain layer, there's an interface between the two, but there's no particular reason to expect a one to one correspondence.

All of the RESTful bits are going to be over there, with the anti corruption logic.

Commands as Resources

 Resources were the second bit that I had flat out gotten wrong.  It had occurred to me that I could just cheat, turn the problem around, and use my commands as resources.

What Jim's talk clarified for me: that's not cheating, it's the whole damn point.

The hand wavy argument is that we are looking for nouns, and the commands, as messages, are the noun that we want.  No kidding, our resources are representations of little pieces of paper, a ToDo on a post it note, that the client is passing to the server.

It still feels like a cheat to me.

But Jim in his talks points out, rightly, that if you are communicating over HTTP, then you are using a document management system to communicate your applications state.  So if you aren't passing documents, you're clearly Doing It Wrong.

That's getting closer, but I needed to make one more connection to sell myself.

Stepping back from the problem; ignore the REST constraint completely.  The client, over there, needs to communicate with the server here.   We crossing a boundary; if we watch on the wire, we're not going to see objects in the transfer, but data.  We see the same thing when the client queries a projection - Data Transfer Objects are being exchanged.

Data Transfer Object is a synonym for document.  Oh.

I find it a little bit easier to sneak up on the idea by looking at the interaction with the read model.  The client sends us a query, and we send back some projection of the model.  It doesn't make sense to think about caching a model (it changes in time), and caching the projection (which also changes in time) is similarly dubious, but caching a snapshot of the projection at some point in time -- that does make sense.  "Get me the report as of Time.now()" sure sounds like a document to me.

Something similar happens with domain events, and the communication between the read model and the write model.  "Stream of Events" might not sound like a document, but journal, ledger, log -- those certainly are.

Commands as documents?  I mentioned the ToDo analog earlier, but another good fit would be orders.

Model Change

Of course, the whole point to this mess is to have an application that can interact with the model, so something needs to connect the two.

The read model, that's easily managed -- the queries arrive, the appropriate event history is loaded into the projection, and report is generated and delivered.  All of these steps are idempotent and safe - we might change some state in memory, like caching the projection data for a time in case we are about to need it, but the model and the event history are not changed at all.

The write model is more difficult - changing the model is a side effect of the arrival of the command, and the command may arrive more than once.

For instance, the client puts a command, the command is received and executed, but the acknowledgement of the command is lost in transit.  As PUT is supposed to be idempotent, the client may send the command document a second time.

The model, it should be running the commands given to it.  So you need either that the commands in the model handle commands idempotently, or you need the anti-corruption layer to do the right thing when the duplicated command arrives.

An example scenario: Alice puts command.id:1 to the server.  The server executes the command, updating the history of the model, and publishes a reply.  That reply is lost in transit.  Bob puts command.id:2 to the server, updating the model further.  Alice times out waiting for the acknowledgment that her command arrived, and resends it.  Charlie puts command.id:3 to the server, but is data is stale because he was working from a state prior to Alice's first command.

What should the responses look like?

Bob, clearly, successfully delivered a command that was executed, he should see a status 201 Created message.   Charlie's command should be rejected, because the preconditions under which he submitted the command were not met, which probably means a 409 or 411 response.  Alice's first command should like Bob, get a 201.  When she resubmits the same command a second time, it is still supposed to be an idempotent operation, so she should still be getting the 201 response (and not the error code seen by Charlie).

That probably means a message store: to cache the response in the application layer for a time so that it can be replayed without interacting with the model.

My feeling is that the command should be considered immutable by the client; a second put that doesn't agree with the first should be rejected.  The delete by the client might be a way to incorporate an acknowledgement, and expire the data in the message store.