Recording of stream about collaboration between microservices

 The other I was on the Manning Twitch talking about collaboration between microservices. You can see recording of stream below. I was there on occasion of the publication of the second edition of my Microservices book, which only just landed in hard copy a few days before. Much of the material in this talk appears in further detail in the book, though I did change it up a bit for the talk.

Hope you enjoy it!

Interview on Adventures in .NET

 I recently had the pleasure of chatting with the folks on the Adventures in .NET podcast and now the episode is out. We talk about some of the things I consider essential to get right to have success with microservices -  like getting the responsibilities right, having good observability, good automation and more. We also talked about when to use microservices and when not to. After all, one does not simply microservice. Anyway, the episode is out and you can find it here.

Interview about Microservices on the Cynical Developer Pordcast

I recently had a chat with James Studdart, host of the Cynical Developer podcast about microservices and the second edition of my microservices book. The podcast is now out and you can listen to it here or any of the usual places you find podcasts:

Identifying and Scoping Microservices talk from DevConfPL

I recently had the pleasure for delivering my Identifying and Scoping Microservices talk at DevConf in Krakow. Here's the video. Enjoy 😉

A pattern for synchronizing data from a legacy system to microservices

Abstract

A recurring need in the teams I work with seems to be to move and continually synchronize data from a legacy system - typically a monolith - to a new microservice based replacement. In this post I will outline a data synchronization solution that allows

• The microservice side to catch up on historical data
• Synchronizes new data to the microservices as it is produced in the legacy system
• Lets us control the speed of synchronization and as a result the load on the legacy system
• Is repeatable, so the same data can easily be synchronized again if need be

In the interest of keeping this post short I will only show a high level view of the solution.

Microservices need legacy data

When replacing an existing legacy system with a modern microservice based system teams (sensibly) tend to do so in steps following the strangler pattern. This means that for a while both systems run in production. Some functionality is handled in the legacy system and some in the microservices. At first very little is handled in the microservices but gradually more and more functionality is handled there. To support those small early steps implemented in the microservices data from the legacy side is often needed. For instance, in a back office system a team in the process of moving to a microservice architecture they might want to implement a new sales dashboard with microservices. To do so we will need order data and possibly other data too, but let's just focus on the order data for now. Orders are still being taken in on the legacy side, so order data is being produced on the legacy side. But the new sales dashboard needs both historical orders and new orders to work correctly.

To make things more interesting let's say the legacy system and the microservices are in different data center - maybe the system is moving from on prem to the cloud as part of the microservice effort.

Solution: A data pump 

A solution to the situation above is to implement a data pump that sends any updates to relevant data in the legacy database over to the microservices. In the example that means new orders as well as changes to orders.

This solution has two components: A data pump which is deployed in the legacy environment and a data sink which is deployed in the microservices environment. The data pump tracks which data has already been sent over to the microservices and sends new data over as it is produced in the legacy system. The data sink simply receives the data from the pump and posts it onto a queue. This enables any and all microservices interested in the data - e.g. new or updated orders - to subscribe to such messages on the queue and to build up their models of that data in their own databases.

With the described solution in place any historical data can be sent over to the microservices. That may take a a while, especially if the legacy database cannot take to much additional load. In such cases the data pump can be throttled. Once the pump has caught up sending over historical data it will continue to send over new data in near real time.

If we put a bit of up front design into the data pump we can also support restarting the process - the pump tracks what it has already sent, resetting that tracking will make it start over. That's sometimes useful if we e.g. don't get the receiving mciroservice right in the first attempt, or if we introduce new microservices that also need data.

This is a solution I have seen used with success in several of my client's systems and that I think is applicable in many more systems too.

Talk: Lightweight Microservice Collaboration using HTTP

I had the pleasure of being at the DotNext conference in Sct. Petersburg a couple of weeks ago, and now the video of the talk I did there is out. Enjoy!

Talk video: "Consolidating Services with middleware" from NDC London 2017

Back in January I did a talk called at NDC London called "Consolidating Services with middleware", and the video of the talk has come out. Enjoy!



Consolidating Services with middleware - Christian Horsdal from NDC Conferences on Vimeo.

In case you are wondering what the talk is about, here is the abstract:

"Have many services? Writing new ones often? If so middleware can help you cut down on the ceremony for writting new services and at same time consolidate the handling of cross cutting concerns. But what is middleware? OWIN and ASP.NET Core both have a concept of middleware. What are they? How do they help? In this talk we will dive into the code, write some middleware and show how middleware helps you handle cross-cutting concerns in an isolated and re-usable way across your services. I'll compare and contrast the OWIN and ASP.NET Core middleware concepts and talk about where each is appropriate."

Event-based Collaboration does not imply Event Sourcing

Abstract

In my microservices book I talk about three styles of collaboration between microservices: Command, Query and Event based collaboration. On the book's forum I recently got a question that boils down to "if a microservice allows event-based collaboration, can all it's state always be recreated from those events?". I'm taking the liberty of re-framing this question as "if a microservice allows event-based collaboration, does it have to use Event Sourcing?" My answer is: No - you can, but you don't have to.

I will try to explain why below.

Event Based Collaboration

When we use a microservice architecture for a system, that system gets broken into lots - usually hundreds - of small narrowly focused services each of which handle one specific capability. In order to deliver a cohesive experience to the users using the applications built on top of all those microservices we have to make them collaborate - the alternative would be to expose the end user to that very fine grained break down into single capability services. That would make for an insane user experience.

One of the ways microservices can collaborate is through events: When something significant happens in a microservice it can choose to publish an event that other microservices can then react to however they need/wish to.

This is a powerful style of collaboration. Events allow for asynchronuous processing, for slow subscriberss to catch up with bursts at their own pace, for some microservices to be down for shorter periods and more. For these reasons event-based collaboration between microservices is quite often a better choice than command- or query-based collaboration. The point in this context, though, is more about what the events are. These events are things that are significant outside of the microservice publishing them. They are published in order to drive collaboration with other microserivces.

Internal Event and External Events

The events published to other microservices in order to drive collaboration are events that can - and should - be published regardless of how the publishing microservice is implemented. Referring to the figure above: The blue microservice publishes events to drive collaboration with other microservices, like the green and yellow ones. That has nothing to do with the inner workings of the blue microservice. The events published to other microservices are external events.
The blue microservice from the figure above can store its state however it wants. One of the ways it can chose to store its state is using Event Sourcing. Using Event Sourcing introduces another set of events: Ones stored internally in the microservice. These events to do not drive collaboration with other microservices, but they capture every little state change in the blue microservice.

I find it helpful to distinguish between these two types of events:

• External events drive collaboration with other microservices. These are also sometimes referred to as integration events.
• Internal events captures all state changes within the microservices and is the basis for event sourcing. These are also sometimes referred to as domain events.

Internal events are usually far more granular than external events. Internal events are also tightly coupled to the implementation details of the microservice, whereas external events are not.

Conclusion

Having established the difference between internal and external events it should be clear that one does not imply the other. A microservice can publish external events without using Event Sourcing. Likewise a microservice can use Event Sourcing without publishing any external events.

Sharing and Caching Assets Across Single-Page BFFs

Over the last couple of posts I showed the single-page BFFs pattern (a variation of the Backend-For-Frontend aka BFF pattern where the BFF serves only one page in a web application), and how you will likely need to place single-page BFFs behind a reverse proxy.

In this post I will outline an approach to handling shared assets - like CSS or Javascript which is used on all or most of the pages in a web application built using a number of single-page BFFs.

When building a web application consisting of a number of pages, there is likely both some look-and-feel and some behavior that we want to share across all pages - buttons should look and behave the same, menus that should be on all pages, colors and fonts should be consistent etc. This calls for sharing some CSS and some Javascript across the pages. When thinking about handling these shared things across a number of single-page BFFs there are some competing requirements to take into consideration:

• Clients - i.e. browsers - should be able to cache the CSS and Javascript as the user moves through the web application. The alternative is that the client downloads the same CSS and JS over and over as the user navigates the web application. That would be too wasteful in most cases.
• Each single-page BFF should be individually deployable. That is we should be able deploy a new version of one single-page BFF without having to deploy any other services. If not the system soon becomes unmanageable. Furthermore developers should be able to run and test each single-page BFF by itself without having to spin up anything else.
• I do not want to have to write the same CSS and JS in each single-page BFF. I want to share it.

In this post I will show an approach that allows me to achieve all three of the above.

First notice that the obvious solution to the first bullet - allowing clients to cache shared CSS and JS - is to put the shared CSS and the shared JS in two bundles - a CSS bundle and a JS bundle - and put the bundles at a known place. That way each single-page BFF can just address the bundle at the known place, as shown below. Since all pages refer to the same bundles the client can cache them.

The problem with this is that it violates bullet 3: Any given single-page BFF only works when both the global bundles are available, so for a developer to work on a single-page BFF, they need not only the BFF, but also something to serve the bundles. If we are not careful, this approach can also violate bullet 2: Since the single-page BFFs depend on the two global bundles, but they do not contain the global bundles themselves, each one has a hard dependency on another service - the one that serves the bundles. That means we can easily end up having to deploy the service that serves the bundles at the same time as a new version of one of the single-page BFFs.

Looking instead a bullet three first suggests that we should put the shared CSS and Javascript into a package - an NPM package, that is - and include that package in each single-page BFF. That means each single-page BFF has the bundles it needs, so developers can work with each single-page BFF in isolation. This also means that each single-page BFF continues to be individually deployable: When deployed they bring along the bundles they need, so there is no dependency on another service to serve the bundles. The problem now becomes how to allow clients to cache the bundles? When the bundle are included in each single-page BFF, each page will use different URsL for the bundles - completely defeating HTTP cache header.

Luckily, as discussed in the last post, the single-page BFFs are already behind a reverse proxy. If we put a bit of smarts into the reverse proxy we can include the bundles in each single-page BFF and still let every page use same the URLs for the bundles, like so:

The trick implementing this is to have the reverse proxy look at the referrer of any request for one of the bundles - if the referrer is frontpage, the request is proxied to the frontpage single-page BFF, if the referrer is the my account page, the request is proxied to the MyAccount single-page BFF. That means that the bundles loaded on the frontpage is the bundles deployed in the frontpage single page BFF. Likewise the bundles loaded on the my account page come from the MyAccount single-page BFF. But from the client perspective the bundles are on the same URL - i.e. the client is able to cache the bundles based on HTTP cache headers.

To sum up: Put shared CSS and JS bundles in an NPM package that all the single-page BFFs use, and making the reverse proxy use the referrer on requests for the bundles to decide where to proxy requests to allows clients the cache the bundles, the single-page BFFs to be individually deployable and developers work with each single-page BFF in isolation.

Using Single-Page BFFs and Hiding It

In the last post I wrote about a variation of the BFF - Backend-for-frontend - pattern: The single-page BFF page, where the BFF does not support a full web frontend, but only a single page. The reason to use this pattern is that even BFFs can grow out of hand, at which point we can consider cutting the BFF into several single-page BFFs, each responsible for supporting just on page.

Now imagine we have a web site that uses the single-page BFF patterns extensively. As users navigate through the site and visit different pages they will in effect interact with different single-page BFFs. Each such single-page BFF is its own service, and conseqeuntly has its own host name - e.g. the frontpage BFF might be at https://frontpage.bff.ajax.com, the "my account" page might be at https://myaccount.bff.ajax.com/ and so on for each logical page on the site. Like this:

The problem we now have is that if users access these adresses - https://frontpage.bff.ajax.com, https://myaccount.bff.ajax.com etc. - we are exposing the way we have chosen to implement the system on the server side to a our users. That's bad: Users will bookmark these and then we are tied to keeping these host names alive, even if we decide to rearchitect the server side to use some other pattern than single-page BFFs. Instead I would like users to just stay of the same domain all the time - like https://www.ajax.com/ - and just visit different pages on that domain - e.g. https://www.ajax.com/ and https:/www.ajax.com/account. So how do we make ends meet? We simply set up a reverse proxy in front of the single-BFFs. Like so:

This is a standard thing to do and can easily be done with widespread technology, like nginx, squid or IIS with ARR. In other words: There is nothing new here, I am only pointing out that using the single-page BFF pattern will lead a need to set a up a reverse proxy in front of your services.

Single-Page BFFs

In this post I describe a variation of the backend for frontend (AKA BFF) pattern that Søren Trudsø made me aware of. With this variation we create not just a backend for each frontend - one backend for the iOS app, one for the Android app and one for the web site, say - but a backend for each page in the web frontend: A single-page BFF.

The BFF patterns is a pattern for building applications on top of a system of microservices. I describe that pattern in my Microservice in .NET Core book, but you can also find good descriptions of the pattern on the web - for instance from Sam Newman. To sum it up the BFF patterns is that you create a microservice for each frontend you have. That is, if you have an iOS app there is an iOS BFF. If you have an Android app you have an Android BFF. If you have an Oculus Rift app there is an Oculus Rift BFF. Each BFF is a microservice with the sole responsibility of serving its frontend - the Android BFF is there solely to serve the Android app and does not care about the Oculus rift app whatsoever. The BFFs do nothing more than gather whatever their frontends - or apps - need and serve that in a format convenient to the frontend. The BFFs do not implement any business logic themselves, all that is delegated to other microservices. This figure illustrates the setup:

In this setup each BFF tends to grow as its frontend grows. That is, the web BFF tends to grow as the web frontend grows: When more functionality is added to existing pages, that functionality might need new endpoints to make AJAX requests to, and thus the web BFF grows a little bit. When new pages are added to the web frontend, the web BFF also grows.

Sidenote: I realize that the term "page" on a web site is somewhat fuzzy these days: Single page apps routinely swap the entire view from one thing to something completely different, giving the user the experience of going to a new "page". In this post I use the term "page" in the more traditional sense of a full page reload. You know, that thing when you follow a link and the browser loads a completely new HTML document from a new URL. I think you've encountered it before :D

The size of the web BFF might not be problem at first (or ever), but a some point enough may have been added to the web frontend to make it a problem. In this situation I have found it useful to break the web BFF down by page boundaries: In stead of having one BFF serve the entire web frontend, I will have a one BFF for each page on the web site, like so:

This way the BFFs are kept small and focused on a single task, namely serving a single page.

Notice that one or more of the pages here can be single page apps that include several views, so there need not be a direct correspondance between what the use perceives a separate views - or pages - and the single page BFFs on the backend. Rather, in such cases, there is a BFF for each single page app.

Book Excerpt: Expecting Failures In Microservices and Working Around Them

This article was excerpted from the book Microservices in .NET.

When working with any non-trivial software systems, we must expect failures to occur. Hardware can fail. The software itself might fail due, for instance, to unforeseen usage or corrupt data. A distinguishing factor of a microservice system is that there is a lot of communication between the microservices.

Figure 1 shows the communication resulting from a user adding an item to his/her shopping cart. From figure 1 we see that just one user action results in a good deal of communication. Considering that a system will likely have concurrent users all performing many actions, we can see that there really is a lot of communication going on inside a microservice system.

We must expect that communication to fail from time to time. The communication between only two microservices may not fail very often, but in regard to a microservice system as a whole, communication failures are likely to occur often simply because of the amount of communication going on

Figure 1 In a system of microservices, there will be many communication paths

Since we have to expect that some of the communication in our microservice system will fail, we should design our microservices to be able to cope with those failures.

We can divide the collaborations between microservices into three categories: Query, command and event based collaborations. When a communication fails, the impact depends on the type of collaboration and way the microservices cope with it:

•  Query based collaboration: When a query fails, the caller does not get the information it needs. If the caller copes well with that, the impact is that the system keeps on working, but with some degraded functionality. If the caller does not cope well, the result could be an error.
• Command based collaboration: When sending a command fails, the sender won’t know if the receiver got the command or not. Again, depending on how the sender copes, this could result in an error, or it could result in degraded functionality.
• Event based collaboration: When a subscriber polls an event feed, but the call fails, the impact is limited. The subscriber will poll the event feed later and, assuming the event feed is up again, receive the events at that time. In other words, the subscriber will still get all events, only some of them will be delayed. This should not be a problem for an event-based collaboration, since it is asynchronous anyway.

Have Good Logs

Once we accept that failures are bound to happen and that some of them may result, not just in a degraded end user experience, but in errors, we must make sure that we are able to understand what went wrong when an error occurs. That means that we need good logs that allow us to trace what happened in the system leading up to an error situation. "What happened" will often span several microservices, which is why you should consider introducing a central Log Microservice, as shown in figure 2, that all the other microservices send log messages to, and which allows you to inspect and search the logs when you need to.

Figure 2 A central Log Microservice receives log messages from all other microservices and stores them in a database or a search engine. The log data is accessible through a web interface. The dotted arrows show microservices sending log messages to the central Log Microservice

The Log Microservice is a central component that all other microservices use. We need to make certain that a failure in the Log Microservice does not bring down the whole system when all other microservices fail because they are not able to log messages. Therefore, sending log messages to the Log Microservice must be fire and forget - that is, the messages are sent and then forgotten about. The microservice sending the message should not wait for a response.

SIDEBAR

Use an Off-the-Shelf Solution for the Log Microservice A central Log Microservice does not implement a business capability of a particular system. It is an implementation of generic technical capability. In other words the requirements to a Log Microservice in systems A are not that different from the requirements to a Log Microservice is system B. Therefore I recommend using an off-the-shelf solution to implement your Log Microservice - for instance logs can be stored in Elasticsearch and made accessible with Kibana. These are well-established and well-documented products, but I will not delve into how to set them up here.

Correlation Tokens

In order to be able to find all log messages related to a particular action in the system, we can use correlation tokens. A correlation token is an identifier attached to a request from an end user when it comes into the system. The correlation token is passed along from microservice to microservice in any communication that stems from that end-user request. Any time one of the microservices sends a log message to the Log Microservice, the message should include the correlation token. The Log Microservice should allow searching for log messages by correlation token. Referring to figure 2, the API Gateway would create and assign a correlation token to each incoming request. The correlation is then passed along with every microservice-to-microservice communication.

Roll forward vs Roll back

When errors happen in production, we are faced with the question of how to fix them. In many traditional systems, if errors start occurring shortly after a deployment, the default would be to roll back to the previous version of the system. In a microservice system, the default can be different. Microservices lend themselves to continuous delivery. With continuous delivery, microservices will be deployed very often and each deployment should be both fast and easy to perform. Furthermore, microservices are sufficiently small and simple so many bug fixes are also simple. This opens the possibility of rolling forward rather than rolling backward.

Why would we want to default to rolling forward instead of rolling backward? In some situations, rolling backward is complicated, particularly when database changes are involved. When a new version that changes the database is deployed, the microservice will start producing data that fits in the updated database. Once that data is in the database, it has to stay there, which may not be compatible with rolling back to an earlier version. In such a case, rolling forward might be easier.

Do Not Propagate Failures

Sometimes things happen around a microservice that may disturb the normal operation of the microservice. We say that the microservice is under stress in such situations. There are many sources of stress. To name a few, a microservice may be under stress because:

•  One of the machines in the cluster its data store runs on has crashed
•  It has lots network connectivity to one of its collaborators
• It is receiving unusually high amounts of traffic
• One of its collaborators is down

In all of these situations, the microservice under stress cannot continue to operate the way it normally does. That doesn’t mean that it’s down, only that it must cope with the situation.

When one microservice fails, its collaborators are put under stress and are also at risk of failing. While the microservice is failing, its collaborators will not be able to query, send commands or poll events from the failing microservice. As illustrated in figure 3, if this makes the collaborators fail, even more microservices are at risk of failing. At this point, the failure has started propagating through the system of microservices. Such a situation can quickly escalate from one microservice failing to lot of microservices failing.

Figure 3 If the microservice marked FAILED is failing, so is the communication with it. That means that the microservices at the other end of those communications are under stress. If the stressed microservices fail due to the stress, the microservices communicating with them are put under stress. In that situation, the failure in the failed microservice has propagated to several other microservices.

Some examples of how we can stop failures propagating are:

• When one microservice tries to send a command to another microservice, which happens to be failing at the time, that request will fail. If the sender simply fails as well, we get the situation illustrated in figure 3 where the failures propagate back through the system. To stop the propagation, the sender might act as if the command succeeded, but actually store the command into a list of failed commands. The sending microservice can periodically go through the list of failed commands and try to send them again. This is not possible in all situations, because the command may need to be handled here and now, but when this approach is possible it stops the failure in one microservice from propagating.
• When one microservice queries another one that’s failing, the caller could use a cached response. In case the caller has a stale response in the cache, but a query for a fresh response fails, it might decide to use the stale response anyway. Again, this is not something that will be possible in all situations, but when it is, the failure will not propagate.
• An API Gateway that is stressed because of high amounts of traffic from a certain client can throttle that client by not responding to more than a certain number of requests per second from that client. Notice that the client may be sending an unusually high amount of requests because it is somehow failing internally. When throttled, the client will get a degraded experience, but will still get some responses. Without the throttling, the API Gateway might become slow for all clients or it might fail completely. Moreover, since the API Gateway collaborates with other microservices, handling all the incoming requests would push the stress of those requests onto other microservices too. Again, the throttling stops the failure in the client from propagating further into the system to other microservices.

As we can see from these examples, stopping failure propagation comes in many shapes and sizes. The important thing to take away from this article is the idea of building safeguards into your systems that are specifically designed to stop from propagating the kinds of failures you anticipate. How that is realized depends on the specifics of the systems you are building. Building in these safeguards may take some effort, but it’s very often well worth the effort because of the robustness they give the system as a whole.

Book Excerpt: What is a Microservice?

Abstract

In this article, excerpted from my upcoming book Microservices in .NET, I will talk about the characteristics that help you recognize a Microservice when you see one and that help you scope and implement your services in way that enable the benefits of Microservices

---

A Microservice is a service with one and only one very narrowly focused capability. That capability is exposed to the rest of the system in a remote API. For example, think of a system for managing a warehouse: Some capabilities that might each be provided by a Microservice in such a system are:

• Receive stock
• Calculate where new stock should be stored
• Calculate placement routes inside the warehouse for putting stock into the right storage units
  
• Assign placement routes to warehouse employees
• Receive orders
• Calculate pick routes in the warehouse given a set of orders
• Assign pick routes to warehouse employees

Each of these capabilities - and most likely many more - are implemented by individual Microservices. Each Microservice runs in separate processes and can be deployed on its own independently of the other Microservices. Likewise, each Microservice has its own dedicated database. Still each Microservice collaborates and communicates with other Microservices.

It is entirely possible that different Microservices within a system will be implemented on different platforms - some Microservices may be on .NET, others on Erlang, and still others on Node.js. As long as they can communicate in order to collaborate this polyglot approach can work out fine. HTTP is a good candidate for the communication: All the mentioned platforms, as well as many others, can handle HTTP nicely. Other technologies also fit the bill for Microservice communication: Some queues, some service buses and some binary protocols for instance. Of these HTTP is probably the most widely supported, is fairly easy to understand and - as illustrated by the World Wide Web - is quite capable, all in all making it a good candidate.

To illustrate, think of the warehouse system again. One Microservice in that system is the Assign Pick Routes Microservices. Figure 1 shows the Assign Pick Route Microservice receiving a request from another collaborating Microservice. The request is for the next pick route for a given employee. The Assign Pick Route Microservice has to find a suitable route for the employee. Calculating an optimal route is done in another Microservice. The Assign Pick Route Microservice simply gets notified of the pick routes and only needs to decide how to assign them to employees. When a request for a pick route for a given employee comes in the Assign Pick Route Microservice looks in its database for a suitable pick route, selects one and returns it to the calling Microservice.

Figure 1 The Assign Pick Route Microservice exposes an API for getting a pick route for an employee. Other Microservices can call that API.

What is a Microservices Architecture?

Microservices as an architectural style is a lightweight form of Service Oriented Architecture where the services are very tightly focused on doing one thing and doing it well.

A system that uses Microservices as its main architectural style is a distributed system of a - most likely large - number of collaborating Microservices. Each Microservice runs on its own in its own process. Each Microservice only provides a small piece of the picture and the system as a whole works because the Microservices collaborate closely. To collaborate they communicate over a lightweight medium that is not tied to one specific platform like .NET, Java or Erlang. As mentioned, all communication between Microservices, in this book, is over HTTP, but other options include a queue, a bus or a binary protocol like Thrift.

The Microservices architectural style is quickly gaining in popularity for building and maintaining complex server side software systems. Understandably so: Microservices offer a
number of potential benefits over both more traditional service oriented approaches and monolithic architectures. Microservices – when done well - are malleable, scalable, resilient and allow a short lead time from start of implementation to deployment to production. A combination which often prove evasive for complex software system.

Microservice Characteristics

So far I have established that a Microservice is a very tightly focused service, but that is still a vague definition. To narrow down the definition of what a Microservice is, let’s take a look at what characterizes a Microservice. In my interpretation of the term, a Microservice is characterized by being:

1. Responsible for one single capability
2. Individually deployable
3. Consists of one or more processes
4. Owns its own data store
5. A small team can maintain a handful of Microservices
6. Replaceable

This list of characteristics both helps you recognize a Microservice when you see one and helps you scope and implement your services in way that enable the benefits of Microservices - a malleable, scalable and resilient system. Let’s look at each in turn.

Responsible for One Single Capability

A Microservice is responsible for one and only one capability in the overall system. Breaking that statement down there are two parts in there: First, a Microservice has a single
responsibility. Second, that responsibility is for a capability. The single responsibility principle has been stated in several ways. One traditional way is:

"There should never be more than one reason for a class to change."

-- Robert C. Martin SRP: Single Responsibility Principle

While this way of putting it specifically mentions “a class” the principle turns out to apply on other levels than that of a class in an object-oriented language. With Microservices, we apply the single responsibility principle at the level of services. Another, newer, way of stating the single responsibility principle, also from Uncle Bob, is:

"Gather together the things that change for the same reasons. Separate those things that change for different reasons."

-- Robert C. Martin The Single Responsibility

This way of stating the principle applies to Microservices: A Microservice should implement exactly one capability. That way the Microservice will have to change only when there is a change to that capability. Furthermore, we should strive to have the Microservice fully implement the capability, such only that one Microservice has to change when the capability is changed.

A capability in Microservice system can mean a couple of things. Primarily, a capability can be a business capability. A business capability is something the system does that contributes to purpose of the system – like keeping track of users shopping carts or calculating prices. A very good way to tease apart which separate business capabilities a system has is to use Domain Driven Design. Secondly, a capability can sometimes be a technical capability that several other Microservices need to make use of – integration to some third-party system for instance. Technical capabilities are not the main drivers for breaking down a system to Microservices, they are only identified as the result of several of Microservices implementing business capabilities needing the same technical capability.

Individually Deployable

Every Microservice should be individually deployable. That is: When you a change a particular Microservice you should be able to deploy that change of the Microservice to the production environment without deploying or in any other way touching any other part of your system. In fact, the other Microservices in the system should continue running and working during the deployment of the changed Microservice as well as after that new version is deployed and up and running.

Consider an ecommerce site. Whenever a change is made to the Shopping Cart Microservice, you should be able to deploy just the Shopping Cart Microservice. Meanwhile the Price Calculation Microservice, the Recommendation Microservice, the Product Catalog Microservice etc. should continue working and serving user requests.

Being able to deploy each Microservice individually is important for several reasons. For one, in a Microservice system, there are many Microservices and each one will collaborate with several others. At the same time development work is done on all or many of the Microservices in parallel. If we have to deploy all or groups of them in lock step, managing the deployments will quickly become unwieldy typically resulting in infrequent, but big risky deployments. This is something we very much want to avoid. Instead, we want to be able to deploy small changes to each Microservice often resulting in frequent and small low risk deployments.

To be able to deploy a single Microservice while the rest of the system continues to function, the build process must be set up with this in mind: Each Microservice has to be built into separate artifacts or packages. Likewise, the deployment process itself must also be set up to support deploying Microservices individually while other Microservices continue running. For instance, a rolling deployment process where the Microservice is deployed to one server at a time in order to reduce downtime can be used.

The way Microservices interact is also informed by the fact that we want to deploy them individually. Changes to the interface of a Microservice must be backwards-compatible in the majority of cases, so that other existing Microservices can continue to collaborate with the new version the same way they did with the old one. Furthermore, the way Microservices interact must be resilient in the sense that each Microservices must expect the other services to fail once in a while and continue working as best it can anyway. One Microservices failing – for instance because of a short period of downtime during deployment – must not result in other Microservices failing, only in reduced functionality or in slightly longer processing time.

Consisting of One or More Processes

A Microservice is made up of one or more processes. There are two sides to this characteristic. First, each Microservice runs in separate processes from the other Microservices. Second, each Microservice can have more than one process.

That Microservices run in separate processes is a consequence of wanting to keep each Microservice as independent of the other Microservices as possible. Furthermore, in order to
deploy a Microservice individually, that Microservice cannot run in the same process as any other Microservice. Consider a Shopping Cart Microservice again. If it ran inside the same process as a Product Catalog Microservice, the Shopping Cart code might have a side effect on the Product Catalog. That would mean a tight and, undesirable coupling between the Shopping Car Microservice and the Product Catalog Microservice.

Figure 2 Running more than one Microservice within a process leads to high coupling in terms of deployment. If two Microservices share the same process, deploying one will directly affect the other and may cause downtime or bugs in that one.

Now consider deploying a new version of the Shopping Cart Microservice. We either would have to redeploy the Product Catalog Microservice too, or would have some sort of dynamic code loading capable of switching out the Shopping Cart code in the running process. The former option goes directly against Microservices being individually deployable. The second option is complex and at the very least puts the Product Catalog Microservice at risk of going down because of a deployment to the Shopping Cart Microservice.

Each Microservice may consist of more than one process. On the surface this may be surprising. We are, after all, trying to make each Microservice as simple to handle as possible, so why introduce the complexity of having more than one process? Let’s consider a Recommendation Microservice in an e-commerce site. It implements the recommendation algorithms that drive recommendations at our e-commerce site. These algorithms run in a process belonging to the Microservice. It also stores the data needed to provide a recommendation. This data could be stored in files on disk, but is more likely stored in a database. That database runs in a second process that also belongs to the Microservice. The need for a Microservice to often have two or more processes comes from the Microservice implementing everything needed to provide a capability including, for example, data storage and possibly background processing.

Owns Its Own Data Store

A Microservice owns the data store where it stores the data it needs. This is another consequence of wanting the scope of a Microservice to be a complete capability. For most business capabilities, some data storage is needed. For a Product Catalog Microservice, for instance, information about each product needs to be stored. To keep the Product Catalog Microservice loosely coupled with other Microservices, the data store containing the product information is completely owned by the Product Catalog Microservice. It is a decision of the Product Catalog Microservice how and when the product information is stored. Other Microservices – the Shopping Cart Microservice for instance – can only access product information through the interface to the Product Catalog Microservice, never directly from the Product Catalog Store.

Figure 3 One Microservice cannot access another’s data store. All communication with a given Microservice happens through its public API; only the Microservice itself is allowed to access its data store directly

Each Microservice owning its own data store opens up the possibility for using different database technologies for different Microservices depending on the needs of each Microservice. The Product Catalog Microservice might use SQL Server to store product information, while the Shopping Cart Microservice might store each user’s shopping cart in Redis and the Recommendations Microservice could use an Elastic Search index to provide recommendations. The database technology chosen for a Microservice is part of the implementation and is hidden from the view of other Microservices. Mixing and matching database technologies with the requirements for each Microservice has the upside of allowing each Microservice to use exactly the database best suited for the job. This can have benefits in terms of development time, performance and scalability, but also comes with a cost. Databases tend to be complicated pieces of technology and learning to use and run one reliably in production is not easy. When choosing database technology for a Microservice, you should consider this trade-off. But also remember that since the Microservice owns its own data store, swapping it out for another database later is feasible.

Maintained by a Small Tema

So far, I have not talked much about the size of a Microservice even though the “micro” part of the term Microservice indicates that they are small. I do not believe, however, that it makes sense to discuss the number of lines of code that a Microservice should have, or the number of requirements, use cases or function points it should implement. All of that depends on the complexity of the capability provided by the Microservice. What makes sense, however, is to consider the amount of work involved in maintaining the Microservice. A rule of thumb that can guide the size of Microservice is that a small team – of, say, 5 people – should be able to maintain a handful or more Microservices. Maintaining a Microservice includes all aspects of keeping it healthy and fit for purpose: Developing new functionality, factoring out new Microservices from ones that have grown too big, running it in production, monitoring it, testing it, fixing bugs and everything else needed. Considering that a small team should be able to perform all of this for a handful of Microservices should give you an idea of the size of a typical Microservice.

Replaceable

That a Microservice is replaceable means it can be rewritten from scratch within a reasonable time frame. In other words, the team maintaining the Microservice can decide to replace the current implementation with a completely new implementation and do so within the normal pace of their work. This characteristic is another constraint on the size of a Microservice: If a Microservice grows too large it will be expensive to replace, only when kept small is it realistic to rewrite.

Why would a team decide to rewrite a Microservice? One reason could be that the code has become a mess. Another is that the Microservice doesn’t perform well enough in production.While these are not desirable situations, they can present themselves. Even if we are diligent while building our Microservices, changes in requirements over time can push the current implementation in ways it cannot handle. Over time, the code can become messy because the original design is bent too much. The performance requirements may increase too much for the current design to handle. If the Microservice is small enough to be rewritten within a reasonable time frame, these situations are OK from time to time. The team simply does the rewrite with the all the knowledge obtained from writing the existing implementation as well as the new requirements in mind.

Excerpted from Microservices in .NET

NCraftsConf Talk Video

This is somewhat late, but still: Here is a recording of the talk I did at NCraftsConf in Paris in May:

LAYERS CONSIDERED HARMFUL - Cristian Horsdal from NCRAFTS Conferences on Vimeo.

Why Write (OWIN) 'Middleware' in Web Applications?

TL;DR

Middleware - in the OWIN sense - can help you modularize your web application code, and maybe even enable you to reuse the plumbing code needed to deal with cross cutting concerns.

What is Middleware?

The context of this post is web applications, and in that context middlewares are pieces of code between the web server and the application code:

When a request comes in it is first handled by the web server, which hands it off to the first piece of middleware, which hands it to the next and so on until the request reaches the application code - i.e. the Nancy Modules or MVC/WebAPI controllers. The response takes the opposite route from the application, through the middlewares and the web server. You can think of that as a pipeline. The pipeline starts and ends with the web server, has the middlewares along the way and the application in the middle:

The trick here is that the middlewares all conform to the same interface; meaning that the pipeline be composed in many different ways - we may omit middleware 3, insert a 4th between 1 and 2 or swap 2 and 3.

Put Cross Cutting Concerns In Middleware

When we look at the handling of an HTTP request as a pipeline there are some things that we can immediately see that we could stick into that pipeline and thereby have applied to all requests. The obvious examples are request and response logging, perfomance logging and authentication. Indeed these are things that there are OSS OWIN middleware implementations for.
Slightly less obvious maybe are the more application specific things like request throttling, caching, redirects of obsoleted URLs, or even things like integration to a payment gateway.
The common theme of these candidates for middleware is they are moving concerns that should otheriwse have been handled by the application code out into the pipeline. This not only cleans up application code but also lends you the flexibility of the pipeline to re-configure the cross cutting stuff in one pipeline across all end points.

Middleware in a Service Oriented Architecture

I'm usually wary of the promise of reuse, but it does depend on where the reuse happens. There is usually not a whole lot of reuse of business logic between applications. But the frameworks they are built on are hugely reused. Middleware sits (well) in the middle.
In the case of service oriented architecture (where HTTP is use for inter-service communication) serving one or more applications there is a potential for reuse across services of the types of plumbing that are specific to this particular architecture but not to each individual service. Things like

• adding monitoring endpoints to each service (middleware can short circuit the pipeline such that requests never reach the application)
• collecting usage data
• opening and closing database sessions
• keeping track of correlation tokens
• caching rules
• ...

These are all examples of things that you don't want to write for every single service, but that you might want every service to have. The more services there are and the smaller each one is the more important the reuse of plumbing becomes. Wrapping each of these cross cutting pieces of functionality into middleware which plugs right into the pipeline makes them readily reusable across services.

OWIN and ASP.NET 5

Both OWIN and ASP.NET 5 supports the idea of middleware. In OWIN it's one the central features. In ASP.NET 5 the same feature is available 1) because ASP.NET 5 supports OWIN and 2) because it also brings it's own notion of middleware to the table.

In future post I'll dig more into both OWIN middleware and the 'proprietary' ASP.NET 5 middleware.

Breaking Domain Layer Dependency on Data Layer: Use Events for Writes

In this final part of my series (part I, II and III) on reversing the dependency from Domain Model to Data Access code found in many traditional layered code bases I will show how raising Domain Events when Domain Objects are updated can further decouple the Data Access component from the other components.

Recap

To recap the initial situation is this:

The two previous posts show how to reverse the dependency between Domain Model and Data Acces either by letting the Presentation Layer talk to the Data Access code or by invoking Dependency Inversion.

Leaking Data Access Concerns?
Reversing the depedency by letting the Presentation component depend directly on the Data Access component works, but leaves the responsibility of making the Data Access component save entities to the Presentaion component. E.g. in this snippet of Presentation Layer code there is a call to wishListRepository.Save(wishList);


This may seem innocent enough, but saving is not a presentation concern, which means presentation code should not be making that .Save call.

Reversing the depedency by Dependency Inversion also works and alleviates the Presentation code from calling .Save. On the other hand the Domain Model code has to make the Data Access code save entitites. Doing this without the compile time dependency pointing from Domain Model to Data Acces is achieved by introducing interfaces like this in the Domain Model:


Again this seems innocent enough, but what about the Save method? On the one hand it does not reveal anything in particular about the underlying Data Access code or database. On the other hand it does imply that WishList objects can be saved - presumably to a database of some kind. Furthermore the Domain Model has to decide when to call .Save, which is not really a Domain Model concern. The Domain Model should be responsible for the Domain logic. The 'what' and the 'how' of persistence is the responsiblity of the Data Access component. The 'when' of persistence is also the responsibility of the Data Access component in simple cases and of Infrastructure code in more complicated cases.

Domain Events Enable Further Decoupling
If the Presentation component is not to call .Save and the Domain Model is not to call .Save then what? -The Domain Model will raise events when Domain Objects are updated - as done at line 10 in this:



The actual saving of the WishList is done in an event handler, which can be something along the lines of - notice that this class is part of the Data Access component and uses the Domain Model:



Now neither the Domain Model or the Presentation component makes any decisions about what, how or when to persist. The Presentation layer only retrieves Domain Objects as needed and interacts with them. The Domain Model just implements the Domain Logic and raises events whenever there are state changes. It is the full responsibility of the Data Access component to handle the 'what', 'how', and 'when' of persistence.

A Caveat
It is not always good enough to simply persist state changes right away as Domain Events are raised. Sometimes a Unit Of Work is needed. In such cases the Domain Event handlers should not persist data directly, but rather interact with the Unit Of Work to record changes. Whether the Unit Of Work eventually ends up being commited or not is not the concern of the Domain Event handlers. They can rely on Infrastructure code to take care of persisting or discarding the changes recorded by the Unit of Work.

Simple Eventing
Implementing a simple mechanism for raising and handling Domain Events amounts to iterating over all Subscribers calling them one after another when a Domain Event is raised, and to register all Subscribers. That is having this:



and registering all Subscribers with that class in the Composistion Root. This can be done by explicitly listing all subscribers, by discovering them by scanning assemblies or in case the application uses a IoC/DI Container by asking the Container to resolve Subscribers.

Conclusion
Rasing Domain Events further decouples the Data Access component from the rest of the components. It frees other components of the responsibility of deciding what and when to save, and it allows both for simply saving immediately and for delegating to a Unit of Work.
Raising Domain Events also allows for similar deploupling of other components - e.g. integrations to third parties.

Breaking Domain Layer Dependency on Data Layer: Dependency Inversion

This 3rd post in the series about moving from an architecture where Domain Model code depends on to Data Access code to an architecture where that dependency is reversed shows how that can be accomplished using Dependency Inversion.

To recap the situation we come from is this:

and we want to get to this:

Before
While the dependency points from the Domain Model to the Data Access component we might have code like this in the Domain Model:


which depends on this interface in the Data Access Component:



Reversing the Dependency
Let's take a look at what the Dependency Inversion Principle as formulated by Uncle Bob tells us:

"Abstractions should not depend upon details. Details should depend upon abstractions"

This tells us that the Data Access code - a detailed level - should not depend on the Domain Model - an abstraction - but vice versa. In other words the Domain Model component should be changed to include this interface


which should be used by the Domain Model code:


That new DomainModel.WishListRepository interface should be defined in the Domain Model and implemented in the Data Access component:


And that inverts the dependency.

Conclusion
We have inverted the dependency between Domain Model and Data Access, such that the Data Access component now depends on the Domain Model, while the Domain Model has one dependency less.
As a side effect the type WishListDTO disappeared, so we end with less code and a better separation of concerns.

Breaking Domain Depedency on Data Access: Let Presentation Depend on Data Access

This is the second post about not letting domain layers be depend on data access layers. This post explores the (maybe) most obvious way to reverse the dependency between domain model and data access in classic three-layer architectures

Which is to basically turning this

into this

Notice that the arrow from the Data Access component to the Domain Model? Why is that there?
To answer that lets take a quick look at an example. Say we have a web enabled e-commerce system with an endpoint that lets clients add an item to a logged in users wishlist (in others words a simplistic version of something like Amazons wishlist). To do that the code for the endpoint needs to:

• Get the user the call is made on behalf of
  
• Retrieve that users wish list from the data access component
  
• Add the item to the wish list
  
• Save the updated wishlist back through the data access component
  

Which is accomplished with code along the lines of:



This code looks almost like it would have in a classic 3-Layer Architecture. So how does this code justify the dependency arrow from the Data Access component to the Domain component? It does so because:

• The WishListRepository is found in the Data Access component
  
• The call wishListRepository.GetByUserName(Context.CurrentUser.UserName)
  returns a Domain object - an instance of the WishList Domain class.
  
• The call wishListRepository.Save(wishList) accepts a Domain object
  

So there you have it: Allowing the Presentation component to talk to the Data Access component allows you to cut the dependency from Domain to Data Access, but introduces the reverse dependency. For the reasons argued in the first post (and in the resources referenced there) this a much better position to be in than the one in the top most diagram.

An Added Benefit 
 Notice that the data access component now returns Domain objects when queried. That is possible because the dependency points from Data Access to Domain. It used to be the other way around. So ... what did the Data Acces code return on queries before? Most likely some object containing the relevant data from the data base. These types of object come under different names: WishLisDTO, WishListEntities (as in NHibernate or Entity Framework entity), DataModels etc. Regardless of the name their main purpose is to move data from the database into the layer on top of the Data Access layer (i.e. the Domain layer in the top most diagram). But now that the Data Access code returns Domain objects directly these are no longer needed to move data into the rest of the system. In the cases where the mapping from the database to objects is not complicated these DTOs/Entities/DataModels have become superfluous and can be deleted. This can turn out to be a lot of code that can be deleted.

Downsides 
The major downside here is that the endpoint code has been given slightly more responsibility, since it now has to fetch Domain objects - which it also did before - and also has to save them again after interacting with them. Depending on the state of affairs before breaking the Domain to Data Access dependency the endpoint may or may not have had to explicitly save changes. That may have been handled by a combination of dirty tracking and infrastructure code.
In the 4th post of this series I will show a way to remove this extra responsibility from the presentation layer again.

Domain Models Should Not Depend on Data Layers

This is the first of 4 posts about how to go about moving from an architecture where the Domain Model depends on the Data Acces component, to an architecture where that dependency is reversed.

In this post I outline the problem that I propose solutions for in the next posts.

Why Write About This?
Because I see this problem repeatedly in code that I get to work with. So I might as well write down what I tend to talk to clients about in those situations.

Why Do So Many Domain Models Depend on a Data Access Component?
I think there are several reasons, but the most important one it seems to me is that for many this seems to be the default for any server side software:

So that's how many systems start out. Or so it seems from my experience. At some point in the life time of the system the Business Logic component is repurposed as a Domain Model - this may happen is an attempt to get away from Transaction Scripts, or simply because having a Domain Model is seen as The Right Thing To Do (™).

Why Reverse the Dependency?
As long as the Domain Model depends on the Data Access component both are hampered. Albeit for different reasons.

Having the Domain Model depend on Data Access layer is a violation of the Dependency Inversion Principle because the Domain Model is a higher level of abstraction than the Data Access component. The consequence is that the Domain Model is tighter coupled to the details of Data Access than it should be leading the poorer maintainability and poorer testability. In fact Domain Models should be POCOs with no other dependencies than the standard library.

Having all other component depend - directly or indirectly - on the Data Access component, makes the Data Access code harder and more risky to change because it potentially effects everything else. The Data Access component is at the edge of the system and just like the entire system should not depend on the Prensentation component (another component at the edge) the entire system should not depend on the Data Access component.

Therefore this is preferable:

For deeper explainations of this I recommend Alistair Cockburns article on Hexagonal Architecture and the GOOS book by Nat Pryce and Steve Freeman.

How To Break the Dependency?
The upcoming posts covering these tactics for breaking the dependency:

1. Allow presentation layer to talk to data layer
   
2. Dependency Inversion
   
3. Use domain events for writes
   

BTW. This applies to other stuff too. E.g. integration components or hardware access components.

Slides from Warm Crocodile Talk

At the awesome Warm Crocodile Developers Conference I did a talk under the title "Does Your Architecture Enable Flow?". These are the slides from my talk. As always the slides make less sense without the words, so YMMV.