Coupling

Consistency

Cohesion

I'm Bob!

Find me on Twitter: @bob_the_mighty
Find slides on Github: github.com/bobthemighty/talk-boundaries
Read my blog: codefiend.co.uk
Read my book: cosmicpython.com

Talk to me about

Climate change
How to find work in climate
How to do your first talk
DDD and serverless architectures

Why does this matter?

Why this talk? Partly because when the organisers asked me to take part, I was halfway through writing a course on software architecture, and this is the stuff I was thinking about. Mostly because I think we don't spend enough time talking about the fundamentals. Lots of people in software talk about libraries you can use, or techniques you can apply, or services that are available, but we rarely go back to basics and talk about the principles that help us to make good choices.

I've been working at Cazoo for a few years. For those of you who don't know, Cazoo is an online car dealership - we buy cars from auction, we fix them and make them look pretty, then we sell them on our website and deliver them to your house. It's like Amazon for second hand cars.

For a while we were probably the fastest growing business in the whole world. We built the first version of our system in about 90 days from start to finish, we made £100M in our first year of trading, and after 18 months we launched on the NYSE for $8bn.

So people would say to me, you've built this thing in about five minutes, what kind of scaling problems do you have?

And I'd inevitably say that actually we'd used Lambda, and Dynamo, and these serverless technologies so scaling wasn't a problem for us at all. Amazon are very good at scaling those technologies and we outsource all the operational stuff to them.

Boundaries are hard to get right

Boundaries are hard to change

Three lenses

Coupling

Consistency

Cohesion

Coupling

"Egg balanced on Spoons - TROML - 647" by Clint__Budd is licensed under CC BY 2.0 .

Is coupling bad?

Coupling is necessary

Re-use is coupling

API Types are repeated!

DRY is not an improvement

Is reuse bad?

Of course not!

but we have to be mindful of coupling

Things that change together ship together

Define your deployment boundaries

Small is beautiful

Microservices are deployment boundaries

Cloudformation stacks are deployment boundaries

Lambdas are (usually) not microservices

Microservices do not share databases

Functions may share a model

We might also have a domain model that's used by multiple functions. Imagine we're asked to build an application for managing books at a library. We need a function for checking books in, a function for checking books out, maybe we need a function that lets us search for a particular book, or a function that lets us reserve a book and so on. It's likely that we want to have a shared codebase across these functions.

Each of the functions uses our shared codebase as a library, and that - again - means that they are coupled, just as they were by the database. When we want to change how we reserve a book, that means changes to the shared model, and those changes ripple out to these functions. Things that change together ship together, so these are all a single deployment boundary.

This is why you should avoid sharing libraries between microservices as much as possible. DRY turns out to be a bad heuristic in microservice architectures, because re-use is coupling, and the whole point of microservices is to avoid coupling these distinct components.

Temporal coupling

Temporal coupling in book borrowing

If one component is temporally coupled to another, then it's unable to return a result independently. It has to _wait_ for the other component to respond. In this scenario, we have a Book service, and a User service, and when you want to check out a book, the checkout service has to call the book service to update the book's state. This kind of RPC-oriented integration is really common amongst microservice people, and it's horrible and painful and bad, because this checkout service can't do its job if the other service is broken.

In other words, the checkout service is coupled to these other services in _time_. It's coupled because you have to wait. A request that comes in here can't be completed until the requests are completed here and here. If one of those requests breaks, then so does your request.

Event-Driven architecture reduces temporal coupling

Small is beautiful

This is how Cazoo works

Consistency

"letter jumble" by Sky Noir is licensed under CC BY-NC 2.0 .

What is consistency?

Requests through a system should see a coherent state

Consistency matters when making decisions

Strong consistency

All requests see all updates in the same order

Eventual consistency

States converge to a single view over time

If a system is _Eventually_ consistent, then all the processes using the system will _Eventually_ agree on the state. A good example of an eventually consistent system is DNS.

If you add a new DNS record to, say, GoDaddy then it takes time to propogate. You might not see your new DNS entry for 10 or 15 minutes. Some parts of the internet might not see your new entry for 24 hours, or even longer. Eventually, though, so long as you don't keep changing your DNS settings, every part of the system will have the same view of state.

In our library system, that might mean that when we check out a book, it doesn't necessarily show as checked out on the website immediately. This is the case for Cazoo - when you buy a car, we use an event to remove the car from our website and, in the early days, that led to occasional problems where a car was bought by more than one customer. Big oops.

Consistency is the first casualty of temporal decoupling

An eventually consistent story

"A lot has happened since 1925" by Neil. Moralee is licensed under CC BY-NC-ND 2.0 .

Handling concurrency

Pessimistic Locking

Concurrent locks are rejected

Problems emerge under load

If you've got lots of grannies, though, all looking at bits of your catalogue, then you won't be able to change anything. That's going to get very frustrating.

In particular, you can get this situation where you granny has locked the book, and wants to lock the check out table, but the librarian has locked the checkout and wants to lock the book. This is a dead-lock, and it results in broken websites.

In a dead lock, we have multiple clients who are requesting locks on multiple items, and they all hold a subset. The result is that nobody can lock all of the resources that they need to do their job and so the database just *hangs* waiting for someone to get bored and give up

I've spoken to teams before who are deeply freaked out by concurrency, and they insist that they need to have serialized access to tables to avoid "race conditions". They're generally wrong about this, but they haven't thought through the consistency boundaries that would make the most sense for their problem.

In the aggregate pattern, we match the design of our domain model to the consistency boundaries that make most sense.

The aggregate pattern

A cluster of objects, Only accessible through the root

An aggregate is a collection of related objects. One of the classic examples is a shopping cart. In a shopping cart, I have the cart itself, which probably has some identifier, is related to a customer session and so on, and then I have the cart items, which relate to a product, have a quantity and so forth. The cart and its items together make up an aggregate. It's a cluster of objects that change together.

We use our database to enforce consistency at the aggregate level. If I've got two shopping carts, they can both be updated concurrently, but only one concurrent write is allowed to an individual cart.

Aggregates are a consistency boundary


              function addItemToCart(cartId, sku, qty) {

                // fetch the whole cart
                const cart = Carts.get(cartId)

                // make a change
                cart.addItem(sku, qty)

                // persist the whole object
                Carts.put(cart)
              }

When we want to make a change - let's say I want to add an item - we do that by loading the whole cart out of the database, making the changes, and then persisting the whole thing again. We never interact with the items directly, we only work through this public interface - the cart. The cart represents the _scope_ of our persistent state.

Typically, we use a Repository pattern here - repositories load aggregates from a database. There is no repository for loading "cart item". There is no way to access a cart item except by loading the cart, and calling methods on the cart. why? Because the cart is our consistency boundary.

Aggregates reduce locking

The only way to make changes to the items is to fetch the cart, and that means we never need to lock the cart items table. We only need to lock the cart row. Because there's no way for someone to modify the items without using the cart, we can use smaller locks and that reduces the scope for deadlocks and performance problems.

Now we only need to lock a single object - the cart - and that means we can't end up in the situation where the librarian has locked the check out table and your gran has locked the book. This effectively prevents deadlocks but the trade-off is that we have to manage the consistency of the check out items ourselves in our domain model.

Dynamo partitions are a consistency boundary

In serverless, we tend not to use relational databases, because they're a nuisance and you need connection pools and usernames and schema migrations, and all that boring stuff we want to leave to Amazon. Instead the database of choice for serverless applications is Dynamo. Here is how to implement an aggregate pattern with Dynamo DB.

Objects in Dynamo DB have two keys - the partition key and the sort key. Every operation in the Dynamo API (except scans) requires a partition key. Why? Because the partition is the unit of consistency. Whenever I make a change, I have to make a change to a single partition.

One aggregate maps to one partition key.

Optimistic concurrency


              class Book {
                checkOut (userId) {
                    this.checkOutBy = userId;
                    this.version ++;
                }

                updateTitle (newTitle) {
                    this.title = newTitle;
                    this.version ++;
                }
              }

Optimistic concurrency


            function checkOutBook (bookId, userId) {

                const book = BookRepository.get(bookId);
                book.checkOut(userId);

                // this throws a ConcurrencyError
                BookRepository.save(book);

            }

Optimistic locks with DynamoDB

Dynamo DB has in-built support for optimistic concurrency control

Inserting a new cart


    {
        Item: {
            pk: cart-123,
            sk: __cart,
            version: 0,
        },
        ConditionExpression: "attribute_not_exists(pk)",
    }

Updating a cart


                  Update: {
                    Key: {
                        pk: cart-123,
                        sk: __cart,
                    },
                    UpdateExpression: "ADD version :inc",
                    ConditionExpression: "version = :prev",
                    ExpressionAttributeValues: {
                        ":inc": 1,
                        ":prev": 0,
                    },
                }

Aggregates balance safety and performance

Strong consistency within aggregates

Eventual consistency between aggregates

Consistency is for decision making

This is because we only need consistency when we're making decisions. In this context a decision is a policy that we apply when we change state in our system.

For example, coming back to our example at Cazoo, it's important that when you pay for a shopping cart, you pay the right amount for the items in the cart. We need to make a decision whether to accept your payment or not, and that decisions depends on the items you've added, and the amount you're paying us. If we don't have strong consistency we could have a bug where a customer can add an air freshener and a car, but only pay us for the air freshener.

Small is beautiful

Finding aggregates in the library

Is our aggregate the book?

Our first thought might be that the aggregate is the _book_. Books are clearly important objects in the context of our library system, and only one person can check out a book at a time. Isn't that a good consistency boundary?

But remember that when I want to mutate the state of an aggregate, I'm going to load the whole thing into memory. do I need the complete history of a book to make borrowing decisions?

What are the operations exposed by a book aggregate? I can check a book out, I can check a book in. If the book is late, do I apply the fine to _the book_? That doesn't sound right - fines are applied to a single customer, not to the book as a whole.

If I want to check out a book at version 3, but someone else has paid a fine on version 3, should my request fail? That doesn't seem correct. The consistency boundary is too large.

Our aggregate is the loan

Pragmatically, it makes more sense to choose the Loan as the aggregate. What operations does a loan have? I can start a loan, I can return a loan, I can report that my dog has eaten the book, and that's _it_. The loan contains all the information I need to know in order to make a decision: was this book returned on time?

We've made our consistency boundary really small. We require that each individual lending is internally consistent - we can't have two updates to the same lending at once - but we _don't_ require a single consistent view of the "book". The book does not exist in our borrowing model. That seems like a problem, though - humans think in terms of books.

Aggregates and view models

Use projections to maintain views

Dynamo Change-streams FTW!

The simplest way to build this kind of system - a CQRS system - in serverless is to use change streams.

In this model we have two dynamo tables, but we could use a single table, it's just easier to show this way.

The _lending_ aggregate lives in this table over on the top. Whenever we insert a record, or update a record, that results in an event on the Dynamo change stream.

We handle that event with a lambda function that publishes it over EventBridge. Our subscriber lambda down at the bottom receives the event and updates the search database.

So this is how we handle consistency issues between boundaries - we identify the boundary where we _need_ things to be strongly consistent, we model those boundaries using the aggregate pattern, and then we use techniques like CQRS to give eventual consistency between those boundaries.

But how do we know what _needs_ to be consistent? How do we find the loan aggregates?

Cohesion

"Time Machine Clockwork" by Pierre J. is licensed under CC BY-NC-SA 2.0 .

What is cohesion?

Cohesion is the flip side of coupling

Cohesion is intra-dependence

Coupling is inter-dependence

Cohesion is intra-dependence - it's the connections between parts within a module. Our

If things are coherent, then they belong together. It's natural for us to put things together that are coherent. Being laser-focused on cohesion actually reduces coupling.

In our borrowing context we need to understand loans and payment fees but over in our catalogue context, we have catalogue items, version histories, and the physical locations of books. Both these bounded contexts are highly coherent, and that means there is very little coupling between them.

Why do utils packages suck?

Utils packages lack cohesion

Reuse is coupling

If you've got a "Borrowing" microservice, and a "Catalogue" microservice in your library, but they both use a shared "utils" package, then any change to the utils package can ripple out to affect both of these systems. We've introduced _coupling_ because we have a shared dependency on this low _cohesion_ lib.

By contrast, a well-designed package has _high_ cohesion. Inside a package, all the components should be closely related, they _should_ be talking to each other. Between packages we want to reduce our surface area. In a well-designed package we have a small number of public types and functions that _abstract_ a large number of private types and functions.

In our borrowing context we have only a couple of public types - loan and penalty fee - and all the internal details are hidden away. By contrast in a utils package, everything is public - by definition.

Bounded contexts

Bounded contexts are conceptual boundaries

Decide what's in and what's out

Again, our natural inclination is to look for nouns that we're familiar with. In our library example, the intuitive solution is to build a Book service that does ... everything.

If we want to avoid building a giant ball of mud, we need to think in terms of the use-cases of our system. Instead of nouns, we have to look for _verbs_.

If we return to our original view of the system, we've got four major use cases:

Customers "Borrow"
Customers "Pay fines"
Librarians "Maintain catalogue items"
Both users "Query"

Each of these is a separate process. Each of these has its own requirements for the model. They're all about books, sort of, but those books don't have much overlap.

Borrowing context

Catalogue context

Search context

Context maps

We can construct a map of these concepts like this. This is called a _context_ map and it describes how the parts of our system interrelate at a conceptual level.

We draw out the relationships between our contexts. Both the borrowing and search contexts are down-stream of the catalogue context because they receive data from it.

The payments context depends on Stripe, and has some code - an anti-corruption layer - that converts between Stripe's API model and our internal domain model.

Notice that there _is no book object_. Catalogue items, loans, reservations, payments, index items.

We don't need a book service, because book is a noun, and our systems should be focused on process.

Each of these three bounded contexts has its own specific domain jargon. That's what a bounded context _is_ - it's the set of concepts that you need to understand in order to solve problems. If you're an archivist, you need to know about classifications, and catalogue items. If you're a librarian, you need to know about stock takes and damage reports. If you're a customer, you need to know when to return an item, and whether you're over due. These things don't overlap at all. They're bounded - inside the boundary we use one set of concepts with a rigorous definition.

Putting it all together

Deployment boundary

Things that change together ship together

Deployment boundary

Find boundaries that align to reasons for change
Remember that reuse is coupling

Temporal Boundary

Do the minimum work per-request

Conceptual boundary

What do you need to hold in your head?

Conceptual boundary

A single service might have multiple contexts

Conceptual boundary

Look for processes
Don't model around nouns

Consistency boundary

What's the smallest unit of correctness?

Consistency boundary

Use aggregates to make consistency explicit
Use eventual consistency between aggregates

Questions?

Find me on Twitter: @bob_the_mighty
Find slides on Github: github.com/bobthemighty/talk-boundaries
Read my blog: codefiend.co.uk
Read my book: cosmicpython.com