I first saw “A to AB to B” in reference to deployments in rachelbythebay’s blog post Some items from my “reliability list”.
This describes quite well a deployment method I frequently see, but rarely has a name put to it.
You see it everywhere, and giving something a name often helps to discuss it.
This technique is not AB testing, but you could use it as a way to do AB testing. It’s also subtly different to Blue-Green deployment, and more frequently used.
Example of an A AB B deployment
Let’s say you want to change the type of a column in a database.
Perhaps you’ve had a lot of traffic, and now want to change the attribute ‘viewcount’ in your ‘pageviews’ table from an integer to a bigint.
You need to add a new column, copy the data from one column to the other, and then once the application is using the bigint column, drop the integer column. There may be extra steps, but that’s the gist.
Getting from world A to world B without breaking the world necessitates a multi-stage deploy.
Where world A has viewcount as an integer, and world B has viewcount as a bigint, you therefore need to create an intermediate state during your deploy where the database supports both worlds, i.e. the AB state.
If you went straight from A to B, your application code using the viewcount field might break. The application might not be using the new field yet. It might take multiple days to copy all of the data to the new column. Something unexpected might happen with the new column. In the majority of cases, you definitely don’t want to create the new column and drop the old column in one go.
Definition of an A AB B deployment
A→AB→B is a multi-stage deployment strategy to change software from one state (A) to another (B) via one or many intermediate states (AB), so that clients dependent on that software experience the least possible disturbance.
Similar to other deployment strategies, the aim of an A AB B deployment is to reliably deploy new code.
By reliably, I mean without clients noticing or at least being minimally affected, without unwanted side effects, and with an easy rollback route should anything go wrong.
Benefits to an A AB B deployment
- By doing atomic changes and supporting multiple states, you don’t break the world during a deploy
- Therefore you can more easily do continuous deployment
- And you can rollback if a step goes wrong
Differences between an A AB B deployment and a Blue-Green deployment
The A AB B deployment strategy sounds similar to the Blue-Green deployment technique.
In fact, Martin Fowler uses the same example above in his blog post on blue green deployments:
Databases can often be a challenge with this technique, particularly when you need to change the schema to support a new version of the software. The trick is to separate the deployment of schema changes from application upgrades. So first apply a database refactoring to change the schema to support both the new and old version of the application, deploy that, check everything is working fine so you have a rollback point, then deploy the new version of the application. (And when the upgrade has bedded down remove the database support for the old version.)
The goal of an A AB B deployment and a Blue-Green deployment is the same: a safe migration.
Martin Fowler’s and Wikipedia’s definition of Blue-Green deployment explicitly states it involves running “two servers” or “environments”.
I think Blue-Green deployment is too narrow. A new definition would help us to discuss this deployment technique more broadly.
Blue-Green deployments run two application versions side-by-side then route all traffic to the new one.
This is useful perhaps when talking about how to actually release the software to users (i.e. a way of doing a hot deployment).
In contrast, A AB B deployments describe an intermediate state to solve the “challenge” in Fowler’s example that isn’t solved by the Blue-Green deployment technique.
An A AB B deployment is less about the migration technique (use a router, or not) and emphasises the AB state (which could be many states) of the system rather than having “two easily switchable environments” as Fowler describes.
You could argue that running multiple servers with different code is still putting the software into an AB state, from the client’s perspective. Therefore, one could see a Blue-Green deployment as an implemention technique of the A AB B strategy.
Further examples of A AB B deployments
Releasing a new version of an API endpoint
Rather than making a breaking change to an endpoint, you introduce a new version so that users can migrate in their own way.
You might run
/v1/cat-pictures (world A) and then also start providing
/v2/cat-pictures (world AB). Until you remove that first endpoint, you’re
living in world AB.
Running an AB test with an A AB B deployment
Since an A AB B strategy necessitates that your server provides both A and B (or more) states during the deploy, you could run an AB test while you’re in that middle state, and rollback to A or roll forward to B depending on the result of the AB test.
Gracefully migrating active clients during deploys
Deploys can take a while. Even deploys that are instantaneous can still impact end users if you’re deploying a high traffic service.
During a deploy you might have two servers running side by side running different code.
Unless you’ve configured your router to guarantee that all of a user’s requests go to the same server (old or new) it’s possible for a user’s requests to be handled by both an old and new server during a deploy.
That produces a problem if, say, you’ve replaced an endpoint URL. For example, perhaps you are deploying a change to the CSS of your website. With Ruby on Rails, the assets pipeline will generate a new URL for that CSS, and may discard the old assets URL.
During the deploy you may have users requesting CSS assets from the old or the new assets endpoint. Those requests would fail if they went to the wrong server, since the server would only have one version of the assets to serve.
There are a few solutions to this (e.g. AssetSync), but they reduce to: don’t be lazy, do an A AB B deploy, support both old and new clients simultaneously.
I frequently see opportunities for A AB B deployments, and many people use this approach without giving it a name.
I hope by giving this approach a name it’ll prompt people to think about it more. A AB B isn’t how most deployments go (or even need to go), but since we all use this strategy at one point, it helps to have a common term for it.
You’ll be able to ask others, do we need to do an A AB B deployment here? Or what does the AB state for this migration look like?