Migrating to Blue/Green with AWS CodeDeploy
The purpose of this document isn’t a guide to standing up CodeDeploy, AWS have brilliant documents for this. I want to share the lows and highs to accelerate my key learnings from adding Blue/Green transforms to an existing stack.
Why bother?
The idea here is that your business has zero down time. It also creates a better development experience for teams, enabling them to safely ship their features.
Jargon:
Deployment Configurations; AWS supports Canary, Linear or All-at-once cut over strategies. I opted for all-at-once which will shift 100% of the production traffic once my tests have passed.
Deployment Types available are isolated where you configure a CodeDeploy Application or Assimilated into existing deployments. Here we examine the later option.
General Design Constraints
Compute platforms of lambda and ECS are supported.
Imports and exports are not exported, though exports seem to work.
A custom lookup on resources won’t work for some resources, like service level roles for running your task sets.
You can only specify a single ECS service as the deployment target.
We typically orchestrate a cluster and then have a separate template for a service, obviously if you embrace lambda you won’t have this burden as you not working with clusters, but nested stacks are not supported.
From a cost perspective you can create more clusters at 0 cost anyway. You will want to understand the path to production for releasing this.
For orchestrating across multiple stacks, you could potentially use hooks to do this although AWS don’t recommend this. We have versioned our lambdas for the rest of our workloads and then injected versions into an nginx proxy, but will cover that another time. I’m currently working on the testing this stack and I think there is good opportunity to bring in serverless testing frameworks which could run in parallel.
Key Lessons
Where is my deployment!? When you create a deploy for the first time, it won’t trigger a blue green deploy until a subsequent deployment.
CFT Parameters; default CFT parameters will result in a failure on a subsequent deploy , note to self its bad practice IMO.
BREAKING changes!?
Redeployments with breaking changes … if theres one thing you take from this before you start its this:
***You cannot include updates to multiple resources at once***.
Updates to the above resources with updates to other resources in the same stack WILL FAIL and no linter will tell you. If you need to update resources in the list above as well as other resources in the same stack, do one of the following scenarios (I recommend the first):
· Perform two separate stack update operations: one that includes only the updates to the above resources, and a separate stack update that includes changes to any other resources.
· Remove the Transform and Hook sections from your template and then perform the stack update. In this case, CloudFormation will not perform a green deployment.
Or else! Yes I did try it, because I don’t want to see this in production.
Lambda imports; need to be hardcoded which is bad, AWS best practice is to use auto naming and I find it makes stacks easier to share as templates. Try intrinsic value lookups and ambiguous stack errors will follow.
Default timeouts are set at 1hr for CodeDeploy, so make sure you have a decent catch block for failed scenarios:
So why use it?
Positives here, its forced our work into a shift toward automated testing for everything, which is obviously best practice. I’m hoping that I can quickly update this article and that AWS mature this offering.
All in all, I like it. You could code all this yourself, but I would rather that AWS reduces our operational burden here and allows us to focus on delivering business functionality.
It’s going to be easier for teams to ship feature changes.
