- Published on
Blue/Green Deployments with ArgoCD and Kubernetes StatefulSets
- Authors
- Name
- Vid Bregar
Argo Rollouts currently lack native support for blue/green deployments with StatefulSets. Learn how to work around this limitation using only standard Kubernetes and ArgoCD resources.
The Challenge
Blue/green deployment is a strategy used to minimize downtime and reduce risk during software updates. It works by running two identical production environments: one with the current version (blue) and one with the new version (green). After fully testing the new version, traffic is switched over from blue to green. If anything goes wrong, you can quickly switch back to the blue version to keep things running smoothly.
One of the easiest ways to implement blue/green deployments with ArgoCD is by using Argo Rollouts. However, Argo Rollouts currently only support Kubernetes Deployments, not StatefulSets (see issue #1635).
The good news? You can implement a blue/green StatefulSet deployment that might fit your use case using only standard Kubernetes and ArgoCD resources.
Note: While blue/green deployments might sound as simple as spinning up a new StatefulSet and shifting traffic, the reality of stateful applications is often more complex. Can the new version be bootstrapped from a snapshot? Can it rebuild the state on its own? Etc., etc... Naturally these application specific details are not covered.
Code
Explore the example source code here.
See It in Action
Let's say your app is currently running version 1.5.0 (the blue version) and you want to upgrade to 2.0.0. The following examples will be demonstrated with Terraform using ArgoCD Provider for Terraform, but of course the same can be achieved using a pure git approach.
This is the initial applied configuration:
module "example" {
source = "../../modules/argocd_blue_green"
project = local.gcp_project_name
blue_version = "1.5.0"
is_blue_up = true
green_version = "2.0.0"
is_green_up = false
route_to_color = "blue"
}
You can verify that both the Kubernetes service pointing to the blue version and the main service exposing the application are returning the expected responses:
curl http://app-blue.test.svc.cluster.local:8080
> Hello from blue
curl http://app-svc.test.svc.cluster.local:8080
> Hello from blue
Now let's switch is_green_up to true and apply:
- is_green_up = false
+ is_green_up = true
Now that green is up, we can perform tests to verify that the green version seems ok. Note that the main service still routes to the blue version:
curl http://app-green.test.svc.cluster.local:8080
> Hello from green
curl http://app-svc.test.svc.cluster.local:8080
> Hello from blue
Finally, let's route to green:
- route_to_color = "blue"
+ route_to_color = "green"
curl http://app-svc.test.svc.cluster.local:8080
> Hello from green
And after some time if all is well, we can remove the blue StatefulSets:
- is_blue_up = true
+ is_blue_up = false
Under the Hood
Here's a breakdown of the important parts:
The Main Service (app-svc):
This is the single point of access for the application. It's responsible for sending traffic to either the blue or the green version. The service is configured to look for StatefulSet's pods with specific labels:
# Inside the app-svc Helm chart
selector:
app.kubernetes.io/name: {{ .Values.appName }}
app.kubernetes.io/color: {{ .Values.routeToColor }}
Blue and Green Apps (app-blue, app-green):
These are the actual StatefulSets. We can have both running at the same time, or only one. The StatefulSets need to attach the following labels to pods:
# Inside the app-blue and app-green Helm charts
labels:
app.kubernetes.io/name: {{ .Values.appName }}
app.kubernetes.io/color: {{ .Values.color }}
ArgoCD Application Set:
It looks at our settings (like blue_version, is_blue_up, green_version, is_green_up, and route_to_color) and decides which Kubernetes resources to create and manage. All the previous parts are tied together here:
resource "argocd_application_set" "list" {
metadata {
name = "blue-green-example"
}
spec {
generator {
list {
elements = concat(
[{
name = "app-svc"
chart = "blue-green-example-svc" // Chart for the routing service
color = "unset"
version = "0.1.0"
}],
(var.is_blue_up ? [
{
name = "app-blue"
chart = "blue-green-example" // Chart for the application
color = "blue"
version = var.blue_version
}
] : []),
(var.is_green_up ? [
{
name = "app-green"
chart = "blue-green-example" // Chart for the application
color = "green"
version = var.green_version
}
] : []),
)
}
}
template {
metadata {
name = "{{name}}"
namespace = "test"
}
spec {
project = "default"
destination {
server = "https://kubernetes.default.svc"
namespace = "test"
}
source {
repo_url = "example.com/my-repo/helm"
chart = "{{chart}}"
target_revision = "{{version}}"
helm {
release_name = "{{name}}"
parameter {
name = "color"
value = "{{color}}"
}
parameter {
name = "routeToColor"
value = var.route_to_color
}
}
}
}
}
}
}
And it's as simple as that. If you're using Terraform to create the ArgoCD Application Sets, you can even add extra checks to prevent mistakes, like trying to send traffic to a version that isn't running:
locals {
at_least_one_up = var.is_blue_up || var.is_green_up
route_to_is_up = (var.route_to_color == "blue" && var.is_blue_up) || (var.route_to_color == "green" && var.is_green_up)
}
resource "null_resource" "pre_checks" {
lifecycle {
precondition {
condition = local.at_least_one_up
error_message = "At least one of blue or green must be up."
}
precondition {
condition = local.route_to_is_up
error_message = "Cannot route to ${var.route_to_color} because it is not up."
}
}
}
Conclusion
While Argo Rollouts don't yet support Kubernetes StatefulSets, we can easily achieve something similar with standard Kubernetes resources and ArgoCD Application Sets. By using labels and a dedicated service, we can switch traffic between the current (blue) and new (green) versions of our stateful apps. This method ensures minimal downtime and a quick rollback option, making updates much safer.
Need help with your Kubernetes project or want to upgrade your team's skills?
Let's connect