Progressive Delivery with Canary and Blue-Green on Kubernetes: Argo Rollouts, Flagger, and Gateway API

Canary and blue-green delivery fail when teams treat them as YAML patterns. This guide shows how rollout controllers, traffic routers, metrics gates, and rollback mechanics work together on Kubernetes.

TL;DR

Progressive delivery on Kubernetes is safest when rollout control, traffic control, and metric evaluation are separated. Native Deployments provide rolling updates and revision rollback, but canary and blue-green releases need explicit traffic splitting, health gates, and promotion rules. Argo Rollouts gives Kubernetes-native Rollout, AnalysisTemplate, and blue-green/canary strategies; Flagger automates service-mesh or Gateway API canaries from Deployment changes. The operational win is not slower rollout. It is aborting bad versions before they own user traffic, while keeping rollback targets, metrics, and database compatibility observable.

Original generated architecture diagram showing GitOps promotion, Argo Rollouts and Flagger controllers, Gateway API and service mesh routing, stable and canary ReplicaSets, metric gates, promotion, and rollback. — Progressive delivery is a control loop: change desired state, route a measured slice of traffic, evaluate health, then promote or abort.

Progressive Delivery Is a Runtime Safety System, Not a YAML Pattern

A Kubernetes progressive delivery system has four jobs:

It creates or updates the new workload version without destroying the known-good version.
It routes a controlled amount of real or synthetic traffic to the new version.
It evaluates health from metrics, logs, traces, Kubernetes conditions, and optional webhooks.
It promotes the version, pauses for human review, or aborts and rolls traffic back.

Native Kubernetes Deployments already solve part of this problem. Any change to a Deployment's .spec.template triggers a rolling update; Kubernetes creates new Pods and gradually terminates old ones. Rollback uses ReplicaSet revision history with kubectl rollout undo, and the docs call out a critical caveat: setting .spec.revisionHistoryLimit to 0 disables rollback entirely.

That matters because canary and blue-green releases are not replacements for Deployment hygiene. They add decision points and traffic control on top of the same operational fundamentals: immutable images, readiness probes, PodDisruptionBudgets, stable labels, revision history, and observability that can distinguish a bad canary from a noisy cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  annotations:
    kubernetes.io/change-cause: "payments-api@sha256:8c4f... from release-2026-06-08"
spec:
  replicas: 8
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api@sha256:8c4f...
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
          startupProbe:
            httpGet:
              path: /startup
              port: 8080
            failureThreshold: 30
            periodSeconds: 5

The important line is not the image. It is the combination of revisionHistoryLimit, maxUnavailable: 0, and probes. Without those, a progressive controller can still shift traffic to Pods that are logically unready, or fail to return to a meaningful previous revision.

Choose the Strategy by Failure Mode

Canary and blue-green are often described as "slow rollout" versus "instant switch." That framing is too shallow. The right strategy depends on what kind of failure you expect and how fast you can detect it.

Canary release: expose version v2 to a small slice of traffic, measure, then increase the slice. Canary is strongest when failures are visible in request-level metrics: 5xx rate, p95 latency, failed checkouts, dependency errors, or queue lag. It is weak when failures are rare, delayed, or tied to a long-running workflow that will not complete during the analysis window.

Blue-green release: run blue and green versions in parallel, validate green, then switch the active Service or route. Blue-green is strongest when you need fast rollback to the previous active environment and can afford duplicate capacity during the cutover. It is risky when the two versions share mutable state that cannot move backward.

Rolling update: replace Pods gradually under one Service. Rolling update is efficient and native, but it does not answer "is this version good?" It only answers "did Kubernetes create available Pods?"

Rolling update:
  Best for: low-risk changes, stateless workloads, mature readiness checks
  Weakness: no traffic-weighted health decision

Canary:
  Best for: request-level regression detection, SLO-gated promotion
  Weakness: needs reliable metrics and a traffic router

Blue-green:
  Best for: fast switch/rollback, pre-warmed capacity, release rehearsals
  Weakness: double capacity and state compatibility pressure

For high-traffic services, a canary can produce useful signal in minutes. For low-traffic services, a 5% canary may produce almost no signal. In that case, use synthetic checks, shadow traffic, scheduled pauses, or a larger initial weight. "5%" is not magic; statistically useful sample size is the real gate.

Argo Rollouts: Make Promotion a Kubernetes Resource

Argo Rollouts introduces a Rollout custom resource that can act as a Deployment replacement with canary and blue-green strategies. The value is not only the CRD. It is the controller state machine: steps, pauses, analysis runs, traffic routing, service selectors, and abort behavior become part of declarative desired state.

A basic Argo canary without external traffic routing manipulates ReplicaSet scale. That can work for simple cases, but it has an important limitation: Kubernetes Service load balancing is instance-based, not request-percentage-based. If you set canary weight to 20 without a traffic router, the result depends on Pod counts and connection behavior. For HTTP services, precise progressive delivery usually needs Gateway API, Istio, NGINX, ALB, SMI, or another supported traffic provider.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments-api
spec:
  replicas: 10
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api@sha256:8c4f...
          ports:
            - containerPort: 8080
  strategy:
    canary:
      stableService: payments-api-stable
      canaryService: payments-api-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: payments-api
            namespace: edge
      steps:
        - setWeight: 5
        - pause:
            duration: 5m
        - analysis:
            templates:
              - templateName: payments-api-slo
            args:
              - name: service-name
                value: payments-api-canary.default.svc.cluster.local
        - setWeight: 25
        - pause:
            duration: 10m
        - setWeight: 50
        - analysis:
            templates:
              - templateName: payments-api-slo
        - setWeight: 100

The Rollout expresses intent, but the router implements request distribution. The Argo Rollouts Gateway API plugin exists so the controller can adjust route weights through Kubernetes Gateway API resources rather than provider-specific APIs. That design matters operationally: platform teams can use the same Rollout model across Gateway API implementations that support the needed route behavior.

AnalysisTemplates Turn Metrics into Promotion Gates

Argo Rollouts analysis is where a canary becomes more than a pause timer. An AnalysisTemplate defines measurements, intervals, success conditions, failure conditions, and providers. The official analysis docs show Prometheus-backed success-rate checks and explain that failed measurements can cause a Rollout to abort and return canary traffic to zero.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: payments-api-slo
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 60s
      count: 5
      successCondition: result[0] >= 0.99
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(http_requests_total{
              destination_service="{{args.service-name}}",
              status!~"5.."
            }[5m]))
            /
            sum(rate(http_requests_total{
              destination_service="{{args.service-name}}"
            }[5m]))
    - name: p95-latency-ms
      interval: 60s
      count: 5
      successCondition: result[0] < 350
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            histogram_quantile(
              0.95,
              sum(rate(http_request_duration_seconds_bucket{
                destination_service="{{args.service-name}}"
              }[5m])) by (le)
            ) * 1000

The thresholds above are intentionally concrete, but they are not universal. A checkout API, a search API, and a webhook receiver need different budgets. The rule is to gate on user-visible indicators and canary-attributable signals. Cluster-wide CPU, node memory, or generic pod restart counts are useful context, but they are blunt rollback triggers unless you can tie them to the new version.

Blue-Green with Argo Rollouts

Argo Rollouts blue-green uses active and preview Services. The controller points the preview Service at the new ReplicaSet so you can run smoke checks before switching active traffic. Promotion changes which ReplicaSet the active Service selects.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments-api
spec:
  replicas: 8
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api@sha256:8c4f...
  strategy:
    blueGreen:
      activeService: payments-api-active
      previewService: payments-api-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300
      prePromotionAnalysis:
        templates:
          - templateName: smoke-test
      postPromotionAnalysis:
        templates:
          - templateName: payments-api-slo

The operational caveat is provider-specific routing. The Argo docs warn that AWS ALB Ingress with blue-green is not supported without a chance of downtime because target group updates are not atomic in that path. That does not make blue-green unsafe everywhere; it means the traffic switch must be understood for the actual ingress, Service, or Gateway implementation in front of the workload.

Flagger: Automate Canary from Deployment Changes

Flagger takes a different approach. Instead of replacing every Deployment with a Rollout CR, Flagger watches a target workload and creates the canary machinery around it. It can shift traffic through supported ingress controllers, service meshes, and Gateway API implementations. During analysis, it periodically runs checks; if failed checks reach the configured threshold, it stops the analysis and rolls back the canary.

That model fits teams that want progressive delivery as a platform behavior around ordinary Kubernetes Deployments.

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: payments-api
  namespace: default
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  service:
    port: 80
    targetPort: 8080
    gateways:
      - edge/public-gateway
    hosts:
      - payments.example.com
  analysis:
    interval: 1m
    threshold: 3
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 350
        interval: 1m
    webhooks:
      - name: smoke-test
        type: pre-rollout
        url: http://flagger-loadtester.flagger/
        timeout: 30s
        metadata:
          type: bash
          cmd: "curl -fsS http://payments-api-canary.default/ready"

Flagger's metrics analysis supports service-level objectives such as availability, error rate, and response time, plus custom metric checks through MetricTemplate. It also supports external providers beyond Prometheus. That matters for organizations that already use Datadog, CloudWatch, New Relic, or another metric backend and want a consistent release gate.

The tradeoff is abstraction. Flagger creates and manages generated resources. Platform teams need to document ownership: which Deployment fields app teams own, which generated Services and routes Flagger owns, and how emergency operators pause or resume a Canary without fighting reconciliation.

Gateway API and Service Mesh: Traffic Weight Is Not Replica Weight

Progressive delivery becomes much more precise when traffic weight is handled at the request routing layer. Gateway API HTTPRoute traffic splitting uses backend weights as proportional values. If foo-v1 has weight 90 and foo-v2 has weight 10, the denominator is 100 and the intended split is 90/10. If the weights add to 10, then 9 and 1 mean the same proportion.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments-api
  namespace: edge
spec:
  parentRefs:
    - name: public-gateway
  hostnames:
    - payments.example.com
  rules:
    - backendRefs:
        - name: payments-api-stable
          namespace: default
          port: 80
          weight: 90
        - name: payments-api-canary
          namespace: default
          port: 80
          weight: 10

Service meshes such as Istio provide similar weighted routing. The Istio traffic shifting docs distinguish route-level traffic migration from orchestrator-level instance replacement: two service versions can scale independently while routing weights determine distribution.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payments-api
spec:
  hosts:
    - payments.example.com
  gateways:
    - edge/public-gateway
  http:
    - route:
        - destination:
            host: payments-api-stable.default.svc.cluster.local
            port:
              number: 80
          weight: 90
        - destination:
            host: payments-api-canary.default.svc.cluster.local
            port:
              number: 80
          weight: 10

This is the heart of canary correctness: replicas and requests are not the same unit. Ten stable Pods and one canary Pod do not guarantee 90/10 request distribution under persistent connections, uneven load, client retries, or gRPC streams. A router with weighted request handling gives the rollout controller a better actuator.

Rollback Mechanics: What Actually Moves Back?

"Rollback" can mean at least five different operations:

Set canary traffic weight back to zero.
Restore the active Service selector to the previous ReplicaSet.
Undo a Deployment revision with kubectl rollout undo.
Revert a Git commit so GitOps desired state points at the previous image or manifest.
Run a data rollback or compensation workflow.

Those are not interchangeable. Argo Rollouts can abort a failed canary and set traffic back to stable. Flagger can roll back its canary when failed checks hit the threshold. Kubernetes can undo a Deployment revision if old ReplicaSets remain. None of those automatically makes a destructive database migration safe.

Use this release contract for production services:

Before rollout:
  - Image is immutable by digest.
  - Previous ReplicaSet is retained.
  - Stable and canary Services are known.
  - Metrics identify stable versus canary traffic.
  - Database changes are backward compatible.

During rollout:
  - Traffic weights are visible in router state.
  - AnalysisRun or Canary status is watched.
  - Alerting includes canary-specific SLOs.
  - Operators can pause, promote, or abort.

After promotion:
  - Old version remains available for a defined delay.
  - Git desired state records the promoted digest.
  - Dashboards compare pre- and post-promotion health.
  - Incident rollback path is documented.

For database changes, favor expand-and-contract:

Add the new schema while old code still works.
Deploy code that can read old and new forms.
Backfill safely.
Switch writes.
Remove old schema only after the previous application version no longer needs it.

If a canary writes data that the stable version cannot read, traffic rollback may make the incident worse. That is the most common progressive delivery blind spot: application rollout is reversible, but state mutation is not.

Operational Caveats That Decide Production Success

Low Traffic Canaries Need Synthetic Signal

A service with 20 requests per hour cannot prove much with a 5% canary in ten minutes. Use synthetic probes, load-test webhooks, or a larger initial canary for low-volume endpoints. Otherwise the controller is making a promotion decision from absence of evidence.

Metrics Need Version Attribution

Canary metrics must distinguish stable and canary. In a mesh, that might be destination service, subset, workload label, or route destination. In Prometheus, queries should filter by labels that actually separate traffic. If both versions share the same metric labelset, the stable version can mask canary errors.

HPA Can Distort Instance-Based Canary

If your canary strategy relies on ReplicaSet scale rather than request routing, Horizontal Pod Autoscaling can change the effective canary share. Prefer traffic-router-based canaries for externally visible HTTP services, and be explicit about HPA behavior for stable and canary ReplicaSets.

Readiness Is Necessary but Not Sufficient

Readiness probes protect traffic from Pods that are not ready to serve. They do not validate payment authorization, search relevance, downstream quota use, or p95 latency under real traffic. Treat probes as admission into the rollout, not proof of release quality.

Blue-Green Doubles More Than Pods

Blue-green may need duplicate cache warmup, connections, queue consumers, certificates, or provisioned concurrency. If green starts cold at promotion time, the "instant" switch can become a latency incident.

Rollout Controllers Need Ownership Boundaries

Do not let CI, Argo CD, Argo Rollouts, Flagger, and a human operator all mutate the same Service or HTTPRoute fields without a contract. Decide which controller owns traffic weights, which owns workload spec, and which owns Git desired state.

A Practical Reference Architecture

For most platform teams, the clean pattern is:

GitOps owns desired workload and progressive delivery resources.
Argo CD or Flux reconciles manifests into the cluster.
Argo Rollouts or Flagger owns progressive state.
Gateway API or a service mesh owns weighted traffic.
Prometheus or another metric provider owns release health signal.
CI owns build, test, signing, and immutable image digests.

CI build -> signed image digest -> Git production PR
       -> GitOps sync -> Rollout or Canary resource
       -> Gateway API or mesh traffic weights
       -> Prometheus analysis
       -> promote, pause, or abort

For Argo Rollouts, app teams usually own the Rollout template and analysis reference, while the platform team owns reusable AnalysisTemplate objects and traffic provider configuration. For Flagger, app teams usually own the target Deployment and a small Canary spec, while the platform team owns metric templates, default thresholds, alert providers, and Gateway or mesh integration.

Minimal Runbook Commands

# Native Deployment state
kubectl rollout status deployment/payments-api
kubectl rollout history deployment/payments-api
kubectl rollout undo deployment/payments-api

# Argo Rollouts state
kubectl argo rollouts get rollout payments-api
kubectl argo rollouts promote payments-api
kubectl argo rollouts abort payments-api

# Flagger state
kubectl get canary payments-api -o yaml
kubectl describe canary payments-api

# Gateway API route weights
kubectl get httproute payments-api -n edge -o yaml

Do not wait for an incident to test these. A progressive delivery platform should have a game-day release that intentionally fails a canary metric and proves that traffic returns to stable.

Why We Built the System This Way

The core design principle is separation of concerns. A workload controller should manage ReplicaSets. A traffic router should manage request distribution. A metrics system should evaluate health. A GitOps controller should reconcile desired state. A progressive delivery controller should coordinate the release decision across those systems.

Putting everything into a single Deployment manifest looks simpler, but it hides the release decision inside Pod churn. Putting everything into a CI script is also tempting, but CI is a poor long-running controller; it does not naturally reconcile cluster drift, watch metric windows, or recover cleanly after process restarts.

Kubernetes progressive delivery works best when the release state is visible as Kubernetes state. You should be able to answer these questions with kubectl:

What version is stable?
What version is canary?
What traffic weight is active?
Which metric gate failed?
Can this rollout be promoted, paused, or aborted?
Which old version would receive traffic after rollback?

If the answer lives only in a pipeline log, the platform is not production-ready.

Frequently Asked Questions

Q: What is progressive delivery in Kubernetes? A: Progressive delivery in Kubernetes is controlled rollout with measurable promotion. Instead of changing all production traffic at once, a controller exposes a new version in stages, evaluates health, and promotes or aborts based on configured policy.

Q: Is Argo Rollouts better than Flagger? A: Neither is universally better. Argo Rollouts is a strong fit when teams are comfortable replacing Deployments with Rollout resources and want explicit step-based strategy definitions. Flagger is a strong fit when the platform wants to wrap existing Deployments with automated canary behavior across ingress, Gateway API, or service mesh providers.

Q: Can I do blue-green with only Kubernetes Services? A: Yes, for simple cases. Run two Deployments with distinct version labels, point a Service selector at the active version, and switch the selector after validation. The missing pieces are automated analysis, promotion history, provider-specific traffic safety, and clear rollback orchestration.

Q: What should trigger automatic rollback? A: Use metrics that are fast, reliable, and attributable to the canary: success rate, 5xx rate, p95 or p99 latency, saturation, and critical business operation failures. Avoid automatic rollback on noisy or slow signals unless the analysis window and failure thresholds are tuned to prevent false aborts.

Q: Does progressive delivery solve database rollback? A: No. It controls application traffic and workload promotion. Database compatibility still needs expand-and-contract migrations, dual-read or dual-write safety where appropriate, backfill planning, and an explicit decision about when old application versions can no longer run.

Search This Blog

DevOpsDreams