Listen to this article
AI-narrated article - 28 min 01 sec. Prose is preserved; code and diagram cues refer to the examples below.

Read the narration transcript - Open audio

Applying SRE Error Budgets to Services Running on EKS: SLIs, Burn Rates, and Release Decisions

Build error budgets around user requests, not healthy pods. Implement Prometheus burn-rate alerts, diagnose EKS failure modes, and turn reliability evidence into release decisions.

TL;DR

Error budgets turn a service reliability objective into an explicit allowance for unsuccessful user events. For EKS workloads, measure requests at a defined user boundary, calculate good and total events consistently, and distinguish application failures from load balancer failures and missing telemetry. This guide implements request-based SLIs, rolling budget calculations, and multiwindow burn-rate alerts with Prometheus. It also explains latency budgets, low-traffic edge cases, Kubernetes disruption controls, safe release gates, and operational validation. Use the resulting evidence to prioritize mitigation and reliability work without treating autoscaling as a universal response to errors.

Healthy Kubernetes objects can still deliver an unreliable service

An EKS deployment can have ready replicas while checkout requests fail because a database rejects connections. It can also have restarting pods while users see no failures because other replicas absorb the traffic. These situations require different responses, and a pod-health dashboard cannot distinguish them by itself.

SRE error budgets on EKS measure the reliability users receive and turn that evidence into operational decisions. The implementation has four connected steps: define the service objective, measure eligible user events, detect dangerous consumption, and agree what changes are appropriate when reliability deteriorates.

An error budget is an allowance for bad events, not a pool of CPU or replicas. Infrastructure capacity can protect that allowance, but adding capacity does not repair invalid releases, broken credentials, or dependency failures.

The examples below use a hypothetical checkout API with a 99.9% availability objective over a rolling 30-day window. The traffic volumes and incident calculations are illustrative arithmetic, not measured production results. Start by reviewing the EKS reliability guidance against your actual application dependencies and failure boundaries.

Measure the user boundary first. Application and infrastructure telemetry explain failures, but they cannot replace requests that never reached the pods.

Step 1: Define a contract before defining a dashboard

Write an SLO document that answers five questions: which user action, which environment, which eligible events, which success conditions, and which reporting window? Avoid a definition such as “the cluster is available,” which leaves business behavior unspecified.

# A service contract; not a Kubernetes CustomResource.
service: checkout
environment: production
journey: authenticated checkout submission
availability:
  objective: 0.999
  window: 30d
  boundary: public API edge
  eligible: valid attempted checkout submissions, including service-caused failures before application admission
  good: successful completion without a service-caused failure
latency:
  objective: 0.99
  window: 30d
  good: successful eligible response within 0.5 seconds
owner: payments-oncall

The latency objective here is separate from availability. A slow success can satisfy the availability objective and fail latency. A failed request also fails this particular latency definition because it requires success. Document that overlap; do not add the two error counts together and call the sum a unique incident count.

For HTTP APIs, status codes require interpretation. A malformed customer request can be excluded or counted as an expected response, depending on the contract. A 429 generated by an unexpectedly small service quota may represent a service failure. A 200 carrying a failed business transaction may be bad. Consistent classification matters more than a blanket “all 4xx are good” rule.

Choose whether retries represent separate attempts or one user operation. Attempt metrics diagnose servers; operation metrics describe whether the user eventually completed checkout. If you use both, give them separate names and denominators. Never count good user operations above bad server attempts.

The arithmetic: allowed events, spent events, and remaining fraction

Let S be the SLO objective, N eligible events, and E bad events during the same reporting window:

observed SLI = (N - E) / N
allowed bad events = (1 - S) × N
budget remaining events = (1 - S) × N - E
budget remaining fraction = 1 - E / ((1 - S) × N)

If N = 20,000,000 and S = 0.999, the allowance is 20,000 bad events. If 8,000 events are bad, 12,000 remain, or 60% of the allowance. Negative remaining budget communicates an actual SLO miss; hiding it with a display clamp loses useful information.

The familiar 43.2-minute allowance comes from 30 × 24 × 60 × 0.001. That is a time-based 99.9% calculation. It is not interchangeable with a request-based budget when traffic varies. Five minutes of failures during a sale can cost more user events than an hour overnight.

A rolling window also changes continuously: older failures and older successes leave the calculation, while new traffic changes N. Budget remaining is a retrospective ratio, not a fixed bank balance guaranteed to grow monotonically. A calendar-month objective has different boundary behavior and should use calendar-aware reporting.

Step 2: Capture the failures your pods cannot see

Place primary accounting at a boundary that sees the complete eligible journey. Instrumentation inside the handler cannot observe TLS failures, requests rejected upstream, or a load balancer with no usable targets. Use edge telemetry and external probes to cover those blind spots, while retaining application counters for diagnosis.

An ALB exposes separate metrics such as HTTPCode_ELB_5XX_Count, HTTPCode_Target_5XX_Count, and RequestCount. Their definitions and dimensions differ; RequestCount only increments when the ALB can choose a target. Adding every 5xx counter and dividing by that metric does not automatically produce a coherent user-facing SLI. Reconcile populations and aggregation periods first. AWS documents ALB metric semantics and statistics.

In the following implementation, a trusted accounting component exports custom counters. These names are examples, not metrics automatically supplied by EKS:

slo_events_total{service="checkout",environment="production",sli="availability",outcome="good"} 19992000
slo_events_total{service="checkout",environment="production",sli="availability",outcome="bad"} 8000

Initialize both outcome series at zero when the component starts, increment exactly one outcome per eligible event, and keep the labeling contract uniform. A missing bad series should not make the numerator disappear during healthy traffic. An unavailable exporter should not be fabricated into a zero-valued healthy series.

Use bounded labels such as service, environment, and a reviewed route template. User IDs, authorization headers, full URLs, and transaction IDs belong neither in these counters nor in public dashboards. Such labels create unbounded cardinality and may expose sensitive information. Prometheus instrumentation guidance explains why instrumentation and label design need deliberate limits.

Latency: count qualifying events instead of averaging percentiles

For a classic histogram whose observations cover the eligible population, configure a bucket exactly at the contractual threshold. This example assumes histogram observations contain only successful eligible requests, while the denominator counts all eligible requests:

sum by (service, environment) (
  rate(checkout_success_duration_seconds_bucket{le="0.5"}[5m])
)
/
sum by (service, environment) (
  rate(slo_events_total{sli="availability"}[5m])
)

The resulting fraction is successful eligible requests completed within 500 ms. Verify that both metrics represent the same ingress population and service labels. Comparing an internal-handler duration with a browser journey objective would understate network and queueing latency.

A histogram bucket supports an event-based threshold directly. Averaging pod-level p99 values does not yield a service-wide p99, and p99 itself is not an error count. For this classic-histogram example, using an exact bucket boundary also avoids interpolation around the objective. Prometheus explains histogram aggregation and threshold fractions.

Step 3: Build recording rules with explicit missing-data behavior

For every alert window, calculate the sum of per-series counter rates before calculating the ratio. This retains counter-reset handling when individual pods or exporters restart. A service-wide average of pod error percentages gives equal weight to a pod receiving one request and a pod receiving thousands.

The following generator produces the full four-window rule file used later. Install PyYAML in an isolated environment before running it:

python -m venv .venv
# Activate .venv using the command appropriate to your shell.
python -m pip install PyYAML

# generate_slo_rules.py
from pathlib import Path
import yaml

windows = ("5m", "30m", "1h", "6h")
rules = []
for window in windows:
    common = 'environment="production",sli="availability"'
    total = (
        'sum by (service, environment) '
        f'(rate(slo_events_total{{{common}}}[{window}]))'
    )
    bad = (
        'sum by (service, environment) '
        f'(rate(slo_events_total{{{common},outcome="bad"}}[{window}]))'
    )
    rules.extend([
        {"record": f"service:slo_total:rate{window}", "expr": total},
        {"record": f"service:slo_bad:rate{window}", "expr": bad},
        {
            "record": f"service:slo_bad:ratio{window}",
            "expr": (
                f"service:slo_bad:rate{window} / "
                f"service:slo_total:rate{window}"
            ),
        },
    ])
payload = {"groups": [{"name": "checkout-slo", "interval": "30s", "rules": rules}]}
Path("slo-recording.yaml").write_text(
    yaml.safe_dump(payload, sort_keys=False), encoding="utf-8"
)

Run python generate_slo_rules.py, then promtool check rules slo-recording.yaml. The service name stays in the output labels; ensure scrape relabeling supplies canonical labels rather than letting individual exporters invent them. These are ordinary Prometheus rule files. A Prometheus Operator installation requires them under spec.groups in a PrometheusRule, plus selectors that actually load the object. Recording-rule configuration is documented here.

The code deliberately leaves zero traffic as 0 / 0, which produces NaN, rather than declaring the service perfect. Missing series remain missing. This distinction is valuable:

Observation	Interpretation	Follow-up
Positive total, zero bad	Observed requests succeeded	Continue normal reporting
Total and bad both zero	No observed eligible requests	Evaluate expected traffic and external probes
Counter absent	Accounting unavailable or label contract broken	Investigate telemetry
Positive bad, zero total	Inconsistent counters or reset/population issue	Reject the SLI until corrected

Use a separate telemetry alert for this named service:

groups:
  - name: checkout-telemetry
    rules:
      - alert: CheckoutSLITelemetryMissing
        expr: absent_over_time(slo_events_total{service="checkout",environment="production",sli="availability",outcome="good"}[5m]) or absent_over_time(slo_events_total{service="checkout",environment="production",sli="availability",outcome="bad"}[5m])
        labels:
          severity: warning
          owner: payments-oncall
        annotations:
          summary: "Checkout SLI accounting is incomplete"

This alert detects an entirely absent outcome population, not partial loss when one exporter disappears but others remain. Monitor scrape health, exporter inventory, edge request reconciliation, and freshness as additional coverage. Do not use or vector(0) as a universal cure for absent metrics. Prometheus defines rate, increase, and absence functions.

Step 4: Alert on burn rate, then decide how to respond

Burn rate normalizes the recent bad-event fraction by the permitted fraction:

burn rate = recent bad-event fraction / (1 - SLO objective)

With a 99.9% objective, 0.1% bad events means burn rate 1; 1% means burn rate 10; 5% means burn rate 50. A threshold of 5% has no special meaning unless it follows from the objective and response policy.

Under a constant request-rate approximation, the fraction of a 30-day allowance consumed over h hours at burn rate b is b × h / 720. Choosing 2% consumption over one hour gives b = 0.02 × 720 / 1 = 14.4. Choosing 5% over six hours gives b = 0.05 × 720 / 6 = 6. Traffic seasonality can make actual event-budget consumption differ, so display event counts alongside these signals.

Burn rate connects error fraction to the objective. The 14.4 and 6 thresholds shown here are starting points for a 30-day window, with explicit traffic assumptions.

Use a short window to confirm that the long-window problem is still active:

# slo-alerts.yaml
groups:
  - name: checkout-burn
    rules:
      - alert: CheckoutAvailabilityBudgetBurn
        expr: |
          (
            service:slo_bad:ratio1h{service="checkout",environment="production"} > 14.4 * 0.001
            and
            service:slo_bad:ratio5m{service="checkout",environment="production"} > 14.4 * 0.001
          )
          or
          (
            service:slo_bad:ratio6h{service="checkout",environment="production"} > 6 * 0.001
            and
            service:slo_bad:ratio30m{service="checkout",environment="production"} > 6 * 0.001
          )
        labels:
          severity: page
          owner: payments-oncall
        annotations:
          summary: "Checkout availability budget is burning rapidly"

Each pair uses and; the pairs use or. All ratio rules retain identical service/environment labels. A long arbitrary for duration would postpone severe outages; evaluate notification noise using historical traffic before adding delays. Add slower ticket conditions when your team can act on sustained lower burn. Google's SLO alerting chapter provides the multiwindow strategy and its limitations.

Low traffic needs a separate policy. One failed operation among ten is a 10% fraction regardless of whether ten requests arrive in a second or a day. Do not blindly suppress high burn with a request-count floor if every transaction matters. Longer windows, business-critical individual-event alerts, and synthetic journey checks serve different needs. Keep synthetic requests identifiable so successful probes cannot dilute actual customer failures.

Calculate rolling budget remaining independently of paging

This query returns the remaining fraction for the availability objective:

1 - (
  sum by (service, environment) (
    increase(slo_events_total{environment="production",sli="availability",outcome="bad"}[30d])
  )
  /
  (
    0.001 * sum by (service, environment) (
      increase(slo_events_total{environment="production",sli="availability"}[30d])
    )
  )
)

Require retention covering the full window, historical label continuity, and accounting coverage checks. Do not compare a newly installed seven-day metric history with a 30-day contract and call the result complete. For large fleets, precompute aggregate counts or use an appropriately retained backend instead of repeatedly scanning every raw series over 30 days. Document reset handling, sampling gaps, and duplicate ingestion: this is operational event estimation, not a financial ledger.

EKS mitigation: follow the failure mechanism

Once a page fires, establish user impact and stop dangerous changes. Then choose a mitigation that fits the failure:

Evidence	Likely mechanism to investigate	Candidate mitigation
Errors begin with a release; old revision succeeds	Application or configuration regression	Roll back after checking schema compatibility
CPU and queueing rise; dependency has headroom	Application saturation	Add validated pod/node capacity or shed optional work
Database latency and connections spike	Shared dependency bottleneck	Limit concurrency, reduce retries, restore dependency capacity
ALB failures rise while app request counts drop	Target availability or edge failure	Check target health, readiness, routing, and recent infrastructure changes
One zone disproportionately fails	Failure-domain placement or dependency	Shift traffic only after verifying surviving capacity

HPA computes desired replicas from configured resource or custom metrics; it does not come with a universal “error budget allocator.” If error ratio feeds an autoscaler, prove that more replicas improve the causal bottleneck and impose a safe limit. Kubernetes documents HPA behavior and custom metrics.

Retries deserve the same scrutiny. If every pod retries database timeouts several times, doubling pods can multiply database pressure. Bound attempts, use jitter, cap concurrent work, and require idempotency for side-effecting operations. A successful retry can protect a user-operation SLI while increasing backend-attempt failures; both views are useful.

Kubernetes controls protect different failure paths

For an existing replicated stateless service, this partial Deployment configuration shows deliberately conservative rollout settings:

# Merge into the existing Deployment; this is not a complete manifest.
spec:
  replicas: 3
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

This strategy needs spare resources for the surge replica and a readiness probe that reflects ability to serve. Progress can stall when capacity is unavailable. The Deployment controller does not guarantee that ready pods are delivering correct business responses. Deployment documentation explains rollout controls.

For supported voluntary evictions, a separate disruption budget can constrain maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
  namespace: production
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout

This object depends on the selector matching the intended replicated workload. It does not prevent involuntary node failure, and workload rollout behavior needs its own strategy. A restrictive budget can also block draining during maintenance. Kubernetes details PDB scope and operational behavior.

Distribute replicas across failure domains when capacity exists. An example scheduling fragment is:

# Under spec.template.spec of the Deployment.
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: checkout

This constrains placement across eligible topology domains; it does not provision nodes or guarantee three-zone placement by itself. Verify node labels, taints, affinity, and actual eligible zones. Hard constraints trade placement consistency against pending pods when capacity disappears. Topology spread documentation describes these scheduling rules.

Turn the budget into a release policy with an accountable owner

Burn alerts protect the service during incidents. A budget policy determines what engineering work takes priority afterward. Agree it with product and service owners before a breach, including exceptions for urgent security work and reliability repairs. Google's example error budget policy illustrates using reliability results to govern change.

The following is a suggested local policy, not an industry mandate:

State	Release decision	Required action
Valid measurement, budget healthy, no active dangerous burn	Normal reviewed delivery	Observe service and canary signals
Budget declining rapidly	Require an explicit risk review	Identify top failure contributors
Budget exhausted	Pause discretionary feature changes	Prioritize recovery and prevention
Telemetry incomplete	No automatic reliability approval	Restore evidence or use a documented human exception

Avoid embedding this decision in a single unchecked Prometheus query. A release check should record the service, environment, objective, query time, telemetry status, result, and policy version. Secure its read access and separate it from deployment permissions. The person or automation approving an exception should leave an auditable reason.

A canary also needs its own outcome comparison. If only 1% of traffic reaches a broken revision, a global error fraction can look acceptable while that revision fails every operation. Compare revision-specific success with the stable baseline and maintain an overall user SLI. Use bounded revision labels and retire old series to control cardinality.

Why this design separates accounting, diagnosis, and policy

The edge accounting answers “what did users receive?” Application traces and EKS metrics answer “where did the failure happen?” The policy answers “what should the team do next?” Keeping these responsibilities explicit prevents a healthy node count from overruling bad user outcomes and prevents a metric gap from granting release permission.

This also makes ownership clearer. The platform team can own collection and access control while the service team owns success classification and mitigation. Shared dependencies need their own reliability objectives, but their budgets do not erase customer-impacting failures from checkout's budget.

Validate the complete loop before relying on it

Use rule checks for syntax and targeted fixtures for behavior. Then perform a controlled nonproduction exercise with real telemetry and notifications. Prometheus supports rule-unit tests so alert behavior can be reviewed with changes.

python generate_slo_rules.py
promtool check rules slo-recording.yaml slo-alerts.yaml
promtool test rules slo-tests.yaml

Build slo-tests.yaml using your actual scrape/evaluation intervals and include these cases:

Scenario	Expected result
Constant 0.05% bad events	No page for the shown 99.9% thresholds
2% bad events across both 1h and 5m	Fast pair pages
1% bad events across both 6h and 30m	Slower pair pages
Elevated long window after traffic recovers	Short-window condition prevents a lingering page
Counter resets during exporter restart	Rate calculation preserves service accounting behavior
Zero traffic	No fabricated healthy percentage; probe/traffic policy handles it
Missing good or bad population	Telemetry warning fires
One exporter missing while others remain	Inventory or scrape alert detects partial coverage loss

During the live exercise, record elapsed time from injected failure to observed metric change, alert evaluation, notification, acknowledgment, and mitigation. Those measured delays determine whether the response can protect the budget. Do not infer notification delivery from an alert expression evaluating true.

For production dashboards, show objective and reporting window next to eligible volume, bad volume, remaining fraction, current burn, and measurement coverage. Add deployment annotations and useful dependency panels below that user-facing view. Review the top budget-consuming causes periodically and fund repairs with demonstrated impact.

Frequently asked questions

What is an error budget for an EKS service?

An error budget is the allowed fraction or count of bad user events under an SLO during a defined window. For example, 99.9% availability allows 0.1% bad eligible events. Kubernetes health and infrastructure uptime help explain behavior but do not define the user-facing service result.

Should the HPA scale an EKS application when error rate rises?

Scale when a validated capacity signal shows that replicas can reduce the bottleneck. Error fraction alone does not identify capacity shortage; database exhaustion and configuration regressions need different mitigations. Extra replicas can intensify pressure on a saturated shared dependency.

How do I calculate error budget remaining in Prometheus?

Count total and bad eligible events over the same window, calculate the allowance as total multiplied by one minus the objective, and subtract observed bad events. Divide that remainder by the allowance to express a remaining fraction. Treat incomplete history, missing telemetry, and zero-event windows explicitly before making release decisions.

Do PodDisruptionBudgets protect my service SLO?

A PDB constrains supported voluntary evictions for matching pods. It does not guarantee correct requests, prevent node failures, or replace Deployment rollout settings. Combine it with readiness, placement, capacity, dependency resilience, and user-facing measurements.

Should missing SLI metrics count as success?

Missing metrics indicate an unknown measurement state. Alert on absent or partial accounting and use external probes to retain visibility when the serving path or telemetry fails. Automatic release approval should require valid evidence or an explicit accountable exception.

Put one service through the entire workflow

Choose a single user journey, review its eligibility and success contract, and reconcile boundary counts with application observations. Generate the rules, replay representative incidents, confirm notification delivery, and agree an owner-backed release policy. Then extend the same accounting conventions across the fleet while keeping the success criteria specific to each service.