Fault Injection for kgateway

LFX Mentorship 2026 Term 1: HTTP fault injection (delay, abort, and response rate limiting) added to the kgateway TrafficPolicy API and translated to the Envoy fault filter.

Mar 2026 – May 2026 · LFX Mentorship 2026 Term 1 with kgateway, mentored by Tim Flannagan and Omar Hammami

Pull request: kgateway-dev/kgateway#13730 — feat: add fault injection support to TrafficPolicy, merged April 2026 (25 files, ~1.8k lines), closing #11188.

Design doc: design/11188-fault-injection-support.md, merged as #13731.

Write-up: My LFX Mentorship Journey with kgateway on the kgateway blog.

Background

kgateway is a CNCF sandbox project implementing the Gateway API on top of Envoy proxy: a Kubernetes control plane that turns Gateway API resources and its own policy CRDs into Envoy configuration, served over xDS.

Services in production run into degraded dependencies, network latency, and upstream failures whether or not anyone planned for it. Retry policies, timeout budgets, and fallback paths are all written against those conditions, but without a way to reproduce them deliberately, teams only find out whether any of it works during a real incident.

Fault injection makes those conditions reproducible on demand. Doing it at the gateway is what makes it cheap: faults are applied to traffic in flight, so resilience can be validated without touching application code or standing up separate chaos tooling next to the workload.

What I built

A faultInjection field on the kgateway TrafficPolicy, covering the fault types that matter to a platform team:

• Delay — a fixed latency injected before the request is forwarded upstream.
• Abort — terminate the request early with an HTTP status or a gRPC status.
• Response rate limit — cap the response body data rate, simulating a slow or degraded connection.

Each of the three takes a percentage, so faults can be applied to a slice of traffic rather than all of it. On top of those sit maxActiveFaults, a ceiling on concurrent active faults, and disable, which turns fault injection off for a route that would otherwise inherit it.

Extending TrafficPolicy instead of adding a CRD

The first design decision was where the API should live. The alternative was a standalone FaultInjectionPolicy CRD, which would have given cleaner separation and dedicated status reporting, but it would also have duplicated the entire policy attachment and merge infrastructure, and handed users one more CRD to manage.

Extending TrafficPolicy won because everything the feature needs already exists there. TrafficPolicy handles per-route and per-gateway traffic manipulation, and it is where CORS, rate limiting, retries, and timeouts already live, so fault injection inherits targetRefs/targetSelectors attachment, merge semantics, status reporting, and validation for free, and lands where users are already looking.

Applying a 500ms delay to half the traffic on a route is then just:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: TrafficPolicy
metadata:
  name: delay-injection
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: my-app
  faultInjection:
    delay:
      fixedDelay: 500ms
      percentage: 50

The fault types compose, and gRPC gets first-class treatment rather than being approximated with HTTP status codes:

spec:
  faultInjection:
    delay:
      fixedDelay: 200ms
      percentage: 25
    abort:
      grpcStatus: 14 # UNAVAILABLE
      percentage: 5
    maxActiveFaults: 100

Validation is pushed into the CRD, so bad configuration is rejected at apply time rather than surfacing as a broken listener. fixedDelay is constrained by CEL rules to a well-formed duration between 1ms and 1h, percentages to 0–100, HTTP status to 200–599, gRPC status to 0–16. abort uses ExactlyOneOf=httpStatus;grpcStatus so the two can’t be set together, and the policy as a whole requires AtLeastOneOf=delay;abort;responseRateLimit;disable.

Mapping onto the Envoy fault filter

Underneath, all of this becomes the Envoy HTTP fault filter. The filter has to be present in the filter chain of the HTTP connection manager to be usable at all, but a filter chain is listener-wide while the policy is per-route, so registering it globally with a real fault configuration would inject faults into every route on the listener.

The way around that is to add the filter at the listener level with an empty configuration:

{
  "name": "envoy.filters.http.fault",
  "typed_config": {
    "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
  }
}

That registers the filter and injects nothing. The actual behaviour is attached per route through typed_per_filter_config, which overrides the empty listener-level config for that route alone:

"typed_per_filter_config": {
  "envoy.filters.http.fault": {
    "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault",
    "delay": { "fixed_delay": "3s", "percentage": { "numerator": 50 } },
    "abort": { "http_status": 503, "percentage": { "numerator": 25 } },
    "max_active_faults": 100
  }
}

Faults are therefore off by default and opt-in per route or virtual host, which is the only safe default for a feature whose entire purpose is breaking traffic.

The translation itself lives in a TrafficPolicy plugin (pkg/kgateway/extensions2/plugins/trafficpolicy/fault_injection.go), which builds an intermediate representation from the CRD spec and emits the Envoy protos. Percentages become a FractionalPercent with a HUNDRED denominator; the IR implements Equals via proto.Equal so the control plane can tell when a policy genuinely changed and avoid pushing redundant xDS updates.

The disable override

disable is the piece that makes the hierarchy usable. A TrafficPolicy attached to a Gateway applies to every route under it, which is exactly what you want for a blanket “inject 5% failures across this environment”, right up until one route needs to be exempt.

It works by inverting the same mechanism: a policy with disable set produces an IR whose fault config is nil, and at translation time that emits an empty HTTPFault as the typed_per_filter_config for the route. Because per-route config takes precedence over the virtual host and listener config, the empty override wins and the route runs clean, while its siblings keep inheriting the Gateway-level faults.

What I deliberately left out

The Envoy fault filter exposes considerably more than what shipped, and a good part of the design work was arguing for a smaller API:

• The headers field, which fires faults only when a request carries matching headers, was left out because Gateway API already does this better. An HTTPRoute that matches on x-fault-test with a TrafficPolicy attached to it expresses the same thing in the idiom users already know, without duplicating a matcher inside the policy.
• Runtime key overrides, downstream node filtering, and upstream cluster filtering are Envoy operational knobs. Runtime keys in particular are not Kubernetes-native, and exposing them would have grown the API surface for little user-facing gain.
• Header-controlled fault values, the HeaderDelay/HeaderAbort modes in Envoy where the caller passes x-envoy-fault-delay-request or x-envoy-fault-abort-request per request, are genuinely useful for ad-hoc testing, but niche. The design records them as a future headerControlled option rather than shipping them speculatively.

Testing

Fault injection is awkward to test because it is probabilistic by design and spans a control plane, an xDS push, and a proxy. The PR covers it at three levels: unit tests over IR construction, equality, and the disable path; golden translator tests pinning the generated Envoy config for route-level, gateway-level, and disable-override inputs; and end-to-end tests running real traffic through a real proxy for HTTPRoute abort, HTTPRoute delay, Gateway-level abort, and route-level disable.

Challenges and limitations

Racing the xDS push

The end-to-end suite applied a TrafficPolicy and immediately sent its test request, which quietly assumed that controller → xDS → Envoy propagation had already completed. On a fast machine it usually had. On slower CI clusters it had not: the request arrived before the fault filter was active and came back 200 instead of the configured fault status, so the suite flaked. The fix is to wait for the configuration to actually reach Envoy before asserting on it. With an eventually-consistent control plane, “apply then assert” is a race, not a test.

Percentage granularity

Percentages are translated as a FractionalPercent with a HUNDRED denominator, so the smallest expressible slice of traffic is 1%. Envoy itself supports finer denominators (TEN_THOUSAND, MILLION), and for a service handling very high request volume, 1% may be a much bigger blast radius than anyone wants. Exposing a finer granularity is a straightforward extension if the demand shows up.

Delay bounds

fixedDelay is capped at one hour and floored at one millisecond by CEL validation. Both bounds are judgement calls rather than Envoy limits: generous enough for realistic latency simulation, tight enough that a typo cannot pin a request open indefinitely.

Beyond the mentorship

Fault injection was the headline deliverable, but most of the value of the mentorship was learning the control plane well enough to keep contributing after it ended. Work that followed in kgateway includes BackendConfigPolicy merge semantics and BackendConfigPolicy/BackendTLSPolicy conflict resolution, configurable filter stage positioning for ExtProc, async fetch and retry for remote JWKS, TLS termination for TLSRoute, upstream PROXY protocol support, and forwarding client certificate details upstream.