Authorization Cache for Envoy Proxy with WebAssembly

Google Summer of Code 2021 project with Red Hat 3scale: an in-proxy authorization cache for Envoy, built as Proxy-Wasm extensions in Rust.

Jun 2021 – Aug 2021 · Google Summer of Code 2021 with Red Hat 3scale

Repo: 3scale-labs/gsoc-wasm-filters

Write-ups: a two-part series on Red Hat Developer, co-authored with Rahul Anand and mentored by Alejandro Martinez Ruiz — Design an authorization cache for Envoy proxy using WebAssembly and How we implemented an authorization cache for Envoy proxy.

Background

Envoy is an open source edge and service proxy, used as an edge proxy, a middle proxy, a service mesh sidecar, and a daemon set inside Kubernetes. When it fronts APIs managed by Red Hat 3scale API Management, every single HTTP request triggers an external call from the proxy to the 3scale Service Management API to authorize the request and report its usage.

That call sits directly in the request path. It costs latency on every request, and it puts the whole API’s traffic on the management service. The idea behind the project was to keep authorization state inside the proxy, so the external service only has to be contacted on a cache miss.

What I built

An in-proxy authorization cache, implemented as two Proxy-Wasm extensions written in Rust and running in the same Wasm VM:

• An HTTP filter — runs in Envoy’s worker threads, one HTTPContext per request. It intercepts requests, authorizes them against the cached state, and performs rate limiting.
• A singleton service — runs on the main thread, outside the request lifecycle, one instance per Envoy process. It handles background synchronization between the proxy and the 3scale Service Management API.

We built these as WebAssembly modules rather than native Envoy filters for the portability, maintainability, and fault isolation Wasm gives you — and because a native filter means maintaining a fork of Envoy itself.

The Envoy filter chain without the cache (top) versus with it (bottom). With the cache in place, a request only leaves the proxy when the record is missing.

How the cache works

The filter object lives exactly as long as the request does, so it does the bare minimum and pushes anything that can tolerate delay onto the singleton. Each request takes one of two paths:

• Cache miss — no record in shared data. The filter calls out to the 3scale Service Management API for the latest application state, deserializes the response, and stores it under the key {ServiceID}_{AppID}. Every later request for that service and application is a hit.
• Cache hit — the serialized record is fetched and deserialized into Rust structs. The authorization decision comes from the quota left on each metric and method, as configured in 3scale. Authorized requests continue down the filter chain; rejected ones get a local response with the appropriate status and headers straight from the filter.

The HTTP filter objects in the worker threads and the singleton service on the main thread, sharing the per-VM key-value store and communicating over the shared queue.

The two extensions coordinate through primitives the Proxy-Wasm ABI provides, both of which are unique per VM:

• Shared data — an in-memory key-value store holding the cache records. Because the filter and singleton run in the same VM, both reach it directly.
• The shared queue — the communication channel from the filter to the singleton. The filter enqueues request metadata and a callback wakes the singleton, so it never has to poll. The shared queue is one of the more under-represented features of Proxy-Wasm; it can just as well be used for sharing data between worker threads or across VMs.

The singleton then does two jobs on a timer or a configured policy: it aggregates request usage over a time window or a memory limit and flushes it to 3scale in bulk via report.xml, and it pulls fresh application state back down via authorize.xml to refresh the cache records. Both happen in the main thread as background work, without blocking the workers. The trade-off is deliberate: batching means usage shows up in the 3scale dashboard slightly later than real time, in exchange for taking the external call out of the request path.

Features worth calling out

• Unique call-out — an opt-in feature guaranteeing that a cache miss on a given record produces exactly one call-out to 3scale, rather than one per in-flight request. It helps under high load and costs nothing under low load.
• Custom metrics — we used Envoy’s metrics API to expose business-level statistics (applications currently cached, total Authorize calls, Authorize timeouts, and so on), scrapeable by Prometheus and chartable in Grafana.
• Visible logs — with the feature enabled, trace logs for a request are emitted in its response headers and matched against regular expressions by the tests, which is how we got unit-test-like visibility into a module that can only run inside a live proxy.
• End-to-end test framework — a Docker Compose setup running every real service with no mocking, able to start a fresh proxy per test or per suite, generating Envoy configuration from a shared template.

Results

The whole point of the project was request latency, so we benchmarked it extensively, across different traffic profiles and configurations.

Without the cache, maximum response time spikes repeatedly, peaking near 390 ms.

With the cache, the same profile settles into single-digit milliseconds after the initial cache-miss spike.

The proxy with the internal cache outperformed the proxy without it by a wide margin. The transition from cache miss to cache hit is visible in the latency curve itself, as a sudden drop in the maximum. Benchmarking the unique call-out feature separately showed what you would expect: 99% of request latencies are the same either way, and the difference shows up in the last 1% — the high-latency requests during the initial cache-miss window. All the benchmark results are in the repository.

Challenges and limitations

Locking across worker threads

Proxy-Wasm gives you no synchronization mechanism between worker threads, which is exactly what the unique call-out feature needs. We raised an issue upstream asking for one, but an ABI change was never going to land inside a Summer of Code, so we built the lock ourselves out of shared data as a placeholder plus message queues to resume waiting contexts on other threads. That meant bending set_shared_data past its intended use — passing a non-zero CAS value so that multiple threads could not initialize the same entry. A vanishingly small chance of double initialization remains, though it always corrects itself.

The cache is a hashmap

Underneath, the cache is a C++ hashmap, so it inherits whatever problems hashmaps have at scale. Running a dedicated store such as Redis alongside Envoy would solve that, at the cost of giving back some of the latency the cache was built to save.

Concurrency has a ceiling too: the Proxy-Wasm host implementation uses one mutex per access type, so at most one read and one write can touch the cache at a time. Per-entry atomic operations would let threads work on different entries in parallel, for a small increase in memory use.

ABI gaps

At the time there was no way to fully delete an entry from the cache — the ABI simply did not expose it, pending a maintainer update. Unit tests were the other casualty: a Proxy-Wasm module only runs inside a host runtime, which is available during integration testing, and at that point individual functions are no longer reachable. Visible logs were our way around it.

Coarse singleton configuration

The singleton can run in one of three modes — container, periodic, and default — which decide how flush and update operations are performed. They are simple enough to be limiting: there is no way to define the behaviour of individual stages within a flush or update. A policy-based system would make the singleton both more configurable and more fault tolerant.