WebSocket Throttling & Analytics for WSO2 Micro Gateway

Software engineering internship project: WebSocket throttling and analytics support for WSO2 Micro Gateway, built on top of Envoy proxy.

Oct 2020 – Mar 2021 · Colombo, Sri Lanka

Repo: wso2/product-microgateway

Pull request: wso2/product-microgateway#1943 — Support web socket throttling, merged April 2021 (50 files, ~5.4k lines).

Background

WSO2 API Micro Gateway, later renamed Choreo Connect, is an API gateway built on top of Envoy proxy. It is made up of three components:

• Router — the data plane. An Envoy proxy extended with custom filters, which is what actually receives and forwards API traffic.
• Enforcer — a remote external authorization (ext_authz) service for the Router. The Router calls out to it to decide whether a request is allowed through.
• Adapter — the xDS control plane for the Router. It translates API definitions into the configuration the Router runs on, and pushes it over xDS.

Choreo Connect architecture: the Adapter serving xDS to the Router and Enforcer over gRPC, the Router wrapping Envoy proxy with custom filters, and the Enforcer’s filter chain handling authentication, authorization, rate limiting, and analytics.

Diagram © WSO2, from the Choreo Connect documentation.

What I built

My software engineering internship project was adding WebSocket throttling and analytics to WSO2 Micro Gateway. The gateway could already throttle regular HTTP APIs, but a WebSocket connection is a single long-lived upgrade request. Once it is established, the traffic that matters is frames, not requests. So the work was to write a custom HTTP filter for Envoy proxy that captures WebSocket frames, throttles an already-open connection based on those frames at API, application and subscription level, and publishes per-frame analytics (frame count and frame size) to the analytics backend.

Implementation

Envoy keeps a separate upgrade filter chain dedicated to upgrade requests such as WebSocket and HTTP CONNECT, so that is where the filter had to sit. It hooks the distinct header and body callbacks the filter API exposes. The header path handles the initial upgrade, and the body path is where each subsequent frame surfaces. It began as a native filter against Envoy’s StreamFilter interface and shipped as a proxy-wasm filter; see Challenges and limitations for why.

The filter talks to a gRPC server I implemented in the Enforcer:

1. On the initial HTTP upgrade request, the filter opens a gRPC bi-directional stream to the Enforcer’s gRPC server, and holds it open for the life of the connection.
2. On every frame thereafter, the filter reuses that same stream to publish the frame count and frame size, rather than paying connection setup per frame.
3. Throttling decisions are pushed back down the same bi-directional stream, and enforcement happens at the filter level. The filter drops or allows frames itself, without a round trip per frame.

Behind the Enforcer, the throttling data is published to the WSO2 Traffic Manager over the Thrift protocol, and the resulting throttle decisions come back to the Enforcer as JMS events.

Put together, the end-to-end flow looks like this:

Challenges and limitations

Maintaining a native C++ filter

The filter began life as a native HTTP filter written in C++, which meant maintaining a fork of the upstream Envoy proxy repository, a cumbersome duty on its own. Building Envoy with the filter in it also demanded a considerable amount of resources, which made local development painful.

So we ported it to a WASM filter, written with the C++ SDK. The options at the time were C++, Rust and TinyGo, and only C++ was stable and ready for production.

Envoy had no WebSocket codec

Back then Envoy had no WebSocket codec, and that shaped the whole implementation. In the filter’s onRequestBody callback you always receive a chunk of bytes, never a properly framed WebSocket message. Envoy’s filters use watermarks and flush for TCP flow control, so nothing guarantees that one callback carries exactly one complete frame:

• If a frame is large, a single callback may hold only part of it.
• If frames are streaming, one callback may carry an aggregated chunk containing several frames at once.

Everything the filter counted therefore had to be recovered from a raw byte stream rather than read off a frame boundary.

Ping/pong heartbeats

WebSocket connections stay open using ping/pong heartbeat frames, and those should not count toward rate limiting. Excluding them came down to a very simple check on the frame header:

bool MgwWebSocketContext::isDataFrame(const std::string_view data){
  int frame_opcode = data[0] & 0x0F;
  if(!(frame_opcode >= 8 && frame_opcode <= 15) && data.length() >= 3){
    return true;
  }else{
    return false;
  }
}

The low four bits of byte zero are the frame’s opcode. Under RFC 6455 everything from 0x8 upward is a control frame (close, ping, pong), so anything below that range is a data frame worth counting, with a small length guard on top.

It is deliberately crude, and the missing codec is exactly why it has to be: the check assumes data[0] is the first byte of a frame. When a callback hands over the middle of a split frame, byte zero is payload rather than a header and the opcode read means nothing. So this check, like the counting itself, is only ever as reliable as the chunk boundaries allow.

A limited proxy-wasm ecosystem

The other limitation was not technical at all. At the time, proxy-wasm and the Envoy proxy WASM SDKs had very little around them: most of the SDKs had a single author or owner, and there was not much to learn from: few examples, thin documentation, almost no write-ups from people who had built something comparable.

That left reading the source as the way to work. Understanding how a hook behaved, or what a callback actually guaranteed, generally meant going through the SDK code and figuring it out, rather than looking it up.