Lahiru De Silva

Zero Trust Networking for OpenChoreo with Istio Ambient Mesh

Lahiru De Silva — Tue, 14 Apr 2026 00:00:00 GMT

How OpenChoreo Cell architecture and Istio ambient mesh create defense-in-depth for Kubernetes workloads, demonstrated with a simulated NASA Artemis II lunar mission.

Introduction

OpenChoreo is an open-source Internal Developer Platform (IDP) for Kubernetes that recently joined the CNCF as a sandbox project. It provides high-level abstractions such as Projects, Components, Endpoints that compile down to Kubernetes resources, letting developers focus on their applications while platform engineers govern the infrastructure.

One of the most powerful architectural concepts in OpenChoreo is the Cell, which is a secure isolated runtime boundary inspired by Domain-Driven Design and Cell-Based Architecture. Each OpenChoreo Project becomes a Cell at runtime: a dedicated Kubernetes namespace with NetworkPolicies that govern ingress/egress traffic of the Cell.

OpenChoreo currently uses standard Kubernetes NetworkPolicies for Cell isolation, which operate at L3/L4. They control which pods can talk to which pods (and on which ports), but not what HTTP requests are allowed. What if a container inside the Cell is compromised? What if a rogue pod can reach your services but shouldn’t be able to call specific endpoints? NetworkPolicies can’t distinguish a legitimate POST /api/orders from a malicious DELETE /api/users.

This is where Istio Ambient Mesh comes in. Istio ambient mesh can handle the full networking layer beneath OpenChoreo, managing both north/south traffic (external ingress/egress via the Istio Gateway API implementation) and east/west traffic (service-to-service communication via ztunnel mTLS and waypoint proxies). It provides transparent encryption and L7 authorization without sidecars, layering perfectly on top of OpenChoreo Cells.

OpenChoreo + Istio Ambient Mesh Architecture

Together, they create a zero trust networking stack where:

OpenChoreo Cells define what is isolated (namespace boundaries with NetworkPolicies at L3/L4)
Istio Gateway handles north/south traffic (external ingress via the standard Gateway API)
Istio ztunnel secures east/west traffic (mTLS encryption with SPIFFE identities for all pod-to-pod communication)
Istio waypoint proxies enforce what requests are allowed (L7 authorization with identity, method, and path)

Why Istio ambient mesh is a natural fit for OpenChoreo

The Cell-waypoint alignment

OpenChoreo Cell architecture maps remarkably well to Istio ambient architecture:

OpenChoreo Concept	Istio Ambient Equivalent	What It Does
Project (Cell)	Namespace with `istio.io/dataplane-mode=ambient`	Defines the isolation boundary
Cell networking	ztunnel mTLS	Encrypts all traffic both within and across Cells
Cell gateway	Waypoint proxy	Enforces L7 policies at the Cell entrance
Component	Pod with SPIFFE identity	Has a cryptographic identity for authorization

The key insight is that OpenChoreo already creates one namespace per Project-Environment combination. When you label that namespace for ambient mesh enrollment, ztunnel automatically encrypts all pod-to-pod traffic with mTLS. Deploy a waypoint proxy in the namespace, and it becomes the Cell gateway evaluating AuthorizationPolicies before any request reaches your services.

Why ambient, not sidecars?

Istio ambient mode (ztunnel + waypoint) is a better fit for OpenChoreo than traditional sidecar injection:

No pod mutation — OpenChoreo manages pod specs through its rendering pipeline. Sidecar injection would require adding an OpenChoreo Trait, losing the transparent encryption and mTLS becomes a developer concern instead of a platform default. Ambient needs only a namespace label.
Namespace-level enrollment — OpenChoreo already manages namespace lifecycle per Cell. Labeling the namespace enrolls all pods automatically.
Separation of concerns — ztunnel handles L4 mTLS for all pods, waypoints add L7 only where needed. This cleanly separates responsibilities: platform engineers get encryption everywhere by default, while developers define fine-grained L7 authorization policies based on their business requirements.
Lower resource overhead — One ztunnel DaemonSet instead of a sidecar per pod.

Using the standard Kubernetes Gateway API

OpenChoreo generates Gateway and HTTPRoute resources using the standard Kubernetes Gateway API. Istio implements the Kubernetes Gateway API via its istio GatewayClass. This means you can swap the default gateway implementation (kgateway) for Istio by changing a single value:

# Data plane Helm values
gateway:
  gatewayClassName: istio # instead of "kgateway"

The developer experience doesn’t change at all. They still declare endpoints with visibility levels, and the platform handles routing. Istio Envoy-based gateway processes the same HTTPRoutes that kgateway would.

Gateway API Swap

From network isolation to zero trust Cells

OpenChoreo Cells already provide strong isolation. Each Project gets its own namespace with NetworkPolicies that restrict which pods can talk to which pods on which ports. But NetworkPolicies operate at L3/L4. They answer “can this IP reach that port?”, not “should this identity be allowed to call that endpoint?”. And all traffic inside the Cell travels as plaintext. Any process that can sniff the network can read every request.

Adding Istio ambient mesh transforms Cells from network-isolated boundaries into zero trust boundaries:

What changes

	OpenChoreo Cell (default)	OpenChoreo Cell + Istio Ambient
Traffic encryption	Plaintext (unencrypted)	mTLS via ztunnel (HBONE)
Identity model	IP-based (source pod IP)	Cryptographic SPIFFE identity per ServiceAccount
Network policy	L3/L4 NetworkPolicies (port + IP)	L3/L4 NetworkPolicies + L7 AuthorizationPolicies (identity + method + path)
Intra-cell visibility	Any pod in the namespace can reach any port	Any pod can connect, but waypoint enforces identity-based access
Cross-cell authorization	Allowed/denied by namespace-level NetworkPolicy	Allowed/denied by SPIFFE identity + HTTP method + path
Compromised container	Full access to all services in the Cell	Blocked — unknown identity is implicitly denied

The three layer defense

These layers are additive, not replacements. A request must pass all three to reach a service:

OpenChoreo NetworkPolicies (L3/L4) — The perimeter. Controls which namespaces and pods can establish connections. Blocks external traffic that shouldn’t reach the Cell at all.
Istio ztunnel (L4 mTLS) — The encryption layer. Every pod-to-pod connection is wrapped in mTLS via the HBONE protocol. Each pod gets a SPIFFE identity (cluster.local/ns//sa/) which is a cryptographic certificate that proves who is making the request, not just where the request comes from.
Istio waypoint proxy (L7 authorization) — The policy enforcement point. Deployed per-namespace, the waypoint acts as the Cell gateway. It evaluates AuthorizationPolicy rules that match on SPIFFE identity, HTTP method, and path before forwarding traffic to the destination service.

Layer	Blocks	Passes
OpenChoreo NetworkPolicy	External namespaces, unknown pods	Same-namespace traffic, declared dependencies
Istio ztunnel (mTLS)	Unencrypted traffic	All traffic (encrypted), assigns SPIFFE identity
Istio Waypoint (AuthzPolicy)	Unknown identities, wrong method/path	Only requests from authorized identities to permitted methods/paths

A rogue pod deployed within the Cell passes Layer 1 (same namespace) and Layer 2 (gets encrypted with its own identity), but is implicitly denied at Layer 3 as its SPIFFE identity doesn’t match any ALLOW rule. Even if an existing pod is compromised, the attacker is constrained to only the methods and paths that pod’s identity is authorized for.

Defense Layers

NASA Artemis II lunar mission demo

To demonstrate the power of OpenChoreo + Istio Ambient Mesh, a simulated NASA Artemis II mission was built with 5 OpenChoreo Projects (Cells) containing 18 microservices:

Mission entities as Cells

Cell (Project)	Components	Role
Houston	flight-director, capcom, telemetry-monitor, flight-dynamics	Mission control — commands, decisions, telemetry
Kennedy	launch-control, ground-systems, weather-station, pad-operations	Launch operations at LC-39B
Orion	guidance-nav, life-support, crew-interface, comms-system	The spacecraft — navigation, crew health, communications
Deep Space Network	signal-router, goldstone, madrid, canberra	Global antenna array — signal relay
ESA Ops	service-module-monitor, mission-relay	European Service Module monitoring

Each Project becomes a Cell at runtime which is an isolated namespace with its own NetworkPolicies:

apiVersion: openchoreo.dev/v1alpha1
kind: Project
metadata:
  name: orion
  namespace: default
spec:
  deploymentPipelineRef:
    name: default

Components are deployed as services inside each Cell:

apiVersion: openchoreo.dev/v1alpha1
kind: Component
metadata:
  name: guidance-nav
  namespace: default
spec:
  owner:
    projectName: orion
  autoDeploy: true
  componentType:
    kind: ClusterComponentType
    name: deployment/service

Artemis II Cell architecture

Artemis II Communication Map

Each Cell communicates only with the Cells its mission role demands. No more, no less:

Houston → Orion: CAPCOM uplinks commands to the spacecraft comms-system
Orion → DSN → Houston: Telemetry flows through the Deep Space Network back to mission control
Kennedy → Houston: Launch control hands off to the flight director after liftoff
ESA → Houston: The European Service Module monitor relays health data to mission control

The demo includes a mission control dashboard where you can simulate adversaries, observe how each defense layer responds, and neutralize threats in real time to see zero trust networking in action.

Artemis II launch

A rogue pod inside the Cell

A rogue pod gets deployed inside the Orion Cell namespace. Maybe an attacker gained access to the cluster through a misconfigured RBAC role, a leaked kubeconfig, or a CI/CD pipeline vulnerability. The pod runs with its own ServiceAccount and has full network access within the namespace.

As OpenChoreo NetworkPolicies operate at L3/L4, they can’t distinguish a legitimate POST /uplink from CAPCOM versus a malicious one from a rogue pod.

Without additional security, it can:

Abort the mission — POST /decision/abort to Houston flight-director
Hijack navigation — POST /maneuver to Orion guidance-nav, sending the spacecraft off course
Steal classified crew data — GET /crew/vitals from Orion life-support
Send fake commands — POST /uplink to Orion comms-system, impersonating CAPCOM

$ curl -X POST http://guidance-nav:8080/maneuver
{"status":"acknowledged","type":"course-correction","deltaV":"2.3 m/s"}
# 200 OK. No questions asked.

Securing the mission with Istio ambient

Layer 1: Encrypt with ztunnel (mTLS + SPIFFE identities)

Label each Cell namespace to enroll in the ambient mesh:

kubectl label namespace <cell-namespace> istio.io/dataplane-mode=ambient

This does two things:

Encrypts all traffic between pods using mTLS via the HBONE protocol
Assigns SPIFFE identities to every pod based on its ServiceAccount

After enrollment, every pod gets a cryptographic identity: cluster.local/ns//sa/

ztunnel mTLS

Verify all pods are enrolled in HBONE:

$ istioctl ztunnel-config workloads | grep dp-default-orion
dp-default-orion-...  comms-system-...    10.42.0.131  HBONE
dp-default-orion-...  guidance-nav-...    10.42.0.132  HBONE
dp-default-orion-...  life-support-...    10.42.0.134  HBONE
dp-default-orion-...  crew-interface-...  10.42.0.133  HBONE

All traffic is now mTLS encrypted. But encryption alone doesn’t stop the adversary, it just ensures they can’t eavesdrop on other traffic. They can still make requests.

Layer 2: Enforce with waypoint proxies

Waypoint proxies add L7 policy enforcement to the Cell. Think of them as the Cell security checkpoint where every request entering the namespace flows through the waypoint, where AuthorizationPolicies are evaluated.

istioctl waypoint apply -n <cell-namespace> --enroll-namespace

This creates an Envoy-based waypoint proxy (using the istio-waypoint GatewayClass) and labels the namespace so ztunnel routes all inbound traffic through it.

The waypoint naturally acts as a Cell gateway which is the L7 entrance point where identity based access control is enforced. This aligns perfectly with OpenChoreo Cell Architecture, where all cross-cell communication should flow through well defined gateways.

Waypoint Proxy

Now the critical part is defining who can access what. Our approach is pure zero trust: allow only known identities, implicitly deny everything else.

For example, refer the AuthorizationPolicy for Orion comms-system. It combines principal-based identity (SPIFFE identities matching specific ServiceAccounts) with HTTP route enforcement (method and path):

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: comms-system-policy
  namespace: dp-default-orion-development-
spec:
  targetRefs:
    - kind: Service
      group: ""
      name: comms-system
  action: ALLOW
  rules:
    # Houston can uplink commands
    - from:
        - source:
            principals:
              - "cluster.local/ns/dp-default-houston-development-/sa/default"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/uplink", "/uplink/*"]
    # DSN can request downlink
    - from:
        - source:
            principals:
              - "cluster.local/ns/dp-default-deep-space-ne-development-/sa/default"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/downlink"]
    # Intra-cell: only default SA
    - from:
        - source:
            principals:
              - "cluster.local/ns/dp-default-orion-development-/sa/default"

Each rule defines who (the SPIFFE principal) and what (the HTTP method and path). Houston CAPCOM can POST /uplink but can’t GET /downlink. The Deep Space Network can POST /downlink but can’t touch /uplink. Intra-cell traffic from the Orion namespace is allowed on all paths but only from the default ServiceAccount.

There are no DENY policies. If your identity isn’t in the allowlist, the waypoint implicitly denies the request. Every service in every Cell gets a policy like this, each one tailored to the exact communication paths that service needs.

Abstracting Istio complexity with OpenChoreo

The AuthorizationPolicies, waypoint proxy configurations, and SPIFFE identity mappings shown above are powerful but too low level for developers to work with directly. Platform engineers shouldn’t expect developers to author these Istio specific resources directly. This is where OpenChoreo programmable abstraction layer ComponentTypes and Traits comes in.

Platform engineers can encode all of this Istio complexity into ComponentTypes (deployment blueprints that define what Kubernetes resources get created) and Traits (reusable cross cutting concerns that attach to Components). For example, a platform engineer could create a zero-trust-service ComponentType that automatically generates the waypoint proxy enrollment and a baseline AuthorizationPolicy alongside the Deployment and Service. An istio-authz Trait could let developers declare allowed callers with a simple interface:

traits:
  - kind: ClusterTrait
    name: istio-authz
    instanceName: comms-policy
    parameters:
      allowFrom:
        - project: houston
          methods: ["POST"]
          paths: ["/uplink", "/uplink/*"]
        - project: deep-space-network
          methods: ["POST"]
          paths: ["/downlink"]

Behind the scenes, the Trait template renders the full AuthorizationPolicy with SPIFFE principals, namespace mappings, and method/path rules. The developer never sees security.istio.io/v1 or cluster.local/ns/.... Instead they work with project names, HTTP methods, and paths. The platform engineer owns the template, controls the security posture, and can update the underlying Istio configuration across all Components without requiring developer changes.

Key takeaways

1. OpenChoreo Cells + Istio ambient = defense in depth

OpenChoreo Cell Architecture provides the L3/L4 perimeter. Istio ambient mesh adds L4 encryption and L7 authorization inside the perimeter. Neither alone is sufficient and together they cover the full stack.

2. Zero trust is “allow known, deny unknown”

The most effective security model doesn’t try to enumerate attackers. It enumerates legitimate callers and denies everything else. Istio AuthorizationPolicies with SPIFFE identities make this practical. You define who should have access, and the cryptographic identity system ensures no one can fake it.

3. Waypoint proxies as Cell gateways

Istio waypoint proxies are a natural fit for OpenChoreo Cell architecture. Each Cell gets its own waypoint that acts as the L7 entrance point, evaluating identity, method, and path before any request reaches the application. This is the missing piece between OpenChoreo network isolation and full zero trust.

4. Developer abstractions hide infrastructure complexity

OpenChoreo ComponentTypes and Traits can encapsulate Istio specific resources like AuthorizationPolicies, waypoint configurations, and SPIFFE identity mappings into developer friendly interfaces. Platform engineers define the templates once, developers consume them with simple parameters like project names and allowed paths, never touching raw Istio manifests. This separation lets the platform team evolve the underlying security infrastructure without breaking the developer experience.

5. Gateway API as the universal contract

OpenChoreo uses the Kubernetes Gateway API for routing. Istio implements the same API. Swapping the gateway implementation is a single gatewayClassName change. The developer experience is identical, the platform engineer chooses the infrastructure.

Try it yourself

The complete Artemis II demo including installation instructions, deployment steps, and all manifests is available at github.com/NomadXD/samples/artemis-ii-istio-openchoreo. The demo use the OpenChoreo single cluster k3d setup and includes:

A configurable Go microservice (one binary, 18 deployments)
5 OpenChoreo Project manifests with 18 Components
Istio ambient mesh configuration, waypoint proxies, and AuthorizationPolicies
An adversary simulation with real API calls through the Istio gateway

OpenChoreo is a CNCF Sandbox project. Learn more at openchoreo.dev.

Istio ambient mesh is GA as of Istio 1.24. Learn more at istio.io/latest/docs/ambient/.

From Broken Network Drivers to a Working Kubernetes Cluster on Decade-Old Hardware

Lahiru De Silva — Tue, 30 Dec 2025 00:00:00 GMT

What began as a simple idea - “let me reuse my old gaming laptop as a Kubernetes homelab”, turned into a series of interesting challenges. On paper, the plan was straightforward: install Proxmox, spin up a couple of virtual machines, bootstrap a Kubernetes cluster using Kubeadm, and call it a day. In reality, the process became a deep dive through Linux kernel headers, broken network drivers, offline package installs, and typical networking subnet issues.

Setup, objectives & motivation

The Setup: A decade-old gaming laptop with an Intel Core i7 processor with 4 cores and 16 GB RAM (ASUS ROG GL 753VD with Intel Core i7-7700HQ @ 3.80 GHz).

The Objective:

Run Proxmox VE as the base hypervisor
Deploy 2-3 virtual machines
Build a multi-node Kubernetes cluster using kubeadm

The Motivation:

A local environment for testing CNIs, K8s configs, linux networking etc.
To prepare for the upcoming CKS exam by doing the course labs in the local cluster
Turn a retired gaming laptop covered with dust into something useful

Proxmox installation - Secure Boot was not the real problem

I started with a clean Proxmox VE installation using a USB flash drive as described in the official docs. The installation went smoothly, but after rebooting, Proxmox failed to boot properly with the following error.

Could not create MokListTrustedRT: Volume full
Something has gone seriously wrong: import_mok_state() failed: Volume Full

For context, MOK (Machine Owner Key) entries are cryptographic keys used by bootloaders to verify signed kernel modules when Secure Boot is enabled. They’re typically stored in EFI NVRAM and allow users to trust custom or third-party drivers that aren’t signed by the hardware vendor.

At first glance, this made no sense. Secure Boot was already disabled in BIOS.

I investigated the issue further and discovered the root cause: even with Secure Boot disabled, stale MOK (Machine Owner Key) entries were still present in EFI NVRAM from previous OS installations. These leftover entries were confusing the boot chain and filling up the NVRAM storage, preventing Proxmox from storing its own boot variables.

The solution was performing a hard reset for NVRAM. I powered off the machine, held the power button for approximately 30 seconds to perform a NVRAM reset, then booted again. This forces the BIOS to re-initialize the NVRAM variables. On many ASUS motherboards, this clears out temporary “garbage” entries in the UEFI storage, freeing up just enough space for the MokList to be created so you could boot again.

After this reset, all the stale entries were cleared, and Proxmox booted cleanly without any issues.

Network interfaces are DOWN

After installation, Proxmox came up BUT networking didn’t.

This turned out to be the major issue that consumed most of my troubleshooting time. I initially tried bringing the interfaces up manually, but nothing worked. After digging through ChatGPT responses, Google searches, and various Proxmox forums, I finally uncovered the root cause.

Proxmox 9 uses Debian 13 under the hood, which defaults to the r8169 kernel module (maintained by the Linux kernel team) for Realtek Ethernet controllers. Unfortunately, this module has compatibility issues with certain Realtek hardware. The solution was to use the r8168 module instead, which is maintained directly by Realtek and provides better hardware support.

The challenge? Without a working network connection, I needed to perform an offline installation of the r8168 module and blacklist the r8169 kernel module to prevent it from loading.

Offline dependency hell

From another machine with internet access, I downloaded the following .deb packages:

1. proxmox-headers-6.17.2-1-pve_amd64.deb

This provides the kernel headers for the exact Proxmox kernel I was running. These headers describe the internal APIs of the kernel and are mandatory for compiling any out-of-tree kernel module. Without them, the driver simply cannot be built.

2. dkms_3.2.2-1~deb13u1_all.deb

This installs the Dynamic Kernel Module Support (DKMS) framework.

DKMS is responsible for: - Building kernel modules from source - Integrating them with the running kernel - Rebuilding them automatically when the kernel changes

In short, DKMS is the glue between driver source code and the kernel.

3. r8168-dkms_8.055.00-1_all.deb

This contains the Realtek r8168 driver source code, along with DKMS metadata.

This package does not ship a precompiled driver. Instead, it provides: - The driver source - Build instructions (dkms.conf) - Hooks that tell DKMS how to compile it into a .ko module

When installed, DKMS uses this package to compile the actual kernel driver.

r8168-dkms.deb
   │
   ├── driver source code
   ├── dkms.conf
   │
   ▼
DKMS
   │
   ├── uses kernel headers
   ├── runs make
   │
   ▼
r8168.ko
   │
   ▼
Kernel module loaded

I copied the three required .deb packages onto a USB flash drive, mounted it on the Proxmox host, and attempted to install them using dpkg. As I suspected, the installation quickly failed due to unresolved dependencies, which couldn’t be fetched because the system was completely offline.

This is where I paused and reassessed the situation and eventually looked for a temporary way to get internet access, just long enough to fix the driver properly.

A temporary network lifeline

At this point, I asked ChatGPT for alternatives to get temporary internet access on the Proxmox host, just long enough to fix the broken NIC driver properly.

Two options came up: - USB tethering using a mobile phone - A USB to Ethernet adapter

I decided to try them in that order.

USB tethering

I started with USB tethering, hoping it would be the quickest path forward. The interface did show up on the system, but it was in a DOWN state. I tried to bring it up manually, and it transitioned from DOWN to UNKNOWN, but there was still no sign of connectivity - ping attempts failed completely.

After hours of debugging already behind me, this felt like yet another rabbit hole. At that point, I made a call to stop digging.

The USB adapter that saved the day

I was feeling hopeless at this point and almost ready to call it off for the day. But then I suddenly remembered something I’d completely forgotten about; the Baseus 6-in-1 USB-C adapter sitting in my bag.

It has an Ethernet port. Ironically, this was something my coworkers and I had joked about at the office before. None of us could quite justify why a USB adapter would even need an Ethernet port, especially in a world dominated by Wi-Fi. Turns out, this was exactly why.

I plugged the adapter into the laptop, connected an Ethernet cable, and within seconds the interface came up cleanly, obtained an IP address, and for the first time, the Proxmox host had stable internet access. No weird logs. No half-working interfaces. It just worked.

The Baseus 6-in-1 USB-C adapter — its Ethernet port provided the temporary network lifeline that saved the day.

Unlike the onboard Realtek NIC, the USB Ethernet adapter uses a completely different chipset and driver, one that’s well-supported by the Linux kernel out of the box (most adapters use ASIX or AX88179 chip, which are supported in kernels ≥5.x).

That clean, temporary network connection was all I needed to: - Perform a system update - Install dkms, pve kernel headers and realtek r8168 driver

After installing the r8168 driver, I blacklisted the built-in r8169 driver to prevent it from loading and performed a reboot. After the reboot, finally, the nic0 and vmbr0 (Proxmox bridge) interfaces came up successfully.

Bridge network subnet mismatch

After fixing the Realtek driver and finally getting nic0 into an UP state, I expected networking to just work. It didn’t. - The interface was up. - The bridge was up. - The link carrier was detected.

And yet nothing could reach the outside world. At first glance, everything looked healthy: - nic0 showed UP - vmbr0 showed UP

But basic connectivity checks failed. A closer look at the routing configuration revealed the real issue.

Ping failure

The Proxmox host was configured for the 192.168.100.0/24 subnet. However, my actual home network - the router, DHCP server, and other devices lived on 192.168.1.0/24.

In other words: - The bridge was working perfectly just on the wrong network. - The system was trying to reach a gateway that didn’t exist.

Why this happened

During the initial Proxmox installation, I accepted the default network configuration. At the time, the nic0 wasn’t working, so there was no immediate feedback that anything was wrong.

Once the driver issue was fixed, the misconfiguration surfaced: - vmbr0 had a static IP in a non-existent subnet - The gateway pointed to an unreachable address - ARP requests never received replies

This is a particularly nasty class of issue because: - Interfaces appear healthy - No errors are logged - Everything looks correct unless you verify the subnet end-to-end.

The solution was straightforward once the problem was clear. I updated the bridge configuration to match my actual LAN in /etc/network/interfaces.

auto lo
iface lo inet loopback

iface nic0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.50/24
    gateway 192.168.1.1
    bridge-ports nic0
    bridge-stp off
    bridge-fd 0

After restarting the network service with systemctl restart networking: - The gateway responded - Internet access worked - The Proxmox web UI became reachable from other machines in the LAN

Spinning up the Kubernetes cluster

With the Proxmox host finally stable and reachable over the LAN, the rest of the setup felt refreshingly straightforward.

From another machine on the same network, I accessed the Proxmox Web UI and created two virtual machines, both running Ubuntu 22.04:

Control Plane - 2 vCPUs - 4 GB RAM

Worker Node - 1 vCPU - 2 GB RAM

ProxMox VMs

Both VMs were attached to the same bridge interface vmbr0, allowing them to behave like first-class citizens on the local network.

Bootstrapping Kubernetes

Once the VMs were up, I used kubeadm to initialize the Kubernetes control plane and join the worker node to the cluster. With the networking issues behind me, the cluster came up cleanly and behaved exactly as expected.

Pods could communicate, services resolved correctly, and the environment finally felt “boringly stable”.

Headroom for growth

Even after provisioning the 2 VMs, Proxmox itself was only using about half of the available memory. That leaves plenty of headroom to: - Add another worker node - Experiment with multi-node or multi-cluster setups - Run additional infrastructure components alongside Kubernetes

Not bad for a decade old laptop after all !!!

References

Distributed Tracing Infrastructure for Cloud-Native Applications

Lahiru De Silva — Wed, 30 Jul 2025 00:00:00 GMT

As modern applications are increasingly built on a foundation of cloud-native infrastructure and microservices, the simplicity of monolithic debugging has given way to a new era of distributed complexity. When a single user request can trigger a cascade of calls across dozens of services, how do you find the root cause of a slowdown or an error? Traditional monitoring tools often fall short, leaving developers in the dark. This is precisely the problem that distributed tracing was designed to solve. By stitching together the entire lifecycle of a request as it hops across services, it provides the end-to-end visibility crucial for maintaining performance and reliability. This article will guide you through the architectural patterns and core components needed to build a robust distributed tracing infrastructure for your own cloud-native ecosystem.

Core components of a distributed tracing system

The foundation of a distributed tracing system consists of four essential components.

1. Instrumentation

This is the process of integrating code into applications to generate and emit trace data. This data usually takes the form of “spans” which represent individual units of work (e.g., an HTTP request or a database call). Each span includes a unique Span ID, the Trace ID of the overall request it belongs to, start and end timestamps, and other contextual metadata (tags or attributes).

Context propagation ensures that the Trace ID and parent Span ID are passed along as requests flow between services, allowing the reconstruction of the entire trace. OpenTelemetry SDKs are the current industry standard for application instrumentation.

Figure 1: Application instrumentation using OpenTelemetry SDKs

OpenTelemetry (OTel)

OpenTelemetry (OTel) has emerged as the leading open standard for observability, regulated by the Cloud Native Computing Foundation (CNCF). It is a vendor neural approach for application instrumentation. It provides a unified set of APIs, SDKs, data formats (like OTLP), and tools for generating, collecting, and exporting telemetry data—specifically traces, metrics, and logs.

Zero-code / automatic instrumentation

Zero-code instrumentation adds the OpenTelemetry API and SDK capabilities to your application typically as an agent or agent-like installation. The specific mechanisms involved may differ by language, ranging from bytecode manipulation, monkey patching, or eBPF to inject calls to the OpenTelemetry API and SDK into your application. Zero-code instrumentation adds instrumentation for the libraries used by an application. This means that requests and responses, database calls, message queue calls, and so forth are what are instrumented. Application code is not typically instrumented. To instrument application code, code-based instrumentation should be used.

Code-based / manual instrumentation

Code-based instrumentation provides deeper insights and visibility into applications itself in contrast to the zero-code instrumentation. Code-based instrumentation requires developers to add OpenTelemetry SDKs to application code and configure to export telemetry data. Also when the application code is instrumented, the libraries will also automatically export trace data depending on the language specific SDKs and the OpenTelemetry support in the used libraries.

Head sampling

Head sampling is a sampling technique used to make a sampling decision as early as possible. A decision to sample or drop a span or trace is not made by inspecting the trace as a whole. Head sampling is usually performed at the client side and is a stateless operation.

The main purpose of head sampling is to reduce the volume of trace data in high-throughput systems that generate large amounts of telemetry. By making early sampling decisions, head sampling safeguards the collector tier from overload and helps prevent excessive data from reaching the OpenSearch storage layer, ensuring both scalability and cost efficiency.

OpenTelemetry SDK built-in samplers can be used to enforce head sampling at the client side by injecting the following environment variables into the applications.

OTEL_TRACES_SAMPLER - always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG - percentage value [0..1]

2. Collection

Collectors, often deployed as agents running alongside applications or as standalone gateway services, are responsible for receiving telemetry data from instrumented applications. They can perform processing tasks such as batching, filtering, sampling, and attribute enrichment before exporting the data to a backend system.

In the context of distributed tracing, tail sampling and rate limiting are the most prominent use cases of collectors. Tail sampling at the collector level is crucial to filter out the events that are meaningful and have more value for troubleshooting application issues like events with errors, higher latencies etc.

Figure 2: Sampling traces with important data for troubleshooting

Tail sampling

Tail sampling is where the decision to sample a trace takes place by considering all or most of the spans within the trace. Tail sampling gives you the option to sample your traces based on specific criteria derived from different parts of a trace, which isn’t an option with Head sampling.

Tail sampling is done by considering all or most of the spans within the trace and hence this is a stateful operation. This causes a well known limitation with scaling these stateful collectors as applications publishing spans of a trace to multiple collectors will cause fragmented traces. The solution to solve this problem involves a 2 tier collector architecture where the Tier 1 load balances all the spans of a trace to a single collector instance in the Tier 2 based on the Trace ID.

Figure 3: 2-Tier Trace Collector Architecture

Rate limiting

Rate limiting at the collector level acts as a critical safety measure for distributed tracing infrastructure. By enforcing limits on the rate of incoming traces, collectors can prevent overload and protect downstream systems including storage backends and visualization tools from excessive traffic during unexpected incidents, such as application bugs or traffic spikes. This not only helps maintain system stability but also reduces operational costs by avoiding unnecessary ingestion and storage of trace data that may result from accidental or malicious events. Implementing rate limiting policies ensures that your tracing pipeline remains resilient and cost-effective, even under adverse conditions.

3. Backend (sink)

The backend represents the persistence layer of the distributed tracing system. It serves as the terminal sink for all processed telemetry from the collection tier and is engineered to perform the essential functions of data storage, retrieval, and indexing that underpin long-term analysis.

A robust tracing backend must perform three primary functions:

Long-term storage: It must persistently store vast quantities of trace data, making it available for historical analysis, performance trend identification, Service Level Objective (SLO) tracking, and regulatory compliance.
Indexing: To facilitate rapid data retrieval, the backend must create and maintain efficient indexes on key trace and span attributes. This includes indexing by service name, operation name, duration, timestamps, and any custom attributes (tags) that are critical for analysis. The quality of this indexing directly determines the power and performance of the query system.
Query API: It must expose a powerful and flexible API that allows visualization tools and human operators to query the stored trace data, enabling the exploration, filtering, and aggregation required for effective troubleshooting and analysis.

Today’s open-source ecosystem offers a spectrum of powerful tracing backends beyond just one or two solutions. Leading projects like Jaeger, Grafana Tempo, Signoz, and Zipkin each represent a distinct set of design choices and priorities. For instance, the architectural philosophies of Jaeger and Tempo are fundamentally different, tailored for vastly different use cases regarding data querying, scalability, and cost.

For organizations that prefer to buy a solution rather than build and manage one, commercial observability platforms offer tracing as a core feature of a broader, managed service. These platforms abstract away the immense complexity of operating a scalable backend. Key players include Datadog, New Relic, Dynatrace and Splunk.

4. Visualization

The final and most human-centric component of a distributed tracing system is the visualization layer. Raw trace data, which can consist of millions of individual spans, is essentially a stream of structured text that is impossible for a human operator to interpret directly. The visualization layer’s critical function is to transform this overwhelming volume of data into intuitive, interactive graphical representations. This transformation is what enables engineers to rapidly comprehend complex system behavior, identify performance bottlenecks, and accelerate the debugging of failures.

Several distinct visualization paradigms have become standard for presenting trace data, each offering a different perspective on system performance.

Gantt / Waterfall Charts: This is the most fundamental and widely used visualization for analyzing a single trace. It presents the trace as a cascading timeline of horizontal bars, where each bar represents a single span. The left-to-right position of a bar indicates its start time and its relationship to other spans, while its length is proportional to its duration. This view is exceptionally effective for latency analysis; the longest bars in the chart immediately draw the eye to the operations that are consuming the most time, providing an instant visual cue to the source of a slowdown.
Service Dependency Graphs: This provides a high-level, aggregated view of the entire distributed system’s architecture. By analyzing a large number of traces, the visualization tool constructs a topological map where nodes represent individual services and directed edges represent the calls between them. These graphs are typically annotated with key health metrics, such as request throughput (requests per second), error rate, and p99 latency for each inter-service connection. This macro-level view is invaluable for understanding service dependencies, visualizing the potential “blast radius” of a failing component, and getting a quick, at-a-glance overview of the system’s overall health.
Flame Graphs: While visually similar to a Gantt chart, a flame graph serves a different purpose: profiling. It aggregates many traces or spans with the same operation name and stacks them vertically to show which code paths are most frequently executed or are consuming the most time across a large sample of requests. The width of each block in the graph is proportional to its contribution to the total time, making it a powerful tool for identifying inefficient code paths within a single service.

Reference implementation

Prerequisites

Access to a Kubernetes cluster
Helm CLI

OpenSearch Helm charts repository:

helm repo add opensearch https://opensearch-project.github.io/helm-charts
helm repo update

See: https://opensearch-project.github.io/helm-charts

This section provides a practical guide for building an end to end distributed tracing system using Opensearch stack. This include Opensearch DataPrepper as the trace collector, Opensearch as the trace sink and Opensearch Trace Analytics plugin with Opensearch Dashboards for trace visualization.

Deploy Opensearch single node

For ease of use, demonstration purposes, and because there is no tracing-specific dependency with the OpenSearch deployment, we are using a single node deployment. This is not recommended for production use cases.

helm install opensearch opensearch/opensearch -f https://raw.githubusercontent.com/NomadXD/samples/refs/heads/main/opensearch-distributed-tracing/values-opensearch.yaml --namespace opensearch-stack --create-namespace
NAME: opensearch
LAST DEPLOYED: Tue Jul 29 20:48:58 2025
NAMESPACE: opensearch-stack
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Watch all cluster members come up.
  $ kubectl get pods --namespace=opensearch-stack -l app.kubernetes.io/component=opensearch-local-dev-master -w

Deploy Opensearch Dashboards

Deploy Opensearch dashboard and configure it to connect to the Opensearch single node deployment.

helm install opensearch-dashboard opensearch/opensearch-dashboards -f https://raw.githubusercontent.com/NomadXD/samples/refs/heads/main/opensearch-distributed-tracing/values-opensearch-dashboard.yaml --namespace opensearch-stack
NAME: opensearch-dashboard
LAST DEPLOYED: Wed Jul 30 10:55:57 2025
NAMESPACE: opensearch-stack
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace opensearch-stack -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboard" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace opensearch-stack $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace opensearch-stack port-forward $POD_NAME 8080:$CONTAINER_PORT

Deploy Opensearch DataPrepper

Instead of deploying a single node of the Opensearch DataPrepper, multiple nodes are deployed in the peer forwarder configuration to demonstrate the scaling capabilities of the Opensearch DataPrepper.

Figure 4: Opensearch DataPrepper Peer Forwarder Configuration

The Peer Forwarder component in OpenSearch Data Prepper logically implements the 2 Tier collector architecture commonly used for scalable distributed tracing. While the classic model describes a Tier 1 that load balances spans and a Tier 2 that performs tail sampling, Peer Forwarder achieves this in a peer-to-peer fashion: Data Prepper nodes communicate directly with each other, using a hash-ring approach to aggregate events and ensure all spans for a given trace are routed to the same node for sampling and processing. This design provides the benefits of load balancing and stateful trace aggregation without explicit tier separation.

For peer discovery, Data Prepper supports DNS-based configuration, where a DNS provider returns a list of Data Prepper hosts for a given domain name. In Kubernetes environments, this is typically implemented using a headless service, which exposes the individual pod IPs via DNS A records, allowing Data Prepper nodes to discover and communicate with all peers in the cluster.

Initially deploy the Opensearch DataPrepper nodes without peer forwarding as we will face the Chicken and Egg problem if we try to deploy in peer forwader configuration at once. The reason for that is because upon startup the dataprepper nodes will try to connect to other peers using the provided domain name and since every other node will wait for others to start up, this causes a deadlock. To mitigate that scenario, first let’s deploy the dataprepper nodes without peer forwarding.

helm install opensearch-data-prepper opensearch/data-prepper -f https://raw.githubusercontent.com/NomadXD/samples/317e716c4ec3f3d2d846ef2f268d7fcefab7eb94/opensearch-distributed-tracing/values-opensearch-dataprepper.yaml --version 0.3.1 --namespace opensearch-stack
I0730 13:26:13.197723   68838 warnings.go:110] "Warning: spec.template.spec.containers[0].ports[5]: duplicate port definition with spec.template.spec.containers[0].ports[4]"
NAME: opensearch-data-prepper
LAST DEPLOYED: Wed Jul 30 13:26:12 2025
NAMESPACE: opensearch-stack
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace opensearch-stack -l "app.kubernetes.io/name=data-prepper,app.kubernetes.io/instance=opensearch-data-prepper" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace opensearch-stack $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace opensearch-stack port-forward $POD_NAME 8080:$CONTAINER_PORT
Inline pipeline configuration is enabled. Please refer to the values.yaml file for the configuration.

Create a Headless service for the DataPrepper peers to connect with each other.

apiVersion: v1
kind: Service
metadata:
  name: data-prepper-headless
  namespace: opensearch-stack
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/instance: opensearch-data-prepper
    app.kubernetes.io/name: data-prepper
  ports:
    - name: peer-forwarder
      port: 4994
      targetPort: 4994
      protocol: TCP

Configure Opensearch DataPrepper to run in peer forwarder configuration.

helm upgrade opensearch-data-prepper opensearch/data-prepper -f https://raw.githubusercontent.com/NomadXD/samples/ca9a5e0c17cd479e40064fb490a22292c022db64/opensearch-distributed-tracing/values-opensearch-dataprepper.yaml --namespace opensearch-stack
I0730 13:37:09.117548   73798 warnings.go:110] "Warning: spec.template.spec.containers[0].ports[5]: duplicate port definition with spec.template.spec.containers[0].ports[4]"
Release "opensearch-data-prepper" has been upgraded. Happy Helming!
NAME: opensearch-data-prepper
LAST DEPLOYED: Wed Jul 30 13:37:08 2025
NAMESPACE: opensearch-stack
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace opensearch-stack -l "app.kubernetes.io/name=data-prepper,app.kubernetes.io/instance=opensearch-data-prepper" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace opensearch-stack $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace opensearch-stack port-forward $POD_NAME 8080:$CONTAINER_PORT
Inline pipeline configuration is enabled. Please refer to the values.yaml file for the configuration.

Analyse trace data using Opensearch Dashboard

Now that you have a complete end-to-end distributed tracing system running, you can instrument your applications with OpenTelemetry SDKs and configure them to publish traces to the Opensearch Data Prepper OTLP trace endpoint. Once traces are flowing, you can use Opensearch Dashboards Trace Analytics to visualize, search, and analyze your distributed traces in real time.

Figure 5: Opensearch Dashboard Trace Analytics

Helm Charts values.yaml: Opensearch Distributed Tracing Helm Charts

Multi cluster networking with Cilium Cluster Mesh

Lahiru De Silva — Fri, 17 May 2024 00:00:00 GMT

As organizations increasingly adopt distributed architectures and scale their Kubernetes deployments, the need for robust networking and security solutions that can seamlessly operate across multiple clusters becomes paramount. In this blog, we will go through how you can use Cilium Cluster Mesh to effectively manage a fleet of Kubernetes clusters spanning across availability zones or regions, thereby achieving unparalleled levels of high availability and fault tolerance across your infrastructure.

🐝 Cilium Cluster Mesh in a nutshell 🐝

Cilium Cluster Mesh allows you to connect the networks of multiple clusters in such as way that pods in each cluster can discover and access services in all other clusters of the mesh, provided all the clusters run Cilium as their CNI. This allows effectively joining multiple clusters into a large unified network, regardless of the Kubernetes distribution or location each of them is running.

This is done by deploying an additional API server called clustermesh-apiserver to synchronize the shared state among the kubernetes clusters. Each kubernetes cluster holds its state in the etcd and the kubernetes clusters in the cluster mesh can access other cluster’s state via the clustermesh-apiserver. For this each cluster should expose its clustermesh-apiserver as a Load balancer service. Cilium agents running in the kubernetes clusters connect to the clustermesh-apiserver of other clusters, watch for changes and replicate the multi-cluster relevant state into their own cluster.

Cilium Cluster Mesh provides the following features as stated in the Cilium documentation.

Pod IP routing across multiple Kubernetes clusters at native performance via tunneling or direct-routing without requiring any gateways or proxies.
Transparent service discovery with standard Kubernetes services and coredns/kube-dns.
Network policy enforcement spanning multiple clusters. Policies can be specified as Kubernetes NetworkPolicy resource or the extended CiliumNetworkPolicy CRD.
Transparent encryption for all communication between nodes in the local cluster as well as across cluster boundaries.

Time to get our hands dirty…. 👨‍🔧

In this blog, we are going to deploy the following setup in your laptop using kind (Kubernetes IN Docker) and demonstrate the multi cluster load balancing and fault tolerance capabilities supported by Cilium Cluster Mesh.

The setup consists of 2 kubernetes clusters which are named as US and EU for ease of relating to real world scenario of multi region clusters. We are going to install Cilium in both of the clusters and then enable Cilium Cluster Mesh on both. Then we are going to deploy a hello world application in both of the clusters along with a busybox pod to access the application within the cluster. Then we are going to make the hello world service accessible across clusters enabling load balancing across clusters. Finally we are going to use service affinity rules in Cilium to demonstrate the fault tolerant capabilities across clusters.

Multi cluster networking

1. Create two Kubernetes clusters using kind.

Note

We will be using 3 terminals for the hands on lab. Terminal 1 has access to the US cluster and the Terminal 2 has access to the EU cluster. Terminal 3 has access to the both clusters.

Create a new directory to host the kubernetes manifests. Let’s name it as cilium-cluster-mesh.
Open a terminal session (Terminal 1) and set the environment variable KUBECONFIG to export KUBECONFIG=./kubeconfig-us.yaml
Create a file with the following content and save it as kind-us-cluster.yaml

---
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
networking:
  disableDefaultCNI: true
  podSubnet: 10.1.0.0/16
  serviceSubnet: 172.20.1.0/24
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 32042
        hostPort: 32042
      - containerPort: 31234
        hostPort: 31234
      - containerPort: 31235
        hostPort: 31235
  - role: worker
  - role: worker

Execute the following command to create the US cluster. If the cluster creation is successful, following logs can be seen.

# US cluster (Terminal 1)
➜ kind create cluster --name us --config kind-us-cluster.yaml

Creating cluster "us" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-us"
You can now use your cluster with:

kubectl cluster-info --context kind-us

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

Next open a new terminal (Terminal 2) and set the environment variable KUBECONFIG to export KUBECONFIG=./kubeconfig-eu.yaml
Create a file with the following content and save it as kind-eu-cluster.yaml

---
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
networking:
  disableDefaultCNI: true
  podSubnet: 10.2.0.0/16
  serviceSubnet: 172.20.2.0/24
nodes:
  - role: control-plane
  - role: worker
  - role: worker

Execute the following command to create the EU cluster. If the cluster creation is successful, following logs can be seen.

# EU cluster (Terminal 2)
➜ kind create cluster --name eu --config kind-eu-cluster.yaml

Creating cluster "eu" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-eu"
You can now use your cluster with:

kubectl cluster-info --context kind-eu

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

2. Install Cilium CNI in the two clusters.

In the terminal 1 (US), execute the following command to install the Cilium CNI in the US cluster.

# US cluster (Terminal 1)
cilium install \
  --set cluster.name=us \
  --set cluster.id=1 \
  --set ipam.mode=kubernetes

🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.20.0"
ℹ️  Using Cilium version 1.15.4
ℹ️  Using cluster name "eu"
🔮 Auto-detected kube-proxy has been installed

Execute the command cilium status to check the status of the Cilium installation.

# US cluster (Terminal 1)
➜ cilium status   
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Containers:            cilium             Running: 3
                       cilium-operator    Running: 1
Cluster Pods:          3/3 managed by Cilium
Helm chart version:    
Image versions         cilium             quay.io/cilium/cilium:v1.15.4@sha256:b760a4831f5aab71c711f7537a107b751d0d0ce90dd32d8b358df3c5da385426: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.15.4@sha256:404890a83cca3f28829eb7e54c1564bb6904708cdb7be04ebe69c2b60f164e9a: 1

In the terminal 2 (EU), execute the following command to install the Cilium CNI in the EU cluster.

# EU cluster (Terminal 2)
cilium install \
  --set cluster.name=eu \
  --set cluster.id=2 \
  --set ipam.mode=kubernetes

🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.20.0"
ℹ️  Using Cilium version 1.15.4
ℹ️  Using cluster name "eu"
🔮 Auto-detected kube-proxy has been installed

Now if you followed the instructions upto this point correctly, you would have 2 kubernetes clusters up and running in your laptop with Cilium installed as the CNI.

3. Enable Cilium Cluster Mesh in both clusters

Next we are going to enable Cilium Cluster Mesh in both US and EU clusters. This will deploy an additional deployment for the clustermesh-apiserver in both of the clusters.

Cilium agents running in the EU cluster will use this clustermesh-apiserver in the US cluster to replicate the states of US cluster to its own EU cluster and vice versa. Therefore this clustermesh-apiserver should be exposed as a Load balancer service for accessing outside the cluster. But for demonstrating purposes, we are going to expose the clustermesh-apiserver as a NodePort service as we don’t have dynamic load balancers available. This is not recommended for production use cases as the service becomes unavailable when the node goes down.

To enable clustermesh-apiserver run the following command in both Terminal 1 (US) and Terminal 2 (EU).

# US cluster (Terminal 1) & EU cluster (Terminal 2)
cilium clustermesh enable --service-type NodePort

If you execute the cilium status command now, you’ll notice that the ClusterMesh status is changed from disabled to OK.

# US cluster (Terminal 1) & EU cluster (Terminal 2)
➜ cilium status                                    
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        OK

DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
Deployment             clustermesh-apiserver    Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-relay             Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-ui                Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium-operator          Running: 1
                       cilium                   Running: 3
                       hubble-relay             Running: 1
                       hubble-ui                Running: 1
                       clustermesh-apiserver    Running: 1
Cluster Pods:          6/6 managed by Cilium
Helm chart version:    
Image versions         cilium                   quay.io/cilium/cilium:v1.15.4@sha256:b760a4831f5aab71c711f7537a107b751d0d0ce90dd32d8b358df3c5da385426: 3
                       hubble-relay             quay.io/cilium/hubble-relay:v1.15.4@sha256:03ad857feaf52f1b4774c29614f42a50b370680eb7d0bfbc1ae065df84b1070a: 1
                       hubble-ui                quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
                       hubble-ui                quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1
                       clustermesh-apiserver    quay.io/cilium/clustermesh-apiserver:v1.15.4@sha256:3fadf85d2aa0ecec09152e7e2d57648bda7e35bdc161b25ab54066dd4c3b299c: 2
                       cilium-operator          quay.io/cilium/operator-generic:v1.15.4@sha256:404890a83cca3f28829eb7e54c1564bb6904708cdb7be04ebe69c2b60f164e9a: 1

Now if you execute kubectl get pods -A, you’ll notice a new pod is created for the clustermesh-apiserver.

# US cluster (Terminal 1) & EU cluster (Terminal 2)
➜ kubectl get pods -A

NAMESPACE            NAME                                         READY   STATUS      RESTARTS   AGE
kube-system          cilium-5glrm                                 1/1     Running     0          4m8s
kube-system          cilium-operator-5d8fb68868-sn9k5             1/1     Running     0          4m8s
kube-system          cilium-rnpmp                                 1/1     Running     0          4m8s
kube-system          cilium-tfjwr                                 1/1     Running     0          4m8s
kube-system          clustermesh-apiserver-68b6c7cc6d-224tw       0/2     Init:0/1    0          14s
kube-system          clustermesh-apiserver-generate-certs-hmpdc   0/1     Completed   0          14s
kube-system          coredns-5d78c9869d-d8drc                     1/1     Running     0          6m36s
kube-system          coredns-5d78c9869d-f85p5                     1/1     Running     0          6m36s
kube-system          etcd-us-control-plane                        1/1     Running     0          6m50s
kube-system          hubble-relay-64d7c9bb65-zwcfk                1/1     Running     0          79s
kube-system          hubble-ui-7b4457996f-pzhtf                   2/2     Running     0          79s
kube-system          kube-apiserver-us-control-plane              1/1     Running     0          6m52s
kube-system          kube-controller-manager-us-control-plane     1/1     Running     0          6m49s
kube-system          kube-proxy-nfmw2                             1/1     Running     0          6m30s
kube-system          kube-proxy-pmgnx                             1/1     Running     0          6m36s
kube-system          kube-proxy-twjbc                             1/1     Running     0          6m29s
kube-system          kube-scheduler-us-control-plane              1/1     Running     0          6m49s
local-path-storage   local-path-provisioner-6bc4bddd6b-qxk2z      1/1     Running     0          6m36s

If you followed everything upto to this point, you would have 2 kubernetes clusters with Cilium installed as the CNI and clustermesh enabled in both the clusters. Next we need to connect the two clusters.

4. Connect the two clusters

To connect the two clusters, we need to use the global terminal with access to both the clusters. (Terminal 3)

Open the terminal 3 and execute the following command to merge the two contexts of the US and EU clusters.

# Global terminal (Terminal 3)
KUBECONFIG=./kubeconfig-us.yaml:./kubeconfig-eu.yaml 
kubectl config view --flatten > merged-kubeconfig.yaml

Next set the KUBECONFIG to the newly created merged kubeconfig.

# Global terminal (Terminal 3)
export KUBECONFIG=./merged-kubeconfig.yaml

Finally execute the following command to connect the two clusters.

# Global terminal (Terminal 3)
➜ cilium clustermesh connect \                     
  --context kind-us \
  --destination-context kind-eu
✨ Extracting access information of cluster eu...
🔑 Extracting secrets from cluster eu...
⚠️  Service type NodePort detected! Service may fail when nodes are removed from the cluster!
ℹ️  Found ClusterMesh service IPs: [172.18.0.7]
✨ Extracting access information of cluster us...
🔑 Extracting secrets from cluster us...
⚠️  Service type NodePort detected! Service may fail when nodes are removed from the cluster!
ℹ️  Found ClusterMesh service IPs: [172.18.0.4]
⚠️ Cilium CA certificates do not match between clusters. Multicluster features will be limited!
ℹ️ Configuring Cilium in cluster 'kind-us' to connect to cluster 'kind-eu'
ℹ️ Configuring Cilium in cluster 'kind-eu' to connect to cluster 'kind-us'
✅ Connected cluster kind-us and kind-eu!

To check the status of the Cilium Cluster Mesh execute the command cilium clustermesh status. You would see an output like the following if everything is working as expected.

# US cluster (Terminal 1) & EU cluster (Terminal 2)
➜ cilium clustermesh status
⚠️  Service type NodePort detected! Service may fail when nodes are removed from the cluster!
✅ Service "clustermesh-apiserver" of type "NodePort" found
✅ Cluster access information is available:
  - 172.18.0.4:32379
✅ Deployment clustermesh-apiserver is ready
✅ All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
🔌 Cluster Connections:
  - eu: 3/3 configured, 3/3 connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

5. Deploy the sample hello world application

Deploy the sample hello world application by executing the following commands in each of Terminal 1 (US) and Terminal 2(EU).

# US cluster (Terminal 1)
kubectl apply -f deployment-us.yaml

# EU cluster (Terminal 2)
kubectl apply -f deployment-eu.yaml

Check whether sample hello world is working as expected by executing the following command in both US and EU clusters to invoke the hello world service from the busybox pod.

# US cluster (Terminal 1)
➜ kubectl exec -it busybox-deployment-b7bc87c95-t4z2c -- /bin/sh -c 'for i in $(seq 1 10); do wget -qO- hello-world:80; echo ""; done'

{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}

# EU cluster (Terminal 2)
➜ kubectl exec -it busybox-deployment-b7bc87c95-xhc6l -- /bin/sh -c 'for i in $(seq 1 10); do wget -qO- hello-world:80; echo ""; done'

{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}

Did you notice that all the responses in the US are returned from the hello world service in US ? And all the responses in the EU are returned from the hello world service in EU ? Eventhough we have enabled Cilium Cluster Mesh in both clusters and connected the two clusters, we haven’t enabled the hello world service as a global service. So the requests are load balanced among the pods within the same cluster.

6. Enable hello world service as a global service to enable cross cluster service discovery and load balancing.

To enable hello world service as a global service, execute the following command in both US and EU clusters.

Cross cluster load balancing

# US cluster (Terminal 1) & EU cluster (Terminal 2)
kubectl annotate service hello-world service.cilium.io/global="true"

Now invoke the hello world service again and notice the responses received.

# US cluster (Terminal 1)
➜ kubectl exec -it busybox-deployment-b7bc87c95-t4z2c -- /bin/sh -c 'for i in $(seq 1 10); do wget -qO- hello-world:80; echo ""; done'

{"message": "Hello world from US!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from US!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from US!"}
{"message": "Hello world from Europe!"}

# EU cluster (Terminal 2)
➜ kubectl exec -it busybox-deployment-b7bc87c95-xhc6l -- /bin/sh -c 'for i in $(seq 1 10); do wget -qO- hello-world:80; echo ""; done'

{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from US!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from US!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from US!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}

Now you can see that the requests from the US cluster are load balanced to US and EU and vice versa. Tadaaa!!! We have successfully enabled cross cluster service discovery and load balacing using Cilium Cluster Mesh.

Now in an ideal scenario, we want to load balance globally for fault tolerance only if the local services are not available. Load balancing cross cluster is not ideal always as the latency may get higher comapared to local services. To achive such behaviour, we can use the service affinity rules in Cilium.

7. Specify hello world service affinity as local for fault tolerant cross cluster load balancing.

Execute the following command to specify the hello world service affinity as local in the US cluster. This will load balance requests to the hello world service among the pods in the US cluster only and route requests to the other cluster (EU) only if the local service is not available.

# US cluster (Terminal 1)
kubectl annotate service hello-world service.cilium.io/affinity="local"

Now if you invoke the hello world service from the US cluster, you will receive responses only from the hello world service in the US cluster.

# US cluster (Terminal 1)
➜ kubectl exec -it busybox-deployment-b7bc87c95-t4z2c -- /bin/sh -c 'for i in $(seq 1 10); do wget -qO- hello-world:80; echo ""; done'

{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}
{"message": "Hello world from US!"}

To demonstrate the fault tolerant behaviour, scale down the hello world service in US cluster and invoke the hello world service from the US cluster.

# US cluster (Terminal 1)
kubectl scale deployment hello-world --replicas 0

# US cluster (Terminal 1)
➜ kubectl exec -it busybox-deployment-b7bc87c95-t4z2c -- /bin/sh -c 'for i in $(seq 1 10); do wget -qO- hello-world:80; echo ""; done'

{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}
{"message": "Hello world from Europe!"}

Cross cluster fault tolerance

Eventhough the hello world service is down in the US, the requests get routed to the hello world service in EU making it fault tolerant across clusters.

So we have successfully demonstrated the multi cluster service discovery, load balancing and fault tolerance that can be achieved with Cilium Cluster Mesh using two kubernetes clusters running locally in your laptop.

Kubernetes manifests used for the hands on lab: cilium-cluster-mesh

References

Deep dive into Kubernetes custom controllers

Lahiru De Silva — Sat, 30 Sep 2023 00:00:00 GMT

Kubernetes offers a rich ecosystem of extension points that allow developers to tailor and extend its functionality. Among these ability to implement Custom Controllers emerge as a powerful mechanism for extending Kubernetes with your own “Kubernetes logic”. While there are frameworks and tools like Kubebuilder and OperatorSDK that abstracts away the complexity of implementing custom controllers, understanding how custom controllers operate at a fundamental level is crucial for those who wish to harness the full potential of Kubernetes. In this article, we will go through how custom controllers work behind the scenes and how you can leverage the basic building blocks of custom controllers for your use cases.

Kubernetes controller overview

Controllers serve as the intelligent core or the brain behind Kubernetes by orchestrating and coordinating the lifecycle of resources to ensure the desired state of applications. When resources like Pods, Deployments, Replicasets are created, the relevant controllers respond by performing the required actions to match the current state of the resource with the desired state of the resource.

For instance, consider the scenario of creating a ReplicaSet using kubectl. The ReplicaSet controller promptly detects this action and delegates the task to the kube-scheduler, which schedules the desired number of pods. Once these pods are up and running, the ReplicaSet controller continuously monitors their status through the Kubernetes API server. If, for any reason, a pod terminates unexpectedly, the ReplicaSet controller initiates a request to the kube-scheduler to replace the terminated pod, ensuring that the desired replica count is maintained.

Similarly, Kubernetes ships with a set of built-in controllers packaged within the kube-controller-manager. Each of these controllers operates by running a reconcile loop, which continuously ensures the alignment of the current state of the watched resource with its desired state.

for {
  desiredState := getDesiredState()
  currentState := getCurrentState()
  makeChanges(desiredState, currentState)
}

Kubernetes custom controllers

It’s essential to be familiar with the following terms to understand the concept of custom controllers in Kubernetes.

Custom Resources (CRs) enable you to extend the Kubernetes API and create domain specific objects that are managed by Kubernetes itself like any other built-in objects like Pods, Deployments etc.
Custom Resource Definition (CRDs) are simply a Kubenetes resource type that is used to register your Custom Resources (CRs) with the Kubernetes API server.
Custom controllers extend the functionality of Kubernetes by implementing custom logic for managing Custom Resources (CRs).

So if you want to extend Kubernetes with your own custom logic,

Define a Custom Resource Definition (CRD) to let Kubernetes know about your Custom Resources (CRs).
Run your own reconcile loop for the custom resource type by implementing a custom controller and deploying it in the Kubernetes cluster.
Create Custom Resources (CRs) via a Kubernetes client like kubectl.

Following diagram shows how custom controllers fit in with the Kubernetes control plane components and their interactions.

Kubernetes Custom Controllers

Custom controllers can be deployed as typical Kubernetes deployments.

Now that you have some basic understading about how the Kubernetes custom controllers fit in with other control plane components, let’s dig deeper into the internals of a custom controller.

Building blocks of a Kubernetes custom controller

The internal components of a custom controller can be categorized into two distinct groups based on their functionalities and implementation:

1. Kubernetes client-go components (Shared Informer)

These components are initialized and provided out of the box by the Kubernetes Go client.
The developer implementing the custom controller has to initialize these components via the Shared Informer interface.
Responsible for watching the Kubernetes API server for resource changes and notifying the event processing components of the custom controller.

Informer vs Shared Informer

Evaluating the current state against the desired state requires the controller to interact with the Kubernetes API server to fetch object details. As the number of Kubernetes resources created grows, the frequency of API server calls made by controllers will significantly rise, potentially increasing exponentially. So to reduce the load on the Kubernetes API server, informers or shared informers are introduced.

Informers address this issue by retrieving object data and storing it in the local cache of the controller. The Informer then watches for any create, modify, and delete events that occur afterward. In scenarios where multiple controllers are monitoring a single object, each controller will independently update its local cache. This can result in excessive memory usage and the creation of multiple, potentially inconsistent cache data stores containing object data. This is where the Sharedinformer comes in.

The SharedInformer, as its name implies, is utilized to create a shared cache data store accessible to all controllers, thus resolving the issue of multiple controllers monitoring and updating a single resource or object within the cluster.

Shared informers consist of 4 main components.

2. Custom controller components

These components are implemented by the developer who’s writting the custom controller.
Responsible for processing the events passed from the Shared Informer.
Contains the actual reconcilation logic of the custom controller.

Consist of 3 main components.

Resource Event handlers.
Work Queue.
Event processing function (Reconcile)

Now let’s see how all these components work together and what tasks each of these components perform to achieve the desired functionality of the custom controller.

Internal functionality of a Kubernetes custom controller

Following diagram shows the internal interactions between the components of a custom controller and the data flow between the components.

Custom controllers internals

1. Reflector watches the Kubernetes API server using ListAndWatch mechanism.

Reflectors continuously watch the Kubernetes API server for changes in specific resources like Pods, Deployments or Custom Resources and enqueues the Delta FIFO store with events like Add, Update, Delete related to the resources being watched. To perform this, reflectors use the ListAndWatch mechanism provided by the Kubernetes API server.

When a reflector is initialized, it starts by listing all the resources it’s interested in. This initial listing provides a snapshot of the current state of those resources. After the initial listing, the reflector switches to a watch mode where it establishes a long-lived connection with the API server. The API server then notifies the reflector of any changes to the resources in real-time.

To examine the List and Watch capability of the Kubernetes API server, you can follow the following steps.

Execute kubectl proxy to call the Kubernetes API server without certificates or tokens.
Execute curl -s http://localhost:8001/api/v1/namespaces/default/pods to list all the pods in default namespace.

{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "582837"
  },
  "items": []
}

Capture a resource version and execute curl -s "http://localhost:8001/api/v1/pods?watch=true&resourceVersion=582837" to watch for pods from a specificed resource version onwards. This will establish a persistence HTTP connection with the Kubernetes API Server and the Kubernetes API server will send the resource updates as chunks.

curl -s -v "http://localhost:8001/api/v1/pods?watch=true&resourceVersion=587961" -v
*   Trying 127.0.0.1:8001...
* Connected to localhost (127.0.0.1) port 8001 (#0)
> GET /api/v1/pods?watch=true&resourceVersion=587961 HTTP/1.1
> Host: localhost:8001
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< Audit-Id: 9f25e00a-1d6b-47d6-b818-be84ce7458eb
< Cache-Control: no-cache, private
< Content-Type: application/json
< Date: Sat, 30 Sep 2023 08:33:01 GMT
< X-Kubernetes-Pf-Flowschema-Uid: 2eeceae7-9692-4905-85ba-1ad01882320d
< X-Kubernetes-Pf-Prioritylevel-Uid: 30e13e51-7784-4c62-acb5-b944f96ae4da
< Transfer-Encoding: chunked

Execute kubectl run nginx --image=nginx and notice the CREATE event received through the other terminal in watch mode.

The received events via the watch are called deltas and the deltas will be enqueued into the Delta FIFO Queue, which is an incremental first-in first-out queue.

2. Delta FIFO queue stores the resource changes as deltas.

This is a special store that maintains deltas, which is a structure containing the changed object and the type of change.

type Delta struct {
    Type   DeltaType
    Object interface{}
}

The Delta objects in the queue will be eventually popped out and processed by the cache controller.

3. Cache controller popping deltas and processing deltas

When the cache controller is initiated, it starts 2 go routines.

Creates a Reflector and runs the reflector in a separate go routine to watch resources in Kubernetes API server.
Blocks its main go routine on the processLoop() method which continously pop deltas from the Delta FIFO queue and process deltas.

processLoop() method will invoke processDeltas() method which will perform 2 main functions.

Update the indexer according to the detected change retrieved from the delta FIFO queue.
Trigger the pre-registered Resource Event Handlers according to the event type. (These are the resource event handlers registered by the developer when creating a SharedInformer. In the resource event handlers, usually only some simple filtering is done, and then the event is added into the Work Queue)

sharedInformer.AddEventHandler(
        &cache.ResourceEventHandlerFuncs{
            AddFunc:    onAdd,
            DeleteFunc: onDelete,
            UpdateFunc: onUpdate,
        })

onAdd, onDelete and onUpdate resource event handlers are functions implemented by the developer.

4. Resource event handlers enqueue the object key to the work queue

Resource event handlers serve as callback functions that the Shared Informer calls to deliver objects to the controller’s event processing functions. The common practice for writing these functions involves retrieving the key associated with the dispatched object (namespace/name) and enqueueing that key in the work queue for subsequent processing.

The work queue decouples the event arrival from processing, allowing the controller to process events in a controlled and predictable manner, preventing event processing bottlenecks. For example, Shared Informers may operate at different speeds than controller event processing functions, and work queues help prevent the Shared Informer from overloading a slower controller event processor.

5. Dequeueing events from the work queue and reconciliation

Event processing functions are the functions that you implement in your code to process items from the work queue. There can be one or more other functions that do the actual processing.

These functions will typically use the Indexer to retrieve the object corresponding to the key and perform the actual event processing logic you have implemented.

Multiple go routines can be spawned to process event updates and each of these worker go routines will call processNextWorkItem which will dequeue object keys from the work queue and call syncHandler method to perform reconcilation logic.

Depending on the reconcilation logic, you may need to call the Kubernetes API server or any other external services. And mostly importantly, once the reconciliation logic is performed, the status of the resource should be updated and typically the performed actions is added into the Events array.

If you are using a framework like Kubebuilder or OperatorSDK, this is the point where you write reconciliation logic inside the Reconcile() method of the generated controller code.

Summary

In summary, the following sequence of events take place inside the Kubernetes custom controller.

Reflector watches Kubernetes API server and add resource changes (deltas) to Delta FIFO Queue.
Cache controller pops these changes out from the Delta FIFO Queue and update the indexer.
Cache controller trigger the pre-registered resource event handlers.
Pre-registered resource event handlers filters the events and add the object keys to the work queue.
Custom controller event processing functions pops out the object keys, retrieve the object using the keys from the indexer and perform reconcilation logic.
After performing reconcilation logic, custom controller updates the status of the object by calling the Kubernetes API server.

Now to get started with implementing a custom controller, you can use this sample controller from the Kubernetes community as the starting point.

How to implement a K8s operator like a Ninja

Lahiru De Silva — Fri, 14 Jul 2023 00:00:00 GMT

Kubernetes has emerged as the widely accepted standard for deploying and running modern cloud-native applications. With its declarative approach using YAML, it offers a straightforward and intuitive method to define the desired infrastructure state for application deployments. However, managing complex application deployments on Kubernetes requires additional automation and control beyond what Kubernetes provides out of the box. This is where Kubernetes operators come in.

Operators extend the capabilities of Kubernetes by encapsulating domain-specific knowledge and best practices into custom controllers. By leveraging operators, organizations can automate the management of complex application lifecycles, enable self-healing, autoscaling, and perform advanced operations such as backups, upgrades, and rolling deployments. Kubernetes operators help streamline and simplify the deployment and management of applications on Kubernetes, enhancing scalability, reliability, and efficiency.

There are several approaches to building a Kubernetes operator. One option is to develop one from scratch using the Kubernetes Go Client, but this can be a challenging and time-consuming task due to the steep learning curve involved. Alternatively, there are tools available that provide boilerplate code and streamline the process of creating operators. Two popular choices are OperatorSDK and Kubebuilder. In this article, we will primarily focus on utilizing Kubebuilder to create an operator, leveraging its features to expedite the development process.

Scaffolding the project with Kubebuilder

Prerequisites

go version v1.20.0+
kubectl version v1.11.3+.
Access to a Kubernetes v1.11.3+ cluster.

Kubebuilder offers a command-line interface (CLI) that facilitates the creation and management of operator projects. To initiate a new project, all you need to do is execute the following command:

kubebuilder init --domain nomadxd.io --plugins=go/v4-alpha --repo=github.com/NomadXD/samples/k8s-operator-kube-builder

Once executed, Kubebuilder will generate a directory structure, enabling you to immediately begin developing the operator.

Creating the Custom Resource and controller

To enable the desired functionality of an operator, Custom Resource Definitions (CRDs) are utilized in conjunction with a controller. By defining CRDs, you can extend the Kubernetes API and introduce custom resources tailored to your specific application or workload. These CRDs serve as the basis for creating and managing instances of your custom resources.

To ensure the desired state of the custom resources is maintained, a controller is implemented. The controller continuously monitors the state of the custom resources and takes actions to reconcile any differences between the desired state and the actual state of the cluster. This may involve creating, updating, or deleting Kubernetes resources based on changes in the custom resources.

Use the following command to create a new Go type which corresponds to a CRD in K8s. It will prompt for 2 options to create resource and controller and input y to both of the cases.

kubebuilder create api --group podrunner --version v1alpha1 --kind PodRunner
Create Resource [y/n]
y
Create Controller [y/n]
y

Exploring the generated code

Now all the boilerplate required to implement the operator is generated. The core components of an operator are found in the following locations:

/api/v1alpha1/podrunner_types.go

Contains the Custom Resource type definition in Go. Spec and Status types should be extended by adding the required feilds. Spec means the desired state and the Status means the current state. A controller basically runs a reconcile loop to match the current state with the desired state.

/*
Copyright 2023.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1alpha1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// PodRunnerSpec defines the desired state of PodRunner
type PodRunnerSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // Foo is an example field of PodRunner. Edit podrunner_types.go to remove/update
    Foo string `json:"foo,omitempty"`
}

// PodRunnerStatus defines the observed state of PodRunner
type PodRunnerStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// PodRunner is the Schema for the podrunners API
type PodRunner struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   PodRunnerSpec   `json:"spec,omitempty"`
    Status PodRunnerStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// PodRunnerList contains a list of PodRunner
type PodRunnerList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []PodRunner `json:"items"`
}

func init() {
    SchemeBuilder.Register(&PodRunner{}, &PodRunnerList{})
}

/controllers/podrunner_controller.go

Extend the Reconcile() method to implement reconcile logic to match the current state with the desired state.

/*
Copyright 2023.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
    "context"

    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    podrunnerv1alpha1 "github.com/NomadXD/samples/k8s-operator-kube-builder/api/v1alpha1"
)

// PodRunnerReconciler reconciles a PodRunner object
type PodRunnerReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=podrunner.nomadxd.io,resources=podrunners,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=podrunner.nomadxd.io,resources=podrunners/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=podrunner.nomadxd.io,resources=podrunners/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the PodRunner object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.13.0/pkg/reconcile
func (r *PodRunnerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    // TODO(user): your logic here

    return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *PodRunnerReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&podrunnerv1alpha1.PodRunner{}).
        Complete(r)
}

Defining the desired state

For this tutorial, I will implement an operator which can run a Pod based on an image name we provide with the Custom Resource. To implement that, I have extended the PodRunnerSpec as follows.

// PodRunnerSpec defines the desired state of PodRunner
type PodRunnerSpec struct {
    // PodName is the name of the pod.
    PodName string `json:"podName,omitempty"`

    // ImageName is the name of the image used to run the pod.
    ImageName string `json:"imageName,omitempty"`

        // Namespace where the pod is scheduled to run.
    Namespace string `json:"namespace,omitempty"`
}

Generating and installing the CRDs

After that execute the following commands.

make generate - This generates the required deep copy methods
make manifests - To generate the Kubernetes CRDs from the defined Go types. Generated CRDs are located in config/crd/bases/
make install - To apply the CRDs to the K8s API server

Now the Kubernetes API server knows about the resources of type PodRunner.

Implementing the reconciliation logic

Now to implement the logic for scheduling the pod on the given namespace using the provided image name, extend the Reconcile() method in the controller boilerplate as follows.

func (r *PodRunnerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    podRunner := &podrunnerv1alpha1.PodRunner{}
    err := r.Get(ctx, req.NamespacedName, podRunner)
    if err != nil {
        // Error reading the PodRunner instance, requeue the request
        logger.Error(err, "Failed to get PodRunner")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Create a Pod based on the PodRunner specification
    podRunnerPod := &corev1.Pod{
        ObjectMeta: metav1.ObjectMeta{
            Name:      podRunner.Spec.PodName,
            Namespace: podRunner.Spec.Namespace,
        },
        Spec: corev1.PodSpec{
            Containers: []corev1.Container{
                {
                    Name:            podRunner.Spec.PodName,
                    Image:           podRunner.Spec.ImageName,
                    ImagePullPolicy: corev1.PullAlways,
                },
            },
        },
    }

    err = ctrl.SetControllerReference(podRunner, podRunnerPod, r.Scheme)
    if err != nil {
        logger.Error(err, "Failed to set controller reference for Nginx Pod")
        return ctrl.Result{}, err
    }

    // Check if the Pod already exists
    foundPod := &corev1.Pod{}
    err = r.Get(ctx, types.NamespacedName{Name: podRunner.Spec.PodName, Namespace: podRunner.Spec.Namespace}, foundPod)
    if err != nil && errors.IsNotFound(err) {
        // Create the Pod
        err = r.Create(ctx, podRunnerPod)
        if err != nil {
            logger.Error(err, "Failed to create Pod")
            return ctrl.Result{}, err
        }
        logger.Info("Pod created")
        return ctrl.Result{}, nil
    } else if err != nil {
        logger.Error(err, "Failed to get Pod")
        return ctrl.Result{}, err
    }

    // Pod already exists, do nothing
    logger.Info("Pod already exists")
    return ctrl.Result{}, nil
}

Running and testing the operator

Now execute the command make run to run the operator locally. If you have followed everything correctly upto now, you should have a Kubernetes operator up and running. Now to run a pod, create a Custom Resource of type PodRunner and do a kubectl apply -f.

kubectl apply -f - <
apiVersion: podrunner.nomadxd.io/v1alpha1
kind: PodRunner
metadata:
  name: podrunner-sample
spec:
  podName: test-pod
  imageName: nginx:latest
  namespace: default
EOF

If everything is in place, you will see podrunner.podrunner.nomadxd.io/podrunner-sample created. Next execute kubectl get pods to see whether the pod is scheduled by the controller we implemented.

kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          6s

If you execute kubectl describe pod test-pod | grep -i "Controlled By", you will get Controlled By: PodRunner/podrunner-sample. So it proves that the pod is scheduled by the controller we implemented.

Conclusion

By leveraging Kubebuilder’s user-friendly CLI and code generation capabilities, developers can rapidly bring their Kubernetes operators to life within a matter of minutes. With just a few simple commands and a bit of code, complex processes and application management can be streamlined and automated without the need for complex time consuming implementation.

Complete source code for the above setup with Kubebuilder generated files: k8s-operator-kube-builder

How I passed my CKA exam on the first attempt

Lahiru De Silva — Thu, 13 Jul 2023 00:00:00 GMT

Passing the Certified Kubernetes Administrator (CKA) exam was an exhilarating milestone in my career. In this article, I’ll take you through my journey of preparing for and cracking the CKA exam.

I passed the CKA exam in July, 2023 in my first attempt after practising for roughly 4 weeks. I had a good level of Kubernetes experience beforehand the exam practise from my work at the company. I worked on Kubernetes native API gateways for the past 1 year and had a solid understanding about the basic concepts in Kubernetes and how everything works. But there were some missing pieces of the puzzle and I decided to systematically learn the concepts from the beginning by following a course.

Exam preparation

Regardless of your prior experience working with Kubernetes, I would recommend to purchase a well structured course made in parallel with the CKA syallabus. There are couple of options out there and I chose to go with Mumshad Mannambeth’s CKA course on Udemy as it offers access to KodeCloud practise tests which will help you to practise in a real Kubernetes environment and get hands on experience. If you are well experienced in Kubernetes and feel the videos are too lengthty or time consuming, you can complete the practise tests and get an idea about your current state.

After completing the practise tests, you get 3 mock exams and you can try to complete the exams on the given time and check your grade. But these exams are just for you to practise your skills in a provided time limit and do not simulate the actual exam environment or can’t be considered as a measure to predict your score in the actual CKA exam.

Once you complete the mock exams in KodeCloud, you can try the Killercoda pratise exercises which are completly free. The playgrounds in Killercoda are much closer to the real environment you get in the real CKA exam and the difficulty of the questions are also more closer to the real CKA exam.

After completing the Killercoda practise scenarios, you can try the Killer.sh mock exams you get with the CKA certification. You get 2 free sessions when you purchase the CKA cerfification. These practise environmemt will be active for 36 hours but the timer will be disabled after the first 120 mins and you’ll be graded after that. Try to time for 90 mins and complete all the questions. But keep in mind that the questions on Killer.sh is way more harder than the actual exam and you will be completly fine even if you does not get a passing score. I got 87 out of 120 in my first try which is equivalent to 72.5%. Ideally use your first session before 1 week of the scheduled date of your CKA exam and use the second session like 2 days or 1 day before your scheduled exam.

Exam questions

You will get somewhere between 15 to 20 questions with different difficulty levels. I got 17 questions and out of them around 6 or 7 of them were difficult like the ones in Killer.sh. Another 6 of the questions were pretty easy and doable within 2 mins. The rest of the questions were like the ones you get in KodeCloud mock exams. I completed all the questions within 90 mins and spend the next 30 mins trying to do 3 of the flagged questions which I wasn’t sure what I have done wrong.

Exam tips and tricks

First do all the easy and moderate quetions and leave the ones that are difficult and requires more time. You can flag the ones that you have left and come back to them after you finish the easy ones.
Always keep in mind to switch the K8s context, otherwise even though your answers are correct, you won’t get marks
Use --dry-run=client -o yaml to get the K8s manifest as a YAML and edit/add the required info. You can use export do="--dry-run=client -o yaml" alias to save time
Always use imperative way and only use declarative way when it’s absoluelty required
Don’t mess up with the namespaces. Sometimes you run a basic command and wonder what’s wrong or why the resource is not there.
Get comfortable with Vim and know the basics like how to write, delete, copy etc.
Use K8s short names as much as possible
Delete pods without waiting by using --force. You can setup an alias for this also. export now="--force --grace-period 0"
Get familiar with the K8s documentation. Specially Concepts, Tasks and Reference sections. You can always search in the K8s docs.
Use kubectl explain to quickly find the feilds of a certain resource.
Remember which node and which context you are in before executing the commands.
Use kubectl -h option to get help for the commands and arguments.
Learn how to do troubleshooting by viewing pod, container and systemd logs. If the kubelet is running in systemd, you’ll need to use journalctl -u kubelet to view kubelet logs. Know how to restart the kubelet with systemctl.
Practise and be ready to complete trivial tasks like backing up and restoring etcd, creating Network policies, PV & PVC etc.
Learn how to view processes with ps and netstat. Also learn how to view network interface details with ip addr.

My CKA certificate

Building a Secure and Scalable API layer using Envoy Proxy

Lahiru De Silva — Wed, 12 Apr 2023 00:00:00 GMT

APIs (Application Programming Interfaces) have become an integral part of modern software development. They allow different applications to communicate and exchange data, enabling developers to create complex systems that integrate multiple services and technologies. APIs are used by companies of all sizes to power their digital products and services, from social media platforms to financial systems.

However, APIs can also be complex and challenging to manage, particularly when it comes to security and scalability. As APIs are exposed to the internet, they are vulnerable to attacks such as injection, denial of service, and man-in-the-middle attacks. Furthermore, as API usage grows, it becomes increasingly challenging to scale them to meet the demands of growing user bases and increasing traffic. This article aims to demonstrate the use of open source technologies, such as Envoy Proxy, to build a secure, scalable, and resilient Application Programming Interface (API) layer for organizations that depend heavily on APIs.

The following design will be implemented within the scope of this article to demonstrate the creation of a secure and scalable API layer.

Solid lines show the request/response flow and dotted lines show the telemetry data flow

First, let’s start with the external authentication service and the related proxy configuration, as it is the first service that Envoy Proxy calls when a request is received.

External authorization with external authorization gRPC service

Envoy Proxy supports external authorization through its built-in external authorization filter. The external authorization filter should be added to the filter chain with a reference to the relevant external gRPC/HTTP service. When a request is received, it is propagated through the filter chain, and upon reaching the external authorization filter, Envoy Proxy sends an authorization check request to the configured external authorization service. The authorization service receives the request, which includes information about the original request from the client/downstream, such as headers and body. After performing authorization, the external authorization service must return the response in the expected format for Envoy Proxy. Based on the response, Envoy Proxy will either pass the request to the next filter or return an unauthorized response to the client/downstream.

This approach allows for a flexible and modular authorization architecture, where the authorization logic can be separated from the core application logic, making it easier to manage and scale. Additionally, it enables integration with a wide range of authorization providers, such as OAuth, LDAP, and more.

Let’s go through the relevant envoy proxy configuration and the sample external authorization gRPC service.

External authorization filter configuration

Add external authorization filter to the filter chain with a reference to the cluster that represents the external authorization service.

http_filters:
  - name: envoy.filters.http.ext_authz
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
      grpc_service:
        envoy_grpc:
          cluster_name: ext_authz
        timeout: 2s
      transport_api_version: V3

External authorization service configuration

Add the external authorization cluster to the cluster configuration.

  clusters:
    - name: ext_authz
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      load_assignment:
        cluster_name: ext_authz
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: ext_authz
                      port_value: 50051

Sample gRPC authorization service written in go

package main

import (
    "context"
    "fmt"
    "log"
    "net"
    "net/http"
    "strings"
    "time"

    auth "github.com/envoyproxy/go-control-plane/envoy/service/auth/v3"
    "github.com/golang-jwt/jwt"
    "google.golang.org/genproto/googleapis/rpc/code"
    "google.golang.org/genproto/googleapis/rpc/status"
    "google.golang.org/grpc"
)

type AuthServer struct{}
 
var secretKey []byte = []byte("my_secret_key")

func (authServer *AuthServer) Check(ctx context.Context, request *auth.CheckRequest) (*auth.CheckResponse, error) {
    log.Printf("Auth server received auth request: %v", request.String())
  // Extract the authorization token from the request
    authHeader, ok := request.Attributes.Request.Http.Headers["authorization"]
    var tokenString string

    if ok {
        tokenString = strings.TrimPrefix(authHeader, "Bearer ")
    }
    fmt.Printf("Token: %s", tokenString)
    _, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
        if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
            return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
        }
        return secretKey, nil
    })
    fmt.Printf("Error: %v", err)
  // If valid token, returns Code_OK
    if err == nil {
        return &auth.CheckResponse{
            Status: &status.Status{
                Code: int32(code.Code_OK),
                Message: "AuthServer authentication successful",
            },
        }, nil
    }
  // If invalid token, returns Code_PERMISSION_DENIED 
    return &auth.CheckResponse{
        Status: &status.Status{
            Code: int32(code.Code_PERMISSION_DENIED),
            Message: "AuthServer authentication unsuccessful",
        },
    }, nil 
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    opts := []grpc.ServerOption{grpc.MaxConcurrentStreams(10)}
    s := grpc.NewServer(opts...)

    auth.RegisterAuthorizationServer(s, &AuthServer{})

    log.Println("Starting gRPC Server at 50051")
    s.Serve(lis)
}

Rate limiting with Envoy ratelimit gRPC service

Envoy Proxy offers multiple rate limiting strategies, including local rate limiting, circuit breaking, and global rate limiting. For this article, we will focus on global rate limiting using the Envoy External Rate Limit Service. Local rate limiting and circuit breakers can also be used in cases where network connection level rate limiting is needed, in addition to application level rate limiting enforced by the global rate limit service.

Add ratelimit filter to the HTTP filter chain

Similar to external authorization, we need to add the Envoy Rate Limit filter to the filter chain to enable the Envoy Proxy to call the global rate limit service. It’s important to add the Rate Limit filter after the External Authorization filter to reduce the load on the external ratelimit service.

http_filters:
  - name: envoy.filters.http.ext_authz
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
      grpc_service:
        envoy_grpc:
          cluster_name: ext_authz
        timeout: 2s
      transport_api_version: V3
  - name: envoy.filters.http.ratelimit
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
      domain: default
      failure_mode_deny: true
      enable_x_ratelimit_headers: DRAFT_VERSION_03
      rate_limit_service:
        grpc_service:
          envoy_grpc:
            cluster_name: ratelimit
        transport_api_version: V3

Provide ratelimit configuration to the external ratelimit service

Envoy Rate Limit Service uses descriptors which are basically key, value pairs to perform rate limiting. In order to use the Envoy Rate Limit Service, we need to provide rate limits related to the descriptors as configuration to the Rate Limit Service.

---
domain: default
descriptors:
  - key: version
    value: v1
    rate_limit:
      unit: second
      requests_per_unit: 5
  - key: version
    value: v2
    rate_limit:
      unit: second
      requests_per_unit: 10

Add ratelimit descriptors to the routes

Additionally, we need a mechanism for Envoy Proxy to inform the External Rate Limit Service to increment a specific descriptor when a resource is accessed. To achieve this, we need to define descriptors for each resource.

 route_config:
  name: local_route
  virtual_hosts:
    - name: upstream
      domains:
        - "*"
      routes:
      - match:
          prefix: "/api/v1"
        route:
          cluster: upstream_service
          rate_limits:
            - actions:
                - generic_key:
                    descriptor_key: version
                    descriptor_value: v1
      - match:
          prefix: "/api/v2"
        route:
          cluster: upstream_service
          rate_limits:
            - actions:
                - generic_key:
                    descriptor_key: version
                    descriptor_value: v2

In this scenario, we have defined two rate limit keys. When resources with prefix /api/v1 are accessed, the rate limit policy with descriptor value v1 is applied. Similarly, for resources with prefix /api/v2, the rate limit policy with descriptor value v2 is applied. This means that API v1 can be accessed 5 times per minute, while API v2 can be accessed 10 times per minute. If the quota is exceeded, Envoy Proxy returns a 429 error response without processing the request any further.

Collecting telemetry data for distributed tracing

As demonstrated in the above scenario, Envoy Proxy makes multiple gRPC/HTTP requests to external services during request processing. Therefore, it’s important to have better visibility into what’s happening behind the scenes for debugging errors or monitoring purposes. In the above scenario, we can configure Envoy Proxy, External Authorization Service, and Rate Limit Service to publish telemetry data to a telemetry backend. For this example, we use Jaeger as the telemetry backend and OpenTelemetry to generate telemetry data.

Envoy proxy tracer configuration

- name: envoy.filters.network.http_connection_manager
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
    codec_type: AUTO
    stat_prefix: ingress_http
    generate_request_id: true
    tracing:
      provider:
        name: envoy.tracers.opentelemetry
        typed_config:
          "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
          grpc_service:
            envoy_grpc:
              cluster_name: jaeger
            timeout: 1s
          service_name: proxy

Publishing telemetry data from external authorization service

func (authServer *AuthServer) Check(ctx context.Context, request *auth.CheckRequest) (*auth.CheckResponse, error) {
    tracer := otel.Tracer("ext-authz")
  // create span
    _, span := tracer.Start(ctx, "ext-authz-span")
    defer span.End()
    log.Printf("Auth server received auth request: %v", request.String())
    authHeader, ok := request.Attributes.Request.Http.Headers["authorization"]
    var tokenString string

    if ok {
        tokenString = strings.TrimPrefix(authHeader, "Bearer ")
    }
  // set attributes to span
    span.SetAttributes(
        attribute.String("example-key", "example-value"),
        attribute.String("guid:x-request-id", request.Attributes.Request.Http.Headers["x-request-id"]),
    )
    fmt.Printf("Token: %s", tokenString)
    _, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
        if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
            return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
        }
        return secretKey, nil
    })
    fmt.Printf("Error: %v", err)
  // Span end
    span.End()
    if err == nil {
        return &auth.CheckResponse{
            Status: &status.Status{
                Code: int32(code.Code_OK),
                Message: "AuthServer authentication successful",
            },
        }, nil
    }
    return &auth.CheckResponse{
        Status: &status.Status{
            Code: int32(code.Code_PERMISSION_DENIED),
            Message: "AuthServer authentication unsuccessful",
        },
    }, nil 
}

Summary

This article mainly focuses on the application-level security and scalability of the API layer. We achieve application-level security using the External Authorization Service, which deals with identity verification and authorization, and the External Rate Limit Service, which ensures the availability of upstream services by performing the configured rate limits.

Scalability can be achieved by scaling the necessary components based on requirements. It’s recommended to identify the bottleneck and scale the components accordingly. For example, if the rate limit service is causing a bottleneck in request processing and adding high latency, the external rate limit service can be scaled independently without scaling other components.

This kind of API layer helps application developers focus on writing business logic in their microservices without worrying about security and scalability.

Complete source code for the above setup with a docker-compose deployment: secure-scalable-api-layer

A Beginner’s Guide to gRPC and protobuf

Lahiru De Silva — Sat, 25 Feb 2023 00:00:00 GMT

Since both protocol buffers and gRPC framework are developed internally at Google and then open sourced, let’s first see what google says about them.

Protocol buffers

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

So the highlighted text defines exactly what protobufs do. But what is data serialization ? Data serialization is the process of converting some in-memory object to another format that could be used to either store in a file or sent over the network. In layman terms, protocol buffers convert application level data objects to binary data that can be stored or transmitted over the wire. So this is where gRPC comes to play. gRPC deals with the process of transmitting the binary data generated from protocol buffers. Let’s see what google says about gRPC.

gRPC

gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication.

So from the above definition, it is clear that gRPC is a RPC framework which has several improvements over traditional RPC frameworks. If you don’t know what a RPC framework is, simply it’s a set of tools that enable the programmer to call a piece of code in a remote process, be it on a different machine or just another process on the same machine.

Developer workflow

Create a .proto file and define the message types and services that we want to work with.A message is analogous to a request/response in the REST world. We have to define messages by using the in-built data types. And then we have to define services. In gRPC there are 4 types of services that we can use.
- gRPC unary
- gRPC client streaming
- gRPC server streaming
- gRPC bi-directional streaming
Since this is an introductory article, I am not going to explain about them further.
Then we have to use the protoc compiler and compile the .proto file that we created in the previous step. For this step, you have to install the protoc compiler. Follow the official installation guide for protoc compiler installation.
After compiling the proto file in the previous step, proto generated files will be there in a location that you specified. For example in C++, two files will be generated with file extensions .pb.h and .pb.cc. In golang it will be a single file with .pb.go. Other programming languages that support protocol buffers will generate one or more files with different file extensions based on the language. These files contain methods that can be used to create messages, send and receive messages etc.
As the final step, the generated files can be imported from the application source files and the relevant methods can be used to build, send and receive messages.

Since this is an introductory article, I’m not going to add an example here. I hope to publish separate articles for the different types of gRPC services with examples.

gRPC vs REST

Now, let’s do a quick comparison of gRPC and REST to see their differences.

gRPC uses HTTP/2 which is, as you know, much faster than HTTP/1.1 used in REST by default. Note that today we can enable HTTP/2 in REST as well, but normally it often goes with HTTP/1.1. Binary framing, compression and multiplexing capabilities of HTTP/2 improves the performance of gRPC significantly when compared to REST with HTTP/1.x.
gRPC uses Protocol buffer to serialize payload data, which is binary and smaller, while REST uses JSON, which is text and larger.
- One of the common questions that arises when you learn about binary serialization of protocol buffers is that “does it matter when everything that is transferred through the wire is binary ?” In trying to answear that question, let’s see how text based serializers work. A text based serializer, serialize everything as text to make it more human readable. This process will add several symbols to make it more clean and human readable which adds more bytes.Since at the bytes level, a string values usually takes more space than other data types,compared to binary serializers, text based serializers generates more bytes. So a text based serializer will first serialize application data objects into a text and then the bytes relevant to that text is transferred through the wire. In contrast to text based serializers, binary serializers serialize application data objects to binary data which lives more closer to the hardware. For example let’s consider the integer 42.

json - {"id":42} - 9 bytes if we assume ASCII or UTF-8 encoding and no whitespace.

xml - 42 - 11 bytes if we assume ASCII or UTF-8 encoding and no whitespace - and no namespace noise like namespaces.

```
protobuf - 0x08 0x2a - 2 bytes
```
Due to small payloads, fast and efficient serialization in protocol buffers, gRPC performs better than REST with HTTP/1.x in most of the scenarios. But at the same time binary serialization makes it more difficult to work with/debug a gRPC API than a REST based API.

The API contract in gRPC is strict, and required to be clearly defined in the proto file. While in REST, it’s often loose and optional. We can define it via OpenAPI if we want, but it’s not mandatory. Having a strict contract makes it easy for the developers to develop and maintain the API when several teams are involved in the process. However for simple APIs, gRPC can consume extra unnecessary time and effort. Also protobuf field numbers can be used to evolve the API with versions while making the API backward compatible.
Code generation is built-in in gRPC with the help of protocol buffer compiler. While in REST, we must use third-party tools like OpenAPI and Swagger. gRPC support is not available in all of the programming languages. gRPC is currently officially supported in most of the widely used programming languages and there are several other community based implementations.
gRPC APIs are not strictly resource/entity oriented although they can be defined in a resource/entity oriented way. In contrast REST APIs are mostly entity/resource oriented and must be resource/entity oriented if it is defined as a RESTful API. Resource/entity orientedness of REST APIs can be a performance bottleneck when several resources are needed to perform a certain operation since a HTTP call is required per operation per resource. (Only if the API is resource/entity oriented). For example if resource B is needed to be updated after creation of resource A, then a POST request is required for resource A and a PUT request is required for resource B. We can implement both of those under one gRPC service since gRPC is not strictly resource/entity oriented.
With the HTTP/2 streams, gRPC has better support for streaming than the HTTP/1.x. With HTTP/1.x server streaming can be achieved using server sent events (SSE) or by maintaining a persistent connection by specifying Transfer Encoding: chunked in the response header. Compared to that, gRPC offers better support for streaming by providing client streaming, server streaming and bi-directional streaming out of the box.

When to use gRPC

From the previous section it is pretty obvious that gRPC has a lot of new features and strengths when compared to HTTP/1.1x based REST APIs. However with the current state of gRPC, browser support is not yet available. It’s impossible to directly call a gRPC service from a browser today. gRPC heavily uses HTTP/2 features and no browser provides the level of control required over web requests to support a gRPC client. For example, browsers do not allow a caller to require that HTTP/2 be used, or provide access to underlying HTTP/2 frames. To overcome this issue, grpc-web was introduced but not all grpc features are supported in grpc-web. The basic idea of grpc-web is to have the browser send normal HTTP requests (with Fetch or XHR) and have a small proxy in front of the gRPC server to translate the requests and responses to something the browser can use. Due to this limitation , grpc is mostly used for internal service to service communication.

gRPC is well suited to the following scenarios.

Microservices: gRPC is designed for low latency and high throughput communication. gRPC is great for lightweight microservices where efficiency is critical.
Point-to-point real-time communication: gRPC has excellent support for bi-directional streaming. gRPC services can push messages in real-time without polling.
Polyglot environments: gRPC tooling supports all popular development languages, making gRPC a good choice for multi-language environments.
Network constrained environments: gRPC messages are serialized with Protobuf, a lightweight message format. A gRPC message is always smaller than an equivalent JSON message.
Inter-process communication (IPC): IPC transports such as Unix domain sockets and named pipes can be used with gRPC to communicate between apps on the same machine. For more information, see Inter-process communication with gRPC.

References

How we implemented an authorization cache for Envoy proxy

Lahiru De Silva — Thu, 25 Nov 2021 00:00:00 GMT

Note

This article was originally published on Red Hat Developer Blogs and co-authored by Rahul Anand and Lahiru De Silva (myself). Read the original article here: Original Article on Red Hat Developer Blogs

This is the second article in a two-part series about an authorization cache we developed for the Envoy open source proxy as a part of Google Summer of Code 2021. The first article in the series showed the design of our cache, based on Proxy-Wasm and integrated with Red Hat 3scale API Management. In this part, we focus on the implementation aspect of the design described in the previous article (check it out if you haven’t already).

Proxy-Wasm extensions

As mentioned in the first part of the series, at this time of writing Proxy-Wasm supports two main types of extensions: filters and singleton services. A filter executes in a worker thread and a singleton service executes in the main thread.

The Proxy-Wasm specification currently defines two types of filters: network filters and HTTP filters. There are also three types of contexts:

RootContext: This context takes care of such tasks as reading the virtual machine (VM) configuration, reading plugin configurations, setting tick periods, creating contexts, and any work outside the request lifecycle. One RootContext gets created for each thread in each plugin.
StreamContext: This context is used when writing network filters. One StreamContext gets created for each request.
HTTPContext: This context is used when writing HTTP filters. One HTTPContext gets created for each request.

Our design uses a singleton service and an HTTP filter. So our implementation uses only a RootContext and an HTTPContext.

Cache filter

A filter object has the same lifecycle as a request. Therefore, to reduce request latency, our implementation shifts any task that can tolerate a delay to the singleton service. The filter does only the bare minimum work required. As mentioned in the first part of the series, there are two main flows for the filter:

Cache miss: Whenever a cache record is not present in the shared data, the filter proceeds with this path. Because the task is to authorize an HTTP request, the service ID and application ID are combined into a string into the format {ServiceID}_{AppID} to create the key in the cache. The filter calls out to the 3scale Service Management API to obtain the latest application state. Following the call-out response, the fetched state is stored as the cache entry value after deserialization, and future requests for the same service and application benefit from cache hits.
Cache hit: If a cache entry is present, its serialized string value is fetched and deserialized into Rust structs. The decision to authorize the request is based on the left quota for each method and metric, which are already configured on the 3scale API Management dashboard. If authorized, the request is allowed to proceed to the next filter in the filter chain, and if not, a local response from the cache filter is sent back with appropriate response headers.

If any anomaly is found during the processing of the request, such as crucial missing information, a failure to serialize or deserialize the application state, or a processing failure by the host proxy, appropriate status codes are sent back.

Figure 1 shows how we map the XML entities in the authorization response to Rust variables and structs.

Relationships between XML entities in the authorization response and Rust variables

Figure 1: Relationships between XML entities in the authorization response and Rust variables

Singleton service

The singleton service is responsible for two major functions: flushing the cache and updating the cache. These support the metrics we report on the use of different resources based on endpoints, URLs, etc.

Without the cache, all the request metrics are sent to 3scale API Management’s Software-as-a-Service (SaaS) platform using the authrep.xml endpoint. The rapid updating of metrics allows users to see usage in almost real-time and hence with higher accuracy of reported metrics and higher latencies of the request being served. By design, if the cache is used, metrics are shown a bit later than the real-time request being served because of batching.

Our cache project eliminates the HTTP calls from the proxy to an external service on each request received. Instead, we rely on the stored cache records inside the proxy to minimize request processing time. To support the reporting of request usage and other related information to an external management service, we aggregate request usage over a particular time duration or based on a memory limit and then send bulk HTTP requests to the management service using the report.xml endpoint.

The local cache is flushed to an external service on a periodic basis or based on a predefined policy. Because we are using a singleton service, this cache flushing process runs in the main thread as a background task without blocking the worker threads.

The singleton service also updates the local cache with new snapshots of the applications and services. After flushing the local cache based on a configured policy, the local cache update process gets initiated. This process fetches the latest state of an application using the authorize.xml endpoint and updates the cache record stored inside the proxy with that latest data.

Integration of cache with singleton

The first part of this series mentioned that the cache filter sends events to the singleton service via a message queue. Let’s discuss that queue in more detail.

Our implementation uses the shared queue feature in the Proxy-Wasm ABI. The shared queue is one of the most under-represented features of Proxy-Wasm. Still, it can be used for more complex applications such as sharing data between worker threads, cross-VM communication, etc. In our case, we use it to transfer request metadata from the cache filter to the singleton service.

A shared queue supports basic enqueue and dequeue operations, as well as a callback that notifies the recipient whenever data is available in the queue. We effectively used this callback mechanism to implement an event-passing system between the filter and the singleton, so the singleton service doesn’t have to check the shared queue for messages.

Testing

For a project with several interconnected components, integration tests and end-to-end tests are essential to verify the functionality of the features and ensure that new development does not break previously implemented functionality. So we invested time and effort to develop a complete end-to-end testing framework that runs all the tests in a Docker Compose setup with all the required services running in Docker containers.

The integration testing framework has the following key features:

Using everything as we use in a real production environment without mocking
Flexibility to start new proxy instances per each test or per each test suite
Generating an Envoy configuration from a common template

Additional cache features

Some of the other neat capabilities of our cache include:

Custom metrics: One of the primary goals of the Envoy proxy is to make the network understandable. Envoy can emit a large number of statistics, depending on how it is configured. We made use of Envoy’s metrics API to define custom metrics that are more related to business logic. For example, we defined custom metrics to record the total number of applications stored in the cache, total Authorize calls to the 3scale Service Management API, total timeouts for the Authorize calls, etc. The metrics defined here can be scraped by Prometheus and visualized with a Grafana dashboard.
Unique call-out: The unique call-out feature ensures that, for a single record, there is only one call-out to the 3scale Service Management API. Because there is no synchronization mechanism between worker threads, we created an issue in the upstream envoy proxy project to request a solution for this use case. But because any change in the ABI will take more time than we had at Google Summer of Code, we found a way to use shared data as a placeholder for the lock and message queues to signify the resumption of waiting contexts between different threads. This use required us to bend the expected use of set_shared_data by sending some non-zero CAS value to avoid the initialization of the entry by multiple threads. There is still a nearly infinitesimal chance of multiple initializations, but their eventual correction is assured.
Visible logs: Writing unit tests for Proxy-Wasm modules was not reasonable because it requires a runtime that is available during only integration, and at that stage, we cannot check individual functions. So we decided to blur the line between unit and integration tests by developing a feature called visible logs. If enabled, trace logs are sent in the response header, which is later matched against regular expressions to see whether the expected strings appear and thus verify the function’s inner workings as unit tests do. This feature was implemented by intercepting and storing the logs for each request and dumping them into a response header when the response is ready.

Benchmarks

Performing reliable and accurate benchmarks is crucial for our project because the main goal of the project is to improve the performance of request latency. So we performed extensive benchmarks for different traffic profiles and configurations. Results clearly show that the proxy with our internal cache outperformed the proxy without the cache. Figure 2 shows results from the proxy without the internal cache, while Figure 3 shows the proxy with our cache.

Response times are noticeably high without the cache

Figure 2: Response times are noticeably high without the cache

Response times stay low with the cache in place

Figure 3: Response times stay low with the cache in place

Also, we can see the transition from cache miss to cache hit in the request latency diagram of the proxy, where it is marked by a sudden drop in the maximum latency curve.

Apart from benchmarking request latencies, we also benchmarked the unique call-out feature to prove that it improves performance during the initial cache miss stage by preventing multiple HTTP calls. Figure 4 compares request latencies for the setup with and without unique call-outs.

Latencies are improved by the unique call-outs feature

Figure 4: Latencies are improved by the unique call-outs feature

The blue and red lines represent two loads without enabling unique callout, and the rest of them are with unique callout enabled. It shows 99% of the request latencies are similar for both of the setups, but the setup with unique call-outs outperforms the default proxy setup during the initial cache miss period. (The final 1% of the diagram denotes the high latency requests during the initial cache miss period.)

All our benchmark results can be found in the 3scale-labs repository.

Limitations and planned improvements

For the most part, our modules do what they are supposed to do But some limitations exist and we can see areas for improvement. Here is a non-exhaustive list:

Under the hood, the cache is just a C++ hashmap. Any problems hashmaps produce at scale are present in our cache as well. One way to improve this scaling problem is to run a dedicated database process such as Redis run alongside the envoy process. However, the dedicated process would also marginally reduce the savings in latency.
Proxy-Wasm’s host implementation uses mutexes per access type (read/write) for concurrency control on the cache, which means that at most one write and one read can have concurrent access to the cache. To improve parallelism, we can define an atomic operation on each entry (slightly increasing memory use). If we made this change, multiple threads would be able to access different entries concurrently, whereas now only one thread at a time has access to one entry.
With the current implementation, the singleton service can be configured to operate in three modes: container, periodic, and default. These modes define how the cache’s flush and update operations are performed. However, these simple configurations don’t allow the developer to define the behavior of individual stages in the cache flush/update process. The singleton service can be made more intelligent and fault-tolerant by implementing a policy-based system for cache flushes and updates.
To overcome the issues and constraints imposed by Wasm, Proxy-Wasm, and Envoy, we could implement an external service extension based on a local remote procedure call (RPC) mechanism.
As of right now, there is no ability to delete an entry fully in the cache due to limitations in the ABI. But this will be fixed soon in the next ABI update by the maintainers.

Conclusion

We worked hard during the Google Summer of Code to get this project where it is today. The cache will hopefully be in production during the next release cycle of Red Hat OpenShift Service Mesh. But without the help of the great 3scale API Management team, especially our mentor Alejandro Martinez Ruiz, it wouldn’t have been possible for us to complete this project and then write an article about it. We would also like to thank Andrew Mackenzie for his support and rigorous reviews of this series.

Finally, we highly encourage students to consider applying to Google Summer of Code, because it changed our lives and careers for the better and can help others too.

Design an authorization cache for Envoy proxy using WebAssembly

Lahiru De Silva — Thu, 18 Nov 2021 00:00:00 GMT

Note

This article introduces a high-level design to implement an authorization cache associated with the Envoy proxy using WebAssembly. The goal of this project is to reduce the latencies of HTTP requests passing through the Envoy proxy by reducing the traffic to the service responsible for authentication and authorization of requests. The cache stores data about authorization so that the external service needs to be contacted only on cache misses, instead of for every HTTP request.

We also provide the source code of an authorization cache that interacts with Red Hat 3scale API Management. The cache was implemented as a part of the Google Summer of Code 2021 project.

This article is the first in a two-part series. This first article introduces a high-level, generic design that will give you a basic idea of the cache’s overall functionality. The second part explains the major design decisions and implementation details.

What is Envoy proxy?

Envoy is an open source edge and service proxy for applications running in the cloud. Envoy is valuable for many use cases, including edge proxy, middle proxy, sidecar for service mesh deployments, and a daemon set within Kubernetes. Among Envoy’s compelling features, performance, extensibility, and API configurability are the most prominent, making it unique in the proxy space. This article mainly focuses on extensibility.

The Envoy proxy authorization cache in action

The implementation uses Proxy-Wasm, implementing the extensions as WebAssembly modules. We made this choice based on the flexibility, portability, maintainability, and isolation (for fault tolerance) that WebAssembly offers when compared with native Envoy filters. At the time of writing this article, Proxy-Wasm supports two types of extensions: Filters and singleton services.

Envoy communicates through the Wasm virtual machines with the 3scale Service Management API

Figure 1: Envoy communicates through the Wasm virtual machines with the 3scale Service Management API

Figure 1 shows the interactions between the proxy, the 3scale Service Management API, and the upstream services when a client sends a request. The request is propagated through the Envoy filter chain. For access to Wasm filters and services, the request interacts with the Wasm Virtual machine (VM) to execute the Wasm plugins.

Designing the authorization cache

In our design, the cache inside the proxy is implemented through two main components: A filter and a singleton service.

The filter is responsible for intercepting HTTP requests, authorizing them based on the stored cache, and performing rate limiting. In the context of the envoy, this component is an HTTP filter and gets executed in the worker threads. For each request, a context object gets created.

The singleton service is responsible for the background synchronization of cached data between the proxy and the 3scale Service Management API. In the context of the envoy, this is a singleton service and gets executed in the main thread outside the request lifecycle. Only one instance of this service gets instantiated in each Envoy process.

The filter’s threads and the singleton service share data and a message queue

Figure 2: The filter’s threads and the singleton service share data and a message queue

Figure 2 shows how the HTTP filter and singleton service interact with the other internal and external components to provide the expected functionality of the in-proxy cache. Shared data is an in-memory key-value store specified by the Proxy-Wasm ABI and provided by the proxy. Each VM contains a shared datastore. Because the two extensions (filter and singleton) are running in the same Wasm VM, both extensions have direct access to the shared data.

I/O between the host and VM is done in binary format and thus requires serialization and deserialization. The Proxy-Wasm ABI provides a shared queue that is also unique per VM. In our design, the shared queue is the main communication channel between the singleton service and filter. Any message enqueued is broadcast to all threads, but only one thread can dequeue the message. The singleton service updates the cache records saved in the shared data, either periodically or based on policies defined in the configuration. The HTTP filter uses the cache records in the shared data to perform authorization and rate-limiting.

Filter design

The default Envoy proxy makes an external HTTP call to the 3scale Service Management API for every HTTP request to perform authorization and reporting. But with the internal cache, we eliminate this need and limit the external HTTP calls to cache misses (Figure 3). That way, we reduce the traffic on the 3scale Service Management API and therefore overall request latency, to a great extent.

Envoy proxy filter chain with and without cache filter

Figure 3: Envoy proxy filter chain with and without cache filter

The filter serves as the entry point for the request authorization with the help of cached data. The filter takes one of two pathways for each request:

Cache miss: This happens when a cache record is not found in the shared data. The filter calls out to the 3scale Service Management API to fetch the latest state.
Cache hit: This happens when a cache record is present. Based on authorization results, metrics are passed to the singleton for reporting to the 3scale Service Management API.

The filter is equipped with an opt-in feature called unique-callout, which ensures that there is only one callout to the 3scale Service Management API on a cache miss. This feature increases performance and accuracy under high load, without any impact under low-load conditions. We’ll look at the results with and without the unique callout feature in the second part of this series, when we get to examine the benchmarks.

Singleton design

The singleton service serves two main functions for the proposed in-proxy cache design:

Collect metrics based on a predefined policy and report them to the external management service.
Update the local cache stored in the proxy by pulling the latest states of applications and services from the external management service.

In our case, the external management service is the 3scale Service Management API.

In Envoy, almost all the functions in worker threads and the main thread get executed in a non-blocking manner. So the methods get invoked as callbacks. In the singleton service, we use two types of events to perform the required functionality:

Periodic events, triggered at predefined intervals via the on_tick() callback
Events triggered by the filter through the message queue

Events that are sent from the filter pass through the message queue to the singleton service. The second part of this article offers further details about the integration of cache with the singleton section.

Coming up: Implementing the cache

In the next part of this series, we’ll dive into the implementation details of the cache, data modeling, various features that made the cache work better, limitations, and planned improvements.

Next article in the series: How we implemented an authorization cache for Envoy proxy

Our journey to open source during Google Summer of Code

Lahiru De Silva — Sat, 23 Oct 2021 00:00:00 GMT

Note

This article was originally published on Opensource.com and co-authored by Rahul Anand and Lahiru De Silva (myself). Read the original article here: Original Article on Opensource.com

How we made it to GSoC and what we learned about open source code.

Every year, Google organizes a program called Summer of Code (GSoC). Students worldwide can write open source code under an open source mentoring organization and get paid to do so! You get to work on cool open source projects, network with talented engineers, and get paid during the summer break. How cool is that!?

In this blog post, we’ll guide you through our GSoC experience and give you tips and tricks so that you can have a fantastic experience like we did. We’ll also share our differing perspectives based on our different interests and challenges.

About us

Rahul: My name is Rahul Anand, and, at the time of writing this, I am in my senior year of college, majoring in computer engineering at Thapar University, Patiala, India. My interest is in distributed computing because it’s one of the significant areas allowing scalability of applications worldwide. Computer vision is sprinkled on top just because AI will take over the world.

Lahiru: I am Lahiru Udayanga, currently a final-year undergraduate at the Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka. My main focus areas in computer science are distributed systems, API gateways, proxies, and cloud-native technologies.

Choosing the proper organization and project

The first and the most crucial decision you’ll make during this journey is choosing the most suitable organization and the project you want to contribute to. This decision will heavily influence your experience, so be very diligent here. Don’t worry—at the end of this section, we have listed factors you should consider for this stage.

We both decided to apply for the same project: 3scale-envoy proxy authorization cache. However, we each had a slightly different thought process behind choosing this project.

Rahul: I discussed the options with a senior who participated in GSoC’20 under the JBoss umbrella and described his experience as very enriching. That’s how I decided which organization I wanted to participate with. I advise juniors to focus on one good organization and start exploring different projects in it. Regarding project choice, a mix of various factors influenced me. Most important was my first interaction with my mentor, Alejandro Martinez Ruiz. I asked him numerous questions, and he was always ready with detailed replies to all of them. (Tip: Be concise in your questions and respect your mentor’s time.) I also wanted to work with distributed systems, learn Rust, and build a project from the ground up.

Lahiru: The main reason I applied to this project is my deep interest in middleware, API gateways, and proxies. Also, I had some previous experience with envoy proxy from my internship project. I believe that envoy proxy will revolutionize the domain of proxies and that there will be a lot of opportunities for envoy-proxy-related technologies. Like Rahul, my first interaction with Alex and the conversations we had before the proposal-submitting phase also had a significant impact on my decision to apply.

Tips for this period:

If you find any technology cool or want to learn more about it, look for those projects.
Ask seniors about their experiences with different organizations.
Consider the future of this project: do you see a future for this area of research?
If you have a good initial connection with a potential mentor, continue to build rapport.

Developing the proposal

After deciding on which project we wanted to work on, we put our research caps on and started digging for any information related to the problem and possible solutions. During this phase, we were constantly interacting with our mentor about any questions, resources, and solution designs that we were thinking of going forward with independently. It’s best to come as close as possible to the actual outcome and document everything in a separate document for your reference, because it will help immensely during your proposal-creation phase.

After researching for about one to two months, as we neared the opening of proposal submissions, our mentor asked us to start putting everything into a separate Google Doc to be presented as a proposal in front of a committee. It was time to make friends with Shakespeare and type hard work into a document that would change our lives forever!

Jokes aside, it’s imperative to have this document look presentable, with excellent diagrams. We used and recommend draw.io (now diagrams.net), an open source diagramming software. Remember that no one can write a perfect document the first time through, and have a few review cycles with your mentor to get their feedback. You can look at our proposals for an example.

Rahul’s Proposal

Lahiru’s Proposal

We remember the day we clicked the submit button on the GSoC dashboard—it was raining fire from the sky in the Indian subcontinent. With our fates now in the hands of the JBoss organization and the 3scale team at Red Hat, we waited until that fateful day when our parents awoke at 2AM to newly minted adults screaming at top of their lungs, “GSOC BABY!!”

Learning by doing

The next day, we received our first email with onboarding information, each other’s proposal to refer to, and an invitation to a call between Alex and us. During the first call, we learned about each other, our respective cultures, and the 3scale team, and we were tasked with creating a shared document with a realistic timeline. Since this project was massive compared to other GSoC projects and we found clashing obligations in the future, we decided to start working before the coding period began. We were able to finalize a design with proofs-of-concept (POCs) showcasing certain guarantees required by our use cases and an initialized GitHub repository before the official start.

To progress faster, we separated our concerns. Rahul took the responsibility of the Cache-Filter and Lahiru of the Singleton-Service, and with a standard interface finalized during an online meeting. To make sure that we were on the same page and to find blockers early on, Alex scheduled two meetings: weekly syncs for progress, blockers, possible design changes, and POCs, and biweekly syncs for just talking about anything other than work. He also asked us to create weekly reports in a separate Google Doc to be shared with other members of the community. We used these reports to give updates and receive advice or any heads-ups. Both of us struggled during this stage at different places.

Rahul: At the start, I struggled a bit with following a standard workflow and using tools (including Git!) because I wanted to move fast. In hindsight, that helped us avoid future pains, so I am glad I learned them early on. The best advice I got from Alex was that it was our project, and we were free to make final decisions after his feedback. That gave me a sense of ownership, and I started experimenting more, creating more POCs, and bending specs to work with us. This change in attitude allowed me to implement features like visible logs and unique callouts that made integration testing easier and increased performance.

Lahiru: For me, the most irritating thing at the beginning was the steeper learning curve of the Rust programming language. When I started programming in Rust, I was mostly yelling at the compiler instead of writing code. But a few weeks later, we were slowly becoming friends. (I think the Rust compiler is more like an overprotective nanny. It has no interest in being your friend, but it will do everything it can to stop you from hurting yourself.) Also, there are vast differences among the Rust code written by a beginner, an intermediate, and an expert. These differences exist in other programming languages, like Java, Golang, and C++, but not to the same extent. So one of the challenges I had was implementing not only something that works but something that works in the most Rustic way possible.

What we learned

If you want a list of things we learned that you can learn too if you participate in the Summer of Code, here it is:

Collaboration: How to interact with the open source community, how to give context to a person new to your work
Best practices for developing software: How to structure PRs and commits, how to set up and write integration or unit tests
New technologies: Rust, Golang, Proxy-WASM, Envoy, etc.
Soft skills: How to interact with senior engineers, how to introduce yourself
Distributed computing theory in action and problem-solving skills in general

Google Summer of Code 2021 was an enriching experience for both of us. We got to interact with great engineers like Piotr Sikora (some say CPP is a language from hell, but he turns it into beautiful prose), Takeshi Yoneda (our savior when we needed it the most), David Ortiz (the myth, the legend, the co-creator of Limitador), and (obviously) Alex Martinez, along with the whole 3scale team. Without these people, we don’t think we could have made it to the final stages and be listed on the GSoC website.