k8wiz
All graphs

Kubernetes graph

Gateway API: GatewayClass, Gateway and HTTPRoute with a canary split

How Gateway API splits ownership between the platform team and app teams: GatewayClass, a shared Gateway, a ReferenceGrant for a cross-namespace certificate, and an HTTPRoute that sends 10% of traffic to v2.

Gateway API replaces one Ingress object with three, one for each team. The infrastructure provider supplies the GatewayClass, the platform team runs a shared Gateway, and app teams attach HTTPRoutes to it from their own namespaces.

Two different mechanisms control what may cross a namespace. A Gateway's allowedRoutes decides which namespaces may attach routes to it. A ReferenceGrant, written in the namespace being pointed at, allows a reference into that namespace, here a Gateway reading a certificate Secret from certs. This example uses Envoy Gateway, but the objects are the same for Istio, Cilium, NGINX Gateway Fabric and the other implementations.

Interactive graph

Click a resource to see what it does, or follow the setup step by step.

Outside the clusterComes with an add-onYou write itCreated by a controllertrafficreferencescreatesselects by label

How it works

All 14 resources and what they do

Client Browser
Calls https://shop.example.com/api/orders.
Load balancer Cloud load balancer
Provisioned by the cloud for the type: LoadBalancer Service that the implementation creates for the Gateway. Its address shows up in the Gateway's status.addresses.
GatewayClass eggateway.networking.k8s.io/v1
Cluster-scoped. Says which controller implements Gateways of this class, through controllerName. The controller sets Accepted: True in its status when it takes the class.
Deployment envoy-gatewayapps/v1
The Gateway API implementation. It watches GatewayClasses with its controllerName, then Gateways and Routes, and runs the Envoy proxies that carry the traffic.
Gateway publicgateway.networking.k8s.io/v1
Owned by the platform team. It opens an HTTPS listener on port 443 for *.example.com, takes its certificate from the certs namespace, and lets routes attach only from namespaces labeled gateway-access: public.
Service envoy-infra-public-…v1
Created by Envoy Gateway for this Gateway, with a Deployment of Envoy Pods behind it. It is type: LoadBalancer, so the cloud gives it a public address. Other implementations create similar objects under other names.
ReferenceGrant allow-infra-gatewaysgateway.networking.k8s.io/v1beta1
Lives in certs, the namespace being referenced. It allows Gateways in infra to read Secrets here. Without it, the listener reports ResolvedRefs: False with the reason RefNotPermitted.
Secret wildcard-example-comv1
The *.example.com certificate, kept in its own namespace so that only a few people can read the private key. cert-manager can issue it with a DNS-01 challenge.
HTTPRoute ordersgateway.networking.k8s.io/v1
Owned by the app team in shop. It attaches to the public Gateway, matches shop.example.com with paths under /api, and splits requests 90/10 between two Services by weight.
Namespace shopv1
Carries the label gateway-access: public, which is what the Gateway's allowedRoutes selector checks. Remove the label and every route in the namespace gets detached.
Service orders-v1v1
Selects the Pods of the current version with app: orders, version: v1. Gets 90% of the requests.
Service orders-v2v1
Selects the canary Pods with app: orders, version: v2. Gets 10% of the requests until you change the weights.
Pod orders-v1-…v1
Pods of the stable version, from a Deployment named orders-v1.
Pod orders-v2-…v1
Pods of the canary version, from a Deployment named orders-v2. Weights split requests, not Pods, so one Pod can take 10% of the traffic.

Common mistakes

  • An HTTPRoute whose parentRefs points at a Gateway that doesn't allow its namespace isn't rejected when you apply it. It just never gets traffic. Check kubectl get httproute orders -n shop -o yaml and read status.parents[].conditions. Accepted: False with the reason NotAllowedByListeners means the namespace selector didn't match.
  • A ReferenceGrant goes in the namespace being referenced, not in the one making the reference. Writing it in infra does nothing.
  • Backends in another namespace need a ReferenceGrant too, this time in the Service's namespace with from: HTTPRoute.
  • hostnames on the route must overlap the listener's hostname. A route for shop.example.org on a *.example.com listener is not accepted.
  • A weight of 0 sends no traffic but keeps the backend in the route, which is handy for preparing a canary before you send it anything.

The manifests you write

Everything above that is marked "You write it", in one file. The rest comes from add-ons and controllers.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: infra
spec:
  gatewayClassName: eg
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: wildcard-example-com
            namespace: certs
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: public
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-infra-gateways
  namespace: certs
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: infra
  to:
    - group: ""
      kind: Secret
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders
  namespace: shop
spec:
  parentRefs:
    - name: public
      namespace: infra
      sectionName: https
  hostnames: ["shop.example.com"]
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      backendRefs:
        - name: orders-v1
          port: 8080
          weight: 90
        - name: orders-v2
          port: 8080
          weight: 10
---
apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    gateway-access: public
---
apiVersion: v1
kind: Service
metadata:
  name: orders-v1
  namespace: shop
spec:
  selector:
    app: orders
    version: v1
  ports:
    - port: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: orders-v2
  namespace: shop
spec:
  selector:
    app: orders
    version: v2
  ports:
    - port: 8080

Keep going

Kubernetes quizTest yourself on Pods, Deployments, Services, scheduling, storage, networking, security and kubectl.ArticleHow to drain Kubernetes nodes safely with PodDisruptionBudgetsArticleOCI images explained: manifests, layers, media types and artifactsArticleKubernetes 1.35 Timbernetes: What Breaks, What's Deprecated, and How to Upgrade Without Surprises

More graphs