k8wiz
All graphs

Kubernetes graph

Agent Substrate: actors, warm workers and snapshots on Kubernetes

How Agent Substrate runs many sandboxed agents on a few warm Pods: a request for a suspended actor reaches the router, the control plane picks a worker, atelet restores the snapshot, and suspend writes it back to object storage.

Agent Substrate splits an agent in two. The Actor is the agent's identity and state, kept as a row in PostgreSQL and a snapshot in object storage. The worker is a Pod that is already running and waits for an actor to restore into it. Kubernetes only ever sees the workers.

Click a resource to see its role. The walkthrough follows one request for a suspended actor, then the suspend that frees the worker again.

Interactive graph

Click a resource to see what it does, or follow the setup step by step.

Outside the clusterComes with an add-onYou write itCreated by a controllertrafficreferencesselects by labelcreates

How it works

All 14 resources and what they do

Your platform agent app or gateway
Whatever sits in front of Substrate: your API, kagent, an MCP gateway. It authenticates end users, picks the actor, and sets the ate-target-actor: <atespace>/<actor> header. Substrate doesn't authenticate your end users.
Deployment atenet-routerapps/v1
Envoy with an ext_proc processor. It reads ate-target-actor, asks the control plane to resume that actor, then opens an mTLS CONNECT tunnel to the worker that got it. When every worker is busy it parks the request for up to 5 seconds (--parked-request-budget) instead of returning 503.
Deployment ate-api-serverapps/v1
The Substrate control plane, a gRPC API. It owns the actor lifecycle (create, resume, suspend, revert, delete), picks a worker with room, and runs each resume or suspend as a workflow. It authorizes every call per atespace with an embedded OpenFGA model. kubectl ate talks to it.
StatefulSet postgresapps/v1
The state store for everything that changes often: atespaces, actor templates, actors, workers and their assignments. These rows would be too many and change too fast for etcd. The install can use Cloud SQL instead of this bundled database.
ActorTemplate ate-demo-counter/counter
The blueprint: digest-pinned images, command, literal env vars, durableDir volumes, CPU and memory limits, snapshot fidelity, and a workerSelector that matches WorkerPool labels. It is immutable and lives in the Substrate database, not in Kubernetes. Create it with kubectl ate create actor-template -f template.yaml. Creating it boots the workload once and saves a golden snapshot that new actors start from.
Actor ate-demo-counter/my-counter-1
One agent instance, addressed as <atespace>/<name>. It starts SUSPENDED and moves through RESUMING, RUNNING and SUSPENDING. Its record holds the worker it runs on, if any, and the snapshot it resumes from. Create it with kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter.
SandboxConfig gvisor-defaultate.dev/v1
Cluster-scoped list of sandbox runtime versions for one class, here gVisor's runsc. The installer creates one per class and marks it the class default. Nodes download the binaries from it, so they are not baked into the worker image.
WorkerPool counterate.dev/v1
Warm capacity. replicas worker Pods of one sandbox class, each with limits that set how much the actors placed on it may book together. Its metadata.labels are what an ActorTemplate's workerSelector matches, not the Pod template's labels.
Deployment counterapps/v1
Created by atecontroller from the WorkerPool. Ordinary Kubernetes from here on: kube-scheduler places the worker Pods, and node autoscalers see them like any other Pods.
Pod counter-…v1
A worker: the ateworker process with atunnel inside it, running and waiting. It hosts actors, each in its own gVisor or microVM sandbox with a private network namespace, up to its capacity and --max-actors (default 1000). Pod and actor lifetimes are separate: a worker outlives many actors, and an actor outlives many workers.
Worker one row per worker Pod
The control plane's record of a worker Pod: ACTIVE, UNAVAILABLE or DRAINING, its capacity, what is allocated, and one ActorAssignment per hosted actor. kubectl ate get workers lists them; kubectl get never will.
DaemonSet ateletapps/v1
The node supervisor, Substrate's kubelet for actors. It pulls images, builds OCI bundles, tells ateworker to run, checkpoint or restore a sandbox, and moves snapshots to and from object storage through a plugin sidecar. It keeps the latest snapshot on the node for a while, so the next resume there skips the download.
Object storage gs:// or s3:// bucket
Where snapshots live while actors are suspended: golden snapshots, each actor's last snapshot, and tags. Only the plugin sidecars of atelet and ate-api-server touch it. Sandboxes never do.
Deployment atenet-egressapps/v1
The only way out. Each actor's outbound traffic is tunneled here, checked against its EgressPolicy, and given credentials on the way out, so API keys never enter the sandbox. If the policy check fails or isn't configured, the traffic is dropped.

Common mistakes

  • kubectl get actors returns nothing. Actors, Workers and ActorTemplates live in Substrate's PostgreSQL, not etcd. Use kubectl ate get …. Only WorkerPool and SandboxConfig are CRDs.
  • Suspend is asynchronous by default: it returns once the snapshot is on the node's disk, before the upload. If the node dies in between, the actor comes back from its previous snapshot. Use kubectl ate suspend actor NAME --mode=sync when losing that state matters.
  • An ActorTemplate's workerSelector matches the WorkerPool's own metadata.labels. Labels under spec.template.labels only end up on the Pods, and actors stay unplaceable.
  • Worker capacity comes from the pool's limits, while kube-scheduler packs nodes by requests. An actor whose template limits don't fit in a worker's remaining capacity is never placed, however idle the node looks.
  • Templates are immutable. To change an image or the size, create a new template and run kubectl ate update actor NAME --template NEW on the suspended actor. Its next resume keeps the durableDir data and cold-boots, so memory and root filesystem writes are lost.
  • Container images in a template must be pinned by digest (name@sha256:…). Env vars take literal values only: no valueFrom, no envFrom, no $(VAR) expansion. Keep credentials out of them anyway, and let the egress gateway inject them.

The manifests you write

Everything above that is marked "You write it", in one file. The rest comes from add-ons and controllers.

apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: counter
  namespace: ate-demo-counter
  labels:
    workload: counter
spec:
  replicas: 3
  workerImage: ghcr.io/agent-substrate/substrate/ateworker-gvisor:v1.0.0
  sandboxClasses:
    - name: gvisor      # or microvm, which needs /dev/kvm on the node
  template:
    resources:
      limits:           # worker capacity shared by its actors
        cpu: "1"
        memory: 1Gi
      requests:         # what kube-scheduler packs nodes by
        cpu: 250m
        memory: 1Gi

Keep going

More graphs