Containerizing the Compass Runner

Status: Draft — freezes on merge. R7’s nested-KVM VM run widened pod and node requirements; a real-node rerun remains required. No capability or privileged mode was needed.

Ledger-impact: mints DL-358

Problem / Intent

Compass ships a Runner that launches each agent session as a microVM (cloud-hypervisor + virtiofsd + passt) rootless, as an ordinary host process. Operators who run Compass on Kubernetes have no supported way to deploy it: there is no Runner container image, no lane that builds one, and no recorded answer to what such a container would have to be granted in order to boot a microVM at all.

That last question is the hard one, and it is why this is a design record rather than a packaging task. A rootless host user has affordances a container does not — setuid newuidmap, unrestricted unshare(2), no seccomp filter — so “the Runner already runs rootless” does not establish that a locked-down pod can run it. This record decides whether the Runner is containerized, states exactly what the container is granted and what it is denied, says where the KVM userland and guest assets live, and defines the Kubernetes object contract for a fleet whose real workload — the session microVMs — is invisible to Kubernetes.

Scope is the core capability: the image, the privilege shape, and the generic Kubernetes object contract any conformant cluster can run. Choosing a cloud, a node provisioner, or a GitOps delivery path for a particular deployment is an operator concern and out of scope here.

Approach

Containerize the Runner as the Kubernetes delivery and lifecycle unit, running rootless inside an unprivileged pod with a scoped /dev/kvm device — never privileged: true.

The apparent conflict with the runtime record’s “launched rootless as an ordinary host process per session” (docs/designs/infra/runtime/compass-elastic-session-runtime/microvm-runner.md, §Approach (a) “VMM shape”) dissolves once the two layers are separated. Rootless is a property of the uid the Runner and its VMM children run as. Ordinary host process distinguishes the VMM from a CRI-owned pod-sandbox runtime. A container is a namespaced process tree: cloud-hypervisor, virtiofsd, and passt run as ordinary children of the containerized Runner at a non-zero uid, exactly as they would under systemd.

What the runtime record forbids is privilege — its D7 discussion pushes anything needing a capability the rootless Runner lacks out to operator provisioning plus preflight verification. That sentence rules out capabilities as a requirement; it does not establish that a drop: ["ALL"] + RuntimeDefault pod grants enough, because a pod denies affordances the host-rootless model silently assumed.

This is a grant-tuning question, not a feasibility one. The composition is already verified to boot on Linux with /dev/kvm: the microVM CI/dev enablement record runs KVM-backed boot tests as a required leg on GitHub Actions’ ubuntu-latest. What a pod adds over that environment is confinement — a cgroup device controller, a seccomp filter, and a memory cgroup. R7’s nested-KVM VM run found that the original pod shape does not boot. The pod now uses hostUsers: false, procMount: Unmasked, appArmorProfile: Unconfined, and the required Localhost seccomp profile. The node must support Kubernetes user namespaces, set kernel.apparmor_restrict_unprivileged_userns=0 where the node image enables it, and set /dev/kvm to mode 0666. This widens the pod and node settings but adds no capability and does not require privileged: true.

Container vs host process: why the container wins

A host systemd service and a container both satisfy the runtime record. The container wins on four properties an operator needs and systemd does not provide:

1. Atomic, rollback-able version delivery. The image digest names the entire userland — Runner binary, cloud-hypervisor, virtiofsd, passt, guest kernel and rootfs — as one unit. A host service versions each of those independently, and a partial upgrade is a supported state.
2. A declarative fleet object. One DaemonSet describes the Runner on every eligible node, with a rollout strategy and a health surface, rather than N nodes’ worth of drifted unit files.
3. Isolation of the userland from the host. The VMM binaries and guest assets live in the image, not in the node’s filesystem, so a node never accumulates Compass-specific packages.
4. No node mutation on upgrade. Changing the Runner version is a pod template change, not a node reprovision.

The cost is the privilege question, which is the subject of §Privilege shape and R7.

Net backend: passt, implemented, not an open pick

Networking needs no capability. A rootless VMM cannot create a host tap, so networking is a userspace concern the backend provides — and the implemented D6 backend is passt, an unprivileged userspace forwarder. Guest networking is in-guest; the Runner dials out to the Server over gRPC with its per-Runner token (go/cmd/compass-runner/main.go), so the pod needs no host port and no inbound service.

Privilege shape (pod spec and required node settings)

Every grant listed here is required; anything not listed is denied. The nested-KVM VM run found this shape, but it does not meet the real-hardware bar. A real-node rerun must verify it.

• /dev/kvm via a device plugin — NOT privileged: true, NOT a raw hostPath char-device mount. /dev/kvm is a world-irrelevant char device the KVM API gates by fd; the runtime record’s D3 requires only that it “exists and is openable by the Runner uid”. A generic device plugin advertises a devices.kubelet.io/kvm-style resource and the container runtime injects the device node with the correct cgroup device-controller allowance; the pod requests it via resources.limits.

  The hostPath char-device route was tested on the nested-KVM VM node. /dev/kvm open() failed with EPERM; the privileged control opened. The device plugin remains required to supply the cgroup device allowance. This measured result replaces the earlier inference; see R7 spike findings.

  The device-plugin implementation is an operator pick, not frozen here; any community plugin image an operator pins is thinly maintained and should be pinned by digest. The contract this record carries is scoped device node, zero capabilities, no privileged mode; the plugin supplies the cgroup allowance.

• Pod and container security: hostUsers: false creates a user-namespaced pod. The container securityContext sets runAsNonRoot: true, fixed runAsUser/runAsGroup, allowPrivilegeEscalation: false, and capabilities.drop: ["ALL"]. procMount: Unmasked lets virtiofsd --sandbox=namespace mount a fresh /proc; masked or read-only /proc submounts block it. Kubernetes requires hostUsers: false for procMount: Unmasked.

  appArmorProfile: Unconfined is required because the runtime default profile denies the mount passt performs inside its own user namespace. A custom AppArmor profile is an optional R3 follow-up (OQ-6).

  seccompProfile.type: Localhost with a custom profile remains required. RuntimeDefault failed with passt reporting Couldn't create user namespace. The profile is staged on-node under the kubelet’s seccomp root by operator node provisioning. R3 must minimize its allow list and assert the reference.

  No capability is added. Project-quota assignment (FS_IOC_FSSETXATTR + quotactl) needs CAP_SYS_ADMIN the rootless Runner lacks, so it stays an operator-provisioning concern verified by Runner preflight (the runtime record’s D7). Networking needs nothing (passt).

  With hostUsers: false, /dev/kvm appeared as 65534:65534 in the pod. The host kvm gid is outside the pod’s id mapping, so a supplementalGroups grant cannot provide access and is dropped. The nested-KVM VM run booted only with node /dev/kvm mode 0666: the device plugin supplies the cgroup allowance, and mode 0666 supplies DAC access. The node must support Kubernetes user namespaces and set kernel.apparmor_restrict_unprivileged_userns=0 where the node image enables that restriction.

• hostNetwork: false, hostPID: false. Guest networking is in-guest and the Runner dials out, so no host ports and no inbound service. hostPID is unnecessary because the VMM/virtiofsd children are the Runner’s own descendants inside the container pid namespace.

  The restart semantics follow deterministically rather than being chosen: without shareProcessNamespace the container’s pid 1 is the Runner, so Runner death tears down the pid namespace and the kernel kills every descendant (VMM, virtiofsd, passt); a kubelet-driven restart kills the container cgroup regardless. “Container restart implies full session teardown” is therefore forced, not elected — and it guarantees no stranded VMM processes. The hostPath runtime dir then correctly serves the reap path: after a restart the backend finds only stale pidfiles, which it already handles by pidfile plus process-liveness check.

• hostPath mounts, exactly two:

  1. the session-volume tree (the project-quota filesystem operator provisioning supplies) — read-write, type: Directory;
  2. the Runner runtime dir (--runtime-dir, default /run/compass) plus the microVM runroot (--microvm-runroot) — a single host tree, read-write.

  Host-visible rather than emptyDir because the tree must outlive the pod, not just the container. An emptyDir is pod-scoped and does survive a container restart, so that axis does not distinguish them; what it does not survive is pod recreation or a node reboot, which is exactly when a stale pidfile from a previous pod must still be visible for orphan-reaping. Nothing else: the VMM/virtiofsd binaries and the guest kernel/rootfs/initrd ship in the image, so no hostPath reaches them.

• Explicitly absent: privileged, every capability, hostNetwork, hostPID, hostIPC, any /dev directory mount, any container-runtime socket.

Pod resources, QoS, and eviction: guest RAM is pod RAM

The VMM is an ordinary child of the Runner inside the pod’s cgroup, so guest memory is the VMM process’s own memory and is charged to the pod’s memory cgroup: every session microVM’s RAM counts against the Runner pod. This holds regardless of the virtio-fs memory mode.

--memory shared=on is nonetheless set unconditionally (the argv construction in go/internal/runtime/microvm/launch.go; BootConfig.MemoryMB in go/internal/runtime/microvm/config.go is documented as always launched with shared=on), because virtio-fs requires it. Its consequence here is an accounting one, not a charging one: guest memory is memfd/shared mappings visible in more than one process, so naively summing RSS across cloud-hypervisor, virtiofsd, and the Runner double-counts it. Size against PSS or against the configured guest total, never a sum of RSS.

With absent or low memory requests the pod is BestEffort (requests and limits fully absent) or Burstable with a large usage-over-requests overage — precisely the pod the kubelet’s node-pressure eviction ranks first — and evicting it kills every session on the node. A priorityClass helps preemption and eviction ranking only if requests are honest. The DaemonSet’s requests and limits MUST account for the aggregate guest RAM of the node’s session capacity, not just the Runner process itself. R3’s test cycle asserts requests and limits are present and sized to the session-capacity model.

Where the KVM userland and guest assets live: in the image

The image carries the Runner binary, cloud-hypervisor, virtiofsd, passt, and the guest kernel/rootfs/initrd. This is deliberately image-heavy: it is what makes the digest name the whole userland (§Container vs host process, property 1) and what keeps hostPath down to two mounts. The guest assets dominate the image size; that cost is accepted in exchange for atomic version delivery.

Kubernetes object contract

Stated as generic Kubernetes so any conformant cluster can run it. An operator supplies the values (node labels, taints, the device-plugin resource name, capacity sizing); this record fixes the shape.

• DaemonSet, so the Runner lands on every eligible node.
  • nodeSelector on an operator-chosen label key, and a matching toleration for the taint that keeps non-Runner workloads off those nodes. Keys and values are the operator’s; the contract is that both exist.
  • updateStrategy with a bounded maxUnavailable: a Runner pod replacement terminates that node’s sessions (§Privilege shape, restart semantics), so a rollout is session-affecting and must be rate-limited.
  • securityContext, device resource request, and the two hostPath mounts per §Privilege shape.
  • spec.nodeName via fieldRef into the environment, so the Runner can identify its node.
  • A scrape annotation for the metrics endpoint.
  • Liveness probe: conservative, or absent. A probe-driven container restart is the same full-session teardown as a rollout (§Privilege shape, restart semantics) — pid 1 dies and every session on the node dies with it — but nothing rate-limits it the way maxUnavailable bounds a rollout, so a transient health blip costs the whole node’s sessions. Prefer a readiness probe for traffic gating and either omit liveness or give it a failure threshold well past any transient stall. A startup probe is unobjectionable.
  • terminationGracePeriodSeconds sized to the reap budget. On termination the kubelet SIGTERMs pid 1 and SIGKILLs after the grace period; that window is what lets the Runner shut its VMM children down in order and notify the Server. Too short and teardown is hard — still no stranded processes (the pid namespace guarantees that), but no graceful session drain. Size it to the per-session reap budget times the node’s session capacity.
• RBAC: a dedicated ServiceAccount with the narrowest role the Runner actually needs. The Runner dials out to the Server and does not drive the Kubernetes API for session work, so this is minimal by construction.
• priorityClass: high enough that a Runner pod is not preempted by ordinary workload, consistent with the eviction reasoning above.

Relationship to the elastic-session-runtime record

This record does not amend the runtime record’s D3/D6/D7. It consumes them and adds the container/pod layer beneath: D3’s “openable by the Runner uid” becomes a device-plugin grant plus node mode 0666; D6’s passt backend is unchanged; D7’s quota assignment stays outside the Runner, satisfied by operator provisioning and checked by preflight.

Alternatives considered

Host-level systemd service — rejected

Satisfies the runtime record and avoids the privilege question entirely, but loses all four properties in §Container vs host process: no atomic digest, no declarative fleet object, node-resident userland, and node mutation on every upgrade. Rejected for operators who are already running Kubernetes; it remains the right shape for a bare-metal single-node deployment, which this record does not address.

privileged: true DaemonSet — rejected

Trivially works and is what most VM-on-Kubernetes stacks ship. Rejected: a privileged pod holds every capability and effectively owns the node, which defeats the isolation the microVM boundary exists to provide. The whole point of the microVM is that a session cannot reach the host; a privileged Runner re-opens that path from the other side.

RuntimeDefault seccomp with no custom profile — rejected

Rejected because the nested-KVM VM run failed with passt reporting Couldn't create user namespace. The required Localhost profile is staged on-node before the pod starts. R3 must minimize its allow list.

Static pod / runner-in-node-image hybrid — rejected

Bakes the Runner into the node image and manages it as a static pod. Gets atomic delivery of a sort, but the unit of rollback becomes the node image, so a Runner version bump is a node reprovision — the exact cost containerizing is meant to remove.

Kata / pod-sandbox runtime class — rejected upstream

Already rejected by the runtime record: the Runner runs the VMM as its own children, not as a CRI-owned pod sandbox. Restated here only because it is the first thing a Kubernetes reader reaches for.

Global Constraints

• No capability, no privileged. R7 widened the pod and node shape along these axes: user namespaces (hostUsers: false), procMount: Unmasked, appArmorProfile: Unconfined, the Localhost seccomp profile, node sysctl, /dev/kvm mode, and device-plugin delivery. None adds a capability or permits privileged mode. A requirement for either would reopen the container-vs-host ruling.
• The image is the unit of version. Anything in the KVM userland or guest asset set ships in the image. The only node requirements are the two hostPath trees, the seccomp profile, Kubernetes user-namespace support, kernel.apparmor_restrict_unprivileged_userns=0 where enabled by the node image, and /dev/kvm mode 0666.
• A rollout is session-affecting — and so is anything else that replaces or restarts the pod. Any change to the pod template terminates sessions on each replaced node, so every delivery mechanism must rate-limit it. The same blast radius applies to a probe-driven restart and to a node drain (autoscaler, node upgrade, manual drain), neither of which is a delivery mechanism and so neither is covered by that rate limit. A fleet PodDisruptionBudget can bound concurrent drains — it applies to eviction-API disruption, though not to the DaemonSet controller’s own rollout — but it cannot keep a pod alive while its own node drains, so draining a Runner node is inherently session-terminating and must be scheduled as such.
• Operator-supplied values stay operator-supplied. This record fixes object shape, not node labels, taint values, capacity numbers, or a GitOps path.
• Deployment-plane concerns are out of scope. Which cluster, which node provisioner, and how the manifests reach a cluster are operator concerns; they are named and deferred here, never described.

Plan

R1 — runner-image/ project

A Dockerfile built on a minimal hardened base (distroless, or Alpine where a shell is genuinely needed), carrying the Runner binary, cloud-hypervisor, virtiofsd, passt, and the guest kernel/rootfs/initrd.

Not a nix image. The image-build mechanisms split by what the image’s runtime is, not by what built the artifact: an image that is a Nix environment (a CI step image, a dev/agent shell) earns nix2container; a prebuilt application on a minimal base is a Dockerfile built by rootless BuildKit. A first-party image does not earn the nix path merely because Nix built it or it ships a compiled binary — build tool and runtime base are independent choices. The prescribed shape for a nix-built artifact is to nix build it and COPY the result onto the base. nix2container for app images is rejected on cost: a per-app regeneration tax plus a maintained fork.

The Runner is squarely the prebuilt-application row: a static CGO_ENABLED=0 Go binary that exec’s three userland binaries and runs no package manager, toolchain, or nix at runtime. The agent-image lane sits on the other row by name, because that container’s job is to be a toolchain. Sharing the *-image/ directory shape is not sharing the mechanism.

Distroless is the stronger posture for a managed multi-tenant cluster and fits the Runner’s shape — but the KVM userland binaries come from nixpkgs and carry store-path interpreter and rpath references, so R1 must settle how they are made runnable on a minimal base (a static or patchelf’d copy, or an Alpine base with the loader present). That is R1’s one real engineering question.

Test cycle: the image builds reproducibly; the resulting image contains each expected binary and guest asset; every carried binary actually executes on the chosen base (the store-path-reference check above, which a layer-contents assertion alone would miss).

R2 — publish lane

Extend the release workflow to build and publish the runner image by digest, built with rootless BuildKit (buildkitd/buildctl) per the spec cited in R1 — never docker build, never a host docker socket. The deployed contract is the resolved immutable digest (repo@sha256:…), not a tag: GHCR has no server-side tag immutability, so R3’s DaemonSet pins the digest.

Test cycle: a tagged run publishes a manifest whose digest is recorded in the run output; a second build of the same input yields the same digest.

R3 — DaemonSet + RBAC manifests as config-as-data

Author the object contract from §Kubernetes object contract as plain manifests, parameterized where §Global Constraints says the operator supplies values. Test cycle: rendered objects assert the full §Privilege shape: no privileged, drop: ["ALL"], runAsNonRoot, hostUsers: false, procMount: Unmasked, appArmorProfile: Unconfined, the device resource, the Localhost profile reference, and exactly two hostPath mounts. Requests and limits are present and sized per the capacity model; maxUnavailable is bounded.

R4 — /dev/kvm device-plugin delivery

Document and encode the device-plugin requirement and resource-name parameterization. The device plugin supplies the device’s cgroup allowance; the node’s /dev/kvm mode 0666 supplies DAC access. The host kvm gid grant is dropped because user namespaces make it ineffective. Test cycle: rendered pod spec requests the device resource and sets hostUsers: false and procMount: Unmasked; assertions fail if any is absent.

R5 — entrypoint fix for --backend microvm

The image entrypoint must pass agent-image semantics correctly under --backend microvm. Test cycle: the container started with --backend microvm resolves its agent image from the documented flag/env precedence, asserted without launching a VM.

R6 — ledger row

Mint DL-358 recording the containerize ruling and the zero-privilege constraint.

R7 — privilege-shape spike and real-node rerun

R7 ran on a nested-KVM VM node, which does not meet the real-hardware bar. A real-node rerun must verify the spec. Findings land in spike-findings.md beside this record. The rerun must answer:

1. Does a pod with the §Privilege shape boot cloud-hypervisor + virtiofsd + passt end to end?
2. Does the hostPath char-device route actually fail open() without device-plugin injection? (If it succeeds, the option set reopens.)
3. Does the pod open /dev/kvm with node mode 0666, without the host gid?
4. Does RuntimeDefault seccomp fail with passt reporting Couldn't create user namespace?
5. Does guest RAM appear in the pod’s memory cgroup as §Pod resources claims?
6. Does a container restart leave zero stranded VMM/virtiofsd/passt processes?

Task ordering

R7’s real-node rerun gates R3 and R4. R1 gates R2. R5 is independent. R6 lands with the freeze.

Tasks

Task	Deliverable	Depends on
R1	runner-image/ Dockerfile on a minimal hardened base	—
R2	publish lane by digest	R1
R3	DaemonSet + RBAC manifests + render tests	R7
R4	device-plugin resource wiring	R7
R5	entrypoint agent-image fix under --backend microvm	—
R6	DL-358 ledger row	R7
R7	real-node rerun of privilege-shape spike + spike-findings.md	R1

Open Questions

OQ-1 [non-load-bearing] — which grants does confinement make necessary?

The nested-KVM VM run found the required pod and node settings now recorded in §Privilege shape. It found that no capability or privileged mode was needed.

OQ-2 [non-load-bearing] — minimize the required seccomp profile

RuntimeDefault failed with passt reporting Couldn't create user namespace, so the Localhost profile and its on-node staging requirement remain required. R3 must minimize the profile’s allow list.

OQ-3 [non-load-bearing] — device-plugin implementation pick

The contract names a resource, not a plugin. Which plugin an operator runs is an implementation choice; the community options are thinly maintained and want a digest pin.

A plugin that sets the device mode per pod would avoid a world-rw node device. No plugin was tested for this. Adopting one is a later narrowing of the node requirement, not a change to the pod spec.

OQ-4 [non-load-bearing] — runner uid and dropped kvm gid grant

The Runner uid is fixed at implementation and single-sourced between the image and manifests. The host kvm gid grant was dropped: with hostUsers: false, the device appears as 65534:65534 in the pod and the host gid is outside its id mapping. Node mode 0666 replaces the gid grant.

OQ-5 [non-load-bearing] — session-volume host path and filesystem

The path is operator-supplied; the filesystem must support project quotas for D7. Default path chosen at implementation.

OQ-6 [non-load-bearing] — custom AppArmor profile

appArmorProfile: Unconfined is required for the current pod because the runtime default profile denies the mount passt performs inside its user namespace. R3 may follow up with a custom AppArmor profile instead. This is optional and does not change the required pod shape.

Resolved decisions

• Containerize the Runner as the Kubernetes delivery unit, rather than a host systemd service. The measured pod spec needs no capability (§Privilege shape).
• Never privileged: true — the microVM isolation boundary is the reason the Runner exists, and a privileged Runner re-opens the host path (§Alternatives considered).
• /dev/kvm by device plugin, not hostPath char device, not privileged mode (§Privilege shape).
• Guest RAM is pod RAM, so DaemonSet requests must be sized to node session capacity, not to the Runner process (§Pod resources).
• Container restart implies full session teardown — forced by the pid namespace, not chosen (§Privilege shape).
• The image carries the whole KVM userland and guest assets, keeping hostPath to exactly two mounts (§Where the KVM userland and guest assets live).
• Use a user-namespaced pod with hostUsers: false and procMount: Unmasked; provision Kubernetes user-namespace support, the node sysctl where enabled by its image, and /dev/kvm mode 0666. The device plugin supplies the cgroup allowance. The supplementalGroups grant is dropped (§Privilege shape).