Development

Prerequisites

Tool	Used for	Tested with
Go	building and testing	see go.mod
Docker	runs the kind nodes	any recent Docker Engine or Docker Desktop
kind	local Kubernetes cluster	v0.33.0
kubectl	talking to the cluster	a version within one minor of the cluster

Local cluster

make kind-up      # create the cluster "vesper-dev" (does nothing if it exists)
kubectl --context kind-vesper-dev get nodes
make kind-down    # delete it (does nothing if it does not exist)

Use another name with make kind-up KIND_CLUSTER=other. The kubectl context is always kind-<name>.

Kubernetes version

The Kubernetes version is set by the node image in hack/kind-config.yaml. It is pinned by tag and by digest so that every contributor and CI test the same version. To upgrade, take the image listed in the release notes of the new kind version and update the file and the table above.

Operator layout

The operator (the “manager”) was scaffolded with Kubebuilder v4 (see ADR-0001). The scaffold that Kubebuilder writes by itself is kept as generated; files that duplicate the repository’s own conventions (README, Makefile, CI workflows, lint config) were not taken.

Path	What it is
cmd/main.go	Entry point of the manager: builds the controller-runtime manager (API client, cache, health probes on :8081, metrics, optional leader election) and runs the controllers. Kubebuilder edits it when an API is added, at the +kubebuilder:scaffold: markers, so it stays at this path.
api/v1alpha1/	The PreviewEnvironment API as Go types. The CRD is generated from them. zz_generated.deepcopy.go is generated: do not edit it.
cmd/vesperctl/	The CLI.
PROJECT	Kubebuilder’s record of what was scaffolded (domain, module, APIs). Its commands read it; do not edit it by hand.
config/	Kustomize manifests to deploy the operator: manager/ (its Deployment), rbac/ (ServiceAccount and roles), default/ (the overlay make deploy applies, with the metrics Service), network-policy/ and prometheus/ (optional, off by default), crd/ (the generated CRD) and samples/ (an example PreviewEnvironment).
config/rbac/role.yaml	Generated by make manifests from the +kubebuilder:rbac markers. Do not edit it by hand.
hack/boilerplate.go.txt	License header of every Go file; make generate puts it on generated files, and make check-license (run by CI) checks them all.
Dockerfile	Builds the manager into a distroless image that runs as a non-root user.

Generated files are rebuilt from the Go code:

make manifests   # CRD and RBAC YAML from the Go types and +kubebuilder markers
make generate    # DeepCopy methods of the API types

make build runs both before compiling, and installs controller-gen (and kustomize for the deploy targets) into bin/ at a pinned version the first time.

Run the operator

make build                  # bin/vesperctl and bin/manager
make run                    # run the manager on your machine, against the current kubectl context
make docker-build IMG=vesper-manager:dev

The version both binaries print (vesperctl version, and the first log line of the manager, Starting Vesper) comes from git describe; set it with VERSION=v0.1.0. The image builds for another architecture without emulation: docker buildx build --platform linux/amd64,linux/arm64 . compiles on your machine and cross-compiles. Its base images are pinned by digest, and Dependabot proposes their updates.

make run uses whatever cluster your current kubectl context points to: check it with kubectl config current-context first. It reconciles every PreviewEnvironment of that cluster, and writes their status. Stop the manager with Ctrl-C. Its health probes answer on curl localhost:8081/healthz.

What the operator does

The controller is in internal/controller. For each PreviewEnvironment it:

1. creates a dedicated namespace named vesper-<namespace>-<name>-<hash> (NamespaceName in naming.go): truncated to 63 characters, with an 8-character hash of <namespace>/<name> so that two previews never share a namespace. The same preview always gets the same name. Once it exists, its name is written to status.namespace (vesperctl get, or kubectl get pe -o wide), even if a later step fails;
2. labels it with the preview labels (see How Vesper tracks preview objects) and with pod-security.kubernetes.io/enforce=restricted, so that the API server rejects privileged or root pods in it. Labels changed by hand are put back;
3. bounds its resources with a LimitRange and a ResourceQuota named preview (quota.go, see Resource limits), and isolates it on the network with a NetworkPolicy named preview (networkpolicy.go, see Network isolation), before any pod exists;
4. creates or updates a Deployment named app in it (deployment.go): the image, the port (named http), the environment variables and the replicas of the spec. Its pods run with the restricted Pod Security settings: runAsNonRoot, no privilege escalation, all capabilities dropped, the RuntimeDefault seccomp profile and no ServiceAccount token. The image must run as a non-root user: an image that runs as root, such as the official nginx, stays in CreateContainerConfigError (nginxinc/nginx-unprivileged works);
5. creates or updates a ClusterIP Service named app in front of the pods (service.go): port 80, which targets the container port by its name (http), so the Service stays the same when spec.port changes. Its address, http://app.<preview namespace>.svc.cluster.local, is written to status.internalURL (kubectl get pe -o wide). It is reachable from inside the cluster only;
6. with a preview domain, creates or updates an Ingress named app (ingress.go) that routes the host of the preview to the Service, and writes its URL to status.url (see Public address);
7. writes the status: the phase, and the conditions Ready, Available and Expired, and records an event when the preview changes state (see Status, conditions and events).

The Deployment is written with controllerutil.CreateOrUpdate, which sets only the fields Vesper owns and keeps the defaults the API server adds, so a reconciliation that changes nothing writes nothing.

If a namespace of that name exists without the labels of that preview, the operator does not touch it: the phase becomes Failed with the reason NamespaceConflict. A namespace, or an object in it, deleted or changed by hand is restored. To reach a preview from your machine without a public address:

kubectl port-forward -n <preview namespace> svc/app 8080:80

Status, conditions and events

kubectl describe pe <name> shows what the operator knows of a preview. The phase sums it up in one word: Pending, Ready, Failed or Expired. Failed is not final: when the cause goes away, the preview comes back. The conditions give the details; each has a status, a reason and a message:

Condition	Status and reason	Meaning
Available	True DeploymentAvailable	every replica runs the requested image
	False Provisioning	the pods are starting (“1/2 replicas available”)
	False PodsRejected	the API server rejects the pods, for example because the quota is full; the message gives its error
	False RolloutStuck	no progress for the progress deadline of the Deployment, 10 minutes: often an image that cannot be pulled, or a container that keeps crashing. Look at the pods of the preview namespace
	False NamespaceConflict	the namespace of the preview exists and is not Vesper’s
Expired	False TTLNotReached	the message gives the expiry
	True TTLPassed	the ttl has passed: the preview is being deleted
Ready	the reason of Available	the summary: True when the application runs
	False ReconcileError	the operator failed to set up the preview, for example because an object it needs exists and is not Vesper’s; the message gives the error, and the operator retries
	False Expired	the ttl has passed

The reasons are constants of the API (api/v1alpha1). Conflicts between two writers are not reported as ReconcileError: the operator retries, and they resolve.

The events, attached to the preview, tell its history. The operator records one when the Ready condition changes, with its reason (Warning when the phase is Failed), and NamespaceCreated and Deleting when it creates and deletes the namespace. Events are written through the events.k8s.io API; the cluster keeps them for an hour by default.

A wrong image or a crashing container is reported after 10 minutes only, by RolloutStuck: reading the pods to report it at once is #107.

Resource limits

A preview runs code from a pull request, so it must not be able to use the whole cluster. Two objects in its namespace bound it, both named preview. The API server applies them when a pod is created (they are admission checks), so the operator creates them before the Deployment:

• A LimitRange gives each container that sets no requests or limits default ones. Vesper sets none on the container of a preview, so its pods get these values: the Deployment stays as Vesper wrote it.
• A ResourceQuota bounds the whole namespace: the sum of the limits of its pods, and their number. It also forbids NodePort and LoadBalancer Services, so that a preview is reached through its Ingress only. A quota on limits rejects any pod without limits, which is why the LimitRange comes first.

Bound	Default	Operator flag
CPU and memory reserved per container (requests)	100m, 128Mi	--default-cpu-request, --default-memory-request
Most CPU and memory per container (limits)	500m, 512Mi	--default-cpu-limit, --default-memory-limit
Sum of the limits of the pods of a preview	4 CPU, 4Gi	--quota-cpu, --quota-memory
Pods of a preview	10	--quota-pods

The quota fits the largest preview, 5 replicas, during a rolling update: the Deployment then runs up to 7 pods (maxSurge of 25%, rounded up), that is 3.5 CPU and 3.5Gi of limits. If you raise the limits per container, raise the quota too. The operator refuses to start if a value is not a valid quantity, is not positive, if a request is more than its limit, or a limit more than its quota.

When the quota is reached, the API server rejects the new pods: the preview becomes Failed with the reason PodsRejected and the message of the quota, and goes back to Ready once the pods fit. Pods that run when the operator is upgraded keep no limits until their next rollout: a quota does not evict pods it already admitted.

Network isolation

Each preview namespace has a NetworkPolicy named preview that denies all traffic to and from its pods, except (ADR-0005):

Direction	Allowed
In	The port http of the application, from the pods of the preview and from any namespace that is not a preview (without app.kubernetes.io/managed-by=vesper), such as the one of the ingress controller
Out	The pods of the preview, DNS (port 53), and the public internet: not the private (10/8, 172.16/12, 192.168/16, fc00::/7), link-local (169.254/16, fe80::/10) or shared (100.64/10) ranges, nor the public addresses of the nodes and of the API server

So a preview cannot call another preview, nor anything inside the cluster: other applications, the API server, the metadata service of the cloud. It can call external APIs. A preview that needs a service inside the cluster does not work today.

When the nodes have public addresses, as on a server with k3s, those addresses lead to the cluster too: the kubelet and NodePort Services listen on them, and a connection to the kubernetes Service reaches the API server through one. The operator reads the addresses of the nodes and the endpoints of the API server, adds the public ones to the denied blocks (/32, /128), and updates the policy of every preview when they change. As a consequence, a preview cannot call a public URL served by the cluster itself, such as another preview or an application behind the same ingress controller.

The CNI enforces the policy, not the API server. kindnet (the CNI of recent kind releases), k3s, Calico and Cilium do; with a CNI that does not, such as flannel alone, the policy is accepted and ignored, and previews are not isolated. A namespace stays a soft boundary: do not run previews of untrusted code next to sensitive workloads on the strength of this policy alone.

make test-e2e checks the isolation on kind: from the namespace of the test, and from the preview itself, the preview answers; from a namespace labelled like a preview it does not, and the preview cannot reach the API server.

Operator permissions

The operator runs with the ServiceAccount vesper-controller-manager. Its ClusterRole, config/rbac/role.yaml, is generated from the +kubebuilder:rbac markers in internal/controller and lists only what the code uses:

Resource	Verbs	Why
previewenvironments	get, list, watch, patch, delete	read the previews, add and remove the finalizer, delete an expired preview
previewenvironments/status	patch	write the status
namespaces	get, list, watch, create, patch, delete	create the namespace of a preview, put its labels back, delete it
deployments, services, limitranges, resourcequotas, networkpolicies	get, list, watch, create, update	create the objects of a preview and restore them
ingresses	get, list, watch, create, update, delete	the same, and delete the Ingress when the preview domain is removed
nodes	get, list, watch	read the addresses of the nodes, to deny them to previews
endpointslices	get, list, watch	read the addresses of the API server, to deny them to previews; the cache holds only the EndpointSlices of the Service kubernetes in default

The Role leader-election-role (config/rbac/leader_election_role.yaml), in vesper-system only, allows get, create and update on leases, and create and patch on events: when several replicas run, the one that holds the Lease works and the others wait. The operator cannot read Secrets or ConfigMaps, nor create pods.

These rights are cluster-wide: RBAC cannot limit them to the namespaces of previews. The code protects the rest of the cluster instead: it modifies or deletes an object only if it carries the labels of the preview, and deletes with a precondition on its UID. The namespace vesper-system enforces the restricted Pod Security Standard too.

Public address

A preview gets a public address when the operator runs with a preview domain:

go run ./cmd --preview-domain=preview.example.com [--preview-tls] [--ingress-class=traefik]

In the cluster, add the flags to the args of the manager Deployment (config/manager/manager.yaml, or a kustomize patch as in test/e2e/kustomization.yaml). Three pieces work together:

• The Ingress. For each preview, the operator creates an Ingress named app that routes <name>-<namespace>-<hash>.<domain> to the Service app, and writes http://<host> to status.url (https:// with --preview-tls, see HTTPS). The name and the namespace share one DNS label (HostName in naming.go), with the hash of the preview namespace, so that the name stays unique and fits in 63 characters.
• An ingress controller. An Ingress only describes the route: the controller (Traefik, included in k3s; HAProxy; Contour…) serves it. --ingress-class selects it; without it the default class of the cluster is used, and the class the cluster fills in is kept.
• A wildcard DNS record. *.preview.example.com must point to the ingress controller, so that every preview is reachable without a DNS change. A wildcard covers one label, which is why the host has a single one.

Without --preview-domain, no Ingress is created and status.url stays empty; an Ingress Vesper created before is deleted. Both flags are validated when the operator starts. For a local try, localtest.me and its subdomains resolve to 127.0.0.1: with an ingress controller in kind and --preview-domain=preview.localtest.me, the URL of a preview opens through a port-forward to the controller.

Vesper uses an Ingress rather than the Gateway API, which may come later (#99). The NetworkPolicy of a preview lets the ingress controller in (see Network isolation).

HTTPS

With --preview-tls (it requires --preview-domain), each Ingress gets a TLS section for its host that names no Secret, and status.url starts with https://. The ingress controller serves the host with its default certificate, which must be a wildcard for *.<preview domain>: it is configured once, and no private key is copied into preview namespaces (ADR-0006). Without the flag, nothing changes; when it is removed, the TLS sections go at the next reconciliation.

With Traefik, two settings, both on by default in its Helm chart and in k3s except the certificate:

• the default certificate, in the default TLS store: with the Helm chart, tlsStore.default.defaultCertificate.secretName: <Secret>, a Secret of type kubernetes.io/tls in the namespace of Traefik;
• TLS on the websecure entrypoint (--entrypoints.websecure.http.tls=true; the chart sets it). Traefik enables TLS on a route from its entrypoint: the TLS section of an Ingress only names certificates. Without this setting, Traefik presents the certificate but answers 404.

The wildcard certificate itself comes from cert-manager with a DNS-01 challenge, or from any other source; the server guide (#96) describes cert-manager. Redirecting http:// to https:// is a setting of the controller too (with the Helm chart of Traefik, ports.web.redirections.entryPoint).

This was checked on kind with Traefik v3.5 and a self-signed wildcard as its default certificate: the page of the preview answers over HTTPS with that certificate, and the preview namespace holds no Secret. The e2e test checks the TLS section and the URL; kind has no ingress controller there.

Preview lifetime (TTL)

A preview expires spec.ttl after its creation: status.expiresAt is metadata.creationTimestamp + spec.ttl, recomputed at every reconciliation (ttl.go, ADR-0004). The Expires column of kubectl get pe and vesperctl get show it.

• The controller does not poll: each reconciliation returns RequeueAfter: expiresAt - now, so controller-runtime reconciles the preview again at its expiry.

• When the ttl has passed, the controller sets the phase to Expired, with the Ready condition False and the reason Expired, then deletes the preview; the finalizer deletes its namespace (see below). An expired preview creates nothing.

• To extend a preview, raise its ttl; the total lifetime stays within 168h:

  vesperctl extend <name> -n <namespace> --by 24h

• If the operator is stopped at the expiry, the preview expires when it starts again: late, never early.

To watch an expiry, create a preview with ttl: 1m. The tests do not wait: the reconciler has a Clock field, and the unit tests set a fake clock (k8s.io/utils/clock/testing) and move it.

Delete a preview

Every preview carries the finalizer vesper.eventide.dev/cleanup, added before anything is created for it (finalizer.go). A finalizer turns a deletion into two steps:

1. vesperctl delete <name> (or kubectl delete pe <name>) only sets the deletionTimestamp of the preview: it is being deleted, but still there.
2. The operator deletes the preview namespace, which deletes the Deployment, the pods and the Service in it. Once the namespace is gone, which takes a few seconds, the operator removes its finalizer, and the API server removes the preview.

So a preview disappears only when everything it created is gone, and a new preview of the same name never meets the old namespace. The operator only deletes a namespace that carries the labels of the preview: after a NamespaceConflict, the foreign namespace is kept. If the deletion of the namespace fails, the finalizer stays and the operator retries.

If the operator is not running, nothing removes the finalizer: the preview stays in deletion (vesperctl delete returns, kubectl delete waits), and so does make uninstall, which deletes every preview with the CRD. Start the operator again (make run) and the deletion completes. To give up the cleanup, remove the finalizer by hand, then delete the preview namespace yourself:

kubectl patch pe <name> -n <namespace> --type=merge -p '{"metadata":{"finalizers":null}}'
kubectl delete namespace -l vesper.eventide.dev/preview-namespace=<namespace>,vesper.eventide.dev/preview-name=<name>

How Vesper tracks preview objects

The reasons for this design, and the alternatives, are in ADR-0002.

Every object Vesper creates for a preview (the namespace, the Deployment, its pods, the Service and the Ingress) carries the same three labels, set by setPreviewLabels in labels.go:

Label	Value
app.kubernetes.io/managed-by	vesper
vesper.eventide.dev/preview-namespace	namespace of the PreviewEnvironment
vesper.eventide.dev/preview-name	name of the PreviewEnvironment

Kubernetes usually ties an object to its owner with an owner reference, and its garbage collector deletes the object when the owner is deleted. Vesper cannot use one:

• a namespaced object can only be owned by an object in the same namespace. The preview lives in your namespace (team-a), its Deployment and Service in the preview namespace (vesper-team-a-pr-42-<hash>). Kubernetes treats such a reference as invalid, and the garbage collector may delete the object, thinking its owner is gone;
• the preview namespace is cluster-scoped, and a cluster-scoped object cannot have a namespaced owner.

The labels replace the owner reference:

• Ownership. Before it modifies an object, the operator checks that it carries the labels of this preview (ownedBy). It never takes over an object someone else created: it reports an error, or NamespaceConflict for a namespace.
• Watches. A change to a labeled object reconciles its preview, found from the labels (previewOfObject).
• Selection. kubectl get ns,deploy,svc -A -l app.kubernetes.io/managed-by=vesper lists everything Vesper created.
• Cleanup. The preview namespace is the cleanup boundary: deleting it deletes everything in it. The finalizer of the preview deletes it when the preview is deleted (see Delete a preview).

Change the API

The Go types in api/v1alpha1/previewenvironment_types.go are the source of truth for the PreviewEnvironment API:

• the JSON tag of a field is its name in YAML;
• its comment becomes the documentation shown by kubectl explain previewenvironment.spec;
• +kubebuilder:validation: markers, including CEL rules (XValidation), and +kubebuilder:default markers become the validation and defaults that the API server applies.

After a change, regenerate and commit the generated files:

make manifests generate   # CRD, RBAC role (from the +kubebuilder:rbac markers) and DeepCopy methods
make docs-reference       # docs/crd-reference.md, from the same types and comments
make verify-generated     # what CI runs: fails if a generated file is not committed

CI runs make verify-generated on every pull request: it regenerates the files with the controller-gen version pinned in the Makefile and fails if they differ from the committed ones, so a forgotten make manifests generate cannot reach main. The year in the license header of generated files is fixed (YEAR in the Makefile) so that the files do not change every January.

To try the validation by hand on the kind cluster:

make install                                  # install or update the CRD
kubectl --context kind-vesper-dev create --dry-run=server -f config/samples/vesper_v1alpha1_previewenvironment.yaml

--dry-run=server runs the API server’s validation and defaulting without storing the object. make uninstall removes the CRD and every PreviewEnvironment with it.

Walkthrough from source

This walkthrough runs Vesper on a local kind cluster and deploys one preview. You need Go, Docker, kind and kubectl; see Prerequisites.

Vesper has two parts, and both must be in place before a preview does anything:

• The API (the CRD): it teaches the cluster the PreviewEnvironment type, so it can store and validate previews. On its own, a stored preview is only a record: nothing runs.
• The operator: the program that watches previews and creates their namespace, Deployment and Service, then writes their status. Here you run it on your machine with make run; to install it in the cluster from a release, see the quickstart.

Use vesperctl to create, read, extend and delete previews (see the cheat sheet), or kubectl, as for any Kubernetes object. The steps below use kubectl and a manifest.

1. Build the binaries. bin/vesperctl is the CLI, bin/manager the operator.

make build
./bin/vesperctl version

2. Create a local cluster and select it. Every kubectl command, and the operator, act on the cluster of the current context.

make kind-up
kubectl config use-context kind-vesper-dev

3. Install the API. The cluster now knows the PreviewEnvironment type (short name pe).

make install
kubectl get crd previewenvironments.vesper.eventide.dev

4. Run the operator, in a second terminal. It stays in the foreground and logs what it does; leave it running. Stop it with Ctrl-C.

make run

5. Create a preview from the sample: an unprivileged nginx image, kept for 24 hours.

kubectl apply -f config/samples/vesper_v1alpha1_previewenvironment.yaml

6. Watch it become Ready. The phase goes from Pending to Ready once its pod runs, usually within a minute while the image downloads. Stop kubectl get -w with Ctrl-C.

kubectl get pe -w
./bin/vesperctl list -A
./bin/vesperctl get previewenvironment-sample

7. Look inside it. Each preview has a namespace of its own, shown on the Runs in: line of vesperctl get. It holds a Deployment named app, its pod, and a Service named app.

NS=$(kubectl get pe previewenvironment-sample -o jsonpath='{.status.namespace}')
kubectl get deploy,pods,svc -n "$NS"

8. Open it. This local cluster has no ingress controller and the operator runs without a preview domain, so the preview has no public address, only an address inside the cluster (kubectl get pe -o wide shows it, in the INTERNAL URL column). Forward a local port to its Service, then open http://localhost:8080 in a browser. To give previews a public address, see Public address.

kubectl port-forward -n "$NS" svc/app 8080:80

9. Clean up. Delete the preview while the operator still runs: it deletes the preview namespace first, and kubectl delete returns once both are gone. Then stop the operator (Ctrl-C), and remove the API or the whole cluster.

kubectl delete -f config/samples/vesper_v1alpha1_previewenvironment.yaml
kubectl get ns "$NS"   # NotFound: the namespace is gone too
make uninstall         # removes the CRD (delete the previews first, see below)
make kind-down         # or delete the whole cluster

A deleted preview waits for the operator to clean it up. If the operator is stopped, kubectl delete and make uninstall wait too: see Delete a preview.

What the operator does, and why, is described in What the operator does.

Tests

make test               # unit and controller (envtest) tests, no cluster needed
make cover              # after make test: coverage of the lifecycle code
make test-integration   # vesperctl against the kind cluster (created if missing)
make test-e2e           # the operator image in a fresh kind cluster (needs Docker)

Controller tests (envtest)

The controller has two kinds of tests in internal/controller:

• Unit tests call Reconcile directly against the fake client of controller-runtime, an in-memory store. They are fast and cover the edge cases, but the fake client does not validate the CRD schema, add server defaults or run the watches.
• envtest tests (envtest_test.go) start a real kube-apiserver and etcd with the generated CRD, and run the controller in a real manager, as in production. A test creates a preview and waits for the result with eventually (retries every 100 ms, fails after 10 s), never with a fixed sleep. TestMain in suite_test.go starts envtest once for the package.

envtest runs no other controller. There is no Deployment controller, so no pod ever runs: a test writes the Deployment status itself to make a preview Ready. There is no namespace controller, so a deleted namespace stays Terminating: each test creates its own namespace (team-<random>) and never deletes it.

The controller reads the time from Reconciler.Clock. The unit tests give it a fake clock; the envtest controller gets envClock, the real time plus a shift that a test sets with shiftTime(t, 25*time.Hour) and that goes back to zero at the end of the test. TestEnvtestExtendAndShortenTTL moves past the expiry of a preview this way, without waiting: raising the ttl saves it, lowering it expires it at once. Shifting the clock alone does not wake the controller up, which comes back at the expiry in real time: change the spec. The tests of the package do not run in parallel, so a shift is seen by one test only.

make test downloads setup-envtest and the kube-apiserver and etcd binaries of Kubernetes ENVTEST_K8S_VERSION into bin/, and points the tests to them with KUBEBUILDER_ASSETS. A plain go test ./... skips the envtest tests; in CI (CI set), they fail instead of being skipped. To run them from your editor or with other flags:

make setup-envtest
export KUBEBUILDER_ASSETS="$(bin/setup-envtest use 1.37.0 --bin-dir "$PWD/bin" -p path)"
go test -race -run Envtest -v ./internal/controller/

Coverage

make test writes the coverage of every package to cover.out (ignored by git). make cover prints, from it, the coverage of each function of the lifecycle of a preview (ttl.go, finalizer.go, status.go) and the total; set COVER_FILES to a regular expression for other files. For a page that shows every line, run go tool cover -html=cover.out.

CI writes the same report to the summary page of each run (the test job). It is a report, not a gate: no minimum fails the build.

End-to-end test

make test-e2e (hack/test-e2e.sh) checks the operator as it runs in production, in a container:

1. it builds vesperctl, and the operator image (vesper-manager:e2e) from the Dockerfile;
2. it creates a fresh kind cluster, vesper-e2e, and loads the image into it;
3. it deploys the CRD, the RBAC rules and the operator with config/default, through the overlay test/e2e/kustomization.yaml, which only sets the image (make deploy would edit config/manager/kustomization.yaml);
4. it runs test/e2e (build tag e2e):
   • through the API: a preview of an unprivileged nginx image must become Ready, with a ready pod admitted by the restricted profile and a ready endpoint behind its Service, and its namespace must go with it; and the network isolation of previews;
   • through vesperctl, as a user would: create, get until Ready, extend (the operator moves expiresAt) and delete --yes; and a preview created with --ttl 1m, which the operator deletes with its namespace after a minute. These two run in parallel;
5. it deletes the cluster, even when the test fails. On failure it first prints the operator logs, the previews and the recent events, and saves the logs of the cluster to _artifacts/e2e (E2E_ARTIFACTS_DIR): the operator logs, the previews, namespaces, pods and events, and kind export logs (node, kubelet and container logs).

The cluster has a kubeconfig of its own, in a temporary file: your kubeconfig and current context are not changed, and the operator you run with make run does not see the test previews. The script stops if a vesper-e2e cluster already exists. Set E2E_KEEP_CLUSTER=1 to keep the cluster after the test; the script prints the KUBECONFIG to use. It takes about five minutes, half to build the images and create the cluster, half for the tests; the preview that expires waits its minute while the other runs. CI runs it on every pull request; when it fails, the logs are attached to the run as the artifact e2e-logs (at the bottom of the run summary page, kept 7 days). They are public, like the repository: the e2e cluster must never hold a secret.

E2E_CNI chooses the network plugin of the cluster: kindnet (the default), calico or cilium, at the versions pinned in hack/test-e2e.sh; cilium needs helm. E2E_RUN runs only the tests whose name matches it. To check the network isolation with Cilium:

E2E_CNI=cilium E2E_RUN=TestPreviewNetworkIsolation make test-e2e

The workflow .github/workflows/isolation.yml runs this check with Calico and Cilium on pull requests that change the isolation, on main, every Monday, and on demand (Actions → Isolation → Run workflow), before a release.

Integration tests

The integration tests in test/integration sit behind the integration build tag, so go test ./... skips them. They build the vesperctl binary and run list and get against a real API server, with the generated CRD of config/crd/bases. They create previews, then write their status through the status subresource, as the operator would.

Each run creates a namespace named vesper-it-<random> and a few previews, and deletes them at the end, even when a test fails. Everything it creates carries the label app.kubernetes.io/managed-by=vesper-integration-tests.

• If no PreviewEnvironment CRD is installed, the tests install it, with that label, and delete it at the end. Deleting a CRD deletes every PreviewEnvironment, so the tests only delete a CRD that carries their label.
• If a CRD without the label is installed (for example with make install), it is left in place, with its previews, and the “not installed” cases are skipped.
• A labeled CRD left by an interrupted run is removed before the tests start. Never add that label to a CRD yourself.
• Stop the operator (make run) before you run them on the same cluster: it would reconcile the test previews, overwrite the status the tests set, and create namespaces for them that stay after the run (until #39).
• The tests only run against a context named kind-...; they refuse any other cluster. VESPER_IT_CONTEXT selects the context, and make test-integration KIND_CLUSTER=other sets it for you.

CI runs them on every pull request, in the integration job.

Release builds

GoReleaser builds the release assets of vesperctl from .goreleaser.yaml: an archive for Linux, macOS and Windows on amd64 and arm64 (.zip for Windows, .tar.gz for the others), each with the binary, LICENSE and README.md, and checksums.txt with the SHA-256 of every archive. The release workflow publishes them: see Releasing.

make release-check      # validate .goreleaser.yaml
make release-snapshot   # build everything into dist/release/, publish nothing

The first run installs GoReleaser into bin/, at the version pinned in the Makefile (GORELEASER_VERSION); compiling it takes a few minutes. A snapshot needs no tag: its version is the next patch after the last tag, with the commit, such as v0.1.1-snapshot+abc1234, and vesperctl version prints it.

To check an archive you downloaded:

shasum -a 256 --check --ignore-missing checksums.txt   # sha256sum on Linux

The binaries are static (CGO_ENABLED=0) and reproducible: the same commit, built with the same Go version, gives the same archives byte for byte, whoever builds them. -trimpath removes the paths of the build machine, and the files in the archives get the date of the commit and the owner root. CI runs a snapshot on every pull request, in the release-snapshot job, and checks the checksums and the version the Linux binary prints.

vesper-github

vesper-github is the GitHub integration of v0.2 (ADR-0007): a second program, cmd/vesper-github, built into the same image as the operator, and deployed next to it by config/github. It is the only part of Vesper that faces the Internet. It receives and authenticates the webhooks of the GitHub App, and turns the pull requests of the repositories that a PreviewSource allows into PreviewEnvironment objects (internal/github).

• A queue of pull requests. A webhook only says which pull request changed; a resync, at start and every 10 minutes (--resync-interval), adds the open pull requests and those of the existing previews of each source. The reconciler then reads the pull request from the GitHub API, never trusting the payload for its state, so that late, repeated or lost deliveries all end the same way.

• The decision (decide.go): no preview for a closed pull request, a draft, a fork, no-preview (trigger All) or no preview (trigger Label); otherwise the preview of the source, once the buildWorkflow succeeded on the head commit; a new commit keeps the old image until its own is built, then resets the ttl. A PreviewEnvironment of the same name that the integration did not create is never touched.

• On the pull request (reporting.go): a GitHub deployment per commit, in an environment named after the preview (acme-app-pr-42), whose status follows the phase (in_progress, success with the URL, failure with the reason, inactive when expired or stopped), and one comment, edited in place, found again by a hidden marker. The preview remembers what it showed in its annotations, so GitHub is written to only on a change. A pull request that never had a preview gets nothing. An expired preview is not created again on the same commit: the status of its deployment says Expired, and a new commit brings it back.

• The GitHub App authenticates with a JWT signed by its private key, and uses installation tokens of one hour, renewed five minutes before they expire. Its ID and key are files of the Secret vesper-github (app-id, private-key), next to webhook-secret. It needs these permissions, and no other:

  Permission	Access
  Metadata	Read-only
  Pull requests	Read and write
  Deployments	Read and write
  Actions	Read-only

  and these events: Pull request and Workflow run.

• The webhook endpoint, POST /webhook on port 8080, exposed through the ingress controller on a host of its own (config/github/ingress.yaml). A delivery must carry a valid X-Hub-Signature-256, an HMAC SHA-256 of its body with the webhook secret, compared in constant time: 401 otherwise. Bodies over 5 MiB get 413. Signed ping, pull_request and workflow_run deliveries are accepted; other signed events are ignored.

• The secret is a file of the Secret vesper-github, key webhook-secret, mounted in the pod by the kubelet: vesper-github has no RBAC right on Secrets, and reads the file at each delivery, so a new secret applies at once. The pod is not ready until it can read it.

• Its rights, in config/github/role.yaml: previews and sources, and what the protected metrics need. internal/github/rbac_test.go fails if a right is added.

• Metrics: vesper_github_webhook_deliveries_total, by result, on port 8443, for authorized clients only.

From a release, install-github.yaml holds the same manifests with the image of the release, pinned by digest. Set the host of the webhook with a kustomize overlay, so that an upgrade keeps it. Save this as vesper-github/kustomization.yaml, with your host:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/eventide-oss/vesper/releases/download/v0.2.0-rc.1/install-github.yaml
patches:
- target:
    kind: Ingress
    name: vesper-github-webhook
  patch: |-
    - op: replace
      path: /spec/rules/0/host
      value: hooks.preview.example.com
    - op: replace
      path: /spec/tls/0/hosts/0
      value: hooks.preview.example.com

kubectl apply -k vesper-github
kubectl -n vesper-system rollout status deploy/vesper-github

The Secret vesper-github must exist first, with the keys app-id, private-key and webhook-secret of the GitHub App.

Try it on the local cluster, with a secret of your own and a signed ping:

# The image holds the operator and vesper-github.
make docker-build IMG=vesper-manager:dev
kind load docker-image vesper-manager:dev --name vesper-dev

# The Secret of the GitHub App. To try the endpoint only, a random webhook
# secret and a throwaway key do: vesper-github calls GitHub only for pull
# requests of an allowed repository.
kubectl create namespace vesper-system --dry-run=client -o yaml | kubectl apply -f -
secret=$(openssl rand -hex 32)
openssl genrsa -out /tmp/throwaway-key.pem 2048
kubectl -n vesper-system create secret generic vesper-github --from-literal=webhook-secret="$secret" \
  --from-literal=app-id=1 --from-file=private-key=/tmp/throwaway-key.pem
rm /tmp/throwaway-key.pem

# vesper-github, with the image just built.
bin/kustomize build config/github | sed "s|image: controller:latest|image: vesper-manager:dev|" | kubectl apply -f -
kubectl -n vesper-system rollout status deploy/vesper-github

# A ping signed as GitHub signs it: 200. Without the signature: 401.
kubectl -n vesper-system port-forward svc/vesper-github-webhook 18080:80 &
body='{"zen":"Keep it logically awesome."}'
sig="sha256=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$secret" | awk '{print $NF}')"
curl -i -X POST -H 'Content-Type: application/json' -H 'X-GitHub-Event: ping' \
  -H "X-Hub-Signature-256: $sig" --data "$body" http://localhost:18080/webhook

Documentation site

The pages of docs/ are also published as a site, at https://eventide-oss.github.io/vesper/. docs/ stays the only source: the site in site/ is built with Astro Starlight, and site/scripts/sync-docs.mjs translates docs/ for it at each build. The first heading of a page becomes its title, links between pages become links of the site, and links to the rest of the repository point to GitHub. Write the docs as before, for GitHub; the site follows. The home page of the site is not in docs/: it is written for the site, in site/src/landing/index.mdx, and docs/README.md becomes its Overview page.

Run it locally, with Node 22.23.4 or later:

cd site
npm ci --ignore-scripts
npm run dev      # http://localhost:4321/vesper/, reloads on change of site/
npm run build    # the site in site/dist, then a check of its internal links

npm run dev translates docs/ when it starts: run it again after editing docs/. A new page of docs/ must also be added to the sidebar of site/astro.config.mjs. The workflow .github/workflows/docs.yml builds the site on every pull request that touches docs/ or site/, and publishes it from main.