Deployment

Configuration, installation, RBAC, and testing for the Propeller Kubernetes Operator.

Configuration

The operator is configured through command-line flags and one environment variable. There is no configuration file.

Environment Variables

Variable	Default	Description
`WATCH_NAMESPACE`	(all namespaces)	When set, the operator only watches resources in this namespace. Leave unset to watch all namespaces.

MQTT Flags

Flag	Default	Description
`--mqtt-address`	—	MQTT broker URL (required, e.g. `tcp://mqtt:1883`)
`--tenant-id`	—	Atom domain / tenant ID (required)
`--channel-id`	—	Atom channel ID (required)
`--entity-id`	—	Manager entity ID for MQTT authentication (required)
`--api-key`	—	Manager API key for MQTT authentication (required)
`--mqtt-qos`	`0`	MQTT QoS level (0–2)
`--mqtt-timeout`	`30s`	Broker operation timeout

Proplet Monitoring Flags

Flag	Default	Description
`--liveliness-interval`	`10s`	How often to check proplet heartbeat timestamps
`--last-seen-threshold`	`30s`	How long without a heartbeat before marking a proplet Offline

Kubernetes Manager Flags

Flag	Default	Description
`--health-probe-bind-address`	`:8181`	Address for `/healthz` and `/readyz` endpoints
`--metrics-bind-address`	`0`	Address for Prometheus metrics (`0` disables)
`--metrics-secure`	`true`	Serve metrics over HTTPS
`--metrics-cert-path`	—	Directory containing the metrics TLS certificate
`--metrics-cert-name`	`tls.crt`	Metrics certificate filename
`--metrics-cert-key`	`tls.key`	Metrics key filename
`--leader-elect`	`false`	Enable leader election for high-availability deployments
`--enable-http2`	`false`	Enable HTTP/2 (disabled by default for security)
`--webhook-cert-path`	—	Directory containing the webhook TLS certificate
`--webhook-cert-name`	`tls.crt`	Webhook certificate filename
`--webhook-cert-key`	`tls.key`	Webhook key filename

Installing and Deploying

For a complete walkthrough of setting up the operator with Atom and running a task, see the Kubernetes Operator Example.

Installing CRDs

cd propeller-k8s-operator
make install

This runs controller-gen to generate CRD manifests from the Go types, then applies them with kubectl via Kustomize.

Deploying the Controller

make deploy IMG=<your-registry>/propeller-k8s-operator:<tag>

This renders the full deployment manifest (controller Deployment, ClusterRole, ClusterRoleBinding, ServiceAccount, CRDs) via Kustomize and applies it. The operator namespace (propeller-k8s-operator-system) is created automatically.

For local development with k3d, build the image and import it directly into the cluster nodes before deploying (no registry required):

make docker-build IMG=propeller-k8s-operator:latest
k3d image import propeller-k8s-operator:latest -c k3s-default
make deploy IMG=propeller-k8s-operator:latest

Set imagePullPolicy: Never in config/manager/manager.yaml when using a locally-loaded image.

Running Locally

go run ./cmd/main.go \
  --mqtt-address="tcp://localhost:1883" \
  --tenant-id="<tenant-id>" \
  --channel-id="<channel-id>" \
  --entity-id="<manager-entity-id>" \
  --api-key="<manager-api-key>" \
  --metrics-secure=false

The operator uses the current kubectl context for cluster access.

RBAC

The operator requires a ClusterRole with the following permissions:

API Group	Resources	Verbs
`""` (core)	`configmaps`	create, delete, get, list, patch, update
`""` (core)	`pods`	get, list
`""` (core)	`secrets`	get
`apps`	`deployments`	create, delete, get, list, patch, update, watch
`batch`	`jobs`	create, delete, get, list, patch, update, watch
`propeller.propeller.absmach.eu`	`tasks`, `proplets`, `propellerjobs`, `federatedjobs`, `trainingrounds`	create, delete, get, list, patch, update, watch
`propeller.propeller.absmach.eu`	`*/status`	get, patch, update
`propeller.propeller.absmach.eu`	`*/finalizers`	update

Container Security

When deployed in-cluster the manager pod runs with:

runAsNonRoot: true
seccompProfile: RuntimeDefault
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
All Linux capabilities dropped

Resource defaults: 10m CPU / 64Mi memory requests; 500m CPU / 128Mi limits.

Health Probes

Endpoint	Purpose
`GET /healthz`	Liveness probe — confirms the process is alive
`GET /readyz`	Readiness probe — confirms the controller manager is ready to serve

Both endpoints are served on --health-probe-bind-address (default :8181).

Testing

The operator includes a controller test suite that uses envtest to spin up a real Kubernetes API server and etcd in-process, with all five CRDs loaded from config/crd/bases.

make test

End-to-end tests run against a real cluster (Kind by default):

make setup-test-e2e   # creates a Kind cluster
make test-e2e         # runs Ginkgo e2e suite
make cleanup-test-e2e # tears down the Kind cluster

Sample Configurations

Sample YAML files covering all CRDs are in config/samples/:

File	CRD	Description
`propeller_v1_proplet.yaml`	Proplet (k8s)	K8s-backed proplet managed as a Deployment
`propeller_v1_proplet_external.yaml`	Proplet (external)	External device proplet via MQTT
`propeller_v1_task.yaml`	Task (file)	WASM file dispatched via MQTT
`propeller_v1_task_with_image.yaml`	Task (registry)	WASM OCI image reference dispatched via MQTT (proplet fetches it via the registry proxy)
`propeller_v1_task_broadcast.yaml`	Task (broadcast)	WASM sent to all proplets
`propeller_v1_task_recurring.yaml`	Task (cron)	Cron-scheduled recurring task
`propeller_v1_task_monitoring.yaml`	Task (monitoring)	Task with metrics collection
`propeller_v1_task_dag_a.yaml`	Task (DAG target)	DAG dependency target
`propeller_v1_task_dag.yaml`	Task (DAG dependent)	Task with dependsOn
`propeller_v1_propellerjob.yaml`	PropellerJob	Batch of parallel tasks
`propeller_v1_federatedjob.yaml`	FederatedJob	Multi-round FL experiment

The operator repo also ships a sample per WASM example — propeller_v1_task_compute.yaml, _string_input.yaml, _http_server.yaml, _http_client.yaml, _attestation_test.yaml, _hal_test.yaml, _tee.yaml, _filesystem.yaml, _wasi_http.yaml — plus propeller_v1_proplet_full.yaml (every optional k8s.env.* flag enabled, for the feature-gated ones) and propeller_v1_proplet_wasi_nn.yaml (dedicated WASI-NN image). See Running Other WASM Examples for what each needs and the verified output.

propeller_v1_task_with_image.yaml and spec.imageUrl generally need the registry proxy service running — propeller_proxy_deployment.yaml is a plain Deployment + Service for it (not a CRD; see Running Other WASM Examples: Deploying the registry proxy).

E2E Tests

The operator ships with a Ginkgo-based e2e test suite that uses Kind:

# Kind (creates a temporary cluster automatically)
make test-e2e

Manual Testing on k3d

Create a k3d cluster:
```
k3d cluster create propeller
```
Install CRDs:
```
make install
```

Start the operator:

make run ARGS="--mqtt-address='tcp://your-mqtt:1883' \
  --tenant-id='<tenant>' \
  --channel-id='<channel>' \
  --entity-id='<entity>' \
  --api-key='<api-key>'"

Create a proplet and run a task:

# k8s proplet
kubectl apply -f config/samples/propeller_v1_proplet.yaml

# WASM task via MQTT
kubectl apply -f config/samples/propeller_v1_task.yaml

# WASM task via an OCI image reference (proplet fetches it via the registry proxy)
kubectl apply -f config/samples/propeller_v1_task_with_image.yaml

# Broadcast task
kubectl apply -f config/samples/propeller_v1_task_broadcast.yaml

# PropellerJob (parallel batch)
kubectl apply -f config/samples/propeller_v1_propellerjob.yaml

# FederatedJob (FL experiment)
kubectl apply -f config/samples/propeller_v1_federatedjob.yaml

Watch results:

kubectl get proplets,tasks,pjob,federatedjob,trainingrounds -w

Cleanup:
```
k3d cluster delete propeller
```

On this page