Self-hosting troubleshooting

Use these checks when the oz-agent-worker daemon won’t start or connect, tasks stay queued, or tasks fail.

Note

The steps below apply to the managed architecture (oz-agent-worker daemon). For unmanaged deployments, refer to the documentation for the environment running oz agent run (e.g., GitHub Actions, Kubernetes).

---

Worker won’t start

Docker backend

Cause: Docker isn’t running, or the daemon platform isn’t supported.

Fix:

1. Verify Docker is running: docker info.
2. Confirm the daemon platform is linux/amd64 or linux/arm64. Windows containers are not supported.
3. If the worker runs inside Docker, confirm the /var/run/docker.sock mount is correct and the mounting user has permission to the socket.

Kubernetes backend

Cause: The worker Deployment couldn’t start, reach the Kubernetes API, or create its preflight Job.

Fix:

1. Run kubectl describe pod -n NAMESPACE WORKER_POD. Replace NAMESPACE with the chart namespace and WORKER_POD with the worker pod name. For CreateContainerConfigError, verify the Secret and key configured by warp.apiKeySecret.
2. Check the worker logs for Kubernetes API or preflight diagnostics: kubectl logs -n NAMESPACE WORKER_POD.
3. Confirm the worker’s namespace has these permissions: create, get, list, watch, delete on jobs; get, list, watch on pods; get on pods/log; list on events.
4. Confirm the task namespace allows pods with a root init container, unless you enabled native image volumes with kubernetesBackend.useImageVolumes=true.
5. If your cluster restricts image sources, set kubernetesBackend.preflightImage to an allowlisted image. The default is busybox:1.36.
6. To pull the preflight image from a private registry, configure imagePullSecrets in kubernetesBackend.podTemplate.

A successful preflight validates the configured pod shape, not task-specific images, Secrets, setup commands, or network access.

Direct backend

Cause: The oz CLI isn’t installed or isn’t on the worker’s PATH.

Fix:

1. Install the Oz CLI on the worker host. See Installing the CLI.
2. If the CLI isn’t on PATH, set oz_path in the config file to the absolute path of the oz binary.

---

Worker won’t connect

Cause: The API key is invalid, expired, or the host cannot reach the Automation Platform‘s backend.

Fix:

1. Confirm your API key is correct, not expired, and has team scope.
2. Regenerate the API key in Settings > Cloud platform > API keys if you suspect it’s invalid.
3. Ensure the host has outbound internet access to oz.warp.dev:443.
4. Check that no firewall rules are blocking WebSocket connections to wss://oz.warp.dev.
5. Increase log verbosity with --log-level debug to see connection details.

See Security and networking for the full list of outbound endpoints the worker needs.

---

Tasks not being picked up

Cause: The worker isn’t running, the --host value doesn’t match the worker’s --worker-id, or the worker and task belong to different teams.

Fix:

1. Confirm the worker is running and connected. Check the worker logs for Successfully connected to server.
2. Verify the --host (or worker_host) value you passed matches your --worker-id exactly. Case-sensitive.
3. Ensure the worker’s team matches the team creating the task.

---

Metrics not appearing

Cause: The worker is running but metrics aren’t showing up in Prometheus or your collector.

Fix:

1. Verify OTEL_METRICS_EXPORTER is set correctly on the worker process. Run curl -s localhost:9464/metrics from the worker host (for prometheus mode) to confirm the endpoint is serving.
2. For Prometheus scrape mode, confirm the bind address is 0.0.0.0 (not localhost) when running in Docker or Kubernetes. localhost is only reachable from inside the container.
3. Confirm no firewall or network policy blocks the metrics port (default 9464).
4. For OTLP push mode, verify OTEL_EXPORTER_OTLP_ENDPOINT points to a reachable collector and that the protocol matches (http/protobuf vs grpc).
5. When using the Helm chart, confirm metrics.enabled=true is set. Check that the Service and optional PodMonitor were created: kubectl get svc,podmonitor -n NAMESPACE. Replace NAMESPACE with the chart namespace.
6. If using metrics.podMonitor.create=true, verify the monitoring.coreos.com CRDs are installed in the cluster. The PodMonitor resource requires the Prometheus Operator.
7. Restart the worker with --log-level debug and look for metrics-related error messages at startup.

See Monitoring for the full setup guide.

---

Task failures

Cause: The backend logs identify why the task failed.

Fix (all backends):

1. Review task logs in the cloud agent dashboard or via session sharing.
2. Use --no-cleanup to keep the container, Job, or workspace indefinitely for inspection. With Kubernetes cleanup enabled, failed Jobs remain for 24 hours by default.
3. Use --log-level debug to see detailed execution logs.
4. Ensure the worker machine or cluster has sufficient resources (CPU, memory, disk).

Docker backend (task failures)

1. Verify Docker is running (docker info).
2. If using a custom image, confirm it is glibc-based (not Alpine/musl) and that its architecture matches the worker’s Docker daemon platform.

Kubernetes backend (task failures)

Start with the pod phase and recent events:

kubectl get jobs,pods -n NAMESPACE
kubectl describe pod -n NAMESPACE TASK_POD
kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME
kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME --previous

Replace NAMESPACE with the task namespace, TASK_POD with the task pod name, and CONTAINER_NAME with the failed container name. Match the reported reason:

• Pending or Unschedulable: Read the scheduling events. Check free CPU and memory, selectors and affinity, taints and tolerations, topology constraints, quotas, and volume binding. Allow enough time for node provisioning and reserve capacity for init containers, DaemonSets, and spikes.
• OOMKilled: Increase memory in the agent profile’s instance shape. The instance shape overrides the task container’s CPU and memory values in the pod template. Check sidecar and init-container resources separately.
• ErrImagePull, ImagePullBackOff, or InvalidImageName: Follow Image pull failures. Preflight does not pull every task image.
• CreateContainerConfigError or FailedMount: Events name the missing Secret, ConfigMap, service account, key, or volume. Verify that it exists in the task namespace.
• Init container failure: Check each init container’s status and logs. The Warp sidecar-loading init container runs as root unless native image volumes are enabled. Custom init containers must finish before the task starts.
• Evicted, DeadlineExceeded, or exit code 143: Check node pressure, lifecycle-tool events, the Job deadline, and termination events. A replacement pod cannot recover the task’s emptyDir workspace. See protecting active task pods from voluntary disruption.
• The task cannot reach a dependency: Test DNS, TLS, and the destination from the task pod. Worker connectivity to Warp does not test the task’s network policies, service mesh, proxy, or dependency egress.

The worker API key authenticates the worker to Warp; it is not a task credential. Provide task credentials through your Secret integration and pod template.

Direct backend (task failures)

1. Verify the Oz CLI is accessible.
2. Verify the workspace root directory has write permissions for the user running the worker.

---

Image pull failures

Docker backend (image pull)

1. If using a private registry, ensure Docker credentials are available to the worker. See Private Docker registries.
2. Try pulling the image manually on the worker host: docker pull <image>.

Kubernetes backend (image pull)

1. Configure imagePullSecrets in the pod_template section of your worker config.
2. Verify the Secret exists in the task namespace and contains valid credentials.

Both backends (image pull)

• Verify the image exists and the tag is correct.
• Check network connectivity from the worker/cluster to the registry.

---

Related pages

• Self-hosting overview — Architecture and decision guide.
• Self-hosted worker reference — CLI flags and config schema, including every flag mentioned here.
• Security and networking — Outbound endpoints the worker needs.
• Agent Session Sharing — Attach to running tasks to debug interactively.