kubectl or a Kubernetes client library. The Ornn CLI is optional.
There are two separate APIs:
- Ornn REST API (
https://api.ornn.com): authenticate with an Ornn API key to create clusters, read their status, retrieve access, add or remove workers, and tear clusters down. - Your cluster’s Kubernetes HTTPS API: authenticate with the issued kubeconfig to manage Pods, Jobs, Deployments, Services, namespaces, and quotas. Use the server address and namespace in that file, not
api.ornn.com. An Ornn API key is not a Kubernetes bearer token.
/controllers). Buying capacity does not launch a cluster.
The cluster table shows cluster name, status, node count, GPU model, network, and last update. Beside search, use Cluster type, Status, and Network to narrow the list. Each chip allows multiple values, starts unfiltered, and combines with the others. Clear a chip with ×. Changing a filter returns to the first page.
On the cluster manage page, the node list fits each page to the available window space, with previous and next controls and a result count. Search and filters apply to the whole cluster and return you to the first page. This also applies to Slurm clusters.
Choose Actions → Documentation on the cluster manage page to open the guide for your cluster type.
Old /compute/kubernetes bookmarks now open Orchestration (/controllers), where you create and manage clusters.
Before you start
- Sign in as a member of the organization that owns the reservation.
- The reservation must be active and have available, eligible nodes.
- For API integration, create an API key and use an HTTPS client. For the terminal workflow, install kubectl and use a version compatible with your cluster.
- Your organization can run any number of Kubernetes and Slurm clusters, one per reservation. To switch a reservation from Slurm to Kubernetes, tear down its Slurm cluster and wait for it to finish first.
- Decide whether you need a Public or Private connection. Private access requires WireGuard on your computer. Choose Private if you need a shared filesystem at
/workspace. - Save any data on selected nodes before launch. If Ornn asks to clean an existing environment, that cleanup can destroy local data.
- Spot VMs can join a cluster like other running VMs, but they stay interruptible. When a Spot reservation ends, its VMs stop and leave the cluster, along with any Pods running on them.
- Running VMs on your reservation can join as workers, alone or alongside nodes. While a VM is in a cluster, its SSH keys are set aside and its open SSH sessions end; its disk and data stay. The keys come back when the VM leaves the cluster, including any key changes you made in the meantime.
- For a VM reservation, select its VMs. The machine a VM runs on cannot join a cluster itself, so Ornn never removes your VMs to add their machine.
- Only one VM per machine can be in a Public cluster. To cluster several VMs that share a machine, use Private networking. Workers in a Public cluster each need their own network address; a VM whose address another worker already uses cannot join.
Provision and manage through REST
Every lifecycle step in the console has an API equivalent. With an Ornn API key, your automation can:- List the hosts on a reservation and choose eligible ones.
- Create a cluster on those hosts.
- Poll the cluster list until its
stateisactive. - Fetch the kubeconfig.
- Run GPU workloads through the cluster’s own Kubernetes API.
- Add or remove workers.
- Tear down the cluster and confirm it is gone.
curl, and Python examples, including polling, safe retries, and teardown confirmation. Keep these rules in mind:
- Creating a cluster for a reservation that already has one returns the existing cluster when the selected workers are already members. Selecting new workers returns
409 cluster_already_exists. Open Manage cluster and use Add nodes, or use the worker endpoints to change membership. - Ornn does not add or remove workers automatically. Pending Pods do not add capacity; scaling is an API call your automation makes against hosts you already reserved.
- The kubeconfig token is valid for seven days. Ornn issues a new one about two days before it expires; download the kubeconfig again, or read access again, to get it. The access response’s
expires_atshows when the current token expires. Do not tear down a working cluster to renew access. - There is no
GETfor one cluster. Read the cluster list and select bycontroller.id.
Launch through the console or Ornn CLI
1
Open Orchestration
Open Orchestration and start a new cluster. Select Kubernetes.
2
Select your nodes
Open Choose available nodes to cluster, select your reservation group, and select the nodes to use. Only eligible nodes appear. The node count is a worker count, not a GPU count.Select VMs in the same dropdown alongside nodes. Only running VMs that are not in another cluster appear.If the picker is empty, check that the term has started and the nodes are available. Launch On-Demand, when shown, opens a separate VM purchase flow; it does not create a Kubernetes cluster.
3
Choose networking and launch
Public makes the cluster API reachable over the internet; authentication is still required. Private requires a connection to the selected organization’s private network. Set up its WireGuard config in the launch dialog or the cluster’s Network tab.Review the selected workers, then click Create cluster. When physical nodes are selected, confirm Allow cleanup and launch: host preparation may stop processes and delete home-directory data. VM-only launches do not require host cleanup confirmation. The manage page shows progress while Ornn creates the control plane, prepares the hosts, joins the workers, and issues access. Image downloads and host preparation can take several minutes. Wait for Download kubeconfig to become available.
--vms <vm-id>,<vm-id>; use --nodes 0 for a cluster of VMs only. For scripts, supply those inputs explicitly. See ornn help launch and Human and automation modes.
Connect and verify access
1
Download your kubeconfig
On the cluster manage page, select the Download kubeconfig icon beside the Actions menu. This file contains the API address, default namespace, and a token with cluster-admin access to your cluster. The token is valid for seven days. Keep it private.The download stays available while the cluster is active, even when it has no nodes. You can connect and queue Jobs before capacity joins.
2
Select the downloaded file
In a macOS or Linux terminal, replace the path with the actual downloaded filename:The commands below use that default namespace. Do not paste a placeholder such as
tenant-<id> into a manifest. If your manifest specifies a namespace, use the Namespace shown on the cluster page.3
Check the cluster
Bring up WireGuard first if the cluster is Private, then run:Check that your selected workers are Ready. The node list also includes managed control-plane nodes; they are not GPU capacity for your workloads.
Run your first GPU job
This NVIDIA example requests one GPU, runsnvidia-smi, and exits. Use it on NVIDIA capacity with a compatible driver. It does not install a training framework.
amd-smi installed, request amd.com/gpu: 1, and run ["amd-smi", "list"] instead. Use a distinct job name. See the Kubernetes GPU scheduling guide for resource-request rules.
On CPU capacity, workers have no GPUs and the namespace’s ornn-gpu-quota is 0, so skip this GPU test. Instead, run the same Job without the nvidia.com/gpu limit, with an image such as busybox and the command ["nproc"], and request CPU and memory. Success means the Job completes and its logs show the CPU count.
Run your application
Use an image that includes your framework and dependencies. Match its CUDA or ROCm requirements to the reserved hardware. Push private images to the Ornn container registry, or keep them in your own registry and add its credentials as animagePullSecret in the workload namespace.
In the supplied namespace, Ornn adds the scheduling rules that place pods on the workers from the reservation the cluster launched on. You do not need to add a reservation node selector or toleration. Those defaults apply to that namespace; a namespace you create yourself does not automatically receive them. Avoid changing managed scheduling policies or placing workloads on control-plane nodes.
Request CPU and memory as well as GPUs. A single container’s GPU request must fit on one worker; requesting eight GPUs does not combine four GPUs on each of two workers. Distributed training needs separate worker pods and an application configured for distributed execution.
Push images to the Ornn container registry
Every Organization has a private namespace in the Ornn container registry. Each of your Kubernetes clusters keeps its own copy of your Organization’s images, and its workers pull from that copy, so pods need noimagePullSecret.
1
Find your registry
Open Orchestration → Registry and copy Image prefix, which reads
<registry-host>/<organization-id>. Every image you push goes under it. The cluster manage page shows the same prefix as Image registry.2
Log in
With the Ornn CLI, run:It creates a read-write API key named Only a read-write key of an approved Organization can push. A read key can pull.
Docker login on <machine> and logs Docker in with it. Running it again on the same machine replaces that key.To use a key you manage yourself, create a read-write key under Account → Developer → API keys. Docker accepts any username; the key is the password:3
Push an image
4
Use it in a workload
Name the pushed image in your manifest, for example
image: <registry-host>/<organization-id>/my-trainer:v1. Each cluster copies a new push within a few minutes, longer for a large image. Until the copy lands, a pod that names it shows ImagePullBackOff and retries on its own.Manage images
Orchestration → Registry lists your repositories. Open one to see each tag’s digest, size, build time, and how many of your clusters already hold it. Storage used counts each layer once, however many tags share it. The same views are available asornn registry repositories list, ornn registry tags list <repository>, ornn registry copies list, and ornn registry usage.
To delete a tag, select its trash icon, or run ornn registry tags delete <repository>:<tag>. Pods and Slurm jobs that name the tag can no longer pull it. Each cluster drops its copy within a minute, and the registry then frees the space no other tag uses. While it does, pushes pause for about a minute and Docker reports that the registry is reclaiming space; push again after that.
MPI and distributed training
The cluster includes the MPI Operator. Check that its resource is available:Autoscaling and queueing
A Horizontal Pod Autoscaler can scale a Deployment, ReplicaSet, or StatefulSet using CPU or memory. Set the corresponding containerresources.requests. GPU utilization needs a separately configured metrics adapter.
Autoscaling changes pod replicas, not your purchased node count. Ornn does not add workers when pods are pending; there is no Cluster Autoscaler or Karpenter integration. To change capacity, add or remove workers yourself or from your own automation through the API.
Pending and running pods count toward the namespace GPU request quota, ornn-gpu-quota. A quota rejection prevents pod creation; an admitted pod can remain Pending while all matching GPUs are busy. Inspect workload events and kubectl describe resourcequota ornn-gpu-quota before increasing replicas. Adding workers and changing a namespace quota are separate operations.
Choose storage before running a long job
Local persistent volumes
The defaultlocal-path class supports ReadWriteOnce and binds when the first consumer pod is scheduled. A Pending PVC without a consumer is expected. Mount the PVC in your workload; do not rely on an arbitrary host path.
Local data stays on the selected worker. It does not follow a workload to another worker after disk or node loss. Requested capacity is not an enforced filesystem limit, and resizing is unsupported. The class uses a Delete reclaim policy: deleting the PVC can delete its backing data. Export important data before deleting a claim, removing a worker, or tearing down the cluster.
Shared filesystem at /workspace
A shared filesystem requires a Private cluster. Contact Ornn to provision one. Ornn confirms the capacity, performance tier, region, storage cost, and reservation networking with you, then provisions and attaches the filesystem for the reservation.
Wait for provisioning and attachment to finish. Once active, the supplied namespace injects /workspace into workload pods. Verify both the claim and access from your application:
Private access with WireGuard
Install WireGuard on your computer. Generate a key pair locally:.conf, replace its private-key placeholder locally with the contents of ornn-wg-private.key, and protect the file. Keep MTU = 1200 and PersistentKeepalive = 25 from the generated config.
The cluster’s Network tab uses the private network selected at launch. You do not need to create another network to add a device. Confirm that the tab shows your cluster’s network before creating a config. Shared (default) means the cluster has no dedicated network selected. A dedicated network that is not active keeps its name and shows its status; Dedicated network (unavailable) means its details could not be loaded. Wait for a pending or provisioning network to become active. Contact support if the network is degraded or unavailable, or if Shared (default) appears when you expected a dedicated network.
On Linux with wg-quick installed:
Add or remove workers
On the cluster manage page, use Add nodes to select eligible nodes owned by your organization or running VMs from the Target nodes dropdown. The picker includes eligible capacity from all active reservations in your organization, including new purchases made after the cluster was created. Choose the reservation groups containing the workers you want to add; selections stay selected across groups. If available nodes cannot be loaded, use Retry in the add dialog. Ornn prepares each Bare Metal node before it joins, which wipes its tenant home directory; a VM joins as it is. A machine running your VMs is not offered; add its VMs instead. In a Public cluster, a VM on the same machine as a VM already in the cluster is not offered either, and a VM whose machine already has a VM in another Public cluster cannot join; create a Private cluster to use both. Wait for the new workers to become Ready, then check their GPU capacity before increasing your workload. After an added VM worker becomes Ready, Ornn refreshes the supplied namespace’s GPU quota to include its GPUs. A host from a different reservation is labeled and tainted with the reservation the cluster launched on, so Pods in that namespace still schedule on it. After new workers are Ready, runkubectl describe resourcequota ornn-gpu-quota, and contact support if the quota does not cover the GPUs you plan to request.
To remove workers, select them in Nodes and use Remove. Removal drains those workers and interrupts workloads using them. Export local data and checkpoint first. Review remaining GPU capacity and pending workloads afterward. Removing a worker does not cancel its reservation. A removed VM keeps running and gets its SSH keys back.
To add and remove workers from your own automation, see Scale workers up or down.
Stop, tear down, or retry
In the cluster list, hover over a row or focus it with the keyboard to reveal Delete cluster. On mobile, the delete button is always visible. Select it and confirm to tear down the cluster. Reservation billing continues. Use Stop launch while provisioning, or Tear down cluster for an active cluster, from the manage page’s actions menu. Confirm the affected nodes and wait for cleanup to finish. Teardown removes workloads, revokes cluster access, and releases workers back to their reservations. A failed or torn-down cluster can disappear from Orchestration after cleanup. Start a new launch from your available reserved nodes. Do not rely on the old URL or downloaded credentials.Save checkpoints and export local data before teardown. Deleting a test Job is different from
tearing down its cluster. Teardown does not cancel the reserved term or delete separate storage
resources.
Troubleshooting
No nodes available to select
No nodes available to select
Confirm the correct organization, an active reservation, and available hosts. Offline nodes and nodes held by another cluster cannot be selected. If capacity is missing, contact support with the reservation ID.
A VM did not join
A VM did not join
If the VM left the cluster with a message that another machine already uses its network address, choose a different VM or relaunch with Private networking. If Ornn says only one VM per machine can be in a public cluster, relaunch with Private networking to include that VM. If its host needs a management update, contact support with the VM ID.
API connection times out
API connection times out
Confirm
KUBECONFIG points to this cluster’s file. For Private access, activate the matching WireGuard tunnel first. Check your local firewall and VPN routes. A small ping succeeding does not prove the API works; keep the generated MTU of 1200.Pod stays Pending or a Job cannot create pods
Pod stays Pending or a Job cannot create pods
Run
kubectl describe pod <pod-name>, kubectl describe job <job-name>, and kubectl describe resourcequota. Check GPU vendor, per-worker capacity, quota, CPU and memory, and volume placement. Use the supplied namespace so its scheduling defaults apply.ImagePullBackOff or ErrImagePull
ImagePullBackOff or ErrImagePull
Read pod events. Check the image name and tag, registry access, and your
imagePullSecrets. A registry can reject a pull because of authentication, permissions, rate limits, or an unavailable image; the error alone does not establish which cause applies. For an image you pushed to the Ornn container registry, confirm the name starts with the Image registry value and wait a few minutes after a push for the cluster’s copy.NCCL cannot register locked memory
NCCL cannot register locked memory
Check the locked-memory limits inside your training container with
ulimit -Sl and ulimit -Hl. Both should report unlimited. If either reports 8192 (8 MiB), contact Support with the cluster and pod IDs. Correcting an affected worker requires a coordinated runtime restart and a new workload pod; changing your login shell’s limit does not update an existing training process.PVC stays Pending or data is missing
PVC stays Pending or data is missing
For local-path, create the consumer pod and inspect its scheduling events. Local volumes cannot move between workers. For
/workspace, check that the cluster is Private, the filesystem is ready, and the attachment is active before testing from a pod.Launch stalls or fails
Launch stalls or fails
Open the progress details and record the failed stage and customer error. Avoid repeated launches while cleanup is in progress. Contact support with the reservation or cluster ID, approximate failure time, and sanitized error text.
Monitor GPUs with Prometheus
Every NVIDIA GPU worker runs NVIDIA DCGM Exporter on port 9400. AMD Instinct workers do not, so they have no series here. Each GPU series carries thepod, namespace, and container of the pod that requested that GPU, plus UUID, gpu, modelName, and Hostname (the worker). The headless Service ornn-dcgm-exporter in the gpu-operator namespace returns one address per NVIDIA GPU worker.
Confirm discovery from a pod in the supplied namespace. Expect one address per NVIDIA GPU worker:
remote_write.
- Attribution. Only GPUs requested through
nvidia.com/gpucarry pod labels. A GPU that no pod requested has an emptypodlabel. Count distinctUUIDvalues for total capacity. - Public clusters. If scrapes of exporters on other workers time out, run one scraper per worker that reads only the exporter on its own worker, then forward with
remote_write. - Prometheus Operator. The cluster ships a placeholder
ServiceMonitordefinition. A chart that installs the Prometheus Operator definitions as templates fails on ownership; a plain scrape job like the one above does not depend on it.
Monitor and get help
Use the cluster’s Observability tab for host metrics and kernel console logs. Both views fill the available tab width. Usekubectl logs, describe, and events for application failures. Host metrics do not prove a training job completed.
Open Support in the console or email support@ornn.com. Include the cluster and reservation IDs, workload name, failed stage, and time. Never include kubeconfig contents, private keys, registry secrets, or tokens.
What’s next

Kubernetes API automation
Provision, scale, and tear down clusters from your own automation with an API key.

Slurm access
Prefer batch scheduling? Run a managed Slurm cluster on your reserved GPUs instead.

Access overview
Compare the ways to access your reserved Ornn compute.