Skip to main content
Launch Kubernetes on your reserved GPU or CPU nodes through the console or the REST API, then run workloads with kubectl or a Kubernetes client library. The Ornn CLI is optional. There are two separate APIs:
  • Ornn REST API (https://api.ornn.com): authenticate with an Ornn API key to create clusters, read their status, retrieve access, add or remove workers, and tear clusters down.
  • Your cluster’s Kubernetes HTTPS API: authenticate with the issued kubeconfig to manage Pods, Jobs, Deployments, Services, namespaces, and quotas. Use the server address and namespace in that file, not api.ornn.com. An Ornn API key is not a Kubernetes bearer token.
To automate the whole lifecycle with an API key, follow Automate managed Kubernetes with the Ornn API. Use the REST API reference for endpoint schemas, or download OpenAPI for Swagger, Postman, or a client generator. Ornn’s provisioning API is REST over HTTPS; there is no gRPC or GraphQL provisioning API. These steps provision an Ornn-managed cluster. They do not attach Ornn workers to an existing EKS, GKE, or AKS cluster. Contact Ornn about external-cluster requirements before designing that integration. If Ornn has connected an existing custom cluster to your organization, open it in Orchestration and use Download kubeconfig. Its manage page identifies it as managed on site. You have cluster-admin access through that kubeconfig; adding or removing servers and deleting the cluster require coordination with Ornn instead of the standard console lifecycle controls. Downloading the file again does not renew an expired credential. Access expired identifies this separately from the cluster’s running status. Contact support for renewed access, then download again to get the current credential without reloading the page. Ornn can link those existing cluster members to an existing reservation owned by your organization so they appear under that reservation in Nodes. Linking preserves the running cluster, workloads and kubeconfig. Continue using Orchestration to download access; coordinate server removal and handoff with Ornn. Ornn creates a control plane for your cluster and joins the nodes you select as workers. On CPU capacity, each worker is a whole node. Your reservation remains visible in Nodes; manage the cluster from Orchestration (/controllers). Buying capacity does not launch a cluster. The cluster table shows cluster name, status, node count, GPU model, network, and last update. Beside search, use Cluster type, Status, and Network to narrow the list. Each chip allows multiple values, starts unfiltered, and combines with the others. Clear a chip with ×. Changing a filter returns to the first page. On the cluster manage page, the node list fits each page to the available window space, with previous and next controls and a result count. Search and filters apply to the whole cluster and return you to the first page. This also applies to Slurm clusters. Choose Actions → Documentation on the cluster manage page to open the guide for your cluster type. Old /compute/kubernetes bookmarks now open Orchestration (/controllers), where you create and manage clusters.

Before you start

  • Sign in as a member of the organization that owns the reservation.
  • The reservation must be active and have available, eligible nodes.
  • For API integration, create an API key and use an HTTPS client. For the terminal workflow, install kubectl and use a version compatible with your cluster.
  • Your organization can run any number of Kubernetes and Slurm clusters, one per reservation. To switch a reservation from Slurm to Kubernetes, tear down its Slurm cluster and wait for it to finish first.
  • Decide whether you need a Public or Private connection. Private access requires WireGuard on your computer. Choose Private if you need a shared filesystem at /workspace.
  • Save any data on selected nodes before launch. If Ornn asks to clean an existing environment, that cleanup can destroy local data.
Kubernetes does not require an SSH key for API access. Nodes already allocated to another cluster are unavailable. A multi-node selection does not guarantee a high-speed interconnect; confirm your workload’s networking requirements before reserving capacity.
  • Spot VMs can join a cluster like other running VMs, but they stay interruptible. When a Spot reservation ends, its VMs stop and leave the cluster, along with any Pods running on them.
  • Running VMs on your reservation can join as workers, alone or alongside nodes. While a VM is in a cluster, its SSH keys are set aside and its open SSH sessions end; its disk and data stay. The keys come back when the VM leaves the cluster, including any key changes you made in the meantime.
  • For a VM reservation, select its VMs. The machine a VM runs on cannot join a cluster itself, so Ornn never removes your VMs to add their machine.
  • Only one VM per machine can be in a Public cluster. To cluster several VMs that share a machine, use Private networking. Workers in a Public cluster each need their own network address; a VM whose address another worker already uses cannot join.

Provision and manage through REST

Every lifecycle step in the console has an API equivalent. With an Ornn API key, your automation can:
  1. List the hosts on a reservation and choose eligible ones.
  2. Create a cluster on those hosts.
  3. Poll the cluster list until its state is active.
  4. Fetch the kubeconfig.
  5. Run GPU workloads through the cluster’s own Kubernetes API.
  6. Add or remove workers.
  7. Tear down the cluster and confirm it is gone.
Automate managed Kubernetes with the Ornn API walks through each step with CLI, curl, and Python examples, including polling, safe retries, and teardown confirmation. Keep these rules in mind:
  • Creating a cluster for a reservation that already has one returns the existing cluster when the selected workers are already members. Selecting new workers returns 409 cluster_already_exists. Open Manage cluster and use Add nodes, or use the worker endpoints to change membership.
  • Ornn does not add or remove workers automatically. Pending Pods do not add capacity; scaling is an API call your automation makes against hosts you already reserved.
  • The kubeconfig token is valid for seven days. Ornn issues a new one about two days before it expires; download the kubeconfig again, or read access again, to get it. The access response’s expires_at shows when the current token expires. Do not tear down a working cluster to renew access.
  • There is no GET for one cluster. Read the cluster list and select by controller.id.

Launch through the console or Ornn CLI

1

Open Orchestration

Open Orchestration and start a new cluster. Select Kubernetes.
2

Select your nodes

Open Choose available nodes to cluster, select your reservation group, and select the nodes to use. Only eligible nodes appear. The node count is a worker count, not a GPU count.Select VMs in the same dropdown alongside nodes. Only running VMs that are not in another cluster appear.If the picker is empty, check that the term has started and the nodes are available. Launch On-Demand, when shown, opens a separate VM purchase flow; it does not create a Kubernetes cluster.
3

Choose networking and launch

Public makes the cluster API reachable over the internet; authentication is still required. Private requires a connection to the selected organization’s private network. Set up its WireGuard config in the launch dialog or the cluster’s Network tab.Review the selected workers, then click Create cluster. When physical nodes are selected, confirm Allow cleanup and launch: host preparation may stop processes and delete home-directory data. VM-only launches do not require host cleanup confirmation. The manage page shows progress while Ornn creates the control plane, prepares the hosts, joins the workers, and issues access. Image downloads and host preparation can take several minutes. Wait for Download kubeconfig to become available.
From an interactive terminal, you can also launch with the CLI:
Replace angle-bracket placeholders with your values. In a terminal, launch prompts for a missing network and node count. When more machines are eligible than you requested, it asks you to choose the exact machines. Add running VMs with --vms <vm-id>,<vm-id>; use --nodes 0 for a cluster of VMs only. For scripts, supply those inputs explicitly. See ornn help launch and Human and automation modes.

Connect and verify access

1

Download your kubeconfig

On the cluster manage page, select the Download kubeconfig icon beside the Actions menu. This file contains the API address, default namespace, and a token with cluster-admin access to your cluster. The token is valid for seven days. Keep it private.The download stays available while the cluster is active, even when it has no nodes. You can connect and queue Jobs before capacity joins.
2

Select the downloaded file

In a macOS or Linux terminal, replace the path with the actual downloaded filename:
The commands below use that default namespace. Do not paste a placeholder such as tenant-<id> into a manifest. If your manifest specifies a namespace, use the Namespace shown on the cluster page.
3

Check the cluster

Bring up WireGuard first if the cluster is Private, then run:
Check that your selected workers are Ready. The node list also includes managed control-plane nodes; they are not GPU capacity for your workloads.

Run your first GPU job

This NVIDIA example requests one GPU, runs nvidia-smi, and exits. Use it on NVIDIA capacity with a compatible driver. It does not install a training framework.
Success means the Job completes and its logs list an NVIDIA GPU. If the wait fails, inspect the job before deleting it:
After checking the result, remove only this test:
For AMD Instinct capacity, use an image with ROCm and amd-smi installed, request amd.com/gpu: 1, and run ["amd-smi", "list"] instead. Use a distinct job name. See the Kubernetes GPU scheduling guide for resource-request rules. On CPU capacity, workers have no GPUs and the namespace’s ornn-gpu-quota is 0, so skip this GPU test. Instead, run the same Job without the nvidia.com/gpu limit, with an image such as busybox and the command ["nproc"], and request CPU and memory. Success means the Job completes and its logs show the CPU count.

Run your application

Use an image that includes your framework and dependencies. Match its CUDA or ROCm requirements to the reserved hardware. Push private images to the Ornn container registry, or keep them in your own registry and add its credentials as an imagePullSecret in the workload namespace. In the supplied namespace, Ornn adds the scheduling rules that place pods on the workers from the reservation the cluster launched on. You do not need to add a reservation node selector or toleration. Those defaults apply to that namespace; a namespace you create yourself does not automatically receive them. Avoid changing managed scheduling policies or placing workloads on control-plane nodes. Request CPU and memory as well as GPUs. A single container’s GPU request must fit on one worker; requesting eight GPUs does not combine four GPUs on each of two workers. Distributed training needs separate worker pods and an application configured for distributed execution.

Push images to the Ornn container registry

Every Organization has a private namespace in the Ornn container registry. Each of your Kubernetes clusters keeps its own copy of your Organization’s images, and its workers pull from that copy, so pods need no imagePullSecret.
1

Find your registry

Open Orchestration → Registry and copy Image prefix, which reads <registry-host>/<organization-id>. Every image you push goes under it. The cluster manage page shows the same prefix as Image registry.
2

Log in

With the Ornn CLI, run:
It creates a read-write API key named Docker login on <machine> and logs Docker in with it. Running it again on the same machine replaces that key.To use a key you manage yourself, create a read-write key under Account → Developer → API keys. Docker accepts any username; the key is the password:
Only a read-write key of an approved Organization can push. A read key can pull.
3

Push an image

4

Use it in a workload

Name the pushed image in your manifest, for example image: <registry-host>/<organization-id>/my-trainer:v1. Each cluster copies a new push within a few minutes, longer for a large image. Until the copy lands, a pod that names it shows ImagePullBackOff and retries on its own.
Workers pull through the cluster’s copy only if they joined after the registry became available to the cluster. Remove and re-add an older worker to give it access. Slurm clusters pull the same images in job containers; see Use images from the Ornn container registry.

Manage images

Orchestration → Registry lists your repositories. Open one to see each tag’s digest, size, build time, and how many of your clusters already hold it. Storage used counts each layer once, however many tags share it. The same views are available as ornn registry repositories list, ornn registry tags list <repository>, ornn registry copies list, and ornn registry usage. To delete a tag, select its trash icon, or run ornn registry tags delete <repository>:<tag>. Pods and Slurm jobs that name the tag can no longer pull it. Each cluster drops its copy within a minute, and the registry then frees the space no other tag uses. While it does, pushes pause for about a minute and Docker reports that the registry is reclaiming space; push again after that.

MPI and distributed training

The cluster includes the MPI Operator. Check that its resource is available:
Use compatible launcher and worker images, request GPUs on workers, and configure the MPI runtime’s network interface. RDMA performance depends on the reserved hosts and interconnect; first confirm a small multi-node job works before scaling training. VM workers follow the same GPUDirect RDMA guidance as VMs.

Autoscaling and queueing

A Horizontal Pod Autoscaler can scale a Deployment, ReplicaSet, or StatefulSet using CPU or memory. Set the corresponding container resources.requests. GPU utilization needs a separately configured metrics adapter. Autoscaling changes pod replicas, not your purchased node count. Ornn does not add workers when pods are pending; there is no Cluster Autoscaler or Karpenter integration. To change capacity, add or remove workers yourself or from your own automation through the API. Pending and running pods count toward the namespace GPU request quota, ornn-gpu-quota. A quota rejection prevents pod creation; an admitted pod can remain Pending while all matching GPUs are busy. Inspect workload events and kubectl describe resourcequota ornn-gpu-quota before increasing replicas. Adding workers and changing a namespace quota are separate operations.

Choose storage before running a long job

Local persistent volumes

The default local-path class supports ReadWriteOnce and binds when the first consumer pod is scheduled. A Pending PVC without a consumer is expected. Mount the PVC in your workload; do not rely on an arbitrary host path. Local data stays on the selected worker. It does not follow a workload to another worker after disk or node loss. Requested capacity is not an enforced filesystem limit, and resizing is unsupported. The class uses a Delete reclaim policy: deleting the PVC can delete its backing data. Export important data before deleting a claim, removing a worker, or tearing down the cluster.

Shared filesystem at /workspace

A shared filesystem requires a Private cluster. Contact Ornn to provision one. Ornn confirms the capacity, performance tier, region, storage cost, and reservation networking with you, then provisions and attaches the filesystem for the reservation. Wait for provisioning and attachment to finish. Once active, the supplied namespace injects /workspace into workload pods. Verify both the claim and access from your application:
A Bound claim alone does not prove every worker can reach the filesystem. Test a disposable file from pods on different workers before using it for distributed checkpoints. Changing a workspace attachment can restart workloads; checkpoint first. Cluster teardown does not delete the separately provisioned filesystem. Detaching removes access; deleting the filesystem is a separate destructive action. Export data first and confirm deletion and the end of storage charges with Ornn. Keep backups outside the cluster.

Private access with WireGuard

Install WireGuard on your computer. Generate a key pair locally:
Paste only the public key into Create config in the cluster’s Network tab. Download the returned .conf, replace its private-key placeholder locally with the contents of ornn-wg-private.key, and protect the file. Keep MTU = 1200 and PersistentKeepalive = 25 from the generated config. The cluster’s Network tab uses the private network selected at launch. You do not need to create another network to add a device. Confirm that the tab shows your cluster’s network before creating a config. Shared (default) means the cluster has no dedicated network selected. A dedicated network that is not active keeps its name and shows its status; Dedicated network (unavailable) means its details could not be loaded. Wait for a pending or provisioning network to become active. Contact support if the network is degraded or unavailable, or if Shared (default) appears when you expected a dedicated network. On Linux with wg-quick installed:
On macOS or Windows, import the completed config into the WireGuard app and activate it. Keep private keys on your own computer. Each device needs its own key pair and peer. To rotate a key, remove that device’s old peer, create a replacement, and import its new config. Removing a peer disconnects that device.

Add or remove workers

On the cluster manage page, use Add nodes to select eligible nodes owned by your organization or running VMs from the Target nodes dropdown. The picker includes eligible capacity from all active reservations in your organization, including new purchases made after the cluster was created. Choose the reservation groups containing the workers you want to add; selections stay selected across groups. If available nodes cannot be loaded, use Retry in the add dialog. Ornn prepares each Bare Metal node before it joins, which wipes its tenant home directory; a VM joins as it is. A machine running your VMs is not offered; add its VMs instead. In a Public cluster, a VM on the same machine as a VM already in the cluster is not offered either, and a VM whose machine already has a VM in another Public cluster cannot join; create a Private cluster to use both. Wait for the new workers to become Ready, then check their GPU capacity before increasing your workload. After an added VM worker becomes Ready, Ornn refreshes the supplied namespace’s GPU quota to include its GPUs. A host from a different reservation is labeled and tainted with the reservation the cluster launched on, so Pods in that namespace still schedule on it. After new workers are Ready, run kubectl describe resourcequota ornn-gpu-quota, and contact support if the quota does not cover the GPUs you plan to request. To remove workers, select them in Nodes and use Remove. Removal drains those workers and interrupts workloads using them. Export local data and checkpoint first. Review remaining GPU capacity and pending workloads afterward. Removing a worker does not cancel its reservation. A removed VM keeps running and gets its SSH keys back. To add and remove workers from your own automation, see Scale workers up or down.

Stop, tear down, or retry

In the cluster list, hover over a row or focus it with the keyboard to reveal Delete cluster. On mobile, the delete button is always visible. Select it and confirm to tear down the cluster. Reservation billing continues. Use Stop launch while provisioning, or Tear down cluster for an active cluster, from the manage page’s actions menu. Confirm the affected nodes and wait for cleanup to finish. Teardown removes workloads, revokes cluster access, and releases workers back to their reservations. A failed or torn-down cluster can disappear from Orchestration after cleanup. Start a new launch from your available reserved nodes. Do not rely on the old URL or downloaded credentials.
Save checkpoints and export local data before teardown. Deleting a test Job is different from tearing down its cluster. Teardown does not cancel the reserved term or delete separate storage resources.

Troubleshooting

Confirm the correct organization, an active reservation, and available hosts. Offline nodes and nodes held by another cluster cannot be selected. If capacity is missing, contact support with the reservation ID.
If the VM left the cluster with a message that another machine already uses its network address, choose a different VM or relaunch with Private networking. If Ornn says only one VM per machine can be in a public cluster, relaunch with Private networking to include that VM. If its host needs a management update, contact support with the VM ID.
Confirm KUBECONFIG points to this cluster’s file. For Private access, activate the matching WireGuard tunnel first. Check your local firewall and VPN routes. A small ping succeeding does not prove the API works; keep the generated MTU of 1200.
For Unauthorized, confirm you selected this cluster’s downloaded file. Its token expires seven days after it is issued. Download the kubeconfig again to get the current token, which Ornn renews before it expires. Contact support if a fresh download is also rejected; do not tear down a working cluster just to renew access. For Forbidden, check the context and identity. After a relaunch, use the new cluster credentials. Do not print or share tokens.
Run kubectl describe pod <pod-name>, kubectl describe job <job-name>, and kubectl describe resourcequota. Check GPU vendor, per-worker capacity, quota, CPU and memory, and volume placement. Use the supplied namespace so its scheduling defaults apply.
Read pod events. Check the image name and tag, registry access, and your imagePullSecrets. A registry can reject a pull because of authentication, permissions, rate limits, or an unavailable image; the error alone does not establish which cause applies. For an image you pushed to the Ornn container registry, confirm the name starts with the Image registry value and wait a few minutes after a push for the cluster’s copy.
Check the locked-memory limits inside your training container with ulimit -Sl and ulimit -Hl. Both should report unlimited. If either reports 8192 (8 MiB), contact Support with the cluster and pod IDs. Correcting an affected worker requires a coordinated runtime restart and a new workload pod; changing your login shell’s limit does not update an existing training process.
For local-path, create the consumer pod and inspect its scheduling events. Local volumes cannot move between workers. For /workspace, check that the cluster is Private, the filesystem is ready, and the attachment is active before testing from a pod.
Open the progress details and record the failed stage and customer error. Avoid repeated launches while cleanup is in progress. Contact support with the reservation or cluster ID, approximate failure time, and sanitized error text.

Monitor GPUs with Prometheus

Every NVIDIA GPU worker runs NVIDIA DCGM Exporter on port 9400. AMD Instinct workers do not, so they have no series here. Each GPU series carries the pod, namespace, and container of the pod that requested that GPU, plus UUID, gpu, modelName, and Hostname (the worker). The headless Service ornn-dcgm-exporter in the gpu-operator namespace returns one address per NVIDIA GPU worker. Confirm discovery from a pod in the supplied namespace. Expect one address per NVIDIA GPU worker:
Add this job to a Prometheus you run in the supplied namespace, where the reservation scheduling defaults apply:
The exporter samples every 30 seconds, so a shorter interval repeats values. To keep data in an existing monitoring stack, run Prometheus in agent mode and send it there with remote_write.
  • Attribution. Only GPUs requested through nvidia.com/gpu carry pod labels. A GPU that no pod requested has an empty pod label. Count distinct UUID values for total capacity.
  • Public clusters. If scrapes of exporters on other workers time out, run one scraper per worker that reads only the exporter on its own worker, then forward with remote_write.
  • Prometheus Operator. The cluster ships a placeholder ServiceMonitor definition. A chart that installs the Prometheus Operator definitions as templates fails on ownership; a plain scrape job like the one above does not depend on it.

Monitor and get help

Use the cluster’s Observability tab for host metrics and kernel console logs. Both views fill the available tab width. Use kubectl logs, describe, and events for application failures. Host metrics do not prove a training job completed. Open Support in the console or email support@ornn.com. Include the cluster and reservation IDs, workload name, failed stage, and time. Never include kubeconfig contents, private keys, registry secrets, or tokens.

What’s next

access

Kubernetes API automation

Provision, scale, and tear down clusters from your own automation with an API key.
access

Slurm access

Prefer batch scheduling? Run a managed Slurm cluster on your reserved GPUs instead.
access

Access overview

Compare the ways to access your reserved Ornn compute.