Automate managed Kubernetes with the Ornn API

You can provision and manage GPU capacity in Ornn-managed Kubernetes entirely through APIs. A platform team needs only an Ornn API key and an HTTPS client. The console and the Ornn CLI are optional. This guide walks through the full lifecycle: find reserved hosts, create a cluster, wait for it, fetch its kubeconfig, run a GPU Job, scale workers, and tear the cluster down. Every step shows the Ornn CLI, curl, and Python.

Two APIs, two jobs

Layer	Address	Credential	Use it to
Ornn REST API	`https://api.ornn.com`	Ornn API key: `Authorization: Bearer ornn_sk_...`	Find reserved hosts, create a cluster, follow its progress, fetch the kubeconfig, add or remove workers, tear down
Kubernetes API	The cluster’s `endpoint`, such as `https://cluster.example.com:6443`	The kubeconfig Ornn issues for the cluster	Run Jobs, Deployments, and Pods that request `nvidia.com/gpu`; read Node readiness, namespaces, and quotas

The Ornn API manages the cluster and its capacity. The standard Kubernetes API runs your workloads on that capacity. Use any Kubernetes client library or kubectl. Never send the Ornn API key to the Kubernetes API server, or the kubeconfig token to api.ornn.com. The API reference documents every Ornn operation used here. Download the OpenAPI contract to generate a client.

What the API does not do

No gRPC or GraphQL. The Ornn API is JSON over HTTPS.
No purchasing. A cluster runs on hosts your organization already reserved. Reserve capacity before you automate launches.
No automatic scaling of Ornn capacity. Ornn does not provide a Cluster Autoscaler or Karpenter integration, and pending Pods do not add workers. To change capacity, your automation calls the add and remove worker endpoints.
One cluster at a time. Your organization can run one Kubernetes cluster, and cannot run a Kubernetes cluster and a Slurm cluster at the same time. A cluster counts until its teardown finishes; a failed launch does not count. Grow the cluster you have by adding workers.
No credential renewal. The kubeconfig token is issued for seven days. Reading access again returns the same token. Contact Ornn support for new credentials.
No external clusters. You cannot attach Ornn workers to an existing EKS, GKE, or AKS cluster.
No single-cluster read. There is no GET for one cluster. Read the cluster list and select by ID.

Before you start

Belong to an approved organization and create an API key in Account Settings → Developer.
Hold an active reservation with any required payment complete. Its hosts must be online.
Save data on the hosts you plan to use. Ornn prepares each host before it joins a cluster: running processes are stopped, the tenant home directory is wiped, and any VM on the host is removed.
For Python, install requests and the Kubernetes Python client: pip install requests kubernetes. For shell, install curl and jq.
If you choose a Private cluster, the machine that calls the Kubernetes API needs an active WireGuard connection to your organization’s network. See Private access with WireGuard.

The Ornn CLI signs in with ornn login and does not accept API keys. In scripts, pass --format json, --yes, and a stable --idempotency-key to CLI commands that change resources. See Human and automation modes. Set your environment once:

export ORNN_API_URL="https://api.ornn.com"
export ORNN_API_KEY="ornn_sk_..."   # load from your secrets manager

The Python examples build on one another. Start with this helper, which raises an error carrying the HTTP status and detail.code, and retries only requests that are safe to retry:

import os
import time

import requests

API = "https://api.ornn.com"
http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['ORNN_API_KEY']}"


class OrnnError(Exception):
    def __init__(self, response):
        self.status = response.status_code
        try:
            body = response.json()
        except ValueError:
            body = {}
        detail = body.get("detail") if isinstance(body, dict) else None
        if isinstance(detail, dict):
            self.code, message = detail.get("code"), detail.get("message")
        else:
            self.code = body.get("code") if isinstance(body, dict) else None
            message = detail
        self.body = body
        super().__init__(f"HTTP {self.status} {self.code}: {message}")


def call(method, path, *, ok=(200,), **kwargs):
    """Send one request. Retry rate limits on any method and 5xx on reads only."""
    for attempt in range(6):
        response = http.request(method, API + path, timeout=30, **kwargs)
        retryable = response.status_code == 429 or (
            method == "GET" and response.status_code >= 500
        )
        if retryable and attempt < 5:
            time.sleep(float(response.headers.get("Retry-After", 2**attempt)))
            continue
        if response.status_code not in ok:
            raise OrnnError(response)
        return response.json() if response.content else None

A 429 is returned before your request reaches Ornn’s services, so any method is safe to retry after Retry-After. Do not blindly retry a write after a timeout or 5xx; read the current state first, as each step below describes.

1. Find a reservation and eligible hosts

List your active reservations, then list the hosts on the one you choose. A host can join as a worker when all of these are true:

status is bare-metal.
controller_id is null, so it is not in another cluster.
health_readiness.ready is true.

health_readiness.reasons containing heartbeat:stale means the host has not reported recently, and launch refuses it. heartbeat:unknown in advisory_reasons means Ornn has no heartbeat evidence either way; it does not block launch, but it does not prove the host is reachable. Node count means hosts, not GPUs.

ornn context --format json
ornn reservations list --status active --format json
# ornn launch lists eligible hosts for review in a terminal.
# Automation passes exact host IDs, which you can read with the API.

curl --fail-with-body -sS "$ORNN_API_URL/commerce/reservations?status=active&limit=100" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq '.reservations[] | {reservationId, gpuSlug, reservedNodeCount, startsAt, endsAt}'

export RESERVATION_ID='11111111-1111-4111-8111-111111111111'

curl --fail-with-body -sS \
  "$ORNN_API_URL/v1/orchestrator/nodes?reservation_id=$RESERVATION_ID" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq '[.nodes[]
      | select(.status == "bare-metal" and .controller_id == null and .health_readiness.ready)
      | {node_id, gpu_type, gpu_count, advisory: .health_readiness.advisory_reasons}]'

reservations, cursor = [], None
while True:
    params = {"status": "active", "limit": 100}
    if cursor:
        params["cursor"] = cursor
    page = call("GET", "/commerce/reservations", params=params)
    reservations += page["reservations"]
    cursor = page.get("nextCursor")
    if not cursor:
        break

RESERVATION_ID = "11111111-1111-4111-8111-111111111111"  # the reservation you chose

nodes = call(
    "GET", "/v1/orchestrator/nodes", params={"reservation_id": RESERVATION_ID}
)["nodes"]
eligible = [
    node["node_id"]
    for node in nodes
    if node["status"] == "bare-metal"
    and node["controller_id"] is None
    and node.get("health_readiness", {}).get("ready") is True
]
print(eligible)

The CLI’s automation form needs your organization ID; ornn context --format json returns it, and so does GET /v1/auth/session with your API key.

2. Create the cluster

Creating a cluster prepares every selected host, which wipes its tenant home directory and removes any VM on it. This request starts provisioning; it is not a dry run.

Send kind, reservation_id, and the exact node_ids to use. Listing hosts explicitly is the safe choice: if you omit both node_ids and virtual_machine_ids, Ornn uses every bare-metal host on the reservation. Set network_mode to private, and optionally network_id, for a cluster reachable only over WireGuard.

export NODE_ID='33333333-3333-4333-8333-333333333333'
export ORGANIZATION_ID='22222222-2222-4222-8222-222222222222'

ornn launch --mode kubernetes --reservation "$RESERVATION_ID" \
  --nodes 1 --machines "$NODE_ID" --network public \
  --organization "$ORGANIZATION_ID" --yes \
  --idempotency-key "k8s-launch-$RESERVATION_ID" --format json

export NODE_ID='33333333-3333-4333-8333-333333333333'

jq -n --arg reservation "$RESERVATION_ID" --arg node "$NODE_ID" \
  '{kind: "kubernetes", reservation_id: $reservation,
    network_mode: "public", node_ids: [$node]}' \
  | curl --fail-with-body -sS -X POST "$ORNN_API_URL/v1/orchestrator/controllers" \
      -H "Authorization: Bearer $ORNN_API_KEY" \
      -H 'Content-Type: application/json' \
      --data-binary @- > launch.json

export CLUSTER_ID="$(jq -er '.id' launch.json)"

LIVE_STATES = ("pending", "provisioning", "active", "deleting")


def find_cluster(cluster_id=None):
    """Return the {controller, nodes} entry for a cluster, or None.

    Without an ID, return your organization's live Kubernetes cluster. An
    organization runs at most one at a time.
    """
    listing = call("GET", "/v1/orchestrator/controllers", params={"kind": "kubernetes"})
    for entry in listing["controllers"]:
        controller = entry["controller"]
        if controller["id"] == cluster_id or (
            cluster_id is None and controller["state"] in LIVE_STATES
        ):
            return entry
    return None


existing = find_cluster()
state = existing["controller"]["state"] if existing else None
if state == "deleting":
    raise RuntimeError("A teardown is still running. Wait for it to finish.")
if state in ("pending", "provisioning", "active"):
    CLUSTER_ID = existing["controller"]["id"]  # reuse it; do not launch again
else:  # no cluster for this reservation yet, or the last launch failed
    accepted = call(
        "POST",
        "/v1/orchestrator/controllers",
        ok=(202,),
        json={
            "kind": "kubernetes",
            "reservation_id": RESERVATION_ID,
            "network_mode": "public",
            "node_ids": eligible[:1],
        },
    )
    CLUSTER_ID = accepted["id"]
print("cluster", CLUSTER_ID)

Expect 202 Accepted with the new cluster’s id, state: provisioning, and its namespace. That response means the launch was accepted, not that anything is ready; node_count and gpu_count are usually 0 until workers attach. Create is safe to repeat for the same reservation, but it is not a scaling operation:

If the reservation’s cluster is active, or provisioning with workers attached, the request returns that cluster unchanged with 202, even if node_ids differs.
If the last launch on the reservation failed, the request relaunches it under the same id.
Ornn does not read an Idempotency-Key header on this route. After a timeout or 5xx, list your Kubernetes clusters before sending the request again, as the Python example does. Your organization runs at most one at a time, so a live cluster in that list is the one your earlier request created.

Response	`detail.code`	What to do
`409`	`cluster_access_mode_conflict`	Your organization runs a Slurm cluster (`live_mode`). Tear it down and wait for it to finish.
`409`	`kubernetes_cluster_ambiguous_group`	Your organization already runs a Kubernetes cluster on another reservation. Add workers to it.
`409`	`launch_node_offline`	The host in `node_id` has not sent a heartbeat recently. Bring it online or choose another.
`409`	`cluster_nodes_not_available`	A host is not in `bare-metal` status or no longer exists. Refresh the host list.
`409`	`reservation_not_launchable`	The reservation ended, was cancelled, or has a pending change.
`404`	`reservation_not_found`, `node_not_found`	The reservation or a host is not visible to your organization. Check the IDs.
`422`	`invalid_network_mode`	Use `public` or `private`.

3. Wait for the cluster to become active

Poll the cluster list every 10 seconds and select your cluster by controller.id. Read three things:

controller.state: pending or provisioning while launching, active when every launch worker has joined and access has been issued, failed when the launch could not finish.
controller.progress.stages: launch stages create_plane, prepare_hosts, join_workers, apply_plane, and issue_access, each pending, running, completed, or failed. A failed stage’s detail carries the reason.
nodes[].state for each worker: joining, then ready; join_failed means that host could not join.

Creating the control plane, preparing hosts, and pulling images take several minutes. A launch whose workers cannot join is marked failed after about 30 minutes, so a client deadline of an hour leaves room for the rest. If your deadline passes while the cluster is still provisioning, contact Ornn support with the cluster ID; do not launch again. If a worker stays joining, check its host’s health_readiness in the host list from step 1. heartbeat:stale means the host went offline.

ornn clusters wait "$CLUSTER_ID" --timeout 60m --format json

deadline=$((SECONDS + 3600))
while :; do
  entry="$(curl --fail-with-body -sS \
    "$ORNN_API_URL/v1/orchestrator/controllers?kind=kubernetes" \
    -H "Authorization: Bearer $ORNN_API_KEY" \
    | jq -c --arg id "$CLUSTER_ID" '.controllers[] | select(.controller.id == $id)')"
  state="$(jq -r '.controller.state // "missing"' <<<"${entry:-null}")"
  echo "state=$state $(jq -c '[.nodes[]? | {node_id, state}]' <<<"${entry:-null}")"
  case "$state" in active|failed|missing) break ;; esac
  [ "$SECONDS" -lt "$deadline" ] || break
  sleep 10
done
jq '.controller.progress' <<<"${entry:-null}"
# Continue only if the last line printed state=active.

deadline = time.monotonic() + 3600
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        raise RuntimeError("The cluster is no longer listed; it may have been torn down.")
    controller = entry["controller"]
    stages = (controller.get("progress") or {}).get("stages", [])
    print(
        controller["state"],
        [(stage["key"], stage["status"]) for stage in stages],
        [(node["node_id"], node["state"]) for node in entry["nodes"]],
    )
    if controller["state"] == "active":
        break
    if controller["state"] == "failed":
        raise RuntimeError(f"Launch failed: {stages}")
    if time.monotonic() > deadline:
        raise TimeoutError("Still provisioning; contact Ornn with the cluster ID.")
    time.sleep(10)

active means Ornn finished its part. It does not prove each Kubernetes Node is Ready or advertises GPUs; step 5 checks that through the Kubernetes API.

4. Fetch the kubeconfig

Request the cluster’s access. The response contains kubeconfig, the API server endpoint, and namespace, the namespace Ornn configured for your workloads. A 404 with kubernetes_credentials_not_ready means access has not been issued yet; keep polling step 3. Any other 404 means the cluster ID is wrong or not visible to your organization. The kubeconfig’s token has cluster-admin rights and is valid for seven days. Store it like a password, with owner-only file permissions.

ornn connect "$CLUSTER_ID" --output ./ornn-kubeconfig.yaml --format json
export KUBECONFIG="$PWD/ornn-kubeconfig.yaml"

umask 077
curl --fail-with-body -sS \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/access" \
  -H "Authorization: Bearer $ORNN_API_KEY" -o access.json
jq -er '.kubeconfig' access.json > ornn-kubeconfig.yaml
export NAMESPACE="$(jq -er '.namespace' access.json)"
rm access.json
export KUBECONFIG="$PWD/ornn-kubeconfig.yaml"

while True:
    try:
        access = call("GET", f"/v1/orchestrator/controllers/{CLUSTER_ID}/access")
        break
    except OrnnError as error:
        if error.status == 404 and error.code == "kubernetes_credentials_not_ready":
            time.sleep(10)
            continue
        raise

KUBECONFIG_PATH = os.path.abspath("ornn-kubeconfig.yaml")
descriptor = os.open(KUBECONFIG_PATH, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
os.fchmod(descriptor, 0o600)  # also tightens a file that already existed
with os.fdopen(descriptor, "w") as handle:
    handle.write(access["kubeconfig"])
NAMESPACE = access["namespace"]
print(access["endpoint"], NAMESPACE)

The CLI writes the file with 0600 permissions and refuses to overwrite an existing file unless you add --overwrite.

5. Run a GPU smoke Job through the Kubernetes API

From here on you talk to the cluster’s own API server with the kubeconfig. Run GPU work in the namespace from step 4: in that namespace, Ornn’s scheduling rules place Pods on your reserved workers, and a ResourceQuota named ornn-gpu-quota caps the GPUs its Pods can request. Namespaces you create yourself do not receive those scheduling rules. This NVIDIA example checks the Nodes, requests one GPU, runs nvidia-smi, and waits up to five minutes. The Node list also shows the managed control-plane Nodes, which are not GPU capacity. For AMD capacity, use an image with ROCm, request amd.com/gpu, and run amd-smi list.

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,GPUS:.status.allocatable.nvidia\.com/gpu'
kubectl describe resourcequota ornn-gpu-quota

kubectl apply -f - <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: ornn-gpu-smoke
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 240
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: nvidia/cuda:12.4.1-base-ubuntu22.04
          command: ["nvidia-smi"]
          resources:
            limits:
              nvidia.com/gpu: 1
EOF

kubectl wait --for=condition=complete job/ornn-gpu-smoke --timeout=300s
kubectl logs job/ornn-gpu-smoke
kubectl delete job ornn-gpu-smoke

# The kubeconfig Ornn returns has one cluster and one user.
KUBE_SERVER="$(awk '/^    server: / {print $2}' ornn-kubeconfig.yaml)"
KUBE_TOKEN="$(awk '/^    token: / {print $2}' ornn-kubeconfig.yaml)"
awk '/^    certificate-authority-data: / {print $2}' ornn-kubeconfig.yaml \
  | base64 --decode > ornn-ca.crt
kube() {
  curl --fail-with-body -sS --cacert ornn-ca.crt \
    -H "Authorization: Bearer $KUBE_TOKEN" "$@"
}

kube "$KUBE_SERVER/api/v1/nodes" | jq '.items[] | {name: .metadata.name,
  ready: ([.status.conditions[] | select(.type == "Ready")][0].status),
  gpus: .status.allocatable["nvidia.com/gpu"]}'

jq -n '{apiVersion: "batch/v1", kind: "Job", metadata: {name: "ornn-gpu-smoke"},
  spec: {backoffLimit: 0, activeDeadlineSeconds: 240, template: {spec: {
    restartPolicy: "Never",
    containers: [{name: "smoke", image: "nvidia/cuda:12.4.1-base-ubuntu22.04",
      command: ["nvidia-smi"], resources: {limits: {"nvidia.com/gpu": 1}}}]}}}}' \
  | kube -X POST -H 'Content-Type: application/json' --data-binary @- \
      "$KUBE_SERVER/apis/batch/v1/namespaces/$NAMESPACE/jobs" > /dev/null

for _ in $(seq 60); do
  status="$(kube "$KUBE_SERVER/apis/batch/v1/namespaces/$NAMESPACE/jobs/ornn-gpu-smoke" | jq -c .status)"
  [ "$(jq -r '(.succeeded // 0) + (.failed // 0)' <<<"$status")" -ge 1 ] && break
  sleep 5
done
echo "$status"   # expect "succeeded":1
pod="$(kube "$KUBE_SERVER/api/v1/namespaces/$NAMESPACE/pods?labelSelector=job-name%3Dornn-gpu-smoke" \
  | jq -er '.items[0].metadata.name')"
kube "$KUBE_SERVER/api/v1/namespaces/$NAMESPACE/pods/$pod/log"
kube -X DELETE \
  "$KUBE_SERVER/apis/batch/v1/namespaces/$NAMESPACE/jobs/ornn-gpu-smoke?propagationPolicy=Foreground" > /dev/null

import uuid

from kubernetes import client, config

config.load_kube_config(config_file=KUBECONFIG_PATH)
core = client.CoreV1Api()
batch = client.BatchV1Api()

for node in core.list_node(_request_timeout=30).items:
    ready = any(c.type == "Ready" and c.status == "True" for c in node.status.conditions or [])
    gpus = (node.status.allocatable or {}).get("nvidia.com/gpu", "0")
    print(node.metadata.name, "Ready:", ready, "GPUs:", gpus)
quota = core.read_namespaced_resource_quota("ornn-gpu-quota", NAMESPACE, _request_timeout=30)
print("GPU quota:", quota.status.hard, "used:", quota.status.used)

name = "ornn-gpu-smoke-" + uuid.uuid4().hex[:8]
batch.create_namespaced_job(
    NAMESPACE,
    {
        "apiVersion": "batch/v1",
        "kind": "Job",
        "metadata": {"name": name},
        "spec": {
            "backoffLimit": 0,
            "activeDeadlineSeconds": 240,
            "template": {
                "spec": {
                    "restartPolicy": "Never",
                    "containers": [
                        {
                            "name": "smoke",
                            "image": "nvidia/cuda:12.4.1-base-ubuntu22.04",
                            "command": ["nvidia-smi"],
                            "resources": {"limits": {"nvidia.com/gpu": 1}},
                        }
                    ],
                }
            },
        },
    },
    _request_timeout=30,
)
deadline = time.monotonic() + 300
while True:
    status = batch.read_namespaced_job_status(name, NAMESPACE, _request_timeout=30).status
    if status.succeeded:
        break
    if status.failed or time.monotonic() > deadline:
        raise RuntimeError(f"Job {name} did not succeed; inspect its Pods and events.")
    time.sleep(5)

pods = core.list_namespaced_pod(NAMESPACE, label_selector=f"job-name={name}", _request_timeout=30)
for pod in pods.items:
    print(core.read_namespaced_pod_log(pod.metadata.name, NAMESPACE, _request_timeout=30))
batch.delete_namespaced_job(name, NAMESPACE, propagation_policy="Foreground", _request_timeout=30)

Success means each worker Node is Ready with GPUs allocatable, the Job completes, and its log lists an NVIDIA GPU. A failed or timed-out Job stays for inspection; check its Pods and events before deleting it. Deleting the Job removes only the test, not the cluster or reservation. For workload guidance such as storage, MPI, images, and queueing, see Run your application.

6. Scale workers up or down

Scaling means adding or removing whole hosts through the Ornn API. Kubernetes autoscalers change Pod replicas, not Ornn hosts.

Add a worker

Choose an eligible host as in step 1, preferably from the reservation the cluster launched on, and add it. A new host normally returns state: cleaning while Ornn prepares it. It then appears in the cluster’s nodes as joining, then ready. Repeating the request is safe: it returns cleaning until preparation finishes, then the member’s current state.

export WORKER_NODE_ID='77777777-7777-4777-8777-777777777777'

ornn clusters eligible-machines "$CLUSTER_ID" --format json
ornn clusters add-node "$CLUSTER_ID" "$WORKER_NODE_ID" --yes \
  --idempotency-key "add-$WORKER_NODE_ID" --format json --wait --timeout 45m

export WORKER_NODE_ID='77777777-7777-4777-8777-777777777777'

curl --fail-with-body -sS -X POST \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/nodes/$WORKER_NODE_ID" \
  -H "Authorization: Bearer $ORNN_API_KEY"

# Repeat until the worker is listed as ready.
curl --fail-with-body -sS "$ORNN_API_URL/v1/orchestrator/controllers?kind=kubernetes" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq --arg id "$CLUSTER_ID" --arg node "$WORKER_NODE_ID" \
      '.controllers[] | select(.controller.id == $id) | .nodes[] | select(.node_id == $node)'

WORKER_NODE_ID = "77777777-7777-4777-8777-777777777777"

change = call(
    "POST", f"/v1/orchestrator/controllers/{CLUSTER_ID}/nodes/{WORKER_NODE_ID}", ok=(202,)
)
print(change["state"])  # usually "cleaning"

deadline = time.monotonic() + 2700
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        raise RuntimeError("The cluster is no longer listed.")
    member = next((n for n in entry["nodes"] if n["node_id"] == WORKER_NODE_ID), None)
    if member and member["state"] == "ready":
        break
    if member and member["state"] == "join_failed":
        raise RuntimeError("The worker could not join; contact Ornn with both IDs.")
    if time.monotonic() > deadline:
        raise TimeoutError("The worker has not joined yet.")
    time.sleep(10)

Adding a worker does not raise ornn-gpu-quota, which Ornn sizes from the cluster’s workers when it first becomes active. A host from a different reservation is labeled with its own reservation, while Pods in your namespace are pinned to the reservation the cluster launched on, so they do not schedule there. After the worker is ready, confirm in Kubernetes that its Node is Ready with GPUs and that the quota covers the GPUs you plan to request. Contact Ornn support if it does not.

Add worker errors include 404 controller_not_found when the cluster is failed or being torn down, 404 node_not_found when the host is not yours, and 409 cluster_nodes_not_available when the host is not in bare-metal status. A 409 with an empty JSON object means the host already belongs to another cluster.

Remove a worker

Removing a worker stops the workloads on it and can lose its local data. Checkpoint first. Removing a worker does not cancel its reservation or stop its billing.

Optionally drain the Node in Kubernetes first so Pods move off it gracefully. Then remove it. The response is 200 with state: draining, which means cleanup was sent. The worker leaves the cluster’s nodes once its host confirms cleanup. cleanup_failed means the host reported that cleanup did not finish; contact Ornn before reusing it. Repeating the request while the worker drains sends cleanup again; after it has left, the request returns 404 node_not_found.

ornn clusters remove-node "$CLUSTER_ID" "$WORKER_NODE_ID" --yes \
  --idempotency-key "remove-$WORKER_NODE_ID" --format json --wait --timeout 30m

curl --fail-with-body -sS -X DELETE \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/nodes/$WORKER_NODE_ID" \
  -H "Authorization: Bearer $ORNN_API_KEY"

# Repeat until this prints nothing.
curl --fail-with-body -sS "$ORNN_API_URL/v1/orchestrator/controllers?kind=kubernetes" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq --arg id "$CLUSTER_ID" --arg node "$WORKER_NODE_ID" \
      '.controllers[] | select(.controller.id == $id) | .nodes[] | select(.node_id == $node)'

call("DELETE", f"/v1/orchestrator/controllers/{CLUSTER_ID}/nodes/{WORKER_NODE_ID}")

deadline = time.monotonic() + 1800
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        raise RuntimeError("The cluster is no longer listed.")
    member = next((n for n in entry["nodes"] if n["node_id"] == WORKER_NODE_ID), None)
    if member is None:
        break
    if member["state"] == "cleanup_failed":
        raise RuntimeError("Cleanup did not finish; contact Ornn before reusing the host.")
    if time.monotonic() > deadline:
        raise TimeoutError("The worker is still draining.")
    time.sleep(10)

Drive scaling from your own controller

Because Ornn does not watch your Pods, a scaling loop lives in your automation. A typical loop reads pending GPU Pods and Node capacity from the Kubernetes API, decides how many hosts it needs, and calls the add or remove worker endpoint for one host at a time, waiting for ready or removal before the next change. Scaling is bounded by the hosts your organization has reserved; it never buys capacity.

7. Tear down the cluster and confirm

Teardown deletes the cluster’s workloads and revokes its kubeconfig. Export data first. Reservation billing continues, and separately provisioned shared filesystems are not deleted.

The teardown response is 202 with state: deleting. Ornn deletes the cluster’s namespace and workloads, then waits up to five minutes for each worker to confirm cleanup before removing the cluster. Hosts that never confirm are released anyway and cleaned when they reconnect. Repeating the request during teardown returns {"state": "deleting"} and does not restart cleanup. Teardown is complete when both of these are true:

The cluster no longer appears in the cluster list.
GET /v1/orchestrator/controllers/{cluster_id}/access returns 404 with cluster_not_found.

After that, teardown itself returns 404 with no body. If the state becomes failed instead, send the teardown again, and contact Ornn support if it fails twice. Wait for teardown to finish before you create another cluster.

ornn clusters teardown "$CLUSTER_ID" --yes \
  --idempotency-key "teardown-$CLUSTER_ID" --format json --wait --timeout 30m

curl --fail-with-body -sS -X POST \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/teardown" \
  -H "Authorization: Bearer $ORNN_API_KEY"

# Repeat until this prints cluster_not_found.
curl -sS "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/access" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq -r '.detail.code? // "teardown still running"'

try:
    call("POST", f"/v1/orchestrator/controllers/{CLUSTER_ID}/teardown", ok=(202,))
except OrnnError as error:
    if error.status != 404:  # 404 means the teardown already finished
        raise

deadline = time.monotonic() + 1800
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        try:
            call("GET", f"/v1/orchestrator/controllers/{CLUSTER_ID}/access")
        except OrnnError as error:
            if error.status == 404 and error.code == "cluster_not_found":
                break
            raise
    elif entry["controller"]["state"] == "failed":
        raise RuntimeError("Teardown failed; send it again, then contact Ornn.")
    if time.monotonic() > deadline:
        raise TimeoutError("Teardown has not finished.")
    time.sleep(10)
print("Cluster removed. The reservation and its billing continue.")

After teardown, the hosts return to your reservation. Delete your local kubeconfig; a new cluster issues new credentials.

Retries and idempotency

Operation	Safe to repeat?	What a repeat returns
List clusters, hosts, access	Yes	The current state. Retry `5xx` with backoff.
Create cluster	Yes, for the same reservation	The existing cluster when it is `active`, or `provisioning` with workers; a relaunch under the same `id` after `failed`.
Add worker	Yes	`cleaning` while the host is prepared, then the member’s current state.
Remove worker	Yes	`draining` while cleanup runs; `404 node_not_found` once the worker has left.
Tear down	Yes	`{"state": "deleting"}` while it runs; `404` with no body once it has finished.

The /v1/orchestrator routes share a limit of 240 requests per minute for each user. Every API key you create counts against your limit, and so does your console session. Polling every 10 seconds uses a small fraction of it. A 429 includes Retry-After. Cluster operations return errors as {"detail": {"code": "...", "message": "..."}}; show detail.message to people and branch on detail.code. Authentication and organization-approval errors use {"detail": "...", "code": "..."}. Malformed JSON, unknown request fields, or a missing Content-Type return 400, 422, or 415 with a plain-text body. See errors and safe retries.

Kubernetes access

Console launch, networking, storage, and running applications on your cluster.

Cluster API reference

Request and response schemas for every cluster and worker operation.

Get Started

Buy and sell

Supplier

Accessing Compute

Storage

Nodes

Account

Automate managed Kubernetes with the Ornn API

Two APIs, two jobs

What the API does not do

Before you start

1. Find a reservation and eligible hosts

2. Create the cluster

3. Wait for the cluster to become active

4. Fetch the kubeconfig

5. Run a GPU smoke Job through the Kubernetes API

6. Scale workers up or down

Add a worker

Remove a worker

Drive scaling from your own controller

7. Tear down the cluster and confirm

Retries and idempotency

Kubernetes access

Cluster API reference

​Two APIs, two jobs

​What the API does not do

​Before you start

​1. Find a reservation and eligible hosts

​2. Create the cluster

​3. Wait for the cluster to become active

​4. Fetch the kubeconfig

​5. Run a GPU smoke Job through the Kubernetes API

​6. Scale workers up or down

​Add a worker

​Remove a worker

​Drive scaling from your own controller

​7. Tear down the cluster and confirm

​Retries and idempotency

​Related

Kubernetes access

Cluster API reference

Two APIs, two jobs

What the API does not do

Before you start

1. Find a reservation and eligible hosts

2. Create the cluster

3. Wait for the cluster to become active

4. Fetch the kubeconfig

5. Run a GPU smoke Job through the Kubernetes API

6. Scale workers up or down

Add a worker

Remove a worker

Drive scaling from your own controller

7. Tear down the cluster and confirm

Retries and idempotency

Related