> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ornn.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Automate managed Kubernetes with the Ornn API

> Provision, scale, and tear down GPU Kubernetes clusters with an API key, then run GPU workloads through the Kubernetes API.

You can provision and manage GPU capacity in Ornn-managed Kubernetes entirely through APIs. A platform team needs only an [Ornn API key](/api-keys) and an HTTPS client. The console and the Ornn CLI are optional.

This guide walks through the full lifecycle: find reserved hosts, create a cluster, wait for it, fetch its kubeconfig, run a GPU Job, scale workers, and tear the cluster down. Every step shows the Ornn CLI, `curl`, and Python.

## Two APIs, two jobs

| Layer | Address | Credential | Use it to |
| - | - | - | - |
| Ornn REST API | `https://api.ornn.com` | Ornn API key: `Authorization: Bearer ornn_sk_...` | Find reserved hosts, create a cluster, follow its progress, fetch the kubeconfig, add or remove workers, tear down |
| Kubernetes API | The cluster's `endpoint`, such as `https://cluster.example.com:6443` | The kubeconfig Ornn issues for the cluster | Run Jobs, Deployments, and Pods that request `nvidia.com/gpu`; read Node readiness, namespaces, and quotas |

The Ornn API manages the cluster and its capacity. The standard Kubernetes API runs your workloads on that capacity. Use any [Kubernetes client library](https://kubernetes.io/docs/reference/using-api/client-libraries/) or `kubectl`. Never send the Ornn API key to the Kubernetes API server, or the kubeconfig token to `api.ornn.com`.

The [API reference](/api-reference/overview) documents every Ornn operation used here. [Download the OpenAPI contract](/openapi.yaml) to generate a client.

## What the API does not do

* **No gRPC or GraphQL.** The Ornn API is JSON over HTTPS.
* **No purchasing.** A cluster runs on hosts your organization already reserved. Reserve capacity before you automate launches.
* **No automatic scaling of Ornn capacity.** Ornn does not provide a Cluster Autoscaler or Karpenter integration, and pending Pods do not add workers. To change capacity, your automation calls the add and remove worker endpoints.
* **One cluster at a time.** Your organization can run one Kubernetes cluster, and cannot run a Kubernetes cluster and a Slurm cluster at the same time. A cluster counts until its teardown finishes; a failed launch does not count. Grow the cluster you have by adding workers.
* **No credential renewal.** The kubeconfig token is issued for seven days. Reading access again returns the same token. Contact Ornn support for new credentials.
* **No external clusters.** You cannot attach Ornn workers to an existing EKS, GKE, or AKS cluster.
* **No single-cluster read.** There is no `GET` for one cluster. Read the cluster list and select by ID.

## Before you start

* Belong to an approved organization and [create an API key](/api-keys) in **Account Settings** → **Developer**.
* Hold an active reservation with any required payment complete. Its hosts must be online.
* Save data on the hosts you plan to use. Ornn prepares each host before it joins a cluster: running processes are stopped, the tenant home directory is wiped, and any VM on the host is removed.
* For Python, install `requests` and the [Kubernetes Python client](https://github.com/kubernetes-client/python): `pip install requests kubernetes`. For shell, install `curl` and `jq`.
* If you choose a Private cluster, the machine that calls the Kubernetes API needs an active WireGuard connection to your organization's network. See [Private access with WireGuard](/guides/kubernetes-access#private-access-with-wireguard).

The Ornn CLI signs in with `ornn login` and does not accept API keys. In scripts, pass `--format json`, `--yes`, and a stable `--idempotency-key` to CLI commands that change resources. See [Human and automation modes](/cli#human-and-automation-modes).

Set your environment once:

```bash theme={null}
export ORNN_API_URL="https://api.ornn.com"
export ORNN_API_KEY="ornn_sk_..."   # load from your secrets manager
```

The Python examples build on one another. Start with this helper, which raises an error carrying the HTTP status and `detail.code`, and retries only requests that are safe to retry:

```python theme={null}
import os
import time

import requests

API = "https://api.ornn.com"
http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['ORNN_API_KEY']}"


class OrnnError(Exception):
    def __init__(self, response):
        self.status = response.status_code
        try:
            body = response.json()
        except ValueError:
            body = {}
        detail = body.get("detail") if isinstance(body, dict) else None
        if isinstance(detail, dict):
            self.code, message = detail.get("code"), detail.get("message")
        else:
            self.code = body.get("code") if isinstance(body, dict) else None
            message = detail
        self.body = body
        super().__init__(f"HTTP {self.status} {self.code}: {message}")


def call(method, path, *, ok=(200,), **kwargs):
    """Send one request. Retry rate limits on any method and 5xx on reads only."""
    for attempt in range(6):
        response = http.request(method, API + path, timeout=30, **kwargs)
        retryable = response.status_code == 429 or (
            method == "GET" and response.status_code >= 500
        )
        if retryable and attempt < 5:
            time.sleep(float(response.headers.get("Retry-After", 2**attempt)))
            continue
        if response.status_code not in ok:
            raise OrnnError(response)
        return response.json() if response.content else None
```

A `429` is returned before your request reaches Ornn's services, so any method is safe to retry after `Retry-After`. Do not blindly retry a write after a timeout or `5xx`; read the current state first, as each step below describes.

## 1. Find a reservation and eligible hosts

List your active reservations, then list the hosts on the one you choose. A host can join as a worker when all of these are true:

* `status` is `bare-metal`.
* `controller_id` is `null`, so it is not in another cluster.
* `health_readiness.ready` is `true`.

`health_readiness.reasons` containing `heartbeat:stale` means the host has not reported recently, and launch refuses it. `heartbeat:unknown` in `advisory_reasons` means Ornn has no heartbeat evidence either way; it does not block launch, but it does not prove the host is reachable. Node count means hosts, not GPUs.

```bash CLI theme={null}
ornn context --format json
ornn reservations list --status active --format json
# ornn launch lists eligible hosts for review in a terminal.
# Automation passes exact host IDs, which you can read with the API.
```

```bash curl theme={null}
curl --fail-with-body -sS "$ORNN_API_URL/commerce/reservations?status=active&limit=100" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq '.reservations[] | {reservationId, gpuSlug, reservedNodeCount, startsAt, endsAt}'

export RESERVATION_ID='11111111-1111-4111-8111-111111111111'

curl --fail-with-body -sS \
  "$ORNN_API_URL/v1/orchestrator/nodes?reservation_id=$RESERVATION_ID" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq '[.nodes[]
      | select(.status == "bare-metal" and .controller_id == null and .health_readiness.ready)
      | {node_id, gpu_type, gpu_count, advisory: .health_readiness.advisory_reasons}]'
```

```python Python theme={null}
reservations, cursor = [], None
while True:
    params = {"status": "active", "limit": 100}
    if cursor:
        params["cursor"] = cursor
    page = call("GET", "/commerce/reservations", params=params)
    reservations += page["reservations"]
    cursor = page.get("nextCursor")
    if not cursor:
        break

RESERVATION_ID = "11111111-1111-4111-8111-111111111111"  # the reservation you chose

nodes = call(
    "GET", "/v1/orchestrator/nodes", params={"reservation_id": RESERVATION_ID}
)["nodes"]
eligible = [
    node["node_id"]
    for node in nodes
    if node["status"] == "bare-metal"
    and node["controller_id"] is None
    and node.get("health_readiness", {}).get("ready") is True
]
print(eligible)
```

The CLI's automation form needs your organization ID; `ornn context --format json` returns it, and so does `GET /v1/auth/session` with your API key.

## 2. Create the cluster

> [!WARNING]
>
> Creating a cluster prepares every selected host, which wipes its tenant home directory and removes
> any VM on it. This request starts provisioning; it is not a dry run.

Send `kind`, `reservation_id`, and the exact `node_ids` to use. Listing hosts explicitly is the safe choice: if you omit both `node_ids` and `virtual_machine_ids`, Ornn uses every bare-metal host on the reservation. Set `network_mode` to `private`, and optionally `network_id`, for a cluster reachable only over WireGuard.

```bash CLI theme={null}
export NODE_ID='33333333-3333-4333-8333-333333333333'
export ORGANIZATION_ID='22222222-2222-4222-8222-222222222222'

ornn launch --mode kubernetes --reservation "$RESERVATION_ID" \
  --nodes 1 --machines "$NODE_ID" --network public \
  --organization "$ORGANIZATION_ID" --yes \
  --idempotency-key "k8s-launch-$RESERVATION_ID" --format json
```

```bash curl theme={null}
export NODE_ID='33333333-3333-4333-8333-333333333333'

jq -n --arg reservation "$RESERVATION_ID" --arg node "$NODE_ID" \
  '{kind: "kubernetes", reservation_id: $reservation,
    network_mode: "public", node_ids: [$node]}' \
  | curl --fail-with-body -sS -X POST "$ORNN_API_URL/v1/orchestrator/controllers" \
      -H "Authorization: Bearer $ORNN_API_KEY" \
      -H 'Content-Type: application/json' \
      --data-binary @- > launch.json

export CLUSTER_ID="$(jq -er '.id' launch.json)"
```

```python Python theme={null}
LIVE_STATES = ("pending", "provisioning", "active", "deleting")


def find_cluster(cluster_id=None):
    """Return the {controller, nodes} entry for a cluster, or None.

    Without an ID, return your organization's live Kubernetes cluster. An
    organization runs at most one at a time.
    """
    listing = call("GET", "/v1/orchestrator/controllers", params={"kind": "kubernetes"})
    for entry in listing["controllers"]:
        controller = entry["controller"]
        if controller["id"] == cluster_id or (
            cluster_id is None and controller["state"] in LIVE_STATES
        ):
            return entry
    return None


existing = find_cluster()
state = existing["controller"]["state"] if existing else None
if state == "deleting":
    raise RuntimeError("A teardown is still running. Wait for it to finish.")
if state in ("pending", "provisioning", "active"):
    CLUSTER_ID = existing["controller"]["id"]  # reuse it; do not launch again
else:  # no cluster for this reservation yet, or the last launch failed
    accepted = call(
        "POST",
        "/v1/orchestrator/controllers",
        ok=(202,),
        json={
            "kind": "kubernetes",
            "reservation_id": RESERVATION_ID,
            "network_mode": "public",
            "node_ids": eligible[:1],
        },
    )
    CLUSTER_ID = accepted["id"]
print("cluster", CLUSTER_ID)
```

Expect `202 Accepted` with the new cluster's `id`, `state: provisioning`, and its `namespace`. That response means the launch was accepted, not that anything is ready; `node_count` and `gpu_count` are usually `0` until workers attach.

Create is safe to repeat for the same reservation, but it is not a scaling operation:

* If the reservation's cluster is `active`, or `provisioning` with workers attached, the request returns that cluster unchanged with `202`, even if `node_ids` differs.
* If the last launch on the reservation `failed`, the request relaunches it under the same `id`.
* Ornn does not read an `Idempotency-Key` header on this route. After a timeout or `5xx`, list your Kubernetes clusters before sending the request again, as the Python example does. Your organization runs at most one at a time, so a live cluster in that list is the one your earlier request created.

| Response | `detail.code` | What to do |
| - | - | - |
| `409` | `cluster_access_mode_conflict` | Your organization runs a Slurm cluster (`live_mode`). Tear it down and wait for it to finish. |
| `409` | `kubernetes_cluster_ambiguous_group` | Your organization already runs a Kubernetes cluster on another reservation. Add workers to it. |
| `409` | `launch_node_offline` | The host in `node_id` has not sent a heartbeat recently. Bring it online or choose another. |
| `409` | `cluster_nodes_not_available` | A host is not in `bare-metal` status or no longer exists. Refresh the host list. |
| `409` | `reservation_not_launchable` | The reservation ended, was cancelled, or has a pending change. |
| `404` | `reservation_not_found`, `node_not_found` | The reservation or a host is not visible to your organization. Check the IDs. |
| `422` | `invalid_network_mode` | Use `public` or `private`. |

## 3. Wait for the cluster to become active

Poll the cluster list every 10 seconds and select your cluster by `controller.id`. Read three things:

* `controller.state`: `pending` or `provisioning` while launching, `active` when every launch worker has joined and access has been issued, `failed` when the launch could not finish.
* `controller.progress.stages`: launch stages `create_plane`, `prepare_hosts`, `join_workers`, `apply_plane`, and `issue_access`, each `pending`, `running`, `completed`, or `failed`. A failed stage's `detail` carries the reason.
* `nodes[].state` for each worker: `joining`, then `ready`; `join_failed` means that host could not join.

Creating the control plane, preparing hosts, and pulling images take several minutes. A launch whose workers cannot join is marked `failed` after about 30 minutes, so a client deadline of an hour leaves room for the rest. If your deadline passes while the cluster is still provisioning, contact Ornn support with the cluster ID; do not launch again.

If a worker stays `joining`, check its host's `health_readiness` in the host list from step 1. `heartbeat:stale` means the host went offline.

```bash CLI theme={null}
ornn clusters wait "$CLUSTER_ID" --timeout 60m --format json
```

```bash curl theme={null}
deadline=$((SECONDS + 3600))
while :; do
  entry="$(curl --fail-with-body -sS \
    "$ORNN_API_URL/v1/orchestrator/controllers?kind=kubernetes" \
    -H "Authorization: Bearer $ORNN_API_KEY" \
    | jq -c --arg id "$CLUSTER_ID" '.controllers[] | select(.controller.id == $id)')"
  state="$(jq -r '.controller.state // "missing"' <<<"${entry:-null}")"
  echo "state=$state $(jq -c '[.nodes[]? | {node_id, state}]' <<<"${entry:-null}")"
  case "$state" in active|failed|missing) break ;; esac
  [ "$SECONDS" -lt "$deadline" ] || break
  sleep 10
done
jq '.controller.progress' <<<"${entry:-null}"
# Continue only if the last line printed state=active.
```

```python Python theme={null}
deadline = time.monotonic() + 3600
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        raise RuntimeError("The cluster is no longer listed; it may have been torn down.")
    controller = entry["controller"]
    stages = (controller.get("progress") or {}).get("stages", [])
    print(
        controller["state"],
        [(stage["key"], stage["status"]) for stage in stages],
        [(node["node_id"], node["state"]) for node in entry["nodes"]],
    )
    if controller["state"] == "active":
        break
    if controller["state"] == "failed":
        raise RuntimeError(f"Launch failed: {stages}")
    if time.monotonic() > deadline:
        raise TimeoutError("Still provisioning; contact Ornn with the cluster ID.")
    time.sleep(10)
```

`active` means Ornn finished its part. It does not prove each Kubernetes Node is Ready or advertises GPUs; step 5 checks that through the Kubernetes API.

## 4. Fetch the kubeconfig

Request the cluster's access. The response contains `kubeconfig`, the API server `endpoint`, and `namespace`, the namespace Ornn configured for your workloads. A `404` with `kubernetes_credentials_not_ready` means access has not been issued yet; keep polling step 3. Any other `404` means the cluster ID is wrong or not visible to your organization.

The kubeconfig's token has cluster-admin rights and is valid for seven days. Store it like a password, with owner-only file permissions.

```bash CLI theme={null}
ornn connect "$CLUSTER_ID" --output ./ornn-kubeconfig.yaml --format json
export KUBECONFIG="$PWD/ornn-kubeconfig.yaml"
```

```bash curl theme={null}
umask 077
curl --fail-with-body -sS \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/access" \
  -H "Authorization: Bearer $ORNN_API_KEY" -o access.json
jq -er '.kubeconfig' access.json > ornn-kubeconfig.yaml
export NAMESPACE="$(jq -er '.namespace' access.json)"
rm access.json
export KUBECONFIG="$PWD/ornn-kubeconfig.yaml"
```

```python Python theme={null}
while True:
    try:
        access = call("GET", f"/v1/orchestrator/controllers/{CLUSTER_ID}/access")
        break
    except OrnnError as error:
        if error.status == 404 and error.code == "kubernetes_credentials_not_ready":
            time.sleep(10)
            continue
        raise

KUBECONFIG_PATH = os.path.abspath("ornn-kubeconfig.yaml")
descriptor = os.open(KUBECONFIG_PATH, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
os.fchmod(descriptor, 0o600)  # also tightens a file that already existed
with os.fdopen(descriptor, "w") as handle:
    handle.write(access["kubeconfig"])
NAMESPACE = access["namespace"]
print(access["endpoint"], NAMESPACE)
```

The CLI writes the file with `0600` permissions and refuses to overwrite an existing file unless you add `--overwrite`.

## 5. Run a GPU smoke Job through the Kubernetes API

From here on you talk to the cluster's own API server with the kubeconfig. Run GPU work in the namespace from step 4: in that namespace, Ornn's scheduling rules place Pods on your reserved workers, and a ResourceQuota named `ornn-gpu-quota` caps the GPUs its Pods can request. Namespaces you create yourself do not receive those scheduling rules.

This NVIDIA example checks the Nodes, requests one GPU, runs `nvidia-smi`, and waits up to five minutes. The Node list also shows the managed control-plane Nodes, which are not GPU capacity. For AMD capacity, use an image with ROCm, request `amd.com/gpu`, and run `amd-smi list`.

```bash kubectl theme={null}
kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,GPUS:.status.allocatable.nvidia\.com/gpu'
kubectl describe resourcequota ornn-gpu-quota

kubectl apply -f - <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: ornn-gpu-smoke
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 240
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: nvidia/cuda:12.4.1-base-ubuntu22.04
          command: ["nvidia-smi"]
          resources:
            limits:
              nvidia.com/gpu: 1
EOF

kubectl wait --for=condition=complete job/ornn-gpu-smoke --timeout=300s
kubectl logs job/ornn-gpu-smoke
kubectl delete job ornn-gpu-smoke
```

```bash Kubernetes REST theme={null}
# The kubeconfig Ornn returns has one cluster and one user.
KUBE_SERVER="$(awk '/^    server: / {print $2}' ornn-kubeconfig.yaml)"
KUBE_TOKEN="$(awk '/^    token: / {print $2}' ornn-kubeconfig.yaml)"
awk '/^    certificate-authority-data: / {print $2}' ornn-kubeconfig.yaml \
  | base64 --decode > ornn-ca.crt
kube() {
  curl --fail-with-body -sS --cacert ornn-ca.crt \
    -H "Authorization: Bearer $KUBE_TOKEN" "$@"
}

kube "$KUBE_SERVER/api/v1/nodes" | jq '.items[] | {name: .metadata.name,
  ready: ([.status.conditions[] | select(.type == "Ready")][0].status),
  gpus: .status.allocatable["nvidia.com/gpu"]}'

jq -n '{apiVersion: "batch/v1", kind: "Job", metadata: {name: "ornn-gpu-smoke"},
  spec: {backoffLimit: 0, activeDeadlineSeconds: 240, template: {spec: {
    restartPolicy: "Never",
    containers: [{name: "smoke", image: "nvidia/cuda:12.4.1-base-ubuntu22.04",
      command: ["nvidia-smi"], resources: {limits: {"nvidia.com/gpu": 1}}}]}}}}' \
  | kube -X POST -H 'Content-Type: application/json' --data-binary @- \
      "$KUBE_SERVER/apis/batch/v1/namespaces/$NAMESPACE/jobs" > /dev/null

for _ in $(seq 60); do
  status="$(kube "$KUBE_SERVER/apis/batch/v1/namespaces/$NAMESPACE/jobs/ornn-gpu-smoke" | jq -c .status)"
  [ "$(jq -r '(.succeeded // 0) + (.failed // 0)' <<<"$status")" -ge 1 ] && break
  sleep 5
done
echo "$status"   # expect "succeeded":1
pod="$(kube "$KUBE_SERVER/api/v1/namespaces/$NAMESPACE/pods?labelSelector=job-name%3Dornn-gpu-smoke" \
  | jq -er '.items[0].metadata.name')"
kube "$KUBE_SERVER/api/v1/namespaces/$NAMESPACE/pods/$pod/log"
kube -X DELETE \
  "$KUBE_SERVER/apis/batch/v1/namespaces/$NAMESPACE/jobs/ornn-gpu-smoke?propagationPolicy=Foreground" > /dev/null
```

```python Python theme={null}
import uuid

from kubernetes import client, config

config.load_kube_config(config_file=KUBECONFIG_PATH)
core = client.CoreV1Api()
batch = client.BatchV1Api()

for node in core.list_node(_request_timeout=30).items:
    ready = any(c.type == "Ready" and c.status == "True" for c in node.status.conditions or [])
    gpus = (node.status.allocatable or {}).get("nvidia.com/gpu", "0")
    print(node.metadata.name, "Ready:", ready, "GPUs:", gpus)
quota = core.read_namespaced_resource_quota("ornn-gpu-quota", NAMESPACE, _request_timeout=30)
print("GPU quota:", quota.status.hard, "used:", quota.status.used)

name = "ornn-gpu-smoke-" + uuid.uuid4().hex[:8]
batch.create_namespaced_job(
    NAMESPACE,
    {
        "apiVersion": "batch/v1",
        "kind": "Job",
        "metadata": {"name": name},
        "spec": {
            "backoffLimit": 0,
            "activeDeadlineSeconds": 240,
            "template": {
                "spec": {
                    "restartPolicy": "Never",
                    "containers": [
                        {
                            "name": "smoke",
                            "image": "nvidia/cuda:12.4.1-base-ubuntu22.04",
                            "command": ["nvidia-smi"],
                            "resources": {"limits": {"nvidia.com/gpu": 1}},
                        }
                    ],
                }
            },
        },
    },
    _request_timeout=30,
)
deadline = time.monotonic() + 300
while True:
    status = batch.read_namespaced_job_status(name, NAMESPACE, _request_timeout=30).status
    if status.succeeded:
        break
    if status.failed or time.monotonic() > deadline:
        raise RuntimeError(f"Job {name} did not succeed; inspect its Pods and events.")
    time.sleep(5)

pods = core.list_namespaced_pod(NAMESPACE, label_selector=f"job-name={name}", _request_timeout=30)
for pod in pods.items:
    print(core.read_namespaced_pod_log(pod.metadata.name, NAMESPACE, _request_timeout=30))
batch.delete_namespaced_job(name, NAMESPACE, propagation_policy="Foreground", _request_timeout=30)
```

Success means each worker Node is Ready with GPUs allocatable, the Job completes, and its log lists an NVIDIA GPU. A failed or timed-out Job stays for inspection; check its Pods and events before deleting it. Deleting the Job removes only the test, not the cluster or reservation.

For workload guidance such as storage, MPI, images, and queueing, see [Run your application](/guides/kubernetes-access#run-your-application).

## 6. Scale workers up or down

Scaling means adding or removing whole hosts through the Ornn API. Kubernetes autoscalers change Pod replicas, not Ornn hosts.

### Add a worker

Choose an eligible host as in step 1, preferably from the reservation the cluster launched on, and add it. A new host normally returns `state: cleaning` while Ornn prepares it. It then appears in the cluster's `nodes` as `joining`, then `ready`. Repeating the request is safe: it returns `cleaning` until preparation finishes, then the member's current state.

```bash CLI theme={null}
export WORKER_NODE_ID='77777777-7777-4777-8777-777777777777'

ornn clusters eligible-machines "$CLUSTER_ID" --format json
ornn clusters add-node "$CLUSTER_ID" "$WORKER_NODE_ID" --yes \
  --idempotency-key "add-$WORKER_NODE_ID" --format json --wait --timeout 45m
```

```bash curl theme={null}
export WORKER_NODE_ID='77777777-7777-4777-8777-777777777777'

curl --fail-with-body -sS -X POST \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/nodes/$WORKER_NODE_ID" \
  -H "Authorization: Bearer $ORNN_API_KEY"

# Repeat until the worker is listed as ready.
curl --fail-with-body -sS "$ORNN_API_URL/v1/orchestrator/controllers?kind=kubernetes" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq --arg id "$CLUSTER_ID" --arg node "$WORKER_NODE_ID" \
      '.controllers[] | select(.controller.id == $id) | .nodes[] | select(.node_id == $node)'
```

```python Python theme={null}
WORKER_NODE_ID = "77777777-7777-4777-8777-777777777777"

change = call(
    "POST", f"/v1/orchestrator/controllers/{CLUSTER_ID}/nodes/{WORKER_NODE_ID}", ok=(202,)
)
print(change["state"])  # usually "cleaning"

deadline = time.monotonic() + 2700
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        raise RuntimeError("The cluster is no longer listed.")
    member = next((n for n in entry["nodes"] if n["node_id"] == WORKER_NODE_ID), None)
    if member and member["state"] == "ready":
        break
    if member and member["state"] == "join_failed":
        raise RuntimeError("The worker could not join; contact Ornn with both IDs.")
    if time.monotonic() > deadline:
        raise TimeoutError("The worker has not joined yet.")
    time.sleep(10)
```

> [!NOTE]
>
> Adding a worker does not raise `ornn-gpu-quota`, which Ornn sizes from the cluster's workers when
> it first becomes active. A host from a different reservation is labeled with its own reservation,
> while Pods in your namespace are pinned to the reservation the cluster launched on, so they do not
> schedule there. After the worker is `ready`, confirm in Kubernetes that its Node is Ready with
> GPUs and that the quota covers the GPUs you plan to request. Contact Ornn support if it does not.

Add worker errors include `404 controller_not_found` when the cluster is failed or being torn down, `404 node_not_found` when the host is not yours, and `409 cluster_nodes_not_available` when the host is not in `bare-metal` status. A `409` with an empty JSON object means the host already belongs to another cluster.

### Remove a worker

> [!CAUTION]
>
> Removing a worker stops the workloads on it and can lose its local data. Checkpoint first.
> Removing a worker does not cancel its reservation or stop its billing.

Optionally drain the Node in Kubernetes first so Pods move off it gracefully. Then remove it. The response is `200` with `state: draining`, which means cleanup was sent. The worker leaves the cluster's `nodes` once its host confirms cleanup. `cleanup_failed` means the host reported that cleanup did not finish; contact Ornn before reusing it. Repeating the request while the worker drains sends cleanup again; after it has left, the request returns `404 node_not_found`.

```bash CLI theme={null}
ornn clusters remove-node "$CLUSTER_ID" "$WORKER_NODE_ID" --yes \
  --idempotency-key "remove-$WORKER_NODE_ID" --format json --wait --timeout 30m
```

```bash curl theme={null}
curl --fail-with-body -sS -X DELETE \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/nodes/$WORKER_NODE_ID" \
  -H "Authorization: Bearer $ORNN_API_KEY"

# Repeat until this prints nothing.
curl --fail-with-body -sS "$ORNN_API_URL/v1/orchestrator/controllers?kind=kubernetes" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq --arg id "$CLUSTER_ID" --arg node "$WORKER_NODE_ID" \
      '.controllers[] | select(.controller.id == $id) | .nodes[] | select(.node_id == $node)'
```

```python Python theme={null}
call("DELETE", f"/v1/orchestrator/controllers/{CLUSTER_ID}/nodes/{WORKER_NODE_ID}")

deadline = time.monotonic() + 1800
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        raise RuntimeError("The cluster is no longer listed.")
    member = next((n for n in entry["nodes"] if n["node_id"] == WORKER_NODE_ID), None)
    if member is None:
        break
    if member["state"] == "cleanup_failed":
        raise RuntimeError("Cleanup did not finish; contact Ornn before reusing the host.")
    if time.monotonic() > deadline:
        raise TimeoutError("The worker is still draining.")
    time.sleep(10)
```

### Drive scaling from your own controller

Because Ornn does not watch your Pods, a scaling loop lives in your automation. A typical loop reads pending GPU Pods and Node capacity from the Kubernetes API, decides how many hosts it needs, and calls the add or remove worker endpoint for one host at a time, waiting for `ready` or removal before the next change. Scaling is bounded by the hosts your organization has reserved; it never buys capacity.

## 7. Tear down the cluster and confirm

> [!CAUTION]
>
> Teardown deletes the cluster's workloads and revokes its kubeconfig. Export data first.
> Reservation billing continues, and separately provisioned shared filesystems are not deleted.

The teardown response is `202` with `state: deleting`. Ornn deletes the cluster's namespace and workloads, then waits up to five minutes for each worker to confirm cleanup before removing the cluster. Hosts that never confirm are released anyway and cleaned when they reconnect. Repeating the request during teardown returns `{"state": "deleting"}` and does not restart cleanup.

Teardown is complete when both of these are true:

* The cluster no longer appears in the cluster list.
* `GET /v1/orchestrator/controllers/{cluster_id}/access` returns `404` with `cluster_not_found`.

After that, teardown itself returns `404` with no body. If the state becomes `failed` instead, send the teardown again, and contact Ornn support if it fails twice. Wait for teardown to finish before you create another cluster.

```bash CLI theme={null}
ornn clusters teardown "$CLUSTER_ID" --yes \
  --idempotency-key "teardown-$CLUSTER_ID" --format json --wait --timeout 30m
```

```bash curl theme={null}
curl --fail-with-body -sS -X POST \
  "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/teardown" \
  -H "Authorization: Bearer $ORNN_API_KEY"

# Repeat until this prints cluster_not_found.
curl -sS "$ORNN_API_URL/v1/orchestrator/controllers/$CLUSTER_ID/access" \
  -H "Authorization: Bearer $ORNN_API_KEY" \
  | jq -r '.detail.code? // "teardown still running"'
```

```python Python theme={null}
try:
    call("POST", f"/v1/orchestrator/controllers/{CLUSTER_ID}/teardown", ok=(202,))
except OrnnError as error:
    if error.status != 404:  # 404 means the teardown already finished
        raise

deadline = time.monotonic() + 1800
while True:
    entry = find_cluster(cluster_id=CLUSTER_ID)
    if entry is None:
        try:
            call("GET", f"/v1/orchestrator/controllers/{CLUSTER_ID}/access")
        except OrnnError as error:
            if error.status == 404 and error.code == "cluster_not_found":
                break
            raise
    elif entry["controller"]["state"] == "failed":
        raise RuntimeError("Teardown failed; send it again, then contact Ornn.")
    if time.monotonic() > deadline:
        raise TimeoutError("Teardown has not finished.")
    time.sleep(10)
print("Cluster removed. The reservation and its billing continue.")
```

After teardown, the hosts return to your reservation. Delete your local kubeconfig; a new cluster issues new credentials.

## Retries and idempotency

| Operation | Safe to repeat? | What a repeat returns |
| - | - | - |
| List clusters, hosts, access | Yes | The current state. Retry `5xx` with backoff. |
| Create cluster | Yes, for the same reservation | The existing cluster when it is `active`, or `provisioning` with workers; a relaunch under the same `id` after `failed`. |
| Add worker | Yes | `cleaning` while the host is prepared, then the member's current state. |
| Remove worker | Yes | `draining` while cleanup runs; `404 node_not_found` once the worker has left. |
| Tear down | Yes | `{"state": "deleting"}` while it runs; `404` with no body once it has finished. |

The `/v1/orchestrator` routes share a limit of 240 requests per minute for each user. Every API key you create counts against your limit, and so does your console session. Polling every 10 seconds uses a small fraction of it. A `429` includes `Retry-After`.

Cluster operations return errors as `{"detail": {"code": "...", "message": "..."}}`; show `detail.message` to people and branch on `detail.code`. Authentication and organization-approval errors use `{"detail": "...", "code": "..."}`. Malformed JSON, unknown request fields, or a missing `Content-Type` return `400`, `422`, or `415` with a plain-text body. See [errors and safe retries](/api-keys#errors-and-safe-retries).

## Related

> **[Kubernetes access](/guides/kubernetes-access)**
>
> Console launch, networking, storage, and running applications on your cluster.

> **[Cluster API reference](/api-reference/clusters/create)**
>
> Request and response schemas for every cluster and worker operation.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.