Runtimes

Platform › Model serving › Runtimes lists the serving runtimes model deployments run on. A runtime is a container image that runs a model server. Each one is pinned by digest and inspected: what it serves, its parsers and its flags are read from the image itself, not typed in by hand.

Who can open it

Platform administrators only.

What a runtime is for

When a tenant deploys a model, its deployment plan resolves to one runtime: the recommended runtime for the hardware, when it serves the model’s architecture, otherwise the smallest registered runtime that does. The plan pins that image by digest, so a running deployment keeps the image it was planned with whatever later happens to the runtime.

Runtimes come in two builds:

Build	Runs on	Serves
GPU	Nodes with NVIDIA GPUs. Each GPU build names the minimum NVIDIA driver it needs.	Text generation; the image lists the architectures and modalities it serves.
CPU	Ordinary nodes, with no GPU request.	Text generation for small models, or embeddings and reranking, depending on the build.

A CPU plan only takes a CPU build, and a GPU plan never takes one, so a GPU build is never started on a node without a driver.

The validated catalog

Each Olympusx release ships a catalog of runtimes it has validated, listed on this page under Not added yet. It works offline: nothing is fetched to show it. The 1.1 catalog holds:

Build	Platforms	Status
GPU, CUDA 13.0	linux/amd64 and linux/arm64	Validated by Olympusx, recommended for GPU text generation. Needs NVIDIA driver 580 or newer on every GPU node.
GPU, CUDA 12.9	linux/amd64 and linux/arm64	Listed with a Known issue: this build cannot start its server. It has no Add and inspect button, only a link to the upstream issue.
CPU, text generation	linux/amd64	Validated by Olympusx, recommended for CPU text generation.
CPU, embeddings and reranking	linux/amd64	Validated by Olympusx, recommended for CPU embeddings.

The runtime names and versions are those shown in the table on the page. Each entry carries its release notes: highlights, security fixes, breaking changes, and what matters for Olympusx AI.

What you see

The header, Runtimes, with Add runtime.
A notice naming the recommended runtime for GPU text generation and the NVIDIA driver it needs.

The registered runtimes, with Search by name, version or image:

Column	What it shows
Runtime	Name and version, a Recommended tag, and how many security fixes its release carries.
Build	GPU with its minimum driver, or CPU.
State	Ready, Disabled, Not pinned (named by tag, not digest) or Not inspected.
Serves	How many architectures, and which of text, embeddings, reranking, vision and speech.
Size	Unpacked size.
Released	The upstream release date.
Deployments	How many deployments are planned on it, in every tenant.

Recent inspections: Image, State (Inspecting, Registered or Failed), Node, Result and Started.
Not added yet: the validated catalog entries this installation has not registered, each with Add and inspect.

An installation with no runtime registered shows No runtime is registered.

Add a runtime

From the validated catalog:

Under Not added yet, select Add and inspect on an entry. The Add a runtime dialog opens with the entry’s image.
Choose Inspect on cluster, then select Inspect and add.

From your own registry:

Select Add runtime. The Add a runtime dialog opens.
Image: one image reference, with no spaces. Name it by digest (<registry>/<path>@sha256:<digest>) so what is registered is what will run; a tag-only image is registered as Not pinned.
Inspect on cluster: a registered cluster that can pull the image. The image stays cached on the node it lands on.
Runs on: GPU nodes, or CPU nodes (a CPU build).
Optionally, under Where it is pulled: Node labels (optional) as key=value pairs separated by commas, to keep the pull to one node pool; and Unpacked size in GiB (optional), used to find a node with room when none has the image yet. Empty assumes 30 GiB.
Select Inspect and add.

The control plane pulls the image onto a node with room and reads it. The runtime appears in the list once it is registered. One inspection runs at a time on a cluster; a large GPU image can take minutes to pull. An inspection that fails is listed under Recent inspections with its reason.

A runtime’s page

Select a runtime. Its page carries Make recommended (or Withdraw recommendation) and Disable (or Enable), then the tabs.

Tab	What it shows
Overview	Version, Build, Released, Image, Platforms, Software, Needs (the NVIDIA driver), Serves, Size, Channel and Inspected.
Availability	Where this runtime is, cluster by cluster. See Copy a runtime into a cluster’s registry.
Release notes	What matters for Olympusx, Highlights, Security fixes, Breaking changes with the effect of each, and a link to the full notes upstream.
Models	The model architectures the image serves natively, as a model’s `config.json` names them.
Flags	Every flag the image’s server accepts, with Group, Default and What it does. Flags marked Set by Olympusx (model, name, address, parallelism, memory) cannot be passed as extra arguments in a deployment plan.
Parsers	Reasoning parsers and Tool-call parsers the image registers. A reasoning parser splits a model’s thinking from its answer; a tool-call parser lets agents use the model. A deployment uses one only if the image lists it.
Deployments	Deployments planned on this image, in every tenant.

Make recommended: new plans on this build resolve to it whenever it serves the model, instead of the runtime recommended now. Running deployments keep the image they were planned with.
Disable: new plans stop resolving to it, and it stops being recommended. Deployments already on it keep running.
Enable reverses Disable.

Copy a runtime into a cluster’s registry

A cluster with no internet access can serve only what is in its own registry. The Availability tab shows, for each cluster, its Network (Internet access, No internet access, or via its bastion), its Registry, the Image it will pull, and an Action:

State	Meaning
In your registry	Copied and checked, by a cluster job, through the bastion or by hand.
Pulls from upstream	The cluster has internet access and no registry; the image is not pinned to your registry.
No registry	The cluster has no internet access and no registry. It pulls only if its nodes mirror the image. Select Connect a registry.
Copying, Checking the copy	A copy is running.
Copy failed, Not in this registry	The copy did not land.

A copy keeps the digest, so the image is the same everywhere. To copy:

Copy runs a job on the cluster that copies the image into its registry, using the registry’s push account. See Clusters.
Copy through the bastion has the cluster’s bastion agent copy it, when the agent is configured for image copies into this registry. See Remote access.
Copy by hand shows two commands: run the first on a machine that can reach upstream, carry the folder to one that can reach the registry, and run the second. Then select I’ve copied it, check it now: Olympusx AI reads the digest back from the registry before it records the copy.

Serving on CPU nodes

A cluster with no GPUs can still serve a small model of its own. A deployment plan with Hardware set to CPU runs the model on ordinary nodes with a CPU runtime, through the same plan, admission, project namespace, routing and metering as a GPU model. It suits models up to a few billion parameters and light traffic.

What it needs:

A CPU runtime registered and Ready. Register one with Add runtime and Runs on set to CPU nodes (a CPU build), or with Add and inspect on a CPU entry of the validated catalog.
A tenant with CPU cores and Memory above 0 in its capacity envelope on the cluster, and a project with a quota there. See Tenants.
The weights reachable without internet: packaged in the model registry. Project namespaces deny internet egress.

CPU nodes and the serving pool

A CPU model needs no node in the serving pool: the scheduler places it on any node with room. The pool still bounds the CPU and memory you can grant tenants on the cluster once any node is in it, because a cluster’s pool counts the CPU and memory of its managed nodes. On a cluster whose pool holds only GPU nodes, that can be less than the CPU nodes offer.

The console’s Managed switch appears on GPU nodes only. To put a CPU node in the pool, use the API as a platform administrator:

curl -X PATCH "https://<console-host>/api/v1/platform/clusters/<cluster-id>/nodes/<node-name>" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"managed": true}'

{"managed": false} takes it out again, and is refused with 409 pool_below_envelopes when tenants’ envelopes on the cluster need the node. An unknown node name answers 404 node_not_found.

Sizing a CPU plan

A CPU plan starts at 4 vCPUs and 16 GiB of memory, a 4096-token context and 8 concurrent sequences. The model server’s requests equal its limits.

Plan for these:

Leave memory headroom. The fit check sizes the weights and the context cache into 90% of the pod’s memory; the runtime itself shares the rest. A plan that barely fits can run out of memory under load.
No architecture pinning. The CPU builds are linux/amd64. On a cluster that mixes amd64 and arm64 nodes, a CPU model can land on a node its image cannot run on.
The weights image adds a small sidecar (10m CPU, 32 MiB requested; 100m, 128 MiB limit). Leave that room in a project quota sized exactly to the plan.

Limits

Limit	Value
Inspections at a time	One per cluster
Default size assumed for an inspection	30 GiB unpacked
CPU plan, minimum	1 vCPU and 1 GiB
Driver for the recommended GPU runtime	NVIDIA 580 or newer

Models and deployments
Deploy a model
Settings for how long a model call may run
Requirements for GPU and driver requirements