Skip to content
Olympusx AIOlympusx AIOlympusx AI documentation

Monitoring and alerts

Olympusx AI exposes Prometheus metrics and ships alert rules for your own monitoring stack. System health answers “is it up right now” in the console; this page is for the monitoring and on-call tooling you already run: Prometheus, Alertmanager and whatever pages your team.

Turn it on

Section titled “Turn it on”

Monitoring needs the Prometheus Operator’s resource types on the cluster: kube-prometheus-stack on Kubernetes, or OpenShift’s user-workload monitoring, which a cluster administrator turns on in OpenShift’s monitoring configuration.

  1. Turn on the monitoring component in the Olympus resource (spec.components.monitoring: true) or the chart. It renders service monitors for the platform’s metrics endpoints, and the alert rules below.
  2. Make sure your Prometheus may reach the metrics ports. The platform’s network policies admit scrapers from the monitoring namespace on Kubernetes, and OpenShift’s monitoring namespaces on OpenShift. If your Prometheus runs elsewhere, list it in spec.networkPolicy.metricsScrapers. See Network and hardening.

Asking for monitoring on a cluster without the Prometheus Operator’s resource types is refused rather than ignored.

If scrapers are not admitted, nothing looks wrong in the cluster: every pod stays ready, and Prometheus shows the targets as down.

Metrics endpoints

Section titled “Metrics endpoints”
Endpoint Port What it carries
Control plane 8086 Requests, authorization decisions, budget blocks, tokens, background loops, leadership
Metering worker 8082 Metering throughput
Platform database 9187 Database statistics

These ports are on internal listeners, not on the public console or gateway hosts, and they are not authenticated: the network policy is the control. They carry per-tenant token volumes, labelled by tenant id.

Control-plane metrics

Section titled “Control-plane metrics”
Metric Type Labels What it tells you
olympus_build_info gauge version, commit The running version
olympus_http_requests_total counter method, route, status Management API requests, by route pattern
olympus_http_request_duration_ms histogram method, route, status Their latency
olympus_authz_decisions_total counter decision, reason Gateway allow and deny decisions. reason matches the error code returned to the caller.
olympus_budget_blocks_total counter scope Requests blocked by a budget: key, model and so on
olympus_tokens_total counter tenant, project, model, direction Tokens served. Resets on restart: use the usage ledger, not this, for billing.
olympus_rate_limit_degraded_total counter Requests allowed because the rate-limit check itself failed
olympus_provider_unmetered_responses_total counter reason Cloud-provider answers that ended without usage, so were metered at zero
olympus_reconcile_total, olympus_reconcile_errors_total counter loop Passes and errors of each background loop
olympus_reconcile_duration_ms histogram loop How long a pass takes
olympus_reconcile_last_success_timestamp_seconds gauge loop When a loop last completed. 0 on a replica that is not its leader.
olympus_reconcile_interval_seconds gauge loop The loop’s own declared interval
olympus_leader gauge lease 1 on the replica that leads a lease. Exactly one replica should.
olympus_leader_transitions_total, olympus_lease_check_failures_total, olympus_leader_loop_exits_total counter lease Leadership changes and failures
olympus_deployments gauge state Model deployments by state

The route label is the registered route pattern, such as /v1/tenants/{tenantId}/keys, so the number of series stays bounded.

The alerts the platform ships

Section titled “The alerts the platform ships”

All are warning except the first.

Alert Fires when Severity
OlympusRateLimitFailedOpen Any request was allowed without a rate-limit check in the last 5 minutes, for 2 minutes critical
OlympusReconcileLoopStalled A background loop has not completed a pass in ten of its own intervals, for 15 minutes warning
OlympusLoopHasNoLeader No replica leads a lease, for 10 minutes warning
OlympusLeaseHasMultipleLeaders More than one replica leads the same lease, for 5 minutes warning
OlympusModelDeploymentNotReady An operator-managed model deployment has not been ready for 15 minutes warning
OlympusModelDeploymentUnderReplicated A model deployment has had fewer available replicas than desired for 20 minutes warning
OlympusOperatorReconcileErrors The operator’s model deployment controller has been erroring for 15 minutes warning
OlympusPlatformNotReady The Olympus resource has not been ready for 30 minutes warning
OlympusPlatformComponentsMissing Some, but not all, of the platform’s components have been available for 45 minutes warning
OlympusPlatformInstallRefused The operator has refused the installation for 15 minutes warning
OlympusProviderUsageMissing Any cloud-provider answer ended without usage warning

The waiting times are chosen not to fire during a normal start: pulling a large model image, or an installation pulling images and migrating its database, legitimately takes many minutes.

Rate limiting failed open

Section titled “Rate limiting failed open”

OlympusRateLimitFailedOpen. Requests are being allowed without a rate-limit check, because the check failed. Every tenant is effectively unlimited while it lasts, and callers see no error: Olympusx AI serves traffic rather than refuse it when the limiter is unavailable, and this alert is how you know.

  1. Check the cache the limiter reads from, on System health. The counter climbs when that read fails.
  2. Meanwhile, watch spend: olympus_tokens_total and the usage ledger stay accurate, because metering is separate from rate limiting.

A background loop stalled, has no leader, or has two

Section titled “A background loop stalled, has no leader, or has two”
  • OlympusReconcileLoopStalled: a loop has stopped completing. Users see a queue that never drains: deployments stay pending, namespaces are not created, imports never finish, while the API keeps accepting work. If no replica leads its lease, see the next item. Otherwise read olympus_reconcile_errors_total and the leader’s log: a loop that errors on every pass is stuck on one record, which the log names. Check it can reach the Kubernetes API, the registry and the database. Restarting the leader hands the lease to another replica, but does not fix a bad record.
  • OlympusLoopHasNoLeader: the loop runs nowhere. olympus_lease_check_failures_total climbing means replicas cannot read or write their lease: check the service account still holds the lease permissions. olympus_leader_loop_exits_total climbing means a replica keeps acquiring the lease and losing it; its log says why. A lease held by a pod that no longer exists resolves when the lease expires.
  • OlympusLeaseHasMultipleLeaders: two replicas do the same work. Usually two installations share one namespace, or the nodes’ clocks are badly skewed. Scale the control plane to one replica while you fix the cause.

Model deployments

Section titled “Model deployments”
  • OlympusModelDeploymentNotReady: requests to that model fail. kubectl describe on the deployment resource shows which condition is false: plan valid false means the plan does not fit its GPUs and nothing is wrong with the cluster; placed false means objects could not be applied (permissions, a missing namespace); rolled out false means the workload never became available, usually scheduling; routable false means the gateway will not publish it.
  • OlympusModelDeploymentUnderReplicated: still serving, with less headroom. A replica cannot be scheduled (no free GPU of the requested type, or the candidate node is cordoned), or a node went away. Free or add a GPU, or lower the replica count deliberately.
  • OlympusOperatorReconcileErrors: the operator is failing, so changes and drift are not applied. Its log names the cause: usually a permission, a missing resource type, or a field the API server refused.

The installation

Section titled “The installation”
  • OlympusPlatformNotReady: the whole platform in its namespace. kubectl describe on the Olympus resource names the failing condition. The usual causes are an image pull failure (check the pull secret and that tags carry no v prefix), a database that never came up (three instances need three schedulable nodes and volumes), or migrations that cannot run (read the control plane’s init container log).
  • OlympusPlatformComponentsMissing: part of the platform is not running. Which part decides the urgency: without the control plane nothing works; without the gateway no inference runs though the console still works; without the metering components inference still serves and nothing is metered or budget-checked, the quietest expensive failure; without the console only the console is lost; without the document converter or reranker, ingestion and reranking degrade.
  • OlympusPlatformInstallRefused: the reason label says why. MasterKeyUnusable: the master key Secret is missing or has no SECRETS_MASTER_KEY entry; if the platform ran before, restore the original key, never a new one. ClusterScopedConflict: another installation with the same resource name owns the cluster-wide names; rename one. PreflightRejected: a resource type the specification needs is missing. DuplicateInstallation: one platform per namespace. ValuesInvalid: the message names the field.

Unmetered cloud-provider usage

Section titled “Unmetered cloud-provider usage”

OlympusProviderUsageMissing. A cloud provider billed your account for tokens the usage ledger recorded as zero. reason is client_disconnected (the caller hung up before the final chunk that carries usage) or no_usage (the provider’s answer carried none). The control plane’s log names the tenant, model and request id of each.

  • The same tenant disconnecting again and again: a client closes streams early. Ask the tenant; if it continues, revoke that key.
  • no_usage on one model: the provider does not report usage for it. Do not price that model until it stays quiet.
  • The ledger cannot be corrected afterwards. Compare the provider’s own usage report with olympus_tokens_total for the period to recover the cost.

Do not silence it: each occurrence is spend that reaches no statement.

What else to watch

Section titled “What else to watch”
  • The retention hold. The control-plane log line retention: usage ledger held back by unstated billing means a retention window is not being met. See Data retention.
  • Backups. Watch that your databases keep a first recoverability point. See Backup and restore.
  • Platform notifications for components going down and imports finishing, in the console. See System health.
Section titled “Related”