Troubleshooting

Symptom, cause, fix, grouped by where the install stands. Most install failures are a credential in the wrong namespace or a value that does not match another; the platform refuses by name wherever it can, so start with what the resource, the pod or the log says.

Where to look first

On an operator install, everything the operator refuses is on the resource:

kubectl -n olympus describe olympuses.olympus.olympusx.ai olympus
kubectl -n olympus get events --field-selector reason=InstallWarning

On either path:

kubectl -n olympus get pods
kubectl -n olympus describe pod <pod>
kubectl -n olympus logs deploy/olympus-control-plane

Images

Symptom	Cause	Fix
Pods in `ImagePullBackOff` with `401 Unauthorized`, `unauthorized` or `invalid username/password`	No valid pull secret in that namespace, the token expired, or the pods were created before the Secret	Create or replace the Secret, then delete the failing pods. A pod keeps the pull secrets it was created with
`ImagePullBackOff` with `no match for platform in manifest`	The pod landed on an arm64 node	Run the platform on `linux/amd64` nodes
`ImagePullBackOff` on a third-party component on an air-gapped cluster	Its image is not in your mirror, or its `image.registry` still names the public registry	Mirror it and point the component at the mirror (Air-gapped installation)
`cosign verify` fails on a mirrored image	The mirror copied the image but not its signature	Copy again with `cosign copy`

The operator

The catalog never becomes READY

Cause: the catalog pod cannot pull. Almost always the pull secret is missing from the catalog’s namespace (openshift-marketplace or olm), or the token is wrong. The CatalogSource reports a state, never a reason.

Fix: ask the pod:

oc -n openshift-marketplace describe pod -l olm.catalogSource=olympus-operators | tail -20

Fix the Secret, then delete that pod. A credential that works with docker login <registry-host> is the test.

No ClusterServiceVersion appears

Cause: the Subscription’s approval is Manual, or the Software Catalog form set it so. The InstallPlan waits at RequiresApproval; the Subscription is healthy, the catalog is READY, and nothing reports an error. A Subscription whose sourceNamespace names a namespace with no matching CatalogSource behaves the same way.

Fix: approve the waiting plan, and set approval to Automatic if you want installs to proceed on their own:

oc -n openshift-operators get installplan
oc -n openshift-operators patch installplan <installplan> \
  --type=merge -p '{"spec":{"approved":true}}'
oc -n openshift-operators patch subscription olympus-operator \
  --type=merge -p '{"spec":{"installPlanApproval":"Automatic"}}'

Check spec.sourceNamespace on the Subscription names the CatalogSource’s namespace.

The ClusterServiceVersion stays at Installing

Cause: the operator’s pod cannot pull: the pull secret is not linked to the olympus-operator-controller-manager ServiceAccount, or the pod was created before the link.

Fix: link the Secret and delete the pod (Install on OpenShift):

oc -n openshift-operators describe pod \
  -l app.kubernetes.io/name=olympus-operator,control-plane=controller-manager | tail -20

The `Olympus` resource reports READY False

READY Unknown with COMPONENTS climbing is the install happening. False is a refusal; the reason names it:

Reason	Cause	Fix
`MasterKeyUnusable`	The Secret in `secrets.masterKey` is missing, or has no `SECRETS_MASTER_KEY` entry	Create it. The operator retries by itself
`ValuesInvalid`	The spec does not map onto values the chart accepts. The message names the field, for example `database.urlSecret is required when database.mode is external`, or `tenantIsolation` without `appRoleSecret`	Correct the field
`RenderFailed`	A chart check fired, such as an `auth.issuer` that does not match the built-in identity service. The message names both issuers	With the built-in identity service, set `auth.issuer` to exactly `https://<console-host>/auth/realms/olympus`
`PreflightRejected`	The API server would not accept one object, so nothing was applied. Almost always a missing CRD: the managed database needs the PostgreSQL operator, monitoring needs the Prometheus Operator	Install the prerequisite (Requirements)
`ClusterScopedConflict`	Another `Olympus` with the same name already owns the cluster-wide roles	Give each install on a cluster a different `metadata.name`
`DuplicateInstallation`	A second `Olympus` in the same namespace	One platform per namespace

The resource is stuck deleting

Cause: the operator was removed before the resource, so nothing runs its finalizer.

Fix: reinstall the operator by applying the catalog and the Subscription again. It completes the deletion by itself (Uninstall).

The Helm chart

Symptom	Cause	Fix
`helm install` stops naming `postgresqlCNPG.backup.destinationPath`	The OpenShift values select the managed database and leave the backup destination for you	Set it, or pass `--set postgresqlCNPG.backup.enabled=false` for a smoke test
`helm install` stops naming `route.consoleHost` or `ingress.consoleHost`	Every token’s issuer is built from the console’s hostname	Set the console’s hostname
The chart refuses `monitoring.enabled`	The Prometheus Operator’s CRDs are missing	Install them, or leave monitoring off
The chart refuses `secrets.gitops`	Under GitOps it will not generate a master key	Set `secrets.existingSecret`
The chart refuses `auth.issuer` with the identity service on	The platform would trust your provider while fetching keys from the built-in service	Turn the built-in identity service off, or leave `auth.issuer` empty
The install runs on the bundled database though you set `DATABASE_URL`	The bundled database is on by default, and the Secret’s URL is ignored while it is	`postgresql.enabled: false` (Configuration)

Pods

Symptom	Cause	Fix
Pods `Pending` with no events	The namespace’s ResourceQuota, or no node with room	Compare against Capacity
A PersistentVolumeClaim stays `Pending`	The StorageClass does not exist, is not ReadWriteOnce, or there is no default StorageClass for the object store	`kubectl -n olympus describe pvc <claim>` names the cause
`CreateContainerConfigError: secret … not found`	The Secret named in `secrets.existingSecret` is missing, or lacks a key	Create it with the expected keys
The identity service’s pod shows `CreateContainerConfigError`	The relay password Secret is missing, or has no `password` key	Create it
The control plane refuses to start	Its configuration can never work, and it says which value: `refusing to start: identity provider misconfigured: …`	Correct the value the message names
The guardrails service takes minutes to become ready	It loads several models on start	Wait; its startup probe allows for it. On a cold node the image pull comes first

Symptom	Cause	Fix
Everything is healthy and nobody can sign in	The issuer is wrong or unreachable. It must equal the tokens’ `iss` byte for byte, and the browser must reach it	`kubectl -n olympus logs deploy/olympus-control-plane \| grep -i "identity provider"`. With the built-in service, check `auth.issuer` against `https://<console-host>/auth`, character for character
Sign-in works, then every API call answers `401`	The audience does not match the token’s `aud`	Set `auth.audience` to what your provider issues
A signed-in person sees one screen refusing everything	Their account has no role, or its `tenant` names a tenant this platform does not have	Grant a role in a tenant (Users and roles)
The browser warns about the certificate	On OpenShift, a host is not directly under the apps domain; elsewhere, no TLS Secret exists and the ingress serves its fallback	Move the host under the apps domain, or create the TLS Secrets or let cert-manager issue them

Mail

Symptom	Cause	Fix
Added, but the email was not sent, with `No sender address configured`	No relay was configured at install	Configure the relay in the identity service’s administration console: realm olympus, Realm settings › Email, then Resend invitation. Meanwhile, Set a temporary password lets the person in
Added, but the email was not sent, with another reason	The relay’s host, port, encryption or login is wrong, or the cluster cannot reach it	Correct it in the same place. Changing the install values does not reach a running realm

The gateway

The gateway answers 404 to every request

Cause: the gateway is Ready but has no routes. It waits up to 15 seconds for its configuration from the control plane, then serves without it. Either the control plane is not running, or a NetworkPolicy drops the gateway’s connection to it, for example after a pod label was changed by hand.

Fix: check the control plane is Ready, then the gateway’s log for connection errors to the control plane:

kubectl -n olympus rollout status deploy/olympus-control-plane
kubectl -n olympus logs deploy/olympus-gateway | tail -20

Requests fail with guardrails_unavailable

Cause: the guardrails service is on and not ready. The scanner fails closed by default, in observe mode too.

Fix: kubectl -n olympus rollout status deploy/olympus-guardrails. Its image is large and it loads models on start.

Health and monitoring

Symptom	Cause	Fix
A System health tile reads `Down · context deadline exceeded` while its pod is Ready	A deadline, not a refusal: a NetworkPolicy is dropping the packets. Kubelet’s probes come from the node and are not subject to these policies, so Ready proves nothing here	Check the NetworkPolicy that covers that component admits the port
ServiceMonitors exist and Prometheus shows no targets	The Prometheus selector does not match their labels, or `networkPolicy.metricsScrapers` does not admit Prometheus’s namespace	Configuration
The managed database reports `Instance Status Extraction Error` and stops failing over	The PostgreSQL operator runs in a namespace the database’s NetworkPolicy does not admit	Set `networkPolicy.cnpgOperatorNamespace` (resource) or `networkPolicy.cnpgOperator` (chart)
Registering a cluster or saving a credential answers `503 sealer_unconfigured`	The platform has no master key	Supply `SECRETS_MASTER_KEY` and restart the control plane
The cluster registration log says `cluster self-registration gave up`	The token Secret was not ready within two minutes, or a cluster of another type already holds the name	Restart the control plane; if the name is taken, set a different `clusterRegistration.name`