Observability and evaluation
Olympusx AI keeps several records of what happened, each for a different question: what an agent did, what was said, what it cost, and whether the answer was any good. This page explains each record, who reads it, and how evaluation builds on it.
The records
Section titled “The records”| Record | The question it answers | Where | Who reads it |
|---|---|---|---|
| Runs | What did this agent do, step by step, and why did it stop? | Operate › Runs | Admin, Builder, Auditor |
| Traces | What exactly was said at each step, how long it took, and how many tokens it used? | Operate › Traces | Admin, Builder, Auditor |
| Conversations | What did this person and this assistant say to each other, as one thing? | Operate › Conversations | Admin, Builder, Reviewer, Auditor |
| Usage | How many tokens did each project, key and model spend? | Govern › Usage | Admin, Builder, Auditor |
| Audit | Who changed what, and when? | Govern › Audit | Admin, Auditor |
| Performance | How many calls, how slow, where the tokens went, how many failed? | Operate › Overview | Admin, Builder, Auditor |
- A run records what an agent did and what it used: each model call, each tool call with its input and output, each hand-off to a collaborator, each approval, the tokens and the outcome, including which ceiling stopped it.
- A trace records what was said: the prompt and answer of each step, with retrieval, reranking and model spans and their timings. Traces are where content lives, so they are open to fewer roles. Tracing is a platform setting, off until a platform administrator turns it on.
- The audit log records every change and every decision: settings, keys, roles, approvals, guardrail findings, exports, erasures. Each entry is chained to the one before it by a hash, so a removed or altered entry breaks the chain, and the console can verify the chain on demand. See Audit.
Evaluation
Section titled “Evaluation”Evaluation is people judging answers, one at a time, on the record above.
- What is judged. Answers from published assistants, and, when a tenant turns on review capture, a sample of gateway traffic: 1 request in every N, chosen by the tenant. Captured turns are stored after the guardrails scan, so in Enforce mode with redaction a reviewer reads redacted text. Traffic from developer keys is never captured.
- Who judges. An Admin or a Reviewer, in Evaluate › Review queue or beside an answer in its conversation.
- The verdict. One of four fixed words: Acceptable, Wrong, Unsafe or Off policy, with an optional note. Each person judges an answer once, and a judgment is final.
- What is kept with it. Who judged, when, and the configuration of the assistant that produced the answer, so judgments before and after a change can be told apart.
- Where it is read. Evaluate › Scores lists every judgment and exports them.
See Review queue and Scores.
How long records are kept
Section titled “How long records are kept”Every record is kept until your platform administrator sets a retention window for its class, except captured gateway turns, which are kept for 30 days by default. Trimming the audit log keeps an anchor, so the remaining chain still verifies. See Data retention.