Review queue

Evaluate › Review queue lists the answers a person still has to look at. Pick one, read the question and the answer whole, and record a verdict: Acceptable, Wrong, Unsafe or Off policy. Recording it takes the answer out of the queue and adds it to Scores.

What a judgment is

A judgment, or score, is one person’s verdict on one stored answer, with an optional note saying why. The verdict is one of four fixed words: Acceptable, Wrong, Unsafe or Off policy. Only Acceptable counts as a pass. Agree with your reviewers what the other three mean for your assistants.

• Each person records at most one judgment per answer.
• A judgment is final. It cannot be edited or withdrawn: two contradictory judgments from one person, with no way to tell which is current, would be a worse record than one.
• Olympusx AI records who judged, when, and the configuration of the assistant that produced the answer, so judgments of an assistant before and after a change can be told apart.
• Recording a judgment is written to the audit trail: the verdict and the answer it judged, never the answer’s text or your note.

Who can open it

Role	Reads the queue	Records judgments
Admin	Yes	Yes
Reviewer	Yes	Yes
Auditor	Yes	No. The panel says recording a judgment takes the Admin or Reviewer role.
Platform admin	Yes, in every tenant	Yes
Builder, End user	No	No

For an Admin and a Reviewer, the Review queue row in the rail shows how many answers are waiting.

How answers reach the queue

The queue is not a list someone fills. It is every stored answer that has no judgment yet, newest first. Everything unjudged is there. Two sources feed it, each on its own tab.

Assistant answers

Every answer a published assistant gave in a conversation, beside the question it answered. An answer appears as soon as it is given. The question is the person’s latest turn before the answer, as the conversation stored it; when the guardrails redact input, it is the redacted text.

Gateway traffic

A sample of the calls your applications make to the gateway’s OpenAI-compatible API. Nothing from the gateway is kept until an Admin turns on Review capture in Guardrails:

1. In Govern › Guardrails, set Keep 1 request in every to a number: 0 keeps nothing (the default), 1 keeps every request, 100 keeps one in a hundred.
2. Save the policy.

Capture has three conditions, and all must hold:

• A content scanner is installed. A captured turn is what the guardrails scan passed, so with no scanner, nothing can be captured, and Guardrails refuses to save a capture rate above 0. The gateway tab then says nothing can be captured until a scanner is installed.
• The rate is above 0. The choice of which requests to keep is made from the request id, so the rate holds across every gateway replica.
• The traffic is not from a developer’s editor. Code Assistant traffic is never captured.

What is kept depends on the guardrails mode. In enforce mode with personal-data redaction, a captured turn holds the redacted text. In observe mode, it is stored as the caller sent it, and Guardrails warns that captured prompts will not be redacted. Each captured turn shows which applies.

Captured turns are deleted when their retention window closes, with any judgment and note on them. See Data retention.

What you see

1. A switcher: Assistant answers or Gateway traffic.
2. How many are waiting, for example Waiting: 12.
3. On Assistant answers, an Assistant filter (Every assistant, or one).
4. The Waiting for a judgment table, with a search field: Search the questions and answers or Search the prompts and answers.

Tab	Columns
Assistant answers	When, Question and answer (the start of each), Assistant
Gateway traffic	When, Prompt and answer (the start of each), Model, Content scan (Redacted by guardrails or As the caller sent it)

5. Load more of the queue, when more is waiting than one page holds.
6. Under the table, the open item:
   • The question and its answer (or The captured turn): What was asked and The answer, the answer rendered as the person saw it. Question shown in part or Answer shown in part marks a text cut for display; Cut off at 32 KB marks a captured turn cut by the gateway.
   • Your judgment: How was this answer?, a Why (optional) note, the reminder A judgment is final, and Record and show the next.

The newest item is open when the page loads. The tab, the assistant filter and the open item are kept in the page address, so a link can open one answer and a reload keeps your place. Your search is not kept in the address, because it is words from customers’ conversations.

When nothing is waiting, the page says Nothing waiting. On the gateway tab, it also says whether capture is off or cannot be turned on.

What you can do

Judge an answer

1. Choose Assistant answers or Gateway traffic.
2. Select a row to open it below the table, or keep the newest one.
3. Read What was asked and The answer in full.
4. Under How was this answer?, choose Acceptable, Wrong, Unsafe or Off policy.
5. Optionally, write Why (optional). The note is deleted with the conversation or captured turn, so it may refer to what was said.
6. Select Record and show the next, or press Ctrl Enter (⌘ Enter on a Mac).

The page says Judgment recorded. Waiting: with the new count, removes the item, and opens the next one.

Work through one assistant’s answers

1. On Assistant answers, choose the assistant in Assistant.
2. Judge from the top. Show every assistant clears the filter when nothing more is waiting for that one.

Find an answer someone complained about

Type part of the question or the answer in the search field. The search runs on the server, so it reaches items beyond the loaded page.

Judge an answer where it was given

You can also judge an answer while reading its conversation, with Score this answer; see Conversations. The verdict is the same, and the answer leaves the queue either way.

What is scored

Olympusx AI records judgments made by people, on two kinds of answer:

Judgment	What it covers
Human verdicts on assistant answers	Every answer a published assistant gives, from the queue or the conversation
Human verdicts on captured gateway traffic	The sampled share of gateway calls, once an Admin sets a capture rate and a scanner is installed

Every unjudged answer is listed, newest first.

Limits

Limit	Value
Items per page	200
Waiting count	Counts up to 10,000, then reads 10,000 or more
Search	200 characters
Note	2,000 characters
Captured prompt or answer	32 KB each; longer text is cut
Capture rate	0 (off) to 1000000 (one in a million)
Judgments per person per answer	1, final
Rail count refresh	Every minute

Related

• Scores
• Conversations
• Guardrails
• Observability and evaluation
• Data retention