Skip to content
Olympusx AIOlympusx AIOlympusx AI documentation

Code assessments

An assessment reviews one repository’s indexed code for weaknesses an attacker could exploit: injection, broken access control, authentication flaws, unsafe deserialization, path traversal, server-side request forgery, cross-site scripting, weak cryptography, hard-coded credentials, races with a security impact, and the like. A model reads the code in batches and reports each weakness tied to the lines it was shown, with a severity, a confidence, a CWE when it names one, and a suggested fix.

Admins and builders start and stop assessments on the Sources tab. Auditors read them.

What it reads

Section titled “What it reads”
  • The indexed code, at the commit the index holds. The code was scanned for secrets when it was indexed, so no secret reaches the model or a finding. Sync the source first to assess the latest commit. See Code sources.
  • Code, not prose or data. Markdown, plain text, CSV, JSON and reStructuredText files, files in languages the index does not parse, and lock files (package-lock.json, yarn.lock, pnpm-lock.yaml, go.sum, poetry.lock, Cargo.lock, Gemfile.lock, composer.lock) are skipped.
  • Your code, not third-party or generated code. Paths under vendor/, node_modules/, third_party/, dist/ or build/, and files ending in .min.js, .pb.go or _generated.go, are skipped.
  • Tests and examples only when you ask. Paths under test/, tests/, __tests__/, testdata/, spec/, fixtures/, example/, examples/, _examples/, sample/ or samples/, and files named like *_test.*, *.test.*, *.spec.* or test_*, are read only with Read tests and examples too. They do not ship, and reading them costs tokens.

Comments and strings in the code are treated as code under review: the model is told never to follow instructions written in them. A finding is kept only if the file and lines it names are ones its batch showed, and its evidence is cut from the index, never taken from the model’s answer.

Start an assessment

Section titled “Start an assessment”
  1. Open Use cases › Security and select the Sources tab.
  2. Under Repositories, select New assessment on the repository’s row, or select New assessment under Assessments and choose the Repository.
  3. Choose the Model. Models of 30B parameters or more and models from connected providers are listed first; smaller ones are marked (may be too small to review code). Choose a model of 30B parameters or more for code review.
  4. Turn on Read tests and examples too only if you want them read.
  5. Select Start the assessment.

Only one assessment of a repository runs at a time; starting a second says one is already running.

Follow it

Section titled “Follow it”

Assessments lists the tenant’s assessments, newest first:

Column What it shows
Repository The repository assessed.
Status Running, Stopping, Finished, Stopped or Failed, with Batches: done of total.
Findings New: findings not reported before · seen before: findings an earlier assessment or review already reported.
Files read How many files the assessment reads.
Model The model, and Tokens spent.
Started When it started.

Select Stop to stop an assessment at its next batch. The findings so far are kept.

An assessment fails, keeping the findings so far, when the model’s budget is spent for the period, when it runs out of time, or when the model answers none of its batches. One whose progress stands still for 10 minutes is marked interrupted.

Read the results

Section titled “Read the results”

Findings appear on the Findings tab as each batch finishes, and on the Repositories list as Open findings by severity. Each finding has:

  • Where: the repository, the file and lines, the function or type they are in, and the weakness class (CWE), with Open in the host and Show in the repository’s graph.
  • Why it may be exploitable: the model’s argument from the code.
  • The code: the lines it rests on, at most 40, from the index.
  • Suggested fix.
  • History: first and last reported, what first found it, and the model.

See Findings.

How repeat reports are matched

Section titled “How repeat reports are matched”

A finding is the same finding across assessments when it names the same file, the same weakness class, and the same function or type (or, outside one, the same first line). A repeat updates Last reported and counts as seen before; it does not add a row. A dismissed finding stays dismissed when it is reported again.

Limits

Section titled “Limits”
Limit Value
Files one assessment reads 3,000. A larger repository is assessed in part, and the rest count as skipped.
Code in one batch About 24,000 characters (about 6,000 tokens). A file too long for one batch is read in windows of lines.
Batches asked about at once 2
Findings one batch may add 25
Evidence lines kept per finding 40
A whole assessment 6 hours
Progress silent before it counts as interrupted 10 minutes
One model call The platform’s Security model call limit: 600 seconds by default
Running assessments per repository 1
Assessments listed The 100 most recent
Section titled “Related”