Pipelines and connectors

An ingestion pipeline reads documents from a source and loads them into one collection. It parses each document, cuts the text into chunks, embeds the chunks with the collection’s embedding model and stores them. Builders create pipelines; anyone who can open Knowledge can follow their runs.

Who can open it

Role	What they can do
Admin	Create, run, edit the source and schedule of, and delete pipelines.
Builder	The same as an Admin.
Auditor	Open the list and every pipeline page, with its runs and documents. Cannot create, run, change or delete.
Reviewer	Cannot open it.
End user	Cannot open it.
Platform admin	Everything an Admin can, for any tenant.

Sources

A pipeline has one source. The Source column and the pipeline’s page name it by its short label.

Source	What it reads	Can it run on a schedule
Upload (Upload documents)	Files you send from the console, one at a time.	No: it runs each time you send a file.
URL (Fetch from a URL)	One `http` or `https` address, fetched as one document.	Yes
S3	Every object in a bucket, or under a prefix, on Amazon S3 or an S3-compatible store.	Yes
NFS (Filesystem or NFS share)	Every file under a path of a share mounted for Olympusx AI.	Yes
SFTP (SFTP server)	Every file under a path on an SFTP server.	Yes

Each source’s fields:

Source	Fields
URL	Source URL (required)
S3	Endpoint (host:port) and Bucket (required); Prefix (optional); Region (optional); Use TLS (https); Access key and Secret key (both required)
NFS	Path (under the configured base directory): relative to the base directory your platform administrator configured; leave it blank to read the whole share
SFTP	Host and Username (required); Port (optional); Remote path (optional); Server host key (required); Password (required)

Credentials (the S3 keys and the SFTP password) are stored encrypted and never shown again. When you edit a pipeline, leave them empty to keep the saved ones.
Server host key is the server’s public-key line, for example the output of ssh-keyscan <host>. Olympusx AI pins it, so a different server answering at that address is refused.
NFS reads only from the share your platform administrator mounted for Olympusx AI. On an installation without one, an NFS pipeline cannot read anything.

Add code from the Code Assistant, which scans every file for secrets before it is indexed. See Code sources.

What you see

The pipelines list

Open Build › Knowledge, then the Pipelines tab: Ingestion pipelines, with a search box and Create pipeline for Admins and Builders.

Column	What it shows
Pipeline	Its name. Opens its page.
Target collection	The collection it loads, with its vector database and embedding model.
Source	Upload, URL, S3, NFS or SFTP.
Schedule	Manual, or its cadence with the next run time.
Last run	How its last run went, Not run yet, or Last run could not be read.
Created	When it was created.
Actions	Edit schedule and Delete pipeline, for Admins and Builders.

Opened from a collection, the list shows only that collection’s pipelines, with “Showing the pipelines that load collection” and Show all pipelines.

Empty states:

No pipelines yet, “A pipeline loads documents into a knowledge collection, and this tenant has none yet”: create a collection first. Go to vector databases takes you there.
No pipelines yet (or No pipeline loads collection yet): collections exist. Create pipeline starts the wizard.

A pipeline’s page

The header says which collection it loads. Admins and Builders see Run now (not on an upload pipeline), Edit schedule and Delete pipeline. The page has three tabs.

Overview:

Its facts: Target collection, Vector database, Embedding model, Parser, Chunking (“1,000 characters, overlapping by 100”), Source, Schedule with its time zone, Next run and Last run.
On an upload pipeline, Add a document (see below).
Run flow: the stages of the latest run, Parse, Chunk, Embed, Store and Index, each with what it handled (documents, characters, chunks, tokens, vectors) and its outcome: succeeded, Partly failed, failed or Not run. A failed run says where, for example “Failed at Embed. The embedding model did not return vectors.”, with Error details.

Runs: newest first, with Started, Status, Trigger (Scheduled, Run now or Upload), Docs, Chunks, Tokens and Duration. Choose a run’s start time to see its flow. Empty: No runs yet.

Documents: each document this pipeline loaded, newest first, with Filename, Status, Chunks, Size and Added, and a search box. Empty: No documents ingested yet.

What you can do

Create a pipeline

Open Build › Knowledge › Pipelines and choose Create pipeline. From a collection, choose Create pipeline there instead: the collection is filled in for you.
Details: enter the Pipeline name, choose the Vector database, then the Target collection.
Processing: choose the Parser and set Chunk size (chars) and Chunk overlap (chars).
Source: choose the Source and fill in its fields.
Schedule: choose Manual (no schedule), Hourly, Daily or Weekly, then the Minute, Hour and Day of week that apply. The step says when it will run and in which time zone.
Choose Create pipeline.

The pipeline does not run when it is created. Run it, wait for its schedule, or send it a file.

You can also create a collection and its first pipeline together; see Collections.

Choose a parser and chunking

Parser	Reads
Local (text, Markdown, HTML)	Text, Markdown, HTML, CSV, JSON and log files. Always available.
The document-converter parser (its label reads “PDF, DOCX, scans”)	PDF, DOCX, PPTX, XLSX, AsciiDoc and scanned images (PNG, JPEG, TIFF, BMP, WebP), as well as everything the local parser reads. Offered only when your installation includes the document converter.

Chunking cuts the parsed text into passages of about Chunk size characters, preferring paragraph and then sentence boundaries. Chunk overlap repeats the end of one chunk at the start of the next, so a sentence split across two chunks is still found. The defaults are 1,000 and 100.

The parser and the chunking are set when the pipeline is created. To change them, create a new pipeline.

Run a pipeline now

Open the pipeline.
Choose Run now.

The run lists the source and loads what it finds. When it ends, Olympusx AI shows Run finished with the documents, chunks and tokens, or The run did not start with the reason.

On every run, Olympusx AI skips a document that has not changed since the last run (same change tag at the source, or same content), and replaces the chunks of a document that has changed. A document removed from the source stays in the collection: delete and reload the collection to drop it.

Send a document to an upload pipeline

Open the upload pipeline. On Overview, under Add a document, choose Choose a file, or enter a Filename and paste the Document text.
Choose Ingest.

Olympusx AI reports “Ingested file. Chunks: n. Tokens: n.” To send several files at once, use Add documents on the collection instead; see Collections.

Change the source or schedule

Open the pipeline and choose Edit schedule, or choose it from the row’s Actions menu.
In Source and schedule, change the source, its fields or the schedule.
Choose Save. Olympusx AI shows Source and schedule saved and recomputes the next run.

Delete a pipeline

Choose Delete pipeline on the pipeline’s page or in the row’s Actions menu.
Confirm with Delete.

The pipeline never runs again, and its schedule, source settings, runs and document list are deleted. What it already loaded stays in the collection, so answers drawn from that collection keep working.

Limits

What	Limit
Pipeline name	Required, at most 60 characters, unique in the tenant
Collections per pipeline	One
One document	At most 16 MiB; Olympusx AI reads at most the first 16 MiB of an object from a source
Chunks per document	At most 2,000; a document that cuts into more fails
Chunk size	100 to 8,000 characters; default 1,000
Chunk overlap	Default 100; an overlap of the chunk size or more is replaced by the smaller of 100 and a quarter of the chunk size
Schedules	Hourly, daily or weekly, at a minute (0-59), hour (0-23) and weekday you choose; the console offers minutes in steps of 5
Schedule time zone	The platform’s schedule time zone, shown on the step; UTC unless your platform administrator set another at installation
URL fetch	One address per pipeline; the fetch gives up after 60 seconds
Budget	Runs are refused once the tenant’s budget for the period is spent