Sources and processing

AI Draft

Choose an input source and follow a file until it is ready.

A file source supplies content. Processing prepares it for retrieval. The source type and file status tell your application what it must do next.

Choose a source

Source

Inline

Text in JSON

Source

Upload

PUT to a signed URL

Source

URI

A readable storage object

Add a file · receive file and job IDs

Pending

Await input

Upload bytes when required.

Indexing

Prepare content

Receive status events.

Ready

Retrieve

Search and Query can use the file.

If processing fails, read the file or job error. Correct the source before submitting a replacement.

Inline text, an upload, and a storage URI are three ways to supply content. Processing reports pending, indexing, ready, or failed states.

Source type	Use it when	Your next action
`inline`	The content is a small UTF-8 text string.	Send text in source.text, then wait for completion.
`upload`	Your application has bytes to upload.	PUT the bytes to the returned upload.url before upload.expires_at.
`uri`	The cluster can read an existing storage object.	Send its supported URI, then wait for completion.

Declare content_type for every source. /info lists the URI schemes available on the cluster. Do not assume that a cloud-storage scheme is supported because it is a valid URI.

Understand acceptance

Inline and URI additions normally return 202 Accepted with a file and a job. Upload creation returns 201 Created with those resources and an upload URL. The upload file remains pending until its bytes arrive.

A successful HTTP response acknowledges the request. It does not guarantee successful processing. Save the file ID and job ID before waiting for events. An upload that receives no bytes before expiry fails.

Observe file status

Status	Meaning
`pending`	The file awaits input or processing. An upload can be waiting for its PUT.
`indexing`	Engine is preparing the file for retrieval.
`ready`	The file can contribute to Search and Query.
`failed`	The file could not finish processing. Read its error and job error.
`deleting`	The file is being removed.

Receive file.status_changed for file state and job.updated for the operation. A recovery GET returns current state after a missed notification. Do not run a GET loop for live status.

Replace and recover

Engine indexes the content supplied by an add request. Changing an external storage object later does not update the collection. Add the same path with a new source and a new idempotency key to submit the replacement.

A URI read failure can return 422 storage_unreachable. An unsupported URI returns 400 invalid_uri. Correct storage access or the URI before another attempt. A processing failure appears on the file and job after acceptance.

Follow Add a file for complete examples.

Was this page helpful?