> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cortex-labs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Choose a data source, dataset, and collection

> Choose a supported collection route and understand how its data reaches a workspace.

A source identifies where material comes from. A dataset holds its versioned content. A collection makes permitted datasets available to a workspace.

| Part | What you choose | What it does |
| - | - | - |
| Source | A file, URL, folder, repository, API or supported feed | Identifies the material to collect. Source addresses contain no credentials. |
| Ingestion job | A published bundle, parameters and target dataset | Defines collection. Each run records the version, principal and output. |
| Dataset | A name, security level and billing team | Holds uploads, records, fragments and processing results. A dataset has no fixed content type. |
| Collection | A topical group of datasets | Establishes workspace reach. The same dataset can appear in several collections without copying. |
| Workspace | Your goal and audience | Supplies the investigation context and permitted consumers. |

## Choose the collection route

Start with a small source you can inspect. Check the **Source** choices in the job form and the selected bundle's parameters.

| Source | Configuration to check | Material limits |
| - | - | - |
| Upload | File format, path and any selected Format bundle | Uploading the same path creates another file version. Extraction support depends on format. |
| Web scrape or crawl | URLs, crawl boundaries, page limits and provider secret status | Collection time does not establish when a page's content was published. |
| Facebook, Twitter, YouTube, Instagram or TikTok | Profile, posts or single-post scope; date window; item and comment limits | Available scopes differ. Private, deleted or provider-restricted content can be unavailable. |
| LinkedIn | Profile URLs and recommendation depth | This source collects profiles. It does not offer every social collection scope. |
| Google Drive | Authorised folder, access and declared secret names | Confirm the selected folder and change behaviour before recurring collection. |
| Git repository | Supported GitHub/GitLab host, branch and include/ignore paths | Binary or oversized blobs can retain metadata without readable bytes. |
| AIS or ADS-B | Supported provider, area, identifiers and rate limits | Inspect feed coverage, provider permissions and collection gaps. |
| Private API or telemetry | Published custom Ingest bundle and bounded parameters | Requires the deployment's confined runtime and declared network destinations. |
| Webhook records | Format bundle and authenticated delivery | The **Webhook records** source accepts its declared JSON envelope. It does not fetch a source itself. |

Use only adapters offered by your deployment. A source family's presence in your task does not establish adapter support. For an unsupported source, use a supported export or [custom ingestion job](/vantage/bring-in-data/custom-ingestion-job).

For social collection, an item limit means sampling. Without an item limit, collection still has provider, spend and runtime bounds. Check comment and reply settings separately. Date filters and comment collection do not have identical behaviour across platforms.

## How collected material becomes evidence

Collection first accepts material into the dataset. Processing then creates addressable fragments, searchable text and eligible enrichment results. Each derived result identifies its inputs and versions.

A collected file can remain unavailable for search while extraction runs. Empty results can reflect access, collection scope, sampling or processing gaps. [Inspect processing](/vantage/bring-in-data/run-ingestion) before interpreting them.

## Access follows the resource

Workspace reach gives members Reader access to the dataset. Ingestion and changes need a Contributor grant on the dataset itself. Every attached dataset must fit the workspace security level.

Secrets are named centrally. An Admin sets tenant secrets in Console; an Operator manages platform-provided secrets. Jobs record the resolved version without showing its value. Attaching a dataset creates no new ingestion job or secret.

Continue with [Organise data through collections](/vantage/set-up-a-workspace/organise-collections) or [Set up ingestion with Kato](/kato/set-up-ingestion).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.