> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cortex-labs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Build and test a custom ingestion job

> Collect a bounded page from your private API with a versioned TypeScript bundle and isolated job test.

This exercise collects one page from your own authorised API. Use a test dataset and a small permitted response.

You need Vantage access, Forge authoring, dataset ingestion permission and a deployment with confined script execution. An Admin must set the named secret in Console.

## Select a small API contract

Choose an HTTPS endpoint returning this shape, or adapt the code and schemas to your API:

```json theme={null}
{"items":[{"id":"sample-1","text":"Your permitted source text"}],"nextCursor":"next-page"}
```

The response is illustrative. Your endpoint supplies the real data. Confirm that each ID identifies one stable source item.

<Frame caption="Illustrative current bundle controls. Choose Ingest for collection or Agent tool for a tool implementation. The type is fixed at creation.">
  <img src="https://mintcdn.com/cortex-ai/vcVF551foITpk__t/images/docs-20261019-bundle-types.jpg?fit=max&auto=format&n=vcVF551foITpk__t&q=85&s=15efe6446551453575838f8e14d7f1d1" alt="Bundle type choices: Ingest, Format, Enrich, Aggregate, Workflow and Agent tool, with their roles explained." width="700" height="345" data-path="images/docs-20261019-bundle-types.jpg" />
</Frame>

<Steps>
  <Step title="Create an Ingest bundle">
    Open **Forge → Bundles**, create an **Ingest** bundle and add `main.ts`. The bundle type is fixed at creation.

    ```typescript theme={null}
    import { z } from "zod";
    import type { BundleContext } from "@ctx/script-sdk";

    const Params = z.object({ endpoint: z.string().url() });
    const Page = z.object({
      items: z.array(z.object({ id: z.string().min(1), text: z.string() })),
      nextCursor: z.string().min(1).nullable(),
    });

    export default async function collect(ctx: BundleContext): Promise<void> {
      const { endpoint } = Params.parse(ctx.params);
      const cursor = ctx.checkpoint.current?.cursor;
      const url = new URL(endpoint);
      if (typeof cursor === "string") url.searchParams.set("cursor", cursor);
      const token = await ctx.secret("private-api-token");
      const response = await ctx.fetch(url.toString(), {
        headers: { Authorization: `Bearer ${token}` },
        signal: ctx.signal,
      });
      if (!response.ok) throw new Error(`API status ${response.status}`);
      const page = Page.parse(await response.json());
      await ctx.emit("item", page.items);
      ctx.checkpoint.propose({ cursor: page.nextCursor });
    }
    ```

    This collects one page per run. A null cursor starts again on the next run. Design a different completion policy for a finite export.
  </Step>

  <Step title="Declare the bundle contract">
    Set runtime `2026.10`, entrypoint `main.ts` and import `zod` at `3.23.8`. Declare these fields in the bundle descriptor:

    ```json theme={null}
    {
      "role": "ingest",
      "runtime": "2026.10",
      "entrypoint": "main.ts",
      "imports": {"zod": "3.23.8"},
      "parameters": {
        "type": "object", "required": ["endpoint"],
        "additionalProperties": false,
        "properties": {"endpoint": {"type": "string"}}
      },
      "secrets": ["private-api-token"],
      "network": [{"host": "api.example.com", "port": 443}],
      "outputs": {
        "item": {
          "shape": "records", "fragment_kind": "text",
          "natural_key": ["id"],
          "schema": {
            "type": "object", "required": ["id", "text"],
            "properties": {"id": {"type": "string"}, "text": {"type": "string"}}
          }
        }
      },
      "checkpoint": {
        "type": "object", "required": ["cursor"],
        "properties": {"cursor": {"type": ["string", "null"]}}
      },
      "resources": {
        "wall_seconds": 60, "cpu_millicores": 250,
        "memory_bytes": 268435456, "output_bytes": 1048576,
        "temporary_storage_bytes": 1048576
      }
    }
    ```

    Replace `api.example.com` with your permitted host. The descriptor names source files separately in its published `files` map. Never put a token in the code or parameters.
  </Step>

  <Step title="Publish and configure the job">
    Inspect the code, declared host, schemas and resource limits. Publish a bundle version.

    Create a job targeting your test dataset. Select and pin that published version. Set `endpoint` to your permitted API URL. Keep triggers disabled.
  </Step>

  <Step title="Run an isolated test">
    Start the job's test run. Check the emitted records, validation errors and proposed cursor.

    Confirm that production records and the production checkpoint did not change. Test malformed input, an empty page and an unavailable endpoint. Inspect the failed cases as well as the successful one.

    A draft bundle cannot be tested. File-output tests retain digests and metadata, rather than file bytes.
  </Step>

  <Step title="Activate deliberately">
    After the same job/version passes, choose one bounded production run. Inspect the destination records and saved checkpoint.

    Repeated IDs should refer to the same source item. Changed content creates another item version. Add a recurring trigger only after checking this behaviour.
  </Step>
</Steps>

## Revise and recover

A changed bundle needs a new publication. Pin the new version and test again before production. Following latest has narrower compatibility rules; inspect changed secrets, hosts, outputs and checkpoint schema.

An Ingest bundle cannot read its dataset. Use its checkpoint for continuity. A failed parse can recover retained bytes without fetching changed source data.

Resetting a production checkpoint changes the next collection position. Pause continuous collection and wait for production runs to finish first.

Continue with [Run ingestion and inspect processing](/vantage/bring-in-data/run-ingestion) and [Set up a schedule or webhook](/vantage/build-and-automate/schedule-or-webhook).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.