Build and test a custom ingestion job

This exercise collects one page from your own authorised API. Use a test dataset and a small permitted response. You need Vantage access, Forge authoring, dataset ingestion permission and a deployment with confined script execution. An Admin must set the named secret in Console.

Select a small API contract

Choose an HTTPS endpoint returning this shape, or adapt the code and schemas to your API:

{"items":[{"id":"sample-1","text":"Your permitted source text"}],"nextCursor":"next-page"}

The response is illustrative. Your endpoint supplies the real data. Confirm that each ID identifies one stable source item.

Bundle type choices: Ingest, Format, Enrich, Aggregate, Workflow and Agent tool, with their roles explained. — Illustrative current bundle controls. Choose Ingest for collection or Agent tool for a tool implementation. The type is fixed at creation.

Create an Ingest bundle

Open Forge → Bundles, create an Ingest bundle and add main.ts. The bundle type is fixed at creation.

import { z } from "zod";
import type { BundleContext } from "@ctx/script-sdk";

const Params = z.object({ endpoint: z.string().url() });
const Page = z.object({
  items: z.array(z.object({ id: z.string().min(1), text: z.string() })),
  nextCursor: z.string().min(1).nullable(),
});

export default async function collect(ctx: BundleContext): Promise<void> {
  const { endpoint } = Params.parse(ctx.params);
  const cursor = ctx.checkpoint.current?.cursor;
  const url = new URL(endpoint);
  if (typeof cursor === "string") url.searchParams.set("cursor", cursor);
  const token = await ctx.secret("private-api-token");
  const response = await ctx.fetch(url.toString(), {
    headers: { Authorization: `Bearer ${token}` },
    signal: ctx.signal,
  });
  if (!response.ok) throw new Error(`API status ${response.status}`);
  const page = Page.parse(await response.json());
  await ctx.emit("item", page.items);
  ctx.checkpoint.propose({ cursor: page.nextCursor });
}

This collects one page per run. A null cursor starts again on the next run. Design a different completion policy for a finite export.

Declare the bundle contract

Set runtime 2026.10, entrypoint main.ts and import zod at 3.23.8. Declare these fields in the bundle descriptor:

{
  "role": "ingest",
  "runtime": "2026.10",
  "entrypoint": "main.ts",
  "imports": {"zod": "3.23.8"},
  "parameters": {
    "type": "object", "required": ["endpoint"],
    "additionalProperties": false,
    "properties": {"endpoint": {"type": "string"}}
  },
  "secrets": ["private-api-token"],
  "network": [{"host": "api.example.com", "port": 443}],
  "outputs": {
    "item": {
      "shape": "records", "fragment_kind": "text",
      "natural_key": ["id"],
      "schema": {
        "type": "object", "required": ["id", "text"],
        "properties": {"id": {"type": "string"}, "text": {"type": "string"}}
      }
    }
  },
  "checkpoint": {
    "type": "object", "required": ["cursor"],
    "properties": {"cursor": {"type": ["string", "null"]}}
  },
  "resources": {
    "wall_seconds": 60, "cpu_millicores": 250,
    "memory_bytes": 268435456, "output_bytes": 1048576,
    "temporary_storage_bytes": 1048576
  }
}

Replace api.example.com with your permitted host. The descriptor names source files separately in its published files map. Never put a token in the code or parameters.

Publish and configure the job

Inspect the code, declared host, schemas and resource limits. Publish a bundle version.Create a job targeting your test dataset. Select and pin that published version. Set endpoint to your permitted API URL. Keep triggers disabled.

Run an isolated test

Start the job’s test run. Check the emitted records, validation errors and proposed cursor.Confirm that production records and the production checkpoint did not change. Test malformed input, an empty page and an unavailable endpoint. Inspect the failed cases as well as the successful one.A draft bundle cannot be tested. File-output tests retain digests and metadata, rather than file bytes.

Activate deliberately

After the same job/version passes, choose one bounded production run. Inspect the destination records and saved checkpoint.Repeated IDs should refer to the same source item. Changed content creates another item version. Add a recurring trigger only after checking this behaviour.

Revise and recover

A changed bundle needs a new publication. Pin the new version and test again before production. Following latest has narrower compatibility rules; inspect changed secrets, hosts, outputs and checkpoint schema. An Ingest bundle cannot read its dataset. Use its checkpoint for continuity. A failed parse can recover retained bytes without fetching changed source data. Resetting a production checkpoint changes the next collection position. Pause continuous collection and wait for production runs to finish first. Continue with Run ingestion and inspect processing and Set up a schedule or webhook.

Getting Started

Kato

Arc

Vantage

Administration

Audit and compliance

Build and test a custom ingestion job

Select a small API contract

Revise and recover

​Select a small API contract

​Revise and recover

Select a small API contract

Revise and recover