Extractors

Extractors are the "entry point" of the Gojjam engine. Their sole responsibility is to connect to a source, fetch data in a memory-efficient way, and stream it into the engine as a series of Apache Arrow tables.

Declaring an Extractor

To extract data using Gojjam, you need two simple files: a YAML configuration to define the connection source, and a pure SQL model to handle the data selection and in-flight transformations.

1. Define the Source Connection (gojjam_ingest_sources.yml)

Register your source, give it a name, and specify its type and connection endpoint:

version: "1.0"
sources:
  - name: "users_api"
    type: "http"
    endpoint: "https://jsonplaceholder.typicode.com/users"
    http_method: "GET"

2. Query and Transform the data from the API (ingest/users_api.sql)

Gojjam treats your declared source name as a virtual SQL table. You can query it directly using standard SQL syntax to handle in-flight transformations like flattening nested JSON payloads, filtering rows, and renaming columns before the data lands in your warehouse:

SELECT
    id,
    name,
    email,
    company.name AS company_name -- Flattening nested JSON payload in-flight
FROM users_api
WHERE id <= 10;

---

The Lifecycle of an Extraction

When you run gojjam run --ingest, the engine performs the following steps for each extractor:

1. Configuration Mapping: The engine reads the type attribute from your YAML and matches it to a specific Python class (e.g., http maps to HTTPExtractor, s3 maps to S3Extractor).
2. Context Injection: If pagination is needed, by utilzing Calculated Model, the engine calculates the required parameters (like date filters, page numbers, or offsets) and injects them into the extractor.
3. Streaming Fetch: The extractor begins fetching data from the endpoint.
4. Yielding Batches: To prevent system memory spikes, extractors do not return a single giant list. Instead, they stream data out, yielding it in efficient batches.

Supported Source Types

Extractor	Source Type
HTTP	REST APIs
PostgreSQL	Relational Database
AWS S3	Cloud Object Storage
Azure Blob Storage	Cloud Object Storage