Skip to main content

Gojjam ingest

The Big Three: Core Components​

Gojjam ingest operates using three fundamental mechanisms. Data is pulled via Extractors, saved via Loaders, and intelligently orchestrated via Calculated Models.

[ Extractors ]  ======( Apache Arrow Streams )======>  [ Loaders (Sinks) ]
      â–˛
      │ (Injects Dynamic State Parameters)
      │
[ Calculated Models (SQL-Driven Cursors) ]

1. Extractors​

Extractors serve as the system ingress layer. Their primary responsibility is establishing connections with source protocols—whether they are REST APIs, object storage buckets, or traditional relational databases.

Instead of dumping massive payloads directly into system memory, Extractors read source records iteratively, chunking data into highly optimized Apache Arrow tables at a default batch threshold of 2,000 rows. This ensures the ingestion engine maintains an incredibly flat memory profile, even when streaming multi-gigabyte files.

2. Loaders (Sinks)​

Loaders act as the target destination layer (Sinks) for your processed streams. A Loader accepts the structured Arrow batches streaming from an Extractor and maps them into permanent physical storage layers like PostgreSQL, DuckDB, or Apache Iceberg tables.

By abstracting write strategies—such as handling structural table merges via APPEND configurations or completely re-initializing destination environments with OVERWRITE definitions—Loaders safely land raw data directly into your analytical bronze/staging layers while guaranteeing transactional integrity.

3. Calculated Models​

Calculated Models are what separate Gojjam from static ELT frameworks. Acting as the runtime orchestrator, Calculated Models let you deploy native SQL files to dynamically compute stateful ingestion parameters (such as API pagination limits, delta-load timestamps, or cursor offsets) on the fly.

Before an Extractor calls an API, Gojjam executes your calculated model SQL against its historical state metadata table, determines the exact parameters needed next, and injects them directly into the ongoing network request.

Component Deep Dives​

Ready to start building? Proceed to the detailed configuration and specification documentation for each module:

  • Read the Extractors Documentation to configure your API connections, storage paths, and database source credentials.
  • Explore the Loaders Documentation to manage storage write modes, data schemas, and transaction structures.
  • Master the Calculated Models Documentation to learn how to write advanced, loop-based SQL pagination structures.