Skip to main content

Gojjam

Gojjam is a high-performance, code-first data pipeline framework built specifically for analytics engineers. It replaces brittle Python boilerplate with a simple, pure-SQL workflow, allowing you to manage both data ingestion and in-warehouse transformations without leaving your comfort zone.

The entire Gojjam philosophy can be summarized in three steps:

  1. Write YAML to configure your data connections.
  2. Write SQL to handle API pagination, cleaning, and modeling.
  3. Run your entire pipeline through a single, lightweight CLI.

Architecture Overview​

Gojjam completely decouples Data Ingestion from In-Warehouse Transformation, passing records through an ultra-fast streaming core.

   [ External Sources ]            [ Data Warehouse ]            [ Analytics Layer ]
(APIs, S3, Postgres Sources)        (Postgres, DuckDB)           (Production Tables)
             │                              │                             │
             ▼                              ▼                             ▼
     ┌───────────────┐              ┌───────────────┐             ┌───────────────┐
     │ gojjam-ingest │ ──────────>  │  Bronze/Raw   │ ──────────> │ gojjam-transform
     └───────────────┘              └───────────────┘             └───────────────┘

1. gojjam-ingest (The Streaming Ingestion Layer)​

The ingestion engine extracts data from external sources and lands it raw into your data warehouse.

  • Pure SQL Ingestion: Treat REST APIs, AWS S3, or Azure Blobs like virtual SQL tables. You can query them and perform in-flight flattening, casting, and filtering directly in SQL before the data hits your database.
  • SQL-Driven Pagination: No more custom Python loops for pagination or incremental tracking. Gojjam uses Calculated Models—simple SQL queries that run against your pipeline's metadata to dynamically compute the next page token, date range, or offset.
  • Memory-Efficient Core: Powered under the hood by Apache Arrow, data streams in small micro-batches. Your laptop or server will never experience memory spikes, whether you are loading 100 rows or millions.

2. gojjam-transform (The In-Warehouse Compute Layer)​

Once your data lands safely in the database (the staging or Bronze layer), Gojjam transitions processing entirely to your warehouse's native compute engine.

  • Pushdown Processing: Gojjam doesn't pull data out of your database to transform it. It executes native SQL directly inside your warehouse (like PostgreSQL or DuckDB) to maximize processing speed.
  • Automatic DAG Generation: Organizes your transformation scripts into an execution graph. Gojjam automatically figures out table dependencies and column lineage, ensuring staging tables finish building before production aggregates run.
  • Transactional Safety: If a SQL script breaks halfway through due to a typo or a bad join, Gojjam rolls back the transaction instantly so your production reporting layer never gets corrupted.

3. gojjam-cli (The Runner)​

The simple command-line interface used to execute your pipelines. It reads your files, maps your environment variables, and executes your pipelines with straightforward commands like gojjam run --ingest and gojjam run --transform.

Core Philosophy​

  • SQL-First Development: If you know SQL, you can build an enterprise-grade data pipeline from source to dashboard. No Python expertise required.
  • Strict Idempotency: By pairing stateful source cursors with fixed destination write strategies (APPEND or OVERWRITE), restarting a failed pipeline will never duplicate or corrupt your data.
  • Clean Separation of Concerns: Ingestion files handle connection settings and transport. Transformation files handle business logic and aggregations. They never cross paths.

Jump straight into the section relevant to your deployment goals:

  • Quickstart: Spin up a complete local project and execute a basic "Hello World" data pipeline in under 5 minutes.
  • Gojjam Ingest: Configure data sources, customize authentication profiles, setup relational loaders, and write SQL-driven calculated models.
  • Gojjam Transform: Manage internal DAG architectures, configure execution environments, and define complex model transformations.
  • Gojjam CLI: Master pipeline command parameters, runtime flags, and debugging workflows.