Skip to main content

Quick Start

Gojjam is a lightweight, architecturally decoupled data engine built for the modern engineer.

Follow these execution steps to clone the project repository, spin up your local database stack, and run a pipeline end-to-end.

Step 1: Clone the Repository​

Pull down the pre-configured starter project files and switch into the root working directory:

git clone https://github.com/gojjam-org/gojjam-starter.git
cd gojjam-starter

Step 2: Install Core Dependencies​

Install the Gojjam execution framework along with its required connector modules:

pip install -r requirements.txt

Step 3: Spin up the Warehouse​

Launch a localized PostgreSQL instance in the background using Docker Compose to serve as your data target:

docker-compose up -d

Step 4: Execute the Pipeline​

Initialize your project workspace and trigger a comprehensive ingestion and transformation run:

gojjam init
gojjam run --all

What Just Happened?​

When you fired off that final gojjam run --all flag, the orchestration engine split processing across its decoupled lifecycle layers:

  • Extract: The engine initiated an asynchronous connection loop to target the JSONPlaceholder REST API endpoint.
  • In-Flight Ingestion: Raw payloads were compiled directly into Apache Arrow columnar tables, handling scalar modifications inside local memory blocks.
  • Load: The micro-batched stream was written directly to the localized PostgreSQL target database instance.
  • Transform: Once landed, Gojjam parsed the SQL transformation models, resolved their lineage configurations, and executed optimal, native table schemas directly inside the warehouse.

Verify the Results​

Connect to your newly initialized local PostgreSQL instance using psql or your preferred SQL editor (like DBeaver or DataGrip) using the connection details specified in your environment settings to see the architecture layers:

  • Check the sample_users table to view the raw data structured from the API layer.
  • Check the agg_users table to view the final, dependencies-ordered business transformation.

Play around with it! You can modify the source configurations or add new .sql script files directly inside the /ingest and /transform folders to test Gojjam against different custom models.

How it Works: The Gojjam Flow​

Gojjam is built on a declarative philosophy. You define your connections in YAML and your logic in standard SQL.

1. Ingestion: Landing the Data​

Register your source in gojjam_ingest_sources.yml. Gojjam treats APIs as virtual tables.

version: "1.0"
sources:
  - name: "users_api"
    type: "http"
    endpoint: "https://jsonplaceholder.typicode.com/users"

Next, create an Ingest Model in ingest/users.sql. You can perform scalar transformations (renaming, casting, formatting, dropping records, imputation, etc.) before the data hits your warehouse.

-- Gojjam maps the API response to a table named 'users_api'
SELECT
    id,
    name,
    email,
    company.name AS company_name,
    UPPER(name) AS formatted_name -- Transform data in-flight
FROM users_api

2. Transformation: Modeling in the Warehouse​

Gojjam decouples the "Where" from the "What." Define your warehouse target in gojjam_transform_settings.yml by referencing your existing sinks.

version: "1.0"
transforms:
  - name: "main_warehouse_transform"
    source_folder: "./transform"
    config:
      ref: "sinks.postgres_silver" -- Reuses your PostgreSQL sink configuration

Now, write your business logic in the ./transform folder. These models execute natively on your warehouse:

-- Aggregate data already landed in the 'sample_users' table
SELECT
    company_name,
    COUNT(id) AS employee_count
FROM sample_users
GROUP BY 1

Project Structure​

  • gojjam_ingest_sources.yml: Registry for REST APIs, DBs, and S3 buckets.
  • gojjam_ingest_sinks.yml: Configuration file for defining your destination targets, specifying exactly where your extracted data should be loaded.
  • gojjam_transform_settings.yml: Configuration file for setting up post-load orchestration, specifying exactly where in-warehouse transformations should execute.
  • ingest/: Houses the SQL models used for extraction. You create SQL files here to explicitly declare what you want to extract—defining the target columns and any light, scalar pre-ingest transformations (like casting, renaming, or filtering) to apply before loading
  • transform/: Houses the SQL models where you define complex, heavy-duty data transformations that execute directly inside your data warehouse after ingestion is complete.

What's Next?​