Gojjam Transform

gojjam-transform is an in-warehouse compute engine. Its sole purpose is to take raw data landed by your ingestion pipelines and transform it into clean, business-ready tables.

---

The Core Transformation Workflow

To build transformations in Gojjam, you need two pieces for every transformation target:

1. A Global Configuration (gojjam_transform_settings.yml): This registers your orchestration target and environment connections.
2. A SQL Model Directory: A directory containing raw SQL queries paired with a configuration manifest detailing how and where they run.

┌─────────────────────────────────┐
│     gojjam_transform_settings   │ ──► [Specifies target warehouse connection]
└─────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────┐
│      ./transform folder         │ ──► [Houses your .sql queries and config manifests]
└─────────────────────────────────┘

---

The Transformation Configuration

The transformation layer is declared inside your global gojjam_transform_settings.yml configuration file. This registers a transformation job block and points Gojjam to the directory containing your business logic.

Example of a Postgres config

version: "1.0"
transforms:
  - name: "main_warehouse_transform"
    source_folder: "./transform"
    config:
      host: "localhost"
      port: 5432
      database: "gojjam_db"
      user: "admin"
      password: "secure_password"
      # OR if you want to reuse you gojjam_ingest_sink configs
      ref: "sinks.postgres_silver" # Links directly to your registered destination connection

Configuration Reference

• name: A unique string identifier for this specific transformation run.
• source_folder: The local directory path where Gojjam will scan for your transformation code models.
• config: A standard key-value credential block (containing host, port, database, user, password, schema)
• config.ref: A string pointer that reference-links back to a destination sink credentials block defined in your gojjam_ingest_sinks.yml file. This prevents you from redefining database passwords and connection hosts in multiple files.

---

Modeling and DAG Execution

Inside your designated source_folder (e.g., ./transform), you organize your transformation definitions using native .sql files.

Gojjam automates execution using an underlying Directed Acyclic Graph (DAG) framework:

• Lineage and Dependencies: Gojjam scans the references in your SQL files to build a structural model lineage map. It maps out upstream dependencies cleanly, ensuring that staging cleaning operations finish executing before production aggregation code blocks begin.
• Transactional Isolation: Transformations run inside managed, isolated database transactions. If an upstream script breaks due to a bad join or syntax error, the execution pipeline immediately drops downstream updates to avoid corrupting your production reports.

SQL Transformation example

Create a file named source_folder/fct_telemetry.sql:

select
    count(*) as total_events,
    avg(temperature_celsius) as avg_temperature,
    max(cpu_utilization_pct) as peak_cpu_pct,
    min(network_latency_ms) as min_latency_ms

from device_telemetry

---

Supported Database

Gojjam is designed to land your structured Arrow data streams into enterprise-grade analytics stores.

Target Database / Lakehouse	Status
PostgreSQL	Suppored
Snowflake	Coming Soon
Apache Iceberg	Coming Soon
AWS Redshift	Coming Soon