Skip to main content

Loaders

Loaders (often referred to as Sinks) are the final destination for your data. A loader takes the Arrow tables produced by an extractor and persists them into a permanent storage layer.

Configuration​

Sinks are declared in a centralized configuration file named gojjam_ingest_sinks.yml.

Below is an example configuration routing all extraction models inside ./ingest to a local PostgreSQL instance:

version: "1.0"
sinks:
  - name: "postgres_silver"
    type: "postgres"
    source_folder: "./ingest" # Routes data from all SQL extraction models in this folder
    write_mode: "APPEND"
    config:
      host: "localhost"
      port: 5432
      database: "warehouse"
      user: "admin"
      password: "password"
      schema: "public"

Core Sink Attributes Reference​

Regardless of the target storage type, every sink block utilizes these structural configuration properties:

AttributeDescription
nameUnique identifier for your target destination environment.
typeThe specific storage driver engine to load into (e.g., postgres, duckdb).
source_folderThis tells Gojjam: "Take the data extracted by any SQL models sitting inside this folder and load them directly into this specific database destination."
write_modeDetermines the operational strategy used when appending or replacing data inside target tables.
configA key-value block containing target-specific network credentials, databases, and schemas (varies depending on your chosen sink type).

Write Strategies​

Gojjam provides granular control over how data is written to the destination. This is configured via the write_mode parameter in your sink.yml.

ModeBehavior
APPENDAdds new records to the end of the existing table.
OVERWRITETruncates (deletes) the existing table and replaces it with the new data.
upsert (Coming Soon)Inserts new records and updates existing records matching a primary key.

Supported Sinks​

  • PostgreSQL: Optimized for production-grade relational storage.
  • DuckDB: Perfect for local development, data science, and "In-process" analytics.
  • Terminal: A "Debug" loader that simply prints the data to your console—essential for testing new SQL models.

Decoupling Logic​

The core philosophy of Gojjam is the separation of Ingestion and Transformation.

  1. The Loader lands the "Raw" data into the warehouse (often referred to as the Bronze or Staging layer).
  2. Once the load is successful, You can use Gojjam triggers the Transformation models or dbt to build your business-ready tables.