Apache Iceberg Loader

The Apache Iceberg Loader is responsible for persisting extracted data into high-performance lakehouse tables. It interfaces directly with Apache Iceberg catalogs to handle metadata management and writes underlying data files efficiently while ensuring seamless schema alignment.

Configuration

Sinks are defined in sink.yml (or gojjam_ingest_sinks.yml). The Apache Iceberg loader dynamically configures the table catalog and routes the data storage layout directly to the defined warehouse infrastructure.

Basic Usage

version: "1.0"
sinks:
  # Local development environment (SQLite / Local Filesystem)
  - name: "iceberg_local_dev"
    type: "iceberg"
    source_folder: "./ingest"
    catalog_type: "sqlite"
    catalog_name: "local"
    catalog_uri: "sqlite:///iceberg.db"
    namespace: "bronze"
    write_mode: "APPEND"
    warehouse_config:
      warehouse_type: "local"
      warehouse: "./warehouse"

  # Cloud Storage with Production PostgreSQL Catalog
  - name: "iceberg_s3_prod"
    type: "iceberg"
    source_folder: "./ingest"
    catalog_type: "postgres"
    catalog_name: "production"
    catalog_uri: "postgresql://admin:password@localhost:5432/warehouse"
    namespace: "bronze"
    write_mode: "APPEND"
    warehouse_config:
      warehouse_type: "s3"
      warehouse: "s3://my-data-bucket/warehouse"
      s3.access-key-id: "your_access_key"
      s3.secret-access-key: "your_secret_key"
      s3.endpoint: "https://s3.amazonaws.com"
      s3.region: "us-east-1"
      s3.force-virtual-addressing: false

---

Key Features

1. Flexible Write Modes

Gojjam supports two native table manipulation strategies matching the Iceberg specification:

• APPEND: Appends new data rows to the target dataset as a new table snapshot.
• OVERWRITE: Performs an atomic snapshot overwrite, replacing the current table data with the newly extracted data block.

2. Multi-Warehouse Routing

The loader supports decoupled storage targets via structural configuration mapping:

• local: Ideal for prototyping, local testing, and file-based pipelines.
• s3: Production-grade cloud storage utilizing robust S3 API endpoints.

3. Native Schema Integrity

Data streaming through the Iceberg loader automatically maintains rigid layout properties. The loader handles target table synchronization directly from PyArrow structures, guarding datasets against type mismatch anomalies or missing system fields.

---

Configuration Reference

Top-Level Fields

Field	Type	Description
name	String	A unique identifier for the Iceberg sink (e.g., iceberg_silver).
type	String	The target system type. Must be iceberg.
source_folder	String	The directory path where the loader looks for data layouts or structures to ingest.
catalog_type	Enum	The database backing the Iceberg catalog. Options: postgres or sqlite.
catalog_name	String	Name of the internal Iceberg catalog instance wrapper.
catalog_uri	String	The full connection URI string to access the metadata database catalog.
namespace	String	The target Iceberg namespace (similar to a database schema or folder grouping).
write_mode	Enum	Determines target table snapshot handling: APPEND or OVERWRITE. Default is APPEND.
warehouse_config	Object	Nested configuration containing backend storage properties.

Warehouse Configurations (warehouse_config)

Local Filesystem Variant (warehouse_type: "local")

Field	Type	Description
warehouse_type	String	Explicit identifier. Must be local.
warehouse	String	Local directory path matching the file layout landing root.

AWS S3 Variant (warehouse_type: "s3")

Field	Type	Description
warehouse_type	String	Explicit identifier. Must be s3.
warehouse	String	The cloud uri path (e.g., s3://my-bucket-name/prefix).
s3.access-key-id	String	AWS user access credential identity token.
s3.secret-access-key	String	AWS user secret token credential payload.
s3.endpoint	String	Custom destination target host (useful for alternative object storage or LocalStack).
s3.region	String	Target AWS geographic region context locator.
s3.force-virtual-addressing	Boolean	Sets URL path structures for bucket paths. Default is true.

---

Implementation Note:

All properties declared inside warehouse_config are tightly mapped and checked at boot time using strict discriminator validation rules. Invalid or misaligned config keys targeting storage options will fail explicitly prior to ingestion setup routines.