Skip to main content

S3 Extractor

The S3 Extractor allows Gojjam to ingest data stored as JSON objects in AWS S3 buckets (or S3-compatible storage like MinIO). It is designed to traverse folders, handle large batches of files, and stream the results into your warehouse.

Configuration​

To use the S3 extractor, define a source in gojjam_ingest_sources.yml with type: s3.

Basic Usage​

sources:
  - name: "raw_events_s3"
    type: "s3"
    bucket_name: "my-data-lake"
    root_folder: "ingest/events/" # The prefix to scan
    region: "us-east-1"
    aws_access_key_id: "your_key"
    aws_secret_access_key: "your_secret"

How it Works​

1. Object Discovery​

Gojjam uses the list_objects_v2 paginator to scan the specified root_folder. It intelligently ignores:

  • Empty objects (Size = 0).
  • Folder markers (Keys ending in /).

2. JSON Parsing​

The extractor currently supports JSON content. It can handle files containing:

  • A Single Object: {"id": 1, "data": "..."}
  • A List of Objects: [{"id": 1}, {"id": 2}]

3. Efficient Batching​

To maintain a low memory footprint while processing thousands of files, Gojjam uses a Batching Buffer:

  • It collects records from multiple S3 objects until it reaches a 2,000-record threshold.
  • Once the threshold is hit, it yields a pyarrow.Table and clears the buffer.
  • This ensures that even if you have millions of small files, your system won't run out of RAM.

Local Development (LocalStack)​

Since Gojjam supports custom endpoints, you can point the S3 extractor to a local instance for testing:

sources:
  - name: "local_s3"
    type: "s3"
    bucket_name: "test-bucket"
    endpoint: "http://localhost:4566" # Pointing to LocalStack
    region: "us-east-1"
    aws_access_key_id: "test"
    aws_secret_access_key: "test"

SQL Mapping​

Like all Gojjam extractors, you can use a SQL model to filter or flatten the S3 data before loading.

-- ingest/raw_events_s3.sql
SELECT
    event_id,
    user_id,
    event_type,
    CAST(event_timestamp AS TIMESTAMP) as event_at
FROM raw_events_s3
WHERE event_type != 'debug'