Azure Blob Storage Extractor

The Azure Blob Storage Extractor enables Gojjam to ingest data from Azure's object storage service. Like the S3 Extractor, it is optimized for high-volume JSON ingestion, using streaming and batching to move data into your warehouse with minimal memory overhead.

Configuration

To use the Azure extractor, define a source in gojjam_ingest_sources.yml with type: azure_blob (or the equivalent identifier in your setup).

Basic Usage

sources:
  - name: "log_events_azure"
    type: "azure_blob_storage"
    container_name: "telemetry-data"
    root_folder: "prod/logs/" # Prefix to scan for blobs
    azure_blob_storage_connection_string: "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;"

---

SQL Transformation

Once the configuration is set up, you define how the extracted data lands in your warehouse by writing a standard SQL SELECT statement. Because Gojjam automatically unpacks and flattens your source JSON objects behind the scenes, you can manipulate raw fields using normal SQL logic. You declare the exact columns you want, rename fields, cast types, and run in-flight data cleansing natively.

Create a file named ingest/device_telemetry.sql:

-- ingest/device_telemetry.sql
-- Gojjam treats the unpacked blob JSON stream as a virtual table matching the source name

select
    -- 1. Explicit Type Casting & Enforcing Column Schemas
    cast(timestamp as timestamp) as observed_at,
    cast(device_id as varchar) as machine_id,

    -- 2. Renaming Unpacked Fields for Clean Destination Tables
    device_type as hardware_generation,
    firmware_version as fw_version,

    -- 3. In-flight Transformation & Arithmetic Manipulations
    temperature_celsius * 1.8 + 32 as temperature_fahrenheit,
    cpu_utilization_pct / 100.0 as cpu_load_ratio,
    memory_used_bytes / 1024 / 1024 as memory_used_mb,

    -- 4. Conditional Data Cleaning & Logic Mapping
    case
        when status_code = 200 then 'HEALTHY'
        when status_code = 500 then 'CRITICAL'
        else 'UNKNOWN'
    end as operational_status,

    -- 5. Standard Column Selections
    network_latency_ms,
    ip_address,
    region_label,
    environment

from device_telemetry
where environment = 'production';

---

Key Features

1. Multi-Record JSON Support

The extractor is designed to be flexible with your data format. It automatically detects and processes:

• Single JSON Objects: Ideal for individual log entries.
• JSON Lists: Ideal for batched exports from other systems.

2. Intelligent Streaming & Memory Management

To handle production-scale data lakes, the Azure extractor employs a Yield-on-Threshold strategy:

• Scanning: It lists all blobs starting with your root_folder prefix.
• Buffering: It downloads and parses blobs into a shared pending_records buffer.
• Flushing: Once the buffer hits 2,000 records, Gojjam yields a pyarrow.Table. This prevents the Python process from consuming excessive RAM when processing large containers.

3. Automatic Validation

The extractor silently skips empty blobs (Size = 0) to prevent pipeline interruptions and ensures that only valid JSON payloads are passed to the transformation layer.

---

Data Flow: From Blob to Warehouse

1. Extract: Gojjam fetches raw JSON from Azure.
2. Schema Enforcement: Using your ingest/log_events_azure.sql model, Gojjam casts types and renames columns.
3. Load: The cleaned Arrow table is pushed into your configured Sink (e.g., Postgres).