HTTP Extractor

The HTTP Extractor enables you to treat REST APIs as virtual SQL tables. It supports multiple authentication methods, HTTP verbs, and stateful pagination strategies driven entirely by Calculated Models.

Configuration

To use the HTTP extractor, define a source in your gojjam_ingest_sources.yml file with type: http.

Basic Usage

sources:
  - name: "my_api"
    type: "http"
    endpoint: "https://api.example.com/v1/data"
    http_method: "GET" # Supports GET, POST, PUT

---

Authentication

Gojjam supports standard authentication strategies out of the box to connect securely with production APIs.

Strategy	auth_type	Required Fields
Basic	basic	username, password
JWT	jwt	api_key

Example (JWT):

auth_type: "jwt"
api_key: "your_bearer_token_here"

---

Stateful Pagination (Cursors)

Gojjam handles complex pagination by maintaining an execution state (cursor). Unlike traditional data tools that rely on static values or rigid code loops, Gojjam calculates the next cursor value using SQL logic declared inside Calculated Models.

Cursor Settings

The cursor block defines how page states are calculated, injected, and persisted between pipeline runs.

• state: Set to STATEFULL to store the cursor in a database (enabling pipelines to resume exactly where they left off), or STATELESS for fresh, full-overwrite runs.

• cursor_type:

• INC (Incremental): Runs the extraction once using the current state value, updates the checkpoint, and finishes.

• SYNC (Synchronous Loop): Continuously extracts data and updates the state in a loop until the source returns an empty dataset.

• calculated_model_name: The name of the SQL file in your calculated models directory that computes the next page marker.

Pagination Configuration

version: "1.0"
sources:
  - name: "users_api"
    type: "http"
    endpoint: "https://api.example.com/v1/posts"
    http_method: "POST"

    # Define the dynamic loop logic for this specific source
    cursor:
      cursor_type: "SYNC" # Use 'SYNC' for continuous looping until the response is empty
      state: "STATEFUL"
      calculated_model_name: "posts_paginator"
      calculated_model_folder_path: "./calculated_models"
      calculated_model_column_names:
        - name: "created_at"
          initial_value: 1715817600 # Starting point if no run history exists
      value_location:
        type: "BODY"
        location:
          timestamps: created_at # Maps the SQL column 'created_at' to the JSON key 'timestamps'

For detailed examples of how to write pagination queries, check out the Calculated Models documentation.

---

Data Mapping & Transformation

Once configured, create a SQL file in your Ingest Model directory (ingest/your_file.sql) to handle in-flight, row-level transformations. Because Gojjam automatically flattens nested data structures behind the scenes, you can easily filter records, cast types, and clean payloads before the data lands in your sink.

-- ingest/my_api.sql
-- Gojjam treats the unpacked API response stream as a virtual table

SELECT
    user.id,
    user.age,
    COALESCE(user.email, 'N/A') as email,
    current_timestamp() as ingested_at
FROM my_api

---

Implementation Note

If your pipeline requires pagination, your primary logic will live within Calculated Models. By declaring your cursor calculations inside a standard SQL SELECT statement, you can handle intricate API architectures—like rolling time windows, delta offsets, or custom cursor responses—natively without writing custom Python looping scripts.