Skip to main content
Version: Next

GoogleBigtable

Google Bigtable source connector

Support Those Engines​

SeaTunnel Zeta

Description​

Reads data from Google Cloud Bigtable using the native Bigtable Data v2 Java client.

Key Features​

tip

The source is bounded. It creates one split for the configured table or row-key range, so increasing job parallelism does not split one Bigtable scan into multiple tablet-range reads. Each scan reads every requested cell for the configured row range and emits one SeaTunnel row per Bigtable row.

Options​

nametyperequireddefault value
project_idstringyes-
instance_idstringyes-
tablestringyes-
credentials_pathstringno-
rowkey_columnlistno-
start_rowkeystringno-
end_rowkeystringno-
start_timestamplongno-
end_timestamplongno-
max_versionsintno1
scan_row_limitintno-1
common-optionsno-

project_id [string]​

Google Cloud project ID.

instance_id [string]​

Bigtable instance ID.

table [string]​

Bigtable table name to read from.

credentials_path [string]​

Path to the Google Cloud service account JSON key file. If omitted, Application Default Credentials (ADC) are used. ADC works automatically on GCE/GKE nodes, in gcloud shell sessions, or when the GOOGLE_APPLICATION_CREDENTIALS environment variable points to a service account JSON file.

rowkey_column [list]​

Optional list of field names that should receive the row key value. If this option is not set, the connector uses a schema field named rowkey as the row-key field.

Each listed field is decoded independently according to its own declared type in schema.fields: BYTES receives the raw row-key bytes; STRING receives a UTF-8 decoded view. Different row-key fields can therefore use different types in the same scan (for example one field exposing the raw key bytes for downstream binary processing, another exposing a UTF-8 view).

start_rowkey [string]​

Inclusive start row key for the scan. If not set, the scan starts from the beginning of the table.

The connector passes the value to the Bigtable client as a UTF-8 string; only lexicographic comparison is supported. Use BYTES for binary row keys that do not encode as UTF-8.

end_rowkey [string]​

Exclusive end row key for the scan. If not set, the scan reads to the end of the table.

start_timestamp [long]​

Inclusive start timestamp filter (microseconds since epoch). Combined with end_timestamp and max_versions, this controls which cell versions Bigtable returns for each column qualifier.

end_timestamp [long]​

Exclusive end timestamp filter (microseconds since epoch).

max_versions [int]​

Maximum number of cell versions to return per column qualifier. Default 1 returns only the latest version. Larger values expose historical cell versions; the source still emits one row per Bigtable row, so older versions of the same cell are flattened into the latest returned cell.

scan_row_limit [int]​

Maximum number of rows to return. -1 (default) means no limit. Use this option together with start_rowkey / end_rowkey to do paginated full-table scans across multiple jobs.

common options​

Source plugin common parameters, please refer to Source Common Options for details.

Schema Mapping​

Field names in the SeaTunnel schema must follow the pattern familyName:qualifier, for example cf:name or stats:age. The row-key field is controlled by rowkey_column; if it is not configured, the special field name rowkey maps to the Bigtable row key.

Schema field nameMapped Bigtable cell
rowkeyRow key
cf:nameColumn family cf, qualifier name
stats:ageColumn family stats, qualifier age
tip

The source reads the latest returned cell for each family:qualifier field. Use start_timestamp, end_timestamp, and max_versions to control the Bigtable scan filter. SeaTunnel field types must match the bytes stored in Bigtable. For example, numeric values written by this connector are binary big-endian values, while STRING, DATE, TIME, TIMESTAMP, and DECIMAL are UTF-8 text.

Task Example​

Read all rows with Application Default Credentials​

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  GoogleBigtable {
    project_id  = "my-gcp-project"
    instance_id = "my-bigtable-instance"
    table       = "events"
    schema {
      fields {
        rowkey    = BYTES
        "cf:type" = STRING
        "cf:ts"   = BIGINT
      }
    }
  }
}

Scan a row-key range with a service account​

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  GoogleBigtable {
    project_id       = "my-gcp-project"
    instance_id      = "my-bigtable-instance"
    table            = "events"
    credentials_path = "/secrets/sa-key.json"
    start_rowkey     = "2024-01-01#"
    end_rowkey       = "2024-02-01#"
    max_versions     = 1
    schema {
      fields {
        rowkey    = STRING
        "cf:type" = STRING
        "cf:data" = STRING
      }
    }
  }
}

Use a custom row-key field name​

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  GoogleBigtable {
    project_id    = "my-gcp-project"
    instance_id   = "my-bigtable-instance"
    table         = "events"
    rowkey_column = ["event_id"]
    schema {
      fields {
        event_id  = STRING
        "cf:type" = STRING
        "cf:data" = STRING
      }
    }
  }
}

Bounded streaming scan with cell-version filtering​

Use STREAMING job mode when you want the scan to run with checkpointing while still being a single bounded read. Combine start_timestamp, end_timestamp, and max_versions to restrict which cell versions Bigtable returns.

env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 60000
}

source {
  GoogleBigtable {
    project_id      = "my-gcp-project"
    instance_id     = "my-bigtable-instance"
    table           = "events"
    start_timestamp = 1704067200000000
    end_timestamp   = 1735689600000000
    max_versions    = 3
    scan_row_limit  = 500000
    schema {
      fields {
        rowkey    = STRING
        "cf:type" = STRING
        "cf:data" = STRING
        "cf:ts"   = BIGINT
      }
    }
  }
}

Changelog​

Change Log
ChangeCommitVersion
[Feature][Connector-V2] Add Google Cloud Bigtable Source and Sink connectorhttps://github.com/apache/seatunnel/commit/8e57c04dev