AmazonDynamoDB

AmazonDynamoDB source connector

Description

Read data from Amazon DynamoDB.

Key features

• batch
• stream
• exactly-once
• column projection
• parallelism
• support user-defined split

Options

name	type	required	default value
url	string	yes	-
region	string	yes	-
access_key_id	string	yes	-
secret_access_key	string	yes	-
table	string	yes	-
schema	config	yes	-
common-options		yes	-
scan_item_limit		false	-
parallel_scan_threads		false	-

url [string]

The URL to read to Amazon Dynamodb.

region [string]

The region of Amazon Dynamodb.

accessKeyId [string]

The access id of Amazon DynamoDB.

secretAccessKey [string]

The access secret of Amazon DynamoDB.

table [string]

The table of Amazon DynamoDB.

schema [Config]

fields [config]

Amazon Dynamodb is a NOSQL database service of support keys-value storage and document data structure,there is no way to get the data type.Therefore, we must configure schema.

such as:

schema {
  fields {
    id = int
    key_aa = string
    key_bb = string
  }
}

common options

Source Plugin common parameters, refer to Source Plugin for details

scan_item_limit

number of item each scan request should return

parallel_scan_threads

number of logical segments for parallel scan

Example

Amazondynamodb {
  url = "http://127.0.0.1:8000"
  region = "us-east-1"
  accessKeyId = "dummy-key"
  secretAccessKey = "dummy-secret"
  table = "TableName"
  schema = {
    fields {
      artist = string
      c_map = "map<string, array<int>>"
      c_array = "array<int>"
      c_string = string
      c_boolean = boolean
      c_tinyint = tinyint
      c_smallint = smallint
      c_int = int
      c_bigint = bigint
      c_float = float
      c_double = double
      c_decimal = "decimal(30, 8)"
      c_null = "null"
      c_bytes = bytes
      c_date = date
      c_timestamp = timestamp
    }
  }
}

Changelog

Change Log

Change	Commit	Version
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817)	https://github.com/apache/seatunnel/commit/55ed90eca	2.3.10
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6ee	2.3.10
[improve] update amazondynamodb connector (#8601)	https://github.com/apache/seatunnel/commit/a69efca0f	2.3.10
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)	https://github.com/apache/seatunnel/commit/6b7c53d03	2.3.9
[Feature][Doris] Add Doris type converter (#6354)	https://github.com/apache/seatunnel/commit/518999184	2.3.6
[Feature][Core] Upgrade flink source translation (#5100)	https://github.com/apache/seatunnel/commit/5aabb14a9	2.3.4
[Improve][Common] Introduce new error define rule (#5793)	https://github.com/apache/seatunnel/commit/9d1b2582b	2.3.4
[Improve] Code clean for AmazonDynamoDB connector (#5791)	https://github.com/apache/seatunnel/commit/a17dd7afc	2.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)	https://github.com/apache/seatunnel/commit/8de740810	2.3.4
[connector-v2] add amazondynamicdb source split (#5275)	https://github.com/apache/seatunnel/commit/740c14422	2.3.4
Support config column/primaryKey/constraintKey in schema (#5564)	https://github.com/apache/seatunnel/commit/eac76b4e5	2.3.4
[Improve]Remove scheduler in Dynamodb sink (#5248)	https://github.com/apache/seatunnel/commit/9e033a824	2.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)	https://github.com/apache/seatunnel/commit/51c0d709b	2.3.4
[Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146)	https://github.com/apache/seatunnel/commit/09995159a	2.3.3
Merge branch 'dev' into merge/cdc	https://github.com/apache/seatunnel/commit/4324ee191	2.3.1
[Improve][Project] Code format with spotless plugin.	https://github.com/apache/seatunnel/commit/423b58303	2.3.1
[improve][api] Refactoring schema parse (#4157)	https://github.com/apache/seatunnel/commit/b2f573a13	2.3.1
[Improve][build] Give the maven module a human readable name (#4114)	https://github.com/apache/seatunnel/commit/d7cd60105	2.3.1
[Improve][Project] Code format with spotless plugin. (#4101)	https://github.com/apache/seatunnel/commit/a2ab16656	2.3.1
[Feature][Connector] add get source method to all source connector (#3846)	https://github.com/apache/seatunnel/commit/417178fb8	2.3.1
[Feature][API & Connector & Doc] add parallelism and column projection interface (#3829)	https://github.com/apache/seatunnel/commit/b9164b8ba	2.3.1
[Hotfix][API] env required option can not set default value (#3584)	https://github.com/apache/seatunnel/commit/c5a23024f	2.3.0
[Feature][Connector-V2][AmazonDynamoDB] Add Factory for AmazonDynamoDB (#3348)	https://github.com/apache/seatunnel/commit/a0068efdb	2.3.0
[Improve][Connector-V2][AmazonDynamoDB] Unified exception for AmazonDynamoDB source & sink connector (#3333)	https://github.com/apache/seatunnel/commit/17bc5adce	2.3.0
[Connector-V2][Chore] Canonical name for AmazonDynamodb (#3321)	https://github.com/apache/seatunnel/commit/e216eb9a6	2.3.0
[Feature][Connector-V2] [Amazondynamodb Connector]add amazondynamodb source & sink connnector (#3166)	https://github.com/apache/seatunnel/commit/183bac02f	2.3.0