Maxcompute

Maxcompute source connector

Description

Used to read data from Maxcompute.

Key features

• batch
• exactly-once
• column projection
• parallelism
• support user-defined split

Options

name	type	required	default value
accessId	string	yes	-
accesskey	string	yes	-
endpoint	string	yes	-
project	string	yes	-
table_name	string	yes	-
partition_spec	string	no	-
split_row	int	no	10000
read_columns	Array	no	-
table_list	Array	No	-
common-options	string	no
schema	config	no

accessId [string]

accessId Your Maxcompute accessId which cloud be access from Alibaba Cloud.

accesskey [string]

accesskey Your Maxcompute accessKey which cloud be access from Alibaba Cloud.

endpoint [string]

endpoint Your Maxcompute endpoint start with http.

project [string]

project Your Maxcompute project which is created in Alibaba Cloud.

table_name [string]

table_name Target Maxcompute table name eg: fake.

partition_spec [string]

partition_spec This spec of Maxcompute partition table eg:ds='20220101'.

split_row [int]

split_row Number of rows per split, default: 10000.

read_columns [Array]

read_columns The columns to be read, if not set, all columns will be read. e.g. ["col1", "col2"]

table_list [Array]

The list of tables to be read, you can use this configuration instead of table_name.

common options

Source plugin common parameters, please refer to Source Common Options for details.

Examples

Read with table

source {
  Maxcompute {
    accessId="<your access id>"
    accesskey="<your access Key>"
    endpoint="<http://service.odps.aliyun.com/api>"
    project="<your project>"
    table_name="<your table name>"
    #partition_spec="<your partition spec>"
    #split_row = 10000
    #read_columns = ["col1", "col2"]
  }
}

Read with table list

source {
  Maxcompute {
    accessId="<your access id>"
    accesskey="<your access Key>"
    endpoint="<http://service.odps.aliyun.com/api>"
    project="<your project>" # default project
    table_list = [
      {
        table_name = "test_table"
        #partition_spec="<your partition spec>"
        #split_row = 10000
        #read_columns = ["col1", "col2"]
      },
      {
        project = "test_project"
        table_name = "test_table2"
        #partition_spec="<your partition spec>"
        #split_row = 10000
        #read_columns = ["col1", "col2"]
      }
    ]
  }
}

Changelog

Change Log

Change	Commit	Version
[Fix][Connector-V2] Fix maxcompute read with partition spec (#8896)	https://github.com/apache/seatunnel/commit/e62bf6c65	2.3.10
[Fix][Connector-V2] Fix MaxCompute cannot get project and tableName when use schema (#8865)	https://github.com/apache/seatunnel/commit/a24fa8fef	2.3.10
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6ee	2.3.10
[Feature][Connector-V2] Support maxcompute source with multi-table (#8582)	https://github.com/apache/seatunnel/commit/0f7824292	2.3.10
[Fix][Connector-V2] Fixed adding table comments (#8514)	https://github.com/apache/seatunnel/commit/edca75b0d	2.3.10
[Improve][Connector-V2] MaxComputeSink support create partition in savemode (#8474)	https://github.com/apache/seatunnel/commit/0b8f9de46	2.3.10
[Improve][Transform] Rename sql transform table name from 'fake' to 'dual' (#8298)	https://github.com/apache/seatunnel/commit/e6169684f	2.3.9
[Feature][Connector-V2] Support MaxCompute save mode (#8277)	https://github.com/apache/seatunnel/commit/44ea675f1	2.3.9
[Improve][dist]add shade check rule (#8136)	https://github.com/apache/seatunnel/commit/51ef80001	2.3.9
[Feature][Core] Rename result_table_name/source_table_name to plugin_input/plugin_output (#8072)	https://github.com/apache/seatunnel/commit/c7bbd322d	2.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)	https://github.com/apache/seatunnel/commit/6b7c53d03	2.3.9
[Fix] Fix dead link on seatunnel connectors list url (#7453)	https://github.com/apache/seatunnel/commit/62b4f16f4	2.3.8
[BugFix][Connector-V2][Maxcompute]fix:Maxcompute sink can't map field(#7164) (#7168)	https://github.com/apache/seatunnel/commit/d5abf8f50	2.3.6
[Feature] Add unsupported datatype check for all catalog (#5890)	https://github.com/apache/seatunnel/commit/b9791285a	2.3.4
FakeSource support generate different CatalogTable for MultipleTable (#5766)	https://github.com/apache/seatunnel/commit/a8b93805e	2.3.4
[Improve][Common] Introduce new error define rule (#5793)	https://github.com/apache/seatunnel/commit/9d1b2582b	2.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)	https://github.com/apache/seatunnel/commit/8de740810	2.3.4
[Improve][Connector] Add field name to DataTypeConvertor to improve error message (#5782)	https://github.com/apache/seatunnel/commit/ab60790f0	2.3.4
[Improve][Test] Move MaxCompute test case file (#5786)	https://github.com/apache/seatunnel/commit/38132f515	2.3.4
[Fix] Fix MaxCompute use not exist SCHEMA option (#5708)	https://github.com/apache/seatunnel/commit/ba4782a67	2.3.4
[Feature] Support catalog in MaxCompute Source (#5283)	https://github.com/apache/seatunnel/commit/946d89cb9	2.3.4
[Bugfix][Connector-V2][maxcompute] sink commit with Block not exsits on server (#4725)	https://github.com/apache/seatunnel/commit/2760cae73	2.3.2
[Bug][Maxcompute] Fix failed to parse some maxcompute type (#3894)	https://github.com/apache/seatunnel/commit/642901f0a	2.3.1
[Improve][build] Give the maven module a human readable name (#4114)	https://github.com/apache/seatunnel/commit/d7cd60105	2.3.1
[Improve][Project] Code format with spotless plugin. (#4101)	https://github.com/apache/seatunnel/commit/a2ab16656	2.3.1
[Feature][Connector] add get source method to all source connector (#3846)	https://github.com/apache/seatunnel/commit/417178fb8	2.3.1
[Feature][API & Connector & Doc] add parallelism and column projection interface (#3829)	https://github.com/apache/seatunnel/commit/b9164b8ba	2.3.1
[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector (#3640)	https://github.com/apache/seatunnel/commit/80cf8f4e4	2.3.0