Skip to main content
Version: Next

Maxcompute

Maxcompute source connector

Description​

Used to read data from Maxcompute.

Key features​

Options​

nametyperequireddefault value
accessIdstringyes-
accesskeystringyes-
endpointstringyes-
projectstringyes-
table_namestringyes-
partition_specstringno-
split_rowintno10000
read_columnsArrayno-
table_listArrayNo-
common-optionsstringno
schemaconfigno

accessId [string]​

accessId Your Maxcompute accessId which cloud be access from Alibaba Cloud.

accesskey [string]​

accesskey Your Maxcompute accessKey which cloud be access from Alibaba Cloud.

endpoint [string]​

endpoint Your Maxcompute endpoint start with http.

project [string]​

project Your Maxcompute project which is created in Alibaba Cloud.

table_name [string]​

table_name Target Maxcompute table name eg: fake.

partition_spec [string]​

partition_spec This spec of Maxcompute partition table eg:ds='20220101'.

split_row [int]​

split_row Number of rows per split, default: 10000.

read_columns [Array]​

read_columns The columns to be read, if not set, all columns will be read. e.g. ["col1", "col2"]

table_list [Array]​

The list of tables to be read, you can use this configuration instead of table_name.

tunnel_endpoint[String]​

Specifies the custom endpoint URL for the MaxCompute Tunnel service.

By default, the endpoint is automatically inferred from the configured region.

This option allows you to override the default behavior and use a custom Tunnel endpoint. If not specified, the connector will use the region-based default Tunnel endpoint.

In general, you do not need to set tunnel_endpoint. It is only needed for custom networking, debugging, or local development.

Example values:

  • https://dt.cn-hangzhou.maxcompute.aliyun.com
  • https://dt.ap-southeast-1.maxcompute.aliyun.com
  • http://maxcompute:8080

Default: Not set (auto-inferred from region)

common options​

Source plugin common parameters, please refer to Source Common Options for details.

Examples​

Read with table​

source {
  Maxcompute {
    accessId="<your access id>"
    accesskey="<your access Key>"
    endpoint="<http://service.odps.aliyun.com/api>"
    project="<your project>"
    table_name="<your table name>"
    #partition_spec="<your partition spec>"
    #split_row = 10000
    #read_columns = ["col1", "col2"]
  }
}

Read with table list​

source {
  Maxcompute {
    accessId="<your access id>"
    accesskey="<your access Key>"
    endpoint="<http://service.odps.aliyun.com/api>"
    project="<your project>" # default project
    table_list = [
      {
        table_name = "test_table"
        #partition_spec="<your partition spec>"
        #split_row = 10000
        #read_columns = ["col1", "col2"]
      },
      {
        project = "test_project"
        table_name = "test_table2"
        #partition_spec="<your partition spec>"
        #split_row = 10000
        #read_columns = ["col1", "col2"]
      }
    ]
  }
}

Changelog​

Change Log
ChangeCommitVersion
[Improve][Connector-V2] Support maxcompute sink writer with timestamp field type (#9234)https://github.com/apache/seatunnel/commit/a513c495e3dev
[Feature][Transform] Support define sink column type (#9114)https://github.com/apache/seatunnel/commit/ab7119e5072.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Improve] maxcompute options (#9163)https://github.com/apache/seatunnel/commit/fdacbae1af2.3.11
[Fix][Connector-V2] Fix maxcompute write with multi parallelism (#9089)https://github.com/apache/seatunnel/commit/9426b7ba2c2.3.11
[Fix][Connector-V2] Fix maxcompute sink write date less than actual date (#8999)https://github.com/apache/seatunnel/commit/fc942a599b2.3.11
[Fix][Connector-V2] Fix maxcompute read with partition spec (#8896)https://github.com/apache/seatunnel/commit/e62bf6c65c2.3.10
[Fix][Connector-V2] Fix MaxCompute cannot get project and tableName when use schema (#8865)https://github.com/apache/seatunnel/commit/a24fa8fef62.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Feature][Connector-V2] Support maxcompute source with multi-table (#8582)https://github.com/apache/seatunnel/commit/0f782429232.3.10
[Fix][Connector-V2] Fixed adding table comments (#8514)https://github.com/apache/seatunnel/commit/edca75b0d62.3.10
[Improve][Connector-V2] MaxComputeSink support create partition in savemode (#8474)https://github.com/apache/seatunnel/commit/0b8f9de4652.3.10
[Improve][Transform] Rename sql transform table name from 'fake' to 'dual' (#8298)https://github.com/apache/seatunnel/commit/e6169684fb2.3.9
[Feature][Connector-V2] Support MaxCompute save mode (#8277)https://github.com/apache/seatunnel/commit/44ea675f1e2.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Core] Rename result_table_name/source_table_name to plugin_input/plugin_output (#8072)https://github.com/apache/seatunnel/commit/c7bbd322db2.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Fix] Fix dead link on seatunnel connectors list url (#7453)https://github.com/apache/seatunnel/commit/62b4f16f4e2.3.8
[BugFix][Connector-V2][Maxcompute]fix:Maxcompute sink can't map field(#7164) (#7168)https://github.com/apache/seatunnel/commit/d5abf8f5062.3.6
[Feature] Add unsupported datatype check for all catalog (#5890)https://github.com/apache/seatunnel/commit/b9791285a02.3.4
FakeSource support generate different CatalogTable for MultipleTable (#5766)https://github.com/apache/seatunnel/commit/a8b93805ea2.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Improve][Connector] Add field name to DataTypeConvertor to improve error message (#5782)https://github.com/apache/seatunnel/commit/ab60790f0d2.3.4
[Improve][Test] Move MaxCompute test case file (#5786)https://github.com/apache/seatunnel/commit/38132f51582.3.4
[Fix] Fix MaxCompute use not exist SCHEMA option (#5708)https://github.com/apache/seatunnel/commit/ba4782a67d2.3.4
[Feature] Support catalog in MaxCompute Source (#5283)https://github.com/apache/seatunnel/commit/946d89cb952.3.4
[Bugfix][Connector-V2][maxcompute] sink commit with Block not exsits on server (#4725)https://github.com/apache/seatunnel/commit/2760cae73c2.3.2
[Bug][Maxcompute] Fix failed to parse some maxcompute type (#3894)https://github.com/apache/seatunnel/commit/642901f0a22.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector] add get source method to all source connector (#3846)https://github.com/apache/seatunnel/commit/417178fb842.3.1
[Feature][API &amp; Connector &amp; Doc] add parallelism and column projection interface (#3829)https://github.com/apache/seatunnel/commit/b9164b8ba12.3.1
[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector (#3640)https://github.com/apache/seatunnel/commit/80cf8f4e422.3.0