Skip to main content
Version: 2.1.3

Jdbc

Description

Read external data source data through JDBC

tip

Engine Supported and plugin name

  • Spark: Jdbc
  • Flink: Jdbc

Options

nametyperequireddefault value
driverstringyes-
jdbc.*stringno
passwordstringyes-
tablestringyes-
urlstringyes-
userstringyes-
common-optionsstringyes-

driver [string]

The jdbc class name used to connect to the remote data source, if you use MySQL the value is com.mysql.cj.jdbc.Driver.

Warn: for license compliance, you have to provide MySQL JDBC driver yourself, e.g. copy mysql-connector-java-xxx.jar to $FLINK_HOME/lib for Standalone.

password [string]

password

url [string]

The URL of the JDBC connection. Refer to a case: jdbc:postgresql://localhost/test

jdbc [string]

In addition to the parameters that must be specified above, users can also specify multiple optional parameters, which cover all the parameters provided by Spark JDBC.

The way to specify parameters is to add the prefix jdbc. to the original parameter name. For example, the way to specify fetchsize is: jdbc.fetchsize = 50000 . If these non-essential parameters are not specified, they will use the default values given by Spark JDBC.

user [string]

username

table [string]

table name

common options [string]

Source plugin common parameters, please refer to Source Plugin for details

Example

jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost:3306/info"
table = "access"
result_table_name = "access_log"
user = "username"
password = "password"
}

Read MySQL data through JDBC

jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost:3306/info"
table = "access"
result_table_name = "access_log"
user = "username"
password = "password"
jdbc.partitionColumn = "item_id"
jdbc.numPartitions = "10"
jdbc.lowerBound = 0
jdbc.upperBound = 100
}

Divide partitions based on specified fields

jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost:3306/info"
table = "access"
result_table_name = "access_log"
user = "username"
password = "password"

jdbc.connect_timeout = 10000
jdbc.socket_timeout = 10000
}

Timeout config