FakeSource

FakeSource connector

Description

The FakeSource is a virtual data source, which randomly generates the number of rows according to the data structure of the user-defined schema, just for some test cases such as type conversion or connector new feature testing

Key features

• batch
• stream
• exactly-once
• schema projection
• parallelism
• support user-defined split

Options

name	type	required	default value
schema	config	yes	-
row.num	int	no	5
split.num	int	no	1
split.read-interval	long	no	1
map.size	int	no	5
array.size	int	no	5
bytes.length	int	no	5
string.length	int	no	5
common-options		no	-

schema [config]

fields [Config]

The schema of fake data that you want to generate

Examples

  schema = {
    fields {
      c_map = "map<string, array<int>>"
      c_array = "array<int>"
      c_string = string
      c_boolean = boolean
      c_tinyint = tinyint
      c_smallint = smallint
      c_int = int
      c_bigint = bigint
      c_float = float
      c_double = double
      c_decimal = "decimal(30, 8)"
      c_null = "null"
      c_bytes = bytes
      c_date = date
      c_timestamp = timestamp
      c_row = {
        c_map = "map<string, map<string, string>>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_decimal = "decimal(30, 8)"
        c_null = "null"
        c_bytes = bytes
        c_date = date
        c_timestamp = timestamp
      }
    }
  }

row.num

The total number of data generated per degree of parallelism

split.num

the number of splits generated by the enumerator for each degree of parallelism

split.read-interval

The interval(mills) between two split reads in a reader

map.size

The size of map type that connector generated

array.size

The size of array type that connector generated

bytes.length

The length of bytes type that connector generated

string.length

The length of string type that connector generated

common options

Source plugin common parameters, please refer to Source Common Options for details

Example

FakeSource {
  row.num = 10
  map.size = 10
  array.size = 10
  bytes.length = 10
  string.length = 10
  schema = {
    fields {
      c_map = "map<string, array<int>>"
      c_array = "array<int>"
      c_string = string
      c_boolean = boolean
      c_tinyint = tinyint
      c_smallint = smallint
      c_int = int
      c_bigint = bigint
      c_float = float
      c_double = double
      c_decimal = "decimal(30, 8)"
      c_null = "null"
      c_bytes = bytes
      c_date = date
      c_timestamp = timestamp
      c_row = {
        c_map = "map<string, map<string, string>>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_decimal = "decimal(30, 8)"
        c_null = "null"
        c_bytes = bytes
        c_date = date
        c_timestamp = timestamp
      }
    }
  }
}

Changelog

2.2.0-beta 2022-09-26

• Add FakeSource Source Connector

2.3.0-beta 2022-10-20

• [Improve] Supports direct definition of data values(row) (2839)
• [Improve] Improve fake source connector: (2944)
  • Support user-defined map size
  • Support user-defined array size
  • Support user-defined string length
  • Support user-defined bytes length
• [Improve] Support multiple splits for fake source connector (2974)
• [Improve] Supports setting the number of splits per parallelism and the reading interval between two splits (3098)