Http

Http source connector

Description

Used to read data from Http.

Key features

• batch
• stream
• exactly-once
• schema projection
• parallelism
• support user-defined split

Options

name	type	required	default value
url	String	Yes	-
schema	Config	No	-
schema.fields	Config	No	-
format	String	No	json
method	String	No	get
headers	Map	No	-
params	Map	No	-
body	String	No	-
poll_interval_ms	int	No	-
retry	int	No	-
retry_backoff_multiplier_ms	int	No	100
retry_backoff_max_ms	int	No	10000
common-options		No	-

url [String]

http request url

method [String]

http request method, only supports GET, POST method.

headers [Map]

http headers

params [Map]

http params

body [String]

http body

poll_interval_ms [int]

request http api interval(millis) in stream mode

retry [int]

The max retry times if request http return to IOException

retry_backoff_multiplier_ms [int]

The retry-backoff times(millis) multiplier if request http failed

retry_backoff_max_ms [int]

The maximum retry-backoff times(millis) if request http failed

format [String]

the format of upstream data, now only support json text, default json.

when you assign format is json, you should also assign schema option, for example:

upstream data is the following:

{"code":  200, "data":  "get success", "success":  true}

you should assign schema as the following:

schema {
    fields {
        code = int
        data = string
        success = boolean
    }
}

connector will generate data as the following:

code	data	success
200	get success	true

when you assign format is text, connector will do nothing for upstream data, for example:

upstream data is the following:

{"code":  200, "data":  "get success", "success":  true}

connector will generate data as the following:

content
{"code": 200, "data": "get success", "success": true}

schema [Config]

fields [Config]

the schema fields of upstream data

common options

Source plugin common parameters, please refer to Source Common Options for details

Example

simple:

Http {
    url = "https://tyrantlucifer.com/api/getDemoData"
    schema {
      fields {
        code = int
        message = string
        data = string
        ok = boolean
      }
    }
}

Changelog

2.2.0-beta 2022-09-26

• Add Http Source Connector