Prometheus

Prometheus source connector

Description

Used to read data from Prometheus.

Key features

• batch
• stream
• parallelism

Options

name	type	required	default value
url	String	Yes	-
query	String	Yes	-
query_type	String	Yes	Instant
content_field	String	Yes	$.data.result.*
schema.fields	Config	Yes	-
format	String	No	json
params	Map	Yes	-
poll_interval_millis	int	No	-
retry	int	No	-
retry_backoff_multiplier_ms	int	No	100
retry_backoff_max_ms	int	No	10000
enable_multi_lines	boolean	No	false
common-options	config	No	-

url [String]

http request url

query [String]

Prometheus expression query string

query_type [String]

Instant/Range

1. Instant : The following endpoint evaluates an instant query at a single point in time
2. Range : The following endpoint evaluates an expression query over a range of time

https://prometheus.io/docs/prometheus/latest/querying/api/

params [Map]

http request params

poll_interval_millis [int]

request http api interval(millis) in stream mode

retry [int]

The max retry times if request http return to IOException

retry_backoff_multiplier_ms [int]

The retry-backoff times(millis) multiplier if request http failed

retry_backoff_max_ms [int]

The maximum retry-backoff times(millis) if request http failed

format [String]

the format of upstream data, default json.

schema [Config]

Fill in a fixed value

    schema = {
        fields {
            metric = "map<string, string>"
            value = double
            time = long
            }
        }

fields [Config]

the schema fields of upstream data

common options

Source plugin common parameters, please refer to Source Common Options for details

Example

Instant

source {
  Prometheus {
    plugin_output = "http"
    url = "http://mockserver:1080"
    query = "up"
    query_type = "Instant"
    content_field = "$.data.result.*"
    format = "json"
    schema = {
        fields {
            metric = "map<string, string>"
            value = double
            time = long
            }
        }
    }
}

Range

source {
  Prometheus {
    plugin_output = "http"
    url = "http://mockserver:1080"
    query = "up"
    query_type = "Range"
    content_field = "$.data.result.*"
    format = "json"
    start = "2024-07-22T20:10:30.781Z"
    end = "2024-07-22T20:11:00.781Z"
    step = "15s"
    schema = {
        fields {
            metric = "map<string, string>"
            value = double
            time = long
            }
        }
    }
  }

Changelog

Change Log

Change	Commit	Version
[Fix][Connector-V2] Fix prometheus check time can not parse double value (#9311)	https://github.com/apache/seatunnel/commit/fbf78721ab	2.3.12
[improve] http connector options (#8969)	https://github.com/apache/seatunnel/commit/63ff9f910a	2.3.10
[Fix][connector-http] fix when post have param (#8434)	https://github.com/apache/seatunnel/commit/c1b2675ab0	2.3.10
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6eeb	2.3.10
[Improve][dist]add shade check rule (#8136)	https://github.com/apache/seatunnel/commit/51ef800016	2.3.9
[Fix][Connector-V2] Fix cdc use default value when value is null (#7950)	https://github.com/apache/seatunnel/commit/3b432125ae	2.3.9
[Feature][Connector-V2] Add prometheus source and sink (#7265)	https://github.com/apache/seatunnel/commit/dde6f9fcbd	2.3.9