Prometheus

Prometheus sink connector

Support Those Engines

Spark
Flink
SeaTunnel Zeta

Key Features

• exactly-once
• cdc
• support multiple table write

Description

Used to launch web hooks using data.

For example, if the data from upstream is [label: {"__name__": "test1"}, value: 1.2.3,time:2024-08-15T17:00:00], the body content is the following: {"label":{"__name__": "test1"}, "value":"1.23","time":"2024-08-15T17:00:00"}

Tips: Prometheus sink only support post json webhook and the data from source will be treated as body content in web hook.And does not support passing past data

Supported DataSource Info

In order to use the Http connector, the following dependencies are required. They can be downloaded via install-plugin.sh or from the Maven central repository.

Datasource	Supported Versions	Dependency
Http	universal	Download

Sink Options

Name	Type	Required	Default	Description
url	String	Yes	-	Http request url
headers	Map	No	-	Http headers
retry	Int	No	-	The max retry times if request http return to IOException
retry_backoff_multiplier_ms	Int	No	100	The retry-backoff times(millis) multiplier if request http failed
retry_backoff_max_ms	Int	No	10000	The maximum retry-backoff times(millis) if request http failed
connect_timeout_ms	Int	No	12000	Connection timeout setting, default 12s.
socket_timeout_ms	Int	No	60000	Socket timeout setting, default 60s.
key_timestamp	Int	NO	-	prometheus timestamp key .
key_label	String	yes	-	prometheus label key
key_value	Double	yes	-	prometheus value
batch_size	Int	false	1024	prometheus batch size write
flush_interval	Long	false	300000L	prometheus flush commit interval
common-options		No	-	Sink plugin common parameters, please refer to Sink Common Options for details

Example

simple:

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  FakeSource {
    schema = {
      fields {
        c_map = "map<string, string>"
        c_double = double
        c_timestamp = timestamp
      }
    }
    plugin_output = "fake"
    rows = [
       {
         kind = INSERT
         fields = [{"__name__": "test1"},  1.23, "2024-08-15T17:00:00"]
       },
       {
         kind = INSERT
         fields = [{"__name__": "test2"},  1.23, "2024-08-15T17:00:00"]
       }
    ]
  }
}


sink {
  Prometheus {
    url = "http://prometheus:9090/api/v1/write"
    key_label = "c_map"
    key_value = "c_double"
    key_timestamp = "c_timestamp"
    batch_size = 1
  }
}

Changelog

Change Log

Change	Commit	Version
[Fix][Connector-V2] Fix prometheus check time can not parse double value (#9311)	https://github.com/apache/seatunnel/commit/fbf78721ab	2.3.12
[improve] http connector options (#8969)	https://github.com/apache/seatunnel/commit/63ff9f910a	2.3.10
[Fix][connector-http] fix when post have param (#8434)	https://github.com/apache/seatunnel/commit/c1b2675ab0	2.3.10
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6eeb	2.3.10
[Improve][dist]add shade check rule (#8136)	https://github.com/apache/seatunnel/commit/51ef800016	2.3.9
[Fix][Connector-V2] Fix cdc use default value when value is null (#7950)	https://github.com/apache/seatunnel/commit/3b432125ae	2.3.9
[Feature][Connector-V2] Add prometheus source and sink (#7265)	https://github.com/apache/seatunnel/commit/dde6f9fcbd	2.3.9