Dynamic HTTP headers

[dolgovnv]

Have you considered the possibility of passing dynamic headers based on the values ​​in the record?
Perhaps something like the headers in Kafka SQL Connector - https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#available-metadata?

INSERT INTO topic
SELECT
  T.id,
  MAP[
    'header1', CAST(T.header1 AS BYTES)
  ] AS `headers`
FROM table AS T

There was no need for such a thing or could there be some difficulties with it?

[grzegorz8]

Hey @dolgovnv! As far as I know no one has requested such feature yet. I'm not familiar with Sink implementation, so at this moment it is hard for me to estimate effort needed to implement dynamic HTTP headers support.

Currently only static headers are supported for sink, for example:

CREATE TABLE http (
  id bigint,
  some_field string
) WITH (
  'connector' = 'http-sink',
  'url' = 'http://example.com/myendpoint',
  'format' = 'json',
  'gid.connector.http.sink.header.Origin' = '*',
  'gid.connector.http.sink.header.X-Content-Type-Options' = 'nosniff',
  'gid.connector.http.sink.header.Content-Type' = 'application/json'
);

The link you provided are readonly columns which contain metadata specific for the source. Currently, David is working on similar feature for HTTP connector: #154

[davidradl]

@dolgovnv I am curious about your use case.

[dolgovnv]

For example, there may be a task to pass a traceId and other service attributes that are not expected in the message body.

[davidradl]

@dolgovnv I see. Issue 154 is more about the headers from the response.
We have LookupQueryInfothat contains information around mapping columns to url paths, body or query params. This would seem a reasonable place to allow content to be mapped to headers. I would envisage that a query creator could then populate the header mapping into this object and the http connector could then honour these mappings. Maybe extending the very capable query creator GenericJsonAndUrlQueryCreator. Feel free to try this out.