Enhancement request - Please provide an OpenSearch destination

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Elastic made it latest libraries not work with OpenSearch
Please make a dedicated OpenSearch destination using the latest OpenSearch libraries.

Attempted Solutions

n/a

Proposal

Elastic made it latest libraries not work with OpenSearch
Please make a dedicated OpenSearch destination using the latest OpenSearch libraries.

References

Please have a look the blog post about client libraries available for OpenSearch: https://opensearch.org/blog/community/2021/08/community-clients/

Version

n/a

One way to get this going in the meantime is to use the AWS sigv4-proxy to sign the requests to OpenSearch. It's the same workaround outlined in #6204, so it's not ideal but it does work.

@jszwedko

@jszwedko I suppose we need some discussion here.

OpenSearch is getting more and more popular in the community for different reasons, so I think OpenSearch is kinda important for Vector.

If we agree to support OpenSearch, the main remaining question here - how it should be implemented in Vector. I suggest create dedicated source/sink for OpenSearch, even if it will share right now a lot of codebase with an existing ElasticSearch source/sink. In the future I guess OpenSearch and ElasticSearch will diverge more and more.

What do you think?

I'm open to creating a new opensearch sink. At present, I think it could largely just wrap the elasticsearch sink and set some defaults. If the HTTP APIs between the two products diverge more, than we could split up the implementation more, but I think just wrapping would be sufficient for now.

Forgive me for not knowing what's under the hood - but please know that elastic supplied libraries after 7.14 have be modified to specifically not work with OpenSearch. So if you are using these libraries - it would be best to start diverging sooner than later.

@jszwedko

@jszwedko as @ryn9 mentioned earlier, it is not possible since Elasticsearch team explicitly broken OpenSearch support in their libraries.

Forgive me for not knowing what's under the hood - but please know that elastic supplied libraries after 7.14 have be modified to specifically not work with OpenSearch. So if you are using these libraries - it would be best to start diverging sooner than later.

Ah, yes, meant to mention in my other comment that we don't rely on any SDKs for Elasticsearch but just make HTTP calls directly using the hyper crate.

@jszwedko

Hmmm, in this case I tend to agree with @jszwedko approach to create opensearch sink which just wraps with some defaults an existing elasticsearch sink. Later, if will be a need, we will be able to divide opensearch and elasticsearch sinks step by step.

Eventually there will be feature divergence for ingestion. When that the time comes - the OpenSearch project does maintain this library https://github.com/opensearch-project/opensearch-rs. The OpenSearch maintainers would love to hear your feedback on it and/or have you speak at one of their meetups: https://www.meetup.com/opensearch/

EDIT:
And that lib has aws sigv4 built in :)

@ryn9

@ryn9 you are right - eventually these projects will diverge a lot. We right now could start with already implemented elasticsearch sink and just wrap it as opensearch. Later, step by step, we can rewrite it with Opensearch-specific details in mind. E.g. start to use opensearch-rs in opensearch sink instead of raw hyper-based requests.

@ryn9

@ryn9 by the way, did you already try to use elasticsearch sink with Opensearch installation? Did you notice any problem?

Apologies - I have not tried for a while - but I believe it was working when I last tested against an OpenSearch 1.x release

I use the existing elasticsearch sink with OpenSearch 1.x in AWS and it works fine, with the caveat that I handle signing separately

I have another question
i have a vector installed in EKS cluster which is sending logs to AWS opensearch but when i see index in opensearch it is showing index only with name vector, so is there any way to configure index in config file of vector and the same we can see in opensearch.
Can you please help me into this?

I have another question i have a vector installed in EKS cluster which is sending logs to AWS opensearch but when i see index in opensearch it is showing index only with name vector, so is there any way to configure index in config file of vector and the same we can see in opensearch. Can you please help me into this?

https://vector.dev/docs/reference/configuration/sinks/elasticsearch/#bulk.index

Hello , i have configured vector in kubernetes cluster and it is taking kubernetes logs as a source and sinks as a elasticsearch, so just wanted to know is logs transferring to elasticsearch instantly ?
As i cannot see latest logs of my applications it is showing old logs for one month old
can you please help me into this?

Technically, Vector won't be sending it instantly, but it should be close enough given the above configuration. The default batch timeout for elasticsearch is just one second, after which it would send anything that has been queued up.

If you run vector with debugging enabled, do you see requests being sent out to the elasticsearch server?

For anyone stumbling upon this thread ... writing back to confirm that 0.26 is suppressing _type to opensearch 2.3, and successfully pushing messages.

live with a config that looks like this..
  sink_opensearch:
    type: elasticsearch
    inputs:
      - transform_remap_for_opensearch
    compression: gzip
    healthcheck: false
    endpoints:
      - "https://<CLUSTERNAME>.<REGION>.es.amazonaws.com:443"
    auth:
      strategy: "basic"
      user: "<USERNAME>"
      password: "<PASSWORD>"
    distribution:
      retry_max_duration_secs: 300
    bulk:
      index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

can we create multiple index from this like:
bulk:
index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"
index2: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

Is it possible to have multiple index?

can we create multiple index from this like: bulk: index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V" index2: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

Is it possible to have multiple index?

You need to create multiple sinks, each with their own index definition

@ryn9

can we create multiple index from this like: bulk: index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V" index2: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

Is it possible to have multiple index?

As @ryn9 pointed out, that's not a valid configuration. However I don't understand what you're trying to do with the two indices that the template fields don't already do. Any unique set of field1, field2, field3 (and the timestamp) will create a new index.

Additionally, please open a new Discussion for questions unrelated to the original issue - thanks.

Adding this issue as a difference between OS and ES we need to handle:
#17690

I'm using the elastic sink with the v7 api definition. It does work, but we would welcome an specific opensearch sink as well :)

Any update on this?

@jonasbartho

Any update on this?

Hi @jonasbartho, no news unfortunately. Have you tried this workaround #11738 (comment)

I also think it's not hard to introduce a dedicated sink based on the existing ElasticSearch sink if someone from the community is interested.

Nope, just using the elasticsearch sink for now.

Hey !
Any news about this issue ?
For the amount of people using opensearch with vector I think it would be worth it to support it "natively" and not a work around through elasticSearch sink... It is probably one of the most popular opensource sinks.

`It works because of the similarity of the ELK and OpenSearch. But at some point they could break it (like logstash, beats agents are not compatible with the latest Opensearch).

So ‘works’ != ‘is supported’ unfortunately. No one can complain that it does not work anymore.`

Thanks !!

Navigation Menu

Repository navigation

Enhancement request - Please provide an OpenSearch destination #11738

A note for the community

Use Cases

Attempted Solutions

Proposal

References

Version

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Enhancement request - Please provide an OpenSearch destination #11738

Description

A note for the community

Use Cases

Attempted Solutions

Proposal

References

Version

Activity

protochron commented on Jun 6, 2022

zamazan4ik commented on Nov 4, 2022

jszwedko commented on Nov 4, 2022

ryn9 commented on Nov 4, 2022

zamazan4ik commented on Nov 4, 2022

jszwedko commented on Nov 4, 2022

zamazan4ik commented on Nov 4, 2022

ryn9 commented on Nov 4, 2022

zamazan4ik commented on Nov 4, 2022

zamazan4ik commented on Nov 4, 2022

ryn9 commented on Nov 7, 2022

protochron commented on Nov 10, 2022

6 remaining items

nike21oct commented on Apr 6, 2023

ryn9 commented on Apr 6, 2023

nike21oct commented on Apr 24, 2023

bruceg commented on Apr 24, 2023

nike21oct commented on May 11, 2023

ryn9 commented on May 11, 2023

spencergilbert commented on May 11, 2023

spencergilbert commented on Jun 27, 2023

sandervandegeijn commented on Jul 20, 2024

jonasbartho commented on Mar 20, 2025

pront commented on Mar 20, 2025

jonasbartho commented on Mar 21, 2025

malhoudmouad commented on Jul 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions