Support more sink

[lispking]

No description provided.

[liyiheng]

HTTP/Vector sink could act as a Vector source, HTTP/Vector source could act as a Vector sink.

Same with other sinks/sources that Vector supports. But only a few sinks communicate with Vector directly

[lispking]

HTTP/Vector sink could act as a Vector source, HTTP/Vector source could act as a Vector sink.

Same with other sinks/sources that Vector supports. But only a few sinks communicate with Vector directly

I've also been thinking about HTTP recently.

Although there are all kinds of HTTP requests, just providing basic HTTP functionality without a bit of customization may not be very useful.

Perhaps we could offer some fundamental HTTP capabilities, such as a retry mechanism and rate - limiting.

Then, for other related issues, like the implementation of gharchive #7 , we can perform secondary processing based on these capabilities.

This way, other HTTP services won't have to handle these basic capabilities again later.

What do you think?

[lispking]

HTTP/Vector sink could act as a Vector source, HTTP/Vector source could act as a Vector sink.

Same with other sinks/sources that Vector supports. But only a few sinks communicate with Vector directly

After some thought, I realize that implementing HTTP functionality as a separate issue #58 is indeed a good idea. This approach will make it more versatile.

[sankred9527]

btw : should we support foreach / foreachBatch features like spark has done ?

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch

[lispking]

btw : should we support foreach / foreachBatch features like spark has done ?

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch

It looks good. However, I don't know what the utilization rate of the sinks is for these two approaches.
(aha, I haven't used Spark for a relatively long time.)