Rust Arroyo adapter

[fpacifici]

A basic Rust Arroyo adapter to urn streaming pipelines.

The basic idea is to have the runtime built in rust and able to run either rust or python
application logic on top of it. Potentially mixing up rust and python on the same pipeline
on the same consumer should be possible.

The adapter is built with pyo3 and maturin and it is packaged inside the sentry_stream
python package as a binary.
The python runner is still the entrypoint. All the pipeline translation logic is still the same
and it is in python so only the basic primitives have to be written in rust.
At this stage only source, sink and map are provided, as a followup there will be a
basic rust Arroyo strategy that runs pyhon code so we can make python primitives
immediately runnable in rust and port them to rust native when we want. Still the
primitives will be implemented only once.

On top of this work we can define how to fully integrate Rust code in the pipeline.

Work to be done after this PR.

• Provide a standard rust/python mixed strategy so we can easily make the existing primitives from rust even if they are built in python
• Support full configuration for sources and sinks.
• Provide a better way to test the Rust consumer side from the python side. Now we only expose the run method which is untestable. Also support the local broker/consumer/producer everywhere
• Provide a way to run Rust native logic in a pipeline defined with the Python DSL
• Better typing
• Some load testing
• Support the missing primitives. This only supports map

Run it this way

python -m sentry_streams.runner \
    -n SimpleMap \
    --adapter rust_arroyo \
    --config ~/code/streams/sentry_streams/sentry_streams/deployment_config/simple_map.yaml \
    ~/code/streams/sentry_streams/sentry_streams/examples/simple_map.py

Architecture:

• The basic principle is to create a data structure similar to ArroyoConsumer (https://github.com/getsentry/streams/blob/main/sentry_streams/sentry_streams/adapters/arroyo/consumer.py#L25) but in Rust. This struct accumulates streaming primitives, reverses the order and builds a ProcessingStrategy. The ProcessingStrategy is an Arroyo Rust one.
• The python Arroyo Rust Adapter builds the ArroyoConsumer accumulating streaming primitives. and runs the rust code via pyO3.
• The application logic, at this point, is all Python, so the rust ArroyoConsumer actually runs the python application logic via pyO3. This allows us to move the consumer into Rust quickly and, step by step, support both rust native and python
• There is a number of data structures written in rust that are meant to allow the python code and the rust code to interact. I cannot expose to python any data structure coming from Arroyo Rust as they have to be flagged for pyO3 and they are not. This is why these structs exist: InitialOffset, PykafkaConsumerConfig, PyKafkaProducerConfig, Route, etc.
• The Arroyo Rust adapter needs to provide the operators to the rust code. This is done via the RuntimeOperator enum. It serves a similar purpose ArroyoStep https://github.com/getsentry/streams/blob/main/sentry_streams/sentry_streams/adapters/arroyo/steps.py#L29 does. Except this is served better by an enum in rust.
• When the Arroyo Rust adapter has composed the chain of operators with the add_step method the cool stuff happens in the run method: https://github.com/getsentry/streams/pull/98/files#diff-3d009ee5960444519bcb6053ac4ae9d659e392e0a5f80bd1a6f4f16e9a681b82R77. This assembles the Arroyo strategies (in rust) and produces a strategy factory, then it runs it.
• We pass python functions to the rust strategies this way: https://github.com/getsentry/streams/pull/98/files#diff-addae6f85d24be771742183cfd70bdbfc7b9a04d01f8b4a7f406bdc7615bb550R25-R45. This is an example of a transformer.

[untitaker]

is this supposed to be removed?

[fpacifici]

Yeah. It was a previous implementation. I forgot to remove.

[ayirr7]

Since the PR is fairly large, I think I'm having a bit of a hard time understanding how all the files in src/ fit together, as well as how they interact with rust_arroyo.py. Would you be able to explain that at a high level in this PR (or however you prefer)?

[ayirr7]

I don't know if I really understand why we have this Adapter? At a high level, does this accomplish something different than ArroyoAdapter in streams/?

[fpacifici]

Conceptually it does not do much differently. Though I created a different one because the data structures involved are quite different:

• ArroyoStep does not exist, it is replaced by a Rust enum.
• ArroyoConsumer is replaced by a rust one, but its interface in Rust is quite different.
• Arroyo Consumer and Producers have to be created by the Rust code because they are the rust ones so I cannot use the logic that creates the python ones. These are Arroyo structs, they cannot be exposed to python.
• I cannot expose the StrategyFactory to python (that's an arroyo interface) so everything has to be hidden behind the run method in the ArroyoConsumer Rust structure.

All in all it could be possible to merge the logic into the python Arroyo adapter but the result would be very brittle as the interfaces to deal with between python and Rust are very different.
The logic is more clear this way.

[fpacifici]

Since the PR is fairly large, I think I'm having a bit of a hard time understanding how all the files in src/ fit together, as well as how they interact with rust_arroyo.py. Would you be able to explain that at a high level in this PR (or however you prefer)?

Done. See the PR description