Support WASM for native zmq in the web browser

[TheButlah]

One of Rust's strengths is is wasm support, letting developers run things natively in the web browser. It would be great if we could plan to support wasm in the future.

Looking into this, tokio currently doesn't support wasm, and async-std has a minimal feature set that supports wasm. It seems like a trivial way to use tcp in the browser is not yet feasible, but we should be on the lookout for ways to support it in the near future. One possible approach might be to use websockets instead of tokio sockets when compiling for wasm, and perhaps unbake the assumption that the sockets are managed by tokio by abstracting them with a common trait.

I also found that using websockets as a transport is an actual RFC, and seems to be the way things are going in the rest of the zmq libraries rather than trying to do tcp with websockets:

• ZWS 1.0 RFC: https://rfc.zeromq.org/spec/39/
• ZWS 2.0 RFC (Draft phase): https://rfc.zeromq.org/spec/45/
• Tracking progress for websocket support in libzmq: WebSocket transport progress libzmq#3581
• Javascript impl of zmq that exclusively uses websockets: https://github.com/zeromq/jszmq

That implies that instead of trying to figure out how to get websockets to work with tcp, we can instead treat them as different transports using the ws:// and wss:// notation. Maybe that would solve these issues?

On the other hand, it looks like maybe tcp with websockets ought to still be supported. The javascript library I linked can do either tcp:// or ws://.

[TheButlah]

I've added ws:// and wss:// to the transport todo list here: #65. Unsure if thats the right approach, but we can figure it out in the future.

[TheButlah]

Update: I've learned a bit more about web sockets vs tcp. Web sockets is a protocol, not a transport (i.e. HTTP vs tcp). It is built on top of TCP. However, you cannot connect a websocket client to a tcp socket unless the server is also using websockets - just like you can't make an HTTP request to a tcp socket unless its actually a HTTP server listening on that socket.

This is why the ws:// and wss:// transports exist - they descibe how ZMQ sockets work over web sockets. Therefore, you will never be able to do tcp:// in the browser, but you can do wss:// in the browser and connect to a server also using wss:://.

[TheButlah]

I've looked into the ecosystem to support this a bit.

First, there is the question of the async executor. Tokio does not support WASM, and doesn't seem to have any plans to do so. async-std currently has paritial support for WASM - importantly, it builds for the platform and task::spawn works (according to here). Its important to note that we technically care only about the executor, not the other functionality of tokio/async-std, but we may consider transitioning as an option. We should try to take care to write our code as runtime-independent as possible.

Next, there is the question of how to get a websocket library that works with the async ecosystem. The best option looks to be async-tungstenite, which can work across either tokio or async-std

We also need to consider how to address the lack of certain transport types on WASM. As previously stated, tcp simply won't exist on wasm - pretty much the only transport mechanism will be ws:// or wss://. We will probably need to use #[cfg()] to prevent the code for those transport types from attempting to compile on WASM.

[TheButlah]

@Alexei-Kornienko would you be open to a PR to get the code to work on both async-std and tokio, via a feature flag? My hope is we can design the code to be mostly runtime-agnostic, and the bits that will actually need one or the other can use either depending on the flag. It doesn't look like tokio has any plans to add wasm support, and async-std already works on wasm (partially). More importantly though, a library as fundamental as zmq ought to work with either, to have the best adoption. Plus, it would just be so cool to get the wasm stuff working! :)

[Alexei-Kornienko]

@TheButlah In theory I'm open for such contributions. However I'm not sure that it will be easy to implement. You can take a look at codec code. We'll have to implement something like Codec and Framed for async-std

[TheButlah]

@Alexei-Kornienko looking into it, the Codec and Framed stuff shouldn't really depend on any particular async runtime - it just needs common traits from futures like Stream and Sink. Tokio provides them from tokio-util, but no alternative appears to exist in async-std, presumably because Codecs aren't a concept in the rust standard library. That being said, conceptually the functionality is quite simple.

I found https://crates.io/crates/futures_codec, which looks to have pretty low development velocity, but the crate is small (about 1k lines of code) and seems feature complete. I think it ought to be feasible to use that crate in lieu of tokio-utils's Framed, since all either of those are doing is providing a helper struct and traits to create Streams and Sinks - no actual runtime-specific code is needed.

Once we do that, I'm sure we can abstract over tokio and async-stds other concepts and try to rely exclusively on futures whenever possible for common functionality like traits and channels.

[Alexei-Kornienko]

@TheButlah I don't think that completely replacing a tokio code is a good idea. Instead I think it can hidden behind a feature flag. So for tokio variant it will compile with tokio provided codec implementation and for async-std it may have a separate codec impl.

[TheButlah]

Oh, I didn't mean to imply that we should replace tokio code across the board - just that we should strive to use runtime-agnostic crates when we can to be as flexible as possible.

Most of the tokio-specific code seems to be in the codec module, as well as when binding/listening on the tcp transport. Since WASM doesn't have tcp, this code can be gated on a feature flag, and doesn't need to be ported to be runtime-agnostic (although doing so anyway might be helpful to avoid fragmenting the async ecosystem further). We can use async-tungstenite to handle the websocket stuff, which should be runtime independent already. Then all thats left is the possibility that we will have some spawn_blocking or block_on calls which are runtime specific scattered in non-transport related code. In those cases, we will have to see what the best way to handle them is.

But overall, I think the path forward is probably the following:

• Update the codec module to use futures_codec instead of tokio-utils (since that is common functionality, cant be feature-gated, and would need to be runtime-agnostic).
• Find any other common code that isn't tcp specific that uses tokio-only features and see if there is anything tricky we need to do, or if making it runtime-agnostic is trivial.
• Feature gate the TCP transport - it can continue to use regular tokio since it wont need to compile on WASM anyway
• Get the code to compile on WASM and add a CI check for that
• Implement the ZWS 2.0 RFC

This should have a minimal impact on code that is heavily dependent on tokio, as most of that code is transport-specific and doesn't need to compile to WASM anyway

[TheButlah]

I've implemented some first steps for this in #77 and #83 - it extracts most of the tokio specific code into one centralized location, and also refactors the rest of the code to use a runtime-agnostic codec type (FramedIo). I suspect it would be simple to get the code to work with both async-std and tokio, if that PR goes through.

[TheButlah]

I've implemented the feature gating in #111

[mrdunky]

Would love to see this project succeed.
However, I have read in a couple of places that webassembly does not have access to the the host environment when running in the browser. So no network APIs etc.
Did this change, or turn out to be incorrect?

https://stackoverflow.com/questions/52102612/is-it-possible-to-submit-http-requests-with-webassembly

[TheButlah]

@mrdunky web assembly does not provide any networking apis in the spec, but because we are running in the browser we can call the browser's networking APIs via wasm-bindgen and web-sys:
https://rustwasm.github.io/wasm-bindgen/examples/fetch.html

So to answer what I think you're asking, yes we can basically do anything we could do in javascript in a frontend context, including accessing network APIs