Proactor event loop warning opt-out message is, weird?

[davetapley]

this is more suited to 'Discussion', but they're not enabled on this repo 🤔

I got this message:

pyzmq/zmq/asyncio.py

Lines 52 to 55 in 206411c

	"Proactor event loop does not implement add_reader family of methods required for zmq."
	" zmq will work with proactor if tornado >= 6.1 can be found."
	" Use `asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())`"
	" or install 'tornado>=6.1' to avoid this error."

And dutifully installed tornado, but then I'm told:

pyzmq/zmq/asyncio.py

Lines 59 to 62 in 206411c

	"Proactor event loop does not implement add_reader family of methods required for zmq."
	" Registering an additional selector thread for add_reader support via tornado."
	" Use `asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())`"
	" to avoid this warning.",

I understand, but if I set WindowsSelectorEventLoopPolicy() to avoid the warning,
then doesn't that defeat the purpose of installing tornado?

... and it also causes this issue:

• WindowsSelectorEventLoopPolicy doesn't raise KeyboardInterrupt immediately python/cpython#101166

[minrk]

Yeah, it's still a warning because the event loop doesn't actually support what's required for zmq sockets, but pyzmq is instantiating an additional selector loop the user didn't ask for to work around that. You can suppress the warning if you are happy with this behavior. It should work, but I think it's worth the user being notified that it's happening.

I think it's worth a warning because this should be considered a bug in asyncio to fail to implement basic asyncio functionality by default. There are lots of good reasons why proactor is better, but it's lacking essential basic functionality (FD reader/writer) that libzmq sockets are impossible to poll without.

[davetapley]

@minrk I think I understand, but when you say:

You can suppress the warning if you are happy with this behavior

Do you mean by setting WindowsSelectorEventLoopPolicy (as the warning suggests)?

[minrk]

Sorry, no. I mean with a warnings filter.

[naseke]

I think the message is wrong because it states:
asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy()) which does not exist while asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) seems better
and

File "C:\Users\naseke\PycharmProjects\reedsolo.venv\Lib\site-packages\zmq\asyncio.py", line 51, in _get_selector_windows
from tornado.platform.asyncio import AddThreadSelectorEventLoop
ModuleNotFoundError: No module named 'tornado'

seems indicate that the tornado module is mandatory

This is also not stated in the documentation:
https://pyzmq.readthedocs.io/en/latest/api/zmq.asyncio.html

can we use zmq without zmq.asyncio in program with lib asyncio ?

libzmq 4.3.5
pyzmq 26.0.3
Python 3.12.2

[minrk]

The message is right, but it assumes an import. It could add the module name, though.

seems indicate that the tornado module is mandatory

And rightly so. pyzmq requires tornado to be compatible with ProactorEventLoop, as stated in the error message when you try to use it without tornado:

Proactor event loop does not implement add_reader family of methods required for zmq.
zmq will work with proactor if tornado >= 6.1 can be found.
Use asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())
or install 'tornado>=6.1' to avoid this error.

which continues to be accurate.

can we use zmq without zmq.asyncio in program with lib asyncio ?

Yes, you can use sync APIs just fine, but if you make a blocking call, it will block the event loop. You can combine polling intervals with zmq.DONTWAIT to avoid blocking, just like most other blocking libraries. For example:

s = ctx.socket(zmq.PULL)
# non-blocking recv with poll interval
async def _async_recv(s, poll_interval=0.2, timeout=5):
    """non-blocking async recv on a blocking zmq sockte"""
    t = time.monotonic()
    deadline = t + timeout
    while True:
        try:
            return s.recv_multipart(zmq.DONTWAIT)
        except zmq.Again:
            if time.monotonic() + poll_interval >= deadline:
                raise
            await asyncio.sleep(poll_interval)

Or you can put sockets in background threads with concurrent.futures, though you have to be very careful to always use a single thread for a given socket and not touch socket methods from the main thread, as libzmq sockets are not threadsafe.

[naseke]

Sorry for my late response... and thank you for your response. ^_^ I'll see how to integrate it.

On the other hand, what do you mean by your first sentence :

The message is right, but it assumes an import. It could add the module name, though.

in the error message we can clearly see that it is "zmq\asyncio.py" line 51 which is trying to import tornado... it is not part of my code

Or you can put sockets in background threads with concurrent.futures, though you have to be very careful to always use a single thread for a given socket and not touch socket methods from the main thread, as libzmq sockets are not threadsafe.

Finally, if I understood correctly, the sequence is as follows:

main -> init socket A -> processing -> socket A close
				|
				|-> thread or process -> init socket B -> processing -> socket B close

and not :

main -> init socket A -> processing -> socket A close
				|
				|-> thread or process -> use socket A -> processing -> End

[minrk]

what do you mean by your first sentence :

Sorry, I only meant that the sample code presented:

asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())

assumes that readers can find how to import:

from asyncio import WindowsSelectorEventLoopPolicy

where it could avoid that assumption with:

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Finally, if I understood correctly, the sequence is as follows:

Yes, that is correct. You can pass sockets across threads, e.g. using locks, etc. You just must make sure that they are not used concurrently across threads (e.g. not waiting in an eventloop in one thread when closed or accessed from another). But it is definitely safest to create/use/close sockets within a single thread.

[naseke]

assumes that readers can find how to import:
from asyncio import WindowsSelectorEventLoopPolicy

When I read this line I said to myself of course pfff ! 😅

Thank you for your time, it's clearer now 👍 😊