Benefit of Gunicorn versus eventlet WSGI server

[lgnashold]

I'm creating a Django web application that also uses python-socketio.

I got the application working with an eventlet WSGI server as in the documentation. I know the traditional django stack is Django + Gunicorn + NGINX. However, I was having difficulties getting Gunicorn to play nice with eventlet and django.

Is there a significant performance benefit to using Gunicorn + eventlet, over just the eventlet WSGI server?**

I continue to receive the same error described here almost verbatim, although the solution (--preload) suggested doesn't help. I know this is more likely on Django's end, rather than python-socketio, but if anyone has resolved this issue further that would be appreciated.

It's also mentioned in the documentation that with gunicorn, monkey patching might need to be used with the database base drivers. Is there an example on how to do this with Django database drivers?

Finally, in the examples folder, I know there's a django example. This is a big ask, but would it okay if I added a database read or write to this example. Then when testing the various WSGI server options on this example, the interaction with Django's model would also be tested.

[miguelgrinberg]

@lgnashold The error is very clear. It indicates that you are taking database objects created in one thread and using them in another. It seems to me you are looking for workarounds instead of actually trying to solve this issue, which is very likely a bug in your application.

Is there a significant performance benefit to using Gunicorn + eventlet, over just the eventlet WSGI server?**

To know the answer to this question you have to run a test on both servers using your own application. In matters of performance there are no absolutes, there's lots of variables that influence performance so I cannot give you the answer you hope for. My guess is that there isn't going to be much of a difference, but Gunicorn is probably more robust than eventlet's WSGI server.

Another suggestion I can make is that if eventlet is making things complicated for you then you just drop it. These days you can run a WebSocket server using the simple-websocket package, which works with Gunicorn alone, without adding eventlet or gevent. See how in the documentation.

[lgnashold]

Thank you for the response, Miguel, and for maintaining python-socketio and the deployment documentation you've created. It's been very helpful.

It indicates that you are taking database objects created in one thread and using them in another.

To clarify, the error is clear. I'm using the Django ORM model, however, and I think the bug is specifically when you try to call ORM functions within a socketio event handler. You can reproduce this with a function as simple as

views.py:

@sio.event
def test_message(sid, message):
    MyModel.objects.create(
        my_field = str(random.random)
    )

models.py:

class MyModel(models.Model):
   my_field = models.CharField(max_length = 128)

Run with:

gunicorn gunicorn -k eventlet -w 1 django_socketio.wsgi:application

Along with the normal Django boiler plate as is in the example in this repository.

This implies that somewhere under the hood in Django, it is re-using the same database connection, and then when the socketio eventlet handlers tries to connect to this model object, it throws an error. I'm not specifically doing anything weird with threads -- this seems like it's the default behavior when you try to connect to a django ORM object from a socketio event handler. The reason I'm looking for a "work around" is because this database write is necessary in this handler, and I don't want to modify the internals of the Django ORM library.

I recognize this is more of an issue with Django rather than python-socketio, but was wondering if you had any feedback since I think writing to a django ORM database from a socketio handler is a relatively common use case. If not, that's okay.

[lgnashold]

I can actually produce a minimally working example based on the examples in this repository.

[lgnashold]

See https://github.com/lgnashold/python-socketio-example for a minimal example. If I'm not supposed to be using the Django ORM model from inside an eventlet handler, then I would be curious what the recommended behavior would be.

[miguelgrinberg]

I still don't know how to help with this. I searched for the error, and there are many reports of this occurring when using Django and eventlet, many times along with Celery. So clearly this isn't something that has to do specifically with Socket.IO, it just seems to be an incompatibility between Django and eventlet.

Many of the Celery reports indicate that the problem does not occur when using gevent instead of eventlet. Have you tried that? Or else, maybe just drop eventlet/gevent and go with standard threads? What is it about eventlet that is so important in your case?