Rewrite backend with Django

[jdemaeyer]

@ubergesundheit @bCyberBasti @ThorbenJensen

This is a complete rewrite of the backend using Django and Django Rest framework instead of Peewee and FastAPI, including OpenAPI spec generation.

To Do:

• Check Dockerfile adjustments make sense and are deployable (may I offload this to you @ubergesundheit?)
• Adjust deployment for static file serving (needed by the web frontend of REST framework and by Django admin)
• Add minDate, maxDate filters for events endpoint (making sure they're picked up by the OpenAPI spec generator)
• Add location and organizer filters for events endpoint
• Update README
• Add Django admin functionality

Closes #250

[ubergesundheit]

Wow! Thanks for this serious effort! I'll take a look at the container image 👍

May I add these points/open questions to the To do list:

• The README of the backend needs to be changed
• Especially: Did something change with the execution of the scrapers?

[jdemaeyer]

You may add them but then they will be on there twice :P Scrapers are now run via ./manage.py crawl

[ubergesundheit]

🤦 I apologize for not reading through until the end

So here are (I think) some real additional TODOs:

• Add DJANGO_SECRET_KEY and DJANGO_HOSTS (maybe even more from settings.py) env to ansible (@bCyberBasti)
• Update CI config to use changed commands (@ubergesundheit)
• The old container did bind itself to 0.0.0.0. Is this preserved in the new version?
• Think about the new entrypoint script of the container image.. I'm not sure if we want to run database migration on every container start.

[jdemaeyer]

I'm hoping to get around to finishing the rewrite sometime this weekend...

[jdemaeyer]

Aaalrighty, this is ready from a "backend code" side! I have implemented everything I wanted to but there are still some tasks required to make this production-ready.

@ubergesundheit @bCyberBasti please take over these (or let me know what exactly you need from me):

Deployment

• Tasks from @ubergesundheit above (I have already adjusted the Django container to bind to all addresses)
• Set up static file generation and serving (roughly: we will need to run collectstatic somewhere inside the Django container, e.g. in serve.sh, configure the output directory of that to be a Docker volume, and make sure we serve files from this volume through a proper webserver, example setup)

Frontend Builds

Although the API responses look the same, the auto-generated OpenAPI spec between the old and new implementations seems different enough to break building the frontend. I don't know enough about the frontend to figure out how severe this is.
0c77151 is the commit that updates the API client and breaks the build. Looking at the diff of the two generated openapi.jsons is probably a good idea but I don't have time to look into that this week, probably not even next week. :/

Admin

I have added some basic Django admin config that allows editing events but is not very pretty. I have also added functionality to preserve edits across crawls and to hide events in a9db02f. It allows marking fields of an event as "dirty" (e.g. after manually editing the value), which will prevent the crawlers from overriding that field. That could serve as a poor man's implementation for some of the issues @ThorbenJensen outlined in #267.

[ubergesundheit]

Sorry for my delayed reaction, but I took some time off. Thus said, thank you very much thus far. I'll now try to resolve the outstanding issues. Starting with the generated frontend client.

[ubergesundheit]

@jdemaeyer Is the openapi generation under your control?

• do you think it is possible to teach django to include some basic error (404,400) cases in the openapi json?
• IMO the previous is related. Seems the openapi spec does not define any response property as required. This results that the generated TypeScript client interfaces define all response properties as such. Meaning additional checks for non null will be required in code in many places. I have to research if this can be done in the frontend, but if its easy to do in django, please do. You can also point me to relevant docs

[ubergesundheit]

Quick update from my side: 111c5a7 finally fixed build issues of the additional nginx container image required for static assets of django.

I'll now wait for changes of @jdemaeyer regarding possible changes to the openAPI spec

[jdemaeyer]

@ubergesundheit I have added hack-ish solutions for adding 400 and 404 responses, and for marking read-only properties as required :) This is likely as far as we can go without putting too much effort in, OpenAPI support in Django Rest Framework is still very new...

[ubergesundheit]

Thanks, I'll check it out 👍

[ubergesundheit]

So IMO this should be ready review. Thanks @jdemaeyer for your additional changes.

@bCyberBasti @ThorbenJensen lets keep this open to review once more

@bCyberBasti This branch also contains changes to the ansible playbook (containing django secret keys env changes), so it is required to execute the playbook on the server.

After this has been merged, we need to re-enable the "Check if generated client is up to date". It depends on the master container image but this is not up to date yet..