Simplify our HTTP dependencies

[kelson42]

We use currently three HTTP libraries in Kiwix library:

• httplib
• libmicrohttpd
• libcurl

These is probably 2 too much.

I would like we consider as well https://nghttp2.org

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

@mgautierfr @veloman-yunkan As we are improving the CI/CD for Windows it appears that we would benefit to make a decision here. On my side, it looks like httplib is quite good, but I wonder a bit if the server part is really good at handling a high throughput traffic (pooling strategy, etc...) like we have at https://library.kiwix.org/. I don't think libcurl has anything to manager a HTTP server.

[kelson42]

On the server side, https://nghttp2.org has always made great first impression to me. Maybe we have here two problems: a lib for the client side and a lib for the server side.

[mgautierfr]

On my side, it looks like httplib is quite good, but I wonder a bit if the server part is really good at handling a high throughput traffic (pooling strategy, etc...)

It seems httplib has everything we need : https://github.com/yhirose/cpp-httplib#default-thread-pool-support and https://github.com/yhirose/cpp-httplib#override-the-default-thread-pool-with-yours

On the server side, nghttp2.org has always made great first impression to me.

It doesn't to me :)
The C api seems really low level and the c++ api is experimental and depends on boost.

[kelson42]

Pinging @legoktm as he might have a feedback about this ticket.

[legoktm]

I have a preference to any library that is already relatively well maintained in Debian :)

Both libcurl and libmicrohttpd are popular and used by other projects and have good track records behind them.

I didn't really give httplib a good look earlier since it was just being used in tests, I don't really know how widely it's used but I expect HTTP servers to be robust and have good security properties because of how often they're attacked. And then any HTTP client should properly use the system certificate store and crypto policies, etc. My quick search through the code indicates that httplib doesn't support stuff like HSTS, which is disappointing (but might not be a concern?).

[kelson42]

My quick search through the code indicates that httplib doesn't support stuff like HSTS, which is disappointing (but might not be a concern?).

This is a quite interesting point because to support SW based ZIM file, we will need to support HTTPS. Not sure what is the value of HTTPS with HSTS?!

[mgautierfr]

Maybe we have here two problems: a lib for the client side and a lib for the server side.

Probably yes. We use libcurl mainly to pilot the aria2c tools (and remote favicon listed in catalog). If we change this, it would be even better to have a websocket client. It would allow us to be notified by aria2c instead of polling it.
(And we use a subprocess because we fail to link/compile with libaria2 on windows. I don't remember why but we may rework on that and simply the client http library).

I didn't really give httplib a good look earlier since it was just being used in tests, I don't really know how widely it's used but I expect HTTP servers to be robust and have good security properties because of how often they're attacked. And then any HTTP client should properly use the system certificate store and crypto policies, etc. My quick search through the code indicates that httplib doesn't support stuff like HSTS, which is disappointing (but might not be a concern?).

On the server side, I would tend to say it (https) is not our problem. If you want https, use a reverse-proxy to do it (it is pretty easy with nginx, httpd or candy). It will be more secured, integrated with the distribution (and made by people who fully understand what they are doing with the certificate).

[kelson42]

Related to kiwix/kiwix-tools#416

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

Related to kiwix/kiwix-desktop#957 (comment).

I suspect that libmicrohhtpd and libcurl might be the final choices.

[kelson42]

libaria2 is also a candidate IMHO, see #1172