KAD firewalled after a period of operation

[Stoatwblr]

I've noticed this for a while and thought it was my firewall/router playing up, however it's not.

After a period of operation, amuled becomes unresponsive on the PtP TCP and UDP ports.

Attempting to connect to them from within the LAN results in connections timing out (not a refusal and not a firewall block) - it works fine for the first few minutes after amuled is restarted and seems to fall over after a number of connections (I haven't worked out the threshold yet)

This seems to have been started happening a couple of years back. I'm not 100% sure, but It appears as if amule isn't closing down connections properly and eventually runs out of handles.

Running (and compiled on) Ubuntu 18.10 at the moment. Built using:

./configure --enable-amule-daemon --enable-amulecmd --enable-webserver --enable-amule-gui --enable-cas --enable-wxcas --enable-alc --enable-alcc --enable-xas --enable-fileview --enable-geoip --enable-profile --enable-optimize --enable-static-boost --enable-mmap --enable-ccache --with-wxdebug --with-toolkit=auto --with-wxshared --with-boost CPPFLAGS="-DBOOST_ASIO_HASH_MAP_BUCKETS=53,389,1543,4093 -DBOOST_ASIO_ENABLE_BUFFER_DEBUGGING "

I've added the Autotools (#43), Upnp delay(#116), crashfix(#125), and libupnp1.8(#135) pulls - it compiles cleanly but the problem is still not solved.

Any ideas?

[biergaizi]

I'm not a developer, but I believe you should increase the maximum connections allowed on your system, regardless of whether there is a bug, you'll need to do it for running all P2P programs efficiently anyway, as they all create lots of connections and open many files, and the default number is often too low.

First, check the global limit imposed by the Linux kernel...

$ sysctl fs.file-max
fs.file-max = 1613718

You should see a similar large number. If not, you'll need to change /etc/sysctl.conf or a file under /etc/sysctl.d to increase it, and reboot the system.

Second, the default ulimit per user often imposes a low number of maximum files opened, and hampers P2P applications, check it by runnig

$ ulimit -a
16384

On many systems, the value is 512 or 1024, which is too low for P2P. You need to increase the limit by changing /etc/security/limits.conf, adding lines like

*               soft    nofile     16384
*               hard    nofile     16384

This increases the limit for every single user on the system. You can change * to your username, to increase the limit just for yourself, or change it to the user used by amule.

Once changed, you need to relogin. Then run ulimit -a again to see whether it has came info effect.

[Stoatwblr]

these are already set quite high (I dialled those up years ago)

Whatever's limiting this seems internal to amule (as in it's not closing connections down properly)

$ sysctl fs.file-max
fs.file-max = 8228647
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 65535
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 200000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

/etc/security/limits.d/local.conf:* hard nofile 100000
/etc/security/limits.d/local.conf:* soft nofile 100000

[biergaizi]

Yep, there's definitely a leak, Kad on my system gets disconnected for every ~24 hours. Someone who has the time and skills should investigate it.

[ValdikSS]

@Stoatwblr, try to compile without Boost support (remove --with-boost). I think the problem is in Boost.ASIO amule code.

[ValdikSS]

@Stoatwblr, and you're definitely don't want to use --enable-profile in non-profiling builds. Profiling slows down the program very much.

[biergaizi]

@ValdikSS No. Without Boost.ASIO, the original wx networking implementation has even more serious bugs. If you don't use Boost, aMule often gets out of control, the user interface stops responding, the Control+C handling timer stops working, and aMule keeps allocating memory and uses a full CPU core until the entire system is crashed.

Here's my previous result (at a downstream project): persmule/amule-dlp#15 I'm not sure if the problem still exists in the current master branch, but I don't have any confidence on non-Boost networking since few people are still using it, and it's likely undertested.

[ValdikSS]

@biergaizi, Yes, I already tried wx networking with amule and got the same result as you. But strangely, amuled with amulegui work fine, contrary to boost.asio. With boost.asio amulegui gets disconnected from amuled after several seconds to minutes, depending on the load. I tried to debug that and it seems that the socket just gets closed. In strace I see poll()/epoll() calls which indicate that some sockets are available for reading or writing but amule does not try to read or write to them. This is most probably why this (#137) bug is happening, too.

So, currently neither wx networking nor boot.asio is working properly.

[Stoatwblr]

Current version bcb39a8 works fine on Ubuntu 20.04 (boost 1.72) - at least as far as not firewalling is concerned.