Cloudflare support

[09173732546]

hi jgmdev,

I installed this on centos 7 and when i tried to simulate ddos in my server/site, the ddos -v lists the various Cloudflare IPs as the attacker, and the connection count range from ~1 - ~100, effectively not blocking anything below the treshold.

Can you help how can this be fixed? The bulk of attack is from Cloudflare IP itself

[jgmdev]

What connection range isn't getting blocked? maybe your ignore.ip.list is permitting the undesired connections. Also you should set the NO_OF_CONNECTIONS flag to the max desired amount of connections per ip.

Also, how are you running ddos? in daemon/service mode or cron job? Just in case service mode is better.

[09173732546]

Hi,

Im using centos 7, in daemon mode. I havent included anything in ignore.ip.list except my shared host ip for remote database: 192.3.x.x. Also, the ddos.conf is modified only for NO_OF_CONNECTIONS=100 and FIREWALL=iptables as i am using vestacp. My domain is under cloudflare-free, and my vps is using vestaCP.

To clarify, when i simulate DOS on my ip/domain, the attacker IP (me) is only 1 hit, and excessive hits are delivered by cloudflare IPs (probably due to reverse proxy of nginx/cloudflare).

I see in issue #27 that the author was able to modify for cloudflare/nginx use, but that was outdated and i am not confident with that fork.

Please see the screenshot: http://prntscr.com/n3swe9 (the IP in red is my isp, which shows 1 hit. the IPs in orange are cloudflare servers with massive hits in just few seconds of running HULK dos)

I cant set NO_OF_CONNECTIONS lower than 100, for possible web visitors who are using shared connections.

[jgmdev]

Ok lets see if I understand correctly...

You are using cloudflare to get some protection but instead cloudflare is distributing the dos attack on multiple servers and delivering it to your server? Isn't cloudflare supposed to prevent a dos attack since your server is behind their network?

How do you know those orange connections are generated from your dos test? If that's the case then cloudflare is making matters worse (maybe on purpose)...

Now I wonder if it's possible to monitor each connection cloud flare ip header (CF-Connecting-IP) using some tool since it is impossible to get that from ss or netstat alone (unless we scan a web server logs which seems unelegant)... that way we could know the real ip and add an iptable rule to drop those packets as I wrote on #27.

Using a rule like
iptables -A INPUT -m string --algo bm --string "CF-Connecting-IP: A.B.C.D" -j DROP
would help but packets would still need to be analyzed which isn't great from a performance perspective. I have toyed with the idea of porting ddos to C to gain lower level access but I would need to learn many networking stuff as kernel things and I don't have enough free time for that so maybe there is a way to analyze each connection packet to see if the cloudflare ip is included and drop if too many of them.

Edit: Maybe using a tool like wireshark to sniff packets containing the CF ip header would be possible.
Edit 2: tcpdump does what is needed with the -n -A flags but have to figure a way to filter the tcpdump data to just obtain what we need.

[09173732546]

Hi there,

You are using cloudflare to get some protection but instead cloudflare is distributing the dos attack on multiple servers and delivering it to your server? Isn't cloudflare supposed to prevent a dos attack since your server is behind their network?

My domain is behind cloudflare, and the vps is using VestaCP nginx+httpd. This is the out-of-box config section of nginx.conf http://prntscr.com/n4bjyh

How do you know those orange connections are generated from your dos test? If that's the case then cloudflare is making matters worse (maybe on purpose)...

Its range on https://www.cloudflare.com/ips , the vps/domain has few to no outside visit coz its still being tested. When i use HULK.py dos tool, my own ip is there, just marked as 1 hit, while the Cloudflare ips populated the ddos.sh logs with several hits and the vps became laggy. as soon as i close the hulk dos, that IPs vanish and vps resumes normal.

Now I wonder if it's possible to monitor each connection cloud flare ip header (CF-Connecting-IP) using some tool since it is impossible to get that from ss or netstat alone (unless we scan a web server logs which seems unelegant)... that way we could know the real ip and add an iptable rule to drop those packets as I wrote on #27.
Using a rule like
iptables -A INPUT -m string --algo bm --string "CF-Connecting-IP: A.B.C.D" -j DROP
would help but packets would still need to be analyzed which isn't great from a performance perspective. I have toyed with the idea of porting ddos to C to gain lower level access but I would need to learn many networking stuff as kernel things and I don't have enough free time for that so maybe there is a way to analyze each connection packet to see if the cloudflare ip is included and drop if too many of them.

The prospect users of the developed site will be chatters.. in that case they could have same IPs.. dropping IPs could have a bad impact, or it could produce some internal errors due to blocking of proxy? I'm not really sure.

I found this on stackoverflow, can a logic like this work on ddos.sh? https://stackoverflow.com/a/47127844

[jgmdev]

Could you try running this command on the vps:

tcpdump -w output.pcap -n -A 'tcp port 80 or tcp port 443' | grep --line-buffered "CF-Connecting-IP:"

to monitor the received packets which include the CF ip header and launch the hulk dos test tool again and report the output?

Or much better run

tcpdump -w output.pcap

launch the dos test with hulk and when finished kill tcpdump with CTRL-C and upload the output.pcap file so I can analyze it with tcpdump locally.

Edit: The issue with these method of obtaining the real ip would be that on https connections I guess the CF-Connecting-IP header would be encrypted... so no way of reading it... making mandatory to read the server logs, but that is why I want you to test the method because I haven't used CF.

[09173732546]

Pre-dos https://budflick.com/output1.pcap
Ongoing-dos: https://budflick.com/output2.pcap
Post-dos: https://budflick.com/output3.pcap

[jgmdev]

Thanks! this worked beautifully:

tcpdump -r output2.pcap -n -A | grep CF-Connecting-IP:

With that we could maybe limit tcpdump for 1 second, grep the CF header and extract the ip's :)

[09173732546]

tcpdump -r output2.pcap -n -A | grep CF-Connecting-IP:

oh. so the logged CF-ips (in ddos -v) are false positive?

With that we could maybe limit tcpdump for 1 second, grep the CF header and extract the ip's :)

pardon me, i dont know :( ... the tcpdump only display my own ip, no cloudflare ip ranges.
but in ddos -v the cloudflare ips show.

if in theory i have 100 users with the same IP accessing the site at the same moment, some of them would be dropped? (i set the connection limit to 100)

update: the problem i see is that when i re-run the hulk.py dos, tcpdump shows the successive hits from my own ip. but in ddos -v my own ip shows 1-3 hits. effectively not hitting the connection limit as the hits are being distributed to that false-positive cf ips :(

[jgmdev]

As it seems cloudflare is distributing each of your request to a different cloudflare server and sending it to your VPS as you wrote before with the orange square of ip's screenshot. So basically we need a new ddos.conf option named CF_MODE=true|false (accompanied with implementation for it) that when true would also use tcpdump to retrieve the list of ip addresses (by reading the CF-Connecting-IP) that are making multiple connections behind the CF servers and block them using iptables string matching module. I will try to implement it but I will need for you to test it to see if it is working.

[09173732546]

As it seems cloudflare is distributing each of your request to a different cloudflare server and sending it to your VPS as you wrote before with the orange square of ip's screenshot. So basically we need a new ddos.conf option named CF_MODE=true|false (accompanied with implementation for it) that when true would also use tcpdump to retrieve the list of ip addresses (by reading the CF-Connecting-IP) that are making multiple connections behind the CF servers and block them using iptables string matching module. I will try to implement it but I will need for you to test it to see if it is working.

I will be on the look-out, if you need my vps i can lend you the login. I think its better that you can also see if i have misconfigs.

edit i checked /var/log/ddos.log , and i saw that CF ips are being banned, and when i stop the hulkdos, my VPS ip is being banned/unbanned

[2019-03-23 05:42:21] unbanned 162.158.111.242 that opened 19 connections
[2019-03-23 05:42:21] unbanned 141.101.105.144 that opened 19 connections
[2019-03-23 05:42:21] unbanned 162.158.154.224 that opened 18 connections
[2019-03-23 05:42:21] unbanned 141.101.107.241 that opened 18 connections
[2019-03-23 05:42:21] unbanned 162.158.89.14 that opened 17 connections
[2019-03-23 05:42:21] unbanned 141.101.104.216 that opened 17 connections
[2019-03-23 05:42:22] unbanned 172.69.55.107 that opened 16 connections
[2019-03-23 05:42:22] unbanned 162.158.89.158 that opened 16 connections
[2019-03-23 05:42:22] unbanned 162.158.155.237 that opened 15 connections
[2019-03-23 05:42:22] unbanned 141.101.98.233 that opened 15 connections
[2019-03-24 02:26:51] daemon stopped
[2019-03-24 02:26:54] daemon started
[2019-03-24 05:14:10] added cron job
[2019-03-24 05:15:20] daemon started
[2019-03-25 04:33:04] daemon stopped
[2019-03-25 04:33:09] daemon started
[2019-03-25 10:25:01] banned VPS_IP with 105 connections for ban period 600
[2019-03-25 10:35:20] unbanned VPS_IP that opened 105 connections
[2019-03-27 11:45:02] banned VPS_IP with 102 connections for ban period 600
[2019-03-27 11:55:51] unbanned VPS_IP that opened 102 connections
[2019-03-27 11:59:02] banned VPS_IP with 116 connections for ban period 600
[2019-03-27 12:09:28] unbanned VPS_IP that opened 116 connections
[2019-03-27 12:12:01] banned VPS_IP with 105 connections for ban period 600

i can still visit my site during and after hulkdos session.

[jgmdev]

Ok, I added the experimental cloudflare support! You will need to update ddos,sh with the most recent one on this repo and add ENABLE_CLOUDFLARE=true on your ddos.conf file, then you can restart ddos service and retest with hulkdos. By the way the -v flag now also includes cloudflare ip addresses found on the tcpdump file when ENABLE_CLOUDFLARE=true so you can test that before running the hulkdos test. The tcpdump file is rotated every DAEMON_FREQ*2 seconds, not sure how well that will work so test and let me know.

About these check #17 to see if that is your case

[2019-03-27 11:55:51] unbanned VPS_IP that opened 102 connections
[2019-03-27 11:59:02] banned VPS_IP with 116 connections for ban period 600
[2019-03-27 12:09:28] unbanned VPS_IP that opened 116 connections
[2019-03-27 12:12:01] banned VPS_IP with 105 connections for ban period 600

[09173732546]

Ok, I added the experimental cloudflare support! You will need to update ddos,sh with the most recent one on this repo and add ENABLE_CLOUDFLARE=true on your ddos.conf file, then you can restart ddos service and retest with hulkdos. By the way the -v flag now also includes cloudflare ip addresses found on the tcpdump file when ENABLE_CLOUDFLARE=true so you can test that before running the hulkdos test. The tcpdump file is rotated every DAEMON_FREQ*2 seconds, not sure how well that will work so test and let me know.

It worked! 👍 but with a note (merely cosmetic) the "normal" and "cloudflare" ip listing is displayed incorrectly.

ddos -v https://prnt.sc/n4kxzz
ddos -b https://prnt.sc/n4l8xc

Could a logic be developed to exclude the cloudflare ips from the ddos -v display list?

#! /bin/bash

CF_NETWORKS="173.245.48.0/20
103.21.244.0/22
103.22.200.0/22
103.31.4.0/22
141.101.64.0/18
108.162.192.0/18
190.93.240.0/20
188.114.96.0/20
197.234.240.0/22
198.41.128.0/17
162.158.0.0/15
104.16.0.0/12
172.64.0.0/13
131.0.72.0/22"

for IP in 141.101.107.229 37.106.149.169 141.101.98.227; do
        grepcidr "$CF_NETWORKS" <(echo "$IP") >/dev/null && \
                echo "" || \
                echo "$IP"
done

Taken from: https://unix.stackexchange.com/questions/274330/check-ip-is-in-range-of-whitelist-array

About these check #17 to see if that is your case

/var/log/ddos.log now shows the correct info, and my server ip was not included on the ban (the fix was from the updated ddos.sh)

I confirm sir that this work regardless of what is displayed on ddos -v and ddos -b now.

Thank you very much 💯

[jgmdev]

It worked! +1 but with a note (merely cosmetic) the "normal" and "cloudflare" ip listing is displayed incorrectly.

The normal listing displays ip addresses of connections directly made to the server, so if you are connecting to the vps using SSH your ip will also be listed on that list, that is why the cloudflare servers are listed because they are making direct connections to your server. On the other side the cloudflare ip list displays real user ip's of people indirectly connecting to the server using cloudflare reverse proxy servers, so the information displayed should be correct. (By the way I commit a change to hide the reading from file... message thrown by tcpdump when using the -v flag).

Could a logic be developed to exclude the cloudflare ips from the ddos -v display list?

You can try adding the cloudflare ip's (103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, etc...) on your /etc/ddos/ignore.ip.list file, and just in case you should add your vps ip to that list also (just in case so the server doesn't bans itself, since I'm not sure if it is retrieving its own ip properly).

Also out of curiosity how much time the server took to ban you when you launched the hulkdos test? Maybe I can remove the experimental message from the readme file :)

[09173732546]

Also out of curiosity how much time the server took to ban you when you launched the hulkdos test? Maybe I can remove the experimental message from the readme file :)

Took around 2~3seconds after the surge of Cloudflare IPs (not yet whitelisted)
[2019-03-30 00:26:30] banned 37.106.149.169 with 297 connections for ban period 600

The tcpdump file is rotated every DAEMON_FREQ*2 seconds, not sure how well that will work so test and let me know.

The default is DAEMON_FREQ=5, can it be lowered, or it will have performance impact?

note: i think i didnt see this before...
Deleting cron job...sed: can't read /ddos/d: No such file or directory
.. (done)

Thanks again

[jgmdev]

The default is DAEMON_FREQ=5, can it be lowered, or it will have performance impact?

that is multiplied by 2 to get 10 seconds total in order to give tcpdump enough time to collect data that can be analyzed, too little time I think would prevent enough data recollection to detect an attack. But yes, you can lower the daemon to 3 seconds interval so tcpdump will rotate every 6 seconds, you can try that and report back.

note: i think i didn't see this before...
Deleting cron job...sed: can't read /ddos/d: No such file or directory
.. (done)

That was the installer? I need more details, in any case I will not be working until Sunday or Saturday late night have a nice day!

[09173732546]

That was the installer? I need more details, in any case I will not be working until Sunday or Saturday late night have a nice day!

uninstall.sh .. im using centos7 and didnt remember seeing that on my first uninstall.

other than that, this issue is now solved, thank you very much

[jgmdev]

Since this seems to be working I will close the issue, by they way make sure to delete the pcap files you shared earlier to keep your server IP private.