incident and response - malicious bots

One of our servers, which hosted a dot net backend as well as some wordpress sites, showed some outages two days in a row, restored by restarting. Feeding the apache access logs to Gemini via aistudio.google.com, Gemini said,

The logs clearly show an Apache HTTP

Server running PHP and hosting a WordPress application.

The logs indicate that your server was hit by an aggressive automated attack

(likely a botnet) starting around 07:33:35, which appears to have either

succeeded in exploiting a vulnerability or overloaded the server, causing it to

crash shortly after.

(Server had run out of memory). 

Gemini suggested the following - 

1. Hide and Protect Your Origin IP - which we may not do
2. Patch and Update the Server Software - which we're doing
3. Implement a Web Application Firewall (WAF) - which is in place
4. Clean Up WordPress Configuration - "Open your wp-config.php file, locate line 63, and remove the    duplicate definition of WP_AUTO_UPDATE_CORE. While this didn't cause the crash, it eats up server I/O and makes reading logs difficult."
5. Restrict PHP Execution in Uploads Directories
6. Implement Intrusion Prevention (Fail2Ban)

Then, feeding the access logs of the dot net api to Gemini 

and as suggested by Gemini,

journalctl -u ourapi.service --since "2026-06-28 07:15:00" --until "2026-06-28 07:45:00"

Gemini gave some recommendations like returning 0 instead of a 500 error for no data found -

"As we saw in the access logs, when the mobile app sees

    this 500 error, its poorly designed error-handling logic says: "Something

    went wrong! Retry the entire sync process!" It then proceeds to download 4MB

    of lesson data, hits the notification endpoint again, gets another 500

    error, and repeats the cycle every 15 seconds until your server runs out of

    memory and crashes."

- this may or may not be actually what is happening here, since Gemini just saw the logs and does not have access to the code.

In any case, I've enabled bot protection on Cloudflare for this domain, and also installed and configured Fail2ban.

On Cloudflare, 

ourdomain.org > Security > Security rules > Custom rules

Block .env scans

URI Path equals .env

Block

 also added Cloudflare's default rate limiting rule,

Leaked credential check [Template]

Password Leaked equals true

Block

also enabled Cloudflare's AI blocking tools - AI Labyrinth enabled, and Block AI training bots on all pages.

Then, installed and set up Fail2ban on the server, monitoring the apache web server and sshd logs. It will temporarily block ip addresses which fail authentication repeatedly, or which repeatedly request non-existent files like malicious bots.

Gemini via aistudio.google.com gave step-by-step instructions for installation and setup of Fail2ban. 

sudo apt install fail2ban -y
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
(we can optionally edit these,

bantime  = 1h

findtime = 10m

maxretry = 5

)

Enable the Apache jails with

sudo nano /etc/fail2ban/jail.local

[apache-auth]

enabled  = true

port     = http,https

logpath  = %(apache_error_log)s

In case the logpath is custom - fail2ban looks by default for

Error logs: /var/log/apache2/*error.log

Access logs: /var/log/apache2/*access.log

In the case of one of our servers, we needed to change this to

logpath  = %(apache_error_log)s

           /var/www/mysite/logs/custom-error*.log

More Apache jails to enable - basically need to add the enabled = true line - 

[apache-badbots]

enabled  = true

port     = http,https

logpath  = %(apache_access_log)s

bantime  = 48h

maxretry = 1

[apache-noscript]

enabled  = true

port     = http,https

logpath  = %(apache_error_log)s

[apache-overflows]

enabled  = true

port     = http,https

logpath  = %(apache_error_log)s

[apache-nohome]

enabled  = true

port     = http,https

logpath  = %(apache_error_log)s

[apache-botsearch]

enabled  = true

port     = http,https

logpath  = %(apache_error_log)s

Then we can test

sudo systemctl start fail2ban

sudo systemctl enable fail2ban
sudo systemctl status fail2ban

and check the enabled jails with

sudo fail2ban-client status

On one server, we had to resolve

ERROR   Failed during configuration: Have not found any log file

for sshd jail

For that, we had to change the block to

[sshd]

enabled = true

port    = ssh

backend = systemd

since that server was using systemd, and traditional text log files like /var/log/auth.log (which Fail2Ban looks for by default for SSH) are no longer created. After the change, after restarting the service, we can check that jail alone with

sudo fail2ban-client status sshd

Then, for using a different port rather than port 22 for sshd - 

[sshd]

enabled = true

port    = 2022

backend = systemd

(if port 2022 is being used) 

and so on, because that is the port which fail2ban would ban using the machine's firewall, iptables or nftables as the case may be.

To see the rules, we can use

sudo nft list ruleset # for nftables, Ubuntu 24.04

sudo iptables -nL # for iptables, earlier Ubuntu etc