We had an issue with a customer who is running magento 2. Bots were scanning for their admin panel hash. In magento the admin panel is not a set string like /admin, instead it is customized by the merchant and hopefully something unique.
So these bots just send request like the below:
This creates a ton of 404 traffic, which is problematic. That’s because 404 traffic typically does not use any caching mechanisms, meaning it by passes all cache that is in place. The veracity of these bots was bolden and they were creating high load.
We took to blocking them using probably not an ideal approach, but I will detail it anyway because there was very little documentation on how to do this when the server is using just straight apache. I saw some solution geared toward cloudflare rules and even an nginx rule, but nothing suggested for straight apache.
As I said, this solution is probably not ideal, but I will document it anyway to give you some ideas of reacting and you can either use the solution or come up with a better one (and please share with me).
We already had fail2ban installed on the server. So what we did was add a custom fail2ban jail for monitoring the apache logs and jailing anyone who hit a threshold of 404s in a certain amount of time.
While you might think no legit use should fire off 50 404 errors in 5 minutes, its possible that things like a feed gone bad, or an seo crawler following some links or an old sitemap could easily, in theory, do this.
Create a file /etc/fail2ban/filter.d/apache-404.conf
[INCLUDES]
before = apache-404.conf
[Definition]
failregex = ^ - .* "(GET|POST|HEAD).HTTP." 404 .$ ignoreregex =.(robots.txt|favicon.ico|jpg|png|js)
Inside /etc/fail2ban/jail.conf:
[apache-404]
enabled = true
port = http,https
filter = apache-404
logpath = %(apache_access_log)s
bantime = 3600
findtime = 600
maxretry = 25
Restart fail2ban:
service fail2ban restart
Check on the new jail:
fail2ban-client status apache-404
You can then test it by wget – ting some fake urls a bunch and seeing your own ip become blocked. (Make sure to have ssh from another ip open to unblock.
To unlock your ip after being blocked (change 1.1.1.1 with your ip):
fail2ban-client set apache-404 unbanip 1.1.1.1
To whitelist your ip or a clients ip from fail2ban forever:
vi /etc/fail2ban/jail.conf
Find the line that starts with ignoreip
Now add all IP you want. Each IP or range IP must be separated here with a whitespace. Ex: 192.168.0.1 192.168.5.0/32, then restart fail2ban.
ignoreip = 192.168.0.1 192.168.5.0/32
service fail2ban restart