In the last few years, website scanning became a normal occurrence. Websites / IPs by some reports are being scanned within minutes of existing on the web.
This means that your site is being looked at by 3rd party to see if you have any vulnerabilities or to scrape your site of it’s content.
To stop these you can implement ASN blocking. Another way of doing this is to block User Agent.
Here are some of the user agents I have blocked through Cloudflare that help me block unwanted scans of servers I manage.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | python github pastebin OgScrper lua-resty-http masscan ZmEu curl Wget Scrapy BrandVerity weborama-fetcher libfetch Go-http-client Corax Java LinuxGetURL kubernetes Faraday nmap special_archiver ruby research Certificate PycURL Wordpress MJ12bot adbeat ltx71 Nimbostratus |
Here is a list for .htaccess if you need it. I have not tested it though.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | # BLOCK USER AGENTS RewriteEngine on RewriteCond %{HTTP_USER_AGENT} python [NC,OR] RewriteCond %{HTTP_USER_AGENT} github [NC,OR] RewriteCond %{HTTP_USER_AGENT} pastebin [NC,OR] RewriteCond %{HTTP_USER_AGENT} OgScrper [NC,OR] RewriteCond %{HTTP_USER_AGENT} lua\-resty\-http [NC,OR] RewriteCond %{HTTP_USER_AGENT} masscan [NC,OR] RewriteCond %{HTTP_USER_AGENT} ZmEu [NC,OR] RewriteCond %{HTTP_USER_AGENT} curl [NC,OR] RewriteCond %{HTTP_USER_AGENT} Wget [NC,OR] RewriteCond %{HTTP_USER_AGENT} Scrapy [NC,OR] RewriteCond %{HTTP_USER_AGENT} BrandVerity [NC,OR] RewriteCond %{HTTP_USER_AGENT} weborama\-fetcher [NC,OR] RewriteCond %{HTTP_USER_AGENT} libfetch [NC,OR] RewriteCond %{HTTP_USER_AGENT} Go\-http\-client [NC,OR] RewriteCond %{HTTP_USER_AGENT} Corax [NC,OR] RewriteCond %{HTTP_USER_AGENT} Java [NC,OR] RewriteCond %{HTTP_USER_AGENT} LinuxGetURL [NC,OR] RewriteCond %{HTTP_USER_AGENT} kubernetes [NC,OR] RewriteCond %{HTTP_USER_AGENT} Faraday [NC,OR] RewriteCond %{HTTP_USER_AGENT} nmap [NC,OR] RewriteCond %{HTTP_USER_AGENT} special_archiver [NC,OR] RewriteCond %{HTTP_USER_AGENT} ruby [NC,OR] RewriteCond %{HTTP_USER_AGENT} research [NC,OR] RewriteCond %{HTTP_USER_AGENT} Certificate [NC,OR] RewriteCond %{HTTP_USER_AGENT} PycURL [NC,OR] RewriteCond %{HTTP_USER_AGENT} Wordpress [NC,OR] RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC,OR] RewriteCond %{HTTP_USER_AGENT} adbeat [NC,OR] RewriteCond %{HTTP_USER_AGENT} ltx71 [NC,OR] RewriteCond %{HTTP_USER_AGENT} Nimbostratus [NC] RewriteRule !^robots\.txt$ - [F] # BLOCK BLANK USER AGENTS RewriteCond %{HTTP_USER_AGENT} ^-?$ RewriteRule ^ - [F] |
The list was generated with: https://freetools.webmasterworld.com/tools/htaccess-block-user-agent