Shopping Cart
There are no more items in your cart
What is meta-webindexer?
meta-webindexer is the crawler of Meta Platforms (the company that controls Facebook, Instagram, and WhatsApp) dedicated to collecting content from the web for training its artificial intelligence models. Unlike facebookexternalhit - which generates previews of links shared on social media - meta-webindexer does not provide any benefit to the site owner: it collects content without sending traffic in return.
The problem: a DDoS-like behavior
Analyzing the SSL access logs on Plesk servers, we detected an extremely aggressive behavior: the bot generates over 180 requests per minute per single domain, operating in rotation on dozens of different IP addresses to evade traditional blocking systems.
Reports worldwide
The problem is far from isolated. According to a Fastly report on AI traffic in 2025, Meta's crawlers represent 52% of all global AI crawler traffic, more than double compared to Google (23%) and OpenAI (20%) combined.
The ratio between crawling and actual traffic sent is about 73,000:1: Meta extracts content from sites at an extraordinary rate without returning practically anything in terms of visits.
On Hacker News and specialized forums, hundreds of webmasters have reported the same behavior: the bot does not respect robots.txt directives, does not apply any progressive slowdown, and in some documented cases, has generated 11 million requests in 30 days on a single site.
How to detect it in your logs
If your server uses Plesk with Apache/SSL, you can check how many requests it has received with this command:
grep -i "meta-webindexer" /var/www/vhosts/yourdomain.com/logs/access_ssl_log | wc -lTo see the distribution hour by hour:
grep -i "meta-webindexer" /var/www/vhosts/yourdomain.com/logs/access_ssl_log \
| grep "13/Apr/2026" \
| awk -F: '{print $2":00"}' | sort | uniq -cTo see how many requests each domain on the server has received:
for log in /var/www/vhosts/*/logs/access_ssl_log; do
count=$(grep -i "meta-webindexer" "$log" | wc -l)
[ $count -gt 0 ] && echo "$count $log"
done | sort -rnHow to block it with Fail2ban
An initial solution adopted was to use Fail2ban, which blocks IPs directly via iptables before the request even reaches the web server. However, as we will see later, this approach presents significant risks that need to be understood before proceeding.
1. Create the filter
Create the file /etc/fail2ban/filter.d/bad-bots.conf with this content:
[Definition]
failregex = ^<HOST> .* ".*" .* ".*(?:meta-webindexer|meta-externalagent|AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|PetalBot|Bytespider|GPTBot|ClaudeBot|anthropic-ai|CCBot|DataForSeoBot|serpstatbot|Scrapy|python-requests|python-urllib|zgrab|masscan|nikto|sqlmap|YandexBot|TTD-Content|Diffbot|magpie-crawler|VelenPublicWebCrawler|Netcraft|WebReaper|WebCopier|HTTrack|AwarioRssBot|AwarioSmartBot).*"$
ignoreregex =2. Configure the jail
Add in /etc/fail2ban/jail.local:
[bad-bots]
enabled = true
filter = bad-bots
logpath = /var/www/vhosts/*/logs/access_ssl_log
maxretry = 3
findtime = 30
bantime = 604800
action = iptables-allports[name=bad-bots, protocol=all]3. Test and activate
# Test the filter
fail2ban-regex /var/www/vhosts/yourdomain.com/logs/access_ssl_log \
/etc/fail2ban/filter.d/bad-bots.conf
# Restart fail2ban
systemctl restart fail2ban
# Check banned IPs
fail2ban-client status bad-bots⚠️ Warning: the risk of false positives with Fail2ban
In practice, we encountered a critical issue: Fail2ban bans IPs at the network level, blocking all connections from that address, regardless of the service or protocol.
The concrete risk is that Meta and other CDN providers share the same IP address ranges. In our case, the range 185.93.2.x is used both by Meta for aggressive crawling and by BunnyCDN for its API (api.bunny.net resolves to 185.93.2.251).
The result: by banning the IPs of the meta-webindexer bot, we accidentally blocked BunnyCDN's API calls as well, causing severe slowdowns in the PrestaShop dashboard for all sites using the CDN on that server. The PHP timeout waited up to 60 seconds for each page load before failing.
The lesson is important: Fail2ban is suitable for blocking brute force attacks on SSH or the control panel, it is not the right tool for blocking HTTP bots identified by user-agent, precisely because of this risk of false positives.
Warning: do not block facebookexternalhit
It is important to distinguish aggressive AI crawlers from legitimate ones. facebookexternalhit is the bot Facebook uses to generate previews of links shared on Facebook, Instagram, and WhatsApp. Blocking it would make previews disappear on social media for all your domains - an unnecessary damage.
Make sure to include only meta-webindexer and meta-externalagent in the filter, which are the crawlers dedicated exclusively to AI training.
Recommended solutions: more precise and risk-free
There are more selective approaches that block bots based on user-agent without touching IPs at the network level, eliminating the risk of blocking legitimate services like BunnyCDN, Cloudflare, or other CDNs that share IP ranges with the bots.
Solution 1 - PrestaShop Security & Bot Shield Module
For those using PrestaShop, the most comprehensive solution is the module PrestaShop Security & Bot Shield which offers an advanced protection system specifically designed for e-commerce.
Main features include:
- Blocking malicious bots and AI scrapers via user-agent - without banning IPs
- Integration with AbuseIPDB to block known dangerous IPs
- Management of trusted IP whitelist to avoid false positives
- Blocking suspicious requests towards WordPress by bots
- Protection via .htaccess or PrestaShop hook
- reverseDNS check to never block official Googlebot and Bingbot
- Logging and recording security events with blocked attempts
- Compatible with PrestaShop 1.7, 8.x, and 9.x
Unlike Fail2ban, the module operates at the application level: the bot is blocked with a 403 error before the request is processed, without ever touching the network connections of IPs shared with legitimate services.
Solution 2 - Cloudflare WAF Rules
For those using Cloudflare as a proxy, it is possible to create WAF (Web Application Firewall) rules directly from the dashboard, without touching the server. Simply go to Security → WAF → Custom Rules and create a rule like:
Field: User Agent
Operator: contains
Value: meta-webindexer
Action: BlockThe advantage of Cloudflare is that the block occurs before the request even reaches the server, with zero impact on resources and no risk of false positives on IPs, because the filter is applied exclusively on the HTTP user-agent. It is possible to add more bots in the same rule using the OR operator.
Solution 3 - Apache Rules via .htaccess
For those managing the server directly, it is possible to block bots at the Apache level by adding these directives in the .htaccess file or in the virtual host configuration:
<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent "meta-webindexer|meta-externalagent|AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|PetalBot|Bytespider|GPTBot|ClaudeBot|anthropic-ai|CCBot|DataForSeoBot|serpstatbot|Scrapy|python-requests|python-urllib|zgrab|masscan|nikto|sqlmap" bad_bot
<RequireAll>
Require all granted
Require not env bad_bot
</RequireAll>
</IfModule>Again, the block is based exclusively on the user-agent, with no impact on IPs shared with CDNs or other services.
Conclusion
The behavior of meta-webindexer represents a real and growing problem for all web server managers. With traffic that can exceed 100,000 requests per day per single domain, ignoring it means wasting bandwidth, CPU, and memory resources to the exclusive benefit of Meta.
However, the choice of blocking tool is crucial: Fail2ban bans IPs at the network level and can accidentally affect legitimate services that share the same IP ranges as the bots. The recommended solutions - the PrestaShop Security & Bot Shield module, Cloudflare WAF rules, or Apache directives - instead act on the HTTP user-agent, ensuring precise protection without the risk of false positives.