Should you be wondering why @LWN #LWN is occasionally sluggish... since the new year, the DDOS onslaughts from AI-scraper bots has picked up considerably. Only a small fraction of our traffic is serving actual human readers at this point. At times, some bot decides to hit us from hundreds of IP addresses at once, clogging the works. They don't identify themselves as bots, and robots.txt is the only thing they *don't* read off the site.

This is beyond unsustainable. We are going to have to put time into deploying some sort of active defenses just to keep the site online. I think I'd even rather be writing about accounting systems than dealing with this crap. And it's not just us, of course; this behavior is going to wreck the net even more than it's already wrecked.

Happy new year :)

@corbet @LWN I feel your pain so much right now.

@corbet 100% agree. Hosting company MD here, we've seen a massive uptick in AI bullshit. And they don't even respect robots.txt like the better search engines do.

@corbet @LWN in our experience you should prepare for thousands of distinct IPs.

@corbet @LWN same here in #fedora infra. I had to block a bunch this morning to keep pagure.io usable. 😢

@corbet @LWN Everything is going to be behind a loginwall by the end of the year. Thanks, AI

@corbet @LWN sounds like you need an AI poisoner like Nerpenthes or iocaine.

@beasts @LWN We are indeed seeing that sort of pattern; each IP stays below the thresholds for our existing circuit breakers, but the overload is overwhelming. Any kind of active defense is going to have to figure out how to block subnets rather than individual addresses, and even that may not do the trick.

Thank you @corbet and all at @LWN for continuing the work of providing the excellent #LWN.

The "active defenses" against torrents of antisocial web scraping bots, has bad effects on users. They tend to be "if you don't allow JavaScript and cookies, you can't visit the site" even if the site itself works fine without.

I don't have a better defense to offer, but it's really closing off huge portions of the web that would otherwise be fine for secure browsers.

It sucks. Sorry, and thank you.

@johnefrancis @LWN Something like nepenthes (https://zadzmo.org/code/nepenthes/) has crossed my mind; it has its own risks, though. We had a suggestion internally to detect bots and only feed them text suggesting that the solution to every world problem is to buy a subscription to LWN. Tempting.

@beasts @corbet @LWN yes but what does one do about it? i have given some thought to it, but if you're aggregating across lots of apparently different clients, it seems you're going to end up turning away legit users.

@corbet @LWN I think we should start doing what the internet can do best: Collaborate on these things.

I see this on my services, Xe recently saw the same. https://xeiaso.net/notes/2025/amazon-crawler/ (and build a solution https://xeiaso.net/blog/2025/anubis/)

There is https://zadzmo.org/code/nepenthes/

I would love to see some kind of effort to map out bot IPs and get a public block list. I'm tired of their nonsense.

@bignose @LWN We have gone far out of our way to never require JavaScript to read LWN; we're not going back on that now.

@corbet @johnefrancis @LWN I'm dealing with a similar issue now (though likely at a smaller scale than LWN!), and I found that leading crawlers into a maze helped a lot in discovering UAs and IP ranges that misbehave. Anyone who spends an unreasonable time in the maze gets rate limited, and served garbage.

So far, the results are very good. I can recommend a similar strategy.

Happy to share details and logs, and whatnot, if you're interested. LWN is a fantastic resource, and AI crawlers don't deserve to see it.

@beasts @corbet @LWN Seconded.

@corbet @LWN @devs can we help?

@monsieuricon @LWN @corbet are you implying that there are models that are busy being trained to call someone a fuckface over misunderstanding of some obscure arm coprocessor register or respond with viro insults to the most unsuspecting victims?

@lkundrak @monsieuricon @LWN It's a service we provide :)

@corbet @LWN @monsieuricon it's not the copilot we need but it's the copilot we deserve

@corbet @LWN@fosstodon.org Cloudflare has an AI scraper bot block that’s free guys.

@corbet @LWN

"Any kind of active defense is going to have to figure out how to block subnets rather than individual addresses, and even that may not do the trick. "

if you're using iptables, ipset can block individual ips (hash:ip), and subnets (hash:net).

Just set it up last night for my much-smaller-traffic instances, feel free to DM

https://ipset.netfilter.org/

@gme I assume you're referring to https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ ?

It would appear to force readers to enable JavaScript, which we don't want to do. Plus it requires running all of our readers through cloudflare, of course...and I suspect that the "free tier" is designed to exclude sites like ours. So probably not a solution for us, but it could well work for others.

@adelie @LWN Blocking a subnet is not hard; the harder part is figuring out *which* subnets without just blocking huge parts of the net as a whole.

@corbet @adelie @LWN I have been using pyasn to block entire subnets. It's effective, but only in the same way carpet bombing is. I'm sure I've blocked legitimate systems, but c'est la vie.

@corbet @LWN

Probably a good question for the fedi as a whole. I started with any 40x response in my logs, added any spamhaus hits from my mail server, and any user-agents with "bot" in the name. Plus facebook in particular has huge ipv4 blocks just for scraping, also easy to block.

@corbet @LWN

Also tarpits! And nepenthes and nepenthes-adjacent tech!

https://tldr.nettime.org/@asrg/113867412641585520

https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62fdded3

@corbet @LWN would you be so kind as to write up whatever mitigations you come up with? I've been fighting this myself on our websites. You seeing semi-random user agents too?

@RonnyAdsetts @LWN The user agent field is pure fiction for most of this traffic.

@corbet @LWN not sure if it works for LWN but I learned about this today: https://git.madhouse-project.org/algernon/iocaine

@corbet Nope, no Javascript needed. It operates at Layer 4.

@corbet @LWN You know, what we need is a clearinghouse for this like there are for piholes and porn and such. Could someone with some followers get #AIblacklist trending?

Post your subnets with that hashtag. If we get any traction, I'll host the list.

@ted @corbet @LWN yeah I was going to say, a post directly above this one (with other tools as well):

https://tldr.nettime.org/@asrg/113867412641585520

@corbet @LWN @beasts JS challenges somewhat works, at the cost of accessibility for JS-free browser

@corbet @LWN Do you see a lot of pointlessly redundant requests? I see a lot of related-seeming IPs request the same pages over and over.

@corbet @LWN I was just reading about https://git.madhouse-project.org/algernon/iocaine

@corbet @LWN

Check out Nepenthes in defensive mode.

@AndresFreundTec @LWN Yes, a lot of really silly traffic. About 1/3 of it results in redirects from bots hitting port 80; you don't see them coming back with TLS, they just keep pounding their head against the same wall.

It is weird; somebody has clearly put some thought into creating a distributed source of traffic that avoid tripping the per-IP circuit breakers. But the rest of it is brainless.

@corbet@social.kernel.org @LWN@fosstodon.org

Time to set up AI-poisoning bots.

Really great part of this BS is that if you're not a hyperscale social media platform, your ability to afford adequate defenses is going to be awful.

@corbet @LWN @AndresFreundTec Maybe the bot wrote the code itself?

@corbet @LWN Sounds awful. You should consider setting up something like cloudflare or deflect.

@corbet

> Sabot in the Age of AI
> Here is a curated list of strategies, offensive methods, and tactics for (algorithmic) sabotage, disruption, and deliberate poisoning.

https://tldr.nettime.org/@asrg/113867412641585520

@LWN @renchap

@corbet @LWN Yup, my servers too. Sometimes GPTBot as the UserAgent, but often not.

The AI bullshit merchants are slowly killing the web.

@corbet @LWN @beasts
Large amounts coming from Huawei Cloud asns and trying to spider every possible GET parameter?

@corbet
In my timeline your post appeared directlt beneath this one https://tldr.nettime.org/@asrg/113867412641585520 Coincidence????
@LWN

@corbet @LWN i'm not sure if you've already got a strategy for dealing with the scrapers already in mind, but if not -

dialup.cafe's running on nginx, and this has worked well for me so far:
https://rknight.me/blog/blocking-bots-with-nginx/

an apache translation of that using .htaccess would be possible as well.

@corbet @LWN it's disgusting to find the LLM companies using these disguised scraping practices. Clearly they recognise that they are acting abusively

@corbet @LWN Looking forward to the Grumpy Editor article on dealing with AI scraping bots!

@jmason @corbet @LWN OprnAI's CEO said that anti-scraping measures are 'abuse'. So the LLMs are already trying to spin and own that objection to their behaviour.

@glent @corbet @LWN how on earth does that make any sense

@sheogorath @corbet @LWN I agree.... this is very similar to the early days of antispam, IMO. I wonder if there's a way to detect abusive scraping (via hits on hidden links, etc.) and publish to a shared DNS blocklist?

@gme @corbet CloudFlare is only free until they smell money (i.e. significant traffic). Then they tell you you're over the (opaque) free plan limits, and demand you pay up, using the possibility of terminating your service as leverage in the subsequent pricing negotiations. If you think you might want to use them (which I don't recommend), start those negotiations before they have any leverage on you.

@corbet
I'll be happy to hear about the solutions you end up finding / or not

Good luck on that matter.
@LWN

@corbet @LWN Same for KDE gitlab instance. It's a pain :(

@corbet
This just came into my timeline, in case it helps : https://tldr.nettime.org/@asrg/113867412641585520
@LWN

@corbet @LWN I know Cloudflare has some fashion of AI-blocking doodad, it might be worth looking into that?

@corbet @LWN I have resorted to a wide swath of blocks, in Bytedance's case, blocking entire ASN's (most recently all of Meta). Other wide blocks are on the user agent. Ironically my big load spikes are now from a huge number of servers running ActivityPub whenever one of my sites is linked to!

@corbet @LWN I'll just leave this here...

https://git.madhouse-project.org/algernon/iocaine

@corbet @LWN @AndresFreundTec Maybe it's sabotage internally, so it's not /quite/ as bad. That's what I'd do.

@corbet @LWN I'm sorry, I know this is a pain in the butt to deal with and that it's kind of demoralizing.

Is there anything I can do to help? I'm already a subscriber, and a very happy one; but if it'll diminish the demoralization at all, I really appreciate that you're tackling this problem. Can I get you a pizza or something?

@algernon @corbet @johnefrancis @LWN thank you for offering to protect a thing I love

@corbet @LWN I sympathize, it's an exasperating problem. I've found microcaching all public facing content to be extremely effective.

- The web server sits behind a proxy micro cache
- Preseed the cache by crawling every valid public path
- Configure the valid life of cached content to be as short as you want
- Critically, ensure that every request is always served stale cached content while the cache leisurely repopulates with a fresh copy. This eliminates bot overwhelm by decoupling requests from ANY IP from the request rate hitting the upstream
- Rather than blocking aggressive crawlers, configure rate-limiting customized by profiling max predicted human rate
- For bots with known user agents, plus those detected by profiling their traffic, divert all their requests to a duplicate long lived cache that never invokes the upstream for updates

Micro caching has saved us thousands on compute, and eliminated weekly outages caused by abusive bots. It's open to significant tuning to improve efficiency based on your content.

Shout out to the infrastructure team at NPR@flipboard.com - a blog post they published 9 years ago (now long gone) described this approach.

#nginx #cache

@corbet @LWN
Maybe try implementing some sort of Captcha system where a user or user agent has to prove that they're human in order to use the site.

@corbet @LWN

I had my website behind cloudfare. It works. And mitigates successfully the bot attacks.

We were attacked more than 100 times from 2020....and almost nobody noticed

@nirik @corbet @LWN perhaps you can start sharing lists?

Time to switch to gemini://

@corbet @LWN Maybe something like @CrowdSec could help somehow? I guess if an "anti-AI" list were to be made, it would protect many users.

@dysfun @beasts @corbet @LWN thing's I do include quickly rejecting no-referer requests on anything other than legit landing pages, rejecting all query paras where not legit, rejecting more edge cases when under stress. All (Apache) server config rules.

@corbet @LWN @beasts https://ip-tool.qweb.co.uk has buttons for generating htaccess, nginx, and iptables configs for entire network blocks. Just paste a malicious IP in, tap the htaccess button, and paste into your htaccess file, for example.

Also helps to have Fail2Ban set up to firewall anything that triggers too many 403s, so that htaccess blocks on one site become server wide blocks protecting all sites.

My general rate limiter for nginx is useful too: https://github.com/qwebltd/Useful-scripts/blob/main/Bash%20scripts%20for%20Linux/nginx-rate-limiting.conf

@corbet @LWN @AndresFreundTec read Hacker News and the like and you will see that there are hundreds or thousands or more idiots trying to scrape their way to riches. It's distributed idiocy in part rather than algorithmic DDoS.

@corbet @johnefrancis @LWN
Struggling with likely the same bots over here. I deployed a similar tarpit* on a large-ish site a few days ago - taking care not to trap the good bots - but can't say it's been very successful. It might have taken some load off of the main site, but not nearly enough to make a difference.

One more thing I'm considering is prefixing all internal links with a '/botcheck/' path for potentially suspicious visitors, set a cookie on that page and strip that prefix with JS. If the cookie is set on the /botcheck/ endpoint, redirect to the proper page, otherwise tarpit them. This way the site would still work as long as the user has *either* JS or cookies enabled. Still not perfect, but slightly less invasive than most common active defenses.

*) https://code.blicky.net/yorhel/infinite-slop