The @lwn web site is currently under the most intense scraper attack I have seen yet. 1.3M unique IP addresses within the last couple of hours, and it's not done yet. The work we have done on defenses appears to be paying off, though; the server is holding up reasonably well — so far.

...just in case anybody wonders why I have a rather dim view of the whole AI industry...

@corbet @lwn

> ...just in case anybody wonders why I have a rather dim view of the whole AI industry...

Me too. Because (and this is just another instance of it), the greed of the few destroy life as it was for the many.

@corbet @lwn if you need more capacity, give a shout, I've a bit i can throw to the cause if you need it

@corbet @lwn same on @blenderartists right now with up to 3.7M requests/hour. Looks more like a DDoS in our case than AI scraping though (which I suspected at first)

@corbet @lwn I hate sites with only a paywall. I like open or your “open later” model far better.

But I’m surprised more sites haven’t gone paywall only at this point. How is anyone supposed to survive this?

(Longtime subscriber 👍)

@corbet @lwn
What's the user agent if the scrapers?
Are you using any captcha or cloud armor?

@corbet @lwn ouch - fingers crossed for you all!

I wonder if this is related to weather.gov going down earlier (and forecast.weather.gov still being down), or just coincidence?

@corbet thank you and the @lwn team for your service. ✌️💙

@corbet @lwn A few months ago one of my clients' sites was getting 5000 requests per second, about 10-20 requests from a single IP, all residential. Server held up quite well until /var filled up because the logs didn't rotate fast enough (daily rotation, /var is 16 GB and normally has around 13 GB free).

@StompyRobot on my site it’s 90% residential proxies that camouflage as some Chrome. Not easy to block. Some pass simple automated challenges. @corbet @lwn

@corbet @lwn Website operators should be especially vocal about this, because too many people who have a positive view of AI companies have no idea..

@corbet @lwn Inkscape also got heated today:

https://floss.social/@doctormo/116825974597784492

@StompyRobot @lwn User agent is whatever random fiction they choose to put in there; there is no useful signal there. We really don't want to inflict captchas or cloudflare or any of that onto our readers, so we've had to find other ways to defend the site.

@corbet@social.kernel.org @lwn@lwn.net Amazing. I'd be interested in a post describing what you folks did, lessons learned, etc. (Assuming one doesn't exist already!)

@BartV @corbet @lwn @blenderartists
Who would have a motive to DDoS LWN or Blender? Other than Microsoft and Adobe, of course.

Most likely those IPs are from residential proxies so you can't do an easy filtering rule like "Block all IPs in AWS/GCP/Azure address spaces". There were revelations last week than half of all Smart TV apps include residential proxy SDKs.

@corbet @lwn I am in the process of rolling out the next major version of my WAF and I've connected to the Abuse IP DB, which I now use to short circuit all the rest of the tests if the score is >=75. It's killing about 95% of the incoming traffic, and the WAF is getting about 95% of the rest (largely through ASN-wide blocks; host an AI scraper and you're dead to me unless you're whitelisted.)

@corbet @lwn
I had these few months back on my sites. Had to send them to hell by blocking out.

@BartV @corbet @lwn @blenderartists What's the difference?

Anyone scrapping to (re)train LLMs is a selfish capitalist who doesn't care who they inconvenience &/or hurt.

We've enough LLMs for the foreseeable future. None are as Free and Open as we'd like, but I'm sure it's not someone trying to build a truly #FOSS LLM that's DDoS'ing #LWN rn.

All new #LLM training should stop *immediately*; continuing now on training is unconscionable. If you work for a company that is still training, I urge you to resign in protest.

Cc: @lwn @Andres4NY
@corbet

#AI

@fazalmajid

At SFC, we've been seeing the primary culprit is .cn IP numbers and Zuckerberg.

& I can confirm User-Agent is fiction, at least from those parties. robots.txt of course ignored.

Cc: @BartV @corbet @lwn @blenderartists

Agreed in the main, @bkuhn.

I imagine an obvious response is “we have to keep putting *new* data into the #LLM so that it stays up to date”. As far as it goes, yes that's true.

But why is that so urgent? Not enough to justify the hammering websites, the bulldozing of consent, the active deception to pass blocks, the refusal to countenance anything except #Hyperscaler interests. Stop it all, now.

@bignose Even more than whether it is urgent, & even whether or not you're pro or against *using* LLM-gen-AI, the world is still figuring out if these monstrosities they are useful *for* (if anything).

The ballyhoo is clearly wrong, but I also think those who say they are not useful for anything are also wrong.

We (humanity) need at least two years to even begin to understand what we have & what it's for. Let's pause and figure that out without capitalists in the driver's seat.

@corbet @lwn From my work experience I can say that the only remediation at that scale is #Blackholing traffic at #IX-level from all malicious ASNs used for said #DDoS and sending angry #AbuseReport mails every originator and their Upstreams.

- Make it THEIR PROBLEM!

Also let us know of the IP ranges so everyone else can block them as well!