Blog post published, with all the funky graphs:

https://people.kernel.org/monsieuricon/creepy-crawlies

@monsieuricon Interesting read, thanks for the write up!

@monsieuricon nice. Has anyone tried to communicate via Anubis pages that the data *is* downloadable via CDN clone bundles and whatnot?
At some points humans or agents are bound to chime in and if they see some trivial way to get the data that's easier than solving Anubis, they might eventually make a better choice.

@monsieuricon Good read, sad situation. Why are shallow clones "awful"? Do they put a lot of strain on the server side?

@monsieuricon we should create a new standard web framework for “I am human” attestation, with legislation making it a crime for a bot to intentionally circumvent it. Then we won’t need captchas at all, we’ll just rely on all the smart website owners to report violations to the authorities.

@forst shallow clones require us to gather about 1GB of random objects, compress them on the fly, then keep that pack in RAM as the client that requested the shallow clone downloads them. If someone does this occasionally, it's no big deal. However, as you can imagine, having 20 or so systems do this all at once (as is common for CI farms), this is super CPU and RAM heavy.

@monsieuricon TIL, thank you!

And is my understanding correct that the preferred approaches are, roughly in order of preference?:

1. Tarball of specific version, where possible
2. Pre-packed git bundle file
3. Full git clone (as that's using pre-existing packs or smth along those lines?)

Was also wondering about blobless and treeless clones, do those have just as bad effect on the server side?

@monsieuricon Wasn't there some file you could put somewhere to give AI bots some hints? I can't quite remember but maybe that can help? But I assume the bots are so dumb that they only blindly do HTML and try to slurp up the whole internet.

@monsieuricon what I’m experimenting with is placing trap links to iocaine on every page of my forge. It doesn’t keep them from reading the real pages, but it at least gives them something cheap to sometimes request instead.

@monsieuricon This is exactly what's bogging down the musl infrastructure too. Utter incompetence on the part of the AI clowns, scraping cgit rather than actual repos.

@monsieuricon I wonder how much of the features you are having to disable could be implemented on the client side? Sure that would suck in many ways, but at least it could possibly reduce server load

@monsieuricon

@monsieuricon but it's okay, it's just a tool

@monsieuricon Loved the funky graphs!

Sad about that even with #Anubis cranked up, there doesn't seem to be a proper solution but to hurt legitimate users' UX.

@monsieuricon i've been noticing more headless headers on my blogger stats, possibly using VPNs from small countries in the past few months, much more than before. https://github.com/apify/crawlee-python/issues/401 most page views are from Hacker News.

@monsieuricon

It occurs to me, perhaps this will eventually lead to cgit and the like to serve static data with a script to offload the pretty rendering to the client.

@monsieuricon All feels vaguely familiar (nothing like your scale, but some large/dynamic sites that are big enough to be hard to cache).

Part of me wonders if login/registration will become more common across various sites to counter this... but I'm sure it would be no problem for the scrapers to create accounts if registration is open.

Each year is getting trickier as the easy defensive solutions just aren't working any more.