I'm sad to say that we're following the lead of many others and putting in proof-of-work proxies into place to protect ourselves against "AI" crawler bots. Yes, I hate this as much as you, but all other options are currently worse (such as locking us into specific vendors).

We'll be rolling it out on lore.kernel.org and git.kernel.org in the next week or so.

You can see it in action on this recently decommissioned system I'm using for testing purposes: https://ams.source.kernel.org/

Difficulty is set at 4 leading zeroes, unless you're coming from US in which case there's also a tariff of 5 more leading zeroes.

@monsieuricon Don't set the difficulty above 5 or it becomes very mathematically unlikely for people to pass the challenge!

@cadey lol,.I wasn't being serious. :)

Or was I?

@monsieuricon I mean, that's funny enough that I'm willing to implement geoip lookups to make it happen :)

@monsieuricon will the JS requirement (or simply the proxy itself) for lore.kernel.org have an impact on `b4` (e.g. via `b4 shazam` or `b4 am`)?

@0leil no, I believe right now Anubis only gets triggered on Mozilla user agents.

@monsieuricon mhm. Is lynx and w3m supposed to go through without any mentions of the challenge?

@KasTasMykolas try it, let me know if you have problems.

@monsieuricon that's the point, tried it expecting some issues due to JS. But no problems whatsoever :)

@monsieuricon You know what, that's funny enough that I'm adding geoIP to the issue tracker: https://github.com/TecharoHQ/anubis/issues/206

@monsieuricon ^ @Klaus_Mueller @BNetzA @bsi yet another day of #OpenSource, here the largest one of them all, the #Linux kernel itself, fighting with #AI crawler traffic. If only there were some organization able to regulate this...

@monsieuricon wtf why does the progress bar go backwards?

@monsieuricon hmm it works fine on my PC, but on my phone it goes forward a bit at 1.2kH/s, stops, goes back to zero as the hashrate drops to 0.012kH/s, then fluctuates a bit, times out, and refreshes :(

@monsieuricon Just out of curiosity, why choose this over the honey-pot traps?

@monsieuricon what about if detecting a bot, just spit out from /dev/random, sure it will hit the server but maybe would get companies to stop harvesting without permission.

@louis @monsieuricon

With honeypot traps they risk getting de-listed from search engines.

A workaround could be that search engines should provide IP lists for their bot traffic (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#automatic) which should be configured to bypass the honeypot.

However, it is possible search engines themselves are the AI crawlers.

@inawhilecrocodile @monsieuricon But wouldn't the proof-of-work solution also have the same issue of preventing search engine bots from indexing?

@louis @monsieuricon

I've not implemented or looked at how Anubis works.

Maybe @cadey can answer.

@inawhilecrocodile @monsieuricon @cadey Just checked the website for Anubis. Apparently it does prevent the server from being indexed.

@KasTasMykolas @monsieuricon Yes, Anubis will skip the challenge if User-Agent string doesn't contain Mozilla.

@jernej__s @monsieuricon yup, thanks! was reading that but wasn't sure of later mentioned configuration & filtering options regarding fancier scrapers ;)

In general not a casual user of lynx, but tested just to cover basis of incoming shitstorm that something is not working anymore :)

@monsieuricon would b4 and git like that?

@monsieuricon interesting! can't wait to see the stats on percentage of requests blocked. any guesses?
also, wish i could get away with this :(

@mvc1095 @monsieuricon i wish *they* could *not* get away with this. crawlers are essentially doing large scale copyright infringement, if not criminal, large scale DOS attacks. how can we even let this happen? WP, one of the largest sites on the internet, has seen a 50% rise in bandwidth usage, that's nonsense https://mastodon.social/@camwilson/114267595008201156 we need to organise a collective response to this, individually putting up what are essentially CAPTCHAs is not a fix.

@mvc1095 @monsieuricon that said i don't want to target kernel.org here, you do what you have to do, and i suspect we (torproject.org) will have to do the same, at least for our gitlab instance, as we're seeing increased load as well. but damnit, we shouldn't individualize this problem, this is a collective issue. together we stand.

@Anarcat @mvc1095 @monsieuricon If you need help with that, please email me@xeiaso.net. I can also get you in touch with the artist I commissioned for the mascot design to get Anubis wearing a Tor hoodie or something.

@cadey @Anarcat @mvc1095 I would actually prefer no graphics at all, because if the goal is to minimize bots' impact, then we should also minimize how many bytes we send them. I know most of them won't download images, but some will and that's wasted bandwidth.

@monsieuricon Can you also email me?

Also if you're looking for devops people in Canada I'm on the market :)

@monsieuricon @Anarcat @mvc1095 @cadey yeah but on the other hand some mascot will look pretty. Especially comparing to a popular plain Cloudflare page.

Maybe somebody should really check how much data is wasted this way.

@yura @monsieuricon @cadey @mvc1095 i'd take an ASCII art version ;) (more seriously, i am not sure i'm ready to put (another) javascript gate in front of our servers, and i was informed that our load issue might come from inside anyways :p, but thanks for the offer!)

@Anarcat @yura @monsieuricon @mvc1095 It's no problem! Tor was critical to my free web browsing in my youth (I proxied Tor over XMPP to get around an iBoss set up by conservatives). I've always wanted to contribute back and it'd be exceptionally hilarious if that ended up happening this way.

@cadey @mvc1095 @monsieuricon @yura this is already pretty hilarious, and your contribution to our entire community is already felt tremendously. thank you. :)

@monsieuricon What solution are you putting in place? I’ve considered exploring the space myself.

@monsieuricon

Right now, I think that these bot deterrents are mostly just functioning similar to a "security thru obscurity" javascript blob.

I don't think the difficulty actually matters at all, you might as well set it to one because if scrapers ever try to solve the proof of work in the future, I think sha256 is categorically not going to work anymore since it's so easy to accelerate and so many accelerators for it already exist (bitcoin).

I actually created a proof of work bot deterrent before the LLM hype even existed. Back then I chose Scrypt as a memory-hard hash function because I wanted it to be as easy as possible for normal website visitors to solve, but as painful as possible for scrapers, even after they perceive it and react to it.

I don't have mine triggering on browser user agents. I just have it trigger all the time by default except for some tools that I allow list like git, npm, go, etc. I also explicitly allow home pages and repository home pages so that search indexers can still find things and display them.

You can see a demo of it here as well as the source code:

https://git.sequentialread.com/forest/pow-bot-deterrent-rp