@monsieuricon Don't set the difficulty above 5 or it becomes very mathematically unlikely for people to pass the challenge!
@monsieuricon I mean, that's funny enough that I'm willing to implement geoip lookups to make it happen :)
@monsieuricon will the JS requirement (or simply the proxy itself) for lore.kernel.org have an impact on `b4` (e.g. via `b4 shazam` or `b4 am`)?
@monsieuricon mhm. Is lynx and w3m supposed to go through without any mentions of the challenge?
@monsieuricon that's the point, tried it expecting some issues due to JS. But no problems whatsoever :)
@monsieuricon You know what, that's funny enough that I'm adding geoIP to the issue tracker: https://github.com/TecharoHQ/anubis/issues/206
@monsieuricon ^ @Klaus_Mueller @BNetzA @bsi yet another day of #OpenSource, here the largest one of them all, the #Linux kernel itself, fighting with #AI crawler traffic. If only there were some organization able to regulate this...
@monsieuricon hmm it works fine on my PC, but on my phone it goes forward a bit at 1.2kH/s, stops, goes back to zero as the hashrate drops to 0.012kH/s, then fluctuates a bit, times out, and refreshes :(
@monsieuricon Just out of curiosity, why choose this over the honey-pot traps?
@monsieuricon what about if detecting a bot, just spit out from /dev/random, sure it will hit the server but maybe would get companies to stop harvesting without permission.
With honeypot traps they risk getting de-listed from search engines.
A workaround could be that search engines should provide IP lists for their bot traffic (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#automatic) which should be configured to bypass the honeypot.
However, it is possible search engines themselves are the AI crawlers.
@inawhilecrocodile @monsieuricon But wouldn't the proof-of-work solution also have the same issue of preventing search engine bots from indexing?
@inawhilecrocodile @monsieuricon @cadey Just checked the website for Anubis. Apparently it does prevent the server from being indexed.
@KasTasMykolas @monsieuricon Yes, Anubis will skip the challenge if User-Agent string doesn't contain Mozilla.
@jernej__s @monsieuricon yup, thanks! was reading that but wasn't sure of later mentioned configuration & filtering options regarding fancier scrapers ;)
In general not a casual user of lynx, but tested just to cover basis of incoming shitstorm that something is not working anymore :)
@monsieuricon interesting! can't wait to see the stats on percentage of requests blocked. any guesses?
also, wish i could get away with this :(
@mvc1095 @monsieuricon i wish *they* could *not* get away with this. crawlers are essentially doing large scale copyright infringement, if not criminal, large scale DOS attacks. how can we even let this happen? WP, one of the largest sites on the internet, has seen a 50% rise in bandwidth usage, that's nonsense https://mastodon.social/@camwilson/114267595008201156 we need to organise a collective response to this, individually putting up what are essentially CAPTCHAs is not a fix.
@mvc1095 @monsieuricon that said i don't want to target kernel.org here, you do what you have to do, and i suspect we (torproject.org) will have to do the same, at least for our gitlab instance, as we're seeing increased load as well. but damnit, we shouldn't individualize this problem, this is a collective issue. together we stand.
@Anarcat @mvc1095 @monsieuricon If you need help with that, please email me@xeiaso.net
. I can also get you in touch with the artist I commissioned for the mascot design to get Anubis wearing a Tor hoodie or something.
@monsieuricon Can you also email me?
Also if you're looking for devops people in Canada I'm on the market :)
@yura @monsieuricon @cadey @mvc1095 i'd take an ASCII art version ;) (more seriously, i am not sure i'm ready to put (another) javascript gate in front of our servers, and i was informed that our load issue might come from inside anyways :p, but thanks for the offer!)
@Anarcat @yura @monsieuricon @mvc1095 It's no problem! Tor was critical to my free web browsing in my youth (I proxied Tor over XMPP to get around an iBoss set up by conservatives). I've always wanted to contribute back and it'd be exceptionally hilarious if that ended up happening this way.
@cadey @mvc1095 @monsieuricon @yura this is already pretty hilarious, and your contribution to our entire community is already felt tremendously. thank you. :)