Conversation
ai bot traffic
Show content

this is just the stupidest shit:

https://social.anoxinon.de/@Codeberg/115033796075422170

and from what i can tell, basically everyone else who provides any kind of public resource on the web is dealing with the same garbage traffic. i can assure you that the people i work with are burning absurd amounts of time and effort on this.

and in service of who and what? do you think the absolute worst people in the world would be all in on this if it were a _good_ idea for the rest of us?

2
3
0
re: ai bot traffic
Show content

@brennen I just wish they gave me literally any advance notice about this. I found out when it showed up on Hacker News.

0
0
0

Jonathan Corbet

re: ai bot traffic
Show content
@brennen This is one of the reasons why I haven't been hugely tempted to add a system like Anubis to @lwn ... it always seems that, once it started actively getting in the way of the scraperbots, they would just add the code to solve the challenges. These people aren't running the scrapers on their own computers, after all, they have little reason to care about burning more CPU time.

What a world we have created for ourselves.

RE: https://federation.p1k3.com/@brennen/115034083662379513
2
2
7
re: ai bot traffic
Show content

@corbet @brennen @lwn It's similar to copy protection measures added to SW in the 1980s - they were annoying to normal users but did little to protect from actual piracy as they always could be cracked by whoever wanted. The problem should be solved at higher level.

1
0
0
re: ai bot traffic
Show content

@aakoskin @corbet @brennen @lwn

Totally agree with the assessments all around here - this isn't a permanent solution - but I think it's important to help operators in the short-term while bringing attention to the need for that higher level solution.

I'm not sure where that higher level solution is going to come from though.

0
0
0
re: ai bot traffic
Show content

@corbet @lwn yeah, hard to see the arms race here as winnable. feels like this is only going to abate once either the incentives to scrape disappear or there are fundamental architectural changes to the network.

0
0
0
ai bot traffic
Show content

@brennen Our distro mirror at work has been hammered so hard in the last week we've denied all external access. It used to be open.

0
1
0