Conversation
K. Ryabitsev
monsieuricon
Yo, ClaudeBot. You see that f'n robots.txt? Yes, it's for you.
Oh, you don't care? Well, then, choke on a bucket of 403s.
Oh, you don't care? Well, then, choke on a bucket of 403s.
5
2
10
@monsieuricon
they don't care .... I ban them with fail2ban
0
0
0
IP over Parrots (with QoS)
rfc2549@fosstodon.org@monsieuricon seems like a common occurrence of abuse, just recently:
- https://fosstodon.org/@kate/112524141250923512
- https://mastodon.social/@nixCraft/112344003877391227
0
0
0
Arjan van de Ven
fenruspdx@fosstodon.org@monsieuricon what really is needed is feeding poison to these bots rather than a 403 -- some kind of cheap Markov model creating english is stuff.... if they ignore robots.txt they then deserve what they get
0
0
1
Thomas Guyot-Sionnest
dermoth@noc.social@monsieuricon In another thread it was said ClaudeBot respected robots.txt... but only when explicitly told so with it's user agent!
It might **also** end up in an infinite loop refreshing your robots.txt, likely hoping you change your mind.
0
0
0