If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse

@osm_tech Hey. Sorry to hear about that. Drop me a line on Signal? username: briankrebs.07. Thanks!

@osm_tech You are definitely not alone: https://lwn.net/Articles/1008897/ The situation is not sustainable but I'm not sure what we do about it beyond waiting for the AI bubble to burst.

@osm_tech You're absolutely not alone, we wish you good luck! ~n

@osm_tech Tell me more. You can reach me at sjvn01 <at> gmail.com

@osm_tech I wonder if there's a way to fail2ban requests coming in faster than typically found in human requests.

@BalooUriza We use fail2ban to handle some of this with custom rules, but eventually fail2ban becomes a bottleneck after 100,000 IP addresses.

@osm_tech @BalooUriza For IPv4, a bitmask of the entire address space is a viable "efficient" implementation of blocking. I wonder if there are tools that can do it that way rather than needing a gigantic list.

@osm_tech what is a embedded-Sdk network?

@utf_7 @osm_tech

App developers can embed some "Sdk" into their apps or games.
The developer receives money.
The "Sdk"-provider proxies requests through these apps and games, to gain residential IPs.
And scrapers can buy these services, to tunnel their requests from residential IPs.

@AliveDevil @utf_7 @osm_tech So ridiculous that Google and Apple won't just permaban any developer embedding one of these "SDKs".

@osm_tech The proxy SDK providers need to be treated like the DDOS providers they are and prosecuted.

@InsertUser @osm_tech Pulling them from app stores and banning developers of the SDKs would be a good start. Save the criminal charges for after the damage control is done.

@osm_tech ugh.... why don't they use the exports...

@pietervdvn Because that would involve a human using their brains or having a shred of conscience and those both go against the basic principles of the companies doing this.

@osm_tech

@InsertUser @pietervdvn @osm_tech It goes against their whole ideology. The ideology says trust the machine to do what it copied from scraped Stack Overflow posts. If you try to intervene to make it do better, you're not trusting it.

@osm_tech @BalooUriza Like, a bitmask of IPv4 space is several times smaller than a Chrome instance. 🙃 🤡

@dalias I'd wish for them to enforce policies, but they get Ad- and IAP-revenue, so why bother.

Also, these "Sdks" probably have kill-switches (or rather, delayed activation) built-in, to not immediately contact their C&C servers.

@AliveDevil Yes but they could still be banned when caught. A few devs getting banned would be a big deterrent for others to ship this malware.

The right *technical* defense, however, is not to allow apps arbitrary network access unless they're declared in the manifest as a "browser" or other "client software" that the user can use with any service they want (like IRC clients, mail clients, Mastodon clients, etc.).

Instead, the manifest should declare a single domain the app can contact, or multiple if the developer is willing to pay for more intensive vetting of them, and only allow network access to the declared domain(s).

@LMieldazis @geerlingguy oooh do we get to show him our out-of-band (remote access) Raspberry Pi with dual power feeds, 4G modem and loads of serial connections? Saved our skin a good few times.

@osm_tech @LMieldazis would love to talk maps ops! I've seen many projects wrapping in map data and adding scripts to dl entire regions

@dalias @BalooUriza But that is one of the points @osm_tech are making in their post. These crawlers resort to using massive amounts of "scrapers hiding behind residential proxy/embedded-SDK networks" - meaning they are using Adware-infested phones all over the world for their scraping attaks. So banning IP ranges won't help much. Playing cat-and-mouse with these scrapers is resource intensive, which is increasingly hard for FOSS projects and is also driving up cost for commercial offerings.

@magezwitscher @BalooUriza @osm_tech Not ranges. Just the single IP, and a short-lived ban. All you need to do is get them down from thousands of requests per minute to one request per hour (because they get banned for an hour each time they start again).

@corbet @osm_tech I don't have answers either but I hope something emerges because waiting for the bubble to burst still may face the "the market can remain irrational longer than you can remain solvent" problem.

@osm_tech might be a thing @davidgerard could do on pivot

@froztbyte @osm_tech yeah i'm getting the same AI assholes

as is @RationalWiki (i'm the sysadmin trying to keep the site up in the face of the hammering - we can either lose Google search listing, or we can be literally unusable for humans)

as is @corbet at Linux Weekly News - OSM might be relevant to LWN, a free content project getting hammered by the AI bots

they botnet suburban Android boxes

covered it a bit previously on Pivot:

https://pivot-to-ai.com/2025/06/02/fighting-the-ai-scraper-bots-at-pivot-to-ai-and-rationalwiki/
https://pivot-to-ai.com/2025/09/07/the-ai-scraper-bots-are-hammering-pivot-to-ai-again-please-test/

@sjvn @osm_tech do contact sjvn!

@davidgerard @osm_tech @RationalWiki @corbet Also getting and handling them (as you know), but I’d be pretty interested to hear how bigger projects have to handle them

Quick check on latest status since last #iocaine restart: 1.49TB across 1.05B requests served

they never ever stop…

@davidgerard @froztbyte @osm_tech @RationalWiki @corbet An aside, but I had no idea you keep Rational Wiki running! I love that site. Thank you for all your hard work! I'm sorry the slopbros are trying to ruin it.

@theorangetheme @froztbyte @osm_tech @RationalWiki @corbet i quit the sysadmin job nine years ago, so of course i still have it