#curl sometimes fails to access some servers. In most situations the problem is not in curl itself but on the server side. Example:

1. Fails: curl https://www.radissonhotels.com

2. Works: curl -A 'Mozilla/5.0 xx Chrome/119' https://www.radissonhotels.com

3. Fails: curl -A 'Mozilla/5.0 xx Chrome/118' https://www.radissonhotels.com

4. Fails, too: curl -A 'Mozilla/5.0 xx Chrome/1189' https://www.radissonhotels.com

Perhaps they perform #filtering to obtain improved #security? It's hard to tell, but any serious attacker surely knows how to spoof the user agent string and bypass such simple #regex

@harrysintonen have a look at --> for an example ;)

https://metalhead.club/@bagder@mastodon.social/112274614049331580

@bagder

@totenlegionChris Yeah, that was stupidity on another scale.

@harrysintonen Filtering is not unheard of. But many do not care, i've been using the Trickbot user agent for years when scraping webpages without anyone noticing.

@untitaker Stopping some scraping might be a reason for it, yes.

As expected Safari and Firefox are also allowed. I guess Edge gets through due to claiming Chrome in the UA.

Works:
'Mozilla/5.0 xx firefox/119'
'Mozilla/5.0 xx version/16.3 safari/'

Doesn't work:
'Mozilla/5.0 xx firefox/118'
'Mozilla/5.0 xx version/16.2 safari/'

@harrysintonen A layered approach is how you do it. First you remove the noise (curl, headless tools, known bad agents), then turn you can enable bot detection (Javascript, behavior), then you can go for url filtering (whitelist/blacklist), and so on.

You will be amazed at how much crap gets filtered out of logs by simple filtering giving you higher log quality for behavioral analysis etc.

Simple is cheap performance wise and that is nice when you have a WAF in the game 😊

@lnxgeek That indeed is the most likely explanation. As long as the ruleset is maintained and everyone understands what this level of filtering is doing (read: not actually providing true security, just weeding out the crap) it is okay.

@harrysintonen I wonder if they do something similar in Tampere University network. I get oauth2 tokens created with Thunderbird in Linux but not with GNOME Evolution or any more sensible other email client. So I don't use tuni address for email at all...

@harrysintonen Maybe they will give you CSS and/or HTML specific for your browser or try to work around bugs in your browser?

You seem to oversimplify things. My list of known User Agents doesn’t have ‘Chrome/118’ and using a proper User Agent like 119 and 121 does work:

@burne Forcibly failing TLS in this case? Quite obtuse way to do it if that's the motivation.

I listed the corner cases which show the weaknesses of the regex the site uses. It checks specific character patterns, rather than interpreting the version number after "Chrome/". 1189 > 119 but yet 119 works and 1189 fails, even though 1189 is "newer". The obvious reason is that the regex matches 3 digits.