Hi,
I had a couple of people mention they were getting an error trying to access a link I posted in c/fountainpens. One of the users (@gemelen@mammut.moe) created a traffic dump of trying to access the link using curl. He posted his results on this Opensuse paste bin.
He added this:
I guess Cloudflare prevents some web clients (like embedded viewer for a Mastodon mobile app) and some networks (like mine Uruguayan Internet provider) - I could open the feddit.online and see your post there in a desktop browser, but being forbidden to access from a mobile or via curl.
George
I’ve just asked gemelen if he can join us to answer your questions.
I was thinking about what you said regarding curl (originally in your deleted post): would setting the user agent have had any effect? I seem to recall having done that in the past for some specific script I wrote (but that was over 15 years ago, so my memory may be shot).
Makes sense that if the URL was wrong in the first place, then the whole thing would fail with a 404 or 403.
gemelen did say that when he tried it from his desktop it worked — hence the reason he suspected it was an issue between cloudflare and his mobile carrier, as he mentions in the quote above.
No, the agent string is not enough.
I just checked. In the past 24 hours, the firewall has dumped 98K requests from AI bots that tried really hard to look like normal traffic and not like AI bots. These bots did a lot more than just set a user agent to try to look like real users, but it didn’t help them.
Not surprising… I’m sure CloudFlare is working overtime to block these ridiculous bots. I’ve seen my analytics provider is blocking tons of bots every day, even low volume sites like mine are getting hit with hundreds of bot requests.