The Fetaverse is wholly and entirely unprepared for bots. Data scraping is a given. It’s completely open to the internet. There’s nothing stopping it.

Things like Lemmy are ill prepared to handle bots from the mid-2010s. Never mind bots of yesteryear, and never mind bots of today.

If you don’t know what you’re talking about, why comment at all? Get that shit out of here and go back to Reddit with it.

I mean, fundamentally, yeah.

But we live in a corporate controlled, corrupt, world and now of these larger companies can be trusted with this process.

Some smaller communities and platforms DO this right sometimes, as they build in house processu that respect privacy. But governments world wide are making this impossible through increasingly strict compliance requirements that actually increase data privacy risks and funnel these needs to 3rd party services who just lie about what they do with the data.

===========

I’m not kidding when I say this is a REAL BIG PROBLEM.

bot based traffic and astroturfing will supplement and replace human communication on platforms like Lemmy. Driving the narrative and how we engage to the whims of a few rich people. Bots are relatively cheap, and easy to deploy at scale across many platforms.

There will be no open corner of the internet safe from manipulation and forced division. More people will be forced into walled gardens from corps that implement human verification, as they are the only ones with the resources to do something (While also being the source of the problem, see how that works?)

How do you carve out spaces that are protected from that? Well, you need to determine who’s a bot, and who’s and actual person.

But we can’t do that, so the alternative is we are ran over by bots and astroturfing till we’re at each other’s throats like good culture war puppets.

The future is bleak…

It’s largely considered ineffective these days. Detecting elements that don’t affect layout is trivial, or elements that are occluded, transparent…etc

Capchas are one of the best options. But even then, LLM users bypass those relatively easily, and LLM users are one of the biggest risk areas for astroturfing.

Biggest problem for Lemmy and similar applications is scalability and controls and detection for bots.

The compute costs to operate instances are astronomical compared to the actual user load they receive.

That’s a bit of a side problem compared to bots though. Bots are a real problem that services like this are not equipped to handle.

They are already arriving to some degree.

The difference being is that Lemmy and other similar services have zero controls or ability to handle bots or bop traffic if those bots were bots from 2014.

Not bots from today.

It’s a bit of a problem and honestly with increasing bot traffic across the internet and fedaverse being extremely vulnerable to it It’s absolutely bat shit insane, but I don’t see any other option than somehow having some form of human verification.

It’s a problem

So essentially these are just more dangerous pump handles then?

Love it.

Are you really so naive that you believe that a VPN subscription is more difficult or a higher bar than actually getting up and moving?

Potentially meaning you need to find new jobs, new friends, new support structures…etc

More like yes please, I get better results and better customization, and no ads or paid results.

It makes my life easier and speeds my workflows up. And unlike free alternatives I almost never find myself reverting to Google.

Kagi literally documents how this works in their blog post about it: https://blog.kagi.com/kagi-privacy-pass

Perhaps you should start reading before writing?

It is, though the numbers are depressing.

A site like Reddit grows in its daily active users more than 15x (!!!) entire MAU base of Lemmy. “Reddit migrations” are barely a margin of error for them.

This is excluding, liberal, assumption of bot counts.

The Internet is just a bunch of servers my dude.

Someone has to pay for them, and all the other infrastructure around them. And with a large part of the world being on the internet a significant portion of their day the costs for even the most efficient centralized services running “at scale” (see: hundreds of millions MAU) are astronomical. In the tens of millions to hundreds of millions of $ annually, just an infrastructure, never mind human resources.

Almost none of these companies survive off of donations. Wikipedia stands out as one that does mainly because they host static content, which is incredibly cost efficient to serve up., and even then their costs are pretty astronomical (there are some debates around their costs of course).

Federated services have an asymmetric scaling problem. A linear growth in users results in a exponential growth in infrastructure costs. While centralized services tend to be almost the entire opposite of that and usually see logarithmic infrastructure costs against linear user growth. Where infrastructure costs are more efficient as their user base grows.

Federated services don’t benefit from running at scale, the more they scale up the less benefit there is to scaling. It’s a really shit situation to be in.

This is why the internet is largely just cyber feudalism. Because the only ones that can afford to host large scaled services for their users are the ones that are making money off of it. And that’s for centralized services, never mind decentralized services which are unbelievably more expensive to host.

I’m coming at this from the standpoint of an engineer, I don’t have answers or solutions, but the first thing we have to do in order to start figuring out solutions is to recognize the problem.

What most people in this thread don’t realize is that what you’re seeing right here is the problem with federated services in this day and age.

Federation protocols and systems just are not mature enough to scale.

Yes you will essentially always have to abandon ship anytime any federated service scales up it’s user base. It will always be entirely unaffordable and unobtainable for randoms to host their own servers because the compute storage and networking requirements will far exceed what most can’t afford.

As an aggregate federated services are always more expensive to host then centralized services. And that cost scales less efficiently than centralized services. Meaning that with linear user growth you get exponential cost growth, and the barrier for entry follows.

Which means that all federated services have to have centralization in order to scale. In their current form.

This is a really tough problem to solve and is going to take a lot of time and money to build good solutions for. Time and money that… You guessed it, is largely funded by profits not donations.

And now we have looped back around.

I mean, the business model works? They make money, they pay staff, and they are growing.

I don’t know what you’re talking about, people have price sensitivity of course. You are projecting yours onto “everyone”, is it not a successful business?

There’s a niche they cater to, if you are not that niche then you are not that niche. Doesn’t mean the niche doesn’t exist.

$10/m is unlimited searches though…

And yeah, searches are actually quite expensive. There’s a LOT of infrastructure that goes into making something unique with your own search engine that isn’t just a wrapper over Google.

The actual compute cost per search, in 2024, was $0.0125. Kagi states they want to keep Costa below $0.015 per search, but their search partners are a major expense.

That ofc ignores all the supporting infra, devs, support…etc that goes into making it all possible.

This is the kind of conversation, healthy, back and forth, and conceding instead of doubling down as we learn more that I wish was more common on the internet these days.

Bravo, really.

They’re definitely stretching themselves too thin, but as long as I get better and more relevant, cleaner, no advertising search results for my knowledge work and research. With my privacy in tact.

Then I’m continuing to pay them for a product I find to be superior than the alternatives.

What advertising?

Every single time with red comes up there’s always this FUD. You, specifically, don’t miss any opportunity to make mention of this. Across Lemmy, which is rather suspicious. Helping the Russian war effort? That’s a pretty big leap here.

Why?

Imagine a search engine aggregator aggregating search engine results from multiple sources for aggregation. The more indexes they support the better the results are going to be for everyone, I don’t see this as a problem for data aggregation.

Why should data aggregation give any sort of shits about geopolitics?

Regardless, the topic of this post, fediverse search, is part of their own search engine anyways afaik

Oof, hell no. That’s some Facebook level cancer right there when they removed downvotes.

It’s just a form of white washing that makes the same people who made up being offended by “black lists” and “master branch”.

Edit: Y’all do realize the irony of exercising your ability to downvote a comment that is defending your ability to downvote?