I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site’s servers. More here.. For that reason, NPR has blocked Fediverse servers like Mastodon.
I have experienced this on my personal server as well (DDOS), but at the time didn’t know why, exactly.
This isn’t the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.
Is this why the automated title retrieval never works for NPR?
Yep
smaller sites like NPR
Norwalk Public Radio. From Norwalk, Iowa.
They can’t afford a $5 CDN?
?
the people have spoken
Well they didn’t seem to have much to say?
smaller sites like NPR
There are two proposed solutions I have seen. I do not claim that this is exhaustive, just only limited to the proposals that I have stumbled across.
FASP - Fediverse Auxiliary Service Provider
The idea that mastodon has is to have a third-party service that does something like generate a link preview once and then share that to fediverse instances that need it. They have a whole spec they wrote up about these kinds of services that can be run to help out fediverse instances with different things like this, search, or discoverability. They call these FASPs - the repo with the spec is here.
I don’t yet know of any FASP that is live and running, so it is kind of a paper spec at the moment. I don’t do too much on the microblogging side of the fediverse though, so I very well could be wrong.
Federate link metadata and the post at the same time
This comes from an issue on PieFed’s repo. It is basically just an idea at this point, but it is a pretty basic one; have the originating instance fetch the link preview information and simply federate it out alongside the post itself.
That second solution is what I thought of as doing as I read this. It doesn’t rely on a third party service being up. And that instance has to do that work anyway. So why not just share the results with others?
It was my first thought too but there are downsides to it. Trust is a small one given the nature of the system but compute and bandwidth costs could be an issue also, especially for larger instances.
The first solution is more fragile but could be CDN’d and those are cheap.
I don’t think compute resources should be an issue, if anything it saves other instances resources.
As far as I understand, right now, all instances are fetching this data individually, so they all have the same compute load from this. Doing it on one and sending it along with federation should keep resource useage pretty much the same on the originating instance, and less for all others.
This would still increase bandwidth of course, but at least with all VPS providers I have used, bandwith is pretty cheap.
WebTorrent kind of solution like PeerTube is the best, because it’s highly scalable yet not that resource-intensive, because everyone (even users) provides their bit; also, as it’s entirely P2P, it acts sort like a distributed and decentralized CDN, so it’s snappier for everyone because it’s more likely that you have a peer near to you like that.
Absolutely LOVE this idea but I feel like a solid implementation would be a pain in the fanny (and it also requires other peers to be online). I dip in and out over the day myself but I don’t know what others use case is.
Also it is far less trustworthy and harder to block bad actors (V’s removing a known bad actor from a relatively small peer group).
Idk, the part of uptime of peers is solved precisely by the fact that not every peer will be connected at the same moments that every other peer. If I’m connected at 10pm, it’s highly likely that there will be at least another single peer at that hour that has what I’m pulling.
As for bad actors, that’s something that at the end of the day is unavoidable. Though with some automated systems could be possible to make it more hard for them, but it’s not that different from a centralised approach, and thus is better because is computationally more efficient.
how would you know the results havent been tampered with?
I’m trusting the user who submitted the link and/or the instance that user is on. I’m doing that anyway by clicking on their link right?
So I receive a post from someone presumably with some comment, along with a preview of a link, and I’m supposed to trust one but not the other ?
The same way you know the rest of the data hasn’t been tampered with - you don’t. And when you find untrustworthy instances, you defederate from them.
That’s not true, you know the data hasn’t been tampered with because of HTTP signatures. The fediverse isn’t just “yolo, just trust me bro”. There are some parts that are a bit more towards that side (particularly with groups on Lemmy) but I don’t think this should be.
HTTP signatures are not a default feature, the baseline is ECDHE key exchange with forward secrecy and deniability. Headers are not typically signed.
There isn’t even an official standard for redistributable signed page metadata. There’s just prototypes. I absolutely hope one of those prototypes becomes official, but we’re not there yet.
The activitypub fediverse is literally very extremely Yolo trust me bro, it works because you can see the originating server with everything and misbehaving ones gets banned.
It could have been FAPS, what a missed opportunity.
Really the only metadata we are interested in is the thumbnail image url. Afaik PieFed already retrieves this from the og:image meta tag before sending the URL along with the other post content to the instance hosting the community.
Another idea - those meta tags are also retrieved during post composition in an ajax call, in order to prepopulate the title and body fields so the image url could be stashed in a hidden field at that point…
I’m a fan of the second one. My thought is, if you trust a server enough to accept messages from it, then you should also trust it enough to let it scrape the page title, article image, etc. for you, too.
Seems like a no-brainer! They could even leave the old behavior as an option for instance admins to re-enable, but if they default it to not DDoSing, that would make a huge difference.
deleted by creator
This is no different from how it is already, though. If a user posts a CSAM link, your server immediately visits that link to scrape it, and you suddenly have that content on your server.
Of course, if you have a blocklist of forbidden sites, then your server will know not to scrape that site. And the same thing applies if we allow servers to send pre-scraped metadata. If the incoming message is for a blocked URL, just drop the message and don’t save any part of it.
The only new threat here is for a malicious server operator to modify outgoing messages to include CSAM metadata on a harmless link. However, that possibility already exists with any other content posted on the fediverse. If a malicious server is modifying its own outgoing messages, the only fix is to defederate from it.
Allowing servers to forward pre-scraped metadata to other servers does not introduce any new threats that do not already exist.
There’s no reason to believe that this mechanism doesn’t check for CSAM (I am also pretty stumped that this is actually a problem with Mastodon according to your comment?) . Lemmy had a phase where messed up stuff got federated, which led to @db0@lemmy.dbzer0.com’s implementation of an anti-CSAM tool: https://github.com/db0/fedi-safety, which is based on the CSAM detection tool for the AI Horde: https://dbzer0.com/blog/ai-powered-anti-csam-filter-for-stable-diffusion/ - it’s probably not so hard to adapt for Mastodon too.
deleted by creator
Both Lemmy and Mbin already attach the preview image to the post via the
imageproperty, which is of typeImagewith a URL pointed at the hosting instance. Receiving instances can then fetch that URL instead of the article.Is there a reason why Piefed doesn’t use this already established approach?
I’ve just had a look at the code to refresh my memory - it does use it :)
they could include the link preview data as part of the federation
Perhaps a modification to the ActivityPub specification?
How do you know that data is authentic?
Lemmy does this to itself, too - when you create a post it federates out and then every Lemmy instance that receives it makes a GET request to retrieve the post from the original server. Even though it just received the exact same data. Some other fedi software is even worse - Akkoma does a GET of the id of every activity it receives, not just for posts but for Likes too.
It’s insane.
Anyway, proper caching at the reverse proxy / Cloudflare level solves it. The same kind of caching setup would do the same for sites serving HTML, like NPR too.
That’s weird. Is that how it’s supposed to work, or did they just implement it in a bad way?
It’s an access control method, called “authorized fetch”. Rather than broadcasting the data you just announce the source of it, and then any client that wants to show it needs to negotiate with the originating server to request the data (And then blocklists and such come into play)
Say you’ve got 3 users, A, B, and C. A has blocked C, B follows A, and C follows B. Now A makes a post, B sees it in their feed and reboosts it, what does C see?
If B’s server sends the full response along, then C can see the contents of A’s post even though they’ve been blocked by A. But if all it sends is the URL of the post, C’s server needs to make a request for it (providing authentication), and then A’s server can reject it, so C sees nothing.
I’m not sure but my guess it’s an attempt to deal with the problem where sometimes federation activities arrive in the wrong order (especially after an instance has been offline for a while) - by making a GET to the origin server you can be sure you’ve got the latest version of the content. There are probably better ways to achieve that, though.
PieFed prioritises efficiency over correctness so it just accepts whatever is sent to it. In 3 years no one has complained about seeing stale content although I’m sure it happens occasionally.
A slight delay would be perfectly sufficient.
If they’re all delayed the same it would have the same effect. In order to prevent that it would require some form of coordination amongst them. And even then would have the same complaint about missing link previews. Better than nothing but not really a “solution”, per se.
Random numbers exist.
wait 3 //randomlydecidedby fair dice roll
Whoa!
Random back off is an ancient and very well understood algorithm. This is not exactly rocket science.
No one said anything about random.
“Random back off” is the name of a technique to avoid this exact problem.
Each instance generates a random number of (say) seconds between (say 15 to 120) and schedules the fetch for then. It is extremely unlikely that any two instances select the same second.
Yes I understand. I just don’t appreciate the demeaning tone this information was delivered in.
Nobody said the delay had to be fixed instead of random either…
Yes that’s the problem…?
If they’re all delayed the same it would have the same effect. In order to prevent that it would require some form of coordination amongst them.
A delay randomized within a certain range could accomplish this without needing coordination.
I added a rule in my .htaccess for their user agents to direct all requests to a static page on my domain.
Heh. Ever heard of the Slashdot effect?
For the fifth fucking time, this is not the same thing .
This probably why my place of work blocked it behind the firewall
Do they block the useragent or the URL? NPR apparently blocks the former.
You’re talking about 2 different kinds of blocking here. NPR can’t block a URL, that concept doesn’t really exist at this level. They could block ip addresses, or user agents as you mentioned.
A workplace can’t block by user agent (at least not on any website using TLS), as this header is not readable by any intermediary devices.
NPR can’t block a URL
I wasn’t talking about NPR, I was talking about their employer…
You mentioned both in the same comment…
[…] NPR apparently blocks the former
I’m just saying, their employer can’t block like NPR does, because they are fundamentally different things. The one is a server receiving a request, the other is a firewall seeing encrypted packets.
Edit: the other could also be a DNS resolver that just does not resolve certain domains.
This was my first thought too.
Same thing used to happen with reddit giving sites the hug of death lol
It’s unfortunate but kind of expected.
Its not at all the same thing
You’re right, it’s not like… thousands of network connections are happening to the same site in both cases resulting in resource exhaustion.
Oh, wait…
Having things in common does not make them the same thing. I explained this in the OP.
You’re right, neither of them are known for providing thousands and thousands of distributed network connections to the same destination resulting in that same destination server running out of resources. They’re clearly not the same thing at all, no, definitely not at all.
Why not have the users fetch the previews themselves. The user’s browser, that is.
For sure there is more users than instances. So the amount of requests would be grater but at least they wouldn’t happen at the same time.
But if fediverse ever grow to be as big as other social media, then a single post/link can be viewed by thousands of users at the same time causing the same effect.
Or get the preview on the first user seeing the post in that instance.
Sounds like every instance is doing the preview generation. Maybe it could grab the preview data it needs on the original instance, and pass that around with the post…
Well you can’t really trust the original instance to be truthful with its preview.
Randomly stagger the times of each instance’s request.
+1-300 seconds
300 seconds is a lot.
That’s basically what’s now implemented, iirc. At least in masto
I’m not exactly sure why a web server can’t handle legit web traffic, just because it’s a lot? I agree that smaller actors should be given a little breastfeeding but solving it on the other side is bad architecture. A legit request can never be compared to a a ddos attack, the overloaded net has to throttle and then the instances need to retry with a generous enough delay.
Options for solutions to another systems problems are all going to introduce more problems. Sure, the requests could honestly be routed to a cache and it would help this problem, att least until it breaks or is exploited. But it would never be truly solved that way.
I’m not exactly sure why a web server can’t handle legit web traffic, just because it’s a lot?
Its not because it’s a lot, it’s because it all happens in the same instant, which does not resemble any sort of normal web traffic.
but solving it on the other side is bad architecture
Solving it on the side that’s creating the problem is bad architecture? 🤔
But it would never be truly solved that way.
Several potential solutions have been posited, if you just look through the comments.
Lots of normal traffic is instant. Throttling must be handled by for in this instance NPR or its providers routers because even if hammering is a bad practice, we are participating in why it happens at all if we just build structures to mitigate it.
Yes that’s why I included a solution, and explained explicitly that it can be solved but only temporarily
Lots of normal traffic is instant
Not instant from a thousand different servers. That’s what a botnet looks like. AKA DDOS. That’s why I explained that. Why are you acting like you don’t understand this?
Throttling must be handled by for in this instance NPR
No. It mustn’t. AP servers can fix it on their end.
Yes that’s why I included a solution
Which you then went on to explain wasn’t a solution…?
You may suffer from irritation of your ego my friend. Know that you are not your ideas or projects, you are more than that. I feel like all of your questions were previously answered sorry.
You may suffer from the dumbs, my friend. My questions were answered. Before you showed up.
Stop with the “you may suffer from” nonsense (both of you). You’re being rude to each other and it’s generating reports.
I’m going to continue being rude to people who are rude to me. Sorry, not sorry.
I remember reading the same thing about reddit, slashdot, fark, and other sites. But I like the FASP idea as a way to relieve this.
Why is that? Those are centralized platforms, they should only need to retrieve the preview once each?
https://en.wikipedia.org/wiki/Slashdot_effect but people were going in, looking at the link, sometimes overwhelming the server
That’s not at all the same thing. That’s just the inability to handle large amounts of unexpected normal user traffic.
This is essentially bot traffic driven by servers all independently but simultaneously fetching the same image, without even a single human necessarily looking at it.
From server’s perspective the only difference between those is the user-agent. Traffic is the same
The origin and purpose of the traffic is not the same. That’s why they block one and not the other.
But what OP is talking about is a technical detail that can be fixed.
Yes, but then a comment mentioned the “hug of death” - high spike of requests resulting from a link being shared on some site, and OP asked why centralized sites were also creating problems
There was another post regarding previews some months/(knowing me probably a year) ago and personally I would go with federating previews. With a button for mods to refresh it from source for when something goes fucky
But social networks have been creating random bursts of requests, this way or another, even in the times of centralized socials
