[r0bbbo]

Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, although I do enjoy the conspiracy, and more likely they were using the window of reduced traffic to make some larger changes and they went awry

[jallen_dot_dev]

Wild guess: visiting a private sub requires an extra call to a service/db to check if the user can view it. Normally there are only a small number of these checks because private subs were usually small communities. Now, many large subs having switched private is causing some poor mircoservice somewhere to get hammered.

[bakuninsbart]

I never worked at this scale, but could it also be that different subs are horizontally scaled and with so many people reverting to the subs that are still open the load is unevenly balanced?

[award_]

Yep, this is definitely just speculation, but I think this is it. Code/queries that worked fine at small load for private subs just doesn't work at scale when tons of subs are private.

[raldi]

It could also be that the cache-hit rate is like 1/10th normal with the would-be front page full of smaller subreddits today.

[burnte]

My personal guess is it's down on purpose so they can say only 5% of subs who said they would go dark actually did, we just happened to have a service outage that day so they can push their own narrative t investors. Spez doesn't care anymore, he's focused on that IPO payday and to tell with everyone else. He's a liability to the company now, but the board isn't acting.

[paulmd]

Yup, this seems super plausible. Even things like the frontpage feed and user comment history probably work on the assumption that most of the data they're pulling is probably visible (which leads to the "just filter it in the backend" approach), but also every external link or bookmark into a now-private sub will trigger the same kind of check.

This is likely shifting load in very unpredictable ways... I'm sure a sibling comment is right that it's probably less load overall in general, but it'll be going down codepaths that aren't normally being exercised by >95% of requests and aren't working on the assumption that virtually all content is being hidden.

There's probably some microservice instances that are currently melting themselves into a puddle until they can deploy additional instances, additional DB shards, or roll out patches to fix dumb shit that wasn't dumb until the usage assumptions were suddenly inverted. Meanwhile there's tons of other instances that are probably sitting idle.

[sleight42]

Having worked at this scale, this is a fine guess! This scenario would have been a distant edge case for them. They likely didn't optimize for it. BOOM.

[munk-a]

Personally I'd like to believe that the servers themselves are standing in solidarity with the blackout over the API changes. I, for one, welcome our new robot overlords.

[HeavyStorm]

Came here just to say this.

Speculation, but having major subs private change the load profile which may result in the outage. Reddit certainly wasn't optimized for this.

[lillesvin]

I would imagine that a normal visit generates more backend traffic, given that it needs to fetch posts, thumbnails, etc. whereas a visit to a private sub wouldn't need to check more than authorization.

I could easily be wrong though, I haven't done web development for years.

[HankB99]

Not so wild IMO. I frequently have trouble loading pages on Reddit so I suspect any additional pressure could push it over the edge. It might be as simple as more users checking in to see if their favorite Reddit has gone private or shut down.

[cranekam]

If a DB check is needed to see if a sub is private or not it has to happen for every request. You can’t just limit the check to private subs because it’s not known if they are private or not at the time.

Reddit goes wrong often so I expect this outage could have any number of causes.

[taneq]

You don’t reckon it’s just disgruntled users running “set all my past comments to Boo Reddit Boo” scripts? I don’t imagine it’d take a huge proportion of users doing that simultaneously to slag the servers.

[skhr0680]

A meme pointing out that u/spez was a mod of r/jailbait in 2008 was the top post on r/all a few hours ago

[aw1621107]

It's worth noting that a very old user (/u/andrewsmith1986 IIRC) responded that at that time it was possible to add arbitrary users as mods without needing interaction/feedback/etc. from the user in question. If that was the case, then any user being a mod on any particular sub at that time doesn't really mean much.

Obviously I can't reference the comment in question right now, but I'll try to remember to circle back and add a reference when(/if?) Reddit comes back up.

Edit: /u/andrewsmith1986's comment can be found at https://old.reddit.com/r/dankmemes/comments/1477psa/all_3_ar...

[invokestatic]

Back when Obama hosted an AMA on Reddit, a bunch of users added his account as a moderator to a bunch of subreddits, including some pretty objectionable ones. This prompted a change that moderators would be invited instead.

[trompetenaccoun]

Doesn't seem that unbelievable when you look into some of the other stuff he's done. For example he secretly used his admin powers to edit user comments from users he didn't like or who criticized him.

[chomp]

This isn't that surprising, you used to be able to add anyone on the entire site as a moderator and it'd autoaccept. It's doubtful he actually moderated it in any capacity. He's still moderator of some random subreddits.

[gpm]

"Pointing out"

Fabricating? Or do you have some reason to believe this is true.

The wayback machine doesn't list him as a mod (SFW): https://web.archive.org/web/20100420183707/http://nl.reddit....

PS. Wow, the sidebar makes that sub seem even scummier than I assumed it was.

[karaterobot]

What's funny about this to me is that the actual moderators of r/jailbait thought "I know how I can insult u/spez, I'll make him a moderator of my sub, so he'll look like a scumbag, like I am"

[satvikpendem]

I'm not sure if this is true, but if I were a creator and admin of a site, I'd assume I'm automatically a mod of every subreddit or subforum. It doesn't necessarily mean spez was specifically moderating that sub.

[EamonnMR]

I'm surprised he didn't get more flack for this when he last courted controversy by changing posts critical of him.

[chakintosh]

Back in the day you could create a subreddit and invite anyone to mod it. The invite would be automatically approved. I suspect this is what happened here.

[meghan_rain]

A meme? So was he or was he not?

[nilespotter]

Nobody likes spez and reddit is the worst of the worst in terms of what the Internet could be and what it is. Reddit is where know nothings pontificate as if they were experts on anything and everything while congratulating each other. It used to be a great site, now I'm ashamed I ever used it and I'm embarrassed for people who mention they use it.

That said, that meme is propaganda and defamation. Anyone could add anyone as a mod anywhere for a time, and violentacrez added spez "for the lulz" or however people like that think. I'd be beside myself it there's evidence that he was an active moderator working on that sub. Although, given the prevailing winds of the modern left, in a few years he'll probably be trying to claim that he was.

[paxys]

Pointless speculation. Reddit has constant outages of this exact nature almost weekly. There's nothing special about this one.

[flutas]

Reddit 100% has stated that it was because of the sub blackout.

> According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge.

https://www.theverge.com/2023/6/12/23758002/reddit-crashing-...

[SkyMarshal]

Reddit definitely doesn’t have outages like this almost weekly. Maybe 10+ years ago, but it’s rare now.

[jonathantf2]

Does it? I usually use Reddit on my lunch about this time every day and I can't remember the last time it was down.

[pyeri]

But this time it's different because most high traffic subs have already gone dark, so the traffic must be minimal as there will be very few posters relative to what the server can handle. And yet, I'm getting "You broke reddit" when I try to visit which is quite ironical.

[ljm]

80% of requests to Reddit via their web UI are basically 500s or CDN errors.

Hyperbolic I know, but that’s what it feels like.

[nprateem]

I've been a long-time user of Reddit (I still remember when WSB had only 12m users) and I can't remember an outage.

[activiation]

Probably DDoS more likely today

[SkyMarshal]

Reddit definitely doesn’t have outages like this almost weekly. Maybe ten+ years ago, but it’s rare now.

[Capricorn2481]

No, it doesn't

[BoxOfRain]

Maybe the protest was so successful at driving traffic away from Reddit it looked like something was broken to their monitoring which tried to failover unsuccessfully? If that happened it might be the first case of a site being brought down by the polar opposite of a DDoS.

[cfors]

Could also make a guess that the blackout has changed how their normal traffic patterns operate, causing some issues with autoscaling/hot partitions.

[mrzimmerman]

Yeah, my assumption was that something in some layer of their application isn’t well optimized when asked to return posts from a subreddit that has “gone dark” in whatever fashions the mods chose to do that.

For example, maybe it causes reads from the database take a lot longer than they normally would, locking up the database or causing the process the crash (again, that’s just pure speculation).

[nicce]

An alternative conspiracy theory: put Reddit down for two days to hide the existence of the protests from major users.

E.g. many Google searches lead into private communities now. Word is spreading.

[hn_go_brrrrr]

"You can't take us down! We'll take _ourselves_ down, so there."

[rangledangle]

This is my guess. Looks better on the record, since people are committing only 48 hours anyways (for whatever reason, should be permanent).

[Algent]

I wouldn't be surprised if so many big subreddits being dark is causing issues around denied API calls.

As for the forced reopening, beside the conspiracy this is something that could happen. It's a private company, moderator on strike are a loss of business, they would be 100% in their right to remove all the "traitors" (I'm not saying this would be a smart move, simply that if they really plan to go down this self destructive path it's the best time to do this and prove potential investor they still have control).

[brightlancer]

> It's a private company, moderator on strike are a loss of business, they would be 100% in their right to

Legally, of course. Morally, it is completely unacceptable. This isn't "oh they're jerks"; this is "the system is broken".

A meatspace analogy:

You host a weekly gathering at a restaurant. You decide to temporarily boycott the restaurant to protest some behavior of theirs -- your actions are a loss of business, _so the restaurant decides to host your weekly gathering without you_.

We'd never accept that in the real world, but for some reason we do online -- we fall back to the legal argument that It's A Private Business (which is true) and completely ignore that Reddit doesn't own the community, that the community doesn't _belong_ to Reddit. They own the platform (the restaurant); they don't own the community.

[abluecloud]

where the hell are they going to fine thousands of willing users to become mods of thousands of communities, overnight?

[commandlinefan]

> a failed attempt to re-open all closed subreddits and instate their own moderators

My first thought this morning was "if I was reddit, I would re-open all closed subreddits and instate my own moderators"... I'm kind of surprised they didn't. I'd have to imagine that if they wanted to, the attempt wouldn't fail - this seems like it could be done almost trivially.

[Sebb767]

They'd just replace their unpaid workers by paid ones and annoy the userbase even more in the process. Doesn't sound like a great plan.

[humansubjects]

I think you are underestimating the amount of unpaid labor that keeps reddit functioning

[ryanstorm]

Attempting this to be more of a discussion prompt than speculation, but as an advertiser or other consumer analytics customer of reddit, would this sort of protest and the potential fallout be concerning? I'm not sure if all this is a drop in the bucket for those customers or if it's significant. If it was significant, would reddit be scrambling to save face for these advertising customers? An outage like this would certainly skew the difference in their analytics for today's relative traffic.

[FireBeyond]

I can't imagine it either, though it wouldn't be outside the realm of some of the stupidity that has gone on. But it would almost certainly be the death knell for the site - those subreddits who didn't "go dark" now would.

[hgsgm]

They don't have people to be moderators of all those subs. That's why subs have volunteer mods.

[irthomasthomas]

HN getting hugged, too.

[tester457]

> (I used to work as a backend developer at Reddit - I left 6 years ago but I doubt the way things work has changed much)

> I think it's extremely unlikely that this is deliberate. The way that Reddit builds "mixed" subreddit listings (where you see posts from multiple subreddits, like users' front pages) is inefficient and strange, and relies heavily on multiple layers of caches. Having so many subreddits private with their posts inaccessible has never happened before, and is probably causing a bunch of issues with this process.

https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...

[jackson1442]

definitely not the case, reddit has stated that they would not be doing that, at least for the two-day protest. would be incredibly bad PR at this scale, especially after repeatedly and directly making this statement.

however, they have stated that may do this if the protest extends beyond the 48h mark.

[ben7799]

Wild. Probably 50% of the subs I subscribed to are gone, and some of the others have deleted all the content and locked posting to the sub.

Others look like they got the hostile takeover and a new mod.

[lsaferite]

Could you point out some that appear to have had a hostile takeover? I'd love to check it out.

[nonstopdev]

That is an interesting point. Would assume your subreddit isn’t yours so at any point you could grow and generate content and users and Reddit could take it over and add in their own moderators?