Facebook employees badges aren't working, unable to enter buildings

[ThePadawan]

Heard a story once about the badge readers at a European Google office being suspiciously slow.

As in, "the duration of a ping to the US and back" slow.

Yep, every badge in and out in the whole office had to go cross-continental (before the door unlocked).

[marginalia_nu]

Unrelated anecdote, I once worked in an office where the light switches were powered by the cloud. If you pressed the on and off button at the same time, there would be some sort of a race condition and you'd get blinking disco lighting as two threads presumably went

while (!bright) { increaseBrightnessBy10%(); sleep(100ms); }

and

while (!dark) { decreaseBrightnessBy10%(); sleep(100ms); }

Then someone had to log into a server and reboot a process before the blinking stopped.

[kevincox]

This is true and surprised me. Even with 100K employees and a separate 4096bit RSA key per badge that is only 50MiB of data. You would think that they would preload the readers with the access list or at least have a copy on site.

Of course it isn't a no-brainer:

- It is still a non-trivial amount of data for an embedded device.

- You want to be able to revoke quickly.

- You still want to log the access if offline (and how big should that buffer be?).

[mc32]

That should not be the case. Any existing user should be in the local cache. Most companies will have a controller in some control room somewhere which manages the door readers. New users yes. Former users should also need a connection. But existing users should not need a round-trip ping.

[MDRobidoux9507]

plumps phone

[asdff]

At what point is this an occupational safety hazard akin to blocking a fire door? Ridiculous that something like access to a building requires a network at all, but leave it to technology companies to overengineer a solution and underthink about the implications.

[coolspot]

To reconfigure misconfigured routers one needs physical access -> routers are in a high-security area of a secure DC -> physical access systems do not let anyone in due to network outage.

[mysterydip]

[conference room, some years ago, maybe]:

technician: "But if we authenticate using the facebook DNS, how will we do our jobs if it goes down?"

boss: "The only way our DNS servers are going down is if facebook closes, in which case you're out of a job anyway."

[leokennis]

Also possible:

technician: "But if we authenticate using the facebook DNS, how will we do our jobs if it goes down?"

boss: "That's a good point but not a blocker for go live, let's remember to add a story to our backlog to revisit this."

[ImBanned]

Now Facebook can see what it's like being banned for 30 days.

[Jcowell]

What I wonder if this isn’t a scheme on Facebook part to show the world what it feels like without it. At first you might think it’ll “wake up” the addicts and whatnot but you can’t just cut off an addict from their supply and expect them to just be ok with it. Instead they’ll relapse harder without help and might dig in deeper, never wanting to feel the withdraw again. I wonder if there’s a phenomenon called Social Withdrawn Symptoms.

[suyash]

You win the best comment award!! Or what is like to be banned indefinitely!

[footlose_3815]

Locking the doors to shred the evidence!

/s But seriously, such odd timing with Pandora Papers and with the Whistleblower.

[gmiller123456]

Seems odd. There is NO ONE that can let people in? Every building I've worked in has required badges, but there's always security inside that can check IDs and let people in. And methods of security getting in even if there's a power failure.

[deadalus]

massive whistleblower leak surfaces about Facebook

public opinion starts to change about fb

Zuck fears people will abandon his platforms

shuts all of the down and claims hax0rs

facebook goes down

divert the story to Russian and Chinese hackers

people instantly forget about whistleblower

facebooks stock balances out after a week

[loa_in_]

People wouldn't abandon Facebook et al. even if Zuckerberg was outed as a creature from outer space. Many people just wouldn't care.

[asdff]

FB is in the spy, stock is practically unsinkable at this point what with everyones retirement plan buying and holding it.

[suyash]

Most sophisticated and biggest hack. They really thought about all use-cases.

[EpicDavi]

I would say this is a bit misleading. Personally, I was able to access my FB building this morning (after the outage started) and have continued to be able to use my badge with no problem since. Not saying that there are no employees experiencing this issue, but it is not affecting all Facebook employees.

[nicoburns]

Apparently this issue is effecting specifically some of the people who are trying to fix the outage https://news.ycombinator.com/item?id=28749244

[agucova]

I imagine the authentication systems for the datacenters are more rigorous than most Facebook buildings, right?

[afavour]

In fairness the tweet doesn’t say all employees but the title here is ambiguous.

[hinkley]

Given it's a routing problem, doesn't it matter where your office is on the network with respect to the servers that control the door locks?

You could be in a zone of stability that's not affected by the overall chaos.

[echelon]

Honest questions!

Do you have any ethical qualms working for Facebook?

Do you feel the criticisms are blown out of proportion?

[eointierney]

What utterly wretched management at every level, inexcusably idiotic, and suggestive of so much worse about which we don't know.

That this company is one of the greatest revenue streams on the planet tells us everything about how badly we evaluate reality.

C'mon Humans, let's do better than this. You're challenged!

[sennight]

This isn't how badge readers work. Every controlled door is wired to a relay panel. That panel connects to a access control db - but it always maintains a cache that is very slow to expire... because the manufacturers weren't clueless morons.

[cududa]

Except some of the readers were made in house. I’ve never been into a Facebook conference room that didn’t use one of their custom touch screen/ card readers.

[kamaal]

>>cache that is very slow to expire...

People have been WFH/Remote working for a while now. Like 1.5+ years.

Besides when everything runs fine it all feels magic. You discover how things can break in a situation like this.

Im pretty sure they are now rushing to restore access for every FB employee to their buildings.

[cromka]

> Every controlled door is wired to a relay panel. That panel connects to a access control db

Absolutely not. E.g. HID is a well-documented protocol and you can do whatever you want using the RS232 interface they come with (depending on the model). I personally wrote a gateway that would dynamically and directly check the access with my own software.

[InsomniacL]

Lots of people drawing conclusions between recent leaks and some sort of attempt to cover them up.

I'd suggest more likely is a disgruntled engineer performing some sort of sabotage because of the content of the leaks.

[cblconfederate]

Internet of Things finally being useful

[katmannthree]

Can't help but feel that the world would be a slightly better place if it stayed this way.

[ferdowsi]

Agreed. The theoretical fat-fingering SRE who stopped all access to FB has probably done more to benefit humanity than any other single engineer in recent memory.

[Denvercoder9]

You should consider that WhatsApp and Facebook Messenger are the primary communication method for large parts of the globe. The world might be better off if we transitioned off them, but having them cease to exist without any warning most definitely won't make the world better off.

[rcurry]

Facebook said they are working hard to “get things back to normal”. But so far things seem more normal without them.

[MarcelOlsz]

My general anxiety is already lifting a little.

[MomoXenosaga]

Everyone in my country uses WhatsApp. It would be a mild inconvenience to switch to Telegram or Signal.

And I'm afraid I'm not naive enough to think that other companies wouldn't turn to the Dark Side. All of Silicon Valley, perhaps all of the tech industry is evil.

[ufmace]

Alas, if Facebook went down for good, they would probably be replaced by something even worse.

[ReptileMan]

You mistake Facebook with Twitter. Twitter dying will be a boon to humanity

[entropyneur]

Have you people considered like, not using it? I, for one, enjoy Facebook and am happy it exists even if some aspects are not to my liking.

[daRealDodo]

The market have never been more ripe for a new player.

[cbtacy]

"And sometimes the universe says, 'go get a different job already.'"

[MDRobidoux9507]

plump phone