_pplp's comments

I appreciate the perspective.

If you're telling me I worked there at literally the worst possible time frame, I'd believe it. I may have my experience skewed through the perspective of Nick's influence, but tbh many of my issues were unrelated to him or his sphere of influence.

The C level thing may not have been a "big" mystery, but it was to me, and as somebody who was running the dev of a flagship software product (UniFi) it set off alarm bells that nobody I talked to could explain who was handling the roles of those execs. I'm not exaggerating when I say I effectively got "I dunno" as a response when I inquired, and I dug.

It is good to know, though, that what I experienced wasn't chronic for the entire company's existence.

To clarify on the China thing, I wasn't trying to imply that anything nefarious was actually happening. Just that it warranted some scrutiny when a security focused product was being developed on the Chinese mainland and by a team of Chinese citizens that are subject to CCP laws. Given some of the things that have happened around that country's involvement in tech in recent years, I don't think such scrutiny is unwarranted, especially when the team has a track record of security "goofs".

Oh god don't remind me about Trace. I had to deal with the Controller side of that and it was a damn nightmare.

He basically dictated that you couldn't use any kind or repo+deployment pipeline except for what his team was building. Which wasn't actually functional for like 8 months. So we never even got a dev or staging tier to test against for months.

And then when I ended up with access to push things along, the actual apps for the trace system we're... not well implemented.

Ugh... I could bitch about this stuff for literal days but I gotta drop my kids off.

> Nick was hired out of his job at Amazon because he was supposed to be the AWS expert.

This always cracked me up. From what I can tell, he was a mid level dev on the Alexa web api team. He knew AWS sure, but he did not have the cred at all to justify the position and responsibility he was given at Ubiquiti.

I've since heard that the repo has been taken down and all the keys rotated, but just kinda makes you wonder how many APs and switches and cloud keys, etc are still out there using compromised keys.

Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).

Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.

But either way it definitely wasn't clear that opting out meant data was still collected. Super sketchy.

Hoo boy, this is gonna be a fun one. For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time.

> * Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams.

The "Cloud Lead" that Nick took over from gave zero fucks. He ran all the AWS stuff for Ubiquiti under his personal AWS account. Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle (my own personal opinion of Robert is... not the greatest).

One thing to understand about Ubiquiti (at least during those times) is that the company had zero C-level execs. There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.

So when Nick came in, a very... let's just say "forceful" personality, he immediately won over Robert and ended up with carte blanche over pretty much all of Ubiquiti's cloud accounts. Which were basically... everything. All the UniFi Network services, UniFi Protect services, you name it. If it was connected to the cloud in any way, Nick had access to it.

So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.

Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah.

> * Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted.

See above for the quality of security processes and practices this company had in place.

> * Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion

See above. (re: "god") Nick answered only to Robert. And he'd already successfully hoodwinked him. He could do whatever he wanted. Eventually he fell from Robert's good graces, but seeing as Ubiquiti as a company didn't really have a ton of checks and balances, he kept his god-level access far longer than he should've.

> * Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable.

Simple. Fear of share price falling. I was constantly given this as a reason we couldn't be transparent. Not by Robert, nor where he could hear. But it was pretty much well known that the company kept shit quiet for fear of the share price dipping.

> All the AWS configuration I'm speaking of above, I would describe as Security 101.

To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.

Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.

And that doesn't really even cover most of it. That year took a toll on my physical, mental, and emotional health, not to mention put a crazy strain on my marriage. I'd rather honestly forget it, but the schadenfreude of what's going on is too delicious to ignore.

> This man was a senior developer

Dude, let's not be generous. Could he write code? Yes. But this is a guy who wrote everything in Node, but absolutely _refused_ to use any existing libraries except for ones he personally wrote. He didn't "trust" them.

He wasn't even hired on as a dev, he was hired to be the "Cloud guy", essentially a sysadmin for AWS, and basically spooked the CEO into giving him the keys to the castle.

I mean, I didn't necessarily agree with all of his methods or reasonings on everything, but I've come to realize a lot of times his hands were just as tied as ours. And the draconian surveillance stuff? Yeah, he was directed to do that. One guess by whom.

He was "in charge" because he convinced Robert that he was the right guy for the job by finding a security flaw that let him log into Robert's personal UniFi Protect setup at his home. At that point Robert basically gave him carte blanche, but also started directing him to lock everything down. More than a bit of paranoia there, in my opinion.

Heh... no. I quit two years ago, well before all this happened. I have ideas about who this "Adam" is, and I also have some suspicions about who they're accusing as the culprit. But that's all they are. Hunches.

If I can vent for a second, this company has no leadership. None. Things may have changed in 2 years, but I doubt it. I was messaged almost daily by random employees asking wtf was going on with the company. They were afraid for their jobs. Practically no one respected the CEO, and he was the only C-suite exec. There. Was. No. Leadership.

There was no company wide communication, and all communication channels were made private, and if you sent an email to more than a couple people you were directly rebuked by the CEO. Nobody felt like they were trusted, and the norm was for most engineers to have absolutely zero idea of what was happening in the company outside of their direct project.

Teams were constantly at odds and pitted against each other, and the CEO never resolved any conflicts between teams or employees. The company (at least the software side) was treated like Thunderdome. Some team leads and office managers took care of their people, but most people were just beaten down. I don't think I'd ever seen a less motivated, more dejected group of software developers than I did during my time there.

IMO, this kind of bullshit clown show starts from the top. And as long as the top doesn't want to fix it, it won't get fixed. And since software almost invariable ends up reflecting the structure of the organization that produced it, you get this kind of security shit show.

I hope this is the last one and they get their act together. But realistically I can't believe that'll happen.

I'm pretty sure I'm safe. I left as soon as I could (almost 2 years ago) once I realized how institutionally broken the company was.

I mean, don't get me wrong, there absolutely _is_ somebody who's responsible for it, but I wouldn't place any money on Ubiquiti being able to figure out who it really was.

They want to brush this under the rug as fast as they can, and that means using the opportunity to pin it on somebody that's been "problematic".

> Ubiquiti also hinted it had an idea of who was behind the attack, saying it has “well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. As we are cooperating with law enforcement in an ongoing investigation, we cannot comment further.”

I personally don't believe this. IMO, this is a company who is looking for a fall guy, and _most likely_ it's going to be somebody who raised a stink about all the security problems during their time there.

Form your own opinion, I'm just a guy who worked at Ubiquiti for a year, raising all kinds of hell about the security, architectural, and operational problems that I saw while I was there.

But what do I know...

Greed. 100% greed. While I was there, the CEO loved to just fly between offices (randomly) on his private jet. You never knew where he'd pop up, and that put everybody on edge, because when he was unhappy he tended to fire people in large chunks (and shut down entire offices). Every decision was motivated by how it affected the stock price.

This is not surprising to me at all.

IMO, the CEO had a bit of a Steve Jobs hero-worship complex, but only all the bad parts. I can absolutely see him putting two teams on the same project, and "may the best product win".

The team that "lost" would get canned, obviously (I saw it happen to two separate offices while I was there).

> can we really trust them

absolutely not

I am 100% not surprised. I spent a year working for Ubiquiti, running the Network Controller team.

Trust me, this whistle-blower "Adam" (I have a few suspicions of who it actually is), toned it down.

The reality is much much worse.

Oh man you have no idea. I spent a year there running the Network Controller team.

"Shit show" doesn't even begin to describe it.