top | item 34965260

LastPass says DevOps engineer’s hacked computer led to security breach in 2022

376 points| mikece | 3 years ago |9to5mac.com | reply

261 comments

order
[+] IncRnd|3 years ago|reply
I'm not sure the description is what actually happened. It doesn't have the ring of truth to it.

That said, LastPass is not deserving of any trust as a password product of any kind. That a password was captured by a keylogger on a Dev Ops home computer shows that they don't understand how to secure remote computers, the meaning of defense in depth, the importance of proper login authentication, or how to secure data at rest. Each of these points are close to the core of their business.

I don't wish them ill. I hope they recover from this, but they need to understand security to produce a security product.

[+] chii|3 years ago|reply
> I hope they recover from this, but they need to understand security to produce a security product.

it's because the market doesn't actually pay for a secure product, but the appearance of one.

The end-user buying cannot really discern whether the company's product is actually secure. There's no third-party standard auditing (let's say, a gov't organization).

Banks are run securely, not because I personally audit them, but that the gov't mandates liability onto the banks for losses from their insecure systems. So the banks are secure, because they stand to lose a lot. The same must be mandated for all companies imho, or insecure companies would continue to exist and thrive.

[+] chaorace|3 years ago|reply
While it's very civil of you to wish recovery upon LastPass, I don't really think the product is deserving of redemption. This is not the first major incident and it demonstrates little growth in relation to prior breaches. The world as a whole would probably be better off if LastPass were to breathe its last.
[+] deathanatos|3 years ago|reply
Yeah, because the description is inadequate. Is this BYOD? (… seems like not the employee's fault.) Is this the employee used the same password on the laptop and home, got credential stuffed, and LastPass isn't using MFA¹? (…seems like not the employee's fault.) Was there some jump from compromised home laptop to corp laptop? (The network is never to be trusted. …seems like not the employee's fault.)

The buck is supposed to stop at security, not at each employee's personal hygiene … if your game plan depends on the latter, it's game over.

There's more here than is being written, and I can only imagine because the truth probably stinks.

¹except TFA mentions MFA … but the mention of it doesn't really make sense.

[+] theamk|3 years ago|reply
Are there any reliable ways to secure remote computers from keyloggers _and_ still provide an efficient software development environment for non-trivial projects?

All of the software engineers I have seen have a fairly unrestricted environment -- Linux machines, with sudo access, often with passwordless root access via "docker" group, and with non-intrusive "endpoint protection" system. It would be normal to for someone to run "npm install" on their machine, or check out a random github repo they read about and run code from it.

Such machine would be a prime target for malware. And endpoint protection I have seen seems to be really stupid -- basically hooking "exec" calls and checking for exact hash match (!). Any serious malware should be able to bypass it without much effort, and if it only stays on a single computer, the detection chance is pretty low.

(I have also seen some poor souls who were stuck on locked-down Windows machines.. but they usually ended up using their machines as remote terminals, doing most their actual work on some remote server. And that server is sudo-capable Linux with light/no protection, and see previous paragraph. I suppose if _that_ is infected, at least Lastpass might not be stolen... unless people start browser on server and log into lastpass there, I've seen this happen)

[+] jstummbillig|3 years ago|reply
> That a password was captured by a keylogger on a Dev Ops home computer shows that they don't understand how to secure remote computers

I tend to disagree. The potential for any single employee to do substantial harm to any business is incredible and designing a system to make that not possible is nigh impossible.

It's neither the humans nor the institutions fault. It's just that systems involving humans are incredibly hard stuff. You are constantly weighing rigidity of the system with the potential to do harm within the system. How much way can it give to make it possible for humans to do their job in a complicated world?

If you go down the path of "we need a process for everything" you are going to end up with a lot of processes. The inherent problem with that approach is that (for most businesses that are not exactly amazon) a lot of processes can not not feasibly be systematically enforced and rely on being honoured by a human a juncture points, to an extent that makes you very uncomfortable when you consider what mechanisms you system has, for when they just don't.

As of now, most system simply rely on humans to do the right thing at the right time, for no other reason than it being the right thing and they also knowing that, for the whole world to not go up in flames.

[+] luxuryballs|3 years ago|reply
Even if they got pwned and the hackers got full system access like this the screw up is still definitely in the implementation right?

Isn't this like basic stuff where you would use encryption that only the end-user can bypass?

[+] Cthulhu_|3 years ago|reply
To me it says their security is not up to par, and that employees were allowed to take secrets home. Which is fine to a point, but it has to be a company managed, remote-wipable and locked down system if it's people that hold the literal keys to the secure castle.

Same with the Canadian ban on Tik Tok, why are they even allowed to have phones where they can install any software on?

BYOD and personalization is fine to a point, but only if you don't have high level access.

[+] LilBytes|3 years ago|reply
Am I assuming in this case the engineer was using his home PC to work?

This isn't unheard of in the industry, Engineers using BYOD devices or similar to work from home. But with a company with a risk profile as high as LastPass this seems _incredibly dumb_.

You would assume anyone with the keys to the kingdom was working on a company provided device, or any device that fits a compliance framework based on their own risk needs (which should be massive!).

E.g., endpoint management, AV/detection, FIDO2 with Yubikeys to access AWS as an admin and otherwise. Just a few off the top of my head.

[+] Johnny555|3 years ago|reply
My company isn't nearly as high profile or security focused and we're not allowed to use our own computers for any work related purposes,and our work laptops run threat detection software and we have a whitelist of software we're allowed to install.

I'm surprised that LastPass's policies aren't at least that strict.

My company has what I think is a big hole in this policy in that we're allowed to use our own phone for email, a few corporate apps (like Jira) and our corporate password manager (not LastPass), but IT doesn't do any management of phones (other than being able to wipe them remotely if you're connected to the company email server). I suspect that the company doesn't want to spend the money on giving everyone a managed phone.

[+] bboygravity|3 years ago|reply
> You would assume anyone with the keys to the kingdom was working on a company provided device, or any device that fits a compliance framework based on their own risk needs (which should be massive!).

You mean one of those company devices that is so locked down that they are close to impossible to work on?

Like if you need to install a new (part of) a toolchain, you need to go through IT which takes between 3 weeks and 6 months?

But your deadline is next week and your manager doesn't care about your IT "excuse", so just use notepad and CLI (or your own PC where you can do in 1 day what would otherwise take you months).

Those devices... yeah...

[+] SOLAR_FIELDS|3 years ago|reply
I could see this kind of sophisticated attack easily working on some random company with fairly lax BYOD policies. Makes sense, it was a rather sophisticated attack when you look at your typical medium sized company. But if your entire company and organization is built around keeping things secure, THIS is what brought you down? This isn’t getting owned by some unforeseen 0day, it’s just sloppy opsec.
[+] metadaemon|3 years ago|reply
I don’t think he’s necessarily working on his pc. He probably just had a shared LastPass account between work and his pc.
[+] scarface74|3 years ago|reply
This is completely unheard of for any company with any level of security. I’ve worked for 60 person startups that wouldn’t allow this.
[+] jhoelzel|3 years ago|reply
i completly agree. Its not so hard to do MFA all the way down either
[+] mcsniff|3 years ago|reply
So, with most password managers, when you authenticate on a new device, you are prompted for MFA.

The user had a keylogger installed on their machine, so the attacker could collect the master password, but how did they login to the vault on a new machine without MFA?

Did they get the MFA seed and login on a different machine, and nobody received a "You're using LastPass on a new machine, if this wasn't you..." message?

Did they exfil all that data to the compromised machine first, then out?

Unless I'm missing something, this doesn't seem right.

[+] lathiat|3 years ago|reply
They had code execution on the persons computer, the encrypted vault is downloaded and stored/cached on the computer - you only need the master password at that point to decrypt it. Or to read the decrypted version out of the process (e.g. your web browsers memory)

The 2FA part in the password managers (and least in the major players currently) is to get a copy of the encrypted vault from the server. The user did that part, and the encrypted vault was not easily accessible locally.

[+] forty|3 years ago|reply
I work for a LastPass competitor.

As far as I know, no popular password manager seriously includes "fully compromised local device" in their threat model. I don't think it can be done without hurting seriously usability (like having one 2fa verification each time you use a credential would work) and the predictable outcome of hurting usability too much is that people will find more handy insecure ways to store their passwords.

[+] throwawaypass|3 years ago|reply
Better trust nobody when it comes to security.

I'm using offline keypass. It's great.

[+] replaceusb|3 years ago|reply
https://pberba.github.io/security/2020/05/28/lastpass-phishi...

From what I can tell, all of lastpass mfa options are based around some form of otp not webauthn.

We tested the above in our own environment, since we had control of the devices we did not need urls to do it. We just grabbed the data locally to confirm if it was true. At the time lastpass told us webauthn was in the pipeline so we stayed.

[+] lrvick|3 years ago|reply
In my experience the majority of hacks are from a compromised laptop of a production engineer. Everyone blindly NPM installs away all their problems and no one checks signatures anymore. Most are using package managers like Brew that don't sign anything to begin with.

At Distrust, my security consulting firm, we train all our clients to build production systems that require a minimum of two engineers to mutate and that only pristine operating systems access production that have only signed reproducible used packages that have never been used for anything else.

Production environments need to be managed like careful methodical clean-room labs with strict accountability. Instead they are managed like collaborative art projects where everyone is trusted and nothing bad can happen.

[+] thomastjeffery|3 years ago|reply
This article is painfully light on the details. The blog post they linked is much more informative: https://support.lastpass.com/help/incident-2-additional-deta...

DevOps itself is a huge (and thin) attack surface. This is a feature of software as a centralized business venture.

If your company is storing user data, then it must be someone's job to do that. They need administrative server access to do that job.

The important takeaway is that the data itself can still be safe. So long as the company did not have a way to decrypt it, that data can rest anywhere. Guaranteeing that reality, however, is very difficult for a business - that is expected to be private about their implementations - to prove to its customers.

When this beach happened, LastPass should have focused on telling their users to never reuse the master password that they had set at that time. That's the biggest vulnerability: the content of their vaults (as copied by the attacker) was, and is, still kept behind that password. The need for each user to keep that specific password secret is the main effect of this situation.

This is a great reminder that you can't trust anyone to keep your data private. You can only trust math.

[+] technick|3 years ago|reply
Only 4 engineers had this level of privilege at lastpass, how did the attacker identify the target? Linkedin... that's why you should not list where you work until you're no longer working there or list a completely different role than what you're currently in.

I'm listed as a janitor of where I work. Only those that know me, know what I really do.

I've tried to sell the policy forbidding employees from listing their positions or where they work on linkedin, each time management frowns and says no. One day they'll come around...

[+] replaceusb|3 years ago|reply
Seems like a lot of their talk of zero knowledge was bs.

>In December, we notified a subset of customers whose SCIM, Enterprise API, and SAML keys were stored in unencrypted form. This only affected customers who joined LastPass and used these services in 2019 or before.

This part just blew my mind.

>Important: Since resetting MFA shared secrets destroys all LastPass sessions and trusted devices for these users, these users will need to log back in, go through location verification, and re-enable their respective MFA apps to continue using the service.

I feel sorry for everyones internal helpdesk. This is going to be brutal.

[+] cebert|3 years ago|reply
> “More specifically, the credentials for the servers were stolen from a DevOps engineer who had access to cloud storage at the company. This made it more difficult for LastPass to detect the suspicious activity.”

This comes off as spin to me by LastPass or LogMeIn’s PR department. Even if this was the case, how is it possible for intrusion detection systems to not observe and report abnormally high egress traffic? Downloading every vault should be a rather noticeable event.

[+] peanutz454|3 years ago|reply
I would expect that they'd setup something to notify them of - "abnormally high egress traffic downloading every vault", and yet due to alert fatigue, they never noticed. The specific thing with getting too many alerts is that you see them, but they aren't anyone's responsibility in general. The entire team gets them through email or sms or slack, and the new guys looks at them and wonders if we should do something about it like tweaking the criteria, and then gets other stuff to work on and learns to mostly ignore the alerts.
[+] mc32|3 years ago|reply
Maybe it was one of those odd and security-violating cases that give CISOs nightmares where they test by copying PROD to the SANDBOX.
[+] execveat|3 years ago|reply
Intrusion detection systems are utter shit and usually undergo even less real-world testing than recovery from a cold backup. Although we don't know LastPass'es architecture, it's also highly likely that with engineer's creds it was possible to exfiltrate database without any registered egress traffic at all.
[+] replaceusb|3 years ago|reply
It did, AWS alerted them on the traffic. It reads like they ignored it and when investigators later started going over that data it jumped up and slapped them.
[+] hinkley|3 years ago|reply
If they only took the vaults after a user accessed them, that would be less than double the traffic in the system. It depends how patient they were, or how broken the system was.
[+] unit_circle|3 years ago|reply
First of all: If you have the title of "DevOps" what you're doing is "Operations", you aren't practicing DevOps.

Anyway, this company has had incident after incident. This will keep happening every few years for them like it has for the past 10. As will lack of transparency/ outright lying.

Some commenters are saying they wish the company the best. I don't. Use something else. LastPass needs to die.

[+] dividedbyzero|3 years ago|reply
> First of all: If you have the title of "DevOps" what you're doing is "Operations", you aren't practicing DevOps.

So what title do you need to have to practice DevOps?

[+] j45|3 years ago|reply
We're reminded that cloud services are ultimately someone else's computer. Putting one's secrets someone else's computer under the marketing of convenience is just that.

This is not just to do with LastPass. It doesn't make sense why one would put their personal, most valuable passwords in the hand of another party. Of course, it's handy on a team between people.

When it comes to our bank accounts, do we trust someone like that with the pin numbers to our credit and bank cards?

One positive that comes out of this is that it raises awareness of thinking about the difference between security and convenience to manage one's own passwords.

One solution? Locating a zero, or no knowledge file storage system which is encrypted at rest and transit and placing your own files on it is the first start.

Spideroak used to fill that slot nicely (not sure if it's available anymore), and others I have heard about are sync.com and syncthing which do this just fine. Are there any other solutions that would be reasonably teachable to the average user?

[+] oceanplexian|3 years ago|reply
I really don't understand the point of these cloud password managers. Use something like KeepassX. Encrypt it with AES256, upload it to Dropbox, Google Drive, whatever. I personally throw the encrypted p/w file in an encrypted MacOS disk image with a secondary, separate memorized passphrase as well. Literally solves the problem without having to trust or pay some random sketchy service.
[+] JadoJodo|3 years ago|reply
> This was accomplished by targeting the DevOps engineer’s home computer and exploiting a vulnerable third-party media software package, which enabled remote code execution capability and allowed the threat actor to implant keylogger malware. The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault.
[+] thehappypm|3 years ago|reply
Bring your own computer for DevOps ?
[+] kc10|3 years ago|reply
I went through the pain of resetting hundreds of my passwords since the last breach during which they lost encrypted data. It was brutal, took several weeks and I had to spend late nights and weekends resetting passwords as a hobby project.

I am glad I went through all the pain.

[+] 8organicbits|3 years ago|reply
For anyone who still needs to do this, consider deleting accounts you no longer need. For some sites it's about the same effort to change a password as it is to delete an account (others... a lot more). The next time you need to bulk change passwords, you'll have fewer to change.
[+] skarz|3 years ago|reply
I think something that should have been in the title is that the breach was facilitated by a vulnerability in an application that many users here might have, Plex. They don’t speak on the nature of the vulnerability in that application. Has it been addressed and fixed through security patches already, or is Plex still potentially dangerous right now?
[+] ccooffee|3 years ago|reply
> Following the incident, LastPass has taken a number of steps to prevent future attacks and investigate what happened. The engineer was assisted in strengthening the security of their personal network [...]

I hope this involved something along the lines of: "This zoom meeting won't end until you update your router firmware".

[+] mekoka|3 years ago|reply
Nevermind the Plex vulnerability. Nevermind that the thieves are called here "hackers" and the theft, an "attack". It all makes it sound like more of a feat than it actually was. The bottom line for me is this: you outsource your security to a company whose sole reason for existing is to be more paranoid about it than you, and they tell you that what should be among their most treasured and garded secrets can casually be found in a coffee shop nearby.
[+] skeeter2020|3 years ago|reply
There's a weird combination of "what happened" and causation in here:

>> the credentials for the servers were stolen from a DevOps engineer who had access to cloud storage at the company. This made it more difficult for LastPass to detect the suspicious activity.

How does A => B?

>> The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault.

The part about authentication and MFA doesn't track with the rest of the sentence. How does a password without also having the MFA channel work? How would this give me access to a LP vault? Why were they on their home machine?

I understand it must be hard to try and come clean in a RCA without injecting some excuses and mitigating factors but you can't attempt soften the blow or the entire thing is a big, smelly mess. It should be a bunch of facts with no emotion, THEN the mitigation and lessons learned. LP just issued a bunch of press releases.

[+] secabeen|3 years ago|reply
> The part about authentication and MFA doesn't track with the rest of the sentence. How does a password without also having the MFA channel work? How would this give me access to a LP vault? Why were they on their home machine?

I think the idea here is that we want the cryptography operations to happen entirely locally, so that LastPass doesn't have any access to them. However, if you do that, someone with root on that system and the Master Password can replicate the operations the local system does on the vault. I'm not aware of any symmetric-encryption algorithm that includes a time-based un-replayable TOTP or HOTP in the key-generation process.

[+] hsbauauvhabzb|3 years ago|reply
Ironically, if they’d used a LastPass generated pass on plex, it would have been infeasible to crack the breached plex password. Using LastPass would literally have prevented a breach of LastPass.
[+] krakenschmidt|3 years ago|reply
No, they breached Plex with a vulnerability. Then they installed a keylogger. Edit: Is there a source about how exactly they got into Plex? I assumed it was via port scanning plus some Plex vulnerability, not that they guess the Plex password.
[+] osigurdson|3 years ago|reply
Welp, I guess it is totally not LastPass's fault then. Darn DevOps engineer should have known better.
[+] bgro|3 years ago|reply
Sometimes I wonder if I shouldn’t google the weather on my work computer because it might have some crazy exploit or cause an alert.

DevOps guy here is just running a Plex server and probably pirating porn on his work laptop because why not at that point. Can’t even bother updating this which Plex makes literally about as easy as they possibly could. It’s a one click fully automated process with zero interruption. Was he using internet explorer too?

Or was it his work laptop or did they just let him log in to work on any random computer?

Did he actually get any work done? I want to hear an interview from this guy. How did he get hired as DevOps while avoiding all the basics of fundamental basic work practices?

[+] jhoelzel|3 years ago|reply
Im not sure that they are just using this as a scapegoat but if your working from home, as a DevOps/Platform engineer, your very first ticket should be to activate MFA.

Kubernetes does MFA, all the Clouds do MFA and the company you work for can afford a "cheap android phone as key".

No matter if bare-metal, cloud or managed. If you habe ANY edit rights you need MFA.