top | item 19793750

Hackers went undetected in Citrix’s internal network for six months

328 points| marcc | 6 years ago |techcrunch.com

118 comments

order
[+] benmarks|6 years ago|reply
> Citrix said in a later update on April 4 that the attack was likely a result of password spraying, which attackers use to breach accounts by brute-forcing from a list of commonly used passwords that aren’t protected with two-factor authentication.

How did Citrix not have 2FA in place?

[+] SCHiM|6 years ago|reply
Haven't read the article, don't know anything about their network. Assuming they use a Windows domain for their corp infrastructure.

Lower level Windows authentication mechanisms can't be configured for 2FA. If your active directory domain is functional at all then at the very least your systems need to be able to talk via SMB and ldap to a domain controller. With sufficient privileges you're able to execute code on other machines via either protocol.

You only need an infected machine, not even user credentials, to be able to perform password spraying or kerberoasting attacks.

[+] gwd|6 years ago|reply
They have had 2FA for years.
[+] todd3834|6 years ago|reply
I fully assume there are more hacks we don’t hear about that ones we do. Not only because of cover ups but it can’t be that hard to cover your tracks if you know what you are doing.
[+] Thorrez|6 years ago|reply
And it sounds like Citrix didn't even notice, it was the FBI who noticed then alerted Citrix.
[+] S_A_P|6 years ago|reply
THIS! My brother works for a large corp that does a lot of government(and private) work. A few years back, they tightened up their security with live monitoring, and as soon as it was enabled they realized that folks from China were actively connected. FBI was involved, but it never made the news. 2-3 more attempts have been made since. While they have an idea how long they had been breached, they don't know for sure...
[+] pandapower2|6 years ago|reply
Its an interesting question. If someone unauthorized was on your network exfiltrating data how would you know?
[+] erlangNewb|6 years ago|reply
Just assume they only catch the dumbest 20%.
[+] m3nu|6 years ago|reply
Security is hard. On the upside, every breach is a chance to learn for everyone else. I hope they release more details on how it happened.

Is there any blog or news that summarizes such post-mortem lessons? Could be a nice project to collect that.

[+] DiffEq|6 years ago|reply
It actually is not if you follow a strict Least Privileged model as a basis for your security architecture...But nobody does...not because it is hard, but because they don't understand it. Security is still based around looking for all the bad; it seems this defunct model will never die.
[+] robbiet480|6 years ago|reply
Has anyone gotten that kind of call from the FBI and can shed light on how the process works? Would be fascinating for a outsider and provide a guide on what next steps look like for those poor souls that receive the call in the future.
[+] 4s2A1tD5|6 years ago|reply
I've been on this call (both sides of it) probably a dozen times by now. Gov agencies are decent at doing research so it's pretty unlikely that the FBI just called their 1800 number or whatever.

Most small start ups don't get to the level where anyone that "big" is looking at them but in the event that something does get flagged the agency will go find their CEO/CTO/counsel on LinkedIn and either message them there or email them. I've never seen an actual vulnerability disclosed in email, if it's a potential legal issue (hello SEC and fintech) they may ask that your lawyer responds to them in writing but more often it's just "this is Agent XYZ with ABC. I have information about your company, please call me immediately."

For someone bigger (like Citrix) the company is hopefully big enough to have a team that is connected to the agencies in someway. Either the agency knows someone who knows them, or they have a designated Security and Compliance team that can handle these inquires.

The real problems come when you're in the middle of sizes - too big to have eyes on every email but too small to have a real security team.

About 5 years I was working for a SaaS company and one of our clients accidentally discovered a pretty serious hole in another company's product. This client wasn't overly tech savy and was basically like "hey is this how this is supposed to work?" when it very much was not... so we killed the API connection and told the client we'd take care of it. It's about 7pm ET by the time we figure out what's going on so we call and email the other company but couldn't find anyone. In the end we got the home phone number of their CTO and had our CTO call him at around 10pm. He thought it was a prank call but once our CTO convinced him this was a problem he was able to get their on call eng to patch it within hours.

Nowadays almost any company involved in security work either has a direct line to FBI/DHS or has a vendor who does. ie if I'm some medium consumer platform I probably don't get to talk to the FBI directly, but if I called up Crowdstrike or any security consulting firm they could do that. In the event that my medium consumer platform was infiltrated by Fancy Bear (and the government decided to tell me, sometimes they don't) an FBI agent would email/call the most likely point of contact for the fastest resolution without causing panic. Lots of time the damage is already done, two vs four hours on a response won't make a big difference in the long term so no need to email info@ or anything.

Over the past 6-8 years the corporation on public/private cyber investigations has definitely changed as red tape has decreased in sharing of info has increased - even more the last 4ish years since the DNC email hacks. I've had a clients get a casual "just a heads up, you should check this out" from the government without no paperwork and no follow up, something that would have been virtually unheard of 8 years ago.

DHS gets a lot of shit in the media (lots of which is deserved) but they've done a pretty good job just opening basic lines of communication and training other agencies that spending 20 minutes looking at a random tip, and following up if needed, is actually a pretty good use of time.

[+] rmason|6 years ago|reply
If you'd like a full perspective of the Citrix hack three security people from Detroit discussed it on a recent episode of their show, How they got hacked:

https://www.youtube.com/watch?v=fMgdrq0xMLk

[+] IncRnd|6 years ago|reply
Did you watch that? They mentioned that they don't know any more than is publicly disclosed how the attack occurred and that they were speculating. That was literally their first sentence about the attack.
[+] axaxs|6 years ago|reply
Having worked with Citrix, I'm shocked. Shocked that they detected it at all...
[+] da_chicken|6 years ago|reply
I was going to say the same thing, but it sounds like it was the FBI that noticed it:

> [T]he hackers had “intermittent access” to its internal network from October 13, 2018 until March 8, 2019, two days after the FBI alerted the company to the breach.

[+] rishabhd|6 years ago|reply
Having responded to multiple incidents across sectors, can confirm that extensive reconnaissance and long term operations are becoming a norm. During one instance, the attackers were present in the system of a client for more than 18 months and had gained an amazing understanding of their operational procedures, policies and security architecture to say the least.
[+] unmole|6 years ago|reply
You can't just make a statement like that without adding colourful anecdotes about your time there!

I'm especially interested because I had an offer from Citrix that I eventually turned down.

[+] empath75|6 years ago|reply
If you have anything of value, I absolutely guarantee you that there are hackers in your network right now.

One thing that frustrates me more than anything else is people assuming that their corporate network is safe. Your firewall and your vpc or whatever is a speed bump at best. You have to assume that you have an attacker on the desk right next to you, because you will eventually.

[+] viraptor|6 years ago|reply
That's a really defeatist attitude. There are different levels of "value" and different levels of protection. Not everything is internet facing. Not everything is managed like a corp where turnover requires lots of access changes. Not everything allows you persistence in the network. And not all access is "access".

I really wish we moved past the "everybody's owned" idea. Your defence should be proportional to the value you can lose. You can monitor for the rest. And you can't guarantee the are hackers in my network. (Unless you're saying you're guilty of breaking in? ;-) )

[+] Spooky23|6 years ago|reply
That phenomenon is worse in environments with lots of compliance, as the security people tend to think like auditors instead of security professionals.
[+] andyv|6 years ago|reply
... and if you don't have anything of value, they came, looked and left.
[+] ngcc_hk|6 years ago|reply
You need network sniffer and pattern recognition. Otherwise basically you hope some of the unusual activities will affect ids/ips (or touch internet). However if it is normal account you need some sort of intelligence to recognise and alert.

Not many software can do this.

[+] 6wKZhFkquv|6 years ago|reply
Throwaway, worked at Citrix. The unfortunate thing about this comment is that they sell Citrix Cloud as having the intelligence to detect anomalies exactly like this in your network.
[+] markholmes|6 years ago|reply
This might not be the right place for this, but where should one get started with security research?
[+] rando444|6 years ago|reply
I feel like an answer to this would depend largely on your age, background, and what you are looking to learn.
[+] inapis|6 years ago|reply
>Citrix said in a later update on April 4 that the attack was likely a result of password spraying, which attackers use to breach accounts by brute-forcing from a list of commonly used passwords that aren’t protected with two-factor authentication.

Wow. This simply reinforces the fact that humans cannot, and should not, be trusted with actively maintaining security of a system especially if there could be significant economic consequences.

Would a password manager help in this? I don't know.

Probably a hardware token which controls all and any access to a system.

*Removed some ambiguous sentences.

[+] Godel_unicode|6 years ago|reply
I was with you up until the last paragraph, but no. That's not 2fa, that's switching one factor for another.

People should use a password manager with an rng to generate and store passwords. IT departments should run password spraying attacks themselves as well as blacklisting known-compromised passwords. There's really good tooling for this (likely the same tooling this adversary used!)

Separately from this, people should use hardware 2fa tokens whose weakest link isn't the cell phone company support.

Edited for clarity.