Sounds like SolarWinds hired a good PR agency with the goal to deflect blame and make it sound like a big conspiracy. JetBrains being a Czech/Russian company makes it the perfect scape goat. As was pointed out, "solarwinds123" hints at very bad security practices which makes some unpatched system or weak password the much likelier scenario. It might well be that the intruders then manipulated the SolarWinds TeamCity config. Would have done the same with Jenkins...
And given that the malware version was signed using SolarWinds' keys, it's likely the attackers just built and deployed their compromised versions using the regular TeamCity CI servers. No need to exfiltrate the signing keys if you can get control of the CI/CD system. It's less likely to be logged, it has the same effect, and such servers regularly contact the outside world to download dependencies. And if the admin password to the CI/CD was "Solarwinds456!" or similar (as seems in character, if possibly overestimating their intelligence by adding "special characters") it wouldn't matter what CI/CD software they'd used.
On a semirelated note, what's with all the articles claiming the attack represents an extreme level of sophistication?
It sounds like the execution was skilled, but I haven't heard anything yet that seems technically novel or extraordinary.
Maybe I haven't read the right articles, but it sounds like solarwinds got pwned, and all these big targets loaded the malware onto their own networks...
Again, skillful execution, but it's not like they cracked an encryption algorithm or even did known-but-still-awesome exploits like rowhammer/spectre
I wouldn't be really shocked if this would turn out an Equifax-style attack (company negligence due to outdated software and lax security policies resulting to big advantages) combined with supply-chain attack (for customers of SolarWinds).
Exactly. As a developer that has used the CI&D infrastructure to finagle some things I didn't quite have permission to do, it doesn't surprise me that CI&D is a good way to escalate permission after the initial hack.
It is non-trivial to write secure deployment scripts/configs and ensure that access keys and credentials can't be leaked to anyone with dev access.
Reading between the lines, does this mean the attack may have simply been an on-prem install that was compromised, rather than every TeamCity install ever, or JetBrains' official SaaS version?
Any developer tools company worth their salt should be dogfooding their own build system, so they likely build IDEs with this tool. In the worst case, IntelliJ/GoLand/etc could be compromised as a result. This would be unlikely to mean there's malicious source code floating around, but it could mean lots of privileged access to software companies' networks. If the attacks are as targeted as the NY Times article makes it out to be, discovering the full extent of the damage may take quite some time ...
> SolarWinds confirmed Wednesday that it used TeamCity software to assist with the development of its software and was investigating the software as part of its investigation. The company said it had yet to confirm a definitive link between JetBrains and the breach and compromise of its own software.
This is a very big "may". Reads like they are simply basing these allegations on the Russian founders part.
it is unbelievable the irresponsibility of the NYT to not provide more substantive evidence before going to print with this "may be", particularly with this title. The reputational hit alone will cost Jetbrains millions including across unrelated products (which they encourage by fluffing out this piece with Jetbrains' customer list without regard as to whether or not they use TeamCity).
I suspect TC may have been leveraged but it doesn’t mean it’s responsible for every horrible misconfiguration that can occur.
Case in point, I have used TC to gain AD administrative privileges before because the idiot who set it up ran a build agent as a domain admin so it could get access to a locked down signing cert. I just created a new build to add me to the right group and ran it on that agent.
These things are really trivial to find and exploit. Also the build agents will obtain and run almost any untrusted software and leave it on disk quite happily for when a later build comes along.
Here's what we need to be thinking instead of getting defensive or extra-optimistic:
If high-end security firms and fairly diligent government agencies were infiltrated, why would we think that smaller dev toolchain organisations not founded as security organisations will somehow be less likely to be targeted and become a vector for introducing supply chain attacks. Sophisticated attackers will go after any soft underbelly or pore they can find, and there's no reason not to believe they'd put significant effort into quietly abusing Jetbrains security just like they did with Solarwinds. I'm less worried about the "Russian" red scare mention other than it may give non-US organizations a few more opportunities to inject badness that we can't get visibility on.
Bottom-line is that it would be the holy grail and are we treating it as the high priority target it is? Having worked for dev tool companies in the past, I know they are a lot more worried about innovation than about their own internal processes.
Does it seem odd to anyone else that the article begins with a big Russian flag? That seems to be implying some sort of official Russian involvement. To me it seems much more likely that Solarwinds just left the username and password as admin:admin, and any detail about Jetbrains are kinda irrelevant
@mods - I changed the title from the original "Russian Software Company May Be Entry Point for U.S. Hack" to be more clear, as "Russian Software Company" felt vague and linkbait-y to me. Apologies if this violates the "No editorialized titles" policy.
The teamcity application is a java application that can be easily analyzed for security threats. Nothing stops you from decompiling it and check the code. This story looks very much like a scapegoat for SolarWinds' poor security practices.
yeah.. and next breaking news - Microsoft and Google also may be entry points for U.S. Hack because Solarwinds used both Windows laptops and Android mobile phones.
I wish tech reporting is written by technical folks, or at least proof read by tech experts.
> I wish tech reporting is written by technical folks, or at least proof read by tech experts.
This comes up for journalism of every single industry.
“Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.”
Is there any published technical evidence to look at which actually implicates JetBrains? CVE or a POC? Publication of the poisoned build of the CI software? Indicators of compromise?
They probably already had access to the network and just used TeamCity since that was SolarWinds build process. Or someone really misconfigured things to allow external access.
What evidence do they have, or is it process of elimination, like source code built normal binaries locally, but malicious binaries when built in CI? Hope they have some VM images of the CI boxes for forensic analysis.
[+] [-] dr_faustus|5 years ago|reply
[+] [-] SAI_Peregrinus|5 years ago|reply
[+] [-] ahepp|5 years ago|reply
It sounds like the execution was skilled, but I haven't heard anything yet that seems technically novel or extraordinary.
Maybe I haven't read the right articles, but it sounds like solarwinds got pwned, and all these big targets loaded the malware onto their own networks...
Again, skillful execution, but it's not like they cracked an encryption algorithm or even did known-but-still-awesome exploits like rowhammer/spectre
[+] [-] zinekeller|5 years ago|reply
[+] [-] ryandvm|5 years ago|reply
It is non-trivial to write secure deployment scripts/configs and ensure that access keys and credentials can't be leaked to anyone with dev access.
[+] [-] gamesbrainiac|5 years ago|reply
[+] [-] vngzs|5 years ago|reply
Any developer tools company worth their salt should be dogfooding their own build system, so they likely build IDEs with this tool. In the worst case, IntelliJ/GoLand/etc could be compromised as a result. This would be unlikely to mean there's malicious source code floating around, but it could mean lots of privileged access to software companies' networks. If the attacks are as targeted as the NY Times article makes it out to be, discovering the full extent of the damage may take quite some time ...
[+] [-] foepys|5 years ago|reply
This is a very big "may". Reads like they are simply basing these allegations on the Russian founders part.
[+] [-] swyx|5 years ago|reply
[+] [-] uncledave|5 years ago|reply
Case in point, I have used TC to gain AD administrative privileges before because the idiot who set it up ran a build agent as a domain admin so it could get access to a locked down signing cert. I just created a new build to add me to the right group and ran it on that agent.
These things are really trivial to find and exploit. Also the build agents will obtain and run almost any untrusted software and leave it on disk quite happily for when a later build comes along.
[+] [-] sbelskie|5 years ago|reply
[+] [-] jen20|5 years ago|reply
[+] [-] thorax|5 years ago|reply
If high-end security firms and fairly diligent government agencies were infiltrated, why would we think that smaller dev toolchain organisations not founded as security organisations will somehow be less likely to be targeted and become a vector for introducing supply chain attacks. Sophisticated attackers will go after any soft underbelly or pore they can find, and there's no reason not to believe they'd put significant effort into quietly abusing Jetbrains security just like they did with Solarwinds. I'm less worried about the "Russian" red scare mention other than it may give non-US organizations a few more opportunities to inject badness that we can't get visibility on.
Bottom-line is that it would be the holy grail and are we treating it as the high priority target it is? Having worked for dev tool companies in the past, I know they are a lot more worried about innovation than about their own internal processes.
[+] [-] Rebelgecko|5 years ago|reply
[+] [-] emptyparadise|5 years ago|reply
[+] [-] ChefboyOG|5 years ago|reply
[+] [-] polka_haunts_us|5 years ago|reply
[+] [-] parhamn|5 years ago|reply
[1] https://twitter.com/nicoleperlroth/status/134690958021993676...
[+] [-] keyle|5 years ago|reply
[+] [-] MrRiddle|5 years ago|reply
[+] [-] nucatus|5 years ago|reply
[+] [-] RhodesianHunter|5 years ago|reply
[+] [-] guru4consulting|5 years ago|reply
I wish tech reporting is written by technical folks, or at least proof read by tech experts.
[+] [-] pfranz|5 years ago|reply
This comes up for journalism of every single industry.
“Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.”
― Michael Crichton
[+] [-] jrs235|5 years ago|reply
[+] [-] twistedpair|5 years ago|reply
[+] [-] throwawaybutwhy|5 years ago|reply
Disclosure: I own a bunch of JetBrains products and love them (even though sometimes there's a lag during typing).
[+] [-] lol768|5 years ago|reply
[+] [-] quaffapint|5 years ago|reply
[+] [-] twistedpair|5 years ago|reply
Really hoping I don't have to go back to Eclipse.
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] fortpoint|5 years ago|reply
[+] [-] konart|5 years ago|reply
[+] [-] tpmx|5 years ago|reply
[+] [-] koreanguy|5 years ago|reply
[deleted]