Probably worth considering here - if this was a state-sponsored attack, it's entirely possible (and exactly what I would do if I was running such a program) that there is one person or department actually writing the code and list messages etc, and another person or team who sits between that department and the internet and whose entire job is to ensure that everything that comes out from the development team takes place at actual times and with timestamps that are consistent with whatever story they're using for the project. Possibly even once in a while intentionally inserting a "mistake" to hint to investigators that the person's real location is another chosen place, which is also different from the real place where the project is being run.
It's likely that there is a whole team behind this operation, and part of that team might include tasks such as "socially engineer XZ project to accept a maintainer of our choice."
This was suggested in a different post[1], where there seem to be newly created accounts that came out of nowhere to create a sense of urgency, which might have accelerated the maintainer role transition.
I don't think he was from Eastern Europe, but if you want to look at UTC+0200/+0300, in Europe this only includes Finland, Baltics, Ukraine, Romania, Moldavia, and Greece. But notably if you look a bit down it also includes a good chunk of the Middle East, including Israel.
Not sure why you’re downvoted. When you think of state actors, Israel comes very high (stuxnet etc), as well as the usual US/Russia/China/NK groups.
Unit 8200 in the IDF especially have a very notable reputation.
That’s not to say other counties don’t have capabilities (and this doesn’t look like you need the resources of a group like say the NSA or GCHQ for this particular attack - indeed it could just be a single lone wolf) but it’s noteworthy.
“Trust no one! The minute God crapped out the third caveman, a conspiracy was hatched against one of them! ” - Gen. Hunter Gathers, OSI (The Venture Bros.)
There are parties who have effectively endless resources and motivation to mock up a false-flag event to steer the responsibility for this a certain way. They're all very, very good at covering their tracks, and even the most experienced security researchers will take months or years to just get an educated guess of who could be responsible. Trying to figure out who did this is a waste of time.
The lesson is this:
1. Never, ever install bleeding-edge software in production, for any reason.
2. Pentesters in your organization should be regularly trying to blow holes in your stack and let both you and package maintainers know the result. If they're not, they're not doing their jobs.
3. FOSS maintainers should audit the new code in each release for security issues, particularly for things that are obviously security-sensitive.
4. Donate to your FOSS maintainers; they do insanely important work.
These rules will pay off no matter who or what wants to break into systems.
This fortunately didn't get to stable. This was as close of a shave as you can possibly get without a security Chernobyl in your systems.
EDIT:
Downvote all you want; I'm right.
There are people who very much want to convince you and everyone you know that a certain party is responsible for this and every other attack you read about. They'll go out of their way to do that. There are geopolitical goals to be achieved by doing so.
We're currently talking about banning foreign ownership of an incredibly popular web app in the US right now. If you are a government or commercial entity that would benefit from such a law, how convenient would it be if there happened to be a GitHub user with a Chinese-sounding name who committed an attack vector to the project with a timestamp that looked like it could have come out of the PRC?
And let's say it is someone in mainland China. We find out they are beyond a shadow of a doubt and who they are on a personal level. Some big-time DA's office like the Southern District of New York puts out an indictment for them and requests extradition. The people who are behind this are almost certainly state-sponsored and there's no way their asses are being handed over to US Marshals, ever. You could even argue that if they managed to stumble into a situation where they could be extradited, the government responsible for backing them would "tie up the loose end" before letting that happen, lest greater knowledge of what the organization has been doing fall into enemy hands.
Besides the Bond-style intrigue, there's no practical application to the knowledge. You already know where your users are coming from, most of the time, and might be geo-blocking based on that alone. If not, you know you should be alert to other threats from that region. If you're not, you aren't doing your job.
GMT+8, the name (Mandarin/mixed-dialect first name + Hokkien surname) and most importantly the circumstance that the user was seen connecting from a likely VPN exit or hosted server in Singapore [1] are all consistent with a Singaporean identity, whether real or cover. For that reason, the opinion of [1]'s source that the name sounds fake should also be taken with some amount of salt (Mainland Chinese are not always well-informed about Diaspora Chinese communities). Worth noting that many of the holidays mentioned, likewise, are not public holidays in SG [2] - Jan 24th was, but Jan 22nd (listed as working) was a Sunday, inconsistent with a "working on regular working days" pattern most anywhere.
Would be interesting to see a greater corpus of Jia Tan's writings, especially older ones. There are fairly distinctive idiosyncrasies that may be used to distinguish Singaporeans, Mainland Chinese and speakers of various Slavic languages (and Anglo natives).
In the few commits from Jia Tan that I've seen, the English parses (to me at least) idiomatically as a native speaker from either the US or UK, and I did not encounter any signs of Singaporean English. Obviously, this is a very small sample so one cannot read too much into it.
Almost 10% of Singaporeans share the family name Tan. [1]
Setting aside the gravity of the situation, I'm 100% gripped by the "mystery novel" side of this. Seeing the internet piecing together the Who? the How? an the Why? is fascinating.
For years I've had `gc` in my terminal mapped to `TZ=UTC0 git commit` for exactly this reason. No need to change system times or git settings. (though perhaps a better way in global config?)
It's not even for nefarious reasons. I travel a lot, and don't like leaking my travel itinerary on public repos.
It not only sets the timezone to UTC, but also sets the time to 00:00. So my commits will only have date information. I think only saving the date permanently in the git commit is good enough. No need to leak the precise time I made commits.
Poor Jia. Two whole years of work down the drain. If you're reading this Jia, remember that you miss 100% of the shots you don't take. Chin up, brother.
The "hacker" stereotype is certainly stale, but subbing "young person" (and implying not "old person") is equally silly.
I'm 41 and have been awake before 7am maybe 20 times in the past 10 years. Half the reason I still put up with computers is because it's one of the few professions where I can work a 11-7pm schedule most days and still excel.
There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
No, stereotypes aside, hackers who code diligently tend to wrap their "actual work" schedule around their external obligations, and while the morning is generally filled with distractions like dealing with "morning people", after lunch (an on into the evening) there are generally far fewer interruptions. I can honestly get much more done with an uninterrupted four hours than I can in the entire eight office hours of random phone calls and emails coming in, and I am no longer a "young person", and I'll do some stuff at night just because I know it'll take half the time if I'm not interrupted and I've had a few hours to at least intermittently mull it over. If you're having peaceful mornings, I envy you.
> Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.
Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
> Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
Yes, and even worse:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
No, this is a difference of almost 10 hours...
Maybe the author inadvertently switched the two analyses? Or maybe the author is just not very good with timezone math... :)
Hi all! One of the authors of the original piece here. Thanks for all your comments – we picked up a lot of cool ideas for future analyses. Just to address four common objections:
- We agree it's unclear whether this was one person, or several ones. But even if it was a team, the timezone idea would still tell us where the team is located (the timings seem too much like regular office hours for a globally distributed team).
- It's also absolutely possible that commit times were changed, or commits were made/uploaded using timed scripts. Still seems like it would be hard in practice to use this to cover up a massive difference in timezone, though, because it would require adding a massive latency -- and xz wasn't developed in a vacuum, but in reaction to other online events like the filing of issues or mailing list posts.
- yes, we did get the two examples mixed up (the two times we say are 10-ish hours apart are actually only minutes apart, and the times we say are minutes apart, are 10-ish hours apart). Update is on the way.
- and finally, yes, UTC+2/3 goes beyond just Eastern Europe and includes part of the Middle East. We also clarified this in an update to the article.
Do you understand the time format there? It lists things as -700 or -800 for me, is that because of my local time zone (Chicago) vs. the mailing list (Finland?).
Here is the a distribution of commit times and days of week (hopefully I didn't mess up time zones and plot everything in UTC): https://imgur.com/a/1js5T8L
Something feels weird about the timezone arithmetic here, though maybe I am nuts. Take this for example:
> Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.
Putting it all into UTC we get:
23:38 - 8 = 15:38
17:27 - 3 = 14:27
That's not 9 hours by my numbers. Or another example:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
Putting it all into UTC we get:
21:53 - 3 = 18:53
17:00 - 8 = 09:00
I think these timezones were swapped (accidentally) by the article's author. If we do the calculation again:
I wonder what the "Lessons Learned" look like from the other side?
Are they whiteboarding the best strategy for distributing the timestamps of commits? Mandatory performance evaluations on all RCEs? Distribute the poisoned commits amongst more authors?
If you don't have your BIOS clock set to UTC and your OS is expecting it to be, you can get strange behavior like this. If they were switching between Windows and Linux, that could explain some of the offsets.
Shame on me for not reading more before my now removed comment. Shout-out for doing this level of analysis and guessing, as with everything in this incident, fascinating and tantalizing to wonder about.
It is interesting, at least one of the things listed as suspicious is something I do with some frequency. I will sometimes work on what is logically three commits and not chunk them out until the very end.
A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
> A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
I don't think much speculation is needed. RPM and deb catches every Enterprise Linux distribution, to the best of my knowledge.
rpm catches Red Hat and all derivatives like CentOS, Alma Linux etc, as well as SuSE, Amazon Linux, and even Microsoft's Mariner Linux (now renamed Azure Linux).
deb catches Debian and Ubuntu.
That's a huge global install base just through those two targets.
It would be interesting to see how the commit timestamps change around time change. If he has a job in a country that observes DST they would likely show an abrupt change.
[+] [-] ufmace|1 year ago|reply
[+] [-] omoikane|1 year ago|reply
This was suggested in a different post[1], where there seem to be newly created accounts that came out of nowhere to create a sense of urgency, which might have accelerated the maintainer role transition.
[1] https://connortumbleson.com/2024/03/31/watching-xz-unfold-fr...
via: https://news.ycombinator.com/item?id=39888854
Search for "Progress will not happen until there is new maintainer" for the relevant text.
[+] [-] bombcar|1 year ago|reply
Including carefully curating things to appear in a location and to NOT appear to be more than one person.
[+] [-] apantel|1 year ago|reply
[+] [-] mik1998|1 year ago|reply
[+] [-] lrasinen|1 year ago|reply
[+] [-] jamebond|1 year ago|reply
Seems to check out to me.
[+] [-] bradleyjg|1 year ago|reply
[+] [-] chgs|1 year ago|reply
Unit 8200 in the IDF especially have a very notable reputation.
That’s not to say other counties don’t have capabilities (and this doesn’t look like you need the resources of a group like say the NSA or GCHQ for this particular attack - indeed it could just be a single lone wolf) but it’s noteworthy.
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] lenerdenator|1 year ago|reply
There are parties who have effectively endless resources and motivation to mock up a false-flag event to steer the responsibility for this a certain way. They're all very, very good at covering their tracks, and even the most experienced security researchers will take months or years to just get an educated guess of who could be responsible. Trying to figure out who did this is a waste of time.
The lesson is this:
1. Never, ever install bleeding-edge software in production, for any reason.
2. Pentesters in your organization should be regularly trying to blow holes in your stack and let both you and package maintainers know the result. If they're not, they're not doing their jobs.
3. FOSS maintainers should audit the new code in each release for security issues, particularly for things that are obviously security-sensitive.
4. Donate to your FOSS maintainers; they do insanely important work.
These rules will pay off no matter who or what wants to break into systems.
This fortunately didn't get to stable. This was as close of a shave as you can possibly get without a security Chernobyl in your systems.
EDIT:
Downvote all you want; I'm right.
There are people who very much want to convince you and everyone you know that a certain party is responsible for this and every other attack you read about. They'll go out of their way to do that. There are geopolitical goals to be achieved by doing so.
We're currently talking about banning foreign ownership of an incredibly popular web app in the US right now. If you are a government or commercial entity that would benefit from such a law, how convenient would it be if there happened to be a GitHub user with a Chinese-sounding name who committed an attack vector to the project with a timestamp that looked like it could have come out of the PRC?
And let's say it is someone in mainland China. We find out they are beyond a shadow of a doubt and who they are on a personal level. Some big-time DA's office like the Southern District of New York puts out an indictment for them and requests extradition. The people who are behind this are almost certainly state-sponsored and there's no way their asses are being handed over to US Marshals, ever. You could even argue that if they managed to stumble into a situation where they could be extradited, the government responsible for backing them would "tie up the loose end" before letting that happen, lest greater knowledge of what the organization has been doing fall into enemy hands.
Besides the Bond-style intrigue, there's no practical application to the knowledge. You already know where your users are coming from, most of the time, and might be geo-blocking based on that alone. If not, you know you should be alert to other threats from that region. If you're not, you aren't doing your job.
[+] [-] 4bpp|1 year ago|reply
Would be interesting to see a greater corpus of Jia Tan's writings, especially older ones. There are fairly distinctive idiosyncrasies that may be used to distinguish Singaporeans, Mainland Chinese and speakers of various Slavic languages (and Anglo natives).
[1] https://boehs.org/node/everything-i-know-about-the-xz-backdo...
[2] https://www.mom.gov.sg/employment-practices/public-holidays
[+] [-] qsi|1 year ago|reply
Almost 10% of Singaporeans share the family name Tan. [1]
[1] https://en.wikipedia.org/wiki/File:Singaporean_surnames_by_f...
[+] [-] gfaure|1 year ago|reply
[+] [-] throwawayyy9237|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] BluSyn|1 year ago|reply
It's not even for nefarious reasons. I travel a lot, and don't like leaking my travel itinerary on public repos.
[+] [-] SmartHypercube|1 year ago|reply
It not only sets the timezone to UTC, but also sets the time to 00:00. So my commits will only have date information. I think only saving the date permanently in the git commit is good enough. No need to leak the precise time I made commits.
[+] [-] nerdponx|1 year ago|reply
[+] [-] sneak|1 year ago|reply
[+] [-] optimalsolver|1 year ago|reply
[+] [-] sva_|1 year ago|reply
[+] [-] tiahura|1 year ago|reply
[+] [-] beanjuiceII|1 year ago|reply
[+] [-] est|1 year ago|reply
But I am sure JiaT didn't only work on the xz project?
[+] [-] lapcat|1 year ago|reply
Me
> For a hacker, it is much more plausible to work in the afternoon and late at night
c/hacker/younger person
[+] [-] jtuple|1 year ago|reply
I'm 41 and have been awake before 7am maybe 20 times in the past 10 years. Half the reason I still put up with computers is because it's one of the few professions where I can work a 11-7pm schedule most days and still excel.
There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
[+] [-] lucb1e|1 year ago|reply
What is this syntax? I know s/a/b/ for sed substitution but not such a thing starting with c
[+] [-] evilDagmar|1 year ago|reply
[+] [-] qiqitori|1 year ago|reply
Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
[+] [-] someplaceguy|1 year ago|reply
Yes, and even worse:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
No, this is a difference of almost 10 hours...
Maybe the author inadvertently switched the two analyses? Or maybe the author is just not very good with timezone math... :)
[+] [-] tete|1 year ago|reply
If I would be pretending something I'd certainly put at least one more indirection in.
[+] [-] rvba|1 year ago|reply
In such setup nothing bad happens in such a leak.
[+] [-] sihe|1 year ago|reply
- We agree it's unclear whether this was one person, or several ones. But even if it was a team, the timezone idea would still tell us where the team is located (the timings seem too much like regular office hours for a globally distributed team).
- It's also absolutely possible that commit times were changed, or commits were made/uploaded using timed scripts. Still seems like it would be hard in practice to use this to cover up a massive difference in timezone, though, because it would require adding a massive latency -- and xz wasn't developed in a vacuum, but in reaction to other online events like the filing of issues or mailing list posts.
- yes, we did get the two examples mixed up (the two times we say are 10-ish hours apart are actually only minutes apart, and the times we say are minutes apart, are 10-ish hours apart). Update is on the way.
- and finally, yes, UTC+2/3 goes beyond just Eastern Europe and includes part of the Middle East. We also clarified this in an update to the article.
[+] [-] broknbottle|1 year ago|reply
https://www.mail-archive.com/[email protected]/
[+] [-] cozzyd|1 year ago|reply
[+] [-] widdma|1 year ago|reply
DST varies by state in Australia. Western Australia (UTC+8) does not observe DST.
[+] [-] dc-programmer|1 year ago|reply
[+] [-] cozzyd|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] crem|1 year ago|reply
~~It has surprisingly many commits on Sundays.~~
UPD. Weekday 0 is actually Monday, not Sunday.
[+] [-] idlephysicist|1 year ago|reply
> Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.
Putting it all into UTC we get:
That's not 9 hours by my numbers. Or another example:> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
Putting it all into UTC we get:
I think these timezones were swapped (accidentally) by the article's author. If we do the calculation again: That _is_ a matter of minutes.[+] [-] sihe|1 year ago|reply
[+] [-] fbdab103|1 year ago|reply
Are they whiteboarding the best strategy for distributing the timestamps of commits? Mandatory performance evaluations on all RCEs? Distribute the poisoned commits amongst more authors?
[+] [-] fullstop|1 year ago|reply
[+] [-] juitpykyk|1 year ago|reply
[+] [-] k8svet|1 year ago|reply
It is interesting, at least one of the things listed as suspicious is something I do with some frequency. I will sometimes work on what is logically three commits and not chunk them out until the very end.
A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
[+] [-] Twirrim|1 year ago|reply
I don't think much speculation is needed. RPM and deb catches every Enterprise Linux distribution, to the best of my knowledge.
rpm catches Red Hat and all derivatives like CentOS, Alma Linux etc, as well as SuSE, Amazon Linux, and even Microsoft's Mariner Linux (now renamed Azure Linux).
deb catches Debian and Ubuntu.
That's a huge global install base just through those two targets.
[+] [-] Dibby053|1 year ago|reply