The affected repo has now been taken down, so I am writing this partly from memory, but I believe the scenario is:
1. An attacker had write access to the tj-actions/changed-files repo
2. The attacker chose to spoof a Renovate commit, in fact they spoofed the most recent commit in the same repo, which came from Renovate
3. Important: this spoofing of commits wasn't done to "trick" a maintainer into accepting any PR, instead it was just to obfuscate it a little. It was an orphan commit and not on top of main or any other branch
4. As you'd expect, the commit showed up as Unverified, although if we're being realistic, most people don't look at that or enforce signed commits only (the real bot signs its commits)
5. Kind of unrelated, but the "real" Renovate Bot - just like Dependabot presumably - then started proposing PRs to update the action, like it does any other outdated dependency
6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds
7. This incident has reminded that many people mistakenly assume that git tags are immutable, especially if they are in semver format. Although it's rare for such tags to be changed, they are not immutable by design
In recent years, it's started to feel like you can't trust third-party dependencies and extensions at all anymore. I no longer install npm packages that have more than a few transitive dependencies, and I've started to refrain from installing vscode or chrome extensions altogether.
Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies.
We need better capabilities.
E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
IMHO, just eliminating Internet access for all tools (e.g. in a power mode), might fix this.
The second problem is that we have merged CI and CD.
The production/release tokens should ideally not be on the same system as the ones doing regular CI.
More users need access to CI (especially in the public case) than CD.
For example, a similar one from a few months back https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj...
This is the death of fun. Like when you had to use SSL for buying things online.
Adding SSL was not bad, don't get me wrong. It's good that it's the default now. However. At one point it was sorta risky, and then it became required.
Like when your city becomes crime ridden enough that you have to lock your car when you go into the grocery store. Yeah you probably should have been locking it the whole time. what would it have really cost? But now you have to, because if you don't your car gets jacked. And that's not a great feeling.
Yes. Same with browser plugins. I've heard multiple free-plugin authors say they're receiving regular offers to purchase their projects. I'm sure some must take up the offer.
The original .NET (and I think Java?) had an idea in them of basically library level capability permissions.
That sort of idea seems increasingly like what we need because reputation based systems can be gamed too easily: i.e. there's no reason an action like this ever needed network access.
Are there examples of these types of actions in other circles outside of the .NET ecosystem? I knew about the FluentAssertions ordeal, but the Moq thing was news to me. I guess I've just missed it all.
Stealing crypto is so lucrative. So there is a huge 'market' for this stuff now that wasn't there before. Security is more important now than ever. I started sandboxing Emacs and python because I can't trust all the packages.
I hope the irony is not completely lost on the fine folks at semgrep that the admittedly "overkill" suggested semgrep solution is exactly the type of pattern that leads to this sort of vulnerability: that of executing arbitrary code that is modifiable completely outside of one's own control.
You should have never trusted them. That ecosystem is fine for hobbyists but for professional usage you can't just grab something random from the Internet and assume it's fine. Security or quality wise.
StepSecurity Harden-Runner detected this security incident by continuously monitoring outbound network calls from GitHub Actions workflows and generating a baseline of expected behaviors. When the compromised tj-actions/changed-files Action was executed, Harden-Runner flagged it due to an unexpected endpoint appearing in the network traffic—an anomaly that deviated from the established baseline. You can checkout the project here: https://github.com/step-security/harden-runner
The advertising in this article is making it actively difficult to figure out how to remediate this issue. The "recovery steps" section just says "start our 14 day free trial".
The security industry tolerates self-promotion only to the extent that the threat research benefits everyone.
It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub. I know they're auditable and you pin versions, but it's crazy to me that the recommended way to ssh to a server is to just give a random package from a random GitHub user your ssh keys, for example.
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
The crazier part is, people typically don't even pin versions! It's possible to list a commit hash, but usually people just use a tag or branch name, and those can easily be changed (and often are, e.g. `v3` being updated from `v3.5.1` to `v3.5.2`).
Having to use actions for ssh/rsync always rubbed me the wrong way. I’ve recently taken the time to remove those in favor of using the commands directly (which is fairly straightforward, but a bit awkward).
I think it’s a failure of GitHub Actions that these third party actions are so widespread. If you search “GitHub actions how to ssh” the first result should be a page in the official documentation, instead you’ll find tens of examples using third party actions.
So much this. I recently looked into using GitHub Actions but ended up using GitLab instead since it had official tools and good docs for my needs. My needs are simple. Even just little scripting would be better than having to use and audit some 3rd party repo with a lot more code and deps.
And if you're new, and the repo aptly named, you may not realize that the action is just some random repo
> It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub.
Right? My mental model of CI has always been "an automated sequence of commands in a well-defined environment". More or less an orchestrated series of bash scripts with extra sugar for reproducibility and parallelism.
Turning these into a web of vendor-locked, black-box "Actions" that someone else controls ... I dunno, it feels like a very pale imitation of the actual value of CI
I am surprised nobody here mentionned immutable github actions that are coming [1]. Been waiting for them since the issue was open in 2022. This would have significantly reduce impact and hopefully github will get it over the finish line.
I always fork my actions or at least use a commit hash.
Doing a bit of investigation with github_events in clickhouse, it is quite clear that the accounts used to perform the attack was "2ft2dKo28UazTZ", "mmvojwip" also seems suspicious:
It seems i forgot to cater for the quota applied to free "play" user in ClickHouse in my previous query... In fact, the threat actor did a lot more... this should give a better list of actions that was performed - Clearly showed he was testing his payload:
Note that these account seems to be deleted now - 2ft2dKo28UazTZ clearly did more than just changed-files and also seem to target coinbase/agentkit as well (Actually .. they might be targeted by the threat actor)
GitHub Actions should use a lockfile for dependencies. Without it, compromised Actions propagate instantly. While it'd still be an issue even with locking, it would slow down the rollout and reduce the impact.
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
This is hilarious, the maven-lockfile project "Lockfiles for Maven. Pin your dependencies. Build with integrity" appears to have auto-merged a PR for the compromised action commit. So the real renovate bot immediately took the exfiltration commit from the fake renovate bot and started auto-merging it into other projects:
> After some cleanup the changed-files (https://github.com/tj-actions/changed-files) action seems to be more work to remove. It would be awesome if it could be added to the allowlist
> Done. Allowed all versions of this action. Should I pin it to one version in the allowlist (won't be convenient if renovate updates this dependency)?
It seems pretty awful that the de-facto way to use GitHub Actions is using git tags which are not immutable. For example to checkout code [1]:
- uses: actions/checkout@v4
Github does advise people to harden their actions by referring to git commit hashes [2] but Github currently only supports SHA-1 as hashing algorithm. Creating collisions with this hashing algo will be more and more affordable and I'm afraid that we will see attacks using the hash collisions during my lifetime.
The repository is back online, with this explanation from the developer:
> This attack appears to have been conducted from a PAT token linked to @tj-actions-bot account to which "GitHub is not able to determine how this PAT was compromised."
> Account Security Enhancements
> * The password for the tj-actions-bot account has been updated.
> * Authentication has been upgraded to use a passkey for enhanced security.
> * The tj-actions-bot account role has been updated to ensure it has only the minimum necessary permissions.
> * GitHub proactively revoked the compromised Personal Access Token (PAT) and flagged the organization to prevent further exploitation.
This kind of auto dependency bump bots are more trouble than their worth. If your app works today, bumping random deps won’t make it work better in any meaningful sense in 95% of cases. With such a small upside, the downside of introducing larger attack surfaces, subtle breakages (despite semver), major breakages, and in the worst cases, compromises (whether it’s a compromised dep, or fake bot commits that people are trained to ignore) just completely outweighs the upside. You’re on the fast lane to compromises by using this kind of crap.
People should really learn from Go’s minimum version selection strategy.
I've always felt uncomfortable adding other people's actions to my GitHub workflows, and this is exactly the kind of thing I was worried about.
I tend to stick to the official GitHub ones (actions/setup-python etc) plus the https://github.com/pypa/gh-action-pypi-publish one because I trust the maintainers to have good security habits.
That's exactly where I stand, and I feel partially vindicated by this outcome. There are so many useful Github actions made by randos, but I am not adding more unvetted dependencies to my project. I will unhappily copy and paste some useful code into my project rather than relying upon yet another mutable dependency.
So this dumps env to stdout using some obfustucated code? And then relies on the fact logs are viewable publicly so the attacker can go scrape your secrets.
If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
I wish Github required some sort of immutability for actions by default as most package managers do, either by requiring reusable actions to be specified via commit hash or by preventing the code for a published tag to be changed.
At the moment the convention is to only specify the tag, which is not only a security issue as we see here, but may also cause workflows to break if an action author updates the action.
You can target `some/action@commithash` already, that's up to you. You're also free to fork or clone each action you use, vet the code, and consume your fork in your workflows. You can also disable the use of third party actions at an org level, or approve them on a case-by-case basis.
This all depends on your threat model and risk tolerance, it's not so much a GitHub problem. There will always be bad code that exists, especially on the largest open source code hosting platform. You defend against it because that's more realistic than trying to eradicate it.
I could have sworn that I've seen other GitHub Actions vulnerabilities that worked the same way, too. And/or HN submissions talking about this specific kind of vulnerability, the standard mitigation strategies, etc.
Feels like the same kind of problem as SQL injection, where everybody kinda knows about it and some people are actively aware and there are standard ways to avoid it but it still happens all the time anyway.
Might also be a good time to mention I'm really not a fan of YAML.
I wish Github didn't just purge entire repositories/accounts in the event like this, although I know why they do this. But there's now no way to analyze the repository/exploit anymore
Helpful update: The gist author has deleted the gist, so https://gist.githubusercontent.com/nikitastupin/30e525b776c4... now results in a 404, and stops the action from any further secrets being leaked. This means you're impacted only if you used the action, and had a build triggered in the last 6 hours or so.
[+] [-] rarkins|1 year ago|reply
The affected repo has now been taken down, so I am writing this partly from memory, but I believe the scenario is:
1. An attacker had write access to the tj-actions/changed-files repo
2. The attacker chose to spoof a Renovate commit, in fact they spoofed the most recent commit in the same repo, which came from Renovate
3. Important: this spoofing of commits wasn't done to "trick" a maintainer into accepting any PR, instead it was just to obfuscate it a little. It was an orphan commit and not on top of main or any other branch
4. As you'd expect, the commit showed up as Unverified, although if we're being realistic, most people don't look at that or enforce signed commits only (the real bot signs its commits)
5. Kind of unrelated, but the "real" Renovate Bot - just like Dependabot presumably - then started proposing PRs to update the action, like it does any other outdated dependency
6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds
7. This incident has reminded that many people mistakenly assume that git tags are immutable, especially if they are in semver format. Although it's rare for such tags to be changed, they are not immutable by design
[+] [-] mubou|1 year ago|reply
Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies.
Just take a look at eslint's dependency tree: https://npmgraph.js.org/?q=eslint
Can you really say you trust all of these?
[+] [-] ashishb|1 year ago|reply
We need better capabilities. E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
IMHO, just eliminating Internet access for all tools (e.g. in a power mode), might fix this.
The second problem is that we have merged CI and CD. The production/release tokens should ideally not be on the same system as the ones doing regular CI. More users need access to CI (especially in the public case) than CD. For example, a similar one from a few months back https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj...
[+] [-] from-nibly|1 year ago|reply
Adding SSL was not bad, don't get me wrong. It's good that it's the default now. However. At one point it was sorta risky, and then it became required.
Like when your city becomes crime ridden enough that you have to lock your car when you go into the grocery store. Yeah you probably should have been locking it the whole time. what would it have really cost? But now you have to, because if you don't your car gets jacked. And that's not a great feeling.
[+] [-] usef-|1 year ago|reply
[+] [-] mh-|1 year ago|reply
And if you turn on devDependencies (top right), it goes from 85 to 1263.
[+] [-] XorNot|1 year ago|reply
That sort of idea seems increasingly like what we need because reputation based systems can be gamed too easily: i.e. there's no reason an action like this ever needed network access.
[+] [-] scrapcode|1 year ago|reply
[+] [-] puffybuf|1 year ago|reply
[+] [-] ozim|1 year ago|reply
Abnormal behavior was to trust by default.
[+] [-] ycombiredd|1 year ago|reply
I hope the irony is not completely lost on the fine folks at semgrep that the admittedly "overkill" suggested semgrep solution is exactly the type of pattern that leads to this sort of vulnerability: that of executing arbitrary code that is modifiable completely outside of one's own control.
[+] [-] YZF|1 year ago|reply
[+] [-] nextts|1 year ago|reply
[+] [-] kurmiashish|1 year ago|reply
StepSecurity Harden-Runner detected this security incident by continuously monitoring outbound network calls from GitHub Actions workflows and generating a baseline of expected behaviors. When the compromised tj-actions/changed-files Action was executed, Harden-Runner flagged it due to an unexpected endpoint appearing in the network traffic—an anomaly that deviated from the established baseline. You can checkout the project here: https://github.com/step-security/harden-runner
[+] [-] cyrnel|1 year ago|reply
The security industry tolerates self-promotion only to the extent that the threat research benefits everyone.
[+] [-] shawabawa3|1 year ago|reply
[+] [-] harrisi|1 year ago|reply
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
[+] [-] remram|1 year ago|reply
[+] [-] sestep|1 year ago|reply
[+] [-] tommasoamici|1 year ago|reply
I think it’s a failure of GitHub Actions that these third party actions are so widespread. If you search “GitHub actions how to ssh” the first result should be a page in the official documentation, instead you’ll find tens of examples using third party actions.
[+] [-] kalaksi|1 year ago|reply
And if you're new, and the repo aptly named, you may not realize that the action is just some random repo
[+] [-] 12_throw_away|1 year ago|reply
Right? My mental model of CI has always been "an automated sequence of commands in a well-defined environment". More or less an orchestrated series of bash scripts with extra sugar for reproducibility and parallelism.
Turning these into a web of vendor-locked, black-box "Actions" that someone else controls ... I dunno, it feels like a very pale imitation of the actual value of CI
[+] [-] sieabahlpark|1 year ago|reply
[deleted]
[+] [-] Sytten|1 year ago|reply
I always fork my actions or at least use a commit hash.
[1] https://github.com/features/preview/immutable-actions
[+] [-] ebfe1|1 year ago|reply
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
Actions taken by the threat actor at the time can be seen here:
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
[+] [-] ebfe1|1 year ago|reply
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
[+] [-] ebfe1|1 year ago|reply
[+] [-] dan_manges|1 year ago|reply
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
[+] [-] themgt|1 year ago|reply
https://github.com/chains-project/maven-lockfile/pull/1111
[+] [-] sureIy|1 year ago|reply
[+] [-] mdaniel|1 year ago|reply
> After some cleanup the changed-files (https://github.com/tj-actions/changed-files) action seems to be more work to remove. It would be awesome if it could be added to the allowlist
> Done. Allowed all versions of this action. Should I pin it to one version in the allowlist (won't be convenient if renovate updates this dependency)?
[+] [-] onnimonni|1 year ago|reply
- uses: actions/checkout@v4
Github does advise people to harden their actions by referring to git commit hashes [2] but Github currently only supports SHA-1 as hashing algorithm. Creating collisions with this hashing algo will be more and more affordable and I'm afraid that we will see attacks using the hash collisions during my lifetime.
I wish that they will add support for SHA-256 soon and wrote product feedback regarding it here: https://github.com/orgs/community/discussions/154056
If this resonates with you please go and give it a thumbs up :)
[1]: https://github.com/actions/checkout?tab=readme-ov-file#usage
[2]: https://docs.github.com/en/actions/security-for-github-actio...
[+] [-] frenchtoast8|1 year ago|reply
> This attack appears to have been conducted from a PAT token linked to @tj-actions-bot account to which "GitHub is not able to determine how this PAT was compromised."
> Account Security Enhancements
> * The password for the tj-actions-bot account has been updated.
> * Authentication has been upgraded to use a passkey for enhanced security.
> * The tj-actions-bot account role has been updated to ensure it has only the minimum necessary permissions.
> * GitHub proactively revoked the compromised Personal Access Token (PAT) and flagged the organization to prevent further exploitation.
https://github.com/tj-actions/changed-files/issues/2464#issu...
[+] [-] oefrha|1 year ago|reply
This kind of auto dependency bump bots are more trouble than their worth. If your app works today, bumping random deps won’t make it work better in any meaningful sense in 95% of cases. With such a small upside, the downside of introducing larger attack surfaces, subtle breakages (despite semver), major breakages, and in the worst cases, compromises (whether it’s a compromised dep, or fake bot commits that people are trained to ignore) just completely outweighs the upside. You’re on the fast lane to compromises by using this kind of crap.
People should really learn from Go’s minimum version selection strategy.
[+] [-] lostmsu|1 year ago|reply
[+] [-] simonw|1 year ago|reply
I tend to stick to the official GitHub ones (actions/setup-python etc) plus the https://github.com/pypa/gh-action-pypi-publish one because I trust the maintainers to have good security habits.
[+] [-] 3eb7988a1663|1 year ago|reply
[+] [-] jasonthorsness|1 year ago|reply
https://news.ycombinator.com/item?id=43367987
[+] [-] londons_explore|1 year ago|reply
If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
[+] [-] cedws|1 year ago|reply
https://cedwards.xyz/github-actions-are-an-impending-securit...
[+] [-] random17|1 year ago|reply
At the moment the convention is to only specify the tag, which is not only a security issue as we see here, but may also cause workflows to break if an action author updates the action.
[+] [-] wutwutwat|1 year ago|reply
This all depends on your threat model and risk tolerance, it's not so much a GitHub problem. There will always be bad code that exists, especially on the largest open source code hosting platform. You defend against it because that's more realistic than trying to eradicate it.
[+] [-] mubou|1 year ago|reply
[+] [-] mirashii|1 year ago|reply
https://nvd.nist.gov/vuln/detail/CVE-2023-51664
[+] [-] zahlman|1 year ago|reply
Feels like the same kind of problem as SQL injection, where everybody kinda knows about it and some people are actively aware and there are standard ways to avoid it but it still happens all the time anyway.
Might also be a good time to mention I'm really not a fan of YAML.
[+] [-] hn8726|1 year ago|reply
[+] [-] captn3m0|1 year ago|reply
[+] [-] lmc|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] rahulr0609|1 year ago|reply
1] We took the public mirror from: https://code.forgejo.org/tj-actions/changed-files/src/tag/v4...
2] Undid the malicious code change: https://code.forgejo.org/tj-actions/changed-files/commit/0e5... - You can see the change here: https://github.com/trmlabs/changed-files/commit/8567847ee196...
3] Published under a v1 tag (since we can't vet historical releases and changes and didn't want folks to get confused)
If you want to contribute or report an issue, file a GH Issue or ping us at [email protected]