This is a cat and mouse game. We add code to detect and disable abuse – sometimes in very clever ways – and then the abusers come up with a new way of circumventing that detection. In order to prevent miners from creating long queues for legitimate free users of GitHub Actions, we have to stay on top of this all the time. So the miners are not just stealing CPU time, they are also stealing engineer time. Because without mitigations the miners will consume all available CPU, and because devising abuse countermeasures is, for whatever reason, a very powerful nerd snipe (including for me!). The sad thing is that it's displacing time that would be spent improving Actions in other ways.
Actions are awesome... but scary as soon as you have a public repos, contractor, rogue employee, etc. They seem to go against security fundamentals. Ex: Actions should allow going into default-deny mode for all basic runtime capabilities and resource use, and only brought back on via RBAC. Today, it's not hard to steal npm/pip/etc creds or get into people's corp runners. Having gone through the browser security policy heyday, this is deja vu, except now for exposing the server side and supply chain.
Ex:
- do not run on any event.. unless user authorized for that event. Same for actions.
- separate out policies and users cannot edit policies unless authorized to do that
- do not get physical/logical resources (runners, disk quota, long runs, ...) unless given
- default-deny network outbound with url safe-listing
That way only trusted users can run them, and a bit harder for them to get hurt when there is a surprising action that they run
The next level would probably be something like sandboxing : allow anyone to run an action , but a sandbox mode can autofail if violated, and have explicit imports/exports to lock down for how it gets used.
A lot possible.. but need to invest in the basics first..
Do I understand correctly that the attacker forks a repository with GitHub Actions enabled, modifies the action, submits a PR, which makes GitHub run the altered action?
If so, I wonder if there is a legit need for running modified GitHub actions from non-collaborators?
Could also subject modified actions coming in via pull requests (from non-collaborators) to heavy resources constraints and timeouts.
I wonder if it would help forming an internal red team that could help stay a step ahead with such and related attacks and abuse scenarios by running such attacks against yourself?
limit the action configurations file to be only editable by a configured set for a specific repo and give us a special folder like .github-action-commands or so that is scoped like that aswell..
I was mostly ambivalent about crypto - it seemed like a lot of speculation but I liked some scenarios it enabled (mostly scenarios where traditional payment was not practical for w/e reason).
But seeing the HW situation, energy burn, scammers, infra hijacking, etc. there are so many negatives I'm more and more in favour of making it illegal.
If that's your critic then Proof-of-Work is the issue, not crypto currencies in general. There are multiple Proof-of-Stake implementations already in production (still very young though), each with its set of properties (see Cardano, Algorand, Tezos, Polkadot, Harmony, etc).
That is a very simplistic, and incorrect, response to the problem.
GitHub provides computing resources, for free, to attract users. This is just one of the challenges of that business model. If this is intolerable for GitHub then it's up to them to find a strategy to counter it.
We don't need to ask the government to punish everyone participating in cryptocurrency.
Since the downsides of proof-of-work get worse based on market price, instead of making it illegal, governments could try to crash it. A 10% wealth tax on holding proof-of-work cryptocurrency would encourage people and businesses who don’t cheat on their taxes to sell.
This problem will resolve itself in 5-10 years when people move to proof-of-stake blockchains. There will be no more demand for proof-of-work cryptocurrencies and attacks like this are no longer economically interesting.
The problem is what is illegal then? Numbers? Cryptography? Because nobody buy sell or own crypto technically, there just happen to be transactions on a distributed network, with some people knowing private keys to generate new transactions. It could be possible to regulate exchanges dealing with fiat, but I feel like the regulable surface area is thin. Also I hope Proof of stake take over so that a big part of these issues will be non existent in a few years.
Like making drugs and alcohol illigal. Declaring something illigal doesn't stop it. It still has value - therefore it will continue on without you, and then only those that deal illigally get the benefits of crypto.
As long as the proposed outlawing is along lines of taxation to make legitimate holdings infeasible, rather than establishing some arbitrary # of SHA256 hashes I am legally allowed to compute per unit time.
The SEC has always been in place supposedly to protect the average citizen from being able to make bad investments - however I don't understand how they dropped the ball so hard with crypto-"currencies;" regulatory capture by VC-finance industrial complex?
Is it that they are global and decentralized that made them just not take a stance on them, also the same reason it obviously would easily and quickly gain popularity as a global, decentralized MLM-Ponzi structure?
Cryptomining is a waste of energy and compute resources. Imagine such an effort being put into protein folding, or finding a cure against rare diseases instead.
What makes it worse is that a tax, higher energy or hardware costs, or increased transaction fees only speed up crypotocurrency inflation relative to other currencies. It's hard for me to comprehend, that others don't see the pyramid scheme behind it.
We’ve been dealing with mining at Replit since the beginning and have gotten really good at handling it. Early on they nearly bankrupt us. There was one month when our compute bill was 500% more than average. So we just had build tooling to catch it and handle it early. It’s a fun problem because it’s not only technical but also psychological.
Mining is one vector of abuse but there are many others when you’re giving free compute to the world. Especially in the radically open way we’re doing it at Replit.
This is pretty much par for the course if you allow others to run arbitrary code. I'd bet that Github deals with many cases of people trying to do this, and the only thing different here is a security researcher found it and talked to a reporter.
When I worked at ZEIT (Vercel) this was a huge issue we'd face. There was one person on GitHub that was shamelessly encouraging others to set up a docker image that ran crypto mining processes.
This is inevitable, and I'm sure every CI system has faced this issue.
I think it could be solved by requiring maintainers to kick off the CI job for every PR coming from a first-time contributor manually. I would not be annoyed by this provided that regular contributors don’t have to go through this. I think GH can further reduce the impact of this by not applying the limitation to first-time contributors who have lots of followers, forks, PR reviews, commits in the last year.
Also, edits to the CI script could be made suspect. I never had a first-time contributor on my projects start by making changes to the CI pipeline.
I am pretty sure smart folks at GH thought of this and just deliberating whether to introduce such a breaking flow to maintainers.
I think that’d definitely help. The Kubernetes project requires a maintainer to reply with /ok-to-test for the CI bot to begin testing a PR. This also helps reduces load on their CI systems, as the project gets a lot of commits.
What's to stop attackers from making a one-off harmless edit ("forgot a comma in readme") and then, once they're whitelisted, deploying a malicious executable to the CI pipeline.
I think the root issue is that people without write access to your repo can queue arbitrary compute on your dime by simply creating a PR and changing the GitHub workflow files (the definition for GitHub actions). This is even a bigger issue for companies with self-hosted runners who can't use those for public repos as an attacker could file a PR with malicious code and compromise a machine (https://docs.github.com/en/actions/hosting-your-own-runners/...)
One possible solution here is that a maintainer needs to okay any change that changes a workflow file before it runs. Not sure if that introduces other problems...
Having the maintainer trigger the CI job moves the problem elsewhere. Attackers will create new repositories (or take over existing ones) and create the PRs, for which they will trigger the CI jobs.
The more apparent problem is that CI jobs can execute arbitrary code and are not limited wrt. their execution time. If limited, it would render them useless when used for cryptomining.
In this specific case, a simple rule change should suffice: CI code committed for the first time does not run until after the PR closes. These guys are exploiting a loophole where you can add CI code and it runs whenever you initially open the PR. Requiring the PR to be closed before the code runs would solve this.
This has been a frustrating thing for us on GitHub , not because of crypto, but for stealing credentials. It's hard to make it so only approved people can run actions and lock down what exposed actions' runtimes can do.
Until it is default-deny for ~all capabilities with say RBAC for enabling, which are basic security principles, they're pretty scary for public repos. It's even scary for private ones as you might want say a contractor or intern to be limited. That they have people publicly dedicated to whackamole response, but seemingly not to security fundamentals (or if they do, not following them / empowered to), is frustrating. Look at the GHA permissions panel and then think like an attacker or defender, it's scary.
The environment variable stuff awhile back was understandable.. but not the big issue. As basically any user can submit a PR that can make current actions that run code (ex: run tests) to run something else by editing the action or code, that means any public user can burn repo $, play in repo-exposed runners (ex: get into corp sandboxes), and if using continuous deployment or service integrations , into their production systems, wherever they publish packages, etc. GitHub is api exposed and git can have commits deleted, so they can even cover most of their tracks. This is SolarWinds all over again, but now a bot can run it automatically on ~everyone!
GHA is both one of my favorite things about GitHub but also been scary. Maybe now that they have access to some. of the best security engineers and researchers in the world, they can fix this...
We are one step closer to having GitHub Actions limited or disabled entirely on free accounts. Really soon it is going to be available on paid plans and for open-source after an approval.
Wouldn't it be great if Crypto was just a giant honey pot. With the intent of creating cyber criminals who think they can get rich with crypto money but instead just end up putting tons of effort into mining coins rather than doing other simpler crimes.
I think if half the effort spent here was spent learning finance, the bad actors would have more money.
I asked this question previously[0] but why are random PRs ran by CIs by default without user intervention? It would make more sense that by default PRs aren't ran by default and an explicit whitelist or configuration option needs to be set.
[0] in a discussion about the dangers of running your own CI server. I thought it were sensible if there was a permission config file which specified who can trigger which actions, where the CI uses the permissions of the last ran commit.
Following the files used reveals the pool and wallet used. The pool lets you look up the stats for this wallet. For this wallet, it says:
> Total Paid: 163000 TRTL - 42.16$
But I guess the culprit is rotating wallets. Judging by the hash rate, the pool estimates a weekly income of 700$. It could be a multiple of this if the culprit uses multiple wallets at once.
If I were them I'd mine unpopular coins hoping that they go up in value during a general bull run in the future. That way you're quite likely to end up with a decent number of coins.
[+] [-] natfriedman|5 years ago|reply
[+] [-] lmeyerov|5 years ago|reply
Ex:
- do not run on any event.. unless user authorized for that event. Same for actions.
- separate out policies and users cannot edit policies unless authorized to do that
- do not get physical/logical resources (runners, disk quota, long runs, ...) unless given
- default-deny network outbound with url safe-listing
That way only trusted users can run them, and a bit harder for them to get hurt when there is a surprising action that they run
The next level would probably be something like sandboxing : allow anyone to run an action , but a sandbox mode can autofail if violated, and have explicit imports/exports to lock down for how it gets used.
A lot possible.. but need to invest in the basics first..
[+] [-] sorbits|5 years ago|reply
If so, I wonder if there is a legit need for running modified GitHub actions from non-collaborators?
Could also subject modified actions coming in via pull requests (from non-collaborators) to heavy resources constraints and timeouts.
[+] [-] anoojb|5 years ago|reply
Could temporarily reduce the population of abusers while we figure out a more sustainable strategy?
[+] [-] kerng|5 years ago|reply
[+] [-] bko|5 years ago|reply
[+] [-] merb|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] reader_mode|5 years ago|reply
But seeing the HW situation, energy burn, scammers, infra hijacking, etc. there are so many negatives I'm more and more in favour of making it illegal.
[+] [-] collsni|5 years ago|reply
Just realize what you're asking for.
[+] [-] Tenoke|5 years ago|reply
Sadly scammers follow both new and old tech and it takes a while for measure to catch up. That's not necessarily a fault of the tech itself.
[+] [-] dgellow|5 years ago|reply
[+] [-] gburnett|5 years ago|reply
GitHub provides computing resources, for free, to attract users. This is just one of the challenges of that business model. If this is intolerable for GitHub then it's up to them to find a strategy to counter it.
We don't need to ask the government to punish everyone participating in cryptocurrency.
[+] [-] skybrian|5 years ago|reply
[+] [-] miohtama|5 years ago|reply
[+] [-] tarsinge|5 years ago|reply
[+] [-] devwastaken|5 years ago|reply
[+] [-] bob1029|5 years ago|reply
[+] [-] shawnz|5 years ago|reply
[+] [-] rodiger|5 years ago|reply
As others said, the answer here is Proof of Stake, not banning crypto outright.
[+] [-] marsrover|5 years ago|reply
[+] [-] loceng|5 years ago|reply
Is it that they are global and decentralized that made them just not take a stance on them, also the same reason it obviously would easily and quickly gain popularity as a global, decentralized MLM-Ponzi structure?
[+] [-] DelightOne|5 years ago|reply
[+] [-] de6u99er|5 years ago|reply
What makes it worse is that a tax, higher energy or hardware costs, or increased transaction fees only speed up crypotocurrency inflation relative to other currencies. It's hard for me to comprehend, that others don't see the pyramid scheme behind it.
[+] [-] knorker|5 years ago|reply
This is why we can't have nice things.
[+] [-] amasad|5 years ago|reply
Mining is one vector of abuse but there are many others when you’re giving free compute to the world. Especially in the radically open way we’re doing it at Replit.
[+] [-] mike_d|5 years ago|reply
[+] [-] junon|5 years ago|reply
This is inevitable, and I'm sure every CI system has faced this issue.
[+] [-] smarx007|5 years ago|reply
Also, edits to the CI script could be made suspect. I never had a first-time contributor on my projects start by making changes to the CI pipeline.
I am pretty sure smart folks at GH thought of this and just deliberating whether to introduce such a breaking flow to maintainers.
[+] [-] praseodym|5 years ago|reply
[+] [-] tomasreimers|5 years ago|reply
What's to stop attackers from making a one-off harmless edit ("forgot a comma in readme") and then, once they're whitelisted, deploying a malicious executable to the CI pipeline.
I think the root issue is that people without write access to your repo can queue arbitrary compute on your dime by simply creating a PR and changing the GitHub workflow files (the definition for GitHub actions). This is even a bigger issue for companies with self-hosted runners who can't use those for public repos as an attacker could file a PR with malicious code and compromise a machine (https://docs.github.com/en/actions/hosting-your-own-runners/...)
One possible solution here is that a maintainer needs to okay any change that changes a workflow file before it runs. Not sure if that introduces other problems...
[+] [-] hrpnk|5 years ago|reply
The more apparent problem is that CI jobs can execute arbitrary code and are not limited wrt. their execution time. If limited, it would render them useless when used for cryptomining.
[+] [-] _fat_santa|5 years ago|reply
[+] [-] lmeyerov|5 years ago|reply
Until it is default-deny for ~all capabilities with say RBAC for enabling, which are basic security principles, they're pretty scary for public repos. It's even scary for private ones as you might want say a contractor or intern to be limited. That they have people publicly dedicated to whackamole response, but seemingly not to security fundamentals (or if they do, not following them / empowered to), is frustrating. Look at the GHA permissions panel and then think like an attacker or defender, it's scary.
The environment variable stuff awhile back was understandable.. but not the big issue. As basically any user can submit a PR that can make current actions that run code (ex: run tests) to run something else by editing the action or code, that means any public user can burn repo $, play in repo-exposed runners (ex: get into corp sandboxes), and if using continuous deployment or service integrations , into their production systems, wherever they publish packages, etc. GitHub is api exposed and git can have commits deleted, so they can even cover most of their tracks. This is SolarWinds all over again, but now a bot can run it automatically on ~everyone!
GHA is both one of my favorite things about GitHub but also been scary. Maybe now that they have access to some. of the best security engineers and researchers in the world, they can fix this...
[+] [-] smarx007|5 years ago|reply
"Secrets are not passed to workflows that are triggered by a pull request from a fork."
[+] [-] nodesocket|5 years ago|reply
If just CPU’s that’s gotta be such a gigantic waste of CPU power for such little crypto.
[+] [-] damsta|5 years ago|reply
[+] [-] gonzo41|5 years ago|reply
I think if half the effort spent here was spent learning finance, the bad actors would have more money.
[+] [-] GordonS|5 years ago|reply
Hope they manage to sort it out - and also hope these arseholes doing the damage don't ruin GitHub Actions for the rest of us.
[+] [-] qqii|5 years ago|reply
[0] in a discussion about the dangers of running your own CI server. I thought it were sensible if there was a permission config file which specified who can trigger which actions, where the CI uses the permissions of the last ran commit.
[+] [-] aleksi|5 years ago|reply
[+] [-] ddevault|5 years ago|reply
[+] [-] woutr_be|5 years ago|reply
[+] [-] photon-torpedo|5 years ago|reply
https://github.com/Bhargav1912/test_1/pull/2/files
Following the files used reveals the pool and wallet used. The pool lets you look up the stats for this wallet. For this wallet, it says:
> Total Paid: 163000 TRTL - 42.16$
But I guess the culprit is rotating wallets. Judging by the hash rate, the pool estimates a weekly income of 700$. It could be a multiple of this if the culprit uses multiple wallets at once.
[+] [-] speedgoose|5 years ago|reply
[+] [-] politelemon|5 years ago|reply
[+] [-] exdsq|5 years ago|reply
[+] [-] junon|5 years ago|reply
[+] [-] peanut_worm|5 years ago|reply
[+] [-] CynicusRex|5 years ago|reply