top | item 45530832

Rubygems.org AWS Root Access Event – September 2025

280 points| ilikepi | 4 months ago |rubycentral.org

159 comments

order

ctoth|4 months ago

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing.

They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window.

You cannot verify supply chain integrity from logs on a system where root was compromised, and you definitely can't verify it when the logs didn't exist (they enabled them during remediation?).

So basically, somebody correct me here if I'm wrong but ... Every gem published Sept 19-30 is suspect. Production Ruby applications running code from that window have no way to verify it wasn't backdoored. The correct response is to freeze publishing, rebuild from scratch (including re-publishing any packages published at the time? Ugh I don't even know how to do this! ) , and verify against offline backups. Instead they rotated passwords and called it done.

tptacek|4 months ago

Isn't the subtext of this post pretty clearly that the unauthorized actor was Andre Arko, who had until days prior all the same access to RubyGems.org already?

The impression I have reading this is that they're going out of their way to make it clear they believe it was him, but aren't naming him because doing so would be accusing him of a criminal act.

placardloop|4 months ago

CloudTrail logs for the last 90 days are enabled by default, cannot be turned off, and are immutable, even by root. If you view this “event” as starting when Arko was supposed to have their access terminated, that’s within the 90 day window and you can indeed trust the logs from that period.

ctoth|4 months ago

Thinking about this a bit more... it sure is interesting that around the time of a competing project launch that something just happens which might reasonably completely compromise trust in the previous incumbent, isn't it? Odd!

coredog64|4 months ago

Not going to bother reading the article, but will chime in here that the recommendation from AWS is to have a separate security account within your organization that only holds your CloudTrail logs. This does potentially double your cost, as you only get one CloudTrail for free, and it's very useful to have an in-account trail for debugging purposes.

Organizations are also useful because you can attach SCPs to your accounts that deny broad classes of activities even to the root user.

shevy-java|4 months ago

You are actually right, I didn't think of this before.

How can they ensure that nobody else did any tampering?

It seems RubyCentral did not think this through completely.

yargzeblog|4 months ago

IMO the only way to avoid doing a total rebuild is to have Andre Arko:

1. Admit that he was the unauthorized actor (which means he's probably admitting to a crime?) 2. Have him attest he didn't exfil or modify the integrity of service while committing a crime.

If I was Ruby Central I would give clemency on #1 in exchange for #2 and I think #2 helps Andre Arko.

akerl_|4 months ago

Given the context of the post, it seems like "Enabled AWS CloudTrail, GuardDuty, and DataDog alerting" means "enabled alerts via CloudTrail, GuardDuty, and Datadog", not "enabled Cloudtrail logging". Otherwise the comment about reviewing Cloudtrail wouldn't make sense.

arianvanp|4 months ago

You can't enable or disable AWS Cloud Trail as far I know?

You can enable the persistent storage of trails. But you can always access 90 days of events regardless of that being enabled

eutropia|4 months ago

They buried the lede...

Arko wanted a copy of the HTTP Access logs from rubygems.org so his consultancy could monetize the data, after RC determined they didn't really have the budget for secondary on-call.

Then after they removed him as a maintainer he logged in and changed the AWS root password.

JeremyNT|4 months ago

What a truly wild situation.

In a certain sense this post justifies why RC wanted so badly to take ownership - I mean, here you have a maintainer who clearly has a desire to sell user data to make a buck - but the way it all played out with terrible communication and rookie mistakes on revoking access undermines faith in RC's ability to secure the service going forward.

Not to mention no explanation here of who legally "owned" the rubygems repo (not just the infra) and why they thought they had the right to claim it, which is something disputed by the "other" side.

Just a mess all around, nobody comes off looking very good here!

picadi|4 months ago

really disappointing. it's such a huge security concern and privacy/ethical lapse, i am super disappointed in him, despite his contributions to the world of Ruby package management

he's now started a competing gem.coop package manager, and while they haven't released a privacy policy it does make me suspicious about how they were planning to fund it

no single person should have Github owner + AWS root password for a major language's package manager and ecosystem just sitting around on their laptop while they fly around to different conferences in Japan e.g. (as Andre did while hacking rubygem's AWS root account to show off)

ebcase|4 months ago

[citation needed]

shevy-java|4 months ago

I'd recommend to people to wait for a response - RubyCentral spins up a gazillion accusations right now and has been in the last days (and, it is also incomplete, because why did they fire every dev here and placed Marty Haught in charge specifically? They never were able to logically explain this; plus, why didn't they release this write-up before? It feels very strange to wait here; they could have clarified things before, but to me it seems they kind of waited and then tried to come up with some explanation that, to me, makes no real sense).

I also highly recommend to not accept RubyCentral's current strategy to post very isolated emails and insinuate that "this is the ultimate, final proof". We all know that email conversation often requires lots of emails. So doing a piecemail release really feels strange. Plus, there also were in-person meetings - why does RubyCentral not release what was discussed here? Was there a conflict of interest due to financial pressure?

Also, as was already pointed out, RubyCentral went lawyering up already - see discussions on reddit. Is this really the transparency we as users and developers want to see? This is blowing up by the day and no matter from which side you want to look at it, RubyCentral sits at the center; or, at the very least, made numerous mistakes, tries to cover past mistakes by ... making more mistakes. I think it would be better to dissolve RubyCentral. Let's start from a clean state here; let's find rules of engagement that doesn't put rich corporations atop the whole ecosystem.

Last but not least - this tactical slandering is really annoying. If they have factual evidence, they need to bring the matter to a court; if they don't, they need to stop slandering people. To my knowledge RubyCentral hasn't yet started a court case, and I have a slight suspicious that they also will not, because we, as the general public, would then demand COMPLETE transparency, including ALL of RubyCentral's members and their activities here. So my recommendation is: wait for a while, let those accused respond.

saghm|4 months ago

Yeah, this is incredibly confusing. The stance that Ruby Central has stated since the takeover of the RubyGems (offline) tooling on Github was that it was necessary for supply chain security, but if this happened literally within a couple of weeks of when they tried (and apparently failed?) to remove all of the previous maintainers, how does this add any amount of confidence in their ability to secure things going forward? If they can't even properly remove the people they already knew had access that they went out of their way to try to remove, it's hard to feel like consolidating their ownership over all of the tooling is going to be an improvement.

dismalaf|4 months ago

> let those accused respond.

Literally all we've heard so far is from the other side...

> If they have factual evidence, they need to bring the matter to a court

I'd be surprised if they aren't. This post feels very much like the amount of disclosure a lawyer would recommend to reassure stakeholders.

> rules of engagement that doesn't put rich corporations atop the whole ecosystem

Right now the only thing stopping us all from being held hostage by rogue maintainers is a rich corporation.

jnewland|4 months ago

This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:"

> 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.

> 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.

TehCorwiz|4 months ago

Right?! Did nobody there think to actually disable the accounts? These are the people who are harping about "security" being the reason for the ham-fisted takeover of the source repos, but they didn't secure the production infrastructure?

colonwqbang|4 months ago

It didn't occur to them that he might have written the password down? That's wild.

deng|4 months ago

If they really have ethical concerns regarding sharing data with third parties, maybe they should update their privacy policies accordingly?

"We collect information related to web traffic such as IP addresses and geolocation data for security-relevant events and to analyze how and where RubyGems.org is used."

(https://rubygems.org/policies/privacy)

"We may share aggregate or de-identified information with third parties for research, marketing, analytics, and other purposes, provided such information does not identify a particular individual."

(https://rubycentral.org/privacy-notice/)

nomdep|4 months ago

> “Following these budget adjustments, Mr. Arko’s consultancy, (…), submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII). The offer would have given Mr. Arko’s consultancy access to that data, so that they could monetize it by analyzing access patterns and potentially sharing it with unrelated third-parties.”

WTF. This is the same guy that is launched gems.coop, a competing index for Ruby gems recently.

On the other hand, RubyCentral actions were truly incompetent, I don’t know anymore who is worse

heartbreak|4 months ago

Anyone who remembers how Ruby Together was managed knows who is worse.

ttfvjktesd|4 months ago

> failed to rotate the AWS root account credentials ... stored in a shared enterprise password manager

Unfortunately, many enterprises follow the poor practice of storing shared credentials in a shared password manager without rotating them when an employee with prior access leaves the company.

kjs3|4 months ago

You might be surprised/horrified at the number of shops I run into that use shared creds without a password manager, still use creds from ex-employees because changing them smells too much like work, and ask "why would I do that?" when you ask about rotation.

tptacek|4 months ago

Presuming, as a group full of security peers kibitzing about this in a chat right now all do, that the "unauthorized actor" here is Andre Arko, this is Ruby Central pretty directly accusing Arko of having hacked Rubygems.org; it depicts what seems to be a black letter 18 USC 1030 violation.

Any part of this narrative could be false, but I don't see a way to read it and take it as true where Arko's actions would be OK.

saltypal|4 months ago

Putting myself in Arko’s shoes, I can imagine (charitably!) the following choice, realizing that I still have access and shouldn’t:

1. Try to get in touch, quickly, with someone with the power to fix it and explain what needs to be rotated.

2. Absent 1, especially if it cannot be done quickly, rotate the credentials personally to get them back to a controlled state (by someone who actually understands the security implications) with the intent to hand them off. Especially if you still _think_ of yourself as responsible for the infrastructure, this is a no-brainer compared to letting anyone else who might be in the same “should have lost access but didn’t, due to negligence” maintain access.

Not a legal defense, but let’s not be too hasty to judge.

hokumguru|4 months ago

Even if Arko wasn't the unauthorized access, this absolutely tarnishes his reputation for the rest of his career, damning email notwithstanding.

Horrible time to own/run a consultancy. Can't imagine what his other customers are thinking right now.

rgreeko42|4 months ago

So is this a smear of Arko (and by extension Ruby Gems' sloppy security) but dressed up like a Security disclosure?

If I'm reading it right, it seems quite petty (and a bit cowardly). Arko was a maintainer was he not? How is that a breach? Presumably his credentials were not misbegotten, or is that the accusation?

fatbird|4 months ago

After Arko's direct access was revoked, Arko retained access via possession of the root password (which RC should have rotated at the same time). Arko then changed the root password, locking RC out of their AWS account, waited a couple weeks, and then Joel Drapper blogged about the situation with proof that the now-fired Arko controlled the account, in order to make RC look bad.

picadi|4 months ago

one assumes he copied the AWS root password out of the RC-provided enterprise password manager / vault onto his own personally controlled password manager before he was locked out, which might be forgivable if it wasn't the root login for a major language's package registry

PapaPalpatine|4 months ago

It’s only a smear if all of the public comments thus far against Shopify and Ruby Central are also smears.

brunosutic|4 months ago

Does anyone know if Arko can have legal troubles (like being sued) if it can be proven he "removed authorized users" from RubyCentral AWS account?

dismalaf|4 months ago

RubyCentral is a legal entity and they paid him for his work so they definitely could take action against him if he was shown to have harmed users or the company in any way.

skywhopper|4 months ago

The most notable thing about this post is the great lengths to which they are willing to go to detail every step of this process, something they never came close to doing in the fallout from the original stripping of rights from the longtime volunteers.

its-summertime|4 months ago

They do seem to have found the perfect scapegoat to point everyone towards to deflect from all the other issues with their actions

RhythmFox|4 months ago

Not only that but they get to use an individual who they have philosophical differences with. You can say it was 'good security practice', tarnish his reputation, and get to switch the narrative to something sympathetic to yourself all in one go. Very convenient for them.

I think they make a lot of overly strong claims here, even though there are plenty of alternative explanations possible. The mere fact that 3 people had AWS root access during this period but they only identify one and never question that it could have been one of the others is telling. They reallllly want you to just take it as obvious that 1) all these actions were taken by 1 individual and 2) that individual was malicious. Then they sprinkle in enough nasty sounding activities and info about Andre to get you to draw the conclusion that he is bad, and did bad things, and they had to do these things the way they did.

Using what reads like a business strategy email as a 'nefarious backstory' is so bad faith. I bet if you got access to all the board's emails you would see a ton of proposals for ways to support RubyGems that may not all sound great in isolation. They are being just transparent enough to bad mouth Andre while hiding any motivations from their end as purely 'security' related.

fareesh|4 months ago

I've read both sides of the story and it does not make sense why he would change the root password and not communicate this with RubyCentral or make any attempt to contact them about the change. That in itself suggests malice.

neverrroot|4 months ago

This, cutting through the noise, this is it.

jmuguy|4 months ago

Well this certainly clarifies a lot. I'm not sure I'm confident that they know nothing was compromised further or the extent of any data exfil, it sounds like a lot of logging was enabled after the incident occurred.

xer0x|4 months ago

The Rubygems take over drama continues!

mylons|4 months ago

[deleted]

voidfunc|4 months ago

This is an absolute disaster for Ruby and the integrity of the Ruby ecosystem...

I dont know if you can build a product with Ruby beyond this without basically having your own highly restricted gem cache.

codegeek|4 months ago

"The root account credentials, essentially the highest level of administrative control, are stored in a shared enterprise password manager in a shared vault to which only three individuals had access: two current Ruby Central staff members and one former maintainer, André Arko"

I am wondering. Did they at least have MFA enabled on the root login or not ?

terracatta|4 months ago

Yes because they state under the section "Root Cause Analysis"

> Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.

mikey_p|4 months ago

So they failed to properly protect their credentials?

This sure doesn't reflect all this supposed professionalism and improvements RC was supposed to make.

Years ago, I decided with all the DHH drama, that using Rails was too much of a liability and this shit just makes the whole Ruby ecosystem a liability to anything build in that ecosystem.

raggi|4 months ago

Honestly I think everyone involved here, and I know most of them, needs to step back and grow up. Despite the formal tone, and the potential legal issues at play here, this situation seems to be descending into needless childishness with potential life damaging side effects. All sides are being disrespectful.

andrewguenther|4 months ago

In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures.

There's no actual control improvements here, just "we'll follow our procedures better next time" which imo is effectively doing nothing.

Also this is really lacking in detail about how it was determined that no PII was accessed. What audit logs were checked? Where was this data stored?

Overall this is a super disappointing postmortem...

nerdjon|4 months ago

> In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures.

I am curious what preventative measures you expect in this situation? To my knowledge it is not actually possible to disable the root account. They also had it restricted to only 3 people with MFA which also seems pretty reasonable.

It is not unheard of that there could be a situation where your ability to login through normal means (like lets say it relies on Okta and Okta goes down) and you need to get into the account, root may be your only option in a disaster situation. Given this was specifically for oncall someone having that makes sense.

Not saying there were not failures because there clearly are, but there have been times I have had to use root when I had no other option to get into an account.

brian_cunnie|4 months ago

Sometimes I log into the root account to see the billing information.

I created an "administrator" account, but apparently it can't see the billing information, including the very-important amount of remaining cloud credits.

Maybe I could spend time fiddling with IAM to get the right privileges, but I have more pressing tasks. And besides, on my personal AWS account I only log in with the root account.

phoronixrly|4 months ago

So we have DHH with his unhinged posts on one side, and Arko wanting to sell PII on the other. Great!

I think we need an f-droid-like project for Rubygems that builds the gems from source, and takes care of signing, and is backed by a non-profit that is independent from Rails/Shopify

dismalaf|4 months ago

Gem can pull in gems from any repository, even straight from a git server like GitHub. And most of the time gems are built from scratch on your computer, Nokogiri is the only one I can think of that isn't.

Imustaskforhelp|4 months ago

Yes!

Not even sure why you are being downvoted, this is such a great idea actually.

F-droid has been so professional and they are just so professional

There was this developer (axet) who recently accused f-droid of somehow convincing the users "maliciously" that the funds are going to the the creator and f-droid when in reality it was going to f-droid and he name called them and what not..

Do you know what f-droid team still said?

They said that they can help him in the donation process and remove theirs and they actually took some feedback from what I know...

They clarified that the donations in their about page that the money that you donate through f-droid in their website's homepage donate goes to f-droid only which should be obvious but for some it wasn't

they also had f-droid donate in the website links of apps and I am not sure when they stopped it but they also stopped it and I deeply deeply respect it.

Like, okay maybe mistakes happen but f-droid is seriously good corporation. We might need something like that for sure. I genuinely think that out of thinking about open source so much, I realized that we need to have priorities to share things about open source.

F-droid is on the top of the list, its just that great, then there is signal/grapheneos or maybe all 3 are on top...

F-droid as an organization is something that I deeply appreciate and its a shame of google's attestation. I genuinely love f-droid nowadays.

shivenigma|4 months ago

RC seems to be incompetent and malicious, the original hostile take over is explained as a security measure. But in reality, you can simply just take away the ability to deploy and leave access to contribute to the repo untouched for people outside of the RC organization.

But claiming that you care about security while missing a basic step of rotating credentials once a member has moved away is pathetic. God help Rubygems.org and the users.

terracatta|4 months ago

That email screenshot is pretty bad for Arko. It clearly shows intent to sell PII data to a third party during a time when Ruby Central had diminished funds and needed help affording basic services.

What the fuck.

mikey_p|4 months ago

Why do they need money? What happened to their funding?

xaxaxa123|4 months ago

interesting timing, isnt it?!

dismalaf|4 months ago

Lol and in previous posts everyone was acting like Arko is above board...

I brought up multiple times that his actions were suspicious, was downvoted. Now proof of that plus an email trying to low-key extort RubyCentral into allowing him to sell user data...

ameliaquining|4 months ago

It's not extortion, because refusing his offer would have left them no worse off than if he hadn't gotten involved in the first place. You can still object on other grounds, of course.

andrewmcwatters|4 months ago

Ethical and legal boundaries? RubyGems Privacy Notice already tells you that they share information with a number of large firms and notably ClickHouse... for "Customer Data Processing."

All this proposal does is request from one of the maintainers/on-call providers? another entry in this Privacy Notice as a part of a payment deal.

This is a mess, but it also unnecessary smears both sides. It calls out that RubyCentral had poor cloud management in place, and it trashes an on-call provider.

This is a terrible postmortem and all it does is advertise to users that RubyCentral doesn't know what it's doing.

RhythmFox|4 months ago

[deleted]

fijiaarone|4 months ago

At least it’s not npm. Or cargo.

busterarm|4 months ago

Props to Ruby Central for taking all of the smears and reputational damage on the chin silently while they mitigated an actual security incident, made absolutely sure and wrote up a proper post-mortem. All of that in line with their original statement that their actions taken were in the interest of security/integrity of their platform.

If there's any evidence that you need to know who the proper stewards of Ruby's gems are, it's this.

Mystery-Machine|4 months ago

An entity that promised security had a security incident due of their incompetence to properly secure their production environment root access?

xaxaxa123|4 months ago

odd timing for such an incident..

queenkjuul|4 months ago

I mean I'd much rather it be people who remember to rotate their passwords after firing high profile staff