"Given the nature of this update and the need to ensure the highest level of security, we have provided limited advanced notice. We understand that the maintenance process impacts our customers, and we apologize in advance for any inconvenience this may cause." -- This is a ridiculous statement. Security does not come from opaque statements made at the last moment.
I fail to understand how updating a "cloud native" service requires downtime at all, much less this sort of outage.
Translation: In the aftermath of the hack last week they found additional major security issues that require downtime to fix. They are limiting the advanced notice to give attackers less time to try to find them before they can be fixed.
I don't think they're saying it's more secure because they didn't tell anyone.
In a prior job, we discovered some automation left ports open that could theoretically be accessed by customers, thus a malicious actor could theoretically have loaded software onto a privileged instance. So we had to treat all devices as potentially compromised. That meant running automation to tear them down and rebuild. This took days.
We were able to do this without visible outage, but the devices affected were mostly doing NAT; if you have a more central component affected, you do need downtime to back it up (that's the read-only section, don't let customers write data that doesn't get backed up) and then rebuild and put it all back together.
Also, it's just easier to get it right with downtime. And they're probably padding these estimates to handle the inevitable things that go wrong during maintenance.
Setting up a registry is easy and actually pretty fun. You can even run it in a docker container (https://hub.docker.com/_/registry)! The complicated/non-trivial part is getting a front-end set up if you want more than the command line. I would recommend most people at least keep backups of their important images in a private registry. It really doesn't take much to keep one running, and you're covered if Docker Hub has an outage.
For most people's needs, command line access to the registry is more than enough though. There are a couple of options out there already for a registry web UI, but I've never had the chance to use any.
If this is related to security, I think users deserve to know what's up.
As someone working in security, I'm fairly distraught that I still don't know exactly what happened last week. The Docker post-mortem[1] states:
> On Thursday, April 25th, 2019, we discovered unauthorized access to a single Docker Hub database storing a subset of non-financial user data. Upon discovery, we acted quickly to intervene and secure the site.
> During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as GitHub and Bitbucket tokens for Docker autobuilds.
> There was a brief period of unauthorized access to a Docker Hub database. During this time some sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of users as well as GitHub and Bitbucket tokens for Docker autobuilds. All these tokens have been revoked.
To the best of my knowledge, the above excerpts are the entirety of information about the incident that Docker has officially released. This is not nearly the level of detail that any security-conscious customer cares about, and in my opinion, it's an egregious violation of trust to not give more information.
For those that may not be working in security day-in, day-out, here's what I mean:
If this incident was, at its core, a DockerHub error that exposed a database to the Internet without proper authentication in place -- and then someone stumbled upon it -- that's an embarrassing mistake, but these things happen. I'd feel comfortable that Docker understands the extent of the breach, and that we can continue to use these products with some degree of confidence.
If, however, the breach was -- and we're going to the other end of the spectrum here -- a state-backed APT that blew 0days to breach the DockerHub network, then the threat model is significantly different. Did Docker bring in incident response professionals? What were the attackers targeting? How confident can we be that they didn't pivot to somewhere else on the Docker infrastructure undetected?
I know that Docker is likely trying to save face, and that the former scenario is significantly more likely than the later -- but guessing about whether or not the breach was severe is a ridiculous situation to be in for major organizations that use the DockerHub service.
In case anyone was wondering, I wasn't personally or professionally impacted by the breach, but we performed full credential rotation anyway. If the breach was more severe than described (e.g., persistent access was established), then that probably wouldn't do much good... but it's better than just assuming that "only 5% of users could have been impacted," and doing nothing.
I'm very unhappy with Docker's communication of this breach/misconfiguration/incident/whatever it may have actually been. I really hope they release more information (perhaps in the form of a real post-mortem) so that DockerHub users can better understand the risks of using Docker products. In my opinion, it's incredibly irresponsible of Docker to keep this information to themselves.
A bit of a tangent: as someone studying cybersecurity at a university in NYC and dealing with a range of topics: virtualization, forensics, incident response, defense, offense, CTFs, bounty programs etc. etc.
And at the pace I am going at it (evening classes), it will take a few years to finish, and even then I might not start in the security industry right away.
I would like to ask, in your opinion, what is a good way to stay up to date, simulating as close as possible to real-world experience through hands-on self-teaching methods ? And what areas would you concentrate on personally, considering how vast the field is.
huslage|6 years ago
I fail to understand how updating a "cloud native" service requires downtime at all, much less this sort of outage.
mbell|6 years ago
ben509|6 years ago
In a prior job, we discovered some automation left ports open that could theoretically be accessed by customers, thus a malicious actor could theoretically have loaded software onto a privileged instance. So we had to treat all devices as potentially compromised. That meant running automation to tear them down and rebuild. This took days.
We were able to do this without visible outage, but the devices affected were mostly doing NAT; if you have a more central component affected, you do need downtime to back it up (that's the read-only section, don't let customers write data that doesn't get backed up) and then rebuild and put it all back together.
Also, it's just easier to get it right with downtime. And they're probably padding these estimates to handle the inevitable things that go wrong during maintenance.
dsfyu404ed|6 years ago
Probably something to do with DB consistency.
chrismeller|6 years ago
paulddraper|6 years ago
"Given the nature of this update and the need to ensure the highest level of security, we have provided limited advanced notice."
mh-|6 years ago
Here's the original from time of submission: https://web.archive.org/web/20190503141604/https://success.d...
Now as of my comment the entire body of the page is:
Docker Hub Maintenance - May 2019
In early May, 2019, Docker Inc. will perform maintenace on the Docker Hub.
This article details actions to be taken, timeline, and any current status/issues/recommendations.
bovermyer|6 years ago
dcchambers|6 years ago
For most people's needs, command line access to the registry is more than enough though. There are a couple of options out there already for a registry web UI, but I've never had the chance to use any.
david_shaw|6 years ago
As someone working in security, I'm fairly distraught that I still don't know exactly what happened last week. The Docker post-mortem[1] states:
> On Thursday, April 25th, 2019, we discovered unauthorized access to a single Docker Hub database storing a subset of non-financial user data. Upon discovery, we acted quickly to intervene and secure the site.
> During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as GitHub and Bitbucket tokens for Docker autobuilds.
> There was a brief period of unauthorized access to a Docker Hub database. During this time some sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of users as well as GitHub and Bitbucket tokens for Docker autobuilds. All these tokens have been revoked.
To the best of my knowledge, the above excerpts are the entirety of information about the incident that Docker has officially released. This is not nearly the level of detail that any security-conscious customer cares about, and in my opinion, it's an egregious violation of trust to not give more information.
For those that may not be working in security day-in, day-out, here's what I mean:
If this incident was, at its core, a DockerHub error that exposed a database to the Internet without proper authentication in place -- and then someone stumbled upon it -- that's an embarrassing mistake, but these things happen. I'd feel comfortable that Docker understands the extent of the breach, and that we can continue to use these products with some degree of confidence.
If, however, the breach was -- and we're going to the other end of the spectrum here -- a state-backed APT that blew 0days to breach the DockerHub network, then the threat model is significantly different. Did Docker bring in incident response professionals? What were the attackers targeting? How confident can we be that they didn't pivot to somewhere else on the Docker infrastructure undetected?
I know that Docker is likely trying to save face, and that the former scenario is significantly more likely than the later -- but guessing about whether or not the breach was severe is a ridiculous situation to be in for major organizations that use the DockerHub service.
In case anyone was wondering, I wasn't personally or professionally impacted by the breach, but we performed full credential rotation anyway. If the breach was more severe than described (e.g., persistent access was established), then that probably wouldn't do much good... but it's better than just assuming that "only 5% of users could have been impacted," and doing nothing.
I'm very unhappy with Docker's communication of this breach/misconfiguration/incident/whatever it may have actually been. I really hope they release more information (perhaps in the form of a real post-mortem) so that DockerHub users can better understand the risks of using Docker products. In my opinion, it's incredibly irresponsible of Docker to keep this information to themselves.
1: https://success.docker.com/article/docker-hub-user-notificat...
Maven911|6 years ago
I would like to ask, in your opinion, what is a good way to stay up to date, simulating as close as possible to real-world experience through hands-on self-teaching methods ? And what areas would you concentrate on personally, considering how vast the field is.