My money is personally on massive, unrecoverable data loss.
If you've ever poked around with the way that Apple's website works, you can see that the entire place is a huge mess. There's old servers running ancient (pre-2004) perl scripts alongside the brand new iCloud gear. I can't imagine how the authentication for AppleID is working as login details still work on the ancient pages (think pinstripes and glassy buttons). Depending what URL you hit, the webserver is using php3, php4, perl, python or maybe WebObjects (java).
At one point I wrote a scraper that was targeting one of their product pages, and kept getting random, unexplainable results. It turned out that one of their product areas was behind a round-robin load balancer, with three completely different apache versions on each server. The page was dying on one but not the other two. In the end I just had to repetitively scrape until I hit a good response.
Even the domain for the "maintenance" page for the developer section is telling. It's just a broken template system regurgitating a bit of the homepage.
I work at an Apple Premium Reseller, and when we order stuff from Apple we use http://reseller.apple.com (which now seems to redirect to https://ecommerce.apple.com/) and all those systems feels hacked together just as you're saying.
The Reseller Store has been down the whole time the Developer Center has been down. Even more interesting perhaps, our company's "master apple id", which is used to place multi 100K USD orders from Apple, got some strange seemingly manual password reset from Apple about the same time as the Developer Portal went down.
Apple's hardware service portal (http://gsx.apple.com) has also been even more flakey this week than normal.
Interesting; the http://devimages.apple.com/ URL seems to return the underlying contents of the file, bypassing the SSI (I'm so happy they use SSI... seriously: I love SSI ;P). You can then see the raw <!--#include virtual=""--> directives, and pull the individual parts. (It isn't quite then fair to say that it is a "broken bit of their template system"; it is more that it is a poor way to setup a static large-file caching endpoint, and may itself lead to a security vulnerability. To be clear: that's probably worse ;P.)
I've spent more time using the developer resources recently, and I completely agree. Clicking one link will take you an entirely different feeling architecture, like parts have been built on top without thinking about what the impacts could be. It's probably been a house of cards for a long and it's now fallen down.
Maybe it's worse? Maybe it's related to an earlier incident?
From Feb this year:
"Earlier today it was reported that Apple’s computers had been compromised by a zero-day exploit in Java. Apple quickly released an update to patch the flaw for all Macs, but not before some of its own employees had been hacked.... A site called iPhoneDevSDK has been revealed as the means by which a dangerous exploit was injected via a Java plugin."
Well if this is the case then there is some money to be made shorting the stock on Monday :-) I really like some of Apple's products (my Macbook pro is my favorite laptop of all time so far) but sometimes, like dealing with them as a guy who buys Macs for everyone who works for me, I think "Man they guys have a ways to go when it comes to this enterprise service stuff."
I hope that what ever it is, it doesn't put too big a crimp in them.
I don't believe they have massive, unrecoverable data loss. Think about that: If they are so retarded that they dont have backup then Apple is really doomed. Really. But I think they have good IT... I think they detected serious security breach (which might be related to some incompetence but security is very very hard).
I think so too. If you look back a couple of years, this kind of things (even if not as severe) happened several times. I actually blogged about it (when their DNS was messed up):
Could you imagine if the Linux Developer Center went down? It'd be nearly impossible to write Linux software, or download Linux development tools, or generate a Linux binary that a device was allowed to run. It'd be awful.
Real businesses are affected by this. Please keep your trolling elsewhere. I came to the comments looking for what people are doing, not for snarky and ill-informed comments.
> It'd be nearly impossible to write Linux software, or download Linux development tools, or generate a Linux binary that a device was allowed to run. It'd be awful.
You clearly don't understand how the Apple Dev center works.
Nice attempt at snark, but something being fault-proof in this regard doesn't magically make it worthy in others.
Like desktop use, for one. People will not adopt Linux for their desktop/laptop even when offered for free. Where by people I mean more than 1% of them.
Haha, great troll, dudebro! Well, gonna go back to making money from iOS apps and running Linux on a backend server since I can still write software, download development tools, and generate binaries.
Could you imagine if people could make the kind of money developing things for Linux that they do developing things for iOS? It would make comments like yours kind of funny and relevant.
Can you imagine writing desktop software for Linux? Inconsistent APIs, API breakage, ABI breakage, sizable variances between distributions, multiple desktops UI implementations, and you may as well just forget about mobile.
I'm not justifying Apple's ever increasing shift towards enacting strict constraints on their platform developers, but the comparison with Linux is pointless.
Apple's websites are among the worst corporate sites I've ever used. I don't say that just to be critical or cynical (ahem), I'm merely pointing out that they have tremendous room for improvement. As opposed to say php.net or stackoverflow.com which still give me that fresh air "why can't all sites be like this?" feeling every time I visit them.
Most of the time when I visit Apple's sites now, I just assume that what I need to find will either be buried in a convoluted maze, or simply won't exist, and I'll find myself on a "this has been deprecated" 404-style page which takes me someplace only loosely related to what I was looking for. So I end up back at google to try to find a copy of the information either cached somewhere or offsite.
Simply being an Apple developer is a chore. Keeping up with yearly certificate expirations is taxing when you are contracting for several clients. And they never really worked out an easy way to allow several developers to share certs. I just assume now that the other developers will invalidate whatever shared cert I made.
The situation is bad enough, and exacerbated by Apple stubbornly refusing to see the flaws, that I wish a startup would encapsulate the friction and just take care of all the minutia for me. I should never have to personally deal with provisioning. Anything short of a one click submission to iTunes Connect is reminiscent of all the TCP/IP details that we used to have to put in our modems in the dialup days, when all that should have been required was a phone number and passcode. I can't gently forgive them for it. So I think this downtime could be a wakeup call for them that the inefficiencies in their system are even costing them now.
What's even worse is that we, the developers, their customers, will never get to know what happened. Because that's the Apple way.
I'd really love that Apple proves me wrong on this one and comes clean on the problem, the cause and prevention measures being put in place so this won't happen again, whatever it is.
I have received 6 "How to reset your Apple ID password" Email from Apple during the last couple of days, none of which was triggered by me. Could this be related?
Was anyone else prompted to "update their expired password" when attempting to login to iTunesConnect?
I did have the same password for about three years so it didn't seem too odd. Of course I double-checked the url and it looked good. Also I waited for a few days in the hopes that someone else would notice if this was a scam of some sort.
Unfortunately after updating my password I forgot the new one. At least I think I forgot as my new password (as best I can remember it) doesn't work.
Last Thursday, an intruder attempted to secure personal information of our registered developers from our developer website. Sensitive personal information was encrypted and cannot be accessed, however, we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed. In the spirit of transparency, we want to inform you of the issue. We took the site down immediately on Thursday and have been working around the clock since then.
In order to prevent a security threat like this from happening again, we’re completely overhauling our developer systems, updating our server software, and rebuilding our entire database. We apologize for the significant inconvenience that our downtime has caused you and we expect to have the developer website up again soon.
Github, Rubygems, Linux Kernel, etc. get hacked; restore from SHA256SUM'ed backups; keep moving.
Turns out what used to be called "hobby" projects matter, because code made without love has a smell, and no one does a "hobby" for anything but. (Remember the 'ama' in 'amateur')
(ok ok so kernel.org was down for a while. But remember all the heavy lifting done by git to keep those commits clean. It was just the server, not the data.)
This reminds me, yesterday i got an obviously spoofed phishing email from "apple" telling me to reset my passwords and reenter my CC info. Anybody else get that?
I just went to check on the dev center and got this e-mail:
Last Thursday, an intruder attempted to secure personal information of our registered developers from our developer website. Sensitive personal information was encrypted and cannot be accessed, however, we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed. In the spirit of transparency, we want to inform you of the issue. We took the site down immediately on Thursday and have been working around the clock since then.
In order to prevent a security threat like this from happening again, we’re completely overhauling our developer systems, updating our server software, and rebuilding our entire database. We apologize for the significant inconvenience that our downtime has caused you and we expect to have the developer website up again soon.
Apple is set to report their earnings on Tuesday, July 23rd. If this isn't back up by Tuesday evening, it's going to be a very interesting earnings conference call.
I would be very interested to see what the implications would be if this had happened to iCloud.
I'm sure Apple are as unhappy about whatever has happened as the rest of us (likely much more so), but I think at least some communication from them about it would be in order.
Instead of fighting about which platform is the best, I hope all of you will agree that Apple should not leave us in the dark for 15min, let alone 3 days, or not even "post mortem" to answer the question "what happened?".
Apple just released an update (developers should be getting it in their inbox) but still no ETA on when it'll be up. Very unfortunate for those of us waiting to release apps before August.
> or device provisioning and certificates (potentially very profitable)
Well, an `App'(, the name of which I would not like to identify, ) which was installed using Safari exploits and whose intended use is to help users search and install apps free of charge from AppStore, is still up and running.
I guess there exist various exploits up and down the App Store chains, till now.
If it weren't Sunday, I wouldn't speculate, but it is. My 2 cents: iOS7 updates galore. Just imagine everything that could be overhauled. Even the old gray textured background we see is not very much like iOS 7.
iOS is in a big transition here. You can't even update apps at all unless you now include 'widescreen' support, for example. If you don't, iTunes reports an Invalid Binary (no default 586 image, etc).
So I would guess a huge overhaul, and typical Apple, is taking care of that vs. arguing or commenting on theories.
So I would guess a huge overhaul, and typical Apple, is taking care of that vs. arguing or commenting on theories.
A design overhaul wouldn't require the dev centre to go down at all - they could just prepare all the assets etc on testing servers and switch them over when they are ready.
Given the normal warning given on any maintenance, and the obvious negative consequences for Apple's business of any extended outage (extended in this case being over a day or so), this is unintended, and most likely caused by a security breach. I think we can safely rule out a design overhaul unless Apple are incredibly incompetent.
NB itunesconnect is still up (the bit which deals with app upload, itunes store metadata etc), only the dev center - dealing with certs/device registration/distribution etc - is down.
If this multi-day outage was planned, and they didn't communicate it to devs (like they usually do) then this is a huge, huge slap to the face. It seems very unlikely.
Being sunday has nothing to do with it - notice the title is "still down" - it's been down since thursday. If this was planned, any sane company would give notice to their developers, Apple's success implies they're sane.
Don't forget that developers get 30%, when Apple show off (rightfully so) a huge sum paid to developers, they've been paid significantly more than that - developers seriously matter.
I really hope you're right and the Dev Center returns without explanation, but with neon icons and a "flat" design. The backlash would help Apple to adjust its priorities again. (Services over fashion)
[+] [-] nwh|12 years ago|reply
If you've ever poked around with the way that Apple's website works, you can see that the entire place is a huge mess. There's old servers running ancient (pre-2004) perl scripts alongside the brand new iCloud gear. I can't imagine how the authentication for AppleID is working as login details still work on the ancient pages (think pinstripes and glassy buttons). Depending what URL you hit, the webserver is using php3, php4, perl, python or maybe WebObjects (java).
At one point I wrote a scraper that was targeting one of their product pages, and kept getting random, unexplainable results. It turned out that one of their product areas was behind a round-robin load balancer, with three completely different apache versions on each server. The page was dying on one but not the other two. In the end I just had to repetitively scrape until I hit a good response.
Even the domain for the "maintenance" page for the developer section is telling. It's just a broken template system regurgitating a bit of the homepage.
http://devimages.apple.com/maintenance/
http://devimages.apple.com/
Truly a hacked together system. Some engineers at Apple must be having a truly awful weekend, no matter the cause and solution.
[+] [-] filleokus|12 years ago|reply
The Reseller Store has been down the whole time the Developer Center has been down. Even more interesting perhaps, our company's "master apple id", which is used to place multi 100K USD orders from Apple, got some strange seemingly manual password reset from Apple about the same time as the Developer Portal went down.
Apple's hardware service portal (http://gsx.apple.com) has also been even more flakey this week than normal.
Something is broken over at Apple.
[+] [-] saurik|12 years ago|reply
[+] [-] nicholassmith|12 years ago|reply
[+] [-] yapcguy|12 years ago|reply
From Feb this year:
"Earlier today it was reported that Apple’s computers had been compromised by a zero-day exploit in Java. Apple quickly released an update to patch the flaw for all Macs, but not before some of its own employees had been hacked.... A site called iPhoneDevSDK has been revealed as the means by which a dangerous exploit was injected via a Java plugin."
http://www.cultofmac.com/216618/this-iphone-developer-forum-...
[+] [-] ChuckMcM|12 years ago|reply
I hope that what ever it is, it doesn't put too big a crimp in them.
[+] [-] tlogan|12 years ago|reply
[+] [-] ychw|12 years ago|reply
http://www.clingmarks.com/apple-please-be-serious-about-itc/... http://www.clingmarks.com/ios-vs-android-mobile-dev-platform...
[+] [-] straight_talk_2|12 years ago|reply
[+] [-] e3pi|12 years ago|reply
Do we read this, you have an AAPL short order in tomorrow's market open?
I find no financial news on this.
[+] [-] NelsonMinar|12 years ago|reply
[+] [-] cscurmudgeon|12 years ago|reply
> It'd be nearly impossible to write Linux software, or download Linux development tools, or generate a Linux binary that a device was allowed to run. It'd be awful.
You clearly don't understand how the Apple Dev center works.
[+] [-] alayne|12 years ago|reply
[+] [-] jammi|12 years ago|reply
[+] [-] coldtea|12 years ago|reply
Like desktop use, for one. People will not adopt Linux for their desktop/laptop even when offered for free. Where by people I mean more than 1% of them.
[+] [-] toyg|12 years ago|reply
[+] [-] vor_|12 years ago|reply
[+] [-] benihana|12 years ago|reply
[+] [-] pbsdp|12 years ago|reply
I'm not justifying Apple's ever increasing shift towards enacting strict constraints on their platform developers, but the comparison with Linux is pointless.
[+] [-] zackmorris|12 years ago|reply
Most of the time when I visit Apple's sites now, I just assume that what I need to find will either be buried in a convoluted maze, or simply won't exist, and I'll find myself on a "this has been deprecated" 404-style page which takes me someplace only loosely related to what I was looking for. So I end up back at google to try to find a copy of the information either cached somewhere or offsite.
Simply being an Apple developer is a chore. Keeping up with yearly certificate expirations is taxing when you are contracting for several clients. And they never really worked out an easy way to allow several developers to share certs. I just assume now that the other developers will invalidate whatever shared cert I made.
The situation is bad enough, and exacerbated by Apple stubbornly refusing to see the flaws, that I wish a startup would encapsulate the friction and just take care of all the minutia for me. I should never have to personally deal with provisioning. Anything short of a one click submission to iTunes Connect is reminiscent of all the TCP/IP details that we used to have to put in our modems in the dialup days, when all that should have been required was a phone number and passcode. I can't gently forgive them for it. So I think this downtime could be a wakeup call for them that the inefficiencies in their system are even costing them now.
[+] [-] qnk|12 years ago|reply
I'd really love that Apple proves me wrong on this one and comes clean on the problem, the cause and prevention measures being put in place so this won't happen again, whatever it is.
[+] [-] kailuowang|12 years ago|reply
[+] [-] sarreph|12 years ago|reply
http://www.zdnet.com/amid-extended-apple-developer-site-down...
[+] [-] swalsh|12 years ago|reply
[+] [-] mrpizzadelivery|12 years ago|reply
I did have the same password for about three years so it didn't seem too odd. Of course I double-checked the url and it looked good. Also I waited for a few days in the hopes that someone else would notice if this was a scam of some sort.
Unfortunately after updating my password I forgot the new one. At least I think I forgot as my new password (as best I can remember it) doesn't work.
[+] [-] tsenkov|12 years ago|reply
[+] [-] FPSDavid|12 years ago|reply
[+] [-] esalman|12 years ago|reply
Last Thursday, an intruder attempted to secure personal information of our registered developers from our developer website. Sensitive personal information was encrypted and cannot be accessed, however, we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed. In the spirit of transparency, we want to inform you of the issue. We took the site down immediately on Thursday and have been working around the clock since then.
In order to prevent a security threat like this from happening again, we’re completely overhauling our developer systems, updating our server software, and rebuilding our entire database. We apologize for the significant inconvenience that our downtime has caused you and we expect to have the developer website up again soon.
[+] [-] undoware|12 years ago|reply
World's premier closed-source shop: (presumably) gets hacked; goes down; stays down.
Github, Rubygems, Linux Kernel, etc. get hacked; restore from SHA256SUM'ed backups; keep moving.
Turns out what used to be called "hobby" projects matter, because code made without love has a smell, and no one does a "hobby" for anything but. (Remember the 'ama' in 'amateur') (ok ok so kernel.org was down for a while. But remember all the heavy lifting done by git to keep those commits clean. It was just the server, not the data.)
[+] [-] Jgrubb|12 years ago|reply
http://imgur.com/hyta4bC
[+] [-] rossjudson|12 years ago|reply
[+] [-] atgm|12 years ago|reply
Last Thursday, an intruder attempted to secure personal information of our registered developers from our developer website. Sensitive personal information was encrypted and cannot be accessed, however, we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed. In the spirit of transparency, we want to inform you of the issue. We took the site down immediately on Thursday and have been working around the clock since then.
In order to prevent a security threat like this from happening again, we’re completely overhauling our developer systems, updating our server software, and rebuilding our entire database. We apologize for the significant inconvenience that our downtime has caused you and we expect to have the developer website up again soon.
[+] [-] Tloewald|12 years ago|reply
[+] [-] slowdown|12 years ago|reply
[+] [-] allwein|12 years ago|reply
[+] [-] unknown|12 years ago|reply
[deleted]
[+] [-] peterkelly|12 years ago|reply
I'm sure Apple are as unhappy about whatever has happened as the rest of us (likely much more so), but I think at least some communication from them about it would be in order.
[+] [-] tsenkov|12 years ago|reply
[+] [-] navs|12 years ago|reply
[+] [-] ttflee|12 years ago|reply
Well, an `App'(, the name of which I would not like to identify, ) which was installed using Safari exploits and whose intended use is to help users search and install apps free of charge from AppStore, is still up and running.
I guess there exist various exploits up and down the App Store chains, till now.
[+] [-] xixora1|12 years ago|reply
[+] [-] vinhnx|12 years ago|reply
[+] [-] tater|12 years ago|reply
[+] [-] trackztar|12 years ago|reply
iOS is in a big transition here. You can't even update apps at all unless you now include 'widescreen' support, for example. If you don't, iTunes reports an Invalid Binary (no default 586 image, etc).
So I would guess a huge overhaul, and typical Apple, is taking care of that vs. arguing or commenting on theories.
[+] [-] grey-area|12 years ago|reply
A design overhaul wouldn't require the dev centre to go down at all - they could just prepare all the assets etc on testing servers and switch them over when they are ready.
Given the normal warning given on any maintenance, and the obvious negative consequences for Apple's business of any extended outage (extended in this case being over a day or so), this is unintended, and most likely caused by a security breach. I think we can safely rule out a design overhaul unless Apple are incredibly incompetent.
NB itunesconnect is still up (the bit which deals with app upload, itunes store metadata etc), only the dev center - dealing with certs/device registration/distribution etc - is down.
[+] [-] untog|12 years ago|reply
[+] [-] tylermac1|12 years ago|reply
[+] [-] mcintyre1994|12 years ago|reply
Don't forget that developers get 30%, when Apple show off (rightfully so) a huge sum paid to developers, they've been paid significantly more than that - developers seriously matter.
[+] [-] nwh|12 years ago|reply
[+] [-] gurkendoktor|12 years ago|reply