Before census day they were predicting two thirds of the 10 million Australian households to complete the census online. They stated their site had been tested with half a million concurrent users and was able to sustain double that.
Unfortunately no one seems to have considered that the majority of users would do it after dinner (say 6-10pm). Take the 6.6 million users divide it by the time frame and it was never going to work with a 1 million per hour capacity.
What is interesting is with IBM paying out they must have failed to deliver the required capacity. I thought it more likely the government had not tendered it properly and requested a lower capacity than needed but obviously IBM just tried to cut costs and under resource the operation.
I noticed the exact same thing. Read about their load testing in the paper, did a quick back of the envelope, and concluded before the census that they hadn't texted big enough.
Which brings us back to another question: was there even a DOS attack? I've seen any little technical details on the attacks, and I know myself, like a lot of other Australians logged on behind vpns. When the site wouldn't load, of course we hit refresh...
I often wonder why companies and governments even trust large organisations like IBM over individuals they've interviewed themselves. I mean, it's not like IBM's consultant's competence is any secret. IBM is also trying to make it worse, or at least, not making the competence factor any priority. Rather, it's cost over everything:
I mean how do you get away with hiring IBM (or accenture, PwC, didata, deloitte or, may God help us, Tata or something like that, where cost of paycheck is literally the only factor in hiring).
That's a pretty hefty fine, but I wonder what the original billings were for developing the system in the first place?
I've heard reports that they spent >AUD$5Million on just the load testing aspect, but I hope someone can verify that. And I trust that they have improved comprehensive testing methodologies that actually work to improve reliability under duress. (<sarcasm>Tip: Perhaps don't build it using Lotus Notes??</sarcasm>)
I guess we can be thankful that is was 'merely' a census and not something like electronic voting!
In the US, Census data is used for redistricting purposes, which CAN influence election outcomes in the future - so the implications of a census may be wider than you think. I don't know exactly the way it works in Australia, but some googling has led me to believe they use a system that is similar.
I'm not sure it is a hefty fine for the situation. Its simply paying the remaining costs for IBM's failures.
I don't see how IBM thought they could pull if off for $9.6 million [0], especially with the ABS requirements for information handling during the census.
"Key measures to safeguard information include strong encryption of data, restricted access on a need-to-know basis and monitoring of all staff, including regular audits. After data collection and processing, the ABS removes names and addresses from other personal and household information. Names and addresses will be stored securely and separate from one another. No one working with Census data will be able to view your personal information (name or address) at the same time as your other Census responses (such as age, sex, occupation, level of education or income). Stored separately and securely, individuals names will also be substituted with a linkage key, a computer generated code, completely anonymising the personal information. Only these anonymous linkage keys will be used by the ABS to bring data sets together." [1]
Currently, it seems IBM is blaming their subcontractors, "While IBM issued a mea culpa, they were then quick to lay the blame at the feet of Australian communications company NextGen Networks and its upstream provider Vocus, blaming them for failing to implement a geoblocking service called Island Australia to prevent traffic reaching the site from outside Australia." [2]
And considering IBM's has a revenue stream of $20.24 billion this quarter [3], the fine won't really impact them.
As to their reputation, IBM doesn't have a brilliant name already:
* Failure to meet deadlines [4]
* In Queensland they were overbudget, overtime, and under scope [5] resulting in their permanent ban from working with the Queensland government.
* They have a great habit of building things that they call incredibly fast, but never perform up to spec in the real world [6].
Anecdotally, National Mutual contracted out to IBM for some "supercomputers" for their COBOL applications, to speed up processing time. The machines were half the speed IBM claimed. IBM's solution? Double the number of machines delivered, and ignore the extra costs of power, maintenance and network load.
Australia - or more specifically - already have a known-vulnerable electronic voting system they used in an election, so that horse has already bolted.
Just wait until IBM's Phoenix Pay system and the scandal here in Canada comes to a head. With 300,000 civil servants waiting six months to be paid and it's still not fixed.
There must be a big penalty looming for such a massive level of incompetence.
Phoenix seems like a rather awkward name for an application.
>Though many government employees work in 9-to-5 office jobs, the public payroll also includes a wide array of people like Mr. Ryan with complicated work schedules and pay rules.
My father used to be in the Coast Guard he worked 28 days on and had 28 days off. If you were injured that meant the days you were off were really two days?? Try explain that to some bureaucrat in Ottawa who can thinks they are speaking English (really French), they're 2,000 miles away and doesn't understand the system either.
Oh God, that sounds an awful lot like two more Australian systems: the Queensland Health payroll debacle, where a bunch of health workers were paid incorrectly or not at all. Or the National Disability Insurance Scheme debacle, where healthcare providers haven't been paid, resulting in small business owners having to take out huge amounts of personal debt just to pay their employees, and sole traders unable to provide services their clients need to live day to day while still keeping a roof over their own heads. (They're both IBM projects too.)
As someone who has negotiated similar contracts with providers like IBM, I'm willing to bet that someone on the customer side gave into the pressure to rush the lawyers while IBM did a nice job of generating loophole-filled statements of work, and I wouldn't hold my breath for that penalty.
Availability and load balancing aside, did any other Australian citizen (who actually managed to complete the census at all) notice that the UX was quite ordinary? Reminded me of a web form circa 1990.
For instance, IIRC the census asked for my and my wife's country of birth and nationality, but later on when filling in the info for our children, it asked for the nationality of the both parents again. Simple logic checking back through the data should have shown that our (the parent's) information was already supplied, and this information for our children could be pre-filled or skipped and auto filled in the database.
Just a few small things that would have made the filling out of the forms a lot smoother, faster, and less complicated for lay people who were already nervous about doing the data entry.
Except you may have step-children, adopted children, foster children, children in the house not part of your family (friends of your child, children of your friends), etc. Keep in mind that the census needs to include everyone who was in the house at the time it was filled out, not everyone who is part of your regular abode/family situation.
You might argue that it would work for the "most common scenario". In which case why not pre-fill all the data to the statistically most common values? (Hint: the census's purpose is to collect those statistics.)
Also, pre-filling might cause confusion as to which fields have already been filled out by the user; if a value is present, they might wrongly assume no action is required on their part.
(Having said all that, I haven't seen the online census.)
I thought the presentation was very good. I was pleasantly surprised that the design was professional and modern looking.
It seemed to be a single page application because once you got started it was fast to get through the process. I suspect the SPA nature was a deliberate decision to offload workload from the servers. Imagine how much more traffic would been needed if every page/section on the form required another call to the server.
See my other comment further down, but I just thought of another issue again:
Design consideration has to be given to how/what people will be using to access said questions.
If it looks like a web form from the 1990s , ask yourself what kind of technology some Australians might be using to try to fill out their census form, and how standard we have to try to keep that interface/experience.
My only disappointment is that they didn't have to pay more for their incompetence. IBM has been on a mission to replace their workforce with cheap labour from India for quite a while. I'm glad the work they deliver to their clients is finally being exposed for what it is - junk code with no quality control.
You have to understand that what StackOverflow gets away with in terms of hardware is not at all reflective of what other shops will be able to do.
SO can work on such a small amount of hardware because they've got people who either know the entire infrastructure from ground up, or have the skillset and interest to figure it out. Plus they have very tight coordination between the folks doing the development, and running it day to day (possibly because it's the same people).
Your average developer working on a government contract through $BIGNAMESERVICESPROVIDER is almost certainly not going to have that skillset or motivation to learn it. Even in the unlikely scenario that they did, they're not going to be in a situation where they can act on it. It'll be a case of developing to specification, throwing stuff over a wall to some sort of QA, and then throwing the result of that over to someone to run it.
The whole thing will be run by project managers who have opex budgets to meet, on infrastructure that was specified well before the thing was ever built.
We requested paper forms long before census night, because we were pretty sure this was going to happen. Didn't put our names or any other personally identifying info on them, either. (Never received a fine.)
[+] [-] stirlo|9 years ago|reply
Unfortunately no one seems to have considered that the majority of users would do it after dinner (say 6-10pm). Take the 6.6 million users divide it by the time frame and it was never going to work with a 1 million per hour capacity.
What is interesting is with IBM paying out they must have failed to deliver the required capacity. I thought it more likely the government had not tendered it properly and requested a lower capacity than needed but obviously IBM just tried to cut costs and under resource the operation.
[+] [-] ACow_Adonis|9 years ago|reply
Which brings us back to another question: was there even a DOS attack? I've seen any little technical details on the attacks, and I know myself, like a lot of other Australians logged on behind vpns. When the site wouldn't load, of course we hit refresh...
[+] [-] candiodari|9 years ago|reply
http://www.businessinsider.com.au/ibm-layoffs-1-month-severa...
I mean how do you get away with hiring IBM (or accenture, PwC, didata, deloitte or, may God help us, Tata or something like that, where cost of paycheck is literally the only factor in hiring).
[+] [-] cyberferret|9 years ago|reply
I've heard reports that they spent >AUD$5Million on just the load testing aspect, but I hope someone can verify that. And I trust that they have improved comprehensive testing methodologies that actually work to improve reliability under duress. (<sarcasm>Tip: Perhaps don't build it using Lotus Notes??</sarcasm>)
I guess we can be thankful that is was 'merely' a census and not something like electronic voting!
[+] [-] timv|9 years ago|reply
See: http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...
[+] [-] unsignedqword|9 years ago|reply
https://en.wikipedia.org/wiki/Redistribution_(Australia)
[+] [-] shakna|9 years ago|reply
I don't see how IBM thought they could pull if off for $9.6 million [0], especially with the ABS requirements for information handling during the census.
"Key measures to safeguard information include strong encryption of data, restricted access on a need-to-know basis and monitoring of all staff, including regular audits. After data collection and processing, the ABS removes names and addresses from other personal and household information. Names and addresses will be stored securely and separate from one another. No one working with Census data will be able to view your personal information (name or address) at the same time as your other Census responses (such as age, sex, occupation, level of education or income). Stored separately and securely, individuals names will also be substituted with a linkage key, a computer generated code, completely anonymising the personal information. Only these anonymous linkage keys will be used by the ABS to bring data sets together." [1]
Currently, it seems IBM is blaming their subcontractors, "While IBM issued a mea culpa, they were then quick to lay the blame at the feet of Australian communications company NextGen Networks and its upstream provider Vocus, blaming them for failing to implement a geoblocking service called Island Australia to prevent traffic reaching the site from outside Australia." [2]
And considering IBM's has a revenue stream of $20.24 billion this quarter [3], the fine won't really impact them.
As to their reputation, IBM doesn't have a brilliant name already:
* Failure to meet deadlines [4]
* In Queensland they were overbudget, overtime, and under scope [5] resulting in their permanent ban from working with the Queensland government.
* They have a great habit of building things that they call incredibly fast, but never perform up to spec in the real world [6].
Anecdotally, National Mutual contracted out to IBM for some "supercomputers" for their COBOL applications, to speed up processing time. The machines were half the speed IBM claimed. IBM's solution? Double the number of machines delivered, and ignore the extra costs of power, maintenance and network load.
[0] http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...
[1] http://www.abs.gov.au/websitedbs/censushome.nsf/home/privacy
[2] http://www.news.com.au/technology/online/security/ibm-faces-...
[3] http://www.businessinsider.com.au/ibm-surprises-with-q2-2016...
[4] http://www.abc.net.au/news/2016-09-27/ibm-unlikely-to-hit-cu...
[5] http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf...
[6] https://en.wikipedia.org/wiki/IBM_7030_Stretch
[+] [-] rodgerd|9 years ago|reply
[+] [-] dghughes|9 years ago|reply
There must be a big penalty looming for such a massive level of incompetence.
Phoenix seems like a rather awkward name for an application.
http://www.nytimes.com/2016/11/18/world/canada/government-wo...
>Though many government employees work in 9-to-5 office jobs, the public payroll also includes a wide array of people like Mr. Ryan with complicated work schedules and pay rules.
My father used to be in the Coast Guard he worked 28 days on and had 28 days off. If you were injured that meant the days you were off were really two days?? Try explain that to some bureaucrat in Ottawa who can thinks they are speaking English (really French), they're 2,000 miles away and doesn't understand the system either.
[+] [-] adambrenecki|9 years ago|reply
[+] [-] nihonde|9 years ago|reply
[+] [-] cyberferret|9 years ago|reply
For instance, IIRC the census asked for my and my wife's country of birth and nationality, but later on when filling in the info for our children, it asked for the nationality of the both parents again. Simple logic checking back through the data should have shown that our (the parent's) information was already supplied, and this information for our children could be pre-filled or skipped and auto filled in the database.
Just a few small things that would have made the filling out of the forms a lot smoother, faster, and less complicated for lay people who were already nervous about doing the data entry.
[+] [-] jestar_jokin|9 years ago|reply
You might argue that it would work for the "most common scenario". In which case why not pre-fill all the data to the statistically most common values? (Hint: the census's purpose is to collect those statistics.)
Also, pre-filling might cause confusion as to which fields have already been filled out by the user; if a value is present, they might wrongly assume no action is required on their part.
(Having said all that, I haven't seen the online census.)
[+] [-] PhilWright|9 years ago|reply
It seemed to be a single page application because once you got started it was fast to get through the process. I suspect the SPA nature was a deliberate decision to offload workload from the servers. Imagine how much more traffic would been needed if every page/section on the form required another call to the server.
[+] [-] ACow_Adonis|9 years ago|reply
Design consideration has to be given to how/what people will be using to access said questions.
If it looks like a web form from the 1990s , ask yourself what kind of technology some Australians might be using to try to fill out their census form, and how standard we have to try to keep that interface/experience.
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] bitmapbrother|9 years ago|reply
[+] [-] justinsaccount|9 years ago|reply
Based on non-cloud setups like http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...
A rack or two would have been able to handle the load.
[+] [-] will_hughes|9 years ago|reply
SO can work on such a small amount of hardware because they've got people who either know the entire infrastructure from ground up, or have the skillset and interest to figure it out. Plus they have very tight coordination between the folks doing the development, and running it day to day (possibly because it's the same people).
Your average developer working on a government contract through $BIGNAMESERVICESPROVIDER is almost certainly not going to have that skillset or motivation to learn it. Even in the unlikely scenario that they did, they're not going to be in a situation where they can act on it. It'll be a case of developing to specification, throwing stuff over a wall to some sort of QA, and then throwing the result of that over to someone to run it. The whole thing will be run by project managers who have opex budgets to meet, on infrastructure that was specified well before the thing was ever built.
[+] [-] suprjami|9 years ago|reply
However it was all non-cloud self-hosted hardware.
[+] [-] beedogs|9 years ago|reply
[+] [-] jacques_chester|9 years ago|reply
Because it's bloody hard to DDoS paper forms. Widely distributed atoms are curiously disinterested in IP packets.
[+] [-] sqldba|9 years ago|reply
It says the damage was estimated at 30 million. It says IBM paid something. It didn't explicitly say IBM paid $30 million.
[+] [-] SyneRyder|9 years ago|reply
"Computer giant IBM will pay more than $30 million in compensation for its role in the bungled census"