top | item 16494387

Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

417 points| RmDen | 8 years ago |itprotoday.com | reply

319 comments

order
[+] dctoedt|8 years ago|reply
Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128 million [2].

[0] https://www.scribd.com/document/100544020/Indiana-IBM-Decisi...

[1] http://www.in.gov/judiciary/opinions/pdf/03221601shd.pdf

[2] https://www.indystar.com/story/news/2017/08/07/court-ibm-owe...

[+] dpweb|8 years ago|reply
Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.

But after selling the deal, their workers have to solve a difficult engineering and organizational management problem simultaneously (the project).

They use contractors (devs) which proves they are not an engineering but sales culture. So, you need to bridge the two worlds. That's the PMs. These are the most incompetent members of the team. The devs very often are very smart.

Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.

Reasons for failure:

1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem. 2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.

[+] wahern|8 years ago|reply
Apropos HN, he then says,

  officials set out to fix Indiana's poorly-performing
  welfare system by inserting an untested, theoretical
  experiment, and substitute personal caseworkers with
  computers and phone calls ("remote eligibility"). This
  is now admitted to be an error, and there is nothing in
  this case, or the Court's power, that can be done to
  correct it, or remedy the lost taxpayer money or
  personal suffering of needy Hoosiers.
[+] hodgesrm|8 years ago|reply
Sometimes, you just gotta love the judges that handle these cases. The first 7 words you quoted plus the ensuing judgement are written very well.
[+] mathattack|8 years ago|reply
I’ve had nothing but terrible experiences with IBM Global Services. With deals like this, the commission check is cut long before the problems arise.
[+] matwood|8 years ago|reply
IBM consulting and many state governments are a mess. It’s no surprise a large project requiring both to function properly failed.
[+] watertom|8 years ago|reply
I was the CTO for a large, Fortune 100 company.

When I first took over the position I would regularly get requirements documents for internal projects that were 3 or 400 hundred pages. The project team would dutifully carry out the requirements gathering process, everything was meticulously documented, with data flows, and process maps, etc. Everything was a requirement, everything was mandatory, and everything had top priority. IT wasn’t allowed to say no, wasn’t allowed to criticize the business, and wasn’t allowed to analyze if what was specified made sense, or would work. Luckily I had a CIO that was willing to back me up when I started rejecting these projects. In six months I killed 10 or so very large multi-year software projects, in each case the business was forced to use existing software and change their processes. We saved a ton of money and implemented the solutions in weeks not years. These would have projects that had upwards of 100 people for 2, 3 or 4 years, all because the business wanted a “perfect” process.

I consulted to the DoD and this pattern was repeated, over and over again.

Most assuredly this is what happened with the Canadian Payroll project. Every person had their pet requirement(s) and they made sure they “got them in”, in the end the system that was specified couldn’t be built.

It’s like SAP, if you buy SAP take it out of the box and adapt your processes to SAP, if you do you’ll have great experience. If you try to customize SAP, it going to get very expensive, you’ll have a ton of problems, you won’t be able to take upgrades and hou’ll Need a ton of staff just go try to keep it running day in and day out.

[+] canada_dry|8 years ago|reply
I've participated on several >$10M Canadian Gov't IT projects over the years and you nailed it.

Business groups will fight tooth-and-nail to keep their existing processes - partly to maintain status-quo, but also because the folks who represent the business know little about system's analysis/design, so they just simply aren't in a mindset to re-engineer.

Combine this with the vendor's (not so hidden) agenda of milking the gov't and it's a recipe for disaster... every damn time.

[+] avinium|8 years ago|reply
Governments everywhere would function a lot better if there were more people like you in them.

I'm probably preaching to the choir, but government managers are deathly afraid to stick their necks out and say "no" to the ridiculous spec-by-committee approach to projects.

It's a lot easier to shift the blame to contractors, particularly when most projects take so long that they're obsolete by the time they go live.

[+] dalbasal|8 years ago|reply
I've seen quite a lot of this too, though from other vantage points.

I don't disagree with anything you say, and would probably go down a similar route. These projects have enormous and unacknowledged risk. One of those risks (the most common pone) is that the project does actually arrive at completion, is good enough to go into use, but it's still pretty crap. Not better than what could have been achieved at a fraction of the cost. You should (as CEO, CIO) take this as a given, and act accordingly. IE, minimize these projects and find alternatives.

So, this isn't a criticism...

Here's the problem as I see it: Custom, "enterprise^" software is important, very. A lot of things could work a lot better if we had better enterprise software, much better. Especially for government, the ability to produce good software that works is tremendous. Transport systems could be better, welfare systems could be better, democratic systems could be much higher information, education.... Google maps really raised the minimum standard for public transport "timetables," especially buses. This changes public transport, enabling multi-stop routes (you would never find them otherwise), casual use, tourist use...

So... How do we do this better? Personally, I think the recurring failure details are probably a red herring. You imply problems like not having a designer, just a dump of requirements from anyone who has the juice to make them. Problems like cargo-cult design around existing processes, rather than adapting to new tools. IMO these are symptoms of bad meta-systems, the "economy" around making decisions, buying services, designing solutions, etc. Also, naturally, the incentives.

Will this be a problem forever?

^I'm at a loss for a good general term. Hopefully everyone understands what I mean.

[+] Cacti|8 years ago|reply
I have had a similar experience. People hate change, and will do their best to not change their processes, ever, unless someone makes them (which rarely happens). They'd rather spends hundreds of millions of dollars inventing some new piece of software just so that don't have to spend a minute pressing a button they don't want to press.

I am convinced 90% of corporate IT software would be irrelevant if the businesses were willing to change 5 or 10% of their process.

[+] tyingq|8 years ago|reply
Been there, and truly understand.
[+] goodroot|8 years ago|reply
As a Canadian who has worked at IBM, I'm not surprised that this didn't find success. In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale.

With that being said, IBM isn't a successful technology company with a proven record building good software products. They were an unwise choice from the get-go. IBM is a successful financial engineering and sales firm that sells things, then scrambles to hire/acquires to get things done to an ever-evolving, always justifiably rough spec. They quack and then start the hack-show.

What do you expect from the organization that invented "FUD" (https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt) as a way to emotionally manipulate perspective clients into making a purchasing decision?

[+] happyopossum|8 years ago|reply
> IBM isn't a successful technology company with a proven record building good products.

Whoa, what? Sure, some of their achievements are in their past, but that claim really overlooks a ton of history.

[+] giobox|8 years ago|reply
> In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale.

How in any way is this "In IBMs defense"? We should think it OK that they pursued and committed to a large government contract that in reality they were not qualified to complete? The sentence reads more as a condemnation.

[+] j45|8 years ago|reply
I know folks who work at IBM as well - don't mean to put you on the spot - but IBM should be far better at leading and guiding customers to success.

Except at $250-350/hour, below average performance is more profitable than successful and a lot of good people suffer on both sides.

[+] ikeboy|8 years ago|reply
Maybe it's time to start firing people for choosing IBM.
[+] Terribledactyl|8 years ago|reply
As an American who has worked at IBM, I'm also not surprised that this didn't find success.

They messed up my own payroll, by sending too little, then too much, then we had a 6 month back and forth trying to resolve it.

[+] ralusek|8 years ago|reply
> perspective clients

prospective

[+] zaroth|8 years ago|reply
Not having ever worked on anything like this, the death knell for these projects must be the inability to ever reduce functionality into a reasonable core set. Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely?

You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

[+] chime|8 years ago|reply
Having worked on similar things on a smaller scale but higher up the chain, I can tell you the list of possible issues is a mile long. Everything from lack of leadership and conflicting goals to ever-changing scopes and insufficient resources is a candidate for the big blame. A lot of people, organizations, and systems have to work well together to succeed.

As techies, we think the technical solution is the most superior solution but that is not always the case. Here is an example of something that just happened to me 2 hours ago today. User called and said osTicket system I manage treats '0' as blank. So when they fill out a form with question 'How many X happened today?' and they enter '0', the system just treats the field as unfilled and does not print '0'. That's a big problem for auditing because they absolutely need to be able to show that it was '0'. I started digging into the source code to see where '0' int was treated as a blank. osTicket being PHP where 0, '', null, and false can all be treated as FALSE, it could've been 8+ hours worth of work digging into the right piece of code to figure out the fix.

So I called the user up and we discussed the best way to handle this. In the end, I added a new question above this one: 'Did one or more X happen today? YES/NO'. If it's 'NO', it doesn't matter the 'How many X happened today?' with '0' doesn't get printed.

To make a major implementation work, this kind of discussion needs to happen 10000x a day among 1000+ people, many of whom may not have the authority to just decide things on the fly.

[+] mseebach|8 years ago|reply
> You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

I read something in a trade journal probably probably about 15 years ago, that really stuck with me: When businesses buy big software solutions (I think the article was about ERP systems), they should prepare to adjust and change their processes (within reason, of course) and not just insist that the software be modified endlessly to suit every little ossified process and preference. Not just because it's very expensive, but because the resulting system will be brittle and incompatible and taking advantage of further developments.

Too many buyers of software would benefit immensely from taking a good, hard look at those remaining 20% and making some hard decisions about what is actually important. Sure, for a government payroll system, there is limited room for movement, but I'd suspect that the majority of the long tail are people on weird, custom contracts and the correct solution is actually to hire a room full of clerks to handle these in Excel - and/or applying political pressure on the hiring parties to get their contracts adjusted to fit standard payroll schemes.

[+] rhizome|8 years ago|reply
Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely?

Zeno's Featureset

[+] trhway|8 years ago|reply
>You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

this is where crucial distinction and choice comes - enterprise software which supposedly have enough flexibility and configurability to cover all the cases or the software which allows and requires a bunch of custom code. The former is a monster and the latter requires good tech chops. And choosing the software with the right balance between what can be configured and what would have to be developed requires deep tech wisdom and experience on the part of the leadership which is usually a very tall order...

[+] microcolonel|8 years ago|reply
Also there's a good chance that the decision to go with IBM as a vendor, and the conduct thereafter, was corrupt from the start. That would go a long way toward explaining why so little due diligence was observed in assessing whether or not to continue the project with them at any of the stages it could've been.

Yes, people do make huge blunders in powering forward with overpowered, unnecessary solutions to problems they largely do not have; but blunders this big usually don't take this long to suss out.

[+] jellicle|8 years ago|reply
Incidentally, the error was here:

"The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs."

It's not software problems, or not primarily software problems anyway. They fired EVERYONE who knew anything about payroll in the entire Canadian government, all at once, and hired 500 brand new people in one call center to work with the brand new software as it rolled out. The new software doesn't include code for a lot of "special cases" which together represent a lot of cases. The new people have no idea how to solve most pay problems (people problems OR software problems). They get four days of training.

http://www.cbc.ca/news/canada/ottawa/phoenix-falling-pay-cen...

"Phoenix was supposed to cut our work in half. It actually doubled our work. So, you can't get rid of 2,000 people ... and double the work and expect the work to get done," one said."

Overall, even if the software worked perfectly the execution would have been a disaster.

[+] kabdib|8 years ago|reply
Was not surprised to see Oracle and PeopleSoft involved.

I hope they get the crap sued out of them. That stuff is toxic, and I pity any organization that Oracle got its claws into using that junk.

[+] manofstick|8 years ago|reply
New Zealand in the 90's was trying to have an integrated policing system designed by IBM which burnt a lot of money (in the order of 100 million) and was abandoned [0].

I was working with one of the companies that came along to replace one part of the system with a much smaller component that just handled that individual piece and vividly remember the meeting with the IBM team. They had no code to show me, but they handed my a massive folder - probably verging on 10cm thick of use case diagrams. So many little stick figures staring up at me I was awestruck - but not in a positive way!

[0] https://en.wikipedia.org/wiki/INCIS

[+] erpellan|8 years ago|reply
The real achievement here is managing to run up a bill of 1B CAD while still failing to deliver. I mean, I can think of a few projects that probably cost several million/yr but 100x? Astonishing.

There's an anecdote floating around about the consultant who heard of a billion-dollar government IT project who promptly offered to do it for 20 million. When asked how, they replied that they would sit on a beach for 5 years then pick up the phone and say, "We failed". Their contact got back in touch 5 years later and said, "I wish we'd given it to you".

[+] kellysutton|8 years ago|reply
This sounds like a good application of Gall's Law:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

[+] vidanay|8 years ago|reply
So, what makes a payroll system at this scale so difficult? On the face of it, there are salary employees and there are hourly employees. How many taxing bodies does Canada have? Federal, provincial, and municipal? Is that a few dozen or a few hundred? Is Canadian tax code crazy complicated that doesn't allow for a rules based system?

I'm honestly curious as to how a "simple" payroll system can go so far off the rails.

[+] BoorishBears|8 years ago|reply
I remember thinking “Why does this simple state Jury Duty site need to be down from 9PM to 9AM”

“I could write a site that did this job with 99.9% reliability in a weekend”

Turns out disability laws require a real person be available to assist with the site. Imagine bidding on a website project and accidentally promising 24/7 live support.

That’s government tech in a nutshell.

[+] dragonwriter|8 years ago|reply
> So, what makes a payroll system at this scale so difficult?

Complex pre-existing business rules of a large, sui generis employer (governments, especially sovereign ones rather than localities, often are because the laws that apply to other employees don't apply to them, and the rules that do apply often aren't coherent across-the-board policies determined by a central org focussed on HR, but often are set in special case laws applied to individual subunits of the government by a legislature then operationalized by the HR staff in that subunit) trying to be put in a COTS system make it hard, but probably what makes it most out of control is that the contractor is optimized in maximizing value extracted not efficient delivery and that governments generally suck at overseeing IT contracts (sometimes this is corruption, but it's often just lacking the knowledge and skills to do it effectively at the direct contract management staff level, often complicated by mandated IT contracting frameworks that are based on the underlying assumption that software engineering is basically structurally the same as civil engineering.)

[+] Demoneeri|8 years ago|reply
There are multiple rules based on collective bargaining (union).

"The software platform itself, a heavily modified version of Oracle’s PeopleSoft program, had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."

[+] blantonl|8 years ago|reply
The software platform itself, a heavily modified version of Oracle’s PeopleSoft program ...

What implementation of Peoplesoft isn't heavily modified?

Adapting business processes to COTS software packages is extremely challenging. Anyone remember the absolute bank that Peoplesoft consultants made back in the day when ERP and Business Process Re-engineering was all the rage?

[+] dman|8 years ago|reply
Could this be the first time someone gets fired for hiring IBM? (Sorry could not resist after having heard of that quip so many times).
[+] bloomingfractal|8 years ago|reply
and that's why governments should hire software engineers that can be in the loop and understand not only the technical but the policy side. Hiring an army of contractors earning 300k/year to deliver "something" in the waterfall model is a recipe for disaster.
[+] giarc|8 years ago|reply
So it doesn't have to be this way.

I work for Alberta Health Services, we have 120,000 employees across the province. There are likely 5 separate unions plus out of scope staff (management). This organization was created after the merger of 5 separate health care organizations all with separate systems etc. They over time rolled everyone into the same pay system and I didn't hear about a single person missing their pay. The federal government transition has resulted in tons of people being not paid at all, being paid too little or being paid too much.

[+] lopmotr|8 years ago|reply
"there is reputational risk for IBM in not helping us fix this,”" - not true. IBM has a long history of exactly this kind of problem. Canada didn't care about their existing poor reputation and the next government to hire them won't either.
[+] endorphone|8 years ago|reply
The Canadian government attempted to build a gun registry, originally predicting it to cost $12M, then $85M, then $1000M, then $2000M, and then it was scrapped. While it's easy to blame vendors like IBM, I have to imagine feature-creep and "what about"-isms lead to such monstrosities.
[+] myrandomcomment|8 years ago|reply
This is sad. IBM used to be the watch word for "It just works!" I started at IBM many years ago. When I went to my 1st ISP (back in the early 90s) and things broke I was like "what do you mean it broke? How is this okay?" It was quite a shock to move from a company where we had systems with 20+ years of uptime to having reboot the USENET server every 5 days.