Like all IBM projects now day the bulk of the work was done in India for likely less than $5 CAD/hr, while they billed out $100 CAD/hr or even more.
The actual numbers may be slightly different, but still these are close to reality.
What a great profit margin for a failed project.
I wouldn't be surprised the code is like a lot I've seen from Indian-based sweat shops (btw there are amazing Indian coders, but you generally won't find many of them in these software houses producing the run of the mill code) - cut and paste code that hard codes every option and combination. And who cares because IBM makes money every time someone asks for a change.
IBM is the winner here, and everyone else is a loser.
Not defending IBM, but IBM's job was to install "off the shelf software called PeopleSoft" to create a payroll system. They 'installed' it and walked away as that was what was required of them. They are now being called in to do more and that has contracting costs associated with it. The Govt of Canada has as much blame in this as anything - poorly written contracts with little knowledge of how software works.
>Like all IBM projects now day the bulk of the work was done in India for likely less than $5 CAD/hr,
This was a PeopleSoft install with customization and IBM typically doesn't use India workers for those types of projects. I don't know of this Canada project in particular but I have knowledge of half a dozen other IBM PeopleSoft contracts with the government and they use onsite developers for it. One of the big reasons is the
consultants must pass a federal background check which requires USA citizenship. I would assume Canada has a similar requirement.
Also, IBM doesn't have a PeopleSoft delivery/solution center in India to do this work. Last I checked, IBM does have a SAP delivery team in India.
I had a small company that did software consulting work for one of the provinces. Around 10+ years ago governments started requesting "off the shelf" solutions instead of custom-developed software. This led to a situation where they didn't own the code, were tied to vendor upgrade schedules, and had to pay a premium for customizations. This had the result of spiralling costs.
Each time there's a scandal, they tighten the rules so that ultimately only large companies with teams dedicated to writing proposals can compete.
To be fair:
- we worked on smaller projects of a scope where one person could maintain a mental model of the system
- nothing so mission critical as payroll
I expect the pendulum to swing back toward more internal development and greater ownership and autonomy. So far, not much has changed.
> btw there are amazing Indian coders, but you generally won't find many of them in these software houses producing the run of the mill code
My favorite story is when, at a company I worked, the maintenance of a security product with the potential to brick a machine was handed over to such a sweat-shop. My then manager, who was behind the outsourcing, called it a success when they had successfully installed the product and bricked their dev machines. "They have already installed it on their machines! (and now they are trying to circumvent it)".
Why doesn't Canada put a rule saying all work must be done by Canadians. Even if its cheaper keeping the money in country seems better than sending it to India.
>IBM Canada was the only company to bid on the government's pay modernization project
If only ONE company bid on your project, then you honestly need to rethink it. Does the government really believe that IBM was the only company interested?
Any project that only receive one bid should be cancelled and rethought. If your payroll regulation is so complex that only one company wants to bid, then maybe you should address that issue first.
Governments are in many cases completely clueless about software and believe it's some magical sauce that once applied correctly will make up for poorly thought out and overly complex regulations introduced by the politicians. They fail to understand that the developers are required to understand the rules and procedures before they're able to implement them correctly.
It's not unusual for governments to create such project requirements that only one or two companies can bid on it. The "winning" company is known long before.
"But when it comes to Phoenix, Aylward isn't focusing his blame on IBM. Like the procurement experts, he faults the scope of the initial contract for the resulting failure."
Twenty-six years of developing software, I haven't worked on a project yet that was any better than 60% specified. This inability of people to specify, in full and in detail, what they want is the reason for the rise of Agile.
A colleague of mine sagely said, "We need to show them something quickly just so they can tell us what they don't want."
Since the projects were done by consultancies you can't call it a failure from the consultancy point of view. They managed to milk the government for multiples of the estimated budged and this is a great success for them.
That's just how it works - underbid, lock in and then massage the client for x10 of the original budget.
Government contracts create perverse incentives. I've worked in the space for the past couple decades, and it's revolting. The government takes bids for a project - a project written in very specific ways to guarantee that only 1 or 2 companies on the planet are capable of bidding on it, usually. I wouldn't be surprised if a necessary condition of the IBM contract was intimate long-term experience with the exact system they were replacing, which was likely built and maintained by IBM. Then once the contract is going on, if the company is incompetent and can't meet their schedule - they get more money. At NO point will the government EVER dare to hold a companies feet to the fire and tell them 'you must finish this project and absorb the entire cost yourself' which is exactly what NEEDS to be done.
Instead, companies are incentivized to hire the dirt cheapest labor they can because getting the job done quicker is actively harmful to their business. Then at the end of it all.. there's usually a maintenance contract to be awarded to keep the system upright and operating. The company doing the dev work is going to bid on that and have an apparent advantage (only apparent though... if they don't get it the actual technical people who matter will just go with whoever gets the contract, the contracting company will never actually relocate anyone) and the worse the system looks, the more bug-ridden and unlikely to succeed it is, the larger the maintenance contract will be. Even more incentive to build a ball of shit. Literally at every single turn, doing whatever makes the software worse is what is most profitable in that space. And eventually some news articles like this will get published, people will complain, and someone on the govenrment side of the fence will decide they want to make bringing the system in as a badge of honor in their career - so they will slash all the mandatory requirements the system doesn't yet meet and sign an agreement that whatever half-functional garbage they managed to get was a full completion of the contract. And then the executives of the contracting company go out and celebrate at an expensive bar.
I once worked on the maintenance side of a system that had just been delivered. The original contract that had been granted was a single contract to develop 3 systems. 3 contracts later, 1 system was delivered. And that system required 9 months of full-time development work to be ready to be used operationally. You know what that's called? An unfettered stellar monumental magnificent success... as far as the contracting company is concerned.
Similar thing happened in Queensland,Australia. IBM is barred from bidding on contracts (actually, the government forbade anyone on awarding contracts to IBM). IBM sued and won, but the government still will ignore all IBM bids. I only wish they would be able to do more than just ignore IBM, a few stupid bureaucrats signed a bad contract cost Australia too much.
Then again, working in the same way as every other bureaucracy.
The public sector in a number of countries seems addicted to these large-scale high risk forms of procurement.
Instead of running a larger number of smaller projects, which would be harder for them to adminster, there's a temptation to lump it all into one mega-project and shift the responsibility to a consulting company.
The problem with that approach is, of course, that its almost impossible to accurately specify all the requirements of the mega-project up front, so there are lots of amendments, and as any consultant knows amendements are where you make your money as by the time they come along the client is committed to the project, so you have the advantage in negotiations.
Unfortunately the litany of failed projects that follow this model, doesn't seem to deter public sector organisations from following them.
I don't doubt similar things happen in the private sector, but failures in the public sector tend to be more visible.
In the US at least, there is a hard cutoff at a certain price point. Beyond that price point the only legal way is to run through the procurement office with all that entails- written specs, detailed contracts, long term execution, etc. Because the procurement process is so complicated, the only companies that WIN the process are those that are optimized for this (rather then, say, writing software or building roads). Often the contracting companies will not even plan on doing actual work, they only specialize in GETTING work and subcontracting it out.
The whole thing is designed to prevent some mid level beauracrat from kicking his brother in law some project for 10% more then "market rate" ... instead we pay 300% more because every project has to get rolled up into some monolithic thing that's big enough for the big contracting companies to get involved.
Indeed many private sector failures are never known to the public.
One small city newspaper burned well over $4 million on business software this was never know to the public. The software caused the repeated delays in delivery, which was explained as a mechanical problem with the press.
At the time the same paper, ran a highly critical story of the county government for a $100,000 software project over run.
I think the way budgeting and management works the people in charge want to have an upfront price and timeline. It would probably be better to say "let's spend $x/year over the next years and see how far we can get".
The mess is probably on both sides. The issue is that management doesn't care about IT until everything breaks apart. This means IT is seen as cost and not worth the trouble to do well and keep in house. Then of course everything starts falling apart and it's easier to put the fault on the vendors. Of course IBM has its faults, but I'd guess they are more on the overselling side (we can do everything and cheaper and better!) than on the pure technical side.
The contracting company who provides the IT people likewise views IT as a cost and not worth the trouble to do well. Only small companies give a damn about technical things. The contracting companies view their value proposition as mountains of documents, tons of forms, processes, ISO9001 certifications, etc. The people who write software or work with hardware? All they do is type. Their work isn't important in any of this.
>The issue is that management doesn't care about IT until everything breaks apart. This means IT is seen as cost and not worth the trouble to do well and keep in house.
It won't make any difference if they had an inhouse team.
In order to choose a competent vendor, or to hire a competent employee the management itself needs to be sufficiently tech savvy to differentiate between mediocre and good candidates.
IT horror stories involving IBM don't really surprise me anymore. I'm more surprised by the companies and governments they continue to game with their low ball bid tactics.
I once worked with a large team from IBM that was staffed with Indians that did the work both onsite and offshore in India and it was without a doubt the worst code quality I've ever seen in my career. There were classes with 10,000+ LOC, methods with over 1000 LOC and dead unreachable code that wasn't even commented out. Speaking of comments, if it wasn't automatically generated by the IDE there would be none. But the most horrific example of their lack of attention to detail and code quality was the absolute absence of any unit tests in this massive project with millions of LOC. The only unit tests ever written were by me and those unit tests were subsequently disabled/commented out when some other developer decided it would be easier to disable the unit tests rather than to modify them to adhere to the changes they were making. It's a shame more clients don't hire independent code auditing companies to inspect the code their paying millions of dollars for.
Apparently if you spend enough on sales, you can sell utter shit to unsuspecting customers. Then, you beat up the engineering team to fix all of the bugs that the customers report.
It seems to me there is a dollar amount above which any software project must surely fail. In other words if it costs more than X, then the complexity implied is too high and the likelihood of success is too low.
I've worked on federal contracts for Canada for a few years in the past. It really doesn't surprise me that this happened.
I saw failures on every level. The client (government), the PMs, the developers. It was mind-blowing how much incompetency was infested into the entire system.
I think the problem with the public sector is that some of them genuinely don't give a shit about their work and there's really zero repercussions to them for not doing their job well. They won't get fired like someone would in the private sector. With that said, I've no doubt some of them do work hard and are comparable to their private sector counterparts.
But until workers in the government can truly be held accountable, these types of IT disasters will continue to happen. Heads need to start rolling.
Ive been on another IBM run project that started off very similarly for a large energy company. A few thoughts about this.
1) IBM pulled the same types of stunts on this project and also failed spectacularly. Its not in the news though, as private companies are reluctant to admit failure.
2) IBM and other big consulting companies pour large amounts of revenue and time into relationship management and networking. This keeps the "nobody ever got fired for hiring consulting shop x" fallacy alive and kicking.
3) There is never a large implementation that is "out of the box and turn key" If a consulting shop tells you this is possible without great sacrifice and compromise to business process, you should run the other direction.
4) IBM and other big consulting shops regularly oversell their specialty domain knowledge, and often end up farming it out to independent vendor/domain experts after they realize they're in over their head.
5) While I think IBM is a shell of its former self, their behavior here is big consulting industry standard.
6) Most private companies know that a project will largely come in way over budget, and often don't care as long as they can afford it. The overages are usually written off and as long as the operational risk is shifted away from the business then people are usually happy.
As an independent consultant, Im obviously biased, but my opinion comes from experience working along side with Deloitte, IBM, PWC, and some other big consulting shops. I am usually one of a few niche independent contractors that are brought in to handle specialty items.
So why does this keep happening?
Most enterprises/governments do not want to place their faith in boutique consulting shops or independent consultants as this seems too risky even if they would likely be able to implement for half the money in half the time. The arbitrage for big consulting shops comes from this. Most big consulting companies are going to put 1 or 2 senior and expensive(to their own payroll) people on the project and leave the grunt work to cheaper junior onsite or offshore resources. That work is still billed at 150+ per hour and the senior folks 300+ per hour.
Payroll isnt "hard" per se, its just hard to model into a single comprehensive process. Complicating this is the fact that laws and taxes vary by employee type and location. Internal business rules vary everywhere, and even though best practices exist, there is always differences from one place to another. Finally, people get really pissed when their money aint right...
Ive noticed, anecdotally, that IBM in particular has had issues with big projects and variances. I chalk that up to being the relative newbie in the big consulting world, and they're likely underbidding to get a foothold.
Undoubtedly, IBM is partially at fault. However, I strongly believe that a vast majority of the blame rests on the Canadian government's managers. My thinking comes from my experience working with government.
---
I work for a contractor that does the software development work for a large government entity (albeit smaller than the entire federal government of Canada). And while I have some incredible horror stories to tell, the following is particularly bad:
One project was particularly bad. During the initial requirements discovery phase, it appeared like a simple CRUD app. So, my firm proposed an architecture that suited the requirements we found. However, when the government employees in charge of managing the contractors looked at it, they tacked on a few requirements and changed several requirements/fundamental assumptions. So my firm adjusted the proposal and resubmitted. The cycle repeated several times. The government managers really wanted this particular project out as quickly as possible, so my firm had started building some of the basic parts of the application (i.e. creating the database tables, importing the models/relationships into the code base, designing the UI, etc.) before the architecture was finalized.
After a month with 5 engineers working on the project full-time, the managers decided to completely change the requirements altogether. So, we had to throw away everything we had made to date. That month of thrown away work costed the government $100k+ and they have literally nothing to show for it.
While we might want to talk about fallacies of a government contract, the reality is, well even software companies fall for it. Without giving much away, my ex-employer a fairly large "product" company wanted to implement a system from Peoplesoft's competitor.
The management wanted to save cost which meant neither full time employee (FTE) were hired nor retrained. So they brought in IBM consultants.
Once the project was over IBM left only for the company to realize they had no one to make incremental changes to the system or maintain it.
They spent extra money to hire another developer and consultancy to maintain the system.
Takeaways:
a. This was not the first time IBM had delivered shoddy product but "low cost" was a driving factor so they get hired again and again. This is true for both private companies and government.
b. Low cost also means companies don't want to hire FTEs rather make do with contractors. This creates dependency.
c. It is not entirely on IBM or other consultancies but general ineptness too. People holding on to their jobs, hiring and creating messes:
During the development of the aforementioned project one of the workflows fell within my team's domain. But the finance team had some "developers" who wanted to justify their salaries. They wanted to develop but had zero understanding of what do words like "standards" and "best practices" meant. I spent hours and hours arguing over why certain things should be done in a certain way. I was always told I was too uptight.
In the end of module written by these "developers":
a. Transmits salary for the whole company via plain text files
b. If a, was not enough the text file is emailed to 60-70 people in the company who have no authorization to look at salary data. It is always maintained that without sufficient background people might not understand the salary data at all because it uses "employee ids" instead of names.
So not everything was IBM's fault on the project.
Currently, the salary in text is a huge security risk but no one cares. All our efforts to write a program to obfuscate data was rejected for being "too complex".
You would think somewhere in the contract it would specify that the payroll software, upon launching, most actually enable you to pay employees. I understand why IBM would make sure the government had some responsibility, but doesn't IBM have a responsibility to deliver something that actually works? Well, apparently not, and I'm sure if that WAS stipulated in the contract, they would never sign it. I have seen to the other side of the coin, I worked for a consulting web shop years ago where they created a bad contract worth 70K, which was at the time their biggest contract. Yay, high fives all around for the sales team. Unfortunately they ended up spending over 300k to get the system to actually work because the contract was so vague and the requirements so complex.
I smell some kind of corrupt kick back shinanegans going on here, I bet you somebody in the government got a new sports car from IBM to get that contract, and to have an endless maintenance part added in (which is where they make their money).
There was once a programmer who was attached to the court of the warlord of Wu. The warlord asked the programmer: "Which is easier to design: an accounting package or an operating system?"
"An operating system," replied the programmer.
The warlord uttered an exclamation of disbelief. "Surely an accounting package is trivial next to the complexity of an operating system," he said.
"Not so," said the programmer, "When designing an accounting package, the programmer operates as a mediator between people having different ideas: how it must operate, how its reports must appear, and how it must conform to the tax laws. By contrast, an operating system is not limited by outside appearances. When designing an operating system, the programmer seeks the simplest harmony between machine and ideas. This is why an operating system is easier to design."
Payroll is incredibly complex. You have different work rules, union bargaining units, vacation and pension accumulation difference, etc.
Local and state governments also need to track time for Federal and State reimbursement and reporting with multiple, conflicting standards.
It’s compounded by the fact that the people who truly understood how things worked (because they were there when the programs were implemented) are mostly retired now.
When you hire contractors to do this, the scope gets defined up front, and the contractor dumps anything not captured on the customer, which generally means a change order and more $.
And that's just scratching the surface of the types of problems encountered.
Statutory compliance makes payroll software complex. There can be many hundreds of rules defined by government and law (and combinations of these rules) that can affect how much money ends up in your pay packet. In addition to this the payroll software and the data held within needs to be continually kept up to date and compliant with any new rule, change of rule or removal of rules. That alone can mean many monthly updates just to keep in step with legislative changes.
I've found a good rule of thumb for complications is that projects that are governed by the law of the universe (e.g. chemistry, physics) might be super hard, at least the answer will probably be a widely agreed to pass / fail (e.g. the rocket guidance worked and the rocket did not blow up).
On the other hand, software defined by bureaucrats (e.g. payroll, regulation compliance) is going to be complicated with loopholes and journeys into minutia and logic defying requirements. Its all head space with very little logical basis. Everyone has an opinion and its all probably valid, or at least the invalid is disguised fairly well.
This is the perfect place to "confess" to something that happened to me, fresh out of college, at my first full time real job at IBM... 1996ish
I got paid for 2 weeks on my first pay check but I was technically on the job for 3 days. I thought about it, biggest paycheck I had ever had in my entire life at that point, and I decided to do the honest thing and tell my manager. He thanked me, made some calls, did whatever he did and 2 or 3 days later just told me to keep the money and not tell anyone. The effort to "fix" the problem was worth more than the problem.
So take that IBM, I got 11 days free pay from you. Enjoy that Canada!
The C3 project, someone else mentioned, is a fascinating project on payroll. It's an odd problem space, it's complex, no denying it. I've built a lot of software and have a hard time comprehending how it could take a payroll cycle to calculate payroll, I believe that the guys that worked on it were all highly intelligent though. There is this odd desire to say "oh, I could do that better" It's also staggering to think that these companies and agencies are willing to spend hundreds of millions of dollars, re-implementing this highly complex business logic rather than somehow streamlining the payroll processes.. even with unions, it's just hard to comprehend it all.
It's insanity to expect not to waste money when asking IBM to install an Oracle CRM/SCM/FMS/HRMS/EPM/ERP suite. God knows it's running on some huge SPARC or Power system with terabytes of RAM and thousands of threads. Maybe a mainframe too. It's the worse possible combination. It's practically taking the cost of IBM consultancy and the cost of Oracle software and multiplying both. Suicide.
Most governments have public portals (or you have to register with your company details but it's basically open apart from that) that they post RF{I,T,Q}s on.
In the EU, there's the Official Journal of the EU (OJEU) which all public sector procurements over a certain size (~£100k but varies) must be publicised in.
The same Canadian goverment that decided to transfer all their IT systems to one vendor Shared Services Canada which turned out to be another failed system that almost every department including statistics canada blame for delays and failures.
[+] [-] bhouston|8 years ago|reply
The actual numbers may be slightly different, but still these are close to reality.
What a great profit margin for a failed project.
I wouldn't be surprised the code is like a lot I've seen from Indian-based sweat shops (btw there are amazing Indian coders, but you generally won't find many of them in these software houses producing the run of the mill code) - cut and paste code that hard codes every option and combination. And who cares because IBM makes money every time someone asks for a change.
IBM is the winner here, and everyone else is a loser.
[+] [-] deskamess|8 years ago|reply
[+] [-] jasode|8 years ago|reply
This was a PeopleSoft install with customization and IBM typically doesn't use India workers for those types of projects. I don't know of this Canada project in particular but I have knowledge of half a dozen other IBM PeopleSoft contracts with the government and they use onsite developers for it. One of the big reasons is the consultants must pass a federal background check which requires USA citizenship. I would assume Canada has a similar requirement.
Also, IBM doesn't have a PeopleSoft delivery/solution center in India to do this work. Last I checked, IBM does have a SAP delivery team in India.
[+] [-] cleaver|8 years ago|reply
Each time there's a scandal, they tighten the rules so that ultimately only large companies with teams dedicated to writing proposals can compete.
To be fair:
- we worked on smaller projects of a scope where one person could maintain a mental model of the system
- nothing so mission critical as payroll
I expect the pendulum to swing back toward more internal development and greater ownership and autonomy. So far, not much has changed.
[+] [-] jguimont|8 years ago|reply
[+] [-] amag|8 years ago|reply
My favorite story is when, at a company I worked, the maintenance of a security product with the potential to brick a machine was handed over to such a sweat-shop. My then manager, who was behind the outsourcing, called it a success when they had successfully installed the product and bricked their dev machines. "They have already installed it on their machines! (and now they are trying to circumvent it)".
[+] [-] haecceity|8 years ago|reply
[+] [-] throwaway_45|8 years ago|reply
[+] [-] mrweasel|8 years ago|reply
If only ONE company bid on your project, then you honestly need to rethink it. Does the government really believe that IBM was the only company interested?
Any project that only receive one bid should be cancelled and rethought. If your payroll regulation is so complex that only one company wants to bid, then maybe you should address that issue first.
Governments are in many cases completely clueless about software and believe it's some magical sauce that once applied correctly will make up for poorly thought out and overly complex regulations introduced by the politicians. They fail to understand that the developers are required to understand the rules and procedures before they're able to implement them correctly.
[+] [-] emerongi|8 years ago|reply
Happens in all sectors.
[+] [-] bpyne|8 years ago|reply
Twenty-six years of developing software, I haven't worked on a project yet that was any better than 60% specified. This inability of people to specify, in full and in detail, what they want is the reason for the rise of Agile.
A colleague of mine sagely said, "We need to show them something quickly just so they can tell us what they don't want."
[+] [-] danmaz74|8 years ago|reply
[+] [-] myth_drannon|8 years ago|reply
Since the projects were done by consultancies you can't call it a failure from the consultancy point of view. They managed to milk the government for multiples of the estimated budged and this is a great success for them. That's just how it works - underbid, lock in and then massage the client for x10 of the original budget.
[+] [-] otakucode|8 years ago|reply
Instead, companies are incentivized to hire the dirt cheapest labor they can because getting the job done quicker is actively harmful to their business. Then at the end of it all.. there's usually a maintenance contract to be awarded to keep the system upright and operating. The company doing the dev work is going to bid on that and have an apparent advantage (only apparent though... if they don't get it the actual technical people who matter will just go with whoever gets the contract, the contracting company will never actually relocate anyone) and the worse the system looks, the more bug-ridden and unlikely to succeed it is, the larger the maintenance contract will be. Even more incentive to build a ball of shit. Literally at every single turn, doing whatever makes the software worse is what is most profitable in that space. And eventually some news articles like this will get published, people will complain, and someone on the govenrment side of the fence will decide they want to make bringing the system in as a badge of honor in their career - so they will slash all the mandatory requirements the system doesn't yet meet and sign an agreement that whatever half-functional garbage they managed to get was a full completion of the contract. And then the executives of the contracting company go out and celebrate at an expensive bar.
I once worked on the maintenance side of a system that had just been delivered. The original contract that had been granted was a single contract to develop 3 systems. 3 contracts later, 1 system was delivered. And that system required 9 months of full-time development work to be ready to be used operationally. You know what that's called? An unfettered stellar monumental magnificent success... as far as the contracting company is concerned.
[+] [-] muro|8 years ago|reply
Then again, working in the same way as every other bureaucracy.
[+] [-] raesene6|8 years ago|reply
Instead of running a larger number of smaller projects, which would be harder for them to adminster, there's a temptation to lump it all into one mega-project and shift the responsibility to a consulting company.
The problem with that approach is, of course, that its almost impossible to accurately specify all the requirements of the mega-project up front, so there are lots of amendments, and as any consultant knows amendements are where you make your money as by the time they come along the client is committed to the project, so you have the advantage in negotiations.
Unfortunately the litany of failed projects that follow this model, doesn't seem to deter public sector organisations from following them.
I don't doubt similar things happen in the private sector, but failures in the public sector tend to be more visible.
[+] [-] pbecotte|8 years ago|reply
The whole thing is designed to prevent some mid level beauracrat from kicking his brother in law some project for 10% more then "market rate" ... instead we pay 300% more because every project has to get rolled up into some monolithic thing that's big enough for the big contracting companies to get involved.
[+] [-] leejoramo|8 years ago|reply
One small city newspaper burned well over $4 million on business software this was never know to the public. The software caused the repeated delays in delivery, which was explained as a mechanical problem with the press.
At the time the same paper, ran a highly critical story of the county government for a $100,000 software project over run.
[+] [-] afterburner|8 years ago|reply
[+] [-] maxxxxx|8 years ago|reply
[+] [-] kfk|8 years ago|reply
[+] [-] otakucode|8 years ago|reply
[+] [-] ksk|8 years ago|reply
It won't make any difference if they had an inhouse team.
In order to choose a competent vendor, or to hire a competent employee the management itself needs to be sufficiently tech savvy to differentiate between mediocre and good candidates.
[+] [-] CupOfJava|8 years ago|reply
> "IBM Canada was the only company to bid on the government's pay modernization project known as Phoenix."
[+] [-] bitmapbrother|8 years ago|reply
I once worked with a large team from IBM that was staffed with Indians that did the work both onsite and offshore in India and it was without a doubt the worst code quality I've ever seen in my career. There were classes with 10,000+ LOC, methods with over 1000 LOC and dead unreachable code that wasn't even commented out. Speaking of comments, if it wasn't automatically generated by the IDE there would be none. But the most horrific example of their lack of attention to detail and code quality was the absolute absence of any unit tests in this massive project with millions of LOC. The only unit tests ever written were by me and those unit tests were subsequently disabled/commented out when some other developer decided it would be easier to disable the unit tests rather than to modify them to adhere to the changes they were making. It's a shame more clients don't hire independent code auditing companies to inspect the code their paying millions of dollars for.
[+] [-] adekok|8 years ago|reply
Testing? Pfft.. that's for losers.
Design specs? Same answer.
Apparently if you spend enough on sales, you can sell utter shit to unsuspecting customers. Then, you beat up the engineering team to fix all of the bugs that the customers report.
[+] [-] scandox|8 years ago|reply
[+] [-] eljimmy|8 years ago|reply
I saw failures on every level. The client (government), the PMs, the developers. It was mind-blowing how much incompetency was infested into the entire system.
I think the problem with the public sector is that some of them genuinely don't give a shit about their work and there's really zero repercussions to them for not doing their job well. They won't get fired like someone would in the private sector. With that said, I've no doubt some of them do work hard and are comparable to their private sector counterparts.
But until workers in the government can truly be held accountable, these types of IT disasters will continue to happen. Heads need to start rolling.
[+] [-] S_A_P|8 years ago|reply
1) IBM pulled the same types of stunts on this project and also failed spectacularly. Its not in the news though, as private companies are reluctant to admit failure.
2) IBM and other big consulting companies pour large amounts of revenue and time into relationship management and networking. This keeps the "nobody ever got fired for hiring consulting shop x" fallacy alive and kicking.
3) There is never a large implementation that is "out of the box and turn key" If a consulting shop tells you this is possible without great sacrifice and compromise to business process, you should run the other direction.
4) IBM and other big consulting shops regularly oversell their specialty domain knowledge, and often end up farming it out to independent vendor/domain experts after they realize they're in over their head.
5) While I think IBM is a shell of its former self, their behavior here is big consulting industry standard.
6) Most private companies know that a project will largely come in way over budget, and often don't care as long as they can afford it. The overages are usually written off and as long as the operational risk is shifted away from the business then people are usually happy.
As an independent consultant, Im obviously biased, but my opinion comes from experience working along side with Deloitte, IBM, PWC, and some other big consulting shops. I am usually one of a few niche independent contractors that are brought in to handle specialty items.
So why does this keep happening? Most enterprises/governments do not want to place their faith in boutique consulting shops or independent consultants as this seems too risky even if they would likely be able to implement for half the money in half the time. The arbitrage for big consulting shops comes from this. Most big consulting companies are going to put 1 or 2 senior and expensive(to their own payroll) people on the project and leave the grunt work to cheaper junior onsite or offshore resources. That work is still billed at 150+ per hour and the senior folks 300+ per hour.
Payroll isnt "hard" per se, its just hard to model into a single comprehensive process. Complicating this is the fact that laws and taxes vary by employee type and location. Internal business rules vary everywhere, and even though best practices exist, there is always differences from one place to another. Finally, people get really pissed when their money aint right...
Ive noticed, anecdotally, that IBM in particular has had issues with big projects and variances. I chalk that up to being the relative newbie in the big consulting world, and they're likely underbidding to get a foothold.
[+] [-] throwaway5050|8 years ago|reply
---
I work for a contractor that does the software development work for a large government entity (albeit smaller than the entire federal government of Canada). And while I have some incredible horror stories to tell, the following is particularly bad:
One project was particularly bad. During the initial requirements discovery phase, it appeared like a simple CRUD app. So, my firm proposed an architecture that suited the requirements we found. However, when the government employees in charge of managing the contractors looked at it, they tacked on a few requirements and changed several requirements/fundamental assumptions. So my firm adjusted the proposal and resubmitted. The cycle repeated several times. The government managers really wanted this particular project out as quickly as possible, so my firm had started building some of the basic parts of the application (i.e. creating the database tables, importing the models/relationships into the code base, designing the UI, etc.) before the architecture was finalized.
After a month with 5 engineers working on the project full-time, the managers decided to completely change the requirements altogether. So, we had to throw away everything we had made to date. That month of thrown away work costed the government $100k+ and they have literally nothing to show for it.
[+] [-] HeyLaughingBoy|8 years ago|reply
Been known for decades, if not centuries.
[+] [-] thisisit|8 years ago|reply
The management wanted to save cost which meant neither full time employee (FTE) were hired nor retrained. So they brought in IBM consultants.
Once the project was over IBM left only for the company to realize they had no one to make incremental changes to the system or maintain it.
They spent extra money to hire another developer and consultancy to maintain the system.
Takeaways: a. This was not the first time IBM had delivered shoddy product but "low cost" was a driving factor so they get hired again and again. This is true for both private companies and government.
b. Low cost also means companies don't want to hire FTEs rather make do with contractors. This creates dependency.
c. It is not entirely on IBM or other consultancies but general ineptness too. People holding on to their jobs, hiring and creating messes:
During the development of the aforementioned project one of the workflows fell within my team's domain. But the finance team had some "developers" who wanted to justify their salaries. They wanted to develop but had zero understanding of what do words like "standards" and "best practices" meant. I spent hours and hours arguing over why certain things should be done in a certain way. I was always told I was too uptight.
In the end of module written by these "developers":
a. Transmits salary for the whole company via plain text files
b. If a, was not enough the text file is emailed to 60-70 people in the company who have no authorization to look at salary data. It is always maintained that without sufficient background people might not understand the salary data at all because it uses "employee ids" instead of names.
So not everything was IBM's fault on the project.
Currently, the salary in text is a huge security risk but no one cares. All our efforts to write a program to obfuscate data was rejected for being "too complex".
[+] [-] darkstar999|8 years ago|reply
Why didn't they just hire Oracle?
[+] [-] alloowishus|8 years ago|reply
I smell some kind of corrupt kick back shinanegans going on here, I bet you somebody in the government got a new sports car from IBM to get that contract, and to have an endless maintenance part added in (which is where they make their money).
[+] [-] lph|8 years ago|reply
[+] [-] mamcx|8 years ago|reply
I like how this totally get it:
Tao of Programming 3.3 http://www.mit.edu/~xela/tao.html
There was once a programmer who was attached to the court of the warlord of Wu. The warlord asked the programmer: "Which is easier to design: an accounting package or an operating system?"
"An operating system," replied the programmer.
The warlord uttered an exclamation of disbelief. "Surely an accounting package is trivial next to the complexity of an operating system," he said.
"Not so," said the programmer, "When designing an accounting package, the programmer operates as a mediator between people having different ideas: how it must operate, how its reports must appear, and how it must conform to the tax laws. By contrast, an operating system is not limited by outside appearances. When designing an operating system, the programmer seeks the simplest harmony between machine and ideas. This is why an operating system is easier to design."
that was posted early here:
https://news.ycombinator.com/item?id=15063511
[+] [-] Spooky23|8 years ago|reply
Local and state governments also need to track time for Federal and State reimbursement and reporting with multiple, conflicting standards.
It’s compounded by the fact that the people who truly understood how things worked (because they were there when the programs were implemented) are mostly retired now.
When you hire contractors to do this, the scope gets defined up front, and the contractor dumps anything not captured on the customer, which generally means a change order and more $.
[+] [-] teh_klev|8 years ago|reply
http://wiki.c2.com/?WhyIsPayrollHard
And that's just scratching the surface of the types of problems encountered.
Statutory compliance makes payroll software complex. There can be many hundreds of rules defined by government and law (and combinations of these rules) that can affect how much money ends up in your pay packet. In addition to this the payroll software and the data held within needs to be continually kept up to date and compliant with any new rule, change of rule or removal of rules. That alone can mean many monthly updates just to keep in step with legislative changes.
[+] [-] protomyth|8 years ago|reply
On the other hand, software defined by bureaucrats (e.g. payroll, regulation compliance) is going to be complicated with loopholes and journeys into minutia and logic defying requirements. Its all head space with very little logical basis. Everyone has an opinion and its all probably valid, or at least the invalid is disguised fairly well.
[+] [-] Nelson69|8 years ago|reply
I got paid for 2 weeks on my first pay check but I was technically on the job for 3 days. I thought about it, biggest paycheck I had ever had in my entire life at that point, and I decided to do the honest thing and tell my manager. He thanked me, made some calls, did whatever he did and 2 or 3 days later just told me to keep the money and not tell anyone. The effort to "fix" the problem was worth more than the problem.
So take that IBM, I got 11 days free pay from you. Enjoy that Canada!
The C3 project, someone else mentioned, is a fascinating project on payroll. It's an odd problem space, it's complex, no denying it. I've built a lot of software and have a hard time comprehending how it could take a payroll cycle to calculate payroll, I believe that the guys that worked on it were all highly intelligent though. There is this odd desire to say "oh, I could do that better" It's also staggering to think that these companies and agencies are willing to spend hundreds of millions of dollars, re-implementing this highly complex business logic rather than somehow streamlining the payroll processes.. even with unions, it's just hard to comprehend it all.
[+] [-] equalunique|8 years ago|reply
[+] [-] jlebrech|8 years ago|reply
How does anyone knows there's something to bid on or do they only tell IBM? In that case it's hardly IBM's fault is it.
[+] [-] paulhart|8 years ago|reply
[0] http://www.merx.com/
[+] [-] Retric|8 years ago|reply
Companies have some listed period that could be as short as 7 days or as long as 6 months to respond with a Proposal of work including costs.
Government then reviews and picks one based on the stated criteria.
This might seem like a reasonable solution, but it adds significant overhead.
[+] [-] protomyth|8 years ago|reply
https://www.federalregister.gov/
[+] [-] NeutronBoy|8 years ago|reply
[+] [-] timthorn|8 years ago|reply
[+] [-] maxxxxx|8 years ago|reply
[+] [-] fortythirteen|8 years ago|reply
The country club
[+] [-] jasonmaydie|8 years ago|reply