top | item 25975110

CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

1167 points| donsupreme | 5 years ago |technologyreview.com | reply

654 comments

order
[+] zaroth|5 years ago|reply
> Deloitte is the only contractor that can meet the project requirements, because configuration of the VAMS application is occurring using Deloitte’s propriety GovConnect platform. Therefore, no other contractor has rights or access to leverage the system to carry out O&M activities, such as the administration of the system, change and defect management and end user support. As the only contractor with the GovConnect proprietary platform, and the need to stand up O&M support for VAMS while concurrently developing future releases, Deloitte is the only contractor that can achieve the O&M scope and contract requirements.”

Can’t make this stuff up.

https://beta.sam.gov/opp/2f5fc512fdde4e22860832337aac420f/vi...

[+] aasasd|5 years ago|reply
Reads like a copy of Russian government contract practices. Put up a contest, list requirements in such a way that only one company would satisfy them without going bust or mad, award the contract to that company. ‘Make the thing using this and that technologies’ does appear regularly.
[+] martincolorado|5 years ago|reply
This right here “configuration of the VAMS application is occurring using Deloitte’s propriety GovConnect platform.”

Because it is so difficult to launch a project and provision the requisite infrastructure in the federal government due to various IT regs and laws and subsequently obtain an Authorization To Operate (ATO) in a timely fashion an existing platform is routinely sole sourced especially when speed is necessary.

Why? This is both legit and utter bs. Legit: the fed gov actually attempts to protect a persons’ private data. BS: after the drain of the acquisition process (led by non-technical folks), ATO, slide decks to more non-technical management (Deloitte and their peers crush this), the competent folks find other jobs either in or outside gov that free their creativity.

Positive: groups like USDS are reducing the drain, but once an exec gets burned they go back to checkbox management and shun the learning organization culture that has made tech the anti-thesis of this type of acquisition in other sectors.

*edited misspelling to show gov tries to protect not product privacy data

[+] closeparen|5 years ago|reply
For your next laptop, car, etc. try doing what you think the government ought to do here. Write down the requirements, give them to a neutral third party who doesn’t doesn’t use computers/drive, and ask them to figure out the cheapest thing that meets the requirements. Commit yourself to buying only that. See how it goes for you.
[+] snarf21|5 years ago|reply
This is a major problem with using contractors. They bill by the hour. Their continued employment is not dependent on doing a good job. They half ass everything because it is about billing and being done. Also, doing a bad job means "overruns" for more free money. The government needs a Department of Technology that runs all these systems for all national systems that are needed. This isn't like building a combat fighter, it would be straightforward to be build a team that can build a scalable website with today's technology.
[+] smsm42|5 years ago|reply
Somebody had written this with exact purpose to give this contract to Deloitte. I would be very surprised if there wasn't bribery involved.
[+] rishabhd|5 years ago|reply
Hate to say this, but having worked in similar industries, RFPs are sometimes deliberately skewed/ created in a manner that make it hard (or downright impossible) for others to participate.
[+] notagoodidea|5 years ago|reply
That is the easiest a company can leverage an invitation to tender ad infinitum. I believe that going for the cheapest is always the wrong way to conducting those projects. On the company side, bargain you first contract, install your proprietary program, lock everyone up in it and higher your fees when you have leverage on the next requirements.

Government, public offers, invitation to tender must not only play by the cheapest but also by the long term impact that changing contractor every 4, 5 years has on their own management and system. This is getting as absurd and stupid. Force contract to use standard/boring technology and leverage a large market of companies for support. Being the cheapest never was a fair decision maker.

[+] animex|5 years ago|reply
US Digital Service Assemble!
[+] faramarz|5 years ago|reply
Wow! the next article should be about who signed that contract!
[+] satya71|5 years ago|reply
Holy cow! Talk about vendor lock-in! Even AWS is easier to get out of.
[+] GizmoSwan|5 years ago|reply
The problem is that any success in delivering past contracts does not guarantee success in future in software technology where a typical software technology becomes obsolete every 3 years.

In software world things change so fast that someone with 6 years of experience developing software will not have enough experience with newest tools, languages, methods, etc. Newer graduates with experience is newest applications in their resume get hired and they sometimes lack real development experience.

Building and delivering software is not like building a road or sidewalk where technology does not change by much every year or decade. They still construct sidewalks like they did say 100 years ago.

So major software contractor's success is pretty much random depending on who gets hired and how good the chosen packages are for the given application and how experienced are the staff in putting it together and having good test specs.

Another problem in software world is that software developers move to new IPOs and so the staffs are ever more transient. Government jobs with complex hiring practices are actually becoming unattractive to developers.

[+] mgh2|5 years ago|reply
Isn't Palantir doing something similar?
[+] Aeolun|5 years ago|reply
> Deloitte is the only contractor that can achieve the O&M scope and contract requirements

Clearly not. Unless... the requirements were garbage to begin with?

[+] Trias11|5 years ago|reply
Yup. Allocate budget. Hire few $10/hr offsore guys to craft some frontend in a few months. Transfer money. Deliver kickbaks. Shutdown project. Rinse and repeat.
[+] ed25519FUUU|5 years ago|reply
Corporations love regulations. Let’s remember this when we support things like net neutrality. They always try and sneak regulatory burden in any of these bills.
[+] didibus|5 years ago|reply
I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad.

I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

[+] lmilcin|5 years ago|reply
I have worked with Deloitte on two projects (on clients' side, I would never willingly work for Deloitte). Generally expect them to do absolute minimum necessary to meet the contract. The code was obfuscated to ensure support contract. The people employed to write it were not interested in the quality in the slightest but were interested a lot in specifics of the contract.
[+] jnathsf|5 years ago|reply
Deloitte is using salesforce for the underlying technology of the system. So almost all of this work is configuration and maybe some customization. We worked with another state that almost selected Deloitte but they bombed the no cost bake off and ultimately chose another vendor which completed 80% of the work during the bake off. This approach lowers risk dramatically and you’re able to set an aggressive and a more accurate launch date.

I hope we see more govts adopting bake-offs for buying IT.

[+] tootie|5 years ago|reply
They probably got sold on it being "off the shelf" and just a matter of customization and not coding. But the minute they run into edge cases (of which there are likely hundreds given the nature of the product) they just end up stuck.
[+] throwarayes|5 years ago|reply
Having seen Federal Govt contracting it’s easy to see how this works. And it’s a 2 way street of dysfunction:

- Many fed agencies have little tech savvy or ability to evaluate solutions

- Feds can have complex, mandated processes for software procurement, development and deployment that must be followed. This drives up the cost and limits the companies that choose to bid.

- There’s not much of a “get sh*t done” attitude. There’s little consequence for the individuals that managed a failure on both sides. Usually “oh well, another day at the office!” is the norm for many federal agencies/contractors.

- buzzword chasing tech ignorant administrators that consulting firms readily take advantage of - “yes! But does it have machine learning!?”

- there’s many fiefs at these agencies that must be navigated: “Oh! Wow - you spent 6 months building and it’s time to deploy? The security team really hates CYZ tech and we’ll have to escalate this to the director”

- so many g-d stakeholder to please and woo: from managers to administrators to political appointees and congresspeople. The relationship cost can be staggering.

These stereotypes aren’t always the case - some govt groups can actually execute really well. But many of these points dominate sadly.

[+] joncrane|5 years ago|reply
>Wow - you spent 6 months building and it’s time to deploy? The security team really hates CYZ tech and we’ll have to escalate this to the director

This right here! The "cyber" folks always throw a wrench in the works at the last moment. They rule by FUD and also have literally ZERO idea how stuff works. There's an old saying about teaching, but these days, and especially in government contracting, it applies to "cybersecurity."

"Those who can, do. Those who can't go into security."

[+] andrewmcwatters|5 years ago|reply
> Why was Deloitte awarded the project on a no-bid basis? The contracts claim the company was the only “responsible source”[1] to build the tool.

[1]: https://www.acquisition.gov/far/6.302-1

[+] stefan_|5 years ago|reply
Deloitte is a whole lot better manipulating and sweet talking the government than it is at building anything, that is for sure.

Surely their track record should see them permanently disqualified?

[+] et-al|5 years ago|reply
Usually government agencies write the contracts in a way that prevents other parties from bidding to streamline the procurement process.

Now what I don't understand, is why wasn't the contract written so the state could refuse broken software, or to get a partial refund if it's doesn't meet the specifications.

Someone got wined and dined pretty well. And government needs better lawyers when dealing with private contractors.

[+] snypher|5 years ago|reply
"because configuration of the VAMS application is occurring using Deloitte’s propriety GovConnect platform. Therefore, no other contractor has rights or access to leverage the system to carry out O&M activities"
[+] jbnorth|5 years ago|reply
I think that usually comes down to the contract being so specifically tailored for a company that they're the only one who could possibly bid and meet the specs for it.
[+] FireBeyond|5 years ago|reply
Which is horseshit (though we know that). I know of another, far smaller, leaner consulting firm that has handled for at least half a dozen states a contact tracing ecosystem tying into DOH, and utilizing the mobile APIs.
[+] yalogin|5 years ago|reply
This kind of looting is common place and is built into the federal contracting situation. After this 48 million or so they will get more to ix the bugs and maintain it. The maintenance is the most lucrative aspect of the contract.

Unfortunately this corrupt behavior is deeply rooted and condoned and expected and there doesn't seem to be much motivation to come up with a solution.

Wonder why the government doesn't just staff up a tech wing themselves instead of outsourcing. There is so much to do and its much more efficient to just create an tech department that serves all branches of government.

[+] antipaul|5 years ago|reply
But then that tech department would become government itself, and would that be better?

Maybe, maybe not.

Though rare, there _are_ private success stories, such as healthcare.gov 2.0 [1] and Palantir, though the latter gets in hot water for some tangential reasons. But maybe figure out how these happened, and scale that.

[1] https://www.theatlantic.com/technology/archive/2015/07/the-s...

[+] oivey|5 years ago|reply
But you see, something something, core business government isn’t tech, something something, private contractors are more efficient.
[+] jandrewrogers|5 years ago|reply
I’ve done a lot of work for multiple national governments on multiple projects. There are two facts that can be true at the same time:

- The government often grotesquely overpays for mediocre product delivered by vendors without holding them accountable in a meaningful way

- The government process drives up costs to an insane level independent of implementor good faith efforts and strongly selects for companies good at the process rather than execution

These are two different problems, and both of them manifest with any large contract. I would make the observation that it is easier to hide mediocre product and execution when you can hide behind a grotesquely inefficient and mediocre government process.

What transpired here should never happen. But I know why and how it happened.

[+] mkl95|5 years ago|reply
Deloitte is well known among devs in my country for the poor working conditions of their employees (including unpaid overtime and low pay) and the overpriced contracts they sign with public bodies. They do things like charging €2M to build a broken CRUD app written by devs fresh out of uni. They then follow up with a "maintenance contract" that consists of not doing much to fix it at all, and replacing the people who are not fresh out of uni anymore with cheaper devs.
[+] martin-t|5 years ago|reply
Incompetence is a source of extreme inefficiency in modern society. People making decisions don't understand the things they're deciding. This is broken. And there's nobody sane in a position where they can say "STOP, this is dumb" and demote people to the level of their competence or fire them.

A harmless example is movie / tv show plots not making sense - there's nobody willing to call out bullshit like obvious plot holes. This is how GoT went from a masterpiece of scheming characters driven by their motivation, making decisions based on information they had and dealing with the consequences to "she kinda forgot".

A dangerous example is the government being full of people who are too incompetent, stupid or useless to get a real job. Sometimes the number of them in the system is so large that competent people can't get anything done to fix things.

We need to do something to fix this. Positions of decision making power should require an IQ and/or knowledge test.

[+] tekkk|5 years ago|reply
I agree with your point but i would also want to point out this has been a problem since the dawn of man. Governments and organizations have been plagued by unending bureucracy and corruption that grows worse the longer it has been in existence. Humans have a tendency to protect their belongings or ego more than altruistically help others so it makes sense.

Democracy has, as a general idea, been a way to solve this problem by creating a counterbalance that should prevent important decisionmakers from being overly selfish and at least show they are being effective and altruistic. But that system isnt really applicable for smale scale power dynamics. Unless efficiency, or in some cases having just common sense, is not monitored by anyone aka. there are no consequences for making poor decisions sooner or later no one will pay attention and they will be doing whatever they _think_ is best (or easiest) but without reality checks.

But how to evaluate performance when dealing with abstract things? At least in the case of GoT the ratings did deteriorate so one could point out a problem in the system (albeit way too late, IMO s6 was when it should have been fixed). In other cases no one can see the damage they are doing (especially if they have never been really doing anything) so how to even tell it needs to be fixed? I think best cure to this problem is a great culture where being efficient is praised and rewarded. But changing already terrible culture, well, that's tough...

(Interesting film on this matter is Ikiru by Kurosawa)

[+] fasteddie31003|5 years ago|reply
The levels of incompetence in government is embarrassing. The government definitely does not attract the best and brightest. However, they control ~30% of the US GDP. I need to make a website that list all the government fails. Looks like http://usagovernmentfails.com/ is available.
[+] singingfish|5 years ago|reply
OK, so the fundamental problem with health IT is a greater problem in the commercial software industry as a whole. The buyer tends not to be the person who uses the software, and the more bureaucratic/ technocratic the process you use to get to the end user the more likely you are to not meet their needs. Put crudely and inaccurately, the procurement team end up going with the consultants who do a good game of golf.

Add government and a big consulting firm like Deloitte in health IT, and you'll be ready for a world of pain. Here [0] is a good paper on the topic.

[0] https://pubmed.ncbi.nlm.nih.gov/22188347/

[+] Ericson2314|5 years ago|reply
I would check out https://pedestrianobservations.com/ for information on other but analogous cost disease.

Basically, no, it's not just a meme. American managerial culture (or even Anglosphere at large, as alleged there) really does suck. And no amount of crafty mechanism design gets around the fact that least exploitable system seems to be one where the civil service isn't maimed and the work is done in-house.

Basically government should fund some free software for various general purposes (or just do a decent UBI), and then any "consulting" should basically be public sector and free software sector exchanging best practices.

The higher end academic consulting (ARPA) is legit[1] and should still exist, but it shouldn't be defense specific, and like drug bounties basically should exist to smooth over the "research...development...product" gap --- development is severely sort-changed current since outside of DARPA and ARPA-E so much research that doesn't contribute to someone's rent-seeking agenda just lies wasted.

[1]: I suppose should say I once was an intern at https://galois.com/

[+] endisneigh|5 years ago|reply
It’s also amazing to me how much a custom skin of Django Admin costs.

Has there been any research into what needs to happen to align government administrators incentives with the population? So much waste - why isn’t the government forcing Deloitte to fix?

I have no familiarity with government contracts, but as far as I know with Big Tech contracts they’re definitely written to favor the Big Tech company in these types of scenarios. The government should take a page from that playbook.

[+] throwaway3699|5 years ago|reply
Cutting taxes to half might force these guys to become more competitive. I don't see any incentives for the government to give up (I'm sure this benefits their friends at these firms) until forced to make hard decisions.
[+] temporallobe|5 years ago|reply
This is not atypical for US government projects. In 2012 I was brought in to “rescue” the UI for a huge multi-contractor system that was supposed to replace an aging legacy system written 40+ years ago.

I worked with the architect tirelessly for almost 3 years, trying to implement various changes but at its core the new system was highly flawed and suffered from various team changes and reboots. The UI/UX “expert” was not a developer and had very little technical background, and was therefore unfamiliar with basic HTML and CSS concepts. For some reason they let end-users dictate style and behavior aspects of the application. For example, the legacy system consisted of green-screen terminals and all the associated keystrokes (think ancient POS systems you still see in some retail stores). The vocal minority of these end-users demanded that we replicate this interface exactly instead of building it from the ground up as a modern best-practices UI using the latest web idioms and paradigms.

It gets worse. At some point someone important said Javascript is evil, and we were to avoid using it as much as possible, despite the application using several embedded and third-party Javascript libraries.

I ended up leaving that project for a slightly less insane one, but the pay was good. Anyway, I heard about a year later that the entire project (a 10-year effort that cost tens of millions of dollars) was abandoned. They continued using their greenscreens but I heard they hired a company to build an interface that reads these screens and presents a modern HTML facade over there screens that operates them kinda like a proxy.

You can’t make this stuff up.

[+] curtis3389|5 years ago|reply
I tend to side with that vocal minority on UI design.

I remember watching my friend who was a manager at a Domino's back in the day place an order for a pizza with their old POS system that was driven by key strokes, and it was ridiculously fast and easy. A pizza was a list of characters that described it.

Fast forward a decade, and I was tapping away at a touch-screen POS system for a smaller pizza chain, and it was woefully slow in comparison for even the most basic pizzas, and more error prone because Anchovies and Artichokes were right beside each other.

If you ever have a user-base where time=money, they will want hotkeys for everything.

[+] schnable|5 years ago|reply
> “The health-care software industry is enormous, and it exists largely because it’s privatized, it’s not standardized,” says Stone. “There are a lot of free-market inefficiencies. And the country doesn’t have a public health infrastructure, so there isn’t any real drive to fix it.”

> “You think about the industries that have been transformed by technology—someone said, How do we get a pizza to your house faster? That’s a competitive advantage,” he says. “That has not happened in American health care.”

The pizza industry is not a exactly standardized industry with public infrastructure either... the failure of VAMS (and healthcare.gov before it, and the unemployment systems, etc) shows that single government solution is not a fix for broken procurement and project management.

[+] alexpetralia|5 years ago|reply
At some point, someone should compile a bullet point list of all these consulting train wrecks on a simple website, like www.consultingfails.com.

Then, whenever you hear another multi-million dollar catastrophe, add it to the list. The brand damage will accrue - as it should - over time.

[+] S_A_P|5 years ago|reply
In my field (ETRM) I’ve worked with Deloitte on a couple of occasions. I have seen this sort of nonsense on more than one occasion.

A specific example that comes to mind was a project a few years back where Deloitte stuck 4 junior consultants along with 5 senior consultants and spent a couple million dollars for 12 months and were unable to produce even one business scenario in the system of choice. They did however produce reams of documents showing how it should work. The software vendor was called back onsite and I happened to be in the right place at the right time. A team of 3 of us ended up doing full implementation in 5 months. That included about 20 scenarios from front to back office along with foreign business.

[+] rainyMammoth|5 years ago|reply
I have worked with Deloitte type consultant in the past. It was exactly what I was expecting. A couple 20-something dressed up in suits, really good at talking and bullshitting high level visions.

Very little technical knowledge overall. It was a disaster.

[+] leephillips|5 years ago|reply
I could have made a broken website for less than half of that.
[+] mikesabbagh|5 years ago|reply
Companies like deloitte sees the CDC as a big cow that needs milking. They will never finish the job. It is like going to the hospital and tell the doctor I have a great insurance. The Doctors will do every possible procedure. It is better to check. CDC needs to employ developers and do the job internally.