top | item 13069775

People’s freedom jeopardised by new software adopted by California’s courts

260 points| dct | 9 years ago |bbc.co.uk | reply

125 comments

order
[+] noir_lord|9 years ago|reply
Software is currently in a strange place legislatively, in the 18th century Civil Engineering exploded (sometimes literally) and the number of disasters went up in radically (for an interesting case https://en.wikipedia.org/wiki/Tay_Bridge_disaster), over time professional standards bodies grew alongside the maturing industry to both ensure that people working in the industry where adequately trained and adequately protected from external factors (people pushing them to do things cheaply, skimp on designs etc).

Given that we've handed a large part of the running of the modern world over to computers, software and the people who write that software I think at some point we need to start looking at a similar system, now the usual argument to this is "We don't want regulation" or "It's not my code, it's other peoples" but that doesn't alter the reality that this isn't going to get better until the way the industry is run changes, it's not a software issue, better tooling won't save us (it might help though) and there is no perfect outcome.

EDIT: It's worth noting that if as an industry we don't work with our customers on this stuff and reach an accord that fits everyone eventually one will be imposed on us from outside, Software Engineering doesn't exist in isolation, other fields of Engineering are covered already.

[+] jdietrich|9 years ago|reply
We need formal codes of practice, we need good institutions, but we also need a cultural change.

We no longer have the luxury of saying "oh, it's just a CRUD app, it doesn't matter if things go wrong". Software has become too important, it has become too deeply intertwined in our daily lives. Errors and leaks from trivial little apps can have life-changing consequences for users.

If your software handles personally identifying data, there's a non-zero chance that you could ruin someone's life through negligence. Legal case management software might be an obvious example, but it's the tip of a very big iceberg.

If you have several million users, one-in-a-million edge cases are going to happen constantly. Weird things happen at scale. We need to treat every Android permission and every Facebook API call as a potential matter of life or death. We need the personal courage and the support of our peers to say "no, I will not implement that feature", "no, I will not store that user data", "no, I will not transmit that in plaintext", "no, I will not commit that to production without unit testing".

https://en.wikipedia.org/wiki/Iron_Ring

[+] pjungwir|9 years ago|reply
We already have a taste of these rules and regulations, for example HIPAA, PCI, and FIPS-140. I'm not saying I disagree with you, but keep in mind that those are all considered quite burdensome, and it's hard to see how that could be different. Also FIPS-140 is a good example of a standard that names specific technologies and so is doomed to lag behind the state of the art. Just be careful what you wish for! But as you say, it seems inevitable for more rules to be imposed eventually, one way or another, so perhaps we should think about how we might write better ones.
[+] pmontra|9 years ago|reply
There are cases where this approach would be very welcome. However it probably won't solve the problem of "a cumbersome user interface was causing the time taken to update a record to jump from around one minute to as much as 30 minutes per entry." That's bad design with maybe the collaboration of the customer who didn't understand the problem or pushed/spent enough to fix it. Not different from building a bridge too narrow to make two cars pass at the same time: perfectly safe but not functional enough. Engineer's fault, customer's fault, both? In the case of this software who's going to be sued, the developer or the customer? The customer by the people in jail and maybe the developer by the customer?

However, let's also try to compare the complexity of that software system with the complexity of a bridge. What would be more complex, that Tay Bridge or California's Odyssey? I bet on the latter and I assume that it would cost much more money and time to properly build and test software systems. It's almost only manpower, compared to concrete, steel and wood, but the costs could go up by one order of magnitude compared to what they are now.

If this approach gets mandated it would be a big slow down for every industry needing custom software. It also means we'll only need 1/10th of customers to keep our shops open, the 10% with more money. Goodbye to the small ones.

There will be the equivalent of a construction site manager, but good luck with the bugs that become apparent or are introduced after years of changes to the first accepted delivery of the project. Bridges are much more static than software.

[+] dispose13432|9 years ago|reply
The threat model of code is different than that of bridges, and legislation won't help.

Civil engineers have to make sure that bridges won't naturally collapse.

Software "engineers" have to ensure that no one will break their code, from the privacy and anonymity of their house, with millions of dollars in payoffs.

If bridges held millions of dollars and would be accessible from around the world, they would be breaking all the time also.

[+] jancsika|9 years ago|reply
Position one: implement professional-training program and whistleblower-protections for employees of the out-of-state companies writing the proprietary software for CA court system

Position two: require court system software to be open source

You might reply that open source is no panacea, and that's true. But the vulnerabilities in current software are wholly different than the ones in civil (physical) engineering-- both your imagined "professional grade" proprietary court app and my imagined open source court app will have critical vulnerabilities. The question is what happens when they are found. I'd like to read a blog entry complete with screen captures by a security expert who is running the exact same software, posted after the exploit has been responsibly reported and addressed. I do not want to read an opaque statement from a company blaming its users, and I don't see how even a set of strong, effective software engineering guidelines is going to get you anything else in this case.

Edit: I said "vulnerabilities", but the logic applies equally to usability issues, data corruption, etc.

[+] gravypod|9 years ago|reply
> professional standards bodies grew alongside the maturing industry > ensure that people working in the industry where adequately trained and adequately protected from external factors > software and the people who write that software I think at some point we need to start looking at a similar system

Who this gets implemented I'll quit and become a welder or something else that's outdoors and filled with less insane propositions. The BS and stupidity that gets setup by these standards organizations is crazy.

I know many people who are PEs who can testify that all of this is completely useless. It's often said by people who give these certifications that you shouldn't go into industry before getting these. That's acceptable until you hear why: you'll forget a lot of stuff that is unused in industry but used in these exams. This is horrible design at it's best.

"Engineering Standardization" in this form is good for a few things:

   1. Driving up the cost by keeping perfectly good engineers out of the field and making the industry talent starved.
   2. Creating a corrupt standardization organization that makes it so their "group" can get certificates while others cant.
   3. A reduction to the speed of new ideas entering the field. Less people allowed in all of which are brainwashed into "THIS IS THE WAY, YOU WILL OBEY" 
   4. A lack of care for checking work; "They're a professional engineer, you know better then me"
   5. Preserving irrelevant information that isn't at all related to the task at hand (A+, I'm looking at you)
I think the real solution is for universities to actually teach people how to program, about software architecture, about software engineering, about maintenance, and less about things that you won't necessarily be doing. I'm in senior level classes at college and I find that most of my peers have problem "doing programming" and I know people in the EE track who have problems "doing electronics". These aren't stupid people, they are just lacking instruction needed to succeed in the field.

Another solution is to have a simple standard: All life endangering software must either be a) formally proven by the programmers and checked by a mathematician or b) have unit testing for every possible case of every testable portion of the code.

If both of these are done then it will save far more lives then useless standards that will be actively ignored. These two standards are already where our field is naturally going and embracing that will only yield higher quality software for everyone. Using stupid standards documents that specify things like "functions can only be 80 chars wide and 100 lines long" or "no memory allocation after startup"

It also avoids forming unnatural accreditation bodies that are harmful to the industry.

Edit: I'd also be perfectly fine with something like UL for software.

[+] 0xdeadbeefbabe|9 years ago|reply
Can one of the standards be that if the software costs money it should perform proportionally better than a free alternative?
[+] dkarl|9 years ago|reply
Similar problems have been reported in Tennessee and also in Indiana - where prosecutors have had a perhaps more troubling issue of inmates being mistakenly released early.

It seems backwards to describe this as more troubling.

[+] strictfp|9 years ago|reply
Better sorry than safe?
[+] kag0|9 years ago|reply
Keep in mind this is a UK site. People are guilty until proven innocent, so from that perspective guilty people going free may indeed seem more troubling.
[+] briandear|9 years ago|reply
Not at all. Do you want a rapist getting out of jail earlier? That would be troubling.
[+] awinter-py|9 years ago|reply
Separate from the question of software liability is the court's responsibility to not make egregious errors.

Historically, courts have been unwilling to assign themselves blame for screwing up. Judicial immunity is untouchable in american case law because it's hard to find a judge willing to rule against it.

The standard of due process is high for 'life and limb' cases but low for misdemeanors & traffic violations. When Fixed discovered that most SF parking tickets are challengeable, SF didn't fix the problem -- they turned off their fax machine to make it harder to challenge tickets. https://techcrunch.com/2015/10/12/fixed-the-app-that-fixes-y...

NYC issued fake parking tickets to the tune of hundreds of thousands of dollars. http://iquantny.tumblr.com/post/144197004989/the-nypd-was-sy...

Jurisprudence doesn't have a concept of 'bulk miscarriage of justice'. You could put NSA surveillance in this camp too. There's a star trek line about genocide which says 'we have no law to fit your crime'. That's where we are with petty crimes mishandled in bulk.

[+] danso|9 years ago|reply
> The software, created by Texas-based Tyler Technologies, costs about $5m (£4m) and is set to gradually replace a decades-old e-filing system that looks like something a hacker would use in a Hollywood movie.

> Tyler Technologies acknowledged in a statement that the upgrade process had been “challenging” - but said poor training was to blame for bad inputting of data and integration with third-party applications that often introduce glitches into the system.

Even as someone who spends the majority of the day at the command prompt, I agree we should always be attentive to user-interface issues. But I've been less than optimistic that people will have the wisdom to know that a new "modern" interface automatically means it's more sophisticated/elegant, or that it's more attuned to the needs of human users.

In particular, it's been alarming to see a rise in unnecessary use of AJAX (nevermind Angular) in government applications. I'm not anti-JS, it's just that client-end development seems to have far more moving parts of the kind that don't get well-tested by workers in a bureaucracy. Especially when that work has been farmed out.

edit: as an example of how government can do modern web-dev/UX well, I can think of no better (at least at the U.S. federal level) example than the CFPB: https://cfpb.github.io/

[+] panic|9 years ago|reply
Tyler Technologies acknowledged in a statement that the upgrade process had been “challenging” - but said poor training was to blame for bad inputting of data and integration with third-party applications that often introduce glitches into the system.

People writing software need to take responsibility for how the software is used, especially when it can impact people's lives to this degree. You can't just blame third party software or the people entering data.

[+] noir_lord|9 years ago|reply
> People writing software need to take responsibility for how the software is used

There is an old joke that it's almost impossible to get someone to understand a problem that would result in a threat to their livliehood.

I don't think people need to "take responsibility" I think organisations need to be made to take responsibility.

In the UK when you purchase a physical good from the retailer the warranty for that good is between you and the retailer, the retailer then has to deal with problems up the line from suppliers, it's not perfect but it's workable.

I think software should be the same, I think if I pay Foo for a system then Foo should be responsible for the system even if it's made of parts from Bar, Fizz and Buzz, if Foo has an issue with Fizz they need to take it up with Fizz.

There is so much crap software out there in every field and the acceleration towards a world run on software continues.

[+] dkarl|9 years ago|reply
I think that's a bit harsh without knowing whether the software vendor was allowed to be effectively involved in the training and integration efforts. Software like this depends on lots of processes around the rollout and usage being properly designed and executed, and the fact that the software is being used successfully in many places suggests that the vendor knows how to deploy the software safely. Whether they were complicit in not doing so in the case of Alameda County is the important question. Maybe the county ran out of money at the last minute and the vendor couldn't stop them from cutting corners on the rollout. Maybe the vendor doesn't think it's good customer relations to go public with the fact that they told the county over and over again that they were creating risks of errors like this. Maybe they even had stipulations in their contract to stop improper integrations from happening and the government ignored them or found a way around them.

Hopefully there will be an investigation and the people will find out who screwed up and why so they can be held accountable. It's tempting to say that everyone involved is "responsible" in some vague way, but that lets people off too easy. If everyone is guilty, then no one is guilty.

[+] CmdrSprinkles|9 years ago|reply
Except, what is the threshold? Are MS responsible for every time a 20 year old machine running windows 98 because nothing else can run the data collection software crashes? Should I give Linus a ring every time my aunt breaks her laptop that I installed Mint on?

Assuming the software itself has no meaningful or significant bugs (a hard thing to prove), it likely IS a problem of training. I know that my travel request time went from 5 minutes to closer to an hour for the first few times I used it after we switched to a new system. And while the admins were trained properly, the rest of us just were told "Ask an admin if you need help"

Similarly, it might actually BE the third party systems. I know that every time we upgrade part of our infrastructure we have to deal with the hell of components that should, but don't, interface well. And as often as I scream profanities and talk about how much I hate a certain vendor, I also know it isn't their responsibility to ensure that their software works with a different vendor's (I do yell at our procurement people though since they should make it so).

Do I think Tyler Tech should try to help? Of course, this is bad PR. And if they were the ones providing the training, address what went wrong and update the training and possibly GUI to resolve this. But so long as they met the requirements of the contract, I see no reason they should be held responsible or penalized.

[+] kafkaesq|9 years ago|reply
You can't just blame third party software or the people entering data.

Indeed, this excuse just doesn't pass the smell test. Responsible vendors do (and not only "do", they advocate and insist on) practices like integration testing, and graduated rollouts (so that when process glitches like these inevitably slip through, they're caught when they affect 1 or 2 people -- not 50).

Never do they blame "bad inputting of data."

[+] nkrisc|9 years ago|reply
I agree that you can't blame the software but disagree with who should be liable. The entity using the software needs to be liable (in this case either the State or the courts). They are responsible for implementing and using the the software thus ought to be liable for any mistakes it makes, whether user error or software bug. If they then want to take that up with the creator of the software, that's between them.
[+] wtbob|9 years ago|reply
Although I don't have any direct evidence regarding this instance, I wouldn't be surprised if the old system was greenscreen & form-based, while the new one is some kind of shiny Java-backed web app. It wouldn't surprise me at all if the old system was faster to use and less error-prone: those old greenscreen apps tended to be optimised for long-term use, rather than for showing off in a board-room demo.
[+] hitgeek|9 years ago|reply
having worked in data entry on a green screen app, as well as the .Net, Windows Form "upgrade", I agree with this sentiment.

the green screen app was highly optimized for efficient and accurate data entry without use of a mouse. The modern alternative looked better, but a large portion of the functionality was much less efficient in comparison.

the only redeeming factor of the new application, was it allowed much of the manual data entry to be automated. This however required considerable time, technical knowledge, and industry contacts to develop and implement, which not all organizational users possessed.

[+] daveleebbc|9 years ago|reply
That's exactly the difference. Mr Woods (the defender I quote in the story) described the old system as something a computer hacker would use in a Hollywood movie... but it worked and was stable.
[+] ptaipale|9 years ago|reply
The headline is somewhat clickbaity and sensational.

Yes, it is bad that there are clerical errors in justice system.

But, if information is lost due to faulty software or user errors or even user error helped by bad UI design, it's still fundamentally just a clerical error. Those errors should be fixed and perhaps some people should be eligible for compensation for being mistreated due to error, but there is no sinister "software is putting people in jail" plan here. Just errors.

Embarrassing ones that should be fixed at a priority.

[+] andrewla|9 years ago|reply
It definitely seems like more of the blame here should be shouldered by the police and the courts, who, knowing that the system has problems (and hopefully assured that those problems will be resolved in the future) should put less faith in them, and double-check (against filed papers, for example) potentially dubious results.

This is more expensive in terms of people's times, but it's just part of the cost of adopting the new software, and should be treated as such, possibly by billing the software vendor for the additional manpower required to work with the software during the transition.

[+] avmich|9 years ago|reply
The point here is perhaps that some errors are less forgivable than others.

May be in the field like this some extra care should be taken to ensure that, despite possible clerical errors, the data entered is sufficiently accurate. Technically that's likely extra checks and more thoughts about clearness of UI. "Just clerical error" doesn't sound like enough justification in cases like this - software should actively help to avoid those errors, and do enough of that help.

[+] mtgx|9 years ago|reply
> “With the old system, it took maybe one or two clicks to complete a process,” she said. “Now it takes 25 clicks, and there are drop-down boxes and all of that.”

> Because the system is so unwieldy, clerks are unable to enter data in the courtroom, she said, so that burden has fallen on other office workers. It’s created a backlog of more than 12,000 files that have not been uploaded — and that number is growing by up to 300 files a day, according to Woods.

Sounds like it's mostly a software design problem to me, rather than "clerk/user stupidity" issue:

http://www.sfchronicle.com/bayarea/article/Alameda-County-s-...

[+] eponeponepon|9 years ago|reply
I expect the truth of this lies somewhere in the outsourcing industry. It has every smell of miscommunicated requirements and half-assed implementation.

If I'm right, I doubt it'll ever be admitted to though.

[+] briandear|9 years ago|reply
I wish I could upvote this a hundred times. I am currently dealing with an Indian dev shop and the code has been atrocious -- as if they didn't even read the requirement. Doing a very basic Stripe integration has taken nearly a week and it was still incorrect. I could provide days of examples.
[+] tremon|9 years ago|reply
Not a system upgrade, but a faulty system. To which branch of the government should the court system be accountable?
[+] cafard|9 years ago|reply
Umm, the courts?

I knew various government techies who went to work for the Administrative Office of the [US] Courts. Honestly, I don't know what the chain of command was there.

[+] FrancoDiaz|9 years ago|reply
Holy Cow! There needs to be some serious consequences for those responsible for this travesty.
[+] threatofrain|9 years ago|reply
Perhaps I'm crazy, but isn't a $5 million contract a bit too low for an overhaul of the California justice software system?

Business-wise there's going to be a lot of things to cut, pushback against government asks for software, and a very skeletal plan for maintenance mode. I sometimes wonder how government models the businesses they do work with, or whether they work as hard as businesses in modelling the other side.

[+] fataliss|9 years ago|reply
I don't get why such software isn't an open source initiative. I wish the government would give more legitimacy to orgs like "code for america". Who wants a proprietary closed janky software to rule wether you are a criminal or not? Nobody, that's who. The only beneficiaries in that story are the people on the other hand of the contract, making $5M for an half baked piece of software!
[+] splicer|9 years ago|reply
While I certainly agree that all government software should be open source, it sounds like a big part of the problem here was a failure in gathering accurate requirements (which is a very expensive endeavor).
[+] forgotpwtomain|9 years ago|reply
I don't see why people in the comments are blaming tech company (Tyler Technologies) for this. There was certainly someone over-seeing procurement and specifications for the government and it was their job to make sure that the product which was procured and delivered was functional and ready to roll-out. This person and their department should be held responsible.
[+] kazinator|9 years ago|reply
> had recently finished a six-month drug programme after he was caught in possession of marijuana and ecstasy.

THAT is what fucking should not have happened in the first place.

Let's not shift the focus to some glitchy software issues that will likely get worked out.

[+] JumpCrisscross|9 years ago|reply
If you care about the welfare of your fellow Californians, consider sending a letter to your U.S. Congressperson, your California State Assemblyperson and Senator [1]. Attach this article as an exhibit. Copy your county court.

Then, and this is very important, set a reminder out one week and call each of those people, confirming they received the letter and understand your concerns.

If this is too much, either accept you don't care about the issue (that's fine) or, if you do, that you may have wrong attitudes about how citizen influence works in a democracy.

[1] http://findyourrep.legislature.ca.gov

[+] jamesvl|9 years ago|reply
I worked as a programmer for a smaller California county court system for about five years, and have seen something very much like this play out before (both in my county and others).

I can't comment on the Tyler product or their training directly; maybe they really are a rock star outfit. But if this is like past attempts, this project has all of of the worst of aspects of software development risks and none of our more "modern" methods to mitigate them.

The court employees - most of whom would _not_ be considered very computer savvy - probably had a lot of training directly with Tyler but are struggling with a system that a) doesn't meet their needs, b) changes years (decades?) of ingrained workflow habits and terminology, and c) may be much slower than what they used to have.

Observations from past projects like this:

* at it's heart it's a database CRUD app, but with hundreds of tables and thousands of fields and business "logic" encoded (in more database fields) to help with validation and workflow

* most of the above fields need to be fully customized for each county, so add in tables and logic to modify your UI on every screen

* this software was not built for Alameda county, but re-purposed from use elsewhere. Terms and concepts for how the law worked in the state this was originally built for may or may not apply here.

* "usability" success metric: "do all 50 fields on the page accept input and save data in less than 60 seconds?" (i.e. no concept of real HCI usability design at all)

* iteration process: waterfall. Vendor sits with court subject experts for 2-3 months, documenting all of the workflow. They customize their product to meet those needs, and a month later show a build that does this. Court can't use it yet (deployment locally would cost way too much), but they've printed out hundred of pages of screen shots to help document how it could be used. Hire external consultants to help with this process. Repeat until a) court money runs out or b) someone's reputation will be tarnished if the system doesn't launch

* There is no staging environment. Deployment is on local hardware only (no cloud). No bug tracker exists that the court can see. Builds are not automated, and "maintenance" may cost the court additional money.

* importing previous cases: worst ETL job you can imagine. Take data from an aging mainframe database that may or may not have any relational integrity at all, and try to plug it into a system as described in point 1

* administrative overhead: your county is given money from the state to do this, and then no choice about which vendor or software to use (because the state wants to roll this out in _all_ counties... each of which is very different from one another, even in CA)

tl;dr This is a horribly difficult software update, subject to the worst practices in our industry.

Personally, I don't think blaming court employees for "clerical errors" is fair at all - not that those haven't happened, but (from my experience) these are hard working people who care about justice yet have really lousy software that impedes their job.

I'd love to see a company do this software right - custom build, real iterative development hand in hand with the users. The Courts really needs it, they've never experienced a high quality product in this area, and the inefficiencies affect the wider economy (because civil matters are faster to resolve).

[edit: formatting]

[+] logfromblammo|9 years ago|reply
Tyler Tech is not a rock star outfit. They may think they are, but they are not.

I interviewed with them in 2008, and that remains, to date, the worst interview experience I have ever had. Everything they did before the interview seemed calculated to convince me to withdraw myself from consideration, and everything after seemed calculated to discourage anyone else I knew from applying.

So I felt a little frisson of schadenfreude from reading the article.

[+] thinkcomp|9 years ago|reply
This stems directly from the opacity of the California Judicial Council Technology Committee and the CCMS debacle it has generally made worse. Public comments are welcome, but secret.

http://www.courts.ca.gov/jctc.htm