[cabernal]

This and the John Deere bug posted earlier make me a bit concerned over the accumulating evidence of unreliable software ruining people's lives...

What can be done? Mandatory audits, pen testing?

If this is an organizational problem, more vacation? limiting overtime? rethinking employee incentives?

[danpalmer]

Pentesting and auditing aren't great solutions here. They can be useful on small scopes but a big system like this, it's unlikely to be hugely impactful – it will find things, but who knows if it finds enough.

In the UK in the wake of the 2008 banking crisis, a number of positions in banks became criminally liable for issues under them. If you're director-level or above (I think?) then you may be ultimately put in prison for negligence or issues like that which occur in your department. This is rare, not sure if it's been used yet, but it effected a cultural change in consumer banking as a bunch of execs suddenly had their necks on the line if someone under them did something wrong. I don't believe this is too hard-line in practice, I think a defence is "look at all these reasonable steps we take, we couldn't have foreseen this", but it had the impact (source, a good friend of mine is bordering on this level in a UK bank).

I wonder if a similar thing could work in a wider way across more industries - not with the intention of criminally punishing lots of people, but with the aim to change the culture around responsibility to the public and other stakeholders in the work that we do.

[viraptor]

> What can be done?

Not taking software results as a fact. Software report stating X in court should be equivalent to "the person who wrote this in a hurry would say X, but it's not a sworn testimony".

We should have the person presenting any report like that be personally responsible for the contents. If they aren't willing, it shouldn't be presented.

[Silhouette]

We should have the person presenting any report like that be personally responsible for the contents. If they aren't willing, it shouldn't be presented.

I don't think making it personal works at scale. You can't reasonably expect everyone giving evidence in court, say every individual police officer who is a witness to a speeding offence, to be a technical expert on the technological tools they are given to do their job.

Instead, as you implied in the previous paragraph, the weight given to any evidence derived from technology should be proportionate to the credibility of that technology. If it's a device that has to be vetted and approved according to strict regulatory standards and in court there are two other concurring sources of evidence, that's clearly a much stronger case than a single reading from a single device whose calibration has reasonably been called into question at trial that is being presented as the only evidence in that trial.

[icegreentea2]

It's not about positive incentives, it's about the lack of negative incentives. More true negative incentives need to be shifted onto the production side, back onto the corporations, its officers, its middle management, and if required down to the individual contributor.

Corporate structure helps diffuse and deflect responsibility. Each group (executive leadership, middle management, and ICs) gets to diffuse and deflect responsibility and liability onto each other.

We already have all the positive incentives in the world - cash money. It's not enough.

[Chris2048]

Standards. Just say certain things, payment systems, need to meet certain levels of auditability (does it record all relevant data, and can I see them after the fact), verification (is the data correct and can I prove that) and privacy.