[ramity]

I'll provide a contrasting, pessimistic take.

> How do you write programs when a bug can kill their user?

You accept that you will have a hand in killing users, and you fight like hell to prove yourself wrong. Every code change, PR approval, process update, unit test, hell, even meetings all weigh heavier. You move slower, leaving no stone unturned. To touch on the pacemakers example, even buggy code that kills X% of users will keep Y% alive/improve QoL. Does the good outweigh the bad? Even small amounts of complexity can bubble up and lead to unintended behavior. In a corrected vibrator example, what if frequency becomes so large it overflows and leads to burning the user? Youch.

The best insight I have to offer is that time is often overlooked and taken for granted. I'm talking Y2K data type, time drift, time skew, special relativity, precision, and more. Some of the most interesting and disturbing bugs I've come across all occurred because of time. "This program works perfectly fine, but after 24 hours it starts infinitely logging." If time is an input, do not underestimate time.

> How do we get to a point to `trust` it?

You traverse the entire input space to validate the output space. This is not always possible. In these cases, audit compliance can take the form of traversing a subset of the input space deemed "typical/expected" and moving forward with the knowledge that edge cases can exist. Even with a fully audited software, oddities like a cosmic bit flip can occur. What then? At some point, in this beautifully imperfect world, one must settle for good enough over perfection.

The astute reading above might be furiously pounding their keyboards mentioning the halting problem. We can't even verifiably prove a particular input will provide an output - moreover an entire space.

> I am convinced that open code, specs and (processes) must be requirement going forward.

I completely agree, but I don't believe this will outright prevent user deaths. Having open code, specs, etc aids towards accountability, transparency, and external verification. I must express I feel there are pressures against this, as there is monumental power in being the only party able to ascertain the facts.

[fjfaase]

I know someone who developed medical devices, not as critical as pacemakers, and he kind of boasts that he probably killed (as in caused premature death of) some people, but also extended the life of many many more.

[kitd]

Tbh, it's the same kind of survival economics as invasive treatments like surgery anyway. No doctor can or should guarantee 100% survival.

[wcunning]

That’s actually the standard in automotive and industrial applications — likelihood of failure vs consequences of failure, set the “acceptable” risk low and show proof that you’re not any higher than that level. Medical devices actually have a much higher “contributes in any way to any patient harm” risk analysis.

[nullc]

https://www.viruscomix.com/page588.html