[dingosity]

Once long ago I worked for a small company doing unified messaging as their Security Architect. In that job it meant I was a software engineer with domain experience in implementing cryptographic protocols. I was well prepared for this role due to my experience at RSA Data Security (when RSA was a middleware vendor) and Certicom. If you've used an ATM machine in the US in the last 30 years, you've likely used the modular exponentiator I wrote for POWER/PowerPC.

So here I am at this small company that's running out of runway. I'm meeting with my boss, let's call him Mark (not his real name). We're discussing what features we can add in the next three months before half the company is laid off. At the top of the list is the device-key-regen feature that would allow the user (through an arcane series of difficult-to-accidentally-perform user inputs) to regenerate a key and re-submit it for certification. It required changes to the device code, the security gateway code and to the manual. Mark is adamant we should not do this. It would "distract us from other high-value tasks." I then explain the devices will eventually brick themselves without some way to update, or at least re-certify device keys.

Fast forward a month or two and I get laid off. No one buys the IP, so it's no great loss. Just one more project on the burning pile of money called Silicon Valley.

Seven years later I'm working at a company in Seattle well known for a ebook reader that rhymes with "Bindle." Their first-gen devices have a problem. In three weeks, the certs for the device keys will expire, effectively disallowing them from authenticating themselves to network services. I get called in because I had the words "certificate" and "X.509" on my resume. It turns out the manager of the team who built the device software was my old boss Mark. The one thing he remembered is that allowing devices to update (or at least re-certify) keys requires changes to a lot of different software and there was no down-side to not doing it last time.

It is true... there was no down-side to not adding this functionality last time. The company died before anyone other than a smattering of companies could see our IP, and the few beta-users we recruited stopped using the product before the device key certs expired.

Back at the company in Seattle whose logo is half of a smiley face: after MANY failed updates, they eventually pushed out an update that re-certified the existing keys (i.e. - it submitted the public key to a third-party registration authority, authenticating the message with the device's private key.) Not a perfect solution, but given that they waited until about three days before devices started failing, not the worst in the world.

But there was a three-day window in which the devices needed to be on, notice there was a software update and convinced their users to press the "install update" button. Some large number of first-gen devices were not even turned on during this period. When they were later turned on, there was no way to get them the update that allowed them to connect to online services.

The solution was just to tell customers they would be given a new ebook reader. Given the engineering time invested and the number of new ebook readers shipped out, our team estimated it cost the company between $40M and $60M.

My old boss Mark was promoted.

I think my point is... with the possible exception of a company whose devices prominently feature a fruit-based logo, no large company understands what a key-pair or certificate is and how it relates to operational processes which enhance their products' security or trust. And I like to kvetch.