[jakobe]

Mac OS X has something similar to this "Software HSM": the Keychain. You can put private keys in your keychain, and apps can use them for signing or encrypting, but they can't extract them. It's quite nicely implemented; when an app tries to access a key the first time, a dialog will pop up saying something like "Mail is trying to use key xyz for decryption. Do you want to allow?".

Of course, this requires using Apple's APIs, which are poorly documented and a pain in the neck even compared to OpenSSL. It's also not suitable for servers.

[DrStalker]

That wouldn't help when there is a bug that lets an attacker read your server's memory; you'd still need to reissue your certificates as a preventative measure because you couldn't guarantee that the bit of memory used by the software HSM hadn't been compromised.

[teacup50]

The keychain operates out-of-process.