[randomswede]

Oh, boy, on the "never log expected failures as errors" front, I once worked with a database system that used opportunistic transactions. Basically, each modification to a row carried effectively the original value of that trow with the update and if it failed, the API call triggered an error saying that the transaction failed. So if you did a "SET column=(column+1) WHERE rowid=unique", the client could basically do an automatic retry.

But, it also logged each and every occurrence of this at "Error" severity, instead of at "Info" severity (it is, after all, expected to happen once in a while).

And of course, once our code switched over to using this, the first few times every team member had to deal with a production issue, the immediate reaction was "oh, no, the data store is unhealthy! look at this mass of error logs, I can see one every few minutes!". Thankfully, after the first team member (me, as it happens) spent half an hour reading the relevant parts of the design and implementation docs, we could frequently short-cut a lengthy investigation by "oh, you think $DB is bad because you are seeing transaction failures? no, that's expected, see $URL".