[whitfin]

tl;dr always make production setup the same as staging.

We once had a production-only crash inside our Elasticsearch setup (we had written plugins). On preproduction it was fine, development fine, etc. But on production it would crash with NoSuchMethodError coming from the Jackson library (I think it was floatValue()). After several hours and rolling back/forward, I took a shot that there was something clashing with the class in the JVM (which I didn't know was possible at the time as I was a fresh dev). After backtracing for a while, it turned out that our production ES environment had some AWS autodiscovery plugin installed. This plugin depended on the AWS SDK, which depended an AWS common library, which depended on a Jackson version just old enough to not have floatValue(). It appears that this was being loaded first, causing our plugin to crash because we were calling a method which didn't exist in their version.

Turns out that I actually filed a ticket for the SDK to update (https://github.com/aws/aws-sdk-java/issues/411) but I think we ended up just migrating away from floatValue() altogether. As a younger developer, this definitely opened my eyes to the need to ensure parity across test environments (I remember this happened whilst I was on a tz that had me trying to debug this at 4am).