[jdm2212]

Error logs and monitors find out because the RPC fails, which causes an error to be logged and a failed request metric to be incremented. (The UncaughtExceptionHandler will do the same for stuff failing in background threads.)

If eventual consistency is important, enqueuing for retry has to be done BEFORE you try to write, not on write failure. If/when the write succeeds, you remove the item from the retry queue.