There has been previous work on deep networks implementing Kalman filters, and another interesting aspect I remember is that unlike a traditional Kalman filter a network is able to maintain multiple hypothesis and so it is less likely to have some of the jittery behavior that a Kalman filter might have under unknown changes of motion, sensor noise, etc. I wonder if the softmax operation in a transformer block might lose this property though, as softmax does tend to push for a single answer.
No comments yet.