[elandau25]

It is important to note that these micro-models are only supposed to be used in the annotation process. During annotation there is a separate process for QA where there will be some form of human supervision. Micro-models are NOT supposed to be used for production environments.

100% agree on the healthcare front, which actually perfectly underlies the point here. These models are overfit to one specific modality but often used for generic purposes. One reason why it is important to define micro-models is to point them out when they are deployed in a live production environment, which I agree is very dangerous. Many healthcare models are truly not ready for live diagnostic settings. On the other hand, these same models often do perform well on assisting the actual annotation of new data when applied to the right domain and with appropriate human supervision. This is a perfect encapsulation of the distinction we are trying to make.

[fny]

You're missing my point. You can make a good micro-model that is very, very targeted and does not overfit. Sure, it won't generalize outside your target, but you know you can use it at clinic X with machine Y to predict Z.

This is why I'm saying you aren't describing overfitting but instead having a very, very specific objective function.