[fcharton]

For training, you need a generator because you want millions of solved examples for deep learning to work.

At test time, you usually want a test set from the same distribution as the training data (or at least related to it in some controllable way), or it becomes very difficult to interpret the results.

Suppose my test set come from a different and unknown distribution (real problems sampled in some way).

If I get good results, is it because the training worked, or because the test set was "comparatively easy"? If I get bad results, is it because the model did not learn, or because the test set was too far away from the training examples?