[CrimpCity]

The paper you shared is WAY more interesting than the original post.

Maybe this is a naive criticism but if the goal of the model is to "emulate the correct reasoning function" by which is implied that it learns prepositional logic rules then shouldn't the model have some architecture that tries to model generic logic rules and then using that score the input data and predict the result?

Currently it seems like the model is simply scoring each predicate since each of it's reasoning layers performs one step of forward chaining, adding some predicates to the Proved Facts.

My naive (and way undercooked) way of conducting this experiment would have been to try to use a GAN where the generator model comes up with new predicates and then the discriminator model that tries to classify them as real/fake. I would then try to train an MLP on top of the generator to classify the result and swap out the MLP depending on the sampling method basically the generator model becomes the pre-trained thing we care about to see if it can generalize.

Another nitpick is that the authors claim that BERT has enough "capacity" to solve SimpleLogic BUT this isn't actually what they want to achieve since solving != learning reasoning. So it feels like a bait and switch since IMO if a model has the capacity for something then it has some architectural aspect to it that can be used to show that it can achieve a smaller version of what you want and they didn't prove BERT can learn ANY prepositional logic rule.