[mlucy]

There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: https://arxiv.org/pdf/1810.04805.pdf

[stared]

Thanks for this reference, I will look it up. Though, from my experience people in NLP still (be default) train from scratch, with some exceptions for tasks on the same dataset:

- https://blog.openai.com/unsupervised-sentiment-neuron/

- http://ruder.io/nlp-imagenet/

[samcodes]

This is true, but rapidly changing. In addition to fine tuneable language models, you can do deep feature extraction with something like bert-as-service [0] ... You can even fine tune Bert on your days, then use the fine tuned model as a feature extractor.

[0] https://github.com/hanxiao/bert-as-service