[stephenroller]

Support for this was also added to [Fairscale](https://fairscale.readthedocs.io/en/latest/) and [Fairseq](https://github.com/pytorch/fairseq) last week. In particular, the Fairscale implementation can be used in any pyotrch project without requiring the use of the Deepspeed trainer.

[diptanu]

What are the relevant commits in Fairseq for this? I couldn't figure out the changes by looking at the commits from last week.

[stephenroller]

https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327