Integrating NVMe Disks in HopsFS (HDFS)

[SirOibaf]

"HDFS and S3 are designed around large blocks (optimized to overcome slow random I/O on disks), while new NVMe hardware supports fast random disk I/O (and potentially small blocks sizes). "

Interesting solution presented here - keep the block size constant, but put the small files on NVMe disks.

[jamesblonde]

Disclaimer, one of the authors here. You can see a video with discussion of the NVMe support in HopsFS in this Spark Summit talk:

https://www.youtube.com/watch?v=tx6HyoUYGL0

[senorcarbone]

super cool. The world needs more success stories for NVM