that looks interesting. we spent a lot of money on FSxL and might save a lot with Weka. Unfortunately, our data access pattern is very random and will likely not benefit from caching unless we cache the entire dataset (100TB)
One thing that we’re considering for these kinds of use cases is providing the ability for users to either (a) request that their data always stays in the hot storage or (b) provide the ability for users to ask us to pre-load their data sets when they begin analysis (and I expect that our large fleet of instances can do this preloading much quicker than any individual instance download). Would either of those options make the product more helpful for you?
huntaub|1 year ago