top | item 35651752

(no title)

aras_p | 2 years ago

I can't comment on the linked github issues, but looking at the turbopfor icapp.c sources, it looks like the blosc2 bytedelta is not used correctly there: the expected order is "shuffle, then bytedelta", but icapp.c does "bytedelta, then shuffle". That might explain the poor result you get :)

discuss

order

powturbo|2 years ago

Thanks for your response. Unfortunatelly blosc benchmark data is not easy accessible and you're not publishing the exact numbers in your survey. This makes it difficult to compare the results. The data in my benchmarks are directly downloadable and is a proposal from academia covering diverse distrutions.

I've now reverted the order to "shuffle, then bytedelta". It looks better but still inferior to TurboPFor floating point compression and is also 35% slower in decompression.

TurboPFor FP with lz4 instead of zstd,15 has similar ratio than blosc+bytedelta+zstd,15, but again it is a lot faster.

Please, publish the exact numbers of your survey at least those with zstd,15. The zfp results in my benchmark don't look good. Are the numbers similar to yours?