[bagrow]

> by filtering any "books" (rather, files) that are larger than 30 MiB we can reduce the total size of the collection from 51.50 TB to 18.91 TB

I can see problems with a hard cutoff in file size. A long architectural or graphic design textbook could be much larger than that, for instance.

[mananaysiempre]

While it’s a bit of an extreme case, the file for a single 15-page article on Monte Carlo noise in rendering[1] is over 50M (as noise should specifically not be compressed out of the pictures).

[1] https://dl.acm.org/doi/10.1145/3414685.3417881

[TigeriusKirk]

I was just checking my PDFs over 30M because of this post and was surprised to see the DALL-E 2 paper is 41.9M for 27 pages. Lots of images, of course, it was just surprising to see it clock in around a group of full textbooks.