[gmuslera]

Reminded me of pi filesystem (https://github.com/philipl/pifs), with enough digits of pi precalculated you might be able to do a decent compression program. The trick is in the amount of reasonable digits for that, if it’s smaller or bigger than that trained LLM.

[GuB-42]

I suspect that the length of the offset of your input data in pi is equal to the length of the input data itself, plus or minus a few bytes at most, regardless of the size of the input data.

That is: no compression, but it won't make things worse either.

Unless the input data is the digits of pi, obviously, or the result of some computation involving pi.

[luke5441]

See http://www.faqs.org/faqs/compression-faq/part1/section-8.htm...

[gmuslera]

What if instead of the index of your full data, you store the index of smaller blocks? Would I need i.e. to use an 8kbytes or larger integer to store the offset all the possible 8k blocks?

It is meant to be a joke anyway.

[noctune]

Some patterns must happen to repeat, so I would assume the offset to be larger, no?

[MrLeap]

You could express the offset with scientific notation, tetration, and other big math number things. You probably don't need the whole offset number all at once!

[miki123211]

In that vein, pi is the true "illegal number"[1], as all the illegal content ever produced can eventually be found in the digits of pi.

I guess when God made the Universe, he filled it with CSAM, hate speech and unsolicited Viagra ads...

[1] https://en.wikipedia.org/wiki/Illegal_number

[anthk]

Pi is not bound to God neither the universe. It's just the ratio between a distance and adding up all the existing points around you at that distance.

[sltkr]

I'm going to be the nerd that points out that it has not been mathematically proven that pi contains every substring, so the pifs might not work even in theory (besides being utterly impractical, of course).

On a more serious note, as far as I understand these compression competitions require that static data is included in the size computation. So if you compress 1000 MB into 500 MB, but to decompress you need a 1 MB binary and a 100 MB initial dictionary, your score would be 500 + 100 + 1 = 601 MB, not 500 MB.

The relevance to this discussion is that the LLM weights would have to be included as static data, since the only way to regenerate them is from the initial training data, which is much larger than the resulting model. By comparison, pi based compression is the other way around: since pi is a natural constant, if your decompressor requires (say) a trillion digits of pi, you could write a relatively small program (a few kb) to generate them. It would be terribly slow, but it wouldn't affect your compression ratio much.

[dataflow]

> I'm going to be the nerd that points out that it has not been mathematically proven that pi contains every substring

Fascinating. Do you know if this has been proven about any interesting number (that wasn't explicitly constructed to make this true)?

[meindnoch]

If we assume pi's digits to be uniformly random, then the expected offset for the first occurrence of a particular N-bit sequence is going to be ~2^N. (This can be proven using a Markov-chain argument. Also note: we're working in binary). So you've converted an N-bit value into an offset on the order of 2^N, which takes again N bits to represent.

[eru]

> I'm going to be the nerd that points out that it has not been mathematically proven that pi contains every substring, so the pifs might not work even in theory (besides being utterly impractical, of course).

Well, either your program 'works', or you will have discovered a major new insight about Pi.

> On a more serious note, as far as I understand these compression competitions require that static data is included in the size computation. So if you compress 1000 MB into 500 MB, but to decompress you need a 1 MB binary and a 100 MB initial dictionary, your score would be 500 + 100 + 1 = 601 MB, not 500 MB.

And that's the only way to do this fairly, if you are running a competition where you only have a single static corpus to compress.

It would be more interesting and would make the results more useful, if the texts to be compressed would be drawn from a wide probability distribution, and then we scored people on eg the average length. Then you wouldn't necessarily need to include the size of the compressor and decompressor in the score.

Of course, it would be utterly impractical to sample Gigabytes of new text each time you need to run the benchmark: humans are expensive writers. The only way this could work would be either to sample via an LLM, but that's somewhat circular and wouldn't measure what you actually want to measure in the benchmark, or you could try to keep the benchmark text secret, but that has its own problems.

[netsharc]

You mentioning the concept of pi containing every substring makes me think of Borges' Library of Babel.

Ha, next: a compression algorithm that requires the user to first build an infinite library...

[charcircuit]

This only does 1 byte, so you only have to prove it contains the bits for 0-255.