The Profound Relationship Between Compression and Prediction, from gzip to LLM

AFFILIATE_PRODUCTS:

The Technical Resonance Between Compression

and Prediction

Unraveling the basic principles of data compression reveals a surprisingly deep relationship with the core technology of large language models. The ngrok.com blog post “Compression is prediction” clearly demonstrates this fact.

Generally, when people hear “data compression,” they think of reducing file size. There are methods like minification that simply remove redundant information. However, true compression based on information theory relies on “predicting” the inherent redundancy and regularity within the data.

The Three-Stage Structure of Compression

Technology

Modern compression tools generally consist of three stages. The first is the transformation stage, which preprocesses the original data into a format that is easier to compress. Next is the model stage, which learns and describes the statistical patterns and structure of the data. Finally, there is the entropy coder, which encodes the output of the learned model into the most efficient bitstream.

As a concrete example, consider the string “AAAAAAAAABBBBCCDAAADDDDDDDDD”. Using run-length encoding to convert it to “A9B4C2D1A3D9” shrinks 28 characters to 12. More advanced methods such as gzip and Brotli combine multiple techniques of this kind.

Principles of Model-Based Compression

The heart of a compression algorithm lies in its “model.” This is a mechanism that predicts data that may appear in the future. For example, a model compressing English text learns that after “th”, there is a high probability that “e” follows. Based on this prediction, it assigns short codes to matching parts and long codes to mismatched parts.

Huffman coding and arithmetic coding are techniques that convert this probabilistic model into an actual bitstream. If the probability predictions are highly accurate, the theoretical minimum number of bits can be approached.

AI Models Are Advanced Compressors

Large language models also essentially solve the same problem. They learn statistical patterns of text and predict the next token (word or subword). This is structurally identical to the task of a compression model predicting the next character.

The report on Lobsters, ngrok.com via gmem, emphasizes this point. The majority of LLM training data is about text compression, and the model’s internal representation can be said to be a sophisticated form of compression that captures the structure of the data.

Theoretical Foundations and Limitations

This relationship is not a mere analogy. According to the “common invariant theorem” of information theory, an optimal compressor and an optimal predictor are equivalent. As compression ratio improves, prediction accuracy also improves.

However, this equivalence also has constraints. Computational cost, memory usage, and the presence or absence of prior knowledge significantly affect practical performance. The remarkable performance of LLMs can be seen as the result of pouring vast amounts of data and computational resources into compressing (i.e., modeling) the structure of text with extremely high precision.

Editorial Opinion

In the short term, this insight has the potential to directly connect to the optimization of AI inference. Understanding the duality of compression and prediction clarifies the development guidelines for LLM distillation and quantization techniques. Part of prompt engineering can effectively be interpreted as an operation that compressionally retrieves the model’s internal knowledge.

In the long term, this relationship calls into question the essence of “intelligence.” Compression condenses information, and prediction unfolds the future. Could intelligence be the ability to build a compressed model of the environment and use it to predict the future? This perspective seems likely to become a key to re-evaluating the limitations and possibilities of current AI architectures.

As a question from the editorial department: Can the equivalence of compression and prediction be applied to higher-order cognition such as consciousness and understanding? Can the process by which the human brain processes information also be regarded as a kind of advanced compression-prediction system? This fundamental question seems likely to extend not only to the future of AI research but also to cognitive science and the philosophy of information.

References

• “Compression is prediction”, by ngrok.com via gmem — Lobsters, 2026-08-11T19:35:12.000Z (ARR)
• Source URL: https://ngrok.com/blog/compression-is-prediction