Being able to compress well is closely related to intelligence as explained below. While intelligence is a slippery concept, file sizes are hard numbers. Wikipedia is an extensive snapshot of Human Knowledge. If you can compress the first 1GB of Wikipedia better than your predecessors, your (de)compressor likely has to be smart(er). The intention of this prize is to encourage development of intelligent compressors/programs as a path to AGI.
Interview with Lex Fridman (26.Feb'20) (Video, Audio, Tweet)
- Create a Linux or Windows compressor comp.exe of size S1 that compresses enwik9 to archive.exe of size S2 such that S:=S1+S2 < L := 100'424'672 (previous record).
- If run, archive.exe produces (without input from other sources) a 109 byte file that is identical to enwik9.
- If we can verify your claim, you are eligible for a prize of 500'000€×(1-S/L). Minimum claim is 5'000€ (1% improvement).
- Restrictions: Must run in ≲50 hours using a single CPU core and <10GB RAM and <100GB HDD on our test machine.
- Discussion forum on the contest and prize
- Compression benchmarks enwik9 and others
- Motivation of compressing the Human Knowledge
- Information about the enwik9 data file
- Wikipedia on the Hutter Prize
- 13??: William of Ockham's razor: Entities should not be multiplied beyond necessity.
- 1964: Ray Solomonoff introduced algorithmic probability for universal prediction.
- 1996: Leonid Broukhis introduced the first compression competition with a prize.
- 2000: Marcus Hutter introduced a compression based universal intelligent agent.
- 2005: Jim Bowery proposed a larger scale compression contest based on the Wikipedia corpus.
- 2006: Matt Mahoney compressed Wikipedia with many state of the art compressors.
- 2006: Marcus Hutter launched the 50'000€ prize.
- 2006-2017: Alexander Rhatushnyak is 4-times winner of the HKCP.
- 2020: Marcus Hutter launched the 500'000€ prize.
- 2021: Artemiy Margaritov is the first winner the 10x HKCP.
- Jim Bowery: verification of claims, public relations, finding sponsors, newsgroups, etc.
- Matt Mahoney: running the compression competition.
- Marcus Hutter: arbiter, current sponsor, and manager of prize fund.
- What is this contest about?
- Is the compression contest still ongoing? (YES)
- Why did you grant a temporary relaxation in 2021 of 5'000 Byte per day?
- Where do I start? How do I develop a competitive compressor?
- What is (artificial) intelligence?
- What does compression has to do with (artificial) intelligence?
- What is/are (developing better) compressors good for?
- The contest encourages developing special purpose compressors
- Why lossless compression?
- I have a really good lossy compressor. (How) can I participate?
- Why aren't cross-validation or train/test-set used for evaluation?
- Why is (sequential) compression superior to other learning paradigms?
- Why is Compressor Length superior to other Regularizations?
- How can I achieve small code length with huge Neural Networks?
- Batch vs incremental/online/sequential compression.
- Why don't you allow using some fixed default background knowledge data base?
- Why is "understanding" of the text or "intelligence" needed to achieve maximal compression?
- Why do you focus on text?
- What is the ultimate compression of enwik9?
- Why recursively compressing compressed files or compressing random files won't work
- Can you prove the claims in the answers to the FAQ above?
- The PAQ8 compressors are hard to beat
- There are lots of non-human language pieces in the file
- Why include the decompressor?
- Why do you require submission of the compressor and include its size and time?
- Why not use Perplexity, as most big language models do?
- Why did you start with 100MB enwik8 back in 2006?
- Why did you go BIG in 2020?
- Why are you limiting (de)compression to less than 100 hours on systems with less than 10GB RAM?
- Why do you restrict to a single CPU core and exclude GPUs?
- The total prize is not exactly 500'000€
- Is 1GB 2^30 byte or 10^9 byte?
- The website looks dated
- Why do you require Windows or Linux executables?
- Why do you require submission of documented source code?
- Where can I find the source code of the past winners and baseline phda9?
- Under which license can/shall I submit my code?
- What if I can (significantly) beat the current record?
- How can I produce self-contained or smaller decompressors?
- Is Artificial General Intelligence (AGI) possible?
- Is Ockham's razor and hence compression sufficient for AI?
- The human brain works very differently from (de)compressors
- I have other questions or am not satisfied with the answer
The time for decompression/compression is estimated for a 2GHz P4 till 2010 and for a 2.7GHz i7 since 2017. The percent (%) improvement is over the baseline previous record. More details on the (de)compressors can be found here.
- Apr-Nov'17: Alexander Rhatushnyak submits another series of ever improving compressors based on phda9, with the final one on 4.Nov'17 improving over his previous record by over 4%!
- Sep'07-...: Alexander Rhatushnyak submits another series of ever improving compressors. Is there nobody else who can keep up with him?
- Nov'06-May'07: Alexander Rhatushnyak submits another improved series of (de)compressors paq8hp6-12 (option -7). On 14.May 2007 he submits paq8hp12 It achieved an improvement of 3.5% over the new baseline paq8hp5 and was finally confirmed as the second winner on 30.June 2007. Congratulations! A detailed description of paq8hp12 can be found here. Most of the time in developing paqhp6-12 went into planning and performing experiments, and studying and understanding the results of these experiments. Alexander Rhatushnyak's current occupation is in software engineering. For him data compression is science and art and sport all together. This was his motivation for participating in the contest. Dr. Rhatushnyak was born in the Siberian Scientific Center (www.nsc.ru), studied data compression and related algorithms since 1991, and graduated from the Moscow State University (www.msu.ru) in 1996. After his PhD in 2002 he lived and worked in various places in the world.
- Aug-Sep'06: Alexander Rhatushnyak of the Moscow State University Compression Project submits an improving series of (de)compressors paq8hp? (option -7), modifications of paq8h with a custom dictionary built from enwik8 and other improvements. Przemyslaw Skibinski contributed to earlier versions. On 25.Sep.2006 Alexander Rhatushnyak submits paq8hp5. It achieved an improvement of 6.8% over the baseline paq8f and was finally confirmed as the first winner on 25.Oct.2006. Congratulations! A detailed description of paq8hp5 can be found here.
- 16.Aug'06: Dmitry Shkarin submits a modification of (de)compressor durilca (option -m1650 -o21 -t2), a modification of ppmd/ppmonstr with filters for text, exe, and data with fixed length records.
- 16.Aug'06: Rudi Cilibrasi submits (de)compressor raq8g.cpp (option -7), a modification of paq8f with additional text modeling.
- Wikipedia: Hutter Prize
- Large Text Compression Benchmark
- Interview on Intelligence & Compression & Contest (10min, video)
- Presentation by past winner Alex Rhatushnyak
- Kolmogorov complexity = the ultimate compression
- Universal Artificial Intelligence (book, 45min/1.5h/3h lecture)
- Interview on Universal AI with Lex Fridman (1.5h)
- Franz&al. (2021) A theory of incremental compression
- Zenil (2020) Compression is Comprehension, and the Unreasonable Effectiveness of Digital Computation in the Natural World
- Yogatama&al. (2019) Learning and Evaluating General Linguistic Intelligence
- Zenil&al (2019) Causal deconvolution by algorithmic generative models (3min video)
- Everitt&Hutter (2018) Universal Artificial Intelligence: Practical agents and fundamental challenges
- Mattern (2016) On Statistical Data Compression
- Mahoney (2011) Data Compression Explained
- Rathmanner&Hutter (2011) A Philosophical Treatise of Universal Induction (slides, recordings)
- Salomon&Motta (2010) Handbook of Data Compression
- Janzing&Schölkopf (2010) Causal Inference Using the Algorithmic Markov Condition
- Hernandez-Orallo&Dowe (2010) Measuring Universal Intelligence: Towards an Anytime Intelligence Test
- Mahoney (2009) Rationale for a Large Text Compression Benchmark (and further references)
- Hutter (2007) Universal Algorithmic Intelligence: A Mathematical Top→Down Approach (slides, recordings)
- Schmidhuber (2007) The New AI: General & Sound & Relevant for Physics
- Cilibrasi&Vitanyi (2005) Clustering by Compression
- Wallace (2005) Statistical and Inductive Inference by Minimum Message Length
- Sanghi&Dowe (2003) A Computer Program Capable of Passing I.Q. Tests
- Catoni (2001) Statistical Learning Theory and Stochastic Optimization
- Hutter (201X) Recommended books & Courses for (Under)Graduate Students
- Twitter Announcement (2023) Progress on the Human Knowledge Compression front
- HKCP GG Announcement (2023) Saurabh Kumar's fast-cmix wins €5187 Hutter Prize Award!
- Slashdot Informal Discussion (2023) Sixth 'Hutter Prize' Awarded for Achieving New Data Compression Milestone
- Mike James Article (2023) Hutter Prize Awarded Again
- Research Snipers Article (2023) New Record Set In Data Compression
- Slashdot Informal Discussion (2021) New Hutter Prize Winner Achieves Milestone for Lossless Compression of Human Knowledge
- Mike James Article (2021) New Hutter Prize Milestone For Lossless Compression
- Discussion (2021) on News YCombinator
- Analytics India Magazine Article (2020) Compress Data And Win Hutter Prize Worth Half A Million Euros
- Mike James Article (2020) Hutter Prize Now 500,000 Euros
- Reddit News (2020) 500,000€ Prize for distilling Wikipedia to its essence
- Language Modelling on Hutter Prize
- Discussion in the AGI mailing list
- Discussion in the Hutter-Prize mailing list
- Technical Discussion in the Data Compression Forum encode.su
- Discussion in Yahoo Group ai-philosophy
- Informal Discussion at Slashdot (13.Aug'06, 29.Oct'06, 10.Jul'07, 21.Feb'20)
- In the Online Heise News
- In the KurzweilAI.net News
- In Mark Nelson's blog (24.Aug'06)
- O'Reilly Radar (29.Sep'06)
- Discussion at the Accelerating Future page
- In the ebiquity news
- In WebPlanet News in Russian
- Wissenschaft-Wirtschaft-Politik, Ausgabe 34/2006 (22.Aug'06)
- Discussion at Newsgroup comp.ai.nat-lang
- Discussion at Newsgroup comp.compression and here
- Discussion at Newsgroup comp.ai
- Prediction market as to when enwik8 will be compressed to Shannon's estimate of 1 bit per character
- many other sites