Researchers propose a recurrent language modeling approach using balanced-tree recursive operators that achieves logarithmic computational depth and linear runtime, offering an alternative to Transformers with fixed depth and quadratic complexity. Experiments show the model extrapolates well to longer sequences and approaches Transformer performance with ALiBi.
A researcher explores whether gzip, a standard compression utility, can perform language modeling by leveraging the mathematical equivalence between compression and prediction. Using beam search over byte sequences scored by gzip's compression length, they demonstrate that gzip can generate text continuations that show understanding of source material, despite producing imperfect output.