This research introduces two methods to convert token logits to byte logits and conducts a large-scale study comparing byte-based and token-based language models across scaling dimensions. The study finds that while token models perform better initially, byte models eventually surpass them with increased compute, achieving superior data efficiency and performance on downstream tasks.
This article examines how foundation models are increasingly trained on synthetic data generated by other models rather than human-curated data, describing this recursive improvement process across five key components: judges, corpora, teachers, curricula, and environments. The shift from human-generated to model-generated training data represents a fundamental change in how state-of-the-art language models are developed, with each generation producing better synthetic data for training the next.
Magic has developed a pretraining recipe that is over 10x more compute-efficient than leading open-weight models, achieving DeepSeek V4 Pro Base performance with ~50x fewer FLOPs (approximately $0.5M on GB200). By scaling up 10x to $4M, the model meaningfully outperforms all publicly available open base models on perplexity evaluations, demonstrating that algorithmic efficiency can enable frontier model development without access to massive compute clusters.