Nunchux introduces VC-Attention, a training-free low-bit attention method that accelerates video generation by 1.6× on NVIDIA B200 compared to BF16 FlashAttention-4. The technique combines V-Smooth for reducing quantization error and ExpCast-FP8 for eliminating the softmax bottleneck, achieving higher fidelity than existing methods like SageAttention2.
SGLang and Miles add day-0 support for DeepSeek-V4.1, a model featuring low-ratio compression, sliding-window attention, manifold hyper-connections, and Engram memory for efficient serving. The implementation includes cross-layer sharing, sparse retrieval mechanisms, and host-memory placement optimizations that increase KV cache capacity by 36% while maintaining comparable throughput.
This paper introduces Effective State-Size (ESS), a metric measuring how well sequence models utilize their memory by analyzing the rank of input-dependent transformation matrices. ESS reveals that models with high memory utilization are harder to distill, and that effective ESS modulation correlates with better performance on recall-intensive tasks.
Singapore launched a five-year national campaign to encourage reading habits by offering small monetary rewards to citizens who log reading sessions on a government website. The initiative uses gamification and aims to combat declining attention spans and the rise of shallow online content consumption, though it has sparked debate about whether financial incentives can truly foster a genuine love of reading.
Article explaining LLM architecture fundamentals, focusing on the transformer model and attention mechanism. Covers how transformers parallelize computation compared to RNNs, and how attention allows tokens to dynamically reference all previous context. Includes code examples and notation for understanding embeddings, queries, keys, and values.
A 2018 essay critiques Waldenponding—the philosophy of retreating from technology to reclaim attention—arguing it's counterproductive both personally and collectively. The author contends that staying plugged into information flows and managing attention actively is preferable to unplugging, and that retreat exploits Fear Of Being Ordinary (FOBO) just as social media exploits Fear Of Missing Out (FOMO).
Research increasingly shows that short-form video consumption is associated with cognitive harms, particularly affecting attention, impulse control, and memory, even as social media companies dispute the scientific evidence. A meta-analysis of 70 studies found stronger associations between short-form video use and negative cognitive changes than with mental health issues, with compulsive use predicting the worst outcomes.
The author reflects on what they initially perceived as internet and smartphone addiction, ultimately concluding they are actually addicted to information discovery. They describe how algorithmic recommendations and short-form content fail to deliver the meaningful, life-changing information they once found online, and express nostalgia for the exploratory internet of the past.
A technical handbook explaining FlashAttention, an optimization technique that accelerates transformer attention mechanisms through tiling, online softmax, and recomputation without approximating the mathematical function. The key insight is that wall-clock speed depends on GPU memory traffic rather than FLOP count alone, achieved by reducing expensive reads and writes to high-bandwidth memory.
OpenArch is a PyTorch repository containing hand-written implementations of modern LLM architectures designed for educational clarity rather than production performance. Each model is implemented from scratch in a single readable file, making architectural choices like attention types, normalization methods, and positional encodings explicit and easy to compare across 72 different architectures.
MoBA (Mixture of Block Attention) is a novel attention mechanism for long-context LLMs that applies Mixture of Experts principles to reduce computational complexity while allowing models to autonomously determine attention patterns. The approach enables seamless transitions between full and sparse attention and has been deployed in Kimi's long-context system.
Recurrent Looped Transformer (RLT) combines a causal encoder with a recurrent decoder that grows temporal depth with each token, traversing tL_D decoder blocks after t tokens while maintaining fixed per-token computation. The architecture integrates global key–value memory, layerwise sliding-window attention caches, and continuous latent computation across prompts and responses, with co-design considerations for hardware efficiency and reinforcement learning scaling.
Neural Turing Machines couple neural networks with external memory resources accessed via attention mechanisms, creating a differentiable system analogous to traditional computing architectures. The approach enables end-to-end training with gradient descent and can learn algorithms like copying, sorting, and associative recall from examples.
Recurrent Looped Transformer (RLT) combines a causal encoder with a recurrent decoder that grows temporal depth with each token, traversing tL_D decoder blocks after t tokens while maintaining fixed per-token computation. The architecture integrates global encoder memory with layerwise sliding-window attention caches in the decoder, enabling co-design with hardware and RL algorithms through shared state transitions across pretraining, fine-tuning, and sampling.
Researchers demonstrate that transformer networks can function as universal computers by programming specific weights and looping the architecture. Using input sequences as instructions and memory, they show that shallow transformers can emulate computing blocks like branches and function calls, enabling execution of algorithms including calculators, linear algebra operations, and backpropagation-based learning.