This paper introduces Complex KDA (CKDA), an enhancement to Kimi Delta Attention that improves the expressivity of linear RNNs while maintaining efficiency. By extending parameter ranges and combining delta-rule transformations with channel-wise gating, CKDA can model 2D rotations and track finite groups, achieving competitive performance on length extrapolation and language modeling tasks compared to Transformers and other linear RNNs.
This article explains Transformer inference, contrasting it with training by introducing latency as a key consideration. It describes how naive token sampling is computationally expensive (O(n²) to O(n³)), but can be optimized using a KV cache to reduce complexity to O(n) to O(n²), enabling efficient sequence generation through separate forward passes for each token.