Kimi Team

Kimi Team is the research division of Moonshot AI dedicated to advancing large language model (LLM) architecture and computational efficiency. The team focuses on addressing fundamental challenges in deep neural network design, particularly those that emerge as models scale to greater depths and complexity. Their work centers on improving the stability and performance of transformer-based architectures through targeted architectural innovations.

Attention Residuals

A key contribution from Kimi Team is the development of Attention Residuals, a technique designed to address pre-norm dilution in deep networks. This method aims to mitigate the degradation of gradient flow and feature representation that often occurs in very deep transformer models. By modifying how attention mechanisms interact with residual connections, the approach seeks to maintain signal integrity across layers, thereby enhancing training stability and model accuracy in large-scale deployments.