Kimi Team

Kimi Team is the research division of Moonshot AI dedicated to advancing large language model (LLM) architecture and computational efficiency. The team focuses on addressing fundamental challenges in deep neural network design, particularly those that emerge as models scale to greater depths and complexity. Their work centers on improving the stability and performance of transformer-based architectures through targeted architectural innovations.

Attention Residuals

A key contribution from Kimi Team is the development of Attention Residuals, a technique designed to address pre-norm dilution in deep networks. Pre-norm dilution refers to a phenomenon where the normalization layers in deep transformer architectures can degrade attention mechanisms and overall model performance. By introducing residual connections specifically within attention components, the team aims to preserve information flow and maintain architectural stability as network depth increases.

Research Focus

The team’s broader research agenda encompasses efficiency improvements across model training and inference, optimization of transformer components, and investigation of architectural scaling laws. Their work contributes to Moonshot AI’s development of the Kimi LLM series and informs the company’s approach to building capable language models with improved computational characteristics.