NemoClaw Knowledge Wiki

Kiki K2 - Prompt Engineering

5 min read

https://www.youtube.com/watch?v=lDXc-zVqN1w The video provides a detailed overview of Moonshot AI’s Kimi K2 model and its research capabilities, then compares various AI research agents, including Gemini, ChatGPT (OpenAI o3), Grok DeepSearch, and Manus, on a specific benchmarking task. Here’s a breakdown of the key points:

  1. Kimi K2 and Kimi-Researcher Overview: Kimi K2: Moonshot AI’s latest Mixture-of-Experts model with 32 billion activated parameters and 1 trillion total parameters. It achieves state-of-the-art performance in knowledge, math, and coding, and is specifically optimized for agentic tasks, meaning it “does not just answer, it acts.” Kimi-Researcher: An autonomous agent excelling at multi-turn search and reasoning. It performs an average of 23 reasoning steps and explores over 200 URLs per task. It achieved state-of-the-art results on Humanity’s Last Exam (26.9% Pass@1) and Pass@4 (40.17%).

  2. Kimi-Researcher’s Architecture and Training: It’s a single agentic and thinking model designed to solve complex problems through multi-step planning, reasoning, and tool use. It leverages three main tools: a parallel, real-time internal search tool, a text-based browser tool for interactive web tasks, and a coding tool for automated code execution. All three can run in parallel. End-to-End Agentic Reinforcement Learning (RL): Kimi-Researcher is trained holistically using RL. Given a query, it explores possible strategies, receives rewards for correct solutions, and learns from full trajectories. This approach helps it handle long-horizon tasks and adapt to changing tools and environments, unlike traditional workflow-based or imitation learning systems. Context Management: Crucially, Kimi-Researcher employs a context-management mechanism that retains important information while discarding unnecessary documents, extending single rollout trajectories to over 50 iterations (compared to 10 for naive agents). This mechanism leads to 30% more iterations and higher performance.

  3. Emergent Agentic Capabilities: Resolving Inconsistencies: When presented with conflicting information from multiple sources, Kimi-Researcher resolves inconsistencies through iterative hypothesis refinement and self-correction. Caution and Rigor: Even for seemingly straightforward questions, it deliberately performs additional searches and cross-validates information before answering, showing a cautious behavior uncommon in LLM-based systems.

  4. Benchmarking Kimi K2 Hosting Providers: The presenter tasked various AI research agents to benchmark public hosts of the open-source Kimi K2 model, requiring discovery of API endpoints, pricing, tokens per second (TPS), context window length, and quantization details, ultimately needing a tabulate Markdown table and a unified coding benchmark suite.

  5. Comparison of AI Research Agents: Gemini: Provided a comprehensive table listing Moonshot AI, Groq, DeepInfra, Fireworks AI, Together AI, OpenRouter, and Hugging Face. It included pricing, access type, free tier info, TPS (with some unknowns), context window, and quantization (mostly unknown). It also generated hypothetical results tables for code generation, comprehension, and tool calling, along with synthesis and strategic recommendations. ChatGPT (OpenAI o3 with DeepSearch): Also generated a similar table of providers including SiliconFlow and Replicate, providing detailed information on pricing, TPS (often with notes like “very low” or “~200 TPS”), and quantization (e.g., FP8, Int8). It was noted that some TPS figures seemed off or providers might be hallucinated. Grok DeepSearch: Did not provide a table but listed information about providers like Moonshot AI, Novita AI, Groq, Together AI, Fireworks AI, DeepInfra, OpenRouter, and Hugging Face. It also failed to provide quantization details for most. Perplexity: Generated a table similar to ChatGPT, including pricing, TPS (with some unknowns), context window, and noted free tiers. Manus: Produced an HTML report with a comprehensive table including Groq, Together AI, Fireworks AI, Moonshot AI, Parasail, and Novita. It also generated the Python code for the benchmark suite. Kimi-Researcher (Moonshot AI’s own agent): Produced an interactive website report with well-formatted sections. It identified Moonshot AI, Together AI, DeepInfra, OpenRouter, Groq, Fireworks AI, Hugging Face, Novita AI, and Parasail. It detailed access, pricing, and technical specs, notably including Chinese web links in its search process, indicating a broader search scope. However, it often couldn’t find specific TPS or quantization details.

Key Takeaways from the Comparison:

  • Comprehensiveness: Gemini and ChatGPT offered the most comprehensive reports in terms of identified providers and data points.
  • Data Accuracy/Completeness: While Gemini was good, ChatGPT sometimes provided potentially hallucinated or inaccurate TPS/quantization data. Kimi-Researcher also had gaps in this specific data.
  • Reporting Format: Kimi-Researcher’s interactive website report was visually impressive and well-structured, akin to Manus’s executable reports.
  • Search Scope: Kimi-Researcher’s ability to utilize and cite Chinese web sources is a unique and valuable feature.
  • Agent Behavior: Kimi-Researcher exhibited cautious and rigorous behavior by cross-validating information, a desirable trait for research agents.
  • Free Tiers/Access: Gemini appeared to identify the most number of DeepSearch uses for free accounts, making it a potentially cost-effective option for extensive research.

The presenter concludes by recommending testing Kimi-Researcher, highlighting its pleasant conversational style and the potential for a subsequent video testing the identified Kimi K2 providers based on quantization levels.

Graph View