🗂️ AI & Agents · View mindmap

Foundation Model

Large-scale models trained on massive datasets that can be adapted to a wide range of downstream tasks.

Recent Developments

• Jamba 1.7 (ai21-labs):
  • Utilizes a unique hybrid SSM-Transformer architecture.
  • Supports an expanded 256k context window.
  • Available in both Mini and Large flavors.
• Large-Scale Diffusion Models (google-deepmind):
  • Insights from Sander Dieleman on building large-scale diffusion models for image and video generation.
  • Focuses on technical architectures and intuitive scaling principles for generative media.
  • See: Dieleman’s DeepMind Insights: Building Large-Scale Diffusion Models.
• Google TabFM: Groundbreaking Zero-Shot Foundation Model for Tabular Data:
  • Introduced by Google as a specialized foundation model for tabular data.
  • Demonstrates groundbreaking zero-shot capabilities, challenging previous assumptions about machine learning limits for structured data.
  • Highlighted in recent analysis by AI with Surya as a significant breakthrough in the field.

References

• Google TabFM: Groundbreaking Zero-Shot Foundation Model for Tabular Data