Gradient-Boosted Trees
Gradient-Boosted Trees (GBT) are an ensemble machine learning technique that builds predictive models in a stage-wise fashion. It generalizes boosting by allowing the optimization of any differentiable loss function.
Core Mechanism
- Ensemble Learning: Combines multiple weak learners (typically decision trees) to form a strong learner.
- Sequential Training: Each new tree is trained to correct the residual errors of the previous ensemble.
- Gradient Descent: Uses gradient descent to minimize the loss function, hence the name “gradient-boosted.”
- Regularization: Often includes shrinkage (learning rate) and subsampling to prevent overfitting.
Comparison with Foundation Models
While gradient-boosted trees have long been the state-of-the-art for structured/tabular data, the landscape is shifting with the advent of tabular foundation models.
- Traditional Approach: GBTs require task-specific training and feature engineering.
- Emerging Paradigm: New models aim to apply the “one model for everything” approach of large-language-models to tabular data.
- Google TabFM: A notable development in this space is Google TabFM: Zero-Shot Foundation Model for Tabular Data, which proposes a zero-shot foundation model for databases and spreadsheets.
- See Google TabFM: Zero-Shot Foundation Model for Tabular Data for detailed analysis.
- Unlike GBTs, TabFM aims to reduce the need for per-task training.
Key Algorithms
- XGBoost
- LightGBM
- CatBoost