For over fourty years, choosing a statistical model thanks to data consisted in optimizing a criterion based on penalized likelihood (H. Akaike, 1973) or penalized least squares (C. Mallows, 1973). These methods are valid for predictive model choice (regression, classification) and for descriptive models (clustering, mixtures). Most of their properties are asymptotic, but a non asymptotic theory has emerged at the end of the last century (Birgé-Massart, 1997). Instead of choosing the best model among several candidates, model aggregation combines different models, often linearly, allowing better predictions. Bayesian statistics provide a useful framework for model choice and model aggregation with Bayesian Model Averaging.
In a purely predictive context and with very few assumptions, ensemble methods or meta-algorithms, such as boosting and random forests, have proven their efficiency.
