Invited talk: Hyperparameter learning in data assimilation systems

MANUEL PULIDO

Reading

Workshop; ECMWF-ESA Workshop on Machine Learning for Earth System Observation and Prediction; 2020

ECMWF (European Centre for Medium-range Weather Forecasts) - ESA (European Space Agency)

Data assimilation combines different information sources using a quantification of the uncertainty of each source to weight them. Therefore, a proper consideration of the uncertainty of observations and model states is crucial for the performance of the data assimilation system. Expert knowledge and expensive offline fine tuning experiments have been used in the past to determine the set of hyperparameters that define aspects of the prior, model error and observational uncertainties and in particular their covariances. In recent years, there is a paradigm shift with several Bayesian and maximum likelihood methods that attempt to infer these hyperparameters within the data assimilation system. In this talk, I will give the foundational basis of these methods which rely on the assumption of slow variations of the hyperparameters compared to the latent state variables. An overview of maximum likelihood methods including the online and batch expectation-maximization algorithm, gradient-based optimization and a  Bayesian hierarchical method based on nested ensemble Kalman filters will be discussed. Finally, some experiments to estimate stochastic parameterizations  that compare the methods in a proof-of-concept dynamical system will be shown.