Fredrik Gustafsson: Reliable regression model uncertainty under real-world distribution shifts

Regression is a fundamental machine learning task with many important applications within computer vision and other domains. In general, it entails predicting continuous targets from given inputs. Within medical imaging, accurate regression models have the potential to automate various tasks, helping to lower costs and improve patient outcomes. Such safety-critical deployment does however require reliable estimation of model uncertainty, also under the wide variety of distribution shifts that might be encountered in practice.

Motivated by this, we set out to investigate the reliability of regression uncertainty estimation methods under various real-world distribution shifts. To that end, we propose an extensive benchmark of eight image-based regression datasets with different types of challenging distribution shifts. We then employ our benchmark to evaluate many of the most common uncertainty estimation methods, as well as two state-of-the-art uncertainty scores from the task of out-of-distribution detection.

We find that while methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and the proposed benchmark therefore serves as a challenge to the research community. It demonstrates that more work is required in order to develop truly reliable uncertainty estimation methods for regression.

Fredrik K. Gustafsson is a PhD student at Uppsala University, supervised by Thomas Schön and Martin Danelljan. He will soon be defending his thesis entitled “Towards Accurate and Reliable Deep Regression Models”. His general areas of interest are machine learning and computer vision. His research focuses on probabilistic deep learning, and often includes regression problems, uncertainty estimation methods or energy-based models. He received his MSc in Electrical Engineering from Linköping University in 2018.