seb

Sebastian Schmon,

DPhil (Oxon)

Welcome to my webpage. I am an Assistant Professor in Statistics at Durham University and a Research Scientist in the Research team at Improbable. My research interests lie at the intersection of statistics, machine learning and probability theory as well as applications to the sciences. In particular, I am interested in constructing and analysing practical algorithms and simulation techniques for complex statistical or machine learning models such as intractable Bayesian models or large scale simulation engines. A list of publications can be found below. I obtained my DPhil (Oxford parlance for PhD) at the Department of Statistics of the University of Oxford under the supervision of Arnaud Doucet and George Deligiannidis.

As a consultant with several years of experience I have supported start-ups as well as large multinational companies to improve their analytical capabilites.

Publications and Preprints

AnoDDPM: Anomaly Detection with Denoising Diffusion Probabilistic Models using Simplex Noise

with Julian Wyatt, Adam Leach and Chris G. Willcocks

NTIRE, New Trends in Image Restoration and Enhancement workshop, CVPR 2022

Generative models have been shown to provide a powerful mechanism for anomaly detection by learning to model healthy or normal reference data which can subsequently be used as a baseline for scoring anomalies. In this work we consider denoising diffusion probabilistic models (DDPMs) for unsupervised anomaly detection. DDPMs have superior mode coverage over generative adversarial networks (GANs) and higher sample quality than variational autoencoders (VAEs). However, this comes at the expense of poor scalability and increased sampling times due to the long Markov chain sequences required. We observe that within reconstruction-based anomaly detection a full-length Markov chain diffusion is not required. This leads us to develop a novel partial diffusion anomaly detection strategy that scales to high-resolution imagery, named AnoDDPM. A secondary problem is that Gaussian diffusion fails to capture larger anomalies; therefore we develop a multi-scale simplex noise diffusion process that gives control over the target anomaly size. AnoDDPM with simplex noise is shown to significantly outperform both f-AnoGAN and Gaussian diffusion for the tumorous dataset of 22 T1-weighted MRI scans (CCBS Edinburgh) qualitatively and quantitatively (improvement of +25.5\% Sørensen–Dice coefficient, +17.6\% IoU and +7.4\% AUC). 
@InProceedings{Wyatt_2022_CVPR,     
author = {Wyatt, Julian and Leach, Adam and Schmon, Sebastian M. and Willcocks, Chris G.},     
title = {AnoDDPM: Anomaly Detection With Denoising Diffusion Probabilistic Models Using Simplex Noise},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2022},
pages = {650-656} 
}

Denoising diffusion probabilistic models on SO(3) for rotational alignment

with Adam Leach, Matteo T. Degiacomi and Chris G. Willcocks

Workshop on Geometrical and Topological Representation Learning, ICLR 2022

Probabilistic diffusion models are capable of modeling complex data distributions on high-dimensional Euclidean spaces for a range applications. However, many real world tasks involve more complex structures such as data distributions defined on manifolds which cannot be easily represented by diffusions on $\mathbb{R}^n$. This paper proposes denoising diffusion models for tasks involving 3D rotations leveraging diffusion processes on the Lie group $SO(3)$ in order to generate candidate solutions to rotational alignment tasks. The experimental results show the proposed $SO(3)$ diffusion process outperforms naïve approaches such as Euler angle diffusion in synthetic rotational distribution sampling and in a 3D object alignment task.

Black-box Bayesian inference for economic agent-based models

with Joel Dyer, Patrick Cannon and Doyne Farmer

https://arxiv.org/abs/2202.00625

Simulation models, in particular agent-based models, are gaining popularity in economics. The considerable flexibility they offer, as well as their capacity to reproduce a variety of empirically observed behaviours of complex systems, give them broad appeal, and the increasing availability of cheap computing power has made their use feasible. Yet a widespread adoption in real-world modelling and decision-making scenarios has been hindered by the difficulty of performing parameter estimation for such models. In general, simulation models lack a tractable likelihood function, which precludes a straightforward application of standard statistical inference techniques. Several recent works have sought to address this problem through the application of likelihood-free inference techniques, in which parameter estimates are determined by performing some form of comparison between the observed data and simulation output. However, these approaches are (a) founded on restrictive assumptions, and/or (b) typically require many hundreds of thousands of simulations. These qualities make them unsuitable for large-scale simulations in economics and can cast doubt on the validity of these inference methods in such scenarios. In this paper, we investigate the efficacy of two classes of black-box approximate Bayesian inference methods that have recently drawn significant attention within the probabilistic machine learning community: neural posterior estimation and neural density ratio estimation. We present benchmarking experiments in which we demonstrate that neural network based black-box methods provide state of the art parameter inference for economic simulation models, and crucially are compatible with generic multivariate time-series data. In addition, we suggest appropriate assessment criteria for future benchmarking of approximate Bayesian inference procedures for economic simulation models. 
@misc{dyer2022blackbox,
	title={Black-box Bayesian inference for economic agent-based models}, 
	author={Joel Dyer and Patrick Cannon and J. Doyne Farmer and Sebastian Schmon},
	year={2022},
	eprint={2202.00625},
	archivePrefix={arXiv},
	primaryClass={econ.EM}
}

Approximate Bayesian Computation with Path Signatures

with Joel Dyer and Patrick Cannon

https://arxiv.org/abs/2106.12555

Simulation models of scientific interest often lack a tractable likelihood function, precluding standard likelihood-based statistical inference. A popular likelihood-free method for inferring simulator parameters is approximate Bayesian computation, where an approximate posterior is sampled by comparing simulator output and observed data. However, effective measures of closeness between simulated and observed data are generally difficult to construct, particularly for time series data which are often high-dimensional and structurally complex. Existing approaches typically involve manually constructing summary statistics, requiring substantial domain expertise and experimentation, or rely on unrealistic assumptions such as iid data. Others are inappropriate in more complex settings like multivariate or irregularly sampled time series data. In this paper, we introduce the use of path signatures as a natural candidate feature set for constructing distances between time series data for use in approximate Bayesian computation algorithms. Our experiments show that such an approach can generate more accurate approximate Bayesian posteriors than existing techniques for time series models. 
@misc{dyer2021approximate,
	title={Approximate Bayesian Computation with Path Signatures}, 
	author={Joel Dyer and Patrick Cannon and Sebastian M Schmon},
	year={2021},
	eprint={2106.12555},
	archivePrefix={arXiv},
	primaryClass={stat.ME}
}

Amortised Likelihood-free Inference for Expensive Time-series Simulators with Signatured Ratio Estimation

with Joel Dyer and Patrick Cannon

The 25th International Conference on Artificial Intelligence and Statistics, AISTATS 2022

Simulation models of complex dynamics in the natural and social sciences commonly lack a tractable likelihood function, rendering traditional likelihood-based statistical inference impossible. Recent advances in machine learning have introduced novel algorithms for estimating otherwise intractable likelihood functions using a likelihood ratio trick based on binary classifiers. Consequently, efficient likelihood approximations can be obtained whenever good probabilistic classifiers can be constructed. We propose a kernel classifier for sequential data using path signatures based on the recently introduced signature kernel. We demonstrate that the representative power of signatures yields a highly performant classifier, even in the crucially important case where sample numbers are low. In such scenarios, our approach can outperform sophisticated neural networks for common posterior inference tasks. 
@InProceedings{pmlr-v151-dyer22a,
  title = 	 { Amortised Likelihood-free Inference for Expensive Time-series Simulators with Signatured Ratio Estimation },
  author =       {Dyer, Joel and Cannon, Patrick W. and Schmon, Sebastian M.},
  booktitle = 	 {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics},
  pages = 	 {11131--11144},
  year = 	 {2022},
  editor = 	 {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel},
  volume = 	 {151},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {28--30 Mar},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v151/dyer22a/dyer22a.pdf},
  url = 	 {https://proceedings.mlr.press/v151/dyer22a.html},
  abstract = 	 { Simulation models of complex dynamics in the natural and social sciences commonly lack a tractable likelihood function, rendering traditional likelihood-based statistical inference impossible. Recent advances in machine learning have introduced novel algorithms for estimating otherwise intractable likelihood functions using a likelihood ratio trick based on binary classifiers. Consequently, efficient likelihood approximations can be obtained whenever good probabilistic classifiers can be constructed. We propose a kernel classifier for sequential data using path signatures based on the recently introduced signature kernel. We demonstrate that the representative power of signatures yields a highly performant classifier, even in the crucially important case where sample numbers are low. In such scenarios, our approach can outperform sophisticated neural networks for common posterior inference tasks. }
}

Learning Multimodal VAEs through Mutual Supervision

with Tom Joy, Yuge Shi, Philip H.S. Torr, Tom Rainforth, and N. Siddharth

International Conference on Learning Representations, ICLR 2022 (Spotlight)

Multimodal VAEs seek to model the joint distribution over heterogeneous data (e.g. vision, language), whilst also capturing a shared representation across such modalities. Prior work has typically combined information from the modalities by reconciling idiosyncratic representations directly in the recognition model through explicit products, mixtures, or other such factorisations. Here we introduce a novel alternative, the MEME, that avoids such explicit combinations by repurposing semi-supervised VAEs to combine information between modalities implicitly through mutual supervision. This formulation naturally allows learning from partially-observed data where some modalities can be entirely missing -- something that most existing approaches either cannot handle, or do so to a limited extent. We demonstrate that MEME outperforms baselines on standard metrics across both partial and complete observation schemes on the MNIST-SVHN (image-image) and CUB (image-text) datasets. We also contrast the quality of the representations learnt by mutual supervision against standard approaches and observe interesting trends in its ability to capture relatedness between data. 
@misc{joy2021learning,
	title={Learning Multimodal VAEs through Mutual Supervision}, 
	author={Tom Joy and Yuge Shi and Philip H. S. Torr and Tom Rainforth and Sebastian M. Schmon and N. Siddharth},
	year={2021},
	eprint={2106.12570},
	archivePrefix={arXiv},
	primaryClass={cs.LG}
}

Optimal scaling of random walk Metropolis algorithms using Bayesian large-sample asymptotics

with Philippe Gagnon

Statistics and Computing, 2022

High-dimensional asymptotics have been shown to be useful to derive tuning rules for finding the optimal scaling in random walk Metropolis algorithms. The assumptions under which weak convergence results are proved are however restrictive; the target density is typically assumed to be of a product form. Users may thus doubt the validity of such tuning rules in practical applications. In this paper, we shed some light on optimal scaling problems from a different perspective, namely a large-sample one. This allows to prove weak convergence results under realistic assumptions and to propose novel parameter dimension dependent tuning guidelines. The proposed guidelines are consistent with previous ones when the target density is close to having a product form, but significantly different when this is not the case. 
@Article{bvm_mcmc,
  author  = {Schmon, Sebastian M and Philippe Gagnon},
  title   = {Optimal scaling of random walk Metropolis algorithms using Bayesian large-sample asymptotics},
  journal = {https://arxiv.org/abs/2104.06384},
  year    = {2021},
}

Chapturing Label Characteristics in VAEs

with Tom Joy, Philipp Torr, Siddharth Narayanaswamy and Tom Rainforth

International Conference on Learning Representations, ICLR 2021

We present an alternative approach to semi-supervision in variational autoencoders (VAEs) that incorporates labels through auxiliary variables rather than directly through the latent variables. Prior work has generally conflated the meaning of labels, i.e. the associated characteristics of interest, with the actual label values themselves—learning latent variables that directly correspond to the label values. We argue that to learn meaningful representations, semi-supervision should instead try to capture these richer characteristics and that the construction of latent variables as label values is not just unnecessary, but actively harmful. To this end, we develop a novel VAE model, the reparameterized VAE (ReVAE), which “reparameterises” supervision through auxiliary variables and a concomitant variational objective. Through judicious structuring of mappings between latent and auxiliary variables, we show that the ReVAE can effectively learn meaningful representations of data. In particular, we demonstrate that the ReVAE is able to match, and even improve on the classification accuracy of previous approaches, but more importantly, it also allows for more effective and more general interventions to be performed. 
@inproceedings{
joy2021capturing,
title={Capturing Label Characteristics in {\{}VAE{\}}s},
author={Tom Joy and Sebastian Schmon and Philip Torr and Siddharth N and Tom Rainforth},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=wQRlSUZ5V7B}
}

Large Sample Asymptotics of the Pseudo-Marginal Method

with Arnaud Doucet, George Deligiannidis and Mike Pitt

Biometrika, 2021

The pseudo-marginal algorithm is a variant of the Metropolis–Hastings algorithm which samples asymptotically from a probability distribution when it is only possible to estimate unbiasedly an unnor- malized version of its density. Practically one has to trade-off the computational resources used to obtain this estimator against the asymptotic variances of the ergodic averages obtained by the pseudo-marginal algorithm. Recent works optimizing this trade-off rely on some strong assumptions which can cast doubts over their practical relevance. In particular, they all assume that the distribution of the additive error in the log-likelihood estimator is independent of the parameter value at which it is evaluated. Under weak regularity conditions we show here that, as the number of data points tends to infinity, a space-rescaled version of the pseudo-marginal chain converges weakly towards another pseudo-marginal chain for which this assumption indeed holds. A study of this limiting chain allows us to provide parameter dimension- dependent guidelines on how to optimally scale a normal random walk proposal and the number of Monte Carlo samples for the pseudo-marginal method in the large sample regime. This complements and vali- dates currently available results. 
@Article{SchmonDeligiannidisDoucet2018a,
	author = {Schmon, S M and Deligiannidis, G and Doucet, A and Pitt, M K},
	title = "{Large-sample asymptotics of the pseudo-marginal method}",
	journal = {Biometrika},
	volume = {108},
	number = {1},
	pages = {37-51},
	year = {2021},
	month = {03},
	issn = {0006-3444},
	doi = {10.1093/biomet/asaa044},
	url = {https://doi.org/10.1093/biomet/asaa044},
	eprint = {https://academic.oup.com/biomet/article-pdf/108/1/37/36440998/asaa044.pdf},
}

Generalized Posteriors in Approximate Bayesian Computation

with Patrick W Cannon and Jeremias Knoblauch

The 3rd Symposium on Advances in Approximate Bayesian Inference, AABI 2020

Complex simulators have become a ubiquitous tool in many scientific disciplines, providing high-fidelity, implicit probabilistic models of natural and social phenomena. Unfortunately, they typically lack the tractability required for conventional statistical analysis. Approximate Bayesian computation (ABC) has emerged as a key method in simulation-based inference, wherein the true model likelihood and posterior are approximated using samples from the simulator. In this paper, we draw connections between ABC and generalized Bayesian inference (GBI). First, we re-interpret the accept/reject step in ABC as an implicitly defined error model. We then argue that these implicit error models will invariably be misspecified. While ABC posteriors are often treated as a necessary evil for approximating the standard Bayesian posterior, this allows us to re-interpret ABC as a potential robustification strategy. This leads us to suggest the use of GBI within ABC, a use case we explore empirically. 
@Article{gbiabc,
  author  = {Schmon, Sebastian M and Patrick W Cannon and Jeremias Knoblauch},
  title   = {Generalized Posteriors in Approximate Bayesian Computation},
  journal = {https://arxiv.org/abs/2011.08644},
  year    = {2020},
}

Neural ODEs for Multi-state Survival Analysis

with Stefan Groha and Alexander Gusev

arxiv.org/abs/2006.04893

Survival models are a popular tool for the analysis of time to event data with applications in medicine, engineering, economics and many more. Advances like the Cox proportional hazard model have enabled researchers to better describe hazard rates for the occurrence of single fatal events, but are limited by modeling assumptions, like proportionality of hazard rates and linear effects. Moreover, common phenomena are often better described through multiple states, for example, the progress of a disease might be modeled as healthy, sick and dead instead of healthy and dead, where the competing nature of death and disease has to be taken into account. Also, individual characteristics can vary significantly between observational units, like patients, resulting in idiosyncratic hazard rates and different disease trajectories. These considerations require flexible modeling assumptions. Current standard models, however, are often ill-suited for such an analysis. To overcome these issues, we propose the use of neural ordinary differential equations as a flexible and general method for estimating multi-state survival models by directly solving the Kolmogorov forward equations. To quantify the uncertainty in the resulting individual cause-specific hazard rates, we further introduce a variational latent variable model. We show that our model exhibits state-of-the-art performance on popular survival data sets and demonstrate its efficacy in a multi-state setting. 
@Article{Groha2020,
  author  = {Stefan Groha, Sebastian M Schmon and Alexander Gusev},
  title   = {Neural ODEs for Multi-state Survival Analysis},
  journal = {https://arxiv.org/abs/2006.04893},
  year    = {2020},
}

Implicit Priors for Knowledge Sharing in Bayesian Neural Networks

with Jack Fitzsimons and Stephen Roberts

4th Neurips workshop on Bayesian Deep Learning 2019

Bayesian approaches offer a robust statistical framework to introduce prior knowledge into learning procedures. However, the tasks introduced above can be challenging in practice since "information" gained by one network such as learned features can be difficult to encode into prior distributions over networks that do not share the same architecture or even the same output dimension. We introduce here a Bayesian viewpoint that centres around features and describe a set of prior distributions derived from the theory of Gaussian processes and deep kernel learning that facilitate a variety of deep learning tasks in a unified way. In particular, we will show that our approach is applicable to knowledge distillation, transfer learning and combining experts. 
@Article{impprior,
  author  = {Jack K Fitzsimons and Sebastian M Schmon and Stephen J Roberts},
  title   = {Implicit Priors for Knowledge Sharing in Bayesian Neural Networks},
  journal = {4th Neurips workshop on Bayesian Deep Learning},
  year    = {2019},
}

Bernoulli Race Particle Filters

with Arnaud Doucet, George Deligiannidis

The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019

When the weights in a particle filter are not available analytically, standard resampling methods cannot be employed. To circumvent this problem state-of-the-art algorithms re- place the true weights with non-negative un- biased estimates. This algorithm is still valid but at the cost of higher variance of the re- sulting filtering estimates in comparison to a particle filter using the true weights. We pro- pose here a novel algorithm that allows for resampling according to the true intractable weights when only an unbiased estimator of the weights is available. We demonstrate our algorithm on several examples. 
@Article{bernrace,
  author  = {Sebastian M Schmon and George Deligiannidis and Arnaud Doucet},
  title   = {Bernoulli Race Particle Filters},
  journal = {AISTATS},
  year    = {2019},
}

Estimating the density of ethnic minorities and aged people in Berlin:

Multivariate kernel density estimation applied to sensitive georeferenced administrative data protected via measurement error

with Marcus Groß, Ulrich Rendtel, Timo Schmid and Nikos Tzavidis

Journal of the Royal Statistical Society: Series A (Statistics in Society) 180.1 (2017): 161-183

Modern systems of official statistics require the timely estimation of area-specific densities of subpopulations. Ideally estimates should be based on precise geocoded information, which is not available because of confidentiality constraints. One approach for ensuring confidentiality is by rounding the geoco‐ordinates. We propose multivariate non‐parametric kernel density estimation that reverses the rounding process by using a measurement error model. The methodology is applied to the Berlin register of residents for deriving density estimates of ethnic minorities and aged people. Estimates are used for identifying areas with a need for new advisory centres for migrants and infrastructure for older people. 
@article{gross2017,
  title={Estimating the density of ethnic minorities and aged people in Berlin: multivariate kernel density estimation applied to sensitive georeferenced administrative data protected via measurement error},
  author={Gro{\ss}, Marcus and Rendtel, Ulrich and Schmid, Timo and Schmon, Sebastian and Tzavidis, Nikos},
  journal={Journal of the Royal Statistical Society: Series A (Statistics in Society)},
  volume={180},
  number={1},
  pages={161--183},
  year={2017},
  publisher={Wiley Online Library}
}

Opportunities

Undergraduate and MSc Projects

I regularly offer projects relevant to my research areas. For the academic year 2022/23 you can find the project description below.

PhD Supervision

I am always looking for talented PhD students with an interested in Bayesian methods and/or Machine Learning. If you are interested in persuing a PhD under my supervision, please contact me at
sebastian [dot] schmon [at] durham [dot] ac [dot] uk.

As part of my position at Improbable I am currently co-supervising PhD students:

  • Joel Dyer, Oxford
  • Dan Ward, Bristol

Teaching

Durham University

Unsupervised Learning

Part of Advanced Statistical and Machine Learning: Foundations and Unsupervised Learning, MSc Scientific Computing and Data Analysis, 2022

University of Oxford

  • Applied Probability, 3rd year Mathematics (Michaelmas 2015)
  • Advanced Simulation, 4th year Mathematics and MSc Statistics (Hilary 2016, 2017, 2018)
  • Graphical Models, 4th year Mathematics and MSc Statistics (Michaelmas 2016, 2017, 2018)
  • Prelims: Probability, 1st year Mathematics (Michaelmas 2016)
  • Part A: Probability, 2nd year Mathematics (Michaelmas 2016)
  • Part A: Statistics, 2nd year Mathematics (Hilary 2017)

Free University Berlin

  • Statistics for Economists (2012, 2013, 2014), Introduction to probability and random variables, summary statistics, graphical data analysis
  • Statistical Inference (2012) Point estimation, large sample properties and limiting theorems, confidence sets and tests, linear regression
  • Statistical Modelling (2013) Generalised linear models, computational issues and implementation, application to economics.