Confirmed Speakers
Oliver Kosut
Title: An Information-theoretic Perspective on Privacy Measures
Abstract: How do you measure information? This question has been central to classical information theory problems, such as data compression and channel coding, and it is also crucial to the modern problem of data privacy. If sensitive data is used to train a learning model, or perform inference, how much private information has been revealed? Privacy differs dramatically from secrecy in that perfect privacy (i.e., close to zero mutual information) is impossible, because some information must be taken from the data. Instead, we must choose a privacy measure, and find how much information has been revealed according to that measure. The first half of the talk will cover a broad selection of information measures, including f-divergences and Rényi divergences, both of which are generalizations of the Kullback-Leibler divergence. The second half will focus on privacy, and how most privacy measures can be viewed as special cases of these information measures. I will discuss the many variants of differential privacy, as well as maximal leakage and maximal alpha-leakage. Finally, I will conclude with a discussion of optimal privacy mechanisms, and recent work deriving optimal mechanisms for differential privacy based on an information-theoretic framework analogous to classical rate-distortion.
Yuejie Chi
Title: Generative Priors in Data Science: From Low-rank to Diffusion Models
Abstract:Generative priors are effective countermeasures to combat the curse of dimensionality, and enable efficient learning and inversion that otherwise are ill-posed, in data science. Focusing on the statistical and algorithmic underpinnings, the tutorial discuss two ubiquitous types of priors for solving inverse problems: low-rank models, which postulate the signal of interest lie in some low-dimensional subspace, and score-based diffusion models, which prescribe the score functions of the marginal distributions of a noise-diffused forward process.
Through the lens of matrix and tensor factorization, the tutorial illuminates the optimization geometry of nonconvex low-rank factorization with the aid of statistical reasoning, and how gradient descent harnesses such geometry in an implicit manner to achieve both computational and statistical efficiency all at once. To further combat with ill-conditioning, we introduce scaled gradient descent (ScaledGD), a method that provably converges linearly at a constant rate independent of the condition number at near-optimal sample complexities, while maintaining the low per-iteration cost of gradient descent, even when the rank is overspecified and the initialization is random.
Going beyond low rank, the tutorial develops a suite of non-asymptotic theory towards understanding the data generation process of diffusion models in discrete time for both deterministic and stochastic samplers, assuming access to inaccurate estimates of the score functions. We further discuss how to provably accelerate data generation without additional training, leveraging higher-order approximation. Last but not least, we introduce a plug-and-play method that is provably robust for nonlinear inverse problems using unconditional diffusion priors.
Yingbin Liang
Title: Theory on Transformer Training
Abstract: Transformers, as foundation models, have recently revolutionized many machine learning (ML) applications such as natural language processing, computer vision, robotics, etc. Alongside their tremendous experimental successes, theoretical studies have emerged to explain why transformers can be trained to achieve the desired performance. This tutorial aims to provide an overview of these recent theoretical investigations that have characterized the training dynamics of transformer-based ML models. Additionally, the tutorial will explain the primary techniques and tools employed for such analyses. Specifically, the tutorial will begin with an introduction to basic transformer models, and then delve into several ML problems where transformers have found extensive application, such as in-context learning, next token prediction, and unsupervised learning. For each learning problem, the tutorial will go over the characterization of the training process, the convergence guarantee, and the optimality and insights of the attention solution. Finally, the tutorial will discuss future directions and open problems in this actively evolving field.
