Pradeep Ravikumar
Professor of Computer Science at Carnegie Mellon University

Banatao Auditorium | 310 Sutardja Dai Hall
Friday April 5, 2024 at 11am

Modern self-supervised representation learning approaches such as contrastive learning and masked language modeling have had considerable empirical successes. A typical approach to evaluate such representations involves learning a linear probe on top of such representations and measuring prediction performance with respect to some downstream prediction task. We add to a burgeoning understanding of such representation learning approaches as learning eigenfunctions of certain Laplacian operators, so that learning a linear probe can be naturally connected to RKHS regression with implicitly specified kernels. This allows us to extend non-parametric tools from RKHS regression analysis to analyze the performance of self-supervised representation learning methods, in a way that completely side-steps grappling with neural network based function classes used in practice for the representation encoders. This contravenes prevailing wisdom that we cannot understand these modern representation learning methods without understanding the inductive bias implicit in the intersection of deep neural networks and the optimization methods used to learn them. We specifically focus on augmentation based self-supervised learning approaches, and isolate key structural complexity characterizations of augmentations that we show can be used to quantitatively inform downstream performance of the learned representations.

Speaker Bio

Pradeep Ravikumar is a Professor in the Machine Learning Department, School of Computer Science at Carnegie Mellon University. His thesis has received honorable mentions in the ACM SIGKDD Dissertation award and the CMU School of Computer Science Distinguished Dissertation award. He is a Sloan Fellow, a Siebel Scholar, a recipient of the NSF CAREER Award, and co-editor-chief of the Journal of Machine Learning Research. His recent research interests are in neuro-symbolic AI, combining statistical machine learning, and symbolic and causal learning.