publications

2025

1. arXiv

   E-Scores for (In)Correctness Assessment of Generative Model Outputs

   Guneet S. Dhillon, Javier González, Teodora Pandeva, and Alicia Curth

   2025

   Abs arXiv Bib PDF

   While generative models, especially large language models (LLMs), are ubiquitous in today’s world, principled mechanisms to assess their (in)correctness are limited. Using the conformal prediction framework, previous works construct sets of LLM responses where the probability of including an incorrect response, or error, is capped at a desired user-defined tolerance level. However, since these methods are based on p-values, they are susceptible to p-hacking, i.e., choosing the tolerance level post-hoc can invalidate the guarantees. We therefore leverage e-values to complement generative model outputs with e-scores as a measure of incorrectness. In addition to achieving the same statistical guarantees as before, e-scores provide users flexibility in adaptively choosing tolerance levels after observing the e-scores themselves, by upper bounding a post-hoc notion of error called size distortion. We experimentally demonstrate their efficacy in assessing LLM outputs for different correctness types: mathematical factuality and property constraints satisfaction.

   @misc{dhillon2025escores,
     title = {E-Scores for (In)Correctness Assessment of Generative Model Outputs},
     author = {Dhillon, Guneet S. and Gonz\'{a}lez, Javier and Pandeva, Teodora and Curth, Alicia},
     year = {2025},
     eprint = {2510.25770},
     archiveprefix = {arXiv},
     primaryclass = {stat.ML},
     url = {https://arxiv.org/abs/2510.25770},
   }

2. ICLR

   L3Ms — Lagrange Large Language Models

   Guneet S. Dhillon, Xingjian Shi, Yee Whye Teh, and Alex Smola

   In International Conference on Representation Learning, 2025

   Abs arXiv Bib PDF Code

   Supervised fine-tuning (SFT) and alignment of large language models (LLMs) are key steps in providing a good user experience. However, the concept of an appropriate alignment is inherently application-dependent, and current methods often rely on heuristic choices to drive optimization. In this work, we formulate SFT and alignment as a constrained optimization problem: the LLM is fine-tuned on a task while being required to meet application-specific requirements, without resorting to heuristics. To solve this, we propose Lagrange Large Language Models (L3Ms), which employ logarithmic barriers to enforce the constraints. This approach allows for the customization of L3Ms across diverse applications while avoiding heuristic-driven processes. We experimentally demonstrate the versatility and efficacy of L3Ms in achieving tailored alignments for various applications.

   @inproceedings{dhillon2025l3ms,
     author = {Dhillon, Guneet S. and Shi, Xingjian and Teh, Yee Whye and Smola, Alex},
     booktitle = {International Conference on Representation Learning},
     editor = {Yue, Y. and Garg, A. and Peng, N. and Sha, F. and Yu, R.},
     pages = {58300--58314},
     title = {{L3M}s --- {L}agrange Large Language Models},
     url = {https://proceedings.iclr.cc/paper_files/paper/2025/file/92d3d2a9801211ca3693ccb2faa1316f-Paper-Conference.pdf},
     volume = {2025},
     year = {2025},
   }

2024

1. AISTATS

   On the Expected Size of Conformal Prediction Sets

   Guneet S. Dhillon, George Deligiannidis, and Tom Rainforth

   In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, 02–04 may 2024

   Abs arXiv Bib PDF Code

   While conformal predictors reap the benefits of rigorous statistical guarantees on their error frequency, the size of their corresponding prediction sets is critical to their practical utility. Unfortunately, there is currently a lack of finite-sample analysis and guarantees for their prediction set sizes. To address this shortfall, we theoretically quantify the expected size of the prediction sets under the split conformal prediction framework. As this precise formulation cannot usually be calculated directly, we further derive point estimates and high-probability interval bounds that can be empirically computed, providing a practical method for characterizing the expected set size. We corroborate the efficacy of our results with experiments on real-world datasets for both regression and classification problems.

   @inproceedings{dhillon2024expected,
     title = {On the Expected Size of Conformal Prediction Sets},
     author = {Dhillon, Guneet S. and Deligiannidis, George and Rainforth, Tom},
     booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics},
     pages = {1549--1557},
     year = {2024},
     editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen},
     volume = {238},
     series = {Proceedings of Machine Learning Research},
     month = {02--04 May},
     publisher = {PMLR},
     url = {https://proceedings.mlr.press/v238/dhillon24a.html},
   }

2021

1. NeurIPS (Spotlight)

   Uniform Sampling over Episode Difficulty

   Sébastien Arnold, Guneet S. Dhillon, Avinash Ravichandran, and Stefano Soatto

   In Advances in Neural Information Processing Systems, 02–04 may 2021

   Abs arXiv Bib PDF Supp Code

   Episodic training is a core ingredient of few-shot learning to train models on tasks with limited labelled data. Despite its success, episodic training remains largely understudied, prompting us to ask the question: what is the best way to sample episodes? In this paper, we first propose a method to approximate episode sampling distributions based on their difficulty. Building on this method, we perform an extensive analysis and find that sampling uniformly over episode difficulty outperforms other sampling schemes, including curriculum and easy-/hard-mining. As the proposed sampling method is algorithm agnostic, we can leverage these insights to improve few-shot learning accuracies across many episodic training algorithms. We demonstrate the efficacy of our method across popular few-shot learning datasets, algorithms, network architectures, and protocols.

   @inproceedings{arnold2021uniform,
     author = {Arnold, S\'{e}bastien and Dhillon, Guneet S. and Ravichandran, Avinash and Soatto, Stefano},
     booktitle = {Advances in Neural Information Processing Systems},
     editor = {Ranzato, M. and Beygelzimer, A. and Dauphin, Y. and Liang, P.S. and Vaughan, J. Wortman},
     pages = {1481--1493},
     publisher = {Curran Associates, Inc.},
     title = {Uniform Sampling over Episode Difficulty},
     url = {https://proceedings.neurips.cc/paper_files/paper/2021/file/0b3f44d9054402de39441e165a4bdfe0-Paper.pdf},
     volume = {34},
     year = {2021},
   }

2020

1. ICLR

   A Baseline for Few-Shot Image Classification

   Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto

   In International Conference on Learning Representations, 02–04 may 2020

   Abs arXiv Bib PDF Code

   Fine-tuning a deep network trained with the standard cross-entropy loss is a strong baseline for few-shot learning. When fine-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100 with the same hyper-parameters. The simplicity of this approach enables us to demonstrate the first few-shot learning results on the ImageNet-21k dataset. We find that using a large number of meta-training classes results in high few-shot accuracies even for a large number of few-shot classes. We do not advocate our approach as the solution for few-shot learning, but simply use the results to highlight limitations of current benchmarks and few-shot protocols. We perform extensive studies on benchmark datasets to propose a metric that quantifies the "hardness" of a few-shot episode. This metric can be used to report the performance of few-shot algorithms in a more systematic way.

   @inproceedings{dhillon2020baseline,
     title = {A Baseline for Few-Shot Image Classification},
     author = {Dhillon, Guneet S. and Chaudhari, Pratik and Ravichandran, Avinash and Soatto, Stefano},
     booktitle = {International Conference on Learning Representations},
     year = {2020},
     url = {https://openreview.net/forum?id=rylXBkrYDS},
   }

2018

1. ICLR

   Stochastic Activation Pruning for Robust Adversarial Defense

   Guneet S. Dhillon, Kamyar Azizzadenesheli, Jeremy D. Bernstein, Jean Kossaifi, Aran Khanna, Zachary C. Lipton, and Animashree Anandkumar

   In International Conference on Learning Representations, 02–04 may 2018

   Abs arXiv Bib PDF Code

   Neural networks are known to be vulnerable to adversarial examples. Carefully chosen perturbations to real images, while imperceptible to humans, induce misclassification and threaten the reliability of deep learning systems in the wild. To guard against adversarial examples, we take inspiration from game theory and cast the problem as a minimax zero-sum game between the adversary and the model. In general, for such games, the optimal strategy for both players requires a stochastic policy, also known as a mixed strategy. In this light, we propose Stochastic Activation Pruning (SAP), a mixed strategy for adversarial defense. SAP prunes a random subset of activations (preferentially pruning those with smaller magnitude) and scales up the survivors to compensate. We can apply SAP to pretrained networks, including adversarially trained models, without fine-tuning, providing robustness against adversarial examples. Experiments demonstrate that SAP confers robustness against attacks, increasing accuracy and preserving calibration.

   @inproceedings{dhillon2018stochastic,
     title = {Stochastic Activation Pruning for Robust Adversarial Defense},
     author = {Dhillon, Guneet S. and Azizzadenesheli, Kamyar and Bernstein, Jeremy D. and Kossaifi, Jean and Khanna, Aran and Lipton, Zachary C. and Anandkumar, Animashree},
     booktitle = {International Conference on Learning Representations},
     year = {2018},
     url = {https://openreview.net/forum?id=H1uR4GZRZ},
   }