Bence graduated from the University of Edinburgh in 2024 with a BSc in Mathematics. His dissertation focused on developing methods for automatic variable selection in generalized additive models. His primary research interests include computational statistics and statistical machine learning. In 2023, Bence completed a research internship at the University of Warwick, where he applied conditional generative neural networks for Bayesian likelihood-free inference.

Outside of mathematics, Bence enjoys reading, drinking overpriced filter coffee, and solving crosswords.

Project title:
In-Context and Curriculum Learning of Formal Languages with Transformers

Supervisor(s):
Michael Murray, Sandipan Roy

Project description:
In-context learning is the phenomenon where a transformer model performs a new task at inference time purely by conditioning on input-output examples in the prompt, without any weight updates. This project focuses on when and how this occurs: which families of functions and formal languages transformers can learn in-context, and how prompt size, token dimension, depth, and attention structure enable or limit that ability. The work is driven by three broad questions: expressivity, construction & mechanisms, and learning, focusing on translating mechanistic insights into provable statements about what attention layers can implement for language recognition and parsing.

We will combine mathematical analysis and experiments to study the ability of transformers to process, recognize and compile strings from formal languages via in-context learning. A second aim is to design training procedures, in particular curricula over language complexity, that improve sample efficiency and promote the emergence of reliable in-context learners and parsers.