About us
We are a group of researchers investigating machine learning, language, and the human mind. We are based at Saarland University and Saarland Informatics Campus, Germany.
Our research studies the following topics:
Abilities of Language Models, and how they are learned:We investigate the abilities and learning biases of transformer models and other architectures, using both theoretical and empirical methods (ICLR 2025, ICML 2025, ACL 2024, NeurIPS 2024, TACL 2020).
Understanding Inner Workings of Language Models:We develop methods for mechanistically reverse-engineering the inner workings of language models (COLM 2025, NeurIPS 2024, TACL 2019).
Computational Cognition and Neuroscience: We use machine learning to model human language proecessing (PNAS 2022a, PNAS 2020, PNAS 2022b) and human perception (Nature Neuroscience 2024).
News
05/2025 Our work on lower bounds for chain-of-thought reasoning in transformers has been accepted at ICML 2025.
01/2025 Our work on provable guarantees for length generalization in transformers has been accepted at ICLR 2025.
09/2024 Three papers accepted at NeurIPS 2024, on mechanistic interpretability, state-space models, and separations between architectures. Our lab organized the ELLIS pre-NeurIPS poster session at Saarbrücken, with 17 NeurIPS posters from across campus.
08/2024 Our work on Why are Sensitive Functions Hard for Transformers? received a Best Paper Award at ACL 2024.
07/2024 Our work on InversionView for reading out information from neural activations received the Second Prize at ICML 2024 Mechanistic Interpretability Workshop.
02/2024 Our work on the mathematics underlying perceptual biases has appeared in Nature Neuroscience.
12/2022 MIT News has an article about our PNAS paper on recursion
01/2025 Our work on provable guarantees for length generalization in transformers has been accepted at ICLR 2025.
09/2024 Three papers accepted at NeurIPS 2024, on mechanistic interpretability, state-space models, and separations between architectures. Our lab organized the ELLIS pre-NeurIPS poster session at Saarbrücken, with 17 NeurIPS posters from across campus.
08/2024 Our work on Why are Sensitive Functions Hard for Transformers? received a Best Paper Award at ACL 2024.
07/2024 Our work on InversionView for reading out information from neural activations received the Second Prize at ICML 2024 Mechanistic Interpretability Workshop.
02/2024 Our work on the mathematics underlying perceptual biases has appeared in Nature Neuroscience.
12/2022 MIT News has an article about our PNAS paper on recursion
Meet the Team
Principal Investigator
Michael Hahn
Assistant Professor
Researchers
Sasha Bakalova
Research Assistant
Xinting Huang
PhD Student
Kate McCurdy
Postdoctoral Researcher
Mark Rofin
Research Assistant
Yash Sarrof
Research Assistant
Xin Tong
Research Assistant
Yana Veitsman
Research Assistant
Entang Wang
Research Assistant
Recent Publications
(2025).
Born a Transformer -- Always a Transformer?.
arXiv preprint.
(2025).
Language models can learn implicit multi-hop reasoning, but only if they have lots of training data.
arXiv preprint.
(2025).
A Formal Framework for Understanding Length Generalization in Transformers.
ICLR 2025.
(2024).
Lossy Context Surprisal Predicts Task-Dependent Patterns in Relative Clause Processing.
Proceedings of the 28th Conference on Computational Natural Language Learning (CoNLL 2024).
(2024).
Toward Compositional Behavior in Neural Models: A Survey of Current Views.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).
(2024).
InversionView: A General-Purpose Method for Reading Information from Neural Activations.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024)
Previously: Mechanistic Interpretability Workshop at ICML 2024 (oral) 🏆 Awarded Second Place Prize.
Previously: Mechanistic Interpretability Workshop at ICML 2024 (oral) 🏆 Awarded Second Place Prize.
(2024).
Linguistic Structure from a Bottleneck on Sequential Information Processing.
arXiv preprint..
(2024).
More frequent verbs are associated with more diverse valency frames: Efficient language design at the lexicon-grammar interface.
Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics..
(2024).
Separations in the Representational Capabilities of Transformers and Recurrent Architectures.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024).
(2024).
The Expressive Capacity of State Space Models: A Formal Language Perspective.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024, poster).
(2024).
Why are Sensitive Functions Hard for Transformers?.
Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics. 🏆 Best Paper Award.
(2024).
A unifying theory explains seemingly contradictory biases in perceptual estimation.
Nature Neuroscience.
(2023).
A theory of emergent in-context learning as implicit structure induction.
arXiv Preprint.
(2023).
A Cross-Linguistic Pressure for Uniform Information Density in Word Order.
Transactions of the Association for Computational Linguistics.
(2023).
Modeling task effects in human reading with neural network-based attention.
Cognition.
(2022).
A resource-rational model of human processing of recursive linguistic structure.
Proceedings of the National Academy of Sciences of the United States of America.
(2022).
Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality.
Proceedings of the National Academy of Sciences of the United States of America.
(2022).
Morpheme ordering across languages reflects optimization for processing efficiency.
Open Mind: Discoveries in Cognitive Science.
Teaching
Every semester, we organize a range of graduate-level seminars where we read current research papers. We also teach some undergraduate classes. See teaching page.
Joining the Lab
If you’re interested in joining, please do not hesitate to contact us at mhahn@lst.uni-saarland.de