About us
We are a group of researchers investigating machine learning, language, and the human mind. We are based at Saarland University and Saarland Informatics Campus, Germany.
Our research studies the following topics:
Understanding machine learning models: We have investigated the expressive power and learning biases of transformer models (ICLR 2025, ACL 2024, TACL 2020) and state space models (NeurIPS 2024 ). We also develop methods for mechanistic interpretability (NeurIPS 2024, TACL 2019).
Computational Cognition and Neuroscience: In recent work, we have augmented GPT-2 with memory limitations to examine what makes recursion difficult for humans (PNAS 2022a), modeled the evolution of grammar (PNAS 2020, PNAS 2022b), and have developed a unifying theory of biases in human perception (Nature Neuroscience 2024).
News
01/2025 Our work on provable guarantees for length generalization in transformers has been accepted at ICLR 2025.
09/2024 Three papers accepted at NeurIPS 2024, on mechanistic interpretability, state-space models, and separations between architectures. Our lab organized the ELLIS pre-NeurIPS poster session at Saarbrücken, with 17 NeurIPS posters from across campus.
08/2024 Our work on Why are Sensitive Functions Hard for Transformers? received a Best Paper Award at ACL 2024.
07/2024 Our work on InversionView for reading out information from neural activations received the Second Prize at ICML 2024 Mechanistic Interpretability Workshop.
02/2024 Our work on the mathematics underlying perceptual biases has appeared in Nature Neuroscience.
12/2022 MIT News has an article about our PNAS paper on recursion
09/2024 Three papers accepted at NeurIPS 2024, on mechanistic interpretability, state-space models, and separations between architectures. Our lab organized the ELLIS pre-NeurIPS poster session at Saarbrücken, with 17 NeurIPS posters from across campus.
08/2024 Our work on Why are Sensitive Functions Hard for Transformers? received a Best Paper Award at ACL 2024.
07/2024 Our work on InversionView for reading out information from neural activations received the Second Prize at ICML 2024 Mechanistic Interpretability Workshop.
02/2024 Our work on the mathematics underlying perceptual biases has appeared in Nature Neuroscience.
12/2022 MIT News has an article about our PNAS paper on recursion
Meet the Team
Principal Investigator
Michael Hahn
Assistant Professor
Researchers
Sasha Bakalova
Research Assistant
Xinting Huang
PhD Student
Kate McCurdy
Postdoctoral Researcher
Mark Rofin
Research Assistant
Yash Sarrof
Research Assistant
Xin Tong
Research Assistant
Yana Veitsman
Research Assistant
Entang Wang
Research Assistant
Recent Publications
(2025).
Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B.
arXiv preprint.
(2025).
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers.
arXiv preprint.
(2025).
A Formal Framework for Understanding Length Generalization in Transformers.
ICLR 2025.
(2024).
Lossy Context Surprisal Predicts Task-Dependent Patterns in Relative Clause Processing.
Proceedings of the 28th Conference on Computational Natural Language Learning (CoNLL 2024).
(2024).
Toward Compositional Behavior in Neural Models: A Survey of Current Views.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).
(2024).
InversionView: A General-Purpose Method for Reading Information from Neural Activations.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024)
Previously: Mechanistic Interpretability Workshop at ICML 2024 (oral) 🏆 Awarded Second Place Prize.
Previously: Mechanistic Interpretability Workshop at ICML 2024 (oral) 🏆 Awarded Second Place Prize.
(2024).
Linguistic Structure from a Bottleneck on Sequential Information Processing.
arXiv preprint..
(2024).
More frequent verbs are associated with more diverse valency frames: Efficient language design at the lexicon-grammar interface.
Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics..
(2024).
Separations in the Representational Capabilities of Transformers and Recurrent Architectures.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024).
(2024).
The Expressive Capacity of State Space Models: A Formal Language Perspective.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024, poster).
(2024).
Why are Sensitive Functions Hard for Transformers?.
Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics. 🏆 Best Paper Award.
(2024).
A unifying theory explains seemingly contradictory biases in perceptual estimation.
Nature Neuroscience.
(2023).
A theory of emergent in-context learning as implicit structure induction.
arXiv Preprint.
(2023).
A Cross-Linguistic Pressure for Uniform Information Density in Word Order.
Transactions of the Association for Computational Linguistics.
(2023).
Modeling task effects in human reading with neural network-based attention.
Cognition.
(2022).
A resource-rational model of human processing of recursive linguistic structure.
Proceedings of the National Academy of Sciences of the United States of America.
(2022).
Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality.
Proceedings of the National Academy of Sciences of the United States of America.
(2022).
Morpheme ordering across languages reflects optimization for processing efficiency.
Open Mind: Discoveries in Cognitive Science.
(2022).
Explaining patterns of fusion in morphological paradigms using the memory--surprisal tradeoff.
Proceedings of the 44th Annual Meeting of the Cognitive Science Society (CogSci).
(2022).
Modeling fixation behavior in reading with character-level neural attention.
Proceedings of the 44th Annual Meeting of the Cognitive Science Society (CogSci).
Teaching
Every semester, we organize a range of graduate-level seminars where we read current research papers. We also teach some undergraduate classes. See teaching page.
Joining the Lab
If you’re interested in joining, please do not hesitate to contact us at mhahn@lst.uni-saarland.de