About us
We are a group of researchers investigating machine learning, language, and the human mind. We are based at Saarland University and Saarland Informatics Campus, Germany.
Our research studies the following topics:
Understanding machine learning models: We have investigated the expressive power and learning biases of transformer models (ACL 2024, TACL 2020) and state space models (NeurIPS 2024 ), and developed methods for mechanistic interpretability (NeurIPS 2024, TACL 2019).
Computational Cognition and Neuroscience: In recent work, we have augmented GPT-2 with memory limitations to examine what makes recursion difficult for humans (PNAS 2022a), modeled the evolution of grammar (PNAS 2020, PNAS 2022b), and have developed a unifying theory of biases in human perception (Nature Neuroscience 2024).
News
09/2024 Three papers accepted at NeurIPS 2024, on mechanistic interpretability, state-space models, and separations between architectures. Our lab organized the ELLIS pre-NeurIPS poster session at Saarbrücken, with 17 NeurIPS posters from across campus.
08/2024 Our work on Why are Sensitive Functions Hard for Transformers? received a Best Paper Award at ACL 2024.
07/2024 Our work on InversionView for reading out information from neural activations received the Second Prize at ICML 2024 Mechanistic Interpretability Workshop.
02/2024 Our work on the mathematics underlying perceptual biases has appeared in Nature Neuroscience.
12/2022 MIT News has an article about our PNAS paper on recursion
08/2024 Our work on Why are Sensitive Functions Hard for Transformers? received a Best Paper Award at ACL 2024.
07/2024 Our work on InversionView for reading out information from neural activations received the Second Prize at ICML 2024 Mechanistic Interpretability Workshop.
02/2024 Our work on the mathematics underlying perceptual biases has appeared in Nature Neuroscience.
12/2022 MIT News has an article about our PNAS paper on recursion
Meet the Team
Principal Investigator
Michael Hahn
Assistant Professor
Researchers
Sasha Bakalova
Research Assistant
Xinting Huang
PhD Student
Kate McCurdy
Postdoctoral Researcher
Mark Rofin
Research Assistant
Yash Sarrof
Research Assistant
Xin Tong
Research Assistant
Yana Veitsman
Research Assistant
Entang Wang
Research Assistant
Recent Publications
(2024).
Lossy Context Surprisal Predicts Task-Dependent Patterns in Relative Clause Processing.
Proceedings of the 28th Conference on Computational Natural Language Learning (CoNLL 2024).
(2024).
A Formal Framework for Understanding Length Generalization in Transformers.
arXiv preprint.
(2024).
Toward Compositional Behavior in Neural Models: A Survey of Current Views.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).
(2024).
InversionView: A General-Purpose Method for Reading Information from Neural Activations.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024)
Previously: Mechanistic Interpretability Workshop at ICML 2024 (oral) 🏆 Awarded Second Place Prize.
Previously: Mechanistic Interpretability Workshop at ICML 2024 (oral) 🏆 Awarded Second Place Prize.
(2024).
Linguistic Structure from a Bottleneck on Sequential Information Processing.
arXiv preprint..
(2024).
More frequent verbs are associated with more diverse valency frames: Efficient language design at the lexicon-grammar interface.
Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics..
(2024).
Separations in the Representational Capabilities of Transformers and Recurrent Architectures.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024).
(2024).
The Expressive Capacity of State Space Models: A Formal Language Perspective.
Annual Conference on Neural Information Processing Systems (NeurIPS 2024, poster).
(2024).
Why are Sensitive Functions Hard for Transformers?.
Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics. 🏆 Best Paper Award.
(2024).
A unifying theory explains seemingly contradictory biases in perceptual estimation.
Nature Neuroscience.
(2023).
A theory of emergent in-context learning as implicit structure induction.
arXiv Preprint.
(2023).
A Cross-Linguistic Pressure for Uniform Information Density in Word Order.
Transactions of the Association for Computational Linguistics.
(2023).
Modeling task effects in human reading with neural network-based attention.
Cognition.
(2022).
A resource-rational model of human processing of recursive linguistic structure.
Proceedings of the National Academy of Sciences of the United States of America.
(2022).
Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality.
Proceedings of the National Academy of Sciences of the United States of America.
(2022).
Morpheme ordering across languages reflects optimization for processing efficiency.
Open Mind: Discoveries in Cognitive Science.
(2022).
Explaining patterns of fusion in morphological paradigms using the memory--surprisal tradeoff.
Proceedings of the 44th Annual Meeting of the Cognitive Science Society (CogSci).
(2022).
Modeling fixation behavior in reading with character-level neural attention.
Proceedings of the 44th Annual Meeting of the Cognitive Science Society (CogSci).
(2022).
Information theory as a bridge between language function and language form.
Frontiers in Communication.
(2021).
Modeling word and morpheme order in natural language as an efficient tradeoff of memory and surprisal.
Psychological Review.
Teaching
Every semester, we organize a range of graduate-level seminars where we read current research papers.
Winter 2024/25 Seminar Processing Morphology (Kate McCurdy).
Winter 2024/25 Seminar Augmenting Language Models with Tools and Knowledge (Xinting Huang).
Winter 2024/25 Lecture Foundations of Mathematics (Michael Hahn).
Summer 2024 Seminar Compositionality in Language and Computation (Kate McCurdy).
Summer 2024 Seminar Probing and Analyzing Neural Language Models (Xinting Huang).
Summer 2024 Seminar Aligning Language Models with Human Preferences, Methods and Challenges (Michael Hahn).
Summer 2024 Seminar Neural Networks in Brains and Computers (Michael Hahn).
Winter 2023/24 Seminar Language Models as Cognitive Models (Kate McCurdy).
Winter 2023/24 Foundations of Mathematics (Michael Hahn).
Winter 2024/25 Seminar Processing Morphology (Kate McCurdy).
Winter 2024/25 Seminar Augmenting Language Models with Tools and Knowledge (Xinting Huang).
Winter 2024/25 Lecture Foundations of Mathematics (Michael Hahn).
Summer 2024 Seminar Compositionality in Language and Computation (Kate McCurdy).
Summer 2024 Seminar Probing and Analyzing Neural Language Models (Xinting Huang).
Summer 2024 Seminar Aligning Language Models with Human Preferences, Methods and Challenges (Michael Hahn).
Summer 2024 Seminar Neural Networks in Brains and Computers (Michael Hahn).
Winter 2023/24 Seminar Language Models as Cognitive Models (Kate McCurdy).
Winter 2023/24 Foundations of Mathematics (Michael Hahn).
Joining the Lab
If you’re interested in joining, please do not hesitate to contact us at mhahn@lst.uni-saarland.de