Course Description: Despite their success, neural language models are still often treated as “black boxes”. This seminar focuses on post-hoc interpretability for transformer-based language models: understanding how models solve problems or why they behave in certain way with techniques like probing representations, attributing outputs to inputs, identifying computational circuits, and testing explanations through causal interventions. We will also combine these methods with recent applications to model editing, behavioral control, context use in retrieval-augmented generation, reasoning, and agent monitoring. Readings include academic papers and research posts.
Prerequisites: Many of our readings will be quite technical. You will need a good background in NLP or machine learning in order to thrive in this course.
Registration:
If you are an LST / CoLi student, and want to take this class, you should directly register in the Course Management System (CMS). Admissions decision will be made around Oct 21.
If you are a Computer Science student, you should initially register via the Computer Science department seminar registration system. Only register in Course Management System (CMS) once you were selected by the assignment system or otherwise admitted by us.
In both cases, please fill in this form with your top-3 preferences among the items in the syllabus, and a brief explanation why you want to take this course and feel prepared for it. If you want, you are welcome to additionally mention any other topic that you would like to present.
Instructors: Xinting Huang
Time: Tuesday, 14:15 - 15:45
Room: C7.3 room 1.14
This is a seminar course. After the introductory class, two students will present in each reading unit. Every student will present exactly once. We expect all students to read the readings every week. Every student submits one question about the readings by Monday noon.
For readings without a selection below, read the main text, including figures and limitations; references, appendices, and supplementary material are optional. You are encouraged to check additional methodological details when needed to explain or assess a result.
The selections below are roughly a conference paper’s worth of reading. Section ranges include all subsections, figures, and captions. Everything else is optional. These selections are approximate; you may need to skim relevant passages in other sections to understand the assigned sections.
A Mathematical Framework for Transformer Circuits: Read from “Transformer Overview” through the end of “One-Layer Attention-Only Transformers”, then read “Induction Heads”.
Towards Monosemanticity: Decomposing Language Models With Dictionary Learning: Read from the beginning through the end of “Detailed Investigations of Individual Features” (stop before “Global Analysis”).
Circuit Tracing: Revealing Computational Graphs in Language Models: Read from the beginning through the end of “Attribution Graphs” (stop before “Global Weights”). Skim “Limitations”.
On the Biology of a Large Language Model: Read from the beginning through the end of “Planning in Poems”, then read “Entity Recognition and Hallucinations”, “Refusals”, and “Limitations”.
Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations: Read from the beginning through the end of “Characterizing NLA confabulations”, then read “Discussion and Limitations”.
IMPORTANT: Study programs may differ in which version(s) of a seminar course you can take. If in doubt, check with your study program coordinator.
For students taking the seminar for 4 credits:
Presentation: 60%
Questions about readings: 40%
For students taking the seminar for 7 credits:
Presentation: 30%
Questions about readings: 20%
Final paper: 50%
Please register on the forum on CMS, and write questions there (click “forum” at the top after logging in CMS)
Starting with the first reading unit, every student submits one question about the readings by Monday noon. Questions are graded on a 3-point scale, 0: no question submitted, 1: superficial question, 2: insightful question (insightful questions don’t mean long questions). Students can also submit more than one questions, the grade will be calculated as the highest score among questions（So you can also ask some basic questions that you want clarification).
We expect that presentations will cover the key points from the readings, such as the main evidence for and against the key claims under consideration in the paper.
We do not expect that presentations will cover all details of the papers. Rather, you should focus on big picture findings and conclusions, and are not expected to include every finding from the paper in your presentation. Select what you consider the key points.
Make sure to motivate the papers’ research question(s). Give background on key concepts, and convey to the audience your understanding of why certain research decisions were made.
Critically engage with the reading: contribute your own opinion on the key findings, and on the paper’s motivation and arguments. In what ways do or don’t you agree with arguments made by the authors?
As you’ll be presenting in teams of two, don’t just present the two papers separately, but make sure to also draw connections and compare if two papers are related.
Aim for 50-60 minutes of presentation (sum of two papers), allowing 30-40 minutes of discussion. You will have sufficient time, so avoid speaking too fast for others to keep up, make sure your audience is following. Making presentation is to convey information, if audience cannot follow, it’s waste of time for both sides.
Generating and moderating in-class discussion is a key component of your presentation – thinking about what will be interesting to your audience will thus be important.
Discussion should happen not just after the presentation, but you should engage the audience and create ample opportunity for discussion during your presentation.
Before the presentation, take a look at the questions that have been posted in the forum and refer to these as needed. You are not expected to answer each question, feel free to cluster similar questions and discuss them during your presentation. These may be useful for getting discussion started. Conversely, when attending other students’ talks, reciprocate by participating actively in the discussion.
Note: We will discuss this in the first meeting. Requirements may be changed based on popular demand.
Term papers will be about a small independent project.
You will investigate some interpretability question about neural networks. You could investigate an existing model, or train and interpret a model of your own. You could use an existing method or try to come up with some new analysis approach. You could also compare different analysis approaches, or investigate interpretability from a theoretical angle.
The report is expected to contain a brief literature review, motivation of your study, a description of what you did, and you found. It’s recommended to include quantitative validation of what you found, so that you show it’s not an illusion.
The report should have 8 pages of main report, plus unlimited appendix, in the NeurIPS style format. The main report should be self-contained, but you can use the appendix to report prompts, further analyses, or other material.
You are required to write the report entirely on your own. You may use LLMs to help with the coding. But you are responsible for correctness of the code and experiments. In the report, you should describe which part of the project is done with the help of LLMs in a separate section called “LLMs Usage”, which doesn’t count towards the page limit.
The report should be uploaded via CMS. The due date is TBD (usually around the beginning of next semester).
Please contact Xinting (xhuang@lst.uni-saarland.de) or Michael (mhahn@lst.uni-saarland.de) for any questions.
If you need any accommodations due to a disability or chronic illness, please either contact Michael at mhahn@lst.uni-saarland.de or the Equal Opportunities and Diversity Management Unit of the university.