Course Description: Today’s large language models can show behavior that looks strikingly intelligent. This raises many questions about their relation to, and differences from, human cognition and intelligence. This seminar course dives into this rapidly emerging field at the intersection of NLP, language, and cognition.
Course Management System: CMS
Instructors: Kate McCurdy
Time: Tue 16:15–17:45
Room: Building C7.3, Seminarraum 1.12
This is a seminar course. Starting from the fourth week, one or two students will present in each unit (except for the Dec 19 session). Every student will present exactly once. We expect all students to read the readings every week. Every student submits one question about the readings by Monday noon.
Note: The syllabus is subject to change. We will discuss this in the first and second meeting.
| Date | Topic | Readings | Slides | Optional Material | Presenter |
|---|---|---|---|---|---|
| 2023-10-24 | First Meeting | Michael | |||
| 2023-10-31 | no class | ||||
| 2023-11-07 | Introduction to (L)LMs | slides | Michael | ||
| 2023-11-14 — 2023-12-12 | Unit 1: LMs as cognitive models of language | ||||
| 2023-11-14 | Past tense debate | Kirov & Cotterell 2018, TACL, Corkery et al. 2019, ACL | Seidenberg & Plaut 2014, Rumelhart & McClelland 1986, Pinker & Prince 1988, McCurdy et al 2020, ACL | Lotta, Julian | |
| 2023-11-21 | Symbolic computation and composition | Santoro and Lampinen et al. 2022, Lake and Baroni 2023, Nature | Lake and Baroni 2018, ICML | Mark, Ravikiran | |
| 2023-11-28 | Surprisal and next-word prediction | Wilcox et al 2020, CogSci, Oh and Schuler 2023, TACL | Hale 2001, NAACL, Levy 2008, Cognition | Asmaa, Anh Tu | |
| 2023-12-05 | Linguistic Capabilities | Hu et al., 2020, ACL | Zhang et al 2023, SIGNLL | Bao | |
| 2023-12-12 | LLM’s capabilities in formal vs functional language use | Mahowald and Ivanova et al., 2023, arXiv | Nicholas, Tyler | ||
| 2023-12-19 | Discussing project ideas | everyone | |||
| 2024-01-02 — 2024-02-06 | Unit 2: LMs as cognitive models of understanding | ||||
| 2024-01-02 | Debate over LLM understanding ~Metalinguistic judgments~ | Mitchell and Krakauer 2023, PNAS ~Hu et al., 2023, EMNLP~ | Mitchell 2023, Science | Nadiia ~Nagasandeepa~ | |
| 2024-01-09 | Using cognitive psychology to understand GPT-3 | Binz and Schulz 2023, PNAS, Binz and Schulz 2023, arxiv | Nellia, Jakob | ||
| 2024-01-16 | Theory of Mind in LLMs | Kosinski 2022, arXiv, Ullman 2023, arXiv | Sarah, Eva | ||
| 2024-01-23 | Debate over Reference in LLMs | Bender and Koller 2020, ACL, Patel and Pavlick 2023, ICLR | Piantadosi and Hill 2022, arXiv, Pavlick 2023 | ~Jilixin~ Aleksandra | |
| ~2024-01-30~ | ~Comparison to children’s cognitive development~ | ~Frank 2023, PsyArXiv, Yiu et al. 2023, arXiv~ | ~Hanna~ | ||
| 2024-01-30 | How next-word prediction mediates LLM reasoning | McCoy et al. 2023, arXiv | Viet Anh ~Thi Thu Uyen~ |
For students taking the seminar for 4 credits:
Presentation: 60%
Questions about readings: 40%
For students taking the seminar for 7 credits:
Presentation: 30%
Questions about readings: 20%
Final paper: 50%
Please register on the forum on CMS.
Starting from the fourth week (Nov 13), every student submits one question about the readings by Monday noon. Questions are graded on a 3-point scale (0: no question submitted, 1: superficial question, 2: insightful question).
We expect that presentations will cover the key points from the readings, such as the main evidence for and against the key claims under consideration in the paper.
We do not expect that presentations will cover all details of the papers. Rather, you should focus on big picture findings and conclusions, and are not expected to include every finding from the paper in your presentation. For instance, instead of a table of numbers, highlight key results. When there are multiple similar results in the paper, synthesize them. If the papers have many studies, you might select a representative subset to explain the paper’s conclusions. On the other hand, if the assigned papers primarily discuss/review other work (as is the case in some weeks), draw on material from the work cited to provide richer content and even details where useful.
Make sure to motivate the papers’ research question(s). Give background on key concepts, and convey to the audience your understanding of why certain research decisions were made.
Select what you consider the key points; you are not expected to cover every part of the paper exhaustively. Include details only to the extent that you believe them to be important.
If you present two papers, do not just present them separately. Rather, draw connections, and compare and contrast where appropriate.
Critically engage with the reading: contribute your own opinion on the key findings, and on the paper’s motivation and arguments. In what ways do or don’t you agree with arguments made by the authors?
Aim for 40-60 minutes of presentation, allowing 30-40 minutes of discussion. If you present on your own, your presentation can be somewhat shorter. Discussion should happen not just after the presentation, but you should engage the audience and create ample opportunity for discussion during your presentation. Before the presentation, take a look at the questions that have been posted in the forum and refer to these as needed. These may be useful for getting discussion started. Conversely, when attending other students’ talks, reciprocate by participating actively in the discussion.
Note: We will discuss this in the first and second meeting. Requirements may be changed based on popular demand.
Term papers will be about a small independent project. You will investigate a question about LLMs’ cognitive ability via prompting. For instance, you might test their ability to perform a certain kind of reasoning, or investigate how they respond to certain kinds of stimuli.
Option 1: You may develop your own question and prompts. In this case, you will be expected to design at least 25 (not more than 50) prompts.
Option 2: You may draw on a larger existing benchmark. In this case, you will be expected to find some new angle on the benchmark, e.g., by tweaking the stimuli or by evaluating the LLM’s behavior in a different way.
The report is expected to contain a brief literature review, motivation of your question, a description of your prompts, and evaluation of the LLM’s behavior. The report is expected to include quantitative evaluation of the LLM’s behavior (e.g., using measures such as accuracy). Additionally including qualitative evaluation can also be beneficial.
The report should have 8 pages of main report, plus unlimited appendix. The main report should be self-contained, but you can use the appendix to report prompts, further analyses, or other material.
The report should be uploaded via CMS. The due date is April 8, 2024, 23:59.
Everyone is expected to report on their project idea in the December 19, 2023, session, and to participate in discussion to give feedback to other students’ ideas. Students may prepare a short slide deck on their idea. This will not be graded; the December 19 session is intended to help improve and finetune project ideas.
Please contact Kate (kmccurdy@lst.uni-saarland.de) or Michael (mhahn@lst.uni-saarland.de) for any questions.
If you need any accommodations due to a disability or chronic illness, please either contact Michael at mhahn@lst.uni-saarland.de or the Equal Opportunities and Diversity Management Unit of the university.