Course Description: We will look into the intersection of machine learning and neuroscience.
The main target audience is students with a background in machine learning who are interested in learning about the brain and how it processes information, how artificial neural networks relate to real neurons, how machine learning can help understand the brain better, and how machine learning models may end up aligning with representations found in the brain.
The focus will mostly be on vision and language. Thus, this seminar may be particularly interesting if you have a background either in NLP/Computational Linguistics or in Computer Vision. Background in one of the two is certainly enough.
The seminar can also be interesting for students with a background in neuroscience.
We will read a range of materials, from both the machine learning and neuroscience literatures.
We will look into questions such as:
how and why do representations of language models align with brain imaging data of people comprehending language? (e.g., Oota et al NeurIPS 2023)
how can we use machine learning for mindreading: decoding thoughts and language from people’s brain recordings, even when they aren’t talking? (e.g., Tang et al, Nat Neuro 2023)
how do ConvNets and other computer vision architectures relate to the way the brain processes visual information? (e.g., Zhuang et al, PNAS 2021)
how do both AI and natural intelligence use reinforcement learning to optimize behavior? (Mnih et al, Nature 2015 and Cross et al, Neuron 2021)
and several more.
Prerequisites: It is sufficient if you have background in either machine learning or neuroscience. However, you should be willing to learn a bit about the other field in preparation for your presentation.
Michael is happy to meet with you once to talk through any questions about the paper you’re assigned. While you are expected to make an effort to understand the paper you’re assigned, and should understand its overall claims and conclusions, it is fully acceptable if some details that aren’t closely related to your own background remain unclear. Not understanding some aspects of your assigned paper won’t affect your presentation grade - as long as you’re transparent and upfront about points you don’t understand about your assigned paper when doing your presentation. In fact, points of confusion can be a starting point for in-class discussion. In choosing papers you want to present, consider not only whether you feel you’d understand the paper in detail, but also how interesting it sounds to you and how motivated you’d be to learn a bit more about its topic.
Registration:
If you are an LST / CoLi student, and want to take this class, you should directly register in the Course Management System (CMS). You may either be directly admitted or waitlisted.
If you are a Computer Science student, you should initially register via the Computer Science department seminar registration system. If you want to take the seminar but were not selected by the assignment system, please apply for the waiting list by emailing mhahn@lst.uni-saarland.de. Only register in Course Management System (CMS) once you were selected by the assignment system or otherwise admitted by us.
In both cases, please email mhahn@lst.uni-saarland.de your top-3 preferences among the items in the syllabus, and a brief explanation why you want to take this course and feel prepared for it. If you want, you are welcome to additionally mention any other topic that you would like to present. If you suggest something interesting, that may boost your chances of being admitted.
Course Management System: Course Management System (CMS)
Instructors: Michael Hahn
Time: Thu 12:15–13:45
Room: Building C7.3, Seminarraum -1.05
This is a seminar course. Starting from the fourth week, one or two students will present in each unit (except for the June 20 session). Every student will present exactly once. We expect all students to read the readings every week. Every student submits one question about the readings by Wednesday noon.
Each week, we will focus on one topic, to be jointly presented by two people. For most topics, we have listed two readings.
Note: The syllabus may still change. You’re welcome to suggest alternative topics or readings that you’re interested in.
In each session, two students will typically present two papers (in the “Readings” column) on a common topic.
The course roughly has three parts: we will look into
| Date | Topic | Readings | Slides | Optional Material | Presenter |
|---|---|---|---|---|---|
| 2024-04-18 | no class | ||||
| 2024-04-25 | Basics of Neuroscience | Gershman, Bio Cyb 2024 | Michael | ||
| 2024-05-02 | no class | ||||
| 2024-05-09 | no class | ||||
| Part 1: Computer Vision and the Visual Cortex | |||||
| 2024-05-16 | Simple and Complex cells (1981 Nobel Prize) | Various online material on Hubel&Wiesel’s findings. Wikipedia: simple cell, Wikipedia: complex cell | Daniyal Ahmed | ||
| ConvNets: LeCun et al, 1989 | Neurocognitron: Fukushima, Biol Cyb 1980 | Furkan | |||
| 2024-05-23 | Grid Cells | Wikipedia, Moser et al | Stensola et al | Jens | |
| Emergence in artifical NNs Cueva and Wei, ICLR 2018 | Suhas | ||||
| 2024-05-30 | Visual Cortex | Wikipedia: Visual Cortex | Lindsay, J Cog Neur 2021 | Tim | |
| Zhuang et al, PNAS 2021 | Qian | ||||
| 2024-06-06 | Aligning Vision Models and Brains | Multimodal encoding Tang et al, NeurIPS 2023 | Lars | ||
| ReAlnet Lu et al, arXiV 2024 | Nellia | ||||
| Part 2: Language Models and the Language Network | |||||
| 2024-06-13 | Aligning Language Models and Brains I | Schrimpf et al, PNAS 2021 | Goldstein et al, Nat Neuro 2022, Lopopolo et al 2024 | Jona | |
| Oota et al NeurIPS 2023 | Finn | ||||
| 2024-06-20 | Project Ideas | Everyone | |||
| 2024-06-27 | Aligning Language Models and Brains II | Antonello et al, NeurIPS 2023 | Ratnadeep | ||
| Driving and suppressing the language network Tuckute et al, Nat Neuro 2024 | TBD | ||||
| 2024-07-04 | Decoding language from the brain | Liu et al, Sci Adv 2023 | Greg | ||
| Tang et al, Nat Neuro 2023 | Syed | ||||
| Part 3: Reinforcement Learning and Behavior | |||||
| 2024-07-11 | Reinforcement Learning | Mnih et al, Nature 2015 | Simon Alexander | ||
| Scanning humans playing Atari: Cross et al, Neuron 2021 | Yuan | ||||
| 2024-07-18 | Inverse Reinforcement Learning | Ng and Russel, ICML 2000 | Arora et al, AI 2021 | Baraah Adil Mohammed | |
| Animal behavior Ashwood et al, NeurIPS 2022 | Asmaa |
For students taking the seminar for 4 credits:
Presentation: 60%
Questions about readings: 40%
For students taking the seminar for 7 credits:
Presentation: 30%
Questions about readings: 20%
Final paper: 50%
Please register on the forum on CMS.
Starting from the fourth week, every student submits one question about the readings by Wednesday noon. Questions are graded on a 3-point scale (0: no question submitted, 1: superficial question, 2: insightful question).
We expect that presentations will cover the key points from the readings, such as the main evidence for and against the key claims under consideration in the paper.
We do not expect that presentations will cover all details of the papers. Rather, you should focus on big picture findings and conclusions, and are not expected to include every finding from the paper in your presentation. For instance, instead of a table of numbers, highlight key results. When there are multiple similar results in the paper, synthesize them. If the papers have many studies, you might select a representative subset to explain the paper’s conclusions. On the other hand, if the assigned papers primarily discuss/review other work (as is the case in some weeks), draw on material from the work cited to provide richer content and even details where useful.
Make sure to motivate the papers’ research question(s). Give background on key concepts, and convey to the audience your understanding of why certain research decisions were made. Draw connections to other topics you’re familiar with (e.g., other concepts in computer science). Explain concepts in ways that will make them more accessible to your – predominantly computing-oriented – audience. Use such connections to other concepts in computer science as a way to stimulate class discussion. If something is unclear to you, say so and try to engage the audience to try and figure it out together.
Select what you consider the key points; you are not expected to cover every part of the paper exhaustively. Include details only to the extent that you believe them to be important.
Critically engage with the reading: contribute your own opinion on the key findings, and on the paper’s motivation and arguments. In what ways do or don’t you agree with arguments made by the authors?
As you’ll be presenting in teams of two, don’t just present the two papers separately, but make sure to also draw connections and compare. Aim for 40-60 minutes of presentation, allowing 30-40 minutes of discussion. Generating and moderating in-class discussion is a key component of your presentation – thinking about what will be interesting to your audience will thus be important. Discussion should happen not just after the presentation, but you should engage the audience and create ample opportunity for discussion during your presentation. Before the presentation, take a look at the questions that have been posted in the forum and refer to these as needed. These may be useful for getting discussion started. Conversely, when attending other students’ talks, reciprocate by participating actively in the discussion.
Note: We will discuss this in the first meeting. Requirements may be changed based on popular demand.
Term papers will be about a small independent project.
Possible topics:
Make a computational simulation of some aspect of brain function
Replicate a research paper, varying some aspect that wasn’t investigated in the original paper
Investigate whether some ML model (e.g. of language or vision) is or isn’t brain-like in some respect
The report is expected to contain a brief literature review, motivation of your study, a description of what you did, and report of your results.
The report should have at most 8 pages of main report, plus unlimited appendix, in the NeurIPS style format. The main report should be self-contained, but you can use the appendix to report prompts, further analyses, or other material.
The report should be uploaded via CMS. The due date is October 13, 2024, 23:59.
Everyone is expected to report on their project idea in the June 20, session, and to participate in discussion to give feedback to other students’ ideas. Students may prepare a short slide deck on their idea. This will not be graded; the June 20 session is intended to help improve and finetune project ideas.
Please contact Michael (mhahn@lst.uni-saarland.de) for any questions.
If you need any accommodations due to a disability or chronic illness, please either contact Michael at mhahn@lst.uni-saarland.de or the Equal Opportunities and Diversity Management Unit of the university.