lacoco-lab

Compositionality in Language and Computation

Course Description: Compositionality — roughly, the ability to correctly process wholes given the ability to correctly process their parts — is a core property of human cognition and especially natural language, where it enables ``infinite use of finite means’’ as known linguistic elements combine to produce novel words and sentences. Recent advances in Natural Language Processing have raised new questions in this domain: are modern artificial neural networks capable of compositional generalization — and for that matter, how capable are humans? This blockseminar briefly reviews foundational and recent work on the core scientific question of compositionality.

If you want to take this class, please register in CMS.

Course Management System: CMS

Instructors: Kate McCurdy. For any questions, please contact me by email: kmccurdy@lst.uni-saarland.de

Time (block seminar): 1-4 pm Monday, Wednesday, and Friday in September 2024, week TBD; possibly Sept. 9, 11, and 13 - or the following week, so Sept. 16, 18, 20. We can adjust the dates based on interest from seminar participants.

In addition, there will be an introductory lecture + coordination session during the summer semester (June or July), date and time TBD.

Room: TBD

Format and requirements

This is a block seminar course.

Every student will give a 10-minute presentation.

Students that do not present on a given day are expected to prepare a two-page high-level overview which summarizes the day’s assigned reading and explains how the papers relate to each other. The summary should conclude with a question for discussion. These summaries will be submitted at the end of each classroom session.

Syllabus

Note: The syllabus is subject to change, and not all of the listed readings here will be required. We will discuss this in the first meeting.

Preliminary topics

• What is compositionality?
  • different views in linguistics -Herbelot 2020
  • ? in current philosophy - Szabo 2020
  • a working approach in NLP - Bender et al 2015
  • ? a recent philosophical treatment w/r/t machine learning - Nefdt 2020
• Foundational issue: abstraction
  • Rule- vs exemplar-based generalization - Dasgupta et al 2022
  • Information compression and model capacity - Resnick et al 2020
  • Trade-off with variation - Conklin and Smith 2023
  • Information-theoretic decomposition of morphology - Socolof et al 2022
• Do ANNs achieve compositional generalization? - evidence of problems
  • an early overview - Baroni 2019
  • ? COGS, an influential benchmark - Kim and Linzen 2020
  • ? maybe not a problem after all - Csordas et al, 2021
  • SLOG, a recent update - Li et al 2023
• Some approaches to increasing ANN compositionality
  • Data augmentation - Akyurek and Andreas 2023
  • Meta learning - will be discussed in human comparison part
  • Explicit composition of distributed representations - Smolensky et al 2022
  • Symbolic methods: Neurosymbolic / program synthesis - reference TBD
• How compositional are humans anyway?
  • Humans generalize recursively to compose longer linguistic sequences than seen in training - McCoy et al 2021
  • Meta-learning better approximates human symbol mapping than symbolic composition - Lake and Baroni 2023
  • ? Humans are biased toward abstraction, ANNs toward pattern-matching - Kumar et al 2023
  • ? LMs are more logical than humans but repeat some fallacies - Eisape et al 2023
  • ? LMs show human-like content effects on reasoning tasks - Dasgupta et al 2023

Evaluation

For students taking the seminar for 4 credits:

Presentation: 50%
Reading summaries: 50%

For students taking the seminar for 7 credits:

Presentation: 25%
Reading summaries: 25%
Final report: 50%

Presentations

Given time limitations, presentations will be strictly kept to 10 minutes each, followed by a general discussion covering all of the papers. The presentation should focus on high-level points from the readings, such as the main argument and evidence for and against key claims under consideration.

Term Papers

Note: We will discuss this in the first meeting. Requirements may be changed based on popular demand.

You will write a report on one of the two following topics:

1. Is compositionality a significant concern for modern artificial neural networks (ANNs), or can we consider this a solved problem? Give reasons for or against one of these perspectives, and motivate your points by citing relevant literature. You should address points considered in class, such as various interventions which have been shown to enhance ANNs’ compositional processing, and evidence for and against compositionality in human behavior. (2000-3000 words)
2. Select an exisiting compositionality benchmark (e.g. SCAN, COGS, SLOG, CFQ) and evaluate at least one proposed approach to improve compositional generalization (e.g. data augmentation, auxiliary tasks/fine-tuning, specialized model architecture) against a standard model baseline. Write up your findings in a technical report, maximally 4 pages in ACL format.

The report should be uploaded via CMS. The due date will be one month following our final in-person session, i.e. Oct. 13 or 20.

Contact

Please contact Kate (kmccurdy@lst.uni-saarland.de) or Michael (mhahn@lst.uni-saarland.de) for any questions.

Accommodations

If you need any accommodations due to a disability or chronic illness, please either contact Michael at mhahn@lst.uni-saarland.de or the Equal Opportunities and Diversity Management Unit of the university.