CS-552: Modern Natural Language Processing

Course Description

Natural language processing is ubiquitous in modern intelligent technologies, serving as a foundation for language translators, virtual assistants, search engines, and many more. In this course, we cover the foundations of modern methods for natural language processing, such as word embeddings, recurrent neural networks, transformers, and pretraining, and how they can be applied to important tasks in the field, such as machine translation and text classification. We also cover issues with these state-of-the-art approaches (such as robustness, interpretability, sensitivity), identify their failure modes in different NLP applications, and discuss analysis and mitigation techniques for these issues.

Quick access links:

• Platforms
• Lecture Schedule
• Exercise Schedule
• Grading
• Contact

Class

Platform	Where & when
Lectures	Wednesdays: 11:15-13:00 [STCC - Cloud C] & Thursdays: 13:15-14:00 [CE16]
Exercises Session	Thursdays: 14:15-16:00 [CE11]
Project Assistance (not every week)	Wednesdays: 13:15-14:00 [STCC - Cloud C]
QA Forum & Annoucements	Ed Forum [link]
Grades	Moodle [link]
Lecture Recordings	Mediaspace [link]

All lectures will be given in person and live streamed on Zoom. The link to the Zoom is available on the Ed Forum (pinned post). Beware that, in the event of a technical failure during the lecture, continuing to accompany the lecture live via zoom might not be possible.

Recording of the lectures will be made available on Mediaspace. We will reuse some of last year's recordings and we may record a few new lectures in case of different lecture contents.

Lecture Schedule

Week	Date	Topic	Suggested Reading	Instructor
Week 1	19 Feb 20 Feb	Introduction | Building a simple neural classifier [slides] Word embeddings	Suggested reading: Introduction to natural language processing, chapter 3.1 - 3.3 & chapter 14.5 - 14.6 Efficient Estimation of Word Representations in Vector Space GloVe: Global Vectors for Word Representation Enriching word vectors with subword information Advances in pre-training distributed word representations	Antoine Bosselut
Week 2	26 Feb 27 Feb	Classical LMs | Neural LMs: Fixed Context Models Neural LMs: RNNs	Suggested reading: Introduction to natural language processing, chapter 6.1-6.4 A Neural Probabilistic Language Model On the difficulty of training recurrent neural networks Introduction to natural language processing, chapter 3.1 - 3.3 & chapter 18.3, 18.4	Antoine Bosselut
Week 3	5 Mar 6 Mar	Sequence-to-sequence Models | Transformers Pretraining: GPT	Suggested reading: Attention Is All You Need The Annotated Transformer The illustrated transformer GPT: Improving language understanding by generative pre-training GPT2: Language Models are Unsupervised Multitask Learners	Antoine Bosselut
Week 4	12 Mar 13 Mar	[Online only] Pretraining: BERT | Transfer Learning Pretraining: T5	Suggested reading: Elmo: Deep Contextualized Word Representations BERT: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding RoBERTa: RoBERTa: A Robustly Optimized BERT Pretraining Approach Transfer Learning in Natural Language Processing T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer BART: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	Antoine Bosselut
Week 5	19 Mar 20 Mar	Transfer Learning: Dataset Biases Generation: Task	Suggested reading: -	Antoine Bosselut
Week 6	26 Mar 27 Mar	Text Generation: Decoding & Training Text Generation: Evaluation	Suggested reading: Decoding: On Using Monolingual Corpora in Neural Machine Translation Decoding: Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Decoding: Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text Decoding: A Diversity-Promoting Objective Function for Neural Conversation Models Decoding: A Deep Reinforced Model for Abstractive Summarization Decoding: Deep Communicating Agents for Abstractive Summarization Decoding: Learning to Write with Cooperative Discriminators Decoding: Hierarchical Neural Story Generation Decoding: Discourse Understanding and Factual Consistency in Abstractive Summarization Decoding: Plug and Play Language Models: A Simple Approach to Controlled Text Generation Decoding: The Curious Case of Neural Text Degeneration Decoding: Generalization through Memorization: Nearest Neighbor Language Models Training: Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks Training: Sequence Level Training with Recurrent Neural Networks Training: Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Training: Deep Reinforcement Learning-based Image Captioning with Embedding Reward Training: Self-critical Sequence Training for Image Captioning Training: Improved Image Captioning via Policy Gradient Optimization of SPIDEr Training: Sentence Simplification with Deep Reinforcement Learning Training: A Deep Reinforced Model for Abstractive Summarization Training: Deep Communicating Agents for Abstractive Summarization Training: Discourse-Aware Neural Rewards for Coherent Text Generation Training: Polite Dialogue Generation Without Parallel Data Training: Paraphrase Generation with Deep Reinforcement Learning Training: The Curious Case of Neural Text Degeneration	Antoine Bosselut
Week 7 / Week 8	2 Apr / 9 Apr 3 Apr / 10 Apr	Midterm / [Online only] LLMs: In-context Learning & Instruction Tuning No Class / Project Description		Antoine Bosselut
Week 9	16 Apr 17 Apr	[Online only]Scaling laws | LLM Efficiency No class (Work on your project)	Suggested reading: Scaling laws for neural language models Training compute-optimal large language models	Antoine Bosselut
		EASTER BREAK
Week 10	30 Apr 1 May	Ethics: Bias & Fairness | Toxicity & Disinformation No class (Work on your project)	Suggested reading: Ethics in NLP United Nations recommendations/overview on responsible AI practice	Anna Sotnikova
Week 11	7 May 8 May	Tokenization | Multilingual LMs No class (Work on your project)	Suggested reading: Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP Byt5: Towards a token-free future with pre-trained byte-to-byte models Neural machine translation of rare words with subword units Unsupervised cross-lingual representation learning at scale Emerging cross-lingual structure in pretrained language models Mad-x: An adapter-based framework for multi-task cross-lingual transfer Composable sparse fine-tuning for cross-lingual transfer The State of Multilingual AI	Antoine Bosselut
Week 12	14 May 15 May	Interpretability No class (Work on your project)	Suggested reading: -	Gail Weiss
Week 13	21 May 22 May	Retrieval-Augmented LLMs | Agents No class (Work on your project)	Suggested reading: Squad: 100,000+ questions for machine comprehension of text Natural questions: a benchmark for question answering research Dense passage retrieval for open-domain question answering Retrieval augmented language model pre-training Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Toolformer: Language models can teach themselves to use tools React: Synergizing reasoning and acting in language models Improving language models by retrieving from trillions of tokens Augmented language models: a survey	Antoine Bosselut
Week 14	28 May 29 May	Multimodality Looking forward	Suggested reading: -	Syrielle Montariol Antoine Bosselut

Exercise Schedule

Week	Date	Topic	Instructor
Week 1	20 Feb	Intro + Setup [code]	Beatriz Borges
Week 2	27 Feb	LMs + Neural LMs: fixed-context models Language and Sequence-to-sequence models	Simin Fan
Week 3	6 Mar	Attention + Transformers, GPT	Badr AlKhamissi
Week 4	13 Mar	Pretraining and Transfer Learning	Badr AlKhamissi
Week 5	20 Mar	Transfer Learning	Simin Fan
Week 6	27 Mar	Generation	Madhur Panwar
Week 7/8		MIDTERM / In-context Learning - GPT-3	Mete Ismayilzad

Exercises Session format:

• TAs will provide a small discussion over the last week's exercises, answering any questions and explaining the solutions. (10-15mins)
• TAs will present this week's exercise. (5mins)
• Students will be solving this week's exercises and TAs will provide answers and clarification if needed.

Note: Please make sure you have already done the setup prerequisites to run the coding parts of the exercises. You can find the instructions here.

Grading:

Your grade in the course will be computed according to the following guidelines.

Submission Format

We will be using the Huggingface Hub as a centralized platform for submitting project artifacts, including model checkpoints and datasets. Please take some time to familiarize yourself with the functionalities of Huggingface Hub in Python to ensure a smooth workflow.

Late Days Policy

All milestones are due by 23:59 on their respective due dates. However, we understand that meeting deadlines can sometimes be challenging. To accommodate this, you will be given 2 late days for the semester to use at your discretion for group project milestones. Additionally, you will have 3 individual late days that can be applied to the individual project milestone. For group projects, if all members still have late days remaining, those days can be pooled and converted to group late days at a rate of one group late day per four individual late days. No extensions will be granted beyond the due dates, except for the final report, code, and data, which have a strict final deadline of June 8th. Late days will be automatically tracked based on your latest commit, so there is no need to notify us. Once all your late days are used, any further late submissions will incur a 25% grade deduction per day.

Midterm (30%):

More details will be announced in the next weeks.

Project (70%):

The project will be divided into 2 milestones and a final submission. Each milestone will be worth 15% of the final grade with the remaining 30% being allocated to the final report. Each team will be supervised by one of the course TAs or AEs.

More details on the content of the project and the deliverables of each milestone will be released at a later date.

Milestone 1:

• Due: 4 May 2025

Milestone 2:

• Due: 18 May 2025

Final Deliverable:

• The final report, code, and date will be due on June 8th. Students are welcome to turn in their materials ahead of time, as soon as the semester ends.

• Due: 8 June 2025

Contacts

Please email us at nlp-cs552-spring2025-ta-team [at] groupes [dot] epfl [dot] ch for any administrative questions, rather than emailing TAs individually. All course content questions need to be asked via Ed.

Lecturer: Antoine Bosselut

Teaching assistants: Angelika Romanou, Badr AlKhamissi, Beatriz Borges, Zeming (Eric) Chen, Simin Fan, Silin Gao, Mete Ismayilzada, Sepideh Mamooler, Madhur Panwar, Auguste Poiroux, Ayush Tarun