Overview
Build, train and evaluate a language model, from n-grams to transformers. The course covers tokenization, attention, training data, inference, post-training, reasoning and tool use. Two individual projects put this into practice.
You need solid Python: functions, classes, file I/O and debugging. Prior machine learning or NLP experience helps but is not required; allow extra time in the first two weeks if these are new to you.
An elective for the Data Science and Mathematics master’s programmes.
- When
- Thursdays, 15:15–17:00
- Where
- SR 127.11 · IDea_LabLeechgasse 34
Schedule
Slides and reading lists will be added as we work through them.
- 01
The guessing game
- 02
Tokenizers
- 03
Attention, by hand
- 04
The transformer block
- 05
Training
Assignment 1 released.
- 06
Data cleaning
- 07
Data mixing
- 08
Inference
- 09
Evaluation
Room: SR 15.05 (room change).
- 10
Post-training
- 16–18 Dec
Assignment 1 defences
Bookable slots, details will follow.
- 11
DPO & GRPO
- 12
Reasoning
- 13
Agents & tools
- 14
Reading a technical report
- 15
Project feedback
- 03–05 Feb
Assignment 2 defences
Bookable slots, details will follow
Contact
For general questions about the course, ask Lorenz. If you have specific requests, you can also contact Ruggero.
The course communication channel will be sent out via Email. For illness or other circumstances affecting a deadline or defence, contact us promptly and provide the relevant documentation.
Assessment
Two individual assignments, each followed by a defence. Active attendance is required.
| Component | Weight | Date |
|---|---|---|
| Assignment 1 | 30% | Wednesday 9 December 2026, 23:59 |
| A1 defence | 10% | 16–18 December 2026 |
| Assignment 2 | 40% | Monday 15 February 2027, 23:59 — proposed |
| A2 defence | 20% | 3–5 February 2027 |
Grading scheme
The final grade is based on the weighted total of the four components above. Passing scores are divided into four equal bands.
| Weighted total | Grade |
|---|---|
| 87.5–100% | 1 |
| 75–<87.5% | 2 |
| 62.5–<75% | 3 |
| 50–<62.5% | 4 |
| <50% | 5 |
Insufficient understanding of the submitted work demonstrated in a defence can lead to an overall negative grade.
The A2 report deadline is not yet confirmed. The A2 code and results deadline and the submission channel will be announced separately. All times are local to Graz.
Using AI
Working with generative AI, especially coding agents, is encouraged in this course. You should think of AI as a tool that can help you achieve bigger, more interesting tasks. The tasks at hand do however require deep understanding of the material, even with the help of AI agents. Therefore you will have to find ways to balance vibecoding and personal engagement with content and code. Additionally, you will need to demonstrate your understanding to us in the defences.
Briefly document which tools you used and for what.
Assignments
A1 · Train a small GPT
Train and evaluate a small language model. Details will be added here shortly.
A2 · Post-train and evaluate
Post-train a model and compare it with the unchanged base model. Details will be added here shortly.
Defences
Each assignment has an individual defence. You should be able to explain your work and results.
Slots will be available for booking within the specified timewindows. Booking and preparation details will follow.
Compute
You will get access to the GSC cluster for the course. Access instructions and the compute allowance will be announced here.
Reference code
Code and setup instructions will be added here.