3rd Fall School of the Methods Center on October 16, 2024
Interdisciplinary Methods
The Methods Center at the University of Tübingen cordially invites PhD candidates, postdoctoral researchers, and professors to join our Fall School.
The topic of this year's workshop is "Psychometrics for Large Language Model Evaluation: Lessons and Challenges" which will be given by Tom Sühr. Tom Sühr is a doctoral candidate at the Max Planck Institute for Intelligent Systems in the Human Aspects of Machine Learning group.
We are looking forward to an exciting and productive gathering!
Arrival in Tübingen
The workshops take place in the "Neue Aula" (Geschwister-Scholl-Platz). We will provide more information soon.
From the train station you can take the busses 1, 2, 3, 4, and 7 to the stop "Geschwister-Scholl-Platz". The "Neue Aula" is on the other site of the street of the bus stop. When you enter the building, you can find the rooms by going straight ahead and turning left before leaving the building again.
The venue will be open from 8:30.
Workshops
Workshop 1: Psychometrics for Large Language Model Evaluation: Lessons and Challenges (in English) - Tom Sühr
As the capabilities of Large Language Models (LLMs) continue to expand, the need for rigorous evaluation methods becomes increasingly critical. This workshop dives into what the machine learning community can learn from psychometrics—specifically Item Response Theory (IRT) and Classical Test Theory (CTT)—to enhance the benchmarking of LLMs. We will also learn about potential pitfalls and critically investigate the application of existing psychometrics to LLMs.
The workshop will begin with a theoretical and practical introduction to LLMs, including hands-on coding examples that demonstrate how to prompt and finetune these models efficiently, with a focus on reducing memory requirements. Participants will then learn how to administer current benchmarks and evaluate LLM responses. Finally, we will analyze existing benchmarks with psychometric tools.
By the end of this workshop, attendees will have a foundational understanding of how LLMs work and how to effectively administer benchmarks for their evaluation. They will also learn how psychometric tools can offer insights into LLM performance, as well as an awareness of the challenges involved in applying these methods.
This session is ideal for machine learning researchers and practitioners looking to adopt or refine psychometric techniques in their work with LLMs. And psychometric or econometric researchers who are interested in an introduction to LLMs. Examples and distributed code will be in Python and R.
Please bring a laptop for the exercises that has Python (version 3.10+) and R installed. We will send an email before the event that includes more information about necessary packages.
Schedule
The workshops will take place on October 16 in the Neue Aula (Geschwister-Scholl-Platz) in Tübingen. The presented program is only an orientation. The specific timetable depends on the workshop and is given at a later time.
Wednesday, October 16
08:30 - 09:30 | Registration and coffee |
09:00 - 12:00 | Workshop (part 1) |
12:00 - 13:00 | Lunch break (included in fee) |
13:00 - 17:00 | Workshop (part 2) |
19:00 | Dinner |
Registration
If you are interested in joining, please write an email to office (including position, affiliation, and dinner attendence). @mz.uni-tuebingen.de
Please be aware that we fill the 20 spots in the workshop in the order of the registration.
The fee for the workshop is 90€ and it includes catering for the breaks and for lunch. On Wednesday evening we offer to go to dinner together at one's own expense.
Contact
If you have any questions please write an email to office @mz.uni-tuebingen.de
Organizing Committee: