Methodenzentrum

3rd Fall School of the Methods Center on October 16, 2024

Interdisciplinary Methods

The Methods Center at the University of Tübingen cordially invites PhD candidates, postdoctoral researchers, and professors to join our Fall School. 
The topic of this year's workshop is "Psychometrics for Large Language Model Evaluation: Lessons and Challenges" which will be given by Tom Sühr. Tom Sühr is a doctoral candidate at the Max Planck Institute for Intelligent Systems in the Human Aspects of Machine Learning group.

We are looking forward to an exciting and productive gathering!

Arrival in Tübingen

The workshop will take place in the "Altbau Theologicum" in the Liebermeisterstr. 14, Tübingen.

From the train station you can take, e.g., the bus 5 (direction “Derendingen Käppele - Feuerhägle - Hauptbahnhof - Kliniken - Wanne - Waldhäuser Ost”) to the stop "Uni-Kliniken Tal - Tübingen". From there the building is in 4 minutes walking distance.

A room map can be found here. The workshop is in Seminarraum 1.

The room will be open from 10:00.

Workshops

Workshop 1: Psychometrics for Large Language Model Evaluation: Lessons and Challenges (in English) - Tom Sühr

As the capabilities of Large Language Models (LLMs) continue to expand, the need for rigorous evaluation methods becomes increasingly critical. This workshop dives into what the machine learning community can learn from psychometrics—specifically Item Response Theory (IRT) and Classical Test Theory (CTT)—to enhance the benchmarking of LLMs. We will also learn about potential pitfalls and critically investigate the application of existing psychometrics to LLMs.

The workshop will begin with a theoretical and practical introduction to LLMs, including hands-on coding examples that demonstrate how to prompt and finetune these models efficiently, with a focus on reducing memory requirements. Participants will then learn how to administer current benchmarks and evaluate LLM responses. Finally, we will analyze existing benchmarks with psychometric tools.

By the end of this workshop, attendees will have a foundational understanding of how LLMs work and how to effectively administer benchmarks for their evaluation. They will also learn how psychometric tools can offer insights into LLM performance, as well as an awareness of the challenges involved in applying these methods.

This session is ideal for machine learning researchers and practitioners looking to adopt or refine psychometric techniques in their work with LLMs. And psychometric or econometric researchers who are interested in an introduction to LLMs. Examples and distributed code will be in Python and R.

Please bring a laptop for the exercises that has Python (version 3.10+) and R installed. We will send an email before the event that includes more information about necessary packages.

 

Schedule

The workshops will take place on October 16 in Seminarraum 1 (EG, ground floor), Liebermeisterstr. 14 in Tübingen. The presented program is only an orientation. The specific timetable depends on the workshop and is given at a later time.

Wednesday, October 16

10:00  - 12:30Workshop (part 1)
12:30 - 13:30Lunch break (included in fee)
13:30 - 16:00Workshop (part 2)
19:00Dinner

Registration

If you are interested in joining, please write an email to officespam prevention@mz.uni-tuebingen.de (including position, affiliation, and dinner attendence).

Please be aware that we fill the 20 spots in the workshop in the order of the registration.

The fee for the workshop is 60€ and it includes catering for the breaks and for lunch. On Wednesday evening we offer to go to dinner together at one's own expense.

Contact

If you have any questions please write an email to officespam prevention@mz.uni-tuebingen.de

 

Organizing Committee:

Holger Brandt

Augustin Kelava