Skip to content
Prof Rod Avatar
The research map

Graham Neubig

Associate Professor · Language Technologies Institute · Carnegie Mellon University

Back to the research map

At Prof Rod’s reading desk

Questions to explore

  • Which differences between language models disappear in an overall score?
  • How can evaluation reveal a specific behavior rather than just rank systems?

Research interests

Graham Neubig works on natural language processing, language models and model evaluation.

  • Natural language processing
  • Language models
  • Model evaluation

A question from Prof Rod

Same score. Same choice?

These models both pass four of six tests. Would you trust them with the same work? Let’s find a difference the average hides.

Six constructed outcomes, not measurements of real models.

What can the overall score tell you?

Prof Rod’s reading notes

Start with a paper

Bring a question of your own

Turn reading into research

Choose two model outputs that an overall accuracy score treats alike. Define a finer behavior you could measure and a held-out check.

Prof Rod’s evaluation lesson helps you ask what was sampled and what a comparison supports. This is our suggested preparation, not the researcher’s admission requirement.

Learn to compare model results

Keep exploring

More at Carnegie Mellon University

Explore the university

Improve this page

Suggest a correction, share a paper to start with, or point readers to a group page or opening.