
Graham Neubig
Associate Professor · Language Technologies Institute · Carnegie Mellon University
At Prof Rod’s reading desk
Questions to explore
- Which differences between language models disappear in an overall score?
- How can evaluation reveal a specific behavior rather than just rank systems?
Research interests
Graham Neubig works on natural language processing, language models and model evaluation.
- Natural language processing
- Language models
- Model evaluation
A question from Prof Rod
Same score. Same choice?
These models both pass four of six tests. Would you trust them with the same work? Let’s find a difference the average hides.
Six constructed outcomes, not measurements of real models.
Prof Rod’s reading notes
Start with a paper
ACL · 2025
BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models
Prof Rod’s reading note: Compare fine-grained differences with an aggregate model score. Ask how discovered behaviors are checked on held-out examples.
Bring a question of your own
Turn reading into research
Choose two model outputs that an overall accuracy score treats alike. Define a finer behavior you could measure and a held-out check.
Prof Rod’s evaluation lesson helps you ask what was sampled and what a comparison supports. This is our suggested preparation, not the researcher’s admission requirement.
Learn to compare model resultsKeep exploring
More at Carnegie Mellon University
- Aaron Steinfeld
Robotics
- Aarti Singh
- Abhinav Gupta
Machine learning · Computer vision
- Aditi Raghunathan
Improve this page
Suggest a correction, share a paper to start with, or point readers to a group page or opening.