AI evaluation and measurement
Testing whether language models do what we say they do, using the habits of psychological measurement: human reference standards, validated judges, and attention to how much a result depends on the analyst’s choices.
QualFrame and the analytic multiverse
Preprint · open toolsAn open-source research prototype for AI-assisted directed content analysis: a language model applies a codebook to transcripts, and a person reviews every proposed code beside its source passage, with provenance, versioned methods and traceable exports.
A screening study of 32 configurations shows how setup choices, and the way agreement is scored, change how well AI coding matches human coding.
LLM feedback on resident scholarly projects
PreprintA rubric-aligned feedback assistant built on open-weight LLMs, tested in a blinded evaluation of 240 feedback reports: 120 LLM-generated and 120 expert-written.
Expert feedback was rated higher overall, and the gap varied by project stage and type, pointing to where expert review remains most important.
AI-assisted admissions scoring
EvaluationEvaluated an AI-assisted scoring pipeline for roughly 2,200 autobiographical sketches per admissions cycle, with bilingual support and equity auditing built into the evaluation framework.
Pre-specified targets: ≥0.70 kappa and ≥0.80 AUC, with ~550 faculty hours a year potentially reclaimed.
Pressure or Persuasion?
Exploratory · InspectAn Inspect evaluation of whether operator instructions push assistants from persuasion towards manipulation when a customer asks to cancel a subscription or opt out of data sharing. Three models, 72 conversations, a deterministic check that the cancel route was given, and two blinded judges validated on 120 reference conversations with planted tactics.
Every model gave the route in its first reply and neither judge flagged manipulation. The result is narrow (two scenarios, three conversations per cell); the reusable part is the measurement.
Intended or Perceived?
Exploratory · not peer reviewedAn exploratory study of whether audio models’ emotion labels track what an actor intended or what listeners heard, using 300 acted clips from CREMA-D and three models, with repeated runs to measure label stability.
The models matched the listener plurality on 64–67% of clips, against 58% for always answering Neutral, and matched the intended emotion on only 23–39%.