A benchmark is an instrument, not a scoreboard
Working on VISTA changed how I report results in a completely different field.
VISTA asks whether vision language models actually understand interactions in video or whether they recognise objects and guess the rest. Building it taught me something that transferred straight into medical imaging, which was not what I expected from a video benchmark.
Where evaluation harnesses actually break
The hard part was never running the models. It was normalising their output. Eleven systems, each producing free form generative text, each with its own conventions about phrasing, hedging and format. Turning that into something comparable is most of the actual engineering, and it is the part that determines whether every downstream number means anything.
Get it right and every number downstream means something. Get it approximately right and you get a leaderboard that looks fine while quietly measuring your string matching rather than their understanding.
What the taxonomy revealed
VISTA decomposes videos into entities, their actions, and the relations between them, which makes it possible to break performance down along complementary axes instead of collapsing everything into one accuracy figure.
Doing that surfaced pronounced spatio-temporal biases that the aggregate had averaged away. Models sitting within a point of each other overall behaved very differently once the axes were separated. One would handle spatial relations and lose the thread on temporal ordering; another the reverse. On a single number they looked like near equals.
Aggregate metrics are a compression, and compression discards exactly the information you need for diagnosis.
The habit it gave me
When I moved to registration, the standard way to report a result is mean Dice over 30 anatomical structures. That is the same compression. A model can post a respectable mean while the small subcortical structures go unrecovered, and those happen to be the ones that matter for most neuroimaging questions.
So my work reports per structure breakdowns, per case distributions, violin plots rather than bars, and critical difference diagrams rather than a ranked table. Not because reviewers ask for it, but because I have seen how much the mean averages away.