The geometry of generalization
Four training runs on paediatric brain tumor MRI, held identical except for the loss and the optimiser, then read through the curvature of the loss surface rather than the leaderboard.
6.84 mm
Boundary error
HD95, from 8.05 mm
84.9%
Whole tumor Dice
baseline 85.9%
about 501
Curvature spike
top Hessian eigenvalue
3500 to 1
Class imbalance
background to cystic
The question
Paediatric tumors are small, multi focal and texturally varied, and the background outnumbers the rarest tumor voxels by roughly 3500 to 1. At that ratio the descent direction is set almost entirely by background, so the minority class contributes a negligible perturbation. The usual response is to try a different optimiser or a boundary aware loss and read the Dice that comes out, which says what happened but never why.
Approach
I ran four arms on identical seeds, splits and augmentations, varying only the objective and the update rule: Dice with cross entropy under AdamW as the baseline, the same plus a signed distance surface loss, that combination under Sharpness Aware Minimisation, and Dice with cross entropy under Lion. Five fold cross validation on an H100. Alongside the usual scores I took the Hessian spectrum with PyHessian at intervals through training, so each arm has a curvature record as well as an accuracy record.
Outcome
The surface loss traded volume for boundary: whole tumor Dice moved 85.9 to 84.9 percent while HD95 improved 8.05 to 6.84 mm, so it bought 1.21 mm of boundary fidelity for one point of overlap. Lion never left its initialisation, which the sign update explains directly, since projecting onto the hypercube discards the gradient magnitude that was the only thing separating a rare tumor voxel from background. Sharpness Aware Minimisation did not help either, and the Hessian says why: the top eigenvalue under the surface loss reached about 501, so the perturbation radius was wider than the valley it was meant to be smoothing and the step climbed out instead. A persistent negative Hessian trace puts the trajectory around saddle points rather than in a convex basin.
What it taught me
The most useful part was that every one of those conclusions is invisible in the Dice column. Two arms can post similar scores for entirely different reasons, and only the curvature separates them. It also produced a concrete proposal rather than an observation: bound the surface loss so its gradient cannot run away, scale the perturbation by weight magnitude, and introduce the geometric term only once the trajectory is already in a stable basin.