From nuclei to tissue
Two specialised models rather than one shared backbone, aimed at putting individual nuclei back into the tissue context that gives them clinical meaning.
0.62
Aggregated Jaccard
five fold mean
0.54
Panoptic Quality
0.65
Macro F1
4
Nuclei classes
tumor, TILs, other, background
The question
Tumor infiltrating lymphocytes are a biomarker for immune checkpoint response in melanoma, but counting them is not enough. What matters is where they sit, since a lymphocyte inside the tumor and one in the surrounding stroma carry different meaning. That needs instance level precision for nuclei and semantic context for tissue at the same time, and a single shared backbone has to compromise between the fine detail one needs and the long range context the other does.
Approach
I split the problem rather than the backbone. Mask R-CNN with a ResNet50-FPN handles nuclei, with anchor sizes from 4 to 256 pixels and aspect ratios from 0.2 to 8.0 to match how varied nuclear shapes actually are, and a separate DynUNet handles the six tissue classes. Work ran on the PUMA melanoma dataset, patchified to 256 pixels at stride 128, with rare classes oversampled by inverse frequency because tumor nuclei alone are over 80 percent of instances. Five fold cross validation, scored with Panoptic Quality, Aggregated Jaccard Index and macro F1.
Outcome
The nuclei stage came out consistent across all five folds at a mean Panoptic Quality of 0.54, Aggregated Jaccard Index of 0.62 and macro F1 of 0.65, which places the AJI slightly ahead of the HoVer-Net figures reported on comparable data. The gap between the AJI and the Panoptic Quality is the honest finding: overlap with the dominant tumor class is strong, and the rarer categories are where the remaining work is. The tissue stage and the merge into a single panoptic mask are specified and partly built, and stand as the next step rather than a finished result.
What it taught me
Choosing two specialists over one generalist was the right call for a reason that is easy to state and easy to forget: the two tasks need features at different scales, and a shared trunk has to pick. I also came away with a much clearer sense of what class imbalance does to an instance metric, since oversampling moved the macro F1 far more than it moved the Dice.