Proposal Specification · Sections 2, 3 & 4

Models & Five-Agent Architecture

GlassBox uses a single shared foundation backbone (UNI) with lightweight per-organ diagnosis heads, fine-tuned SAM2 for promptable cell segmentation, and an adversarial multi-agent verification loop.

SECTION 4

The Five Agents

Launch Live Workbench →

AGENT 01

Orchestrator

Slide Intelligence & ReAct Loop

Indexes gigapixel whole-slide pyramids, extracts candidate tissue patches, and coordinates the multi-agent investigation state machine.

OpenSlideOtsu FilterReAct Controller

in WSI → out Tile[]

AGENT 02

Visual Analysis

Morphology & SAM2 Segmentation

Extracts deep visual embeddings and segments individual cell nuclei and tissue boundaries into addressable masks.

UNI BackboneSAM2 (HEST fine-tuned)CellViT

in Tile → out Mask[]

AGENT 03

Evidence Investigation

Transcriptomics & Multimodal Fusion

Aligns spatial transcriptomic barcode sequencing with morphology to surface diagnostically relevant regions.

CLAM Attention MILk-NN RetrievalVisium Spatial

in Region → out Evidence[]

AGENT 04

Diagnosis and Verification

Adversarial Calibration & Loopback Gate

Formulates diagnostic hypotheses and adversarially challenges them across 7 confidence components, triggering loopback on discordance.

Conformal Predictor (90%)Temperature ScalingMahalanobis OOD

in Bundle → out Verdict

AGENT 05

Explanation and Report

Evidence-Grounded Reporting & Q&A

Synthesizes formal 6-section clinical diagnostic reports citing verified evidence and powers interactive clinician Q&A.

CONCH BioGPT TokenizerEvidence-Constrained RAG

in Verified → out Report

SECTION 2

Per-Organ Diagnosis Heads (Frozen UNI Backbone)

UNI is a shared foundation backbone, not retrained per organ. A lightweight classification head is fine-tuned per organ on top of frozen UNI embeddings — standard practice, avoids retraining a large model per organ, and is straightforward to defend.

ORGAN	DATASETS (PRIMARY + REFS)	DIAGNOSIS MODEL	CLINICAL TARGET (DEMO CASE)	ACTION
Breast	HEST-1k (Breast subset) + TCGA-BRCA, CAMELYON16/17, BreakHis	UNI embeddings + fine-tuned classification head	Invasive Ductal Carcinoma (IDC), Nottingham Grade II, ER+/HER2-	Inspect in Demo →
Lung	HEST-1k (Lung subset) + TCGA-LUAD, TCGA-LUSC	UNI embeddings + fine-tuned classification head	Invasive Lung Adenocarcinoma (LUAD), Acinar predominant with lepidic margin	Inspect in Demo →
Colorectal	HEST-1k (Colon subset) + TCGA-COAD/READ, NCT-CRC-HE-100K	UNI embeddings + fine-tuned classification head	Colorectal Adenocarcinoma, Moderately Differentiated, MSI-High phenotype	Inspect in Demo →
Prostate	HEST-1k (Prostate subset) + PANDA, TCGA-PRAD	UNI embeddings + fine-tuned classification head	Prostatic Adenocarcinoma, Gleason Score 4+3=7 (ISUP Grade Group 3)	Inspect in Demo →

Organ Selection Rule: Choose the four organs from those HEST covers well at cell resolution — verify before committing.

SECTION 3

Pipeline-Wide Models (Shared Across All Organs)

Core models and methods shared across the diagnostic pipeline regardless of organ context.

Segmentation (A2 Cell analysis, A3 Grounding)Agent 2 (Visual Analysis), Agent 3 (Grounding)

SAM2 (Segment Anything 2)

Fine-tuned on HEST-1k cell/nuclei masks. Promptable, interactive segmenter producing cell- and region-level contours from point/box prompts. Evaluated fine-tuned vs. zero-shot.

Cell- and region-level segmentation producing promptable contours from point/box prompts.

Note: Segmentation is a core component and a contribution, not a helper. Classification / diagnosis sits on top of the segmented, grounded regions rather than beside it.

Segmentation benchmark / label sourceBenchmark / Label Source

CellViT, HoVer-Net, StarDist, Cellpose

Specialist nucleus segmenters used as the reference benchmark and (via CellViT) as the label source.

Reference benchmark and ground truth label source. Not the runtime deliverable.

Note: Disclosed honestly. SAM2 provides interactive promptable runtime segmentation.

Image encoder (A2, A3)Agent 2 (Visual Analysis), Agent 3 (Evidence)

UNI Pathology Foundation Model

Pathology foundation model (Chen et al., Nature Medicine 2024, Mahmood Lab). Self-supervised tile embeddings for grounding sampled patches to slide coordinates.

Shared backbone providing frozen embeddings for grounding sampled patches and feeding organ diagnosis heads.

Note: UNI is a shared backbone, not retrained per organ. Lightweight classification heads are fine-tuned on top of frozen embeddings.

Molecular region grouping (A3)Agent 3 (Evidence Investigation)

Transcriptomics-Guided Clustering

Cluster segmented cells by spatial-transcriptomic profile (on HEST samples with transcriptomics) to delineate regions by molecular identity, not appearance alone.

Delineates regions by molecular identity, not appearance alone.

Note: The project's distinguishing contribution over SlideSeek.

Report writer (A5/A9)Agent 5 (Explanation & Report)

CONCH / Transformer Decoder + BioGPT

CONCH (vision-language; Lu et al. 2024) or a Transformer decoder over UNI embeddings with a BioGPT tokenizer. Approach under evaluation.

Generates grounded impressions constrained to cite exact regions.

Note: Constrained to cite verified evidence only; eliminates hallucinated assertions.

Verifier / confidence (A4/A7)Agent 4 (Diagnosis & Verification)

Temperature Scaling & Conformal Prediction

Temperature scaling or conformal prediction over model confidence, targeting distribution-free coverage (e.g. 90%) under standard conformal assumptions.

Independent hypothesis verification; rejection/escalation logic lives in the orchestrator.

Note: Rejection/escalation threshold logic lives in the Orchestrator, triggering automated loop re-entry.