You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, and thanks for HEST — the dataset and the benchmark harness have been
genuinely useful. This issue is about the gene-expression probing protocol,
not the data.
I am writing it up because I could not tell from the repo which of two
official protocols is the intended one, and it turns out the choice changes
the top-1 model on every task I tested.
I am aware of #140 ("fix(bench): respect alpha kwarg + leaderboard row (UNI2-h α-CV → 0.4338)") and #141 ("Clarify HEST public benchmark protocol"). Neither has a reply yet,
so I am adding measurements rather than opening a duplicate question.
The ambiguity
Two places in the official material describe the ridge probe differently:
Both use Ridge(solver='lsqr', fit_intercept=False) on 50 highly variable
genes. I will call them A1 (README) and A2 (Table A13).
Observation 1: the paper's own two printed tables already disagree
This needs no code from me — only the published PDF. Comparing Table 1 (p6)
with Table A13 (p26), over the 10 models present in both:
the top-1 model differs in 10/10
columns (all 9 tasks plus the Average column);
per-task Spearman between the two tables goes negative on 4 tasks
(PAAD -0.4667,
SKCM -0.5152,
LUAD -0.5394,
COAD -0.2970);
UNIv1.5 Average goes 0.4090 -> 0.2719
(best -> worst in that column), while CONCH goes
0.3709 -> 0.3541.
Is Table A13 meant to be a different protocol, a different data revision, or
an earlier run? The caption did not settle it for me.
Observation 2: re-running both protocols reproduces the swap on 9/9 tasks
9 tasks x 9 encoders, paired bootstrap over test spots (n=2000, shared
resample indices across arms, resampled per fold, point estimate from
unresampled data). "hard" = the two arms disagree in sign and both
confidence intervals exclude 0.
task
top-1 under A1 (README)
top-1 under A2 (Table A13)
differs
hardness
delta (A1 view)
CI
delta (A2 view)
CI
CCRCC
virchow2
phikon
yes
hard
+0.0338
[+0.0316, +0.0361]
-0.0390
[-0.0416, -0.0366]
COAD
uni2h
resnet50
yes
hard
+0.0638
[+0.0603, +0.0673]
-0.0235
[-0.0272, -0.0200]
IDC
virchow2
uni2h
yes
hard
+0.0040
[+0.0027, +0.0052]
-0.0235
[-0.0252, -0.0217]
LUNG
virchow2
phikon
yes
hard
+0.0185
[+0.0155, +0.0213]
-0.1855
[-0.1914, -0.1795]
LYMPH_IDC
uni2h
phikon
yes
hard
+0.0307
[+0.0284, +0.0329]
-0.0078
[-0.0104, -0.0053]
PAAD
uni2h
phikon
yes
hard
+0.0580
[+0.0531, +0.0624]
-0.0111
[-0.0174, -0.0049]
PRAD
gigapath
uni2h
yes
hard
+0.0131
[+0.0101, +0.0160]
-0.0354
[-0.0386, -0.0320]
READ
uni2h
gigapath
yes
hard
+0.0254
[+0.0212, +0.0294]
-0.0065
[-0.0108, -0.0020]
SKCM
uni2h
resnet50
yes
hard
+0.1843
[+0.1742, +0.1945]
-0.2281
[-0.2425, -0.2144]
top-1 differs on 9/9 tasks; 9 of them hard; paired bootstrap n=2000
Top-1 moves away from the pathology foundation models toward Phikon and
ResNet-50. On SKCM, ImageNet ResNet-50 beats every foundation model under A2.
Reproduction check, so you can tell whether I am running it wrong. My A1
scores against the printed leaderboard values, over the 63 model-task
cells where the two overlap (9 tasks): median absolute difference
0.0028, max 0.0144. That median is slightly larger than
the median adjacent gap between leaderboard rows (0.0025), which is itself part
of the point: at this resolution the ranking is not robust to protocol choice.
The A2 arm differs from A1 by far more than this reproduction error.
Observation 3: under A1 the ridge penalty is numerically inert
alpha = 100 / (256 * 50) = 0.0078. Against PCA-256 singular
values this sits below the smallest one, so the penalty does essentially
nothing: edf/d = sum_i s_i/(s_i + alpha) stays at
0.999999 (min 0.998954
over 243 fold-cells), and the A1 coefficients match
the unregularized least-squares solution to a median
4.41e-08 in predicted score.
protocol
fold-cell
edf/d median
edf/d min
edf/d>0.99
abs(A1-OLS) median
max
<1e-3
cond median
verdict
A1: README (PCA-256)
243
0.999999
0.998954
243/243
4.41e-08
1.69e-04
243/243
5.56e+02
SUPPORTED
A2: paper Table A13 (no PCA)
243
0.999184
0.504671
232/243
4.27e-05
1.28e-02
232/243
1.27e+15
REJECTED
criterion: edf/d>0.99 and |A1-OLS|<0.001 must hold in every cell. Under no-PCA, 18 cells fail at least one condition (6 explained by rank deficiency, 7 by an inert penalty, 5 unexplained)
So under the README protocol the reported scores are effectively OLS on
PCA-256 features, and the alpha formula is not doing the work its form
suggests. Under A2 (no PCA) the picture is different — the design is
rank-deficient (median condition number
1.27e+15) and 18 of
243 fold-cells fail the inertness test:
6 explained by rank deficiency,
7 by an inert penalty, and
5 I cannot currently explain — all of the
latter in a randomly-initialized ViT arm I carry as a negative control, so they
may well be my artifact rather than yours.
Observation 4: the formula's alpha is far from the alpha that maximizes score
Sweeping alpha over logspace(-6, 8, 29) per fold and reading the plateau
(averaging per-fold curves first, then taking the peak; plateau = within 0.002
of it, a threshold fixed in advance from the median adjacent gap on the
leaderboard, 0.0025):
plateau = within 0.002 of the peak (threshold fixed in advance from the 0.0025 median adjacent gap on the leaderboard); 9 tasks x 9 encoders x 6 dims = 486 cells
The official alpha lands in the plateau in 43/486
fold-cells, and in only 6/324
when dim >= 128. Median mismatch is 6.61 orders of
magnitude. The score left on the table is 0.0414 at
dim=256 and 0.1181 at full dimensionality (medians
across cells).
Two caveats that cut against reading too much into this. The cost is not
uniform across tasks: ccRCC has both the largest training folds and the most
folds of the 9 tasks, yet
shows the smallest cost (0.0114 at dim=256,
0.0230 at full dim) and alone supplies
3 of the
6 dim>=128 plateau hits. And the cost only
increases monotonically with dim in 51/81
model-task units, so "higher dim makes it worse" is a tendency, not a rule.
What I am actually asking
Which protocol is the intended one for the public leaderboard — A1 (README)
or A2 (Table A13)?
Would you accept a PR that (a) documents the protocol in one place, and
(b) adds an opt-in cross-validated-alpha path, leaving the current default
untouched so existing leaderboard numbers stay comparable?
Happy to share the full per-fold outputs and the scripts. I am preparing a
short write-up of this analysis; I would rather cite a maintainer answer than
speculate about intent, so any correction is welcome — including "you are
running it wrong", which is a real possibility I would like to rule out.
Hi, and thanks for HEST — the dataset and the benchmark harness have been
genuinely useful. This issue is about the gene-expression probing protocol,
not the data.
I am writing it up because I could not tell from the repo which of two
official protocols is the intended one, and it turns out the choice changes
the top-1 model on every task I tested.
I am aware of #140 ("fix(bench): respect alpha kwarg + leaderboard row (UNI2-h α-CV → 0.4338)") and
#141 ("Clarify HEST public benchmark protocol"). Neither has a reply yet,
so I am adding measurements rather than opening a duplicate question.
The ambiguity
Two places in the official material describe the ridge probe differently:
HEST-Benchmark results (03.04.26)leaderboard)100 / (dim * n_genes)Both use
Ridge(solver='lsqr', fit_intercept=False)on 50 highly variablegenes. I will call them A1 (README) and A2 (Table A13).
Observation 1: the paper's own two printed tables already disagree
This needs no code from me — only the published PDF. Comparing Table 1 (p6)
with Table A13 (p26), over the 10 models present in both:
columns (all 9 tasks plus the Average column);
(PAAD -0.4667,
SKCM -0.5152,
LUAD -0.5394,
COAD -0.2970);
(best -> worst in that column), while CONCH goes
0.3709 -> 0.3541.
Is Table A13 meant to be a different protocol, a different data revision, or
an earlier run? The caption did not settle it for me.
Observation 2: re-running both protocols reproduces the swap on 9/9 tasks
9 tasks x 9 encoders, paired bootstrap over test spots (n=2000, shared
resample indices across arms, resampled per fold, point estimate from
unresampled data). "hard" = the two arms disagree in sign and both
confidence intervals exclude 0.
top-1 differs on 9/9 tasks; 9 of them hard; paired bootstrap n=2000
Top-1 moves away from the pathology foundation models toward Phikon and
ResNet-50. On SKCM, ImageNet ResNet-50 beats every foundation model under A2.
Reproduction check, so you can tell whether I am running it wrong. My A1
scores against the printed leaderboard values, over the 63 model-task
cells where the two overlap (9 tasks): median absolute difference
0.0028, max 0.0144. That median is slightly larger than
the median adjacent gap between leaderboard rows (0.0025), which is itself part
of the point: at this resolution the ranking is not robust to protocol choice.
The A2 arm differs from A1 by far more than this reproduction error.
Observation 3: under A1 the ridge penalty is numerically inert
alpha = 100 / (256 * 50) = 0.0078. Against PCA-256 singularvalues this sits below the smallest one, so the penalty does essentially
nothing:
edf/d = sum_i s_i/(s_i + alpha)stays at0.999999 (min 0.998954
over 243 fold-cells), and the A1 coefficients match
the unregularized least-squares solution to a median
4.41e-08 in predicted score.
criterion: edf/d>0.99 and |A1-OLS|<0.001 must hold in every cell. Under no-PCA, 18 cells fail at least one condition (6 explained by rank deficiency, 7 by an inert penalty, 5 unexplained)
So under the README protocol the reported scores are effectively OLS on
PCA-256 features, and the alpha formula is not doing the work its form
suggests. Under A2 (no PCA) the picture is different — the design is
rank-deficient (median condition number
1.27e+15) and 18 of
243 fold-cells fail the inertness test:
6 explained by rank deficiency,
7 by an inert penalty, and
5 I cannot currently explain — all of the
latter in a randomly-initialized ViT arm I carry as a negative control, so they
may well be my artifact rather than yours.
Observation 4: the formula's alpha is far from the alpha that maximizes score
Sweeping alpha over
logspace(-6, 8, 29)per fold and reading the plateau(averaging per-fold curves first, then taking the peak; plateau = within 0.002
of it, a threshold fixed in advance from the median adjacent gap on the
leaderboard, 0.0025):
plateau = within 0.002 of the peak (threshold fixed in advance from the 0.0025 median adjacent gap on the leaderboard); 9 tasks x 9 encoders x 6 dims = 486 cells
The official alpha lands in the plateau in 43/486
fold-cells, and in only 6/324
when dim >= 128. Median mismatch is 6.61 orders of
magnitude. The score left on the table is 0.0414 at
dim=256 and 0.1181 at full dimensionality (medians
across cells).
Two caveats that cut against reading too much into this. The cost is not
uniform across tasks: ccRCC has both the largest training folds and the most
folds of the 9 tasks, yet
shows the smallest cost (0.0114 at dim=256,
0.0230 at full dim) and alone supplies
3 of the
6 dim>=128 plateau hits. And the cost only
increases monotonically with dim in 51/81
model-task units, so "higher dim makes it worse" is a tendency, not a rule.
What I am actually asking
or A2 (Table A13)?
alpha = 100/(dim * n_genes)intended as a fixed heuristic, or as astand-in for cross-validated alpha? fix(bench): respect alpha kwarg + leaderboard row (UNI2-h α-CV → 0.4338) #140 suggests the
latter was intended at some point.
(b) adds an opt-in cross-validated-alpha path, leaving the current default
untouched so existing leaderboard numbers stay comparable?
Happy to share the full per-fold outputs and the scripts. I am preparing a
short write-up of this analysis; I would rather cite a maintainer answer than
speculate about intent, so any correction is welcome — including "you are
running it wrong", which is a real possibility I would like to rule out.