Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

Static version: the interactive figure needs JavaScript.

Show the numbers as tables

Block-3 separation and test accuracy (%) on CIFAR-1004 blocks, width 256 · 3 seeds · mean ± SD · Tab. 3

Arm	Block-3 separation	Stage 1: FF’s own sum		Stage 2: probe (not FF)
		single crop	+ flip	single crop	+ flip
cumulative (γ = 0.7)	0.96 ± 0.01	66.78 ± 0.28	67.41 ± 0.35	68.51 ± 0.51	68.96 ± 0.43
hardness-gated (κ = 0)	1.71 ± 0.01	66.35 ± 0.41	67.35 ± 0.66	68.44 ± 0.51	69.07 ± 0.61
history-free (γ = 0)	4.78 ± 0.02	66.37 ± 0.20	66.93 ± 0.14	68.54 ± 0.25	69.23 ± 0.34
MGC	5.61 ± 0.04	65.73 ± 0.35	66.58 ± 0.25	68.24 ± 0.51	69.09 ± 0.27

Accuracy change (pp) with paired 95% CIsCIFAR-100: paired, 3 seeds × 10,000 test images (n = 30,000) · 5000-resample bootstrap, conditional on the trained models · Tab. 3 · CIFAR-10 row: 4 blocks of width 256 with 32 experts; its history-free seed-42 run used λ_curr = λ_depth = 0 (no depth-scaled auxiliary and no depth-order term; seeds 123 and 456: 0.25 and 0.2; Sec. 4.1, App. Tab. 22)

Comparison	Stage 1: FF’s own sum		Stage 2: probe (not FF)
	single crop	+ flip	single crop	+ flip
hardness-gated − cumulative	−0.43 [−0.79, −0.08]	−0.07 [−0.40, +0.27]	−0.07 [−0.39, +0.24]	+0.11 [−0.20, +0.42]
history-free − cumulative	−0.42 [−0.76, −0.07]	−0.48 [−0.82, −0.15]	+0.04 [−0.26, +0.34]	+0.28 [−0.02, +0.56]
MGC − cumulative	−1.05 [−1.46, −0.62]	−0.83 [−1.23, −0.42]	−0.27 [−0.68, +0.14]	+0.13 [−0.27, +0.53]
CIFAR-10, 32 experts: cumulative − history-free (CP-FAIR − γ=0)	−0.06 [−0.35, +0.23]	+0.05 [−0.20, +0.29]	+0.03 [−0.26, +0.31]	−0.07 [−0.32, +0.19]

CIFAR-10 control: where MGC’s separation gain comes from4 blocks, width 128 · 3 seeds per arm · SAM on · accuracy in %, Δ in pp with paired 95% CIs · App. B.1.1, Tab. 6 · The runs mix GPU types and software stacks, so (b) minus (a) is confounded with them; the separation effects are large against this variation, the accuracy differences are not.

Arm	Block-3 separation	Stage 1: FF’s own sum		Stage 2: probe (not FF)
		single crop	+ flip	single crop	+ flip
(a) cumulative + auxiliary	0.75 ± 0.01	85.74 ± 0.37	86.75 ± 0.34	85.98 ± 0.52	87.04 ± 0.51
(b) auxiliary removed, no compensator	4.88 ± 0.05	85.31 ± 0.10	86.47 ± 0.04	85.88 ± 0.34	86.90 ± 0.22
MGC	5.25 ± 0.05	85.45 ± 0.37	86.87 ± 0.23	86.07 ± 0.38	87.27 ± 0.29
(c) MGC + block-0 auxiliary	5.28 ± 0.03	85.32 ± 0.19	86.53 ± 0.19	85.66 ± 0.25	86.93 ± 0.09
Δ (b) − (a)	+4.13 ± 0.05	−0.43 [−0.74, −0.12]	−0.28 [−0.54, −0.01]	−0.10 [−0.40, +0.21]	−0.14 [−0.40, +0.11]
Δ MGC − (a)	+4.50 ± 0.04	−0.29 [−0.59, +0.02]	+0.13 [−0.14, +0.40]	+0.09 [−0.21, +0.40]	+0.24 [−0.02, +0.50]
Δ (c) − (a)	+4.54 ± 0.03	−0.42 [−0.74, −0.12]	−0.22 [−0.49, +0.06]	−0.32 [−0.62, −0.02]	−0.11 [−0.37, +0.16]

What moves accuracy (CIFAR-10 test, single crop, %)Tab. 1 (cumulative 4 seeds, history-free 3 seeds) and Tab. 2 (3 seeds; 89.03%: 1 seed, FF’s augmentation) · Backprop with strong augmentation uses FF’s per-view recipe (one view; FF draws three), and both plain-CNN rows use random crop and flip. † Seed 42 used λ_curr = λ_depth = 0 (no depth-scaled auxiliary and no depth-order term), seeds 123 and 456 used 0.25 and 0.2 (Sec. 4.1, App. Tab. 22).

Suspect	Setting	Accuracy
Training rule	FF, cumulative (γ = 0.7, 32 experts; 4 seeds)	90.36 ± 0.18
	FF, history-free (γ = 0; 3 seeds)†	90.50 ± 0.05
Architecture	FF on a plain CNN	29.85 ± 0.50
	FF on our stripped backbone (1 seed)	89.03%
	Backprop on the same plain CNN	90.08 ± 0.28
Augmentation	Backprop on the stripped backbone, weak augmentation	87.43 ± 0.62
	Backprop on the stripped backbone, strong augmentation	93.85 ± 0.18

Calibration, not a leaderboard

Measured

Our FF backbone vs. the strongest prior FF result we list · single crop; ours: γ = 0, 3 seeds

Calibration: our FF backbone vs. the strongest prior FF result listed

Dataset	Our FF backbone (γ = 0, Stage 1, single crop), %	Strongest prior FF result listed	Gap
CIFAR-10	90.50 ± 0.05†	ASGE, 90.62%	within 0.12 pp
CIFAR-100	66.37 ± 0.20	ASGE, 65.42%	+0.95 pp
Tiny ImageNet	48.54 ± 0.19	SCFF, 35.67% (unsupervised)	+12.9 pp

SCFF is unsupervised: not like-for-like. Calibration, not a leaderboard: architectures, readouts, and augmentation differ. None of the listed prior FF works report test-time augmentation, so our flip and probe numbers are not in this table. Our CIFAR-10 column is the 32-expert model; CIFAR-100 and Tiny ImageNet use the dense backbone. Tiny ImageNet accuracies are on its labeled validation split. † The CIFAR-10 mean pools seed 42, trained with λ_curr = λ_depth = 0 (no depth-scaled auxiliary and no depth-order term), with seeds 123 and 456 (0.25 and 0.2; Sec. 4.1, App. Tab. 22).

Sec. 4.2, App. Tab. 29

Every prior FF result listed (App. Tab. 29)

Prior Forward-Forward results as each work reports them (%)Paper order, not sorted · headline accuracies as each work reports them; calibration, not a leaderboard · App. Tab. 29 · SCFF’s Tiny ImageNet value as printed in note f

Prior FF work	CIFAR-10	CIFAR-100	Tiny ImageNet
Hinton (2022)	59	—	—
Lorberbom et al. (2024)	48.4	—	—
Papachristodoulou et al. (2024), CwC	78.11	51.23	—
Dooms et al. (2024), Trifecta	∼84	35.79	—
Aghagolzadeh and Ezoji (2025), CFFM	85.37	—	—
Wu et al. (2024), Distance-Forward	88.15	59.01	—
Sun et al. (2025), DeeperForward	86.22	53.09	—
Krutsylo (2025), SFF	81.38	55.34	—
Chen et al. (2025), SCFF (unsupervised)	80.8	—	35.67
Sarode et al. (2026), HFF	83.08	54.34	—
Gong et al. (2026), ASGE (linear readout)	90.62	65.42	—

Does the dissociation hold elsewhere?

Across seeds, deepest-block separation shows no detectable association with test accuracy (exploratory).

1. QDid you just repair something that does not matter?The pathology is real and repairable, but in the settings we study it is not the dominant accuracy bottleneck.

   The pathology is real and repairable, but in the settings we study it is not the dominant accuracy bottleneck. That is a tested result: “If free-riding limited accuracy, repairing it should help. It does not: block health improves dramatically …” The attenuation is exact (Theorem 3.1) and measurable (free-riding index 0.930–0.955 at blocks 1–3 under γ = 0.7). Removing or gating the history raises block-3 separation up to 5× on CIFAR-100, yet every paired 95% CI of the history rules lies inside ±1 pp (Tab. 3). The repairs still change what the network learns (retrained variants disagree on 13–26% of test predictions), so they are “diagnostic and mechanism-isolation tools, not accuracy gains.”

   Paper: Sec. 1, Sec. 5.

2. QWhy does accuracy not move if the network changes?FF predicts from the sum of block goodness, so moving evidence between blocks without changing any class’s summed score leaves predictions unchanged (Prop. 3.2).

   FF predicts from the sum of block goodness, so moving evidence between blocks without changing any class’s summed score leaves predictions unchanged (Prop. 3.2): re-splitting a trained CIFAR-100 model’s goodness so the deepest block’s own share moves from 14% to 98% flips 0 of 10,000 predictions. The repairs are not exact redistributions, so their stable accuracy, under this rule and under the structured trained readouts we test, is empirical. The history rules disagree on ∼13% of test predictions (probe + flip), nearly balanced in correctness.

   Paper: Sec. 1, Sec. 3.2, App. B.7.

3. QDo the repaired networks differ at all?Yes. About 13% of test predictions change between history rules (probe + flip).

   Yes. About 13% of test predictions change between history rules (probe + flip). Pooled-embedding, block-3-only and k-NN probes separate the variants by more than 1 pp (block-3-only logistic: 8.40 pp; k-NN on raw goodness: 8.58 pp), while fitted readouts of the per-block goodness keep them within 0.38 pp. “The invariance claim is therefore scoped to the deployed rule and the structured readouts, not to every probe.”

   Paper: App. B.5, Tab. 10, Tab. 11.

4. QWhere does MGC’s separation gain come from?On CIFAR-100, MGC raises deepest-block separation 5.9× (5.61 vs. 0.96) at a consistent Stage-1 cost that the Stage-2 probe recovers.

   On CIFAR-100, MGC raises deepest-block separation 5.9× (5.61 vs. 0.96) at a consistent Stage-1 cost (−1.05 [−1.46, −0.62] pp) that the Stage-2 probe recovers (+0.13 [−0.27, +0.53] pp with flip). In a smaller CIFAR-10 control (4 blocks of width 128, 3 seeds per arm), removing the depth-scaled auxiliary alone reaches 4.88 against MGC’s 5.25 and the cumulative arm’s 0.75: of MGC’s gain of 4.50, 4.13 (92%) comes from that removal and 0.37 (8%) from the compensation. “Our prediction, registered before the runs, that it would stay below 4.72 failed.” How the two interact is open, and the CIFAR-100 gain is not decomposed.

   Paper: Sec. 4.5, App. B.1.1.

5. QIs this specific to the softplus loss?The gradient identity of Theorem 3.1 uses only the chain rule; the barrier sets only the decay profile.

   The gradient identity of Theorem 3.1 uses only the chain rule: for any barrier whose |ℓ′| is non-increasing in the accumulated margin, R ≤ 1 whenever γP ≥ 0, and the barrier sets only the decay profile. The squared hinge max(0, θ − u)² with θ = 2 decays linearly to a hard zero and reproduces both free-riding (deepest-block separation 1.67 at γ = 0.7 vs. 4.44 at γ = 0) and the dissociation (87.08 vs. 86.93% with probe + flip) on CIFAR-10 with 4 blocks of width 128 and 3 seeds per arm. Alternative goodness statistics remain open.

   Paper: Remark after Theorem 3.1, App. B.2.

6. QWhat about deeper networks and harder data?At 8 blocks, hardness gating raises block-7 separation ∼45× over γ = 1 without the auxiliary, yet accuracy moves +0.05 pp.

   At 8 blocks, hardness gating raises block-7 separation ∼45× over γ = 1 without the auxiliary (6.94 vs. 0.15, seed 42), yet accuracy moves +0.05 pp (86.84% vs. 86.79%, probe + flip). With 32 experts at 8 blocks all three regimes stay within 1 pp; the depth-scaled repair reaches 89.61% vs. 89.22% and 89.20% on one seed, so whether repairs begin to matter at greater depth is open. On Tiny ImageNet (γ = 0, 3 seeds; 52.32 ± 0.34% top-1 with probe + flip), separation peaks at block 1 and declines deeper, a depth effect outside Theorem 3.1 whose cause is open, yet per-depth accuracy rises with every block.

   Paper: Sec. 4.3, App. C, App. F.1.

7. QCould the co-designed recipe, not the mechanism, produce the dissociation?Four controls bear on this; they mitigate, but do not eliminate, the concern.

   Four controls bear on this: the central CIFAR-100 trio is dense, with no MoE and no SAM; removing SAM or SupCon, or setting γ = λ_curr = λ_depth = 0, each changes accuracy (probe + flip) by +0.25 pp (CIFAR-10, seed 42); FF on the stripped backbone still reaches 89.03% (one seed); and the squared-hinge barrier reproduces the dissociation, while MGC moves no protocol toward a gain. “These controls mitigate, but do not eliminate, the concern that the recipe drives the result.”

   Paper: Sec. 4.5.

8. QDo deep blocks get no gradient at all?No. Theorem 3.1 concerns the cumulative discrimination term only.

   No. Theorem 3.1 concerns the cumulative discrimination term only. Under γ = 0.7 that term’s gradient share falls from 0.247 at block 0 to 0.030 at block 3, while the current-block auxiliary has the largest share at blocks 1–3 (0.52–0.72), and the current-block term gives unresolved examples a gradient floor (Theorem A.2). The theorem is “a surrogate-loss statement, not a convergence claim.”

   Paper: Sec. 3.2, App. A, App. B.3.

9. QOnly three seeds: how solid is “inside ±1 pp”?The comparisons are paired: per-example correctness differences on the same test images, pooled over the three seeds both arms share.

   The comparisons are paired: per-example correctness differences on the same test images, pooled over the three seeds both arms share (n = 30,000 predictions per pair on CIFAR-100), with 5000-resample bootstrap 95% CIs judged against a stated ±1 pp band, plus exact McNemar tests per seed. The CIs are conditional on the trained models; seed-to-seed spread is reported separately (mean ± SD in Tab. 3, per-seed differences in Tab. 5 and Tab. 12). A CI outside ±1 pp is not read as no change: MGC’s two Stage-1 CIs reach below −1 pp and are reported as a cost.

   Paper: Tab. 3, App. B.7, Checklist 7.

10. QWhy a Stage-2 probe?It is a representation probe, not part of FF, and unlike the FF sum it is not bound by Proposition 3.2.

    It is a representation probe, not part of FF, and unlike the FF sum it is not bound by Proposition 3.2, so the paper reports both stages. It is attentive because a single local readout block, trained with the FF objective or with cross-entropy, stays 5.3–6.4 pp below it (seed 42): the gap is architectural.

    Paper: Sec. 4.1, App. B.4.

11. QDoes this cover Hinton’s original Forward-Forward, or FF in general?No: the claim is scoped to cumulative-goodness FF and to the architectures and datasets evaluated.

    No. “The claim is scoped to cumulative-goodness FF with softplus and squared-hinge barriers and to the architectures and datasets evaluated here, not to FF in general.” ImageNet scale and classic FF architectures are untested, and our models inject the label at every block, unlike Hinton (2022).

    Paper: Sec. 1, Sec. 3.1, Sec. 5.

12. QHow does the FF backbone compare with prior Forward-Forward results?As calibration, not a leaderboard: the listed works differ in backbone size, supervision, strict locality, augmentation, and test-time augmentation.

    As calibration, not a leaderboard: the listed works differ in backbone size, supervision, strict locality, augmentation, and test-time augmentation. Single crop, the history-free backbone is within 0.12 pp of the strongest single-crop FF result listed on CIFAR-10 (90.50% vs. ASGE’s 90.62%, which uses a linear readout; our mean is 0.12 pp lower), +0.95 pp on CIFAR-100 (66.37% vs. 65.42%), and +12.9 pp above SCFF’s unsupervised Tiny ImageNet result (48.54% vs. 35.67%; not like-for-like). None of the listed prior FF works report test-time augmentation. The 91.98% in Tab. 1 is a seed-42 validation accuracy, not a test number. The CIFAR-10 mean pools seed 42, trained with λ_curr = λ_depth = 0, with seeds 123 and 456 (0.25 and 0.2; App. Tab. 22).

    Paper: Sec. 2, Sec. 4.2, App. Tab. 29.

@inproceedings{yousefiramandi2026cumulative,
  title     = {Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant},
  author    = {Yousefiramandi, Amirhossein},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS 2026)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.06240}
}

The same metadata is in CITATION.cff in the code repository.

The paper’s NeurIPS 2026 forum on OpenReview: openreview.net/forum?id=TN0FvatX6A