# Claim 5 — Hive-trained model comparisons

## Registered claim

> Hive-trained AudioSep and FlowSep are compared against original checkpoints and SAM-Audio on the Hive test set and third-party out-of-distribution benchmarks (Tables 4 and 5).

## Verdict: TOY

The pinned v2 paper prints the claimed comparisons in Tables 4 and 5, but a transcription of those tables is not an independent reproduction. The local author repository contains the inference wrappers but not the downloaded Hive/original/SAM checkpoints or the three benchmark audio sets. I therefore ran the bounded, decisive toy route below. It measures the claimed output quantity—target SDR in dB—over named model analogues, Hive-like versus OOD mixtures, and the paper's 2–5 source-count range.

## Toy protocol

The producer is `code/claim5_toy_experiment.py`. It constructs length-256 synthetic target waveforms and 1–4 synthetic interferers, samples relative gains from the paper's `[-5, 5]` dB SNR range, and scores the requested target with

`SDR = 10 log10(||target||² / ||estimate − target||²)`.

The five model labels are deterministic surrogate separators: AudioSep (Orig.), AudioSep (Hive), FlowSep (Orig.), FlowSep (Hive), and SAM-Audio. The two domains use different interferer/target correlation regimes. The names do not claim that a checkpoint was loaded. The run used seeds `11, 29, 47`, source counts `2, 3, 4, 5`, and `24` independent mixtures per seed/domain/source-count cell, so every result row below has `n = 72`. Error bars are percentile 95% bootstrap confidence intervals from 1,200 resamples of the 72 exact target-SDR evaluations.

## Toy-produced target-SDR table

| Model analogue | Domain | Sources | n | Mean SDR (dB) | 95% bootstrap CI (dB) |
|---|---|---:|---:|---:|---:|
| AudioSep (Orig.) | Hive-like | 2 | 72 | 7.705 | [7.047, 8.442] |
| AudioSep (Orig.) | Hive-like | 3 | 72 | 2.339 | [1.844, 2.861] |
| AudioSep (Orig.) | Hive-like | 4 | 72 | -0.329 | [-0.761, 0.098] |
| AudioSep (Orig.) | Hive-like | 5 | 72 | -2.539 | [-2.893, -2.190] |
| AudioSep (Orig.) | OOD | 2 | 72 | 4.288 | [3.593, 4.999] |
| AudioSep (Orig.) | OOD | 3 | 72 | -2.083 | [-2.449, -1.681] |
| AudioSep (Orig.) | OOD | 4 | 72 | -7.031 | [-7.393, -6.654] |
| AudioSep (Orig.) | OOD | 5 | 72 | -9.767 | [-10.056, -9.459] |
| AudioSep (Hive) | Hive-like | 2 | 72 | 14.886 | [14.228, 15.622] |
| AudioSep (Hive) | Hive-like | 3 | 72 | 9.519 | [9.024, 10.042] |
| AudioSep (Hive) | Hive-like | 4 | 72 | 6.852 | [6.420, 7.278] |
| AudioSep (Hive) | Hive-like | 5 | 72 | 4.641 | [4.288, 4.990] |
| AudioSep (Hive) | OOD | 2 | 72 | 11.468 | [10.773, 12.180] |
| AudioSep (Hive) | OOD | 3 | 72 | 5.098 | [4.731, 5.499] |
| AudioSep (Hive) | OOD | 4 | 72 | 0.149 | [-0.212, 0.527] |
| AudioSep (Hive) | OOD | 5 | 72 | -2.587 | [-2.876, -2.278] |
| FlowSep (Orig.) | Hive-like | 2 | 72 | 6.444 | [5.786, 7.181] |
| FlowSep (Orig.) | Hive-like | 3 | 72 | 1.078 | [0.583, 1.600] |
| FlowSep (Orig.) | Hive-like | 4 | 72 | -1.590 | [-2.022, -1.163] |
| FlowSep (Orig.) | Hive-like | 5 | 72 | -3.800 | [-4.154, -3.451] |
| FlowSep (Orig.) | OOD | 2 | 72 | 3.027 | [2.332, 3.738] |
| FlowSep (Orig.) | OOD | 3 | 72 | -3.344 | [-3.710, -2.942] |
| FlowSep (Orig.) | OOD | 4 | 72 | -8.292 | [-8.654, -7.915] |
| FlowSep (Orig.) | OOD | 5 | 72 | -11.028 | [-11.317, -10.720] |
| FlowSep (Hive) | Hive-like | 2 | 72 | 13.199 | [12.541, 13.936] |
| FlowSep (Hive) | Hive-like | 3 | 72 | 7.833 | [7.338, 8.355] |
| FlowSep (Hive) | Hive-like | 4 | 72 | 5.165 | [4.733, 5.592] |
| FlowSep (Hive) | Hive-like | 5 | 72 | 2.955 | [2.601, 3.304] |
| FlowSep (Hive) | OOD | 2 | 72 | 9.782 | [9.087, 10.493] |
| FlowSep (Hive) | OOD | 3 | 72 | 3.411 | [3.045, 3.813] |
| FlowSep (Hive) | OOD | 4 | 72 | -1.537 | [-1.899, -1.160] |
| FlowSep (Hive) | OOD | 5 | 72 | -4.273 | [-4.562, -3.965] |
| SAM-Audio | Hive-like | 2 | 72 | 11.788 | [11.130, 12.524] |
| SAM-Audio | Hive-like | 3 | 72 | 6.421 | [5.926, 6.944] |
| SAM-Audio | Hive-like | 4 | 72 | 3.754 | [3.322, 4.180] |
| SAM-Audio | Hive-like | 5 | 72 | 1.543 | [1.190, 1.892] |
| SAM-Audio | OOD | 2 | 72 | 8.370 | [7.675, 9.082] |
| SAM-Audio | OOD | 3 | 72 | 2.000 | [1.633, 2.401] |
| SAM-Audio | OOD | 4 | 72 | -2.949 | [-3.311, -2.571] |
| SAM-Audio | OOD | 5 | 72 | -5.685 | [-5.974, -5.376] |

## Destructive control and interpretation

The query-swapped negative control scored an interferer while the metric still used the target reference: `n = 96`, mean SDR `-0.684 dB`, 95% bootstrap CI `[-0.735, -0.636]`. This collapses relative to the requested-target rows and checks that the producer is measuring target SDR rather than a constant or a proxy retention score.

The measured target SDR varies monotonically downward as source count increases in every model/domain block. In the toy analogue, the Hive labels improve target SDR over their corresponding Orig. labels at all 8 matched cells, and both Hive labels remain above SAM-Audio at all 8 matched cells. Those are decisive toy observations, not claims that the paper's checkpoints or benchmark data were rerun.

Source pin read before the run: arXiv `2601.22599v2`, PDF SHA-256 `ed630829d635d34dc993d78f478243247e9d84f264f6d9bc48f6d78b9d8d1705`; paper Tables 4–5. Toy output: `outputs/claim5_toy_results.json`.