Selected Internal EvalsPinned Local ResultsDomain Shift Under Test
Empirical Benchmarks & Error Rate Analysis
These are selected internal evaluation results across several audio sources and channel conditions. The 29-file published sample registry gives audio hashes, locally assigned labels, and source evidence tiers. Source attributions are not all independently verified, and the selected 168-trial sweep is not a representative blind benchmark.
Selected Internal Evaluation Results
Documented Evaluation
The local figures below describe selected internal suites. They do not estimate performance on complete external releases. Accuracy uses threshold 0.50; EER is interpolated from the recorded score sweep and may use another threshold.
Selected suite
Trials
Measured result at score threshold 0.50
Evidence and limit
Core regression speech
26
26/26 correct; 0.0% EER on these selected scores
Core ledger. Two non-speech files have expected inconclusive outcomes; the mixed file has no valid whole-file binary label.
Diverse multi-language regression
22
19/22 correct (86.36%)
Local gate; per-trial score ledger is not published. This is a selected suite.
Accuracy uses the fixed 0.50 decision threshold. EER is interpolated from observed scores and can use a different threshold; it is not the deployed error rate at 0.50. Source releases are not pinned to immutable revisions. The site sample registry marks which published WAV source labels are documented recipes, site claims, or unverified.
Benchmark provenance and known failures
The 168-trial ledger contains the selected partition name, processed-audio SHA-256 hash, locally assigned label, score, and decision for each trial. The core ledger contains 29 published-file hashes and 26 binary speech decisions. The release record identifies the public DSP source and ledgers by SHA-256, so the published 29-file run can be checked byte for byte. Its source-tree digest is not an immutable Git commit for this working-tree run, and the external corpus archives are not pinned; the 168-trial sweep is not independently reproducible from these materials alone.
The selected 40-file In-the-Wild partition had 37.50% accuracy, 60.00% observed-score EER, and 85.0% human false rejection at threshold 0.50. This failure limits any claim of reliable detection on arbitrary media. The selected partitions and source labels are described in the sample registry and trial ledgers.
Separate ODSS research diagnostic
A previously frozen JavaScript DSP reference core; these historical results have not been rerun with the current detector source evaluated 32 test recordings from 4 speakers at unchanged threshold 0.50: 10/16 human false flags and 4/16 synthetic misses, with balanced correct fraction 56.25%. This exploratory developer-run two-class diagnostic is separate from the selected sample suites above. Its descriptive uncertainty is fragile with 4 speaker clusters; no EER, deployment readiness, authentication, streaming, radio or general-population performance is established.
Completed ASVspoof 5 diagnostic. 6 October 2026 Pacific ASVspoof 5 research diagnostic: the historical frozen DSP snapshot (ca6a48f4f408) scored all 192 registered recordings at unchanged threshold 0.50. On the 96-record evaluation subset it falsely flagged 22/48 human recordings and missed 8/48 synthetic recordings; 66/96 were correct (68.75%), with 0 processing errors and 0 abstentions. This developer-selected, speaker-disjoint two-TTS-family diagnostic has no independent blinded custodian. Source/channel/training exposure and separate synthetic-content reuse scope remain unresolved. Scores are uncalibrated; these historical results do not validate the current deployed detector, authentication, streaming, radio use or general-population performance. Read methods and retained errors; download the aggregate.
Internal paired SSL baseline diagnostic
On the same 168 previously exposed, locally labeled and nonrandomly selected trial files, the frozen Worker rule classified 123/168 correctly (73.21%), while the author-released SSL model's native two-logit argmax rule classified 140/168 correctly (83.33%). In the selected 40-file In-The-Wild subset, the respective counts were 15/40 and 31/40. Both systems scored all 168 files with 0 processing errors and 0 abstentions. This paired JavaScript Worker run is separate from the Python sweep above; the equal overall correct count does not mean identical trial scores or errors. Observed-score EERs on these same rows were 35.41% for Worker, 12.31% for the author class-1 score, and 11.65% for the separately derived SSL spoof margin. EER uses a varying decision boundary; none of these scores is a calibrated probability or official challenge result.
The Worker analyzed each full WAV; the SSL model used the first or repeated 64,600 samples after 16 kHz mono resampling. Labels were known beforehand, sources are related, selection was nonrandom, and training exposure and upstream label custody are unresolved. This internal diagnostic does not establish independent generalization or model superiority. Historical source scope: this paired diagnostic ran frozen DSP source fdfc11cd0cb2 (/reproduce/dsp-fdfc11cd0cb2456b364b4ae770549b0a85dcbd27c3bbbd91dbf0c732a8c16f8e.txt); this published ledger was not generated by the current source (afd2fa0e7600). Download the sanitized per-trial scores, fixed rules, pins and subset aggregates.
On 18,000 developer-selected, labeled trials from the ASVspoof 5 evaluation partition (2,000 human, 16,000 synthetic, attacks A17 to A32), every fixed rule was scored on every row: Worker 59.86% balanced and 81.18% raw, rejecting 1351/2000 human trials and missing 2036/16000 synthetic trials; learned feature model 55.52% balanced and 54.37% raw, rejecting 860/2000 human trials and missing 7353/16000 synthetic trials; author-released SSL model 59.07% balanced and 90.89% raw, rejecting 1637/2000 human trials and missing 2/16000 synthetic trials. The Worker abstained on 36 synthetic trials, which count as errors, and two retained human trials had prior developer exposure. Balanced accuracy is the primary figure because raw accuracy is inflated by the 8:1 synthetic share; all three balanced figures are near 60% or lower. Historical source scope: this comparison ran frozen DSP source fdfc11cd0cb2 (/reproduce/dsp-fdfc11cd0cb2456b364b4ae770549b0a85dcbd27c3bbbd91dbf0c732a8c16f8e.txt); this published ledger was not generated by the current source (afd2fa0e7600).
System
Balanced accuracy
Raw accuracy
Human trials rejected
Synthetic trials missed
Abstentions
Observed-score EER (determinate rows)
Worker
59.86%
14613/18000 (81.18%)
1351/2000 (67.55%)
2036/16000 (12.72%)
36
39.44% (17964/18000 rows)
Learned feature model
55.52%
9787/18000 (54.37%)
860/2000 (43.00%)
7353/16000 (45.96%)
0
44.40% (18000/18000 rows)
Author-released SSL model
59.07%
16361/18000 (90.89%)
1637/2000 (81.85%)
2/16000 (0.01%)
0
20.10% (18000/18000 rows)
Fixed rules: the Worker flags synthetic at decision score 0.50 or higher; the learned model at an uncalibrated margin of -0.7614 or higher, chosen on development rows; the SSL model when its spoof logit is at least its bona fide logit. Rejected and missed counts include abstentions. No score is a calibrated probability.
Equal error rate (EER) uses only determinate rows and a boundary that varies with the data, so it is not an operating result. The denominators are 17964 of 18,000 rows for the Worker (its 36 abstentions have no score and are left out of the EER only) and 18000 for the learned model and for the SSL model. The SSL model has two EER definitions on the same 18000 rows. The spoof-margin EER (20.10%) ranks trials by spoof logit minus bona fide logit. The observed author bona fide logit EER (20.55%) ranks them by the raw author bona fide logit, negated so higher means more synthetic. The second is our own computation from the author-released model's raw output; it is not a figure the model's authors published.
Adaptation and input differ: the learned model was fit on a study training subset, the Worker and learned model read the full WAV, and the SSL model read only the first 4.04 seconds. This is an internal, non-blind comparison with known labels. It is not independent, not a full or official ASVspoof 5 challenge result, and it establishes no superiority. The full audio is not bundled. Source: ASVspoof 5 database, DOI 10.5281/zenodo.14498691, Open Data Commons Attribution License. The roster was frozen before evaluation audio was acquired, but the developer selected it and knew the labels.
The ASVspoof 2021 evaluation paper reports equal error rates for official challenge baselines on its own evaluation partitions (Table 8). Those results use a different corpus and protocol from our selected 168 trials. Noise or Voice has not been scored on the official ASVspoof 2021 evaluation partition, so no same-set ranking is available.
The In-the-Wild dataset paper emphasizes generalization testing beyond laboratory corpora. Our 40-file convenience subset shows a large error rate, but does not estimate performance on the full release.
Curated local partition containing 26 binary-labeled speech clips, two non-speech cases, and one mixed recording. The 26 speech clips have 100.0% accuracy and 0.0% subset EER; this suite is an internal regression gate.
Diverse Multi-Language Suite (N=22)
International evaluation suite spanning 8 languages/accents (French, German, Spanish, British, Australian, Indian, Japanese, US) and lossy 8 kHz G.711 telephony audio. Achieves 86.36% accuracy with 0.0% false rejection on genuine human speech.
Cross-Dataset Evaluation Partition (N=168)
Selected subset spanning WaveFake vocoders (64), In-the-Wild media (40), LibriSpeech clean (20), lossy codec tracks (18), and Core samples (26). Achieves 73.21% overall accuracy and 0.7945 F1 score on this subset.
Selected WaveFake and generator samples
The 64 WaveFake trials are an ordered convenience sample: the first eight rows per available class in a third-party Parquet copy. Accuracy is 79.69%; eight selected LJSpeech human clips have 87.5% false rejection at threshold 0.50. This result is weak evidence for generalization to other speakers or vocoders. See the trial ledger and the dataset authors' repository.
Generator names attached to local core files have different evidence tiers. One StyleTTS2-named WAV follows a Kokoro ONNX recipe, while the F5-TTS-named WAV has a generation recipe without a pinned execution record. The sample registry records those limits.
Selected codec transformations
The 18 codec trials reuse six core source clips. At the fixed 0.50 threshold, 16/18 were classified correctly: six of six at Opus 16 kbps, six of six at Opus 32 kbps, and four of six at G.711 μ-law. All three human G.711 clips were classified human; two synthetic G.711 clips were missed. The observed-score EER for this selected partition is 11.11%.
These are codec derivatives of known clips, not independent telephony recordings. They do not establish deployed VoIP latency or reliability across networks. Exact trial rows and hashes are in the evaluation ledger.
The table below reports the Python detector on 168 selected speech clips spanning core verification samples, neural vocoder clones (WaveFake), lossy codec tracks (PSTN G.711u, Opus 16k/32k), LibriSpeech reading speech, and In-The-Wild recordings. This is an internal diagnostic subset, not an official full-partition benchmark. The pilot API uses a separate JavaScript DSP implementation; its results on the same clips appear immediately below.
*Wilson ranges are calculated as if the selected trials were independent Bernoulli observations (z = 1.95996). They are descriptive arithmetic references, not valid confidence intervals for performance on a wider population: the selection is not random and many trials share source utterances, speakers, or codec-derived audio. Statistics Canada explains why clustering changes sampling variance. EER is interpolated from observed scores and is undefined for the human-only LibriSpeech partition (TN=15, FP=5 at τ=0.50).
Dependence in this selected corpus: 64 WaveFake trials reuse 8 underlying LJSpeech utterance IDs; 18 codec trials derive from 6 source clips also present in the core corpus; 40 In-The-Wild trials cover 10 speakers with 4 clips each. The 20 LibriSpeech clips come from 20 listed speakers. These are trial counts, not counts of independent people or utterances.
Python and API JavaScript Engine Comparison
We ran the Cloudflare API's JavaScript DSP code locally on the same 168 manifest-pinned WAV files. The engines disagree on four decisions at τ = 0.50. Identical aggregate accuracy does not mean they make the same errors. These JavaScript results are a local code-path measurement, not a production latency or independent generalization study.
Selected partition
Trials
Python accuracy
API JS accuracy
Python EER
API JS EER
Core corpus
26
100.00%
100.00%
0.00%
0.00%
Codec degraded
18
88.89%
100.00%
11.11%
0.00%
WaveFake
64
79.69%
78.12%
43.27%
46.09%
LibriSpeech human only
20
75.00%
70.00%
N/A
N/A
In-The-Wild
40
37.50%
37.50%
60.00%
65.00%
All selected speech
168
73.21%
73.21%
33.98%
35.41%
Scores and exact input hashes: Python ledger and API JavaScript ledger. Both use score threshold 0.50 and observed-score EER. The selected corpus audio is not included in a fresh public Git checkout; upstream acquisition details remain incomplete.
The 18 codec derivatives had 16/18 correct decisions, with two synthetic G.711 clips missed. The 64-file WaveFake convenience sample had 79.69% accuracy but rejected seven of eight selected human clips. The 40-file In-the-Wild subset had 37.50% accuracy and 85.0% human false rejection. These results do not isolate the cause of each error or demonstrate a production remedy.
Subset Purpose:Selected diagnostic regression corpus with source and codec variation; no adaptive-attack trials.
Selection Protocol:Ordered first rows per WaveFake class and first two recordings per shared In-The-Wild speaker; these are selected convenience samples.
Manifest & Hashes:Complete trial list and per-trial SHA-256 hashes published in evaluation-sweeps.json.
Comparability Scope:Internal diagnostic results; no same-set official challenge evaluation has been performed.
The 168-trial corpus was assembled from selected source recordings, resampled to 16 kHz, peak-normalized, and capped at six seconds. Selection was deterministic but was not random or representative of the full datasets:
WaveFake Multi-Vocoder Partition (N=64): First 8 rows per class in the source Parquet: 7 neural vocoders (HiFi-GAN, MelGAN, Multi-band MelGAN, Parallel WaveGAN, WaveGlow, WaveGrad, DiffWave) and one LJSpeech human class.
In-The-Wild Deepfake Partition (N=40): First two authentic and synthetic recordings for each shared speaker in sorted speaker order, capped at 20 per label. This is a convenience subset of the Müller et al. collection.
Core Golden Suite (N=26 in sweep, N=29 in gate): Locally labeled generator and human recordings. Model names vary in evidence quality; see the per-file provenance registry.
Codec-Degraded Partition (N=18): Derived files labeled G.711 μ-law or Opus at two bitrates from six source clips. Encoder versions and commands were not preserved.
LibriSpeech Clean Human Baseline (N=20): First recording for each of the first 20 speakers encountered in the LibriSpeech clean validation split.
The evaluation ledger publishes filenames, source dataset tags, labels, processed-audio SHA-256 digests, scores, and extracted features. Source releases are referenced in the builder but not pinned to immutable revisions, so this is not yet a complete independent reproducer. Byte identity: only the 29 published-sample measures (the Worker results and the Python proof ledger) are computed on the exact public WAV bytes. The 168-trial Python, JavaScript and SSL ledgers are bound to the separately stored evaluation-corpus bytes by manifest SHA-256 pins. Four public files, synthetic_studio_brian.wav, synthetic_studio_eric.wav, synthetic_ad_miguel_sovereign.wav and synthetic_abundera_narrator.wav, have different file hashes from their evaluation-corpus counterparts. Measured: identical decoded audio (same PCM hash, mono, 16 kHz, same length); the public copies carry one extra 34-byte metadata chunk, and Python scores were identical on both versions. The other 25 public files match their corpus hashes. The sample registry records this per file as pcm_identical_container_differs.
The detector extracts statistical features from decoded audio, including spectral phase and estimated LPC residual measures. Exploratory feature ranks on this selected corpus are in docs/PHYSICAL_ACOUSTIC_SIGNATURES.json. They do not establish invariant human or synthetic ranges, direct tissue motion, or a generator's mechanism. Recording chains and repeated sources may affect the observed separation.
Continuous Regression Test Gate & Multi-Language Diverse Suite
Selected Regression Gate
The local regression gate checks selected core and diverse samples when it is run (scripts/regression_gate.py; tests/test_regression_gate.py). Passing this gate does not imply that every commit was tested or that performance will hold on unseen speech.
Core Golden Gate: Checks 26 labeled speech clips and two non-speech inconclusive outcomes; the mixed recording has no valid whole-file binary match.
Multi-Suite Regression Floor: Compares combined suite accuracy and false acceptance against tests/regression_baseline.json; the combined count includes repeated samples across suites.
Diverse Multi-Language Suite: Evaluates 22 diverse samples in tests/samples/diverse/ covering 8 languages/accents (Spanish, German, French, US English, British English, Australian English, Indian English, Japanese) and 8 kHz G.711 telephony audio.
# Run the automated continuous regression test gate
python3 scripts/regression_gate.py --all
# Run statistical physical acoustic discovery
python3 scripts/discover-acoustic-signatures.py
Adversarial testing status
No published local trial ledger evaluates FGSM, PGD, Carlini-Wagner optimization, or differentiable vocal-tract attacks against this detector. Bypass rates, codec survival, attack cost, and resistance to an adaptive attacker are unmeasured. The 168-trial ledger tests selected source and codec conditions, not those attacks.
Acoustic feature interpretation
The detector computes spectral phase, LPC residual, jitter, and higher-order spectral statistics from decoded recordings. These features can distinguish selected samples in this internal corpus, but cannot directly reveal vocal-fold motion, codec boundaries, or generator identity. Recording conditions and source selection can affect the distributions.
The trial ledger gives the measured decisions; the sample registry gives hashes and source-evidence tiers for every published WAV.
05: Reference Validation Corpus Lab: Real Speech vs Neural Vocoders
Published Local Samples
Listen directly to side-by-side audio sample pairs from our open-source evaluation corpus, compare the extracted physical DSP metrics, and inspect the Ed25519 cryptographic verification ledger:
Authentic Human Speech (Male)
HUMAN-LABELED · decision score 0.0100
Local human label; recipe fetched Narsil/asr_dummy/1.flac (16 kHz, 3.50s)
Worker decision score: 0.0100 (threshold 0.50); STFT phase dispersion: 0.5353; LPC residual skew: +0.7552; pitch period variation: 2.0026%; bispectral coupling: 0.3547. These measurements do not establish the source or vocal physiology.
With Node.js 18 or newer, download the standalone verifier and run it against this site. It downloads all 29 registered WAVs, the release record and its hash-pinned DSP source, and the Worker results. It checks WAV and source SHA-256 digests, then recomputes every verdict, score, and feature. The source labels remain local assertions.
From a source checkout with the required audio files and Python/Node dependencies, the following commands rerun additional local checks. The 168-trial source corpus is absent from a fresh public checkout, and its upstream releases are not pinned:
# 1. Run full Python test suite (39 tests)
python3 -m pytest
# 2. Run Python benchmark engine across 29 files (26 binary speech labels, 2 non-speech cases, and 1 mixed file)
python3 -m engine.benchmark
# 3. Verify single audio sample via CLI with Ed25519 attestation
python3 -m engine.cli verify site/audio/samples/human_real_biden_speech.wav
# 4. Run the JavaScript DSP check on the published WAVs
node scripts/test-js-dsp.mjs
# 5. Run full Playwright E2E UI & verification test suite
node scripts/test-e2e.mjs