Skip to content
VERASPEC
Repository
Reference setsstable

What this does and does not claim

Claimed on every commit, by tests/test_record_refset_manifests.py in a tier with no torch and no network:

  • Three manifests whose every content_sha256 and pixel_sha256 recomputes from the committed image bytes.
  • Three set_manifest_sha256 values that recompute from the manifests and are found in this file.
  • The three space_id values, against verd.space_cache.SPACE_IDS, so a space the pipeline can load with no set published for it fails the suite.

Claimed by --check, which regenerates: ten committed images whose bytes are exactly what the closed-form generator draws.

Claimed by measurement, and re-checkable on demand: thirty vector_sha256 values, each the digest of a vector that was actually produced by the named checkpoint. Not one is illustrative — errata E26 is the item that catalogues shipping placeholder digests as though they were real, and a reference set of invented vectors would repeat that defect at the one place §8.2 makes load-bearing. CI cannot afford ~1.5 GB of weights, so it re-checks none of them; --check --vectors re-checks all thirty, and the test suite re-checks three (one per space) wherever torch is installed.

Not claimed. These manifests make set_manifest_sha256 satisfiable. They do not by themselves make a conformance claim, because a conformance claim is made by a space descriptor, not by a manifest:

  • §8.2's tolerances — tolerance_cosine, tolerance_cosine_p5, tolerance_cosine_worst, tolerance_hamming_max — are **declared by a spaceDescriptor.reference block**. Nothing in this directory declares one, and nothing in this directory can: a tolerance is a producer's assertion about its own reproducibility, and the manifest is the evidence that assertion is measured against.
  • This project's pipeline does not publish reference-bearing descriptors. verd.models.SpaceDescriptor.to_json() has no reference key at all, so every Record verd currently emits warns VER404 (space-reference-missing), and errata E13 rates such a descriptor as making no conformance claim. Wiring reference into the descriptor — set_uri, the set_manifest_sha256 values above, and tolerances measured by re-embedding this set across the runs a producer wants to claim reproducibility over — is follow-up work that has not been done, not a property this directory confers.
  • No tolerance number appears anywhere in this directory, deliberately. The cosine figures above are *sensitivity measurements of one implementation against itself* over thirty vectors. A tolerance_cosine is a floor a producer is prepared to be held to across checkpoint reloads, library upgrades and hardware — an empirical corpus RELEASE.md lists as its own outstanding ratification prerequisite. Publishing 0.9999999 as a tolerance on the strength of this run would be exactly the "three round figures with a coherence argument and no data" the draft criticises.