The fixture corpusstable

Running the corpus

Against a single fixture, with the validator CLI. No install is needed — from the repository root:

text

PYTHONPATH=packages/ver-validator/src .venv/bin/python -m ver_validator.cli \
    conformance/valid/l3-full.json

pip install -e packages/ver-validator provides the ver-validate entry point on PATH in an environment whose virtualenv has pip (this repository's .venv does not). The two invocations are the same program, and the examples below use the short spelling:

text

ver-validate conformance/valid/l3-full.json
ver-validate --format json conformance/invalid/ver501-undeclared-space.json

Exit status is 0 when every named record is conformant, 1 when any carries an error-severity issue, 2 for a bad invocation. Warnings never change the exit status. Every report also carries a status of conformant, nonconformant or indeterminate; no fixture in this corpus can produce the third, which is why VER001 is exempt from needing one.

Three of the 1.0.1 fixtures need a flag to reach the rule they test; the manifest entry names it in each case. Without the flag all three are conformant — the size and registry fixtures emit nothing at all, and the signature fixture reports VER1403 rather than a failed verification:

text

ver-validate --max-bytes 1024 conformance/invalid/ver103-record-exceeds-hard-cap.json
ver-validate --spaces conformance/registries/conflicting-spaces.json \
             conformance/invalid/ver402-space-registry-conflict.json
ver-validate --signature-key conformance/keys/corpus-signing-key.pub.pem \
             conformance/invalid/ver1402-signature-verification-failed.json

Every 1.1-draft fixture needs --schema 1.1-draft, and four of them need a flag beyond it; the 1.1-draft fixtures section lists those invocations.

Across the whole corpus, from the repository root:

text

pytest tests/test_conformance_fixtures.py -q

That suite walks manifest.json and asserts the contract below. To run the same checks without pytest — for instance against a validator that is not this one:

python

import json
import pathlib
from ver_validator import validate_file
from ver_validator.validator import load_record, load_registry, load_registry_snapshot
root = pathlib.Path("conformance")
manifest = json.loads((root / "manifest.json").read_text())
def run(entry, override):
    spec = {**entry, **override}
    kwargs = {"schema": spec.get("schema", manifest["default_schema"])}
    for key in ("level", "max_bytes", "profile"):
        if spec.get(key):
            kwargs[key] = spec[key]
    if spec.get("spaces_registry"):
        kwargs["spaces_registry"] = load_registry(root / spec["spaces_registry"])
    if spec.get("signature_key"):
        kwargs["signature_key"] = (root / spec["signature_key"]).read_bytes()
    if spec.get("registry"):
        kwargs["registry_snapshot"] = load_registry_snapshot(root / spec["registry"])
    if spec.get("lineage_peers"):
        kwargs["lineage_peers"] = [load_record(root / p) for p in spec["lineage_peers"]]
    return validate_file(root / entry["file"], **kwargs)
for entry in manifest["valid"]:
    for override in [{}, *entry.get("also", [])]:
        report = run(entry, override)
        allowed = set(override.get("expect_codes", entry["allowed_warnings"]))
        assert report.ok and set(report.codes) <= allowed, (entry["file"], report.codes)
for entry in manifest["invalid"]:
    for override in [{}, *entry.get("also", [])]:
        report = run(entry, override)
        spec = {**entry, **override}
        assert set(report.codes) == set(spec["expect_codes"]), (entry["file"], report.codes)
        assert report.ok == (spec["expect_severity"] != "error"), entry["file"]

invalid/ver101-unparseable-json.json is deliberately not parseable JSON. Any sweep that parses every *.json in the tree must skip it; the manifest flags it in notes.