The tests
Each of Phosphoric’s claims is backed by a test that would make it fall. Here is what runs, from a chip’s registers up to the browser, and how a skipped test is kept from going unnoticed.
On this page
The suite
make tests runs 70 suites, 1,362 checks, all green. They cover every component, from the
processor to the F1 menu:
| Area | Targets |
|---|---|
| Processor | test-cpu (including the 105 illegal opcodes), test-cycle (65x02 oracle), the 6502 functional test |
| Memory and I/O | test-memory, test-io (VIA timing vectors), test-via-lazy |
| Picture | test-video, test-renderer, test-raster-split, test-ula-ng, test-avi |
| Sound | test-audio (signal measurement), test-sp0256, test-mea8000, test-midi, test-smf |
| Storage | test-storage, test-jasmin, test-tape-signal, test-sedoric-tools, test-dsk2hfe |
| Clock and states | test-clock, test-savestate, test-savestate-determinism |
| Serial and network | test-serial, test-serial-backends, test-dtl2000, test-picowifi, test-acia6850, test-pia6821 |
| LOCI | test-loci (163 tests), test-loci-sdimg, test-loci-hw, test-loci-e2e (12 end-to-end scenarios) |
| Debugging and control | test-debugger, test-gdbstub, test-control, test-httpapi, test-http-parse |
| Menu and configuration | test-iomenu, test-iomenu-glue, test-iomenu-cli, test-cards, test-card-template |
| Browser | test-web-loci, test-web-iomenu, test-web-picowifi (headless Chrome) |
External oracles
A test written by the project can be wrong along with it. So Phosphoric also relies on references that do not come from it:
- SingleStepTests/65x02: 10,000 cases per opcode, with the expected bus trace cycle by cycle. Four properties scored separately, and both cores judged in the same run, so that neither can progress by breaking the other.
6502_functional_test: the 6502’s reference functional test, which passes in full.- The vectors are not in the repository (about 1 GB, third-party):
make fetch-vectorsdownloads them, and the CI has a separate step for them.
Measuring instead of comparing
Several tests measure a behaviour instead of comparing frozen bytes:
test-audiomeasures a voice’s frequency (within 0.1%), an envelope’s duration (within 2%), the balance of the noise;test-clockchecks that a clock call moves exactly one cycle forward, that peripherals receive one cycle at a time, and that the processor counter equals the scan position over a frame;test-raster-splitruns a real 6502 program and looks for lines cut in their middle.
Safety nets
| Net | What it prevents |
|---|---|
make test-bench | a frame costing more than 1,000 µs (5% of the 20 ms budget) on the reference machine |
make test-corpus | a picture that would change: local programs replayed at fixed cycles, compared with a manifest of screen fingerprints |
make test-cli-golden | a change of behaviour during refactoring: about 116 command lines replayed on the old and the new binary; exit code, outputs and produced files must be identical |
make test-docs-claims | an unqualified accuracy wording in the showcase documents |
make test-comment-diff | an English branch that would differ from the French one other than by comments |
make tests-strict | a test that would go silent: any skip not justified in tests/allowed_skips.txt fails |
make coverage-check | a line coverage of src/ that would drop below its floor |
Robustness
- Fuzzing (
make fuzz, libFuzzer and clang) of the file readers: configuration, floppies, HFE images, HTTP requests, save states, replay movies, MIDI files, symbols, tapes.make test-fuzz-replayreplays the seeds and the regressions found. - Sanitisers:
make SANITIZE=1 testsruns the whole suite under ASan and UBSan. - Leaks:
make valgrind; the project reports zero leaks. - Static analysis: compiler warnings, cppcheck, flawfinder.
The CI
Four workflows run on the English branch, whose code is identical to the French one (decision 0005):
| Workflow | What it does |
|---|---|
linux-ci | two builds, full suite in strict mode, oracle and functional test, performance budget, Valgrind; then the suite under ASan and UBSan, 30 s of fuzzing per target, and the coverage floor |
windows-build | an .exe cross-built on Linux, with SDL2.dll, and its tests |
macos-build | native build and full suite, on real Apple Silicon |
release | a version’s packages: Linux x86_64 and Windows x86_64 (without the ROMs) |
Replay movies
--record records a session’s keyboard inputs, frame by frame; --replay replays them. The only
non-deterministic input of the emulation is the keyboard matrix (8 bytes): by replaying it bit for bit,
a session reproduces exactly. Two replays give screenshots identical to the byte.
It is a regression tool: record once, replay with no display, compare a reference screenshot. The format is readable text, which can be compared line by line.