11. Checks#

zig build test # type-check every declaration + comptime unit tests zig build develop # test/setu_class.so (every interpreter) + setu_class.pyi /opt/python/3.15t/bin/python3 -m pytest # then 3.15, 3.14t, 3.14: the same .so tool/check.py # everything, offline, on all four interpreters python3 tool/bench.py --ref HEAD # per-call cost A/B against a commit zig build profile -Doptimize=fast -Dcpu=x86_64_v3 # instructions per call, per line (callgrind); # -Drelease with a preferred optimize mode zig build check-symbol -Dpython=/opt/python/3.14/bin/python3 # CPython symbols, per interpreter

A project using Setu runs its own tool/check.sh, which includes zig build setu-build-check, setu-doc-check and setu-guide-check (its copies of Setu's script, docs helpers and this guide match the Setu it pins), and zig build check-symbol -Dpython=<each interpreter> (addSymbolCheck, Setu's tool/check_symbol.py on the extension as built: every CPython symbol it imports is exported by that interpreter and, on 3.15+, in the stable ABI for free-threading (abi3t); needs nm, and cc on 3.15+). After a Setu bump: zig build setu-build-sync setu-doc-sync setu-guide-sync.

Line coverage: zig build coverage [-Dpython=...] [-- --missed --partial pytest args] (addCoverageSteps; kcov 43) prints a per-file table, with --missed the lines that never ran, with --partial the yellow ones, and writes the numbers to zig-out/coverage.json (totals, per interpreter, per file and line, each file's CRC-32; the format is in tool/check_coverage.py). Files are named relative to the project, a dependency's as <package>/<path> (setu/src/arg.zig), wherever it lives. kcov's own output (an HTML report, XML, JSON: some 70 files per interpreter) goes to a temporary directory, removed after the run; -- --html DIR keeps its HTML report. On the docs site (coverage = 'zig-out/coverage.json') the Coverage page has a section per owner (the project, then each dependency), and a click on a file shows its source, Zig highlighted, the missed lines red; a source changed since the run (its CRC-32) is flagged changed. .badge = "doc/coverage-badge.json" also writes the total as a shields.io endpoint file; committed, it is a README badge: the image https://img.shields.io/endpoint?url=<URL>, <URL> the file's raw.githubusercontent.com address, percent-encoded, linked to the docs site's Coverage page (<site>/coverage.html) (run with every --python first: the badge shows the merged total). .history = "doc/coverage-history.json" also keeps the totals per project version (build.zig.zon's .version; a run sets its version's entry, one line each), with the project's own files' (no dependency's) beside them; committed, with coverage_history = 'doc/coverage-history.json' in [tool.setu_doc] the Coverage page charts the own lines (the donut's figure) next to the project's donut, each point naming the total with dependencies too. In CI it needs kcov, packaged for Ubuntu 25.04+: on GitHub's ubuntu-26.04 runner the apt package kcov, then the step with every --python; Setu's .github/workflows/ci.yml also fails when the committed badge or history differs from the run's (git diff --exit-code doc/coverage-badge.json doc/coverage-history.json). Notes:

• It measures the extension as built, in Debug. The default Debug backend (self-hosted x86_64) pads its line tables with ops kcov (elfutils) misreads, so kcov runs a copy with the padding rewritten (the build is not changed). That backend gives each line its own code, so every red line is code that never ran; an LLVM build (.use_llvm = true, aarch64) folds a line like orelse return ... into the one above (its error path shows as run) and marks some comment and fn lines red (a jump nothing takes).
• kcov records nothing from a shared library loaded once about 24 are mapped, and import pytest maps more, so tool/check_coverage.py loads the extension first, as .module_name; a test importing another copy fails.
• Zig compiles only referenced code: a function nothing calls is not counted as missed.
• kcov's colours (--html; the runner's terms too): green ran, red never ran, yellow ran at some of its addresses only (counted as covered). In Zig, yellow is mostly a generic line (fromPython(T), a trampoline) whose instance for some type never ran: hits of possible addresses, e.g. 48 of 56 int conversions.
• Code the tests are not meant to run (benchmark helpers, out-of-memory branches) goes between // coverage-off and // coverage-on comments and is not counted, so every red line is a gap.
• coverage needs pytest in the interpreter it runs (and in each --python), else it stops with that error.
• The tests' child processes (a test running python3 example.py) are not measured: they import the normal build, and kcov parsing its self-hosted-backend DWARF grew to 25 GB in one of Liburing's runs, so the runner keeps kcov's preload out of them. A run is stopped once kcov and the tests use more than -- --max-memory GB (default 8).
• -- --python /opt/python/3.14/bin/python3 (repeatable) runs the tests under that interpreter too; the table merges the runs (a line ran if it ran in any), so code only one interpreter reaches (3.14's PyInit_ entry) counts. --html DIR then keeps one report per interpreter, DIR/<3.15t, 3.14, ...>/index.html; there is no merged HTML report (kcov --merge counts the coverage-off lines again).

Zig unit tests (code that does not call libpython) get the same through addZigTestCoverage: kcov over a b.addTest executable as built, the same table, options after -- (--missed, --html DIR, the rest to the executable) and JSON format, in zig-out/coverage-zig.json:

const tests = b.addTest(.{ .root_module = core });
@import("setu").addZigTestCoverage(b, setu_dep, tests, .{ // zig build coverage-zig
    .include = &.{b.path("src/core")}, // the sources to report
    .step_name = "coverage-zig", // the default; the result is zig-out/<step_name>.json
});

Instructions per call: zig build profile -Doptimize=fast -Dcpu=x86_64_v3 [-- -k TEXT --lines K --python PATH --baseline FILE] (addProfileSteps; valgrind 3.25) runs each case of .cases under callgrind, N and 2N times (the difference over N is one call, Python's startup and the loop taken out), and prints a table per interpreter: total (everything one call costs) and ext (the part in the extension's .so: its code, Setu's and std's, inlined there); --lines K lists each case's K costliest lines with their source. The numbers go to zig-out/profile.json (per case, file and line; the format is in tool/check_profile.py). Setu's own cases include bench_nop / bench_recv written by hand against the C API (test/hand.zig, hand: ...): generated code costs 4-26 fewer instructions per call than those on every interpreter (2026-10-28).

zig build profile -Doptimize=fast -Dcpu=x86_64_v3 -- --lines 8 -k recv cp zig-out/profile.json /tmp/before.json # then change the code zig build profile -Doptimize=fast -Dcpu=x86_64_v3 -- --baseline /tmp/before.json # Δ per case

• ext is the same on every run, total within about 0.3 per call: --baseline fails when either grew by more than --tolerance (default 2).
• The build must be optimized, with debug info, for a CPU valgrind runs (no AVX-512): the step fails otherwise.
• .cases is a Python file with CASES, a list of (label, statement), and namespace(module), which returns the names the statements use; Setu's tool/bench.py is one, so bench.py times the cases profile counts.
• A line 0 is code the compiler added (no source line).
• profile = 'zig-out/profile.json' in [tool.setu_doc] adds a Profile page to the docs site: a row per case with a bar per interpreter (the extension's part dark); a click on a case shows its files' sources, each costly line with its instructions per call, shaded by them.

Memory errors and leaks: valgrind memcheck over the project's pytest, by hand (no build step):

zig build develop -Dcpu=x86_64_v3 # valgrind can not run AVX-512 (SIGILL, exit 132) PYTHONMALLOC=malloc valgrind --leak-check=full --errors-for-leak-kinds=definite --error-exitcode=99 \ python3 -m pytest -q zig build develop # the native build back

• Clean is "0 errors" and "definitely lost: 0 bytes". "Possibly lost" 56-byte blocks from module init (PyType_FromMetaclass, one per class and exception) are the type objects kept for the process: valgrind sees them only through a pointer past CPython's GC header.
• The default Debug backend's line table is partly unreadable to valgrind ("Badly formed extended line op"): build through LLVM (.use_llvm = true on the extension's library) for full source lines in its stacks.