test: unify the test harness, discover suites, fix a real hermeticity leak #134

Merged
rootiest merged 10 commits from test/test-harness-overhaul into main 2026-09-08 05:26:30 +00:00
10 Commits
Author SHA1 Message Date
rootiest 1f847bfce0 test(guards): cover always/on, always/off and AND via a synthetic registry
14 cases. The production registry has 65 entries all carrying exactly one
tag and no always/* anywhere, so these three documented semantics are
unreachable from real data -- the synthetic keys/values table is the only
way to reach them. Reasoning recorded at the fixture.

Also pins the degenerate no-slash tag and reaches C5 through
__fish_config_op_enabled, which is the path production code takes.
2026-09-07 20:12:22 -04:00
rootiest 7a6a7339dc test(guards): cover op_enabled against the real registry
5 cases: default-on, .fish suffix stripping, category off, subcategory
overriding it, and the documented fail-open on a missing entry.
2026-09-07 20:11:06 -04:00
rootiest 2efbcfc53f test(guards): cover registry lookup, including identity:site pairing
5 cases. The wrong-site case pins that the key is the identity:site pair
rather than the identity alone.
2026-09-07 20:10:15 -04:00
rootiest 639d139799 test(guards): pin C5 logging as opt-in, including that the master cannot enable it
8 cases on the real __fish_config_op_logging name. AGENTS.md records this
as a deliberate special case agents keep trying to 'fix' out; three of
these cases exist specifically to turn that into a test failure.

Includes the three subcategory cases proving the opt-in is inherited
through chain[-1] rather than special-cased per subcategory.
2026-09-07 20:09:26 -04:00
rootiest 2b1e99ab47 test(guards): cover the cascade's subcategory -> category -> master order
13 cases on invented variable names, so nothing ambient can perturb them:
default-on, explicit truthy/falsy, unrecognized-defers, subcategory
overriding the category in both directions, and the four cases that
together pin the master as an off switch only.
2026-09-07 20:07:37 -04:00
rootiest a8d564803b test(guards): cover __fish_variable_check's four exit codes
21 cases: a precondition asserting the fork's registry is loaded, then
truthy/falsy/unset/empty/unrecognized/no-argument/list.

Runs isolated, which is what lets these cases manipulate real guard
variable names without being able to reach the user's universal variables.
2026-09-07 20:06:42 -04:00
rootiest 0a9cfebcad refactor(tests): discover suites instead of naming them
run-tests.fish globs tests/test-*.fish and runs each in the mode the suite
declares in its own header. Isolated is the default and no typo can
promote a suite to in-session: detection is case-insensitive so a
near-miss is caught, the value comparison is exact.

In-session suites share one sandboxed session; isolated suites each get a
--no-config child with temp XDG dirs. Counts come back through a file so
output keeps streaming.

functional.fish becomes test-session.fish, its 15 predicates rewritten as
check calls. 332 assertions, unchanged per-phase.
2026-09-07 20:02:59 -04:00
rootiest d3028d7703 refactor(tests): extract the shared assertion core into tests/lib.fish
check/section/report and the counters move out of the vault suite
unchanged. report also writes its counts to $FISH_CONFIG_TEST_COUNTS so a
driver can aggregate without parsing stdout, and ends on an explicit
boolean per AGENTS.md item 5.

All 317 vault assertions and every fixture helper are untouched.
2026-09-07 19:59:23 -04:00
rootiest 8fc27fa9c2 fix(tests): route the runner's utilities through command
run-tests.fish executes under the config it tests, which shadows cp, rm
and cat. The real hazard is cp: the config aliases it to 'cp -i', which on
a non-empty destination reads EOF in a non-interactive runner, silently
skips the copy and exits 0 -- a sandbox missing config files, reported as
success.

rm -rf and cat were measured and behave correctly as-is (the rm wrapper
bails to command rm on any non-recursive flag, so -rf really deletes and
does not trash). Prefixed anyway: a test runner must not depend on the
configuration under test.
2026-09-07 15:00:10 -04:00
rootiest dd672ded3b fix(tests): isolate the vault suite from the live config and universal variables
The suite ran under a plain `fish`, which loads the user's real
~/.config/fish and their universal variables -- this repo doubles as that
config. A test manipulating a guard variable could erase a real universal
variable out of the running shell.

Override XDG_CONFIG_HOME/XDG_DATA_HOME and pass --no-config. HOME stays
real on purpose: overriding it makes the suite's two hermeticity
assertions vacuous. Reasoning recorded at the call site.

Vault suite still 317/317 with byte-identical stderr.
2026-09-07 14:59:05 -04:00