From dd672ded3b3ba1d034739b92d62fb29f189ab616 Mon Sep 17 00:00:00 2001 From: Rootiest Date: Mon, 7 Sep 2026 14:59:05 -0400 Subject: [PATCH 01/10] fix(tests): isolate the vault suite from the live config and universal variables The suite ran under a plain `fish`, which loads the user's real ~/.config/fish and their universal variables -- this repo doubles as that config. A test manipulating a guard variable could erase a real universal variable out of the running shell. Override XDG_CONFIG_HOME/XDG_DATA_HOME and pass --no-config. HOME stays real on purpose: overriding it makes the suite's two hermeticity assertions vacuous. Reasoning recorded at the call site. Vault suite still 317/317 with byte-identical stderr. --- tests/run-tests.fish | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/tests/run-tests.fish b/tests/run-tests.fish index 59bc260..35336af 100755 --- a/tests/run-tests.fish +++ b/tests/run-tests.fish @@ -93,11 +93,43 @@ end # the suite builds its own throwaway git repos and binds the vault, claude # and agy roots to them, so it needs no loaded config and must never see # the real ~/.claude. +# +# HOME is deliberately NOT overridden here. Read this before "improving" it. +# +# Overriding XDG_CONFIG_HOME/XDG_DATA_HOME plus --no-config is what makes +# this run isolated: the universal-variable file fish can reach is a fresh +# empty one, and no config.fish/conf.d is loaded. Without that, an +# "isolated" suite runs against the user's LIVE config and real universal +# variables -- this repo doubles as a real ~/.config/fish -- so a test +# doing `set -e __fish_config_op_logging` would erase a real universal +# variable out of the running shell. Measured: +# $__fish_config_op_registry_keys has 65 entries under a plain `fish`, 0 +# under `fish --no-config`. +# +# `env -i HOME=$sandbox` was tried and REJECTED. It looks strictly more +# hermetic, but test-agents-vault.fish's hermeticity floor snapshots the +# real $HOME/.claude/memory and $HOME/.gemini/antigravity-cli and asserts +# them unchanged at the end. Point HOME at a sandbox and both snapshots +# read "absent" before and after: the assertions still pass while +# asserting nothing. A change that turns a real assertion into a tautology +# without turning anything red is the worst failure mode a test harness +# has. Keeping HOME real is what keeps those two assertions biting. +# +# Overriding XDG_DATA_HOME is a hermeticity gain on top of the isolation: +# _agents_vault_dir falls back to +# ${XDG_DATA_HOME:-$HOME/.local/share}/agent-vault, so a vault path that +# no test overrode lands in a temp dir instead of the user's real +# ~/.local/share/agent-vault. echo "" echo "== Vault helper tests ==" -fish $repo_root/tests/test-agents-vault.fish +set -l vault_xdg (mktemp -d) +env XDG_CONFIG_HOME=$vault_xdg/cfg XDG_DATA_HOME=$vault_xdg/data \ + fish --no-config $repo_root/tests/test-agents-vault.fish if test $status -ne 0 set overall_failed 1 end +# `command rm`, not bare `rm`: this driver runs under the config it tests, +# which shadows rm/cp/cat. See the Phase 2 note for the full reasoning. +command rm -rf $vault_xdg exit $overall_failed -- 2.54.0 From 8fc27fa9c276f9a43a54747873016cce7d410fe8 Mon Sep 17 00:00:00 2001 From: Rootiest Date: Mon, 7 Sep 2026 15:00:10 -0400 Subject: [PATCH 02/10] fix(tests): route the runner's utilities through command run-tests.fish executes under the config it tests, which shadows cp, rm and cat. The real hazard is cp: the config aliases it to 'cp -i', which on a non-empty destination reads EOF in a non-interactive runner, silently skips the copy and exits 0 -- a sandbox missing config files, reported as success. rm -rf and cat were measured and behave correctly as-is (the rm wrapper bails to command rm on any non-recursive flag, so -rf really deletes and does not trash). Prefixed anyway: a test runner must not depend on the configuration under test. --- tests/run-tests.fish | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/tests/run-tests.fish b/tests/run-tests.fish index 35336af..4ebb2d1 100755 --- a/tests/run-tests.fish +++ b/tests/run-tests.fish @@ -53,12 +53,22 @@ mkdir -p $sandbox_cfg # no-op on missing paths), so give it $HOME/.local/bin to find. mkdir -p $sandbox/home/.local/bin -cp $repo_root/config.fish $sandbox_cfg/ +# Every utility below goes through `command`. This driver runs under the +# very config it tests, which shadows these: `cp` is an alias for `cp -i`, +# `rm` is a trash wrapper, `cat` resolves to bat. Only `cp` is an actual +# hazard today -- `-i` on a non-empty destination reads EOF in a +# non-interactive runner and SILENTLY SKIPS the copy while exiting 0, +# which would leave the sandbox missing config files and report success. +# `rm -rf` and `cat` were measured and behave correctly as-is (the rm +# wrapper bails to `command rm` on any non-recursive flag, so -rf really +# deletes and does not trash). Prefixed anyway: a test runner must not +# depend on the configuration under test. +command cp $repo_root/config.fish $sandbox_cfg/ test -f $repo_root/fish_plugins -and cp $repo_root/fish_plugins $sandbox_cfg/ +and command cp $repo_root/fish_plugins $sandbox_cfg/ for d in functions conf.d completions integrations themes data test -d $repo_root/$d - and cp -r $repo_root/$d $sandbox_cfg/ + and command cp -r $repo_root/$d $sandbox_cfg/ end set -l err_file (mktemp) @@ -72,8 +82,8 @@ env -i \ 2>$err_file set -l functional_status $status -set -l stderr_out (cat $err_file) -rm -rf $sandbox $err_file +set -l stderr_out (command cat $err_file) +command rm -rf $sandbox $err_file if test -n "$stderr_out" # Diagnostic only, not a gate: on machines with vendor fish configs -- 2.54.0 From d3028d770389f314c00b7fece724a8980490a117 Mon Sep 17 00:00:00 2001 From: Rootiest Date: Mon, 7 Sep 2026 19:59:23 -0400 Subject: [PATCH 03/10] refactor(tests): extract the shared assertion core into tests/lib.fish check/section/report and the counters move out of the vault suite unchanged. report also writes its counts to $FISH_CONFIG_TEST_COUNTS so a driver can aggregate without parsing stdout, and ends on an explicit boolean per AGENTS.md item 5. All 317 vault assertions and every fixture helper are untouched. --- tests/lib.fish | 62 ++++++++++++++++++++++++++++++++++++ tests/test-agents-vault.fish | 21 ++---------- 2 files changed, 64 insertions(+), 19 deletions(-) create mode 100644 tests/lib.fish diff --git a/tests/lib.fish b/tests/lib.fish new file mode 100644 index 0000000..2edfc91 --- /dev/null +++ b/tests/lib.fish @@ -0,0 +1,62 @@ +#!/usr/bin/env fish +# Copyright (C) 2026 Rootiest +# SPDX-License-Identifier: AGPL-3.0-or-later +# +# Shared assertion and reporting core for tests/test-*.fish. +# +# One assertion: `check