Repository navigation
fork-kit: pc-agents setup, bootstrap and owner steps (part 2 of 2) - #234
scuttletoby wants to merge 19 commits into
Conversation
A WSL2 distro on a Windows PC as the wall around T3's agents: the files the kit owns (wsl.conf, resolv.conf, managed policy, uid-matched nft table, a root system unit with User=agent and a sandbox, a daily check timer), rendered from the pc-agents section of the machine file, and `pc-agents check [--enforce]`: the fleet's wsl2 walls (run, not copied) plus the kit's own. The unit runs the check before every start. A server test pins that the managed policy locks the fork's settings (agentSlots, linkedEnvironmentsEnabled, automaticUpdates). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`pc-agents setup` makes a fresh WSL2 distro into the wall: refuses a distro that is not fresh with nothing changed, adds the agent user, installs the kit's release under /opt/pc-agents/<sha>, writes and records every file atomically, masks interop/cron/at/ssh, installs the configured T3 version for the agent, loads the firewall, and enables T3 only after every wall holds. A second run changes nothing; a run stopped at any write finishes on the next. The bootstrap fetches a ci-release commit; the README has the owner steps, the Tailscale tag and ACL, and the state answers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Two lenses at 9d46a34: security lens234s NOT SAFE, correctness lens234c NOT SAFE (Opus 5.5 high, read-only). Must fix:
Should fix: Nits: setup says "done" while T3 crash-loops on a rejected Both proof tests (lost restart, stale firewall) go into the fix round. A fix round follows; two lenses re-check. |
- Agent code runs only in T3's unit: default distro user root, the agent's user manager masked, walls for the mask, linger and user units. - T3's runtime is root's under /opt/pc-agents/t3; the unit runs it from there; a wall checks its owner, mode and recorded hash. - A check that cannot run or crashes stops T3 (ExecStopPost on the walls service); a failed stop is reported. - Second network wall in the unit (IPAddressDeny/IPAddressAllow with the new peers key); the firewall wall compares the loaded table with the file; check timer every 2 minutes. - Units wall asks systemctl show; the file scan covers system.control, system.attached, generators, /run, /usr/lib and prefix drop-ins. - Gateway wall fails when allow or peers cover the gateway. - lockedSettings takes whole objects and record entries; blank organization refused; StartLimit for a policy T3 cannot apply. - run() kills the whole process group on a timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merges fork/pc-agents-walls (fix round 1: masks, linger and user-unit walls, root-owned T3 runtime, loaded-table firewall wall, ExecStopPost stop, systemd's own view of the units) and fixes the setup and owner steps: - B1: owner steps never open a shell as agent; `pc-agents as-agent -- CMD` runs logins as the agent through systemd-run under the unit's own walls (lib sandbox()); default distro user is root. - B2: root-owned /etc/pc-agents/restart-pending written before the first change, removed only after T3 restarted and is active, so a stopped run or a failed install still restarts T3 next time. - B3: the firewall is loaded right after its file is written, before T3's download. - B5/B6: step 6 --accept-dns=false; 30-minute install and 15-minute apt timeouts; bootstrap curl timeouts. - B7: apt-get update before installing nftables; the error names the fix. - B8: is-active after the restart (points at journalctl); the record keeps old and new hashes during a write; setup refuses unsafe /opt/pc-agents, /etc/systemd/system and parents; first-launch user hint; README steps fetch the bootstrap from ci-release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fix round 1 (head 6f840d6; merges part 1's round 1 first, no force-push)
Each new test was checked by taking its fix out: it fails. setup.test.py 26 OK, check.test.py 23 OK. The PR body has the failure table. |
Lens: correctness re-check after fix round 1 (lenspcac1), read at: 364edb6 (#232) and 6f840d6 (#234) — SAFE WITH NITS (both)The kernel walls, the root-owned runtime, the crash stop and the firewall re-check hold. Every round-0 must-fix and should-fix is closed or honestly documented. Tests: Round 0: 232-1 FIXED ( Findings (lines at 6f840d6)
State: killed at any write → marker first, record keeps both hashes; killed in |
Lens: security re-check after fix round 1 (lenspcas1), read at: 364edb6 (#232) and 6f840d6 (#234) — #232 SAFE WITH NITS, #234 NOT SAFETests at both heads: Round 0: #232 A1-A5 FIXED; A6 FIXED for the gateway (Windows host LAN address inside Findings
State: again → |
Loopback only to T3's port and a devPorts range (unit SocketBindDeny/Allow, nft drop on lo); walls service ordered after T3 with a start timeout; agent login shell nologin, sudoers and running-timer walls; README limits (hairpin NAT, D-Bus, shm, sock_diag, resolved stub, OSC 52) and how to restart T3 after a wall stopped it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SocketBindDeny/Allow move into sandbox(), so as-agent gets them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ute limit The serial run had grown to about 500 s of tests on a 4-core runner and was cancelled at the job limit. Test files now run FORK_CI_TEST_JOBS at once (default: the CPU count), each one's output printed whole in its own group. A process a test leaves behind holding its output no longer keeps the run waiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runAll test drives controlled promises and checks start order and the most running at once; the script test has two files that each wait for the other's marker file, so they pass only when run side by side (and fail with FORK_CI_TEST_JOBS=1). No timers or Date.now in the tests. Header cut to 3 lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`sed ... | grep -q` under pipefail fails when grep exits at the match while sed still writes: a race (5 in 2000 runs idle, 45 under load) that the parallel runner made show up in fork-follow.test.sh. Ten such pipes (grep -q, head) now read the whole output first; two of them failed open (`! ... | grep -q`). fork-ci-tests.test.cjs fails on any new one in a listed shell test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- restart wall: setup marks T3 for a restart (boot id + monotonic time) before and after every change; check fails, and --enforce stops T3 and as-agent units, while a T3 that started before the marker runs. - after the restart setup watches T3 for 15 s before "done" and removes the marker only then; reset-failed first, so a rerun within the start limit still starts T3. - as-agent runs the command through /usr/bin/env with the agent's PATH (~/.local/bin first); the agent gets /usr/sbin/nologin (usermod for an agent an older kit made). - SIGTERM/SIGHUP kill the running child's process group too. - README: the restart wall, Windows logon task to keep the distro up, limits, step 6 wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fix round 2 (head 4f70d86; part 1 merged in at d6f9e20) Each fix below was also checked by taking it out of the new code alone: the tests named fail every time (9 of 9).
State: the README's "Runs again, stops half-way" ( Tests (fake distro only): |
Main's fork-run-alert.test.sh had the same early-exit pipe the new guard refuses; read the whole output first, as in the other shell tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pc-agents-kit # Conflicts: # fork-kit/README.md # fork-kit/common/machine.example.json
# Conflicts: # .github/fork-ci-tests.txt # fork-kit/pc-agents/README.md # fork-kit/pc-agents/lib/pc_agents.py # fork-kit/pc-agents/sbin/pc-agents # fork-kit/pc-agents/tests/check.test.py # fork-kit/pc-agents/tests/fake_distro.py
Part 2 of 2 of the pc-agents kit, stacked on part 1 (#232):
pc-agents setup, which makes a fresh WSL2 distro on a Windows PC into the wall around T3's agents, the bootstrap that starts it, and the owner steps. Review part 1 first; this PR's base is part 1's branch (merged in after part 1's fix round 1).What is in it
sbin/pc-agents setup --source DIR --sha SHA(as root):agentwith another uid or in another group; a file the kit would write that it did not write or that was edited since; a cron/at/ssh unit where the kit masks one;/,/opt,/opt/pc-agents,/etc,/etc/systemd,/etc/systemd/system,/etc/pc-agentsor/etc/t3codenot root's alone);agent(no sudo, its own group only, login shell/usr/sbin/nologin; an older kit's agent gets it withusermod); ifnftis missing,apt-get updatethenapt-get install -y nftables;/opt/pc-agents/<sha>/(root-owned) and switches thecurrentlink;/opt/pc-agents/t3with upstream'sscripts/install.sh(T3CODE_HOME,T3CODE_VERSION,T3CODE_RELEASE_BASE_URL, clean environment, 30-minute limit);pc-agents check; only when every wall holds does it enable T3 and the check timer. It clears the start limit (systemctl reset-failed) first. If anything changed (the restart marker is there) it restarts T3, then watches for 15 s that T3 stays up (active, the same main process, no automatic restart; on failure it points atjournalctl -u t3code-agents.service), and only then removes the marker. On a failed wall it disables both and says what to do (the first run always ends here:wsl --terminatesowsl.confapplies, then run again).pc-agents as-agent -- CMD(as root): runs CMD asagentthroughsystemd-run(a transient system unit namedpc-agents-as-agent-<pid>, whichcheck --enforcestops with T3) with the same sandbox lines as T3's unit, after a passing check. CMD runs through/usr/bin/env --, so a bare name is found in the agent'sPATH(~/.local/binfirst), not root's. The owner steps use it for the Claude Code install and the Claude, Codex andghlogins, so no step opens a login shell or starts the agent's user manager. The distro's default user is root.bootstrap.sh: fetches a commit that is on the fork's protectedci-releasebranch and runs the setup from it (the heavy kit's bootstrap, adapted; curl has connect and total time limits). The README fetches the bootstrap itself fromci-release, so the script that verifies the sha is a reviewed one.--accept-dns=false, Claude andghlogins throughas-agent, linking with Full access, a logon task that keeps the distro running), the Tailscale tag and ACL, the state answers, the limits.fork-kit/common/machine.example.json: apc-agentssection (made-up values,peersincluded; a test keeps it equal to the README's example and valid).Decisions in fix round 1
/etc/pc-agents/restart-pending, root-owned, written before the first changing write and removed only after T3 restarted and is active. Any stopped run, failed install or T3 that did not start leaves it, so the next good run restarts T3 even with nothing left to write.as-agent(B1):systemd-run --uid=agentwith every--property=of the unit's sandbox (libsandbox(), shared with the unit), notrunuser: the logins run inside the same walls as T3. Part 1 masksuser@<uid>.serviceand fails on linger and user units.Runs again, stops half-way, meets old or empty state
test_second_run_changes_nothingtest_stopped_half_way_then_run_again(every write in turn)policy.jsonis writtencheckfails on it (the 2-minute check stops it); the marker makes the next run restart it, oncetest_killed_after_the_policy_is_written_still_restarts_t3,RestartPending.test_killed_after_the_policy_is_written_check_failscheckfailing onrestartuntil setup runs again; thencheckpassesRestartPending.test_upgrade_killed_at_any_write_fails_check_until_run_again(every write in turn)checkpasses; the marker stays and the next run restarts T3 once moreRestartPending.test_check_passes_on_its_own_restart_and_a_kill_while_watching_restarts_againSIGTERMorSIGHUP(closed window) during the downloadSignals.test_sigterm_or_sighup_kills_the_running_download_tootest_killed_at_a_write_then_the_machine_file_edited(before and after the write)checkfails onrestartwhile the old T3 runs,--enforcestops it andas-agentrefuses; the next run installs and restarts T3test_failed_t3_install_on_upgrade_still_restarts_t3_on_the_next_run,RestartPending.test_interrupted_upgrade_keeps_old_t3_and_check_fails,StaleFirewall.test_tightened_allow_with_failed_install--enforcestops T3StaleFirewall.test_a_loaded_table_that_is_not_the_files_failsjournalctl -u t3code-agents.service; the marker staystest_t3_that_does_not_start_is_reportedtest_t3_that_dies_a_moment_after_the_restart_is_reportedreset-failedbefore enable/restart, so T3 startstest_rerun_within_the_start_limit_still_restarts_t3run())Runtestnft, apt offlineapt-get updatethen install; on failure the message names the command to runtest_nftables_installed_after_a_package_list_updatecheckfails onrestartwhile T3 runs; the next setup run restarts T3 and removes itRestartPending.test_marker_of_an_older_kit_fails_until_setup_runs_again,Restart.test_marker_of_an_older_kit_fails(check.test.py)Walls.test_all_holdExecMainStartTimestampMonotonic, unreadable marker,systemctl showfailscheckfails onrestartRestart.test_no_start_time_fails,test_unreadable_marker_fails,test_systemctl_show_failsRestart.test_marker_of_an_earlier_boot_passes,test_t3_started_before_the_marker_fails_and_enforce_stops_itusermod --shell /usr/sbin/nologin, oncetest_agent_of_an_older_kit_gets_no_login_shelluserdel -rhint)Refused.*,test_a_distro_with_its_own_user,test_folders_that_are_not_roots_alonetest_bad_configas-agentrunsAsAgent.test_refused_when_a_wall_does_not_holdDecisions in fix round 2
CLOCK_MONOTONICtime, and is rewritten before and after every change (and after a T3 install, and right before the restart).checkgets arestartwall: it fails while the marker is there and T3's main process started before that time (systemctl show -p MainPID,ExecMainStartTimestampMonotonic); an unreadable marker, start time orsystemctl showfails too. No T3 process (stopped, or in itsExecStartPrecheck) and a marker of an earlier boot pass. Setup's own first check skips the wall. Monotonic time, not the wall clock: WSL's clock jumps on resume and time sync, and the monotonic one is what systemd reports.SETTLE= 15 s, polled every second; the marker is removed only after it.fork-kit/desk/windows/start-desk.ps1, inline there, so not reusable as a file),conhost.exe --headless, at the owner's logon, no administrator.What can be locked
Nothing had to be left unlocked: the managed policy locks any T3 setting, the fork's included (part 1's server test). Since part 1's round 1, T3 runs from root's copy, so the locks bind the agent; the limits that are WSL2's, not the kit's, are in the README's Limits.
General vs project
General: any Windows PC with WSL2 and any fork of this repo. The fork's
OWNER/NAME, the commit, the distro name and every machine fact (T3 version and release URL, port, uid, peers, child ceiling and other locked settings, caps, nameservers, Tailscale tag) are the owner's input: the bootstrap's arguments and the machine file. Nothing project-specific is in the kit; the examples use made-up values.Upstream fit
The T3 install is upstream's
scripts/install.sh(run by root withT3CODE_HOME); the locks are upstream's managed policy; T3 runs with upstream'sT3CODE_*variables. No upstream file changes. The unit is the kit's own root system unit rather thant3 service install's user unit, so the agent cannot override it.Tests
python3 -I -B fork-kit/pc-agents/tests/setup.test.py: 35 tests on a fake distro (fakeuseradd/usermod,envas T3's installer,apt-get,systemctlthat knows which ExecStart and policy a running unit has, its start time and restarts,systemd-runthat looks commands up in the caller's PATH,nft; never the real system).check.test.py: 36 tests, with therestartwall's 8 cases. Round 2: the new tests fail on the round-1 code (setup 29, check 10), and each fix taken out alone fails its own test (9 of 9).Size
Non-test code added over part 1: about 410 lines (
sbin/pc-agents+250/-6,bootstrap.sh51, lib +98/-33 forsandbox(), the marker path and the restart wall, the machine file example 13). Part 1 has 493; together over 400, hence the split.Follow-up (in the README's Limits): no hash manifest of the kit's code yet, and the bootstrap duplicates the heavy kit's.
🤖 Generated with Claude Code