Repository navigation
fork fleet: seed verb and seeds in the fleet config (D11) - #236
scuttletoby wants to merge 46 commits into
Conversation
442e00e to
01b61e4
Compare
cb91fdd to
91abd6e
Compare
01b61e4 to
0781efb
Compare
91abd6e to
f877ff9
Compare
0781efb to
ca286df
Compare
fork-kit/desk: bootstrap, install, PASS/FAIL check and uninstall for a desktop's side of a desk pool (runner users, launcher, walls, desk agent, gate hook, settings, key, units), from one release commit. Safe to run again and to stop at any step; refuses an install it did not make; uninstall removes only what it made. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
start-desk.ps1 sets up a WSL2 distro for the desk kit from Windows PowerShell 5.1: a memory and swap cap in .wslconfig, NAT, the WSL2 walls' settings inside the distro, the packages, the machine file, then the desk kit's bootstrap from one release commit, a logon task that keeps the distro running, and a PASS/FAIL check. -Uninstall removes it. Every step skips what is already right; WSL restarts only on a change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One small contract for every tool that asks whether a machine may take more work: no arguments, 3 s, exit 0 free / 75 busy, one JSON object on stdout. fork-kit/common/free_contract.py is the one reader (judge, ask, and a CLI for shell callers); free-fixtures.json holds the cases every reader must agree on. Two examples: memory (pressure, swap) and busy-while-running (named programs). The launcher's fits also asks FREE_COMMAND when launch.conf names one (default none: the memory rule alone). The desk kit installs the reader and the memory example and writes FREE_COMMAND from the machine file's desk.freeCommand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A desk pool may list warm caches ("seeds") in its fleet config entry. The
always-on host's forced command answers a desktop's `seeds` (the pool's list)
and `seed NAME` (that one ready-made tar, with a size and SHA-256 trailer) in
an ssh session beside the agent's. Only names of the list, never a path;
exit 3 while another stream for the pool runs, 4 when the file may not be
sent. The desktop's fetcher follows in the next change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f877ff9 to
f18fbda
Compare
ca286df to
57e1cb5
Compare
|
Review round 1 at 57e1cb5: two independent lenses (security, correctness), both SAFE WITH NITS. Nothing serves a path the desktop picks; the existing
State question: runs again (read-only, lock reused), killed mid-stream (flock dies, no trailer), broken pipe (exit 1, files closed), missing/broken config (exit 2), pool not in config (exit 2), no |
- install refuses a systemd drop-in it did not write for its units; check fails on one - re-run restarts the agent only when a file, setting or key changed - check compares the settings files with the machine file - uninstall stops every runner, keeps the nft table, users and record while one still runs, removes every fork-fleet-* namespace and ffh- link, removes only what the kit made (deny files it created, its users' deny lines), names what it leaves, and removes its record last - the record is read only when it has this kit's shape - --release-fleet-config goes back to the release's fleet config - bootstrap parses --fleet-config like the other flags Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- .wslconfig is read and written as UTF-8 without a BOM (never ASCII), by a temp file and a rename, with a dated backup; only [wsl2] keys are judged - the restart decision reads the live state (PID 1, interop, wsl.conf newer than boot, memory, networking mode, .wslconfig newer than the boot) - a distro the starter did not make is named and must be typed to take over - MemoryGB above the PC's RAM less 2 GB is refused - exit 1 on a failed check; -Uninstall needs no -Repo/-Sha; UNC paths work Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ask() runs the file unsafe() judged (the link's target), never the link - NaN and Infinity are not JSON: not available (two fixture cases) - a child that left the group with setsid and still holds stdout is ended; one that also closed stdout is the documented bound - poll() instead of select() (descriptors above 1024) - busy-while-running matches names longer than 15 characters - FREE.md: the bound, links, ACLs, the memory example's pressure caveat Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontract # Conflicts: # fork-kit/desk/sbin/desk-install
- seeds live in FILES/POOL/NAME.tar: pools never share a file - a file dated more than five minutes in the future is refused (exit 4) - the stream stops at the size that was checked; ctime and inode join the changed-file check - the forced command sends every command that starts with `seed` to the seed handler, so `seeds x` and `seed` exit 2 and never reach a session - a desktop that takes nothing for STALL seconds is cut off (exit 1) and the pool's lock let go - the lock file must be this user's; a clear refusal otherwise Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e ip (D8) Uninstall now deletes the namespaces ip netns list shows, not one per recorded user; the older test listed none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A whole-folder removal took the record first only for some directory orders, so the test passed on the old code by luck. Listing by name puts the record first, which the old order fails every time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… flag (D9) The text check that no ASCII encoding is left matched the comment that explains the bug. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The round-1 fix resolved the link again after the check, so a link repointed between the two still changed what ran. The new test caught it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ool (D11) The lock check after a stall sent 1 MB into a pipe nobody read until the call returned, and the second pool reused the first one's label and beacon, which the config check refuses. The pool test also puts a tar at the old shared path, so the old code fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fix round 1 (commits 7026afd and 099f438, the second a test-only fix; D10's round 1 merged in; head 099f438). Tests: desk-seed 12 OK, desk 42 OK.
Fails before the fix: the seed tests run against the round-0 code give 10 failures and 1 error (every test above). Some fail there partly because the per-pool path moved the tar; PR body updated. Still draft. |
|
Fix round 2 at 3fb8617 (D10 round 2 merged in first) Lens 236s r1 (security):
Lens 236c r1 (correctness):
State question:
Local run (round 2):
Red-check: the new seed tests run against round-1 code give 3 failures and 2 errors. |
Lens: security re-check r2 (lens236sr2), read at: 3fb8617 — SAFE WITH NITSEarlier findings (new tests run against round-1 099f438: 3 failures + 2 errors, as the PR claims)
New findings (nits)
Checked safe: exact command parse ( State question: runs again: read-only, lock reused ( Tests: desk-seed 16 OK, desk 42 OK (no PowerShell). |
Lens: correctness re-check r2 (lens236cr2), read at: 3fb8617 — SAFE WITH NITSRound-1 findings (round-1
Focus: matches the PR body and Nits
State question: runs again: read-only, lock reused ( Tests: desk-seed 16 OK, desk 42 OK, fork-fleet OK (7 skipped). |
…ock, stopped choice) (D8) - --fleet-config: the folders of the link and of the file it names are both checked. - A runner user made by hand must have no login shell, a locked password and no cron.allow/at.allow line: install refuses otherwise, check FAILs. - check looks at modes (files, folders, desk key); install puts a changed mode or owner back from the release. - One install or uninstall at a time (flock on /opt/fork-fleet); a second is refused and does not sweep the first one's staging folder. - The restart mark is set before a new fleet config choice is saved, and check compares the project's copy with the installed one: a run killed between the choice and the swap fails check. - An old record that names a project config that is gone: a readable refusal. - Tests for the runner-user branches of the ownership check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ges listed, PID 1 start) (D9) - networkingMode/dnsTunneling under [experimental] (the pre-2.0 form that can still win over [wsl2]) are removed by the merge, listed in the question, and the Windows-side check FAILs while one is there. - The question lists each .wslconfig key it changes, old -> new. - The owed-shutdown mark is one for every distro, like .wslconfig; uninstall keeps it. - wsl.conf is compared with PID 1's start (/proc/1/stat field 22 + btime), not the mtime of /proc/1; unreadable means restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontract # Conflicts: # fork-kit/desk/sbin/desk-install # fork-kit/desk/tests/install.test.py
…ailing-/ link case Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no-seeds test; document trust, rate and links The stall alarm could go off between the end of the send loop and alarm(0), so Stalled escaped as a traceback. Now alarm(0) runs inside the try and a late alarm after everything read was written counts as done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix round 3Commit e5e0000 (head; D10 merged in as e8bdc7c). Findings from lens236sr2 (security) and lens236cr2 (correctness):
State question: runs again: read-only, the lock is reused. Stops half-way: flock let go by the kernel, no trailer, the desktop keeps its old copy; the race fix changes nothing here. Old/empty: a pool without Tests (local, at e5e0000): |
Lens: security + correctness re-check r3 (lens236r3), read at: e5e0000 — SAFE WITH NITSRound-2 findings (round-2 head 3fb8617; round-2
New finding
def test_a_desktop_that_stops_reading_a_one_piece_seed_is_exit_1(self):
self.make('m2', {'a': os.urandom(512 << 10)}) # under PIECE: sent == size at the stall
r, w = os.pipe() # nobody reads r
err = io.StringIO()
try:
with mock.patch.object(seed, 'STALL', 1):
code = seed.offer(POOL, 'seed m2', os.path.join(self.t, 'run'), out=w, err=err,
config=os.path.join(self.t, 'fork-fleet.json'), files=os.path.join(self.t, 'files'))
finally:
os.close(r); os.close(w)
self.assertEqual(code, 1, err.getvalue())
self.assertNotIn('seed m2 ', err.getvalue()) # no trailer for a cut streamThe PR body's red-check list ("all 11 red") does not cover this guard. What the round could have broken (checked, fine)
State question
Tests at head: desk-seed 18 OK, desk 42 OK. Round-2 seed code under the head tests: 1 error (the race test). Two mutations of |
…, runner users as strict as the launcher, lock in /etc/fork-fleet, owner of units and settings) (D8) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, question and merge decide the same way) (D9) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontract Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd 4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix round 4Commit 70822b5 (head; D10's round 4 merged in first as e56ad75). Findings from lens236r3:
State question: test only; the code did not change. Runs again: read-only. Stopped half-way: a cut one-piece stream gets no trailer, so the desktop keeps its old copy. Old or empty state: unchanged from round 3. Tests (local, at 70822b5): |
Lens: re-check r4 (lensd8minr4), read at: 70822b5 — SAFE
|
…main group named after the user, public key in check, no FIFO opened) (D8) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lens: re-check (lensd8minr5), read at: 22a6433 — SAFERound-5 merge is clean:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Row D11 of the desktop pool plan, part 1 of 2: the host side of seeds, the warm caches for a desk pool's jobs. It adds
seedsto a pool's fleet config entry and aseeds/seed NAMErequest on the desk link's forced command. Stacked on #230 (D10). Critical: it adds a second request to the desk link's forced command.The D11 row came to 518 non-test lines, over the 400 limit, so it is split:
Why
A job's copy starts empty, so every job on a desktop downloads its tools and dependencies again. That makes it slow and wastes the PC's network. With seeds, the always-on host offers ready-made caches and the desktop keeps a recent copy that its jobs start from.
pools[].seedsis a list of{"name", "every_hours", "max_age_hours", "max_mb"}.check_configrefuses bad names, values that are not positive whole numbers (bools too),max_age_hoursbelowevery_hours,max_mbabove 16384, missing fields and duplicate names.fork-fleet-desk-seed.py(standard library only) answers two requests.seedsprints the pool's list as one JSON object: for each seed,bytesandmadewhen it is offered now, andwhywhen it is not.seed NAMEstreams/var/lib/fork-fleet/seed-files/POOL/NAME.tar(a folder per pool: two pools never share afile), followed by a trailer line on stderr:
seed NAME BYTES SHA256 MADE.fork-fleet-desk.py serve POOLalready runs as the desk-link user for the desk key. WhenSSH_ORIGINAL_COMMANDstarts withseed(soseeds,seed NAME, and alsoseeds x,seed,seedsx), it hands the request to the seed module instead of serving a session; the module refuses anything butseedsandseed NAMEwith exit 2. Every other request keeps the old path.Safety (critical)
seedslist is served. The name never becomes part of a path unless it is on that list, and names match[a-z0-9_-](the POOL pattern).seed ../m2,seed /etc/passwd,seed M2,seed m2 x,seeds xand the like all exit 2 and send nothing (tested, also through the real forced command:seeds x,seed,seedsxandseedsexit 2 and make no socket). A command that does not start withseedkeeps the old path (a session request).O_NOFOLLOW. It must be a plain file that no group or other user may write, in a pool folder and a files folder (not links) that no group or other user may write, at mostmax_mb, younger thanmax_age_hours, and not dated more than 5 minutes in the future. Otherwise it exits 4 and sends nothing. The folders' owner is not checked, only their mode (documented).st_nlink0), so only its ctime (changed by the unlink) is not compared, and the old whole file is sent (tested). The trailer's SHA-256 is of exactly the bytes sent.POOL.seed.lock, undersafe_dir). A second stream exits 3, so a misbehaving desktop cannot open many. A desktop that takes less than 64 KiB in 120 s, or not the whole file within 600 s plus one second per 256 KiB, is cut off (exit 1), so it cannot hold the lock for long. A lock file that cannot be opened or that another user owns is a clear refusal (exit 1, tested).What happens when this runs again, stops half-way, or meets existing, old or empty state?
flock, so it goes with the process, and the next stream gets it (tested).max_age_hoursis not offered. A file replaced by rename while being sent is fine, because the open fd still reads the old whole file. A file written in place is refused. A fleet config withoutseedsmeans an empty list. Older desktops that never ask are unaffected.why: not made yetin the list, andseed NAMEexits 4. A pool with no seeds lists[].General vs project
General. The verb, the config check and the offer name no project. Which caches a pool has, and how the host makes each tar, are the project's own config and its own timer. The doc example (
gradle) is only an example.Upstream fit
Only the fork-only
.github/scripts/fork-fleet*files,.github/fork-fleet.mdand the fork CI test list change. Nothing in upstream T3 changes.Tests
.github/scripts/fork-fleet-desk-seed.test.py(new, in.github/fork-ci-tests.txt). It uses temp folders only: no root, no ssh, no network.max_mbat its cap and 11 bad ones.seeds,seed NAMEand every otherseed-prefixed command. It serves no session for them.The existing
fork-fleet-desk.test.pyandfork-fleet.test.pywere run too.Local run (round 2):
.github/scripts/fork-fleet-desk-seed.test.py: 16 OK.github/scripts/fork-fleet-desk.test.py: 42 OK.github/scripts/fork-fleet.test.py: 201 OK (7 skipped)seed/seedsdivert, the stall deadline). Some of these fail on the round-0 code only because the per-pool path changed where the test puts the tar.max_mbcap). Mutations of the round-2 code: without the future-date check, without the lock-owner check, and with round-0's path (no pool folder), the matching tests fail (the future-dated case now says why it fails: "not made yet" instead of "dated in the future").Red-checks
Each rule was broken in a copy, one at a time, and the named test failed (all 11 red):
test_other_commands_and_names_are_refused_and_send_nothingO_NOFOLLOW) ->test_a_file_that_may_not_be_sent_is_exit_4max_age_hoursis not sent -> the same testtest_a_file_changed_while_sent_is_exit_4test_a_second_stream_for_the_pool_is_exit_3test_a_lock_of_another_user_is_refusedtest_a_file_that_may_not_be_sent_is_exit_4test_never_as_root_and_a_bad_config_is_exit_2max_age_hoursat leastevery_hours->Configtest_the_forced_command_diverts_seed_requestsThe first run found one gap: with
O_NOFOLLOWremoved the link case still passed, because the link's target wasgroup-writable and refused for that reason. The test now makes the target 0644, so only the link can be the reason.
Round 3
signal.[]).fork-fleet.md"Seeds": make a seed only from trusted code, never in a job that runs PR code; a hard link kept to the old tar means exit 4; at least about 256 KiB/s is needed (a slower link is backed off by fork fleet: a desktop fetches its desk pool's seeds (D11b) #237); the pool lock can be held about 18 h for 16 GiB at the slowest rate.fork-fleet-desk-seed.test.py18 OK,fork-fleet-desk.test.py42 OK,fork-fleet.test.py201 OK (7 skipped).Round 4
test_a_desktop_that_stops_reading_a_one_piece_seed_is_exit_1covers thewrote == senthalf of the stall guard (fork-fleet-desk-seed.py:172): a seed under one piece hassent == sizeat the stall, so only that half refuses the cut stream. It passes at head and fails when that half is removed (checked in a copy). The red-check list above does not include it; this line does.fork-fleet-desk-seed.test.py19 OK,fork-fleet-desk.test.py42 OK; the merged kit testsinstall.test.py44 OK,windows.test.py24 OK (4 skipped),free_contract.test.py17 OK.🤖 Generated with Claude Code