Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
04a398d
fork fleet: desk setup pack kit (D8)
scuttletoby Nov 15, 2026
3a91ca2
fork fleet: Windows starter for a desk pool (D9)
scuttletoby Nov 15, 2026
f18fbda
fork fleet: free/v1 "is this machine free?" contract and examples (D10)
scuttletoby Nov 15, 2026
57e1cb5
fork fleet: `seed` verb and `seeds` in the fleet config (D11)
scuttletoby Nov 15, 2026
5f64eb9
fork kit desk: fix round 1 (drop-ins, uninstall, re-run, bootstrap) (D8)
scuttletoby Nov 15, 2026
3e3e314
fork kit desk: Windows starter fix round 1 (D9)
scuttletoby Nov 15, 2026
dd54df1
Merge branch 'fork/desk-d8-setup-pack' into fork/desk-d9-windows-starter
scuttletoby Nov 15, 2026
8a312ae
fork kit: free/v1 fix round 1 (D10)
scuttletoby Nov 15, 2026
1232b6c
Merge branch 'fork/desk-d9-windows-starter' into fork/desk-d10-free-c…
scuttletoby Nov 15, 2026
7026afd
fork fleet: seed fix round 1 (D11)
scuttletoby Nov 15, 2026
9c12570
Merge branch 'fork/desk-d10-free-contract' into fork/desk-d11-seed
scuttletoby Nov 15, 2026
7e73543
fork kit desk: the uninstall test gives its namespace through the fak…
scuttletoby Nov 15, 2026
99bf235
fork kit desk: the record-last test lists folders by name (D8)
scuttletoby Nov 15, 2026
8348cde
Merge branch 'fork/desk-d8-setup-pack' into fork/desk-d9-windows-starter
scuttletoby Nov 15, 2026
a884923
fork kit desk: the starter's comment no longer names the old encoding…
scuttletoby Nov 15, 2026
c0fb90c
Merge branch 'fork/desk-d9-windows-starter' into fork/desk-d10-free-c…
scuttletoby Nov 15, 2026
f706ea4
fork kit free: resolve the link once, then judge and run that file (D10)
scuttletoby Nov 15, 2026
6e6e40d
Merge branch 'fork/desk-d10-free-contract' into fork/desk-d11-seed
scuttletoby Nov 15, 2026
099f438
fork fleet: seed tests read the small stream and use a valid second p…
scuttletoby Nov 15, 2026
4a13be5
fork kit desk: fix round 2 (restart mark, unit check, deny lines, fle…
scuttletoby Nov 15, 2026
af1c3b1
Merge branch 'fork/desk-d8-setup-pack' into fork/desk-d9-windows-starter
scuttletoby Nov 15, 2026
e88ff71
fork kit desk: Windows starter fix round 2 (WSL 2 only, inputs first,…
scuttletoby Nov 15, 2026
7ee9a8d
Merge branch 'fork/desk-d9-windows-starter' into fork/desk-d10-free-c…
scuttletoby Nov 15, 2026
83849bc
fork kit free/v1: judge the path as written, bounded wait (D10 fix ro…
scuttletoby Nov 15, 2026
71e30a4
Merge branch 'fork/desk-d10-free-contract' into fork/desk-d11-seed
scuttletoby Nov 15, 2026
3fb8617
fork fleet: seed fix round 2 (rename while sent, deadline, max_mb cap…
scuttletoby Nov 15, 2026
8c7524f
fork kit desk: fix round 3 (link check, locked runner users, modes, l…
scuttletoby Nov 16, 2026
adcc265
Merge branch 'fork/desk-d8-setup-pack' into fork/desk-d9-windows-starter
scuttletoby Nov 16, 2026
f211bcd
fork kit desk: Windows starter fix round 3 ([experimental] keys, chan…
scuttletoby Nov 16, 2026
44c1b9c
Merge branch 'fork/desk-d9-windows-starter' into fork/desk-d10-free-c…
scuttletoby Nov 16, 2026
de6837e
free/v1: say busy-while-running matches the process name; note the tr…
scuttletoby Nov 16, 2026
e8bdc7c
Merge branch 'fork/desk-d10-free-contract' into fork/desk-d11-seed
scuttletoby Nov 16, 2026
e5e0000
fleet seeds: a last stall alarm after the last write is not a stall; …
scuttletoby Nov 16, 2026
1ccced0
fork kit desk: fix round 4 (linked or foreign folders and key refused…
scuttletoby Nov 16, 2026
4a493a5
Merge branch 'fork/desk-d8-setup-pack' into fork/desk-d9-windows-starter
scuttletoby Nov 16, 2026
b59601f
fork kit desk windows: fix round 4 (owed mark before wsl.conf changes…
scuttletoby Nov 16, 2026
6c91db5
Merge branch 'fork/desk-d9-windows-starter' into fork/desk-d10-free-c…
scuttletoby Nov 16, 2026
297a2e2
free/v1: Wine names its process after the .exe, not wine (D10 fix rou…
scuttletoby Nov 16, 2026
e56ad75
Merge branch 'fork/desk-d10-free-contract' into fork/desk-d11-seed
scuttletoby Nov 16, 2026
70822b5
fork fleet desk seed: test the one-piece stall guard (D11 fix round 4)
scuttletoby Nov 16, 2026
d4184a9
fork kit desk: fix round 5 (subfolders refused like the top folders, …
scuttletoby Nov 16, 2026
60cd894
Merge branch 'fork/desk-d8-setup-pack' into fork/desk-d9-windows-starter
scuttletoby Nov 16, 2026
afc4d44
Merge branch 'fork/desk-d9-windows-starter' into fork/desk-d10-free-c…
scuttletoby Nov 16, 2026
22a6433
Merge branch 'fork/desk-d10-free-contract' into fork/desk-d11-seed
scuttletoby Nov 16, 2026
ed4e6db
Merge main (after #230) into fork/desk-d11-seed
scuttletoby Nov 16, 2026
9bf7c64
Merge remote-tracking branch 'origin/main' into fork/desk-d11-seed
scuttletoby Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/fork-ci-tests.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
.github/scripts/fork-contracts.test.py
.github/scripts/fork-fleet-desk-agent.test.py
.github/scripts/fork-fleet-desk-key.test.py
.github/scripts/fork-fleet-desk-seed.test.py
.github/scripts/fork-fleet-desk.test.py
.github/scripts/fork-fleet.test.py
.github/scripts/fork-fleet-refresh.test.py
Expand Down
46 changes: 46 additions & 0 deletions .github/fork-fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,7 @@ done
files="fork-fleet.json:fork-fleet.json:644 scripts/fork-fleet.py:fork-fleet.py:644
scripts/fork-fleet-hook.sh:hook.sh:755 scripts/fork-fleet-launch.sh:fork-fleet-launch.sh:755
scripts/fork-fleet-desk.py:fork-fleet-desk.py:644 scripts/fork-fleet-desk-wire.py:fork-fleet-desk-wire.py:644
scripts/fork-fleet-desk-seed.py:fork-fleet-desk-seed.py:644
scripts/fork-fleet-refresh.service:fork-fleet-refresh.service:644
scripts/fork-fleet-refresh.timer:fork-fleet-refresh.timer:644
scripts/fork-fleet-job-paths.txt:fork-fleet-job-paths.txt:644"
Expand Down Expand Up @@ -685,6 +686,51 @@ Bun, Node and Rust are installed per job by the setup actions, into the job's
own home, which goes with its copy. Leave a few tens of GB of disk for
`/var/lib/fork-fleet/runners`.

#### Seeds (warm caches for a desk pool)

A job's copy starts empty, so each job would download its tools and
dependencies again. A desk pool may list **seeds** in its fleet config entry,
ready-made caches the desktop keeps and its jobs start from:

```json
"seeds": [{"name": "gradle", "every_hours": 6, "max_age_hours": 72, "max_mb": 4096}]
```

`every_hours` is how often a desktop takes a new one, `max_age_hours` (at
least `every_hours`) the age, from when it was made, at which it is dropped,
`max_mb` its largest size (at most 16384). Nothing in the fleet makes a seed: on the always-on
host the project's own timer or job writes a tar of a cache folder's contents
to `/var/lib/fork-fleet/seed-files/POOL/NAME.tar` (a folder per pool, so two
pools never share a file), by a temp file and a rename. A file changed in place
while it is sent (size, mtime or ctime) is refused; one replaced by a rename while
it is sent is fine (the old one is sent whole). A file dated more than five minutes in the future, older
than `max_age_hours` or above `max_mb` is refused. Both folders and the files are
root's (or the project's), readable by the desk-link user and writable by no one
else (the owner itself is not checked, only that no one else may write); never
put a secret in a seed: every job of the pool reads it. **Make a seed only from
trusted code (the default branch), never in a job that runs pull request code:**
every later job of the pool starts from it, so a poisoned seed reaches them all.
A file replaced by a rename while another hard link to the old one stays (a
backup copy, say) is still "in place" for this check and is refused (exit 4):
keep no second link to a seed file.

The desktop asks for them with the desk link's key, in an ssh session of its
own: the same forced command sends every command that starts with `seed` to
`fork-fleet-desk-seed.py` (which step 3 installs; its header has the exit
codes), never to a session. It answers `seeds` with the pool's list and
`seed NAME` with that one file, only a name of that list, and a trailer with its
size and SHA-256; anything else is refused. One stream per pool at a time; a
desktop that takes less than 64 KiB in two minutes, or not the whole file within
ten minutes plus one second per 256 KiB, is cut off and the next one may go.
So a link must carry at least about 256 KiB a second (2 Mbit/s) for a large
seed: at 100 KiB a second, a seed above about 96 MiB is always cut off. Size
`max_mb` to the desktop's link. A desktop just fast enough not to stall can hold
its pool's lock for a long time (16 GiB at 256 KiB a second is about 18 hours);
only that pool's own seed streams wait, nothing else. The host does not limit
how often a desktop asks: the key belongs to the pool's own desktop, one stream
per pool runs at a time, and each is cut off at its deadline, so the most it
costs is one file's reads and upload at a time.

## Owner steps on the repository

Run these from a machine where `gh` is logged in as the owner. None of them
Expand Down
181 changes: 181 additions & 0 deletions .github/scripts/fork-fleet-desk-seed.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
#!/usr/bin/env python3
"""Fork only: warm caches ("seeds") for a desk pool's jobs (.github/fork-fleet.md, "Seeds").

A project lists the caches its desk pool's jobs may start from in the pool's `seeds` (fork-fleet.json): a name, how
often a desktop takes a new one (every_hours), the age at which it is dropped (max_age_hours, from when it was made)
and its largest size (max_mb). On the always-on host something of the project's own (a timer, a job) makes each one:
a tar of a folder's contents at FILES/POOL/NAME.tar (a folder per pool: pools never share a file), written to a temp
file and renamed. Nothing here makes one. The desktop pulls them with the desk link's key, in an ssh session of its
own beside the agent's:

ssh ... seeds the forced command (fork-fleet-desk.py serve POOL, the desk-link user) sees it in
SSH_ORIGINAL_COMMAND and prints the pool's list (one JSON object; `bytes` and `made` of each
seed the host offers now, `why` for one it does not)
ssh ... seed NAME streams FILES/POOL/NAME.tar: only a name of that list, never a path the desktop names. The last
stderr line is the trailer `seed NAME BYTES SHA256 MADE` (MADE: the file's mtime, epoch seconds)

The forced command sends every command that starts with `seed` here. The host side exits 2 for any other command,
3 while another stream for the pool runs, 4 when the file may not be sent (missing, a link, in a folder or a file
others may write, above max_mb, older than max_age_hours or dated in the future, or changed while it was sent: made
in place, not by rename; a file replaced by rename while it is sent is fine, the old one is sent whole), 1 when the
desktop takes less than WRITE bytes in STALL seconds or the whole file not by its deadline (GRACE seconds plus
MIN_RATE bytes a second): the pool's lock is then let go.
Standard library only.
"""
import fcntl, hashlib, importlib.util, json, math, os, signal, stat, sys, time

sys.dont_write_bytecode = True
HERE = os.path.dirname(os.path.realpath(__file__))


def module(name, file):
spec = importlib.util.spec_from_file_location(name, os.path.join(HERE, file))
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod


FILES = '/var/lib/fork-fleet/seed-files' # the always-on host's: POOL/NAME.tar, made by the project
CONFIG = os.path.join(HERE, 'fork-fleet.json')
PIECE, WRITE, STALL, AHEAD = 1 << 20, 64 << 10, 120, 300 # AHEAD: seconds a file's date may be ahead of now
GRACE, MIN_RATE = 600, 256 << 10 # a stream's deadline: GRACE seconds plus one second per MIN_RATE bytes
ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'}


class Refused(Exception):
def __init__(self, msg, code):
super().__init__(msg)
self.code = code


def pool_seeds(pool, config):
try:
cfg = module('fork_fleet', 'fork-fleet.py').load(config)
except (OSError, ValueError) as e:
raise Refused(f'the fleet config: {e}', 2) from None
for p in cfg['pools']:
if p['name'] == pool:
return {s['name']: s for s in p.get('seeds', [])}
raise Refused(f'pool {pool} is not in {config}', 2)


def offered(seed, folder, now):
"""An open fd and the stat of FOLDER/NAME.tar (FOLDER: FILES/POOL); Refused (4) when it may not be sent."""
path = os.path.join(folder, seed['name'] + '.tar')
try:
for d in (os.path.dirname(folder), folder):
st = os.lstat(d)
if not stat.S_ISDIR(st.st_mode) or st.st_mode & 0o022:
raise Refused(f'{d} must be a folder (not a link) that only its owner may write', 4)
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC)
except OSError as e:
raise Refused(f'{path}: {"not made yet" if isinstance(e, FileNotFoundError) else e.strerror}', 4) from None
st = os.fstat(fd)
hours = seed['max_age_hours']
why = ('not a plain file' if not stat.S_ISREG(st.st_mode) else 'others may write it' if st.st_mode & 0o022
else f'above max_mb ({seed["max_mb"]})' if st.st_size > seed['max_mb'] << 20
else f'older than max_age_hours ({hours})' if now - st.st_mtime > hours * 3600
else 'dated in the future' if st.st_mtime > now + AHEAD else None)
if why:
os.close(fd)
raise Refused(f'{path}: {why}', 4)
return fd, st


def offer(pool, command, lock_dir, out=1, err=sys.stderr, config=CONFIG, files=FILES, now=time.time):
try:
if os.geteuid() == 0:
raise Refused('serve never runs as root', 2)
word, _, name = command.partition(' ')
seeds, folder = pool_seeds(pool, config), os.path.join(files, pool)
if command == 'seeds':
got = []
for s in seeds.values():
item = {'name': s['name'], **{k: s[v] for k, v in ITEM.items()}}
try:
fd, st = offered(s, folder, now())
os.close(fd)
item.update(bytes=st.st_size, made=int(st.st_mtime))
except Refused as e:
item['why'] = str(e)
got.append(item)
os.write(out, (json.dumps({'v': 1, 'seeds': got}) + '\n').encode())
return 0
if word != 'seed' or name not in seeds:
raise Refused(f'refused: {command[:60]!r} (only `seeds`, or `seed NAME` of pool {pool}\'s seeds)', 2)
lock_path = os.path.join(lock_dir, f'{pool}.seed.lock')
try:
lock = os.open(lock_path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600)
except OSError as e:
raise Refused(f'{lock_path}: {e.strerror} (a leftover of another user? remove it)', 1) from None
try:
if os.fstat(lock).st_uid != os.geteuid():
raise Refused(f'{lock_path} belongs to uid {os.fstat(lock).st_uid}, not this user: remove it', 1)
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
raise Refused(f'another seed stream for pool {pool} runs', 3) from None
fd, st = offered(seeds[name], folder, now())
try:
sha, sent = send(fd, out, st.st_size)
after = os.fstat(fd)
finally:
os.close(fd)
finally:
os.close(lock)
# A rename over NAME.tar unlinks the open file (st_nlink 0), which changes its ctime but not its bytes:
# only a file still in place must keep its ctime (a rewrite with the old mtime put back).
same = ('st_size', 'st_mtime_ns') + (('st_ctime_ns',) if after.st_nlink else ())
if sent != st.st_size or any(getattr(after, k) != getattr(st, k) for k in same):
raise Refused(f'{name}: it changed while it was sent (made in place, not by rename?)', 4)
print(f'seed {name} {sent} {sha} {int(st.st_mtime)}', file=err, flush=True)
return 0
except Refused as e:
print(f'fork-fleet-desk seed: {e}', file=err, flush=True)
return e.code


class Stalled(Exception):
pass


def stalled(*_):
raise Stalled


def send(fd, out, size, stall=None):
"""Sends at most SIZE bytes of FD (the size that was checked: a file that grows is cut there, and then fails the
changed-file check) to OUT; Refused (1) when OUT takes less than WRITE bytes in STALL seconds, or not all of it
within GRACE seconds plus SIZE / MIN_RATE."""
stall = STALL if stall is None else stall
sha, sent = hashlib.sha256(), 0
deadline = time.monotonic() + GRACE + size / MIN_RATE
wrote = 0
old = signal.signal(signal.SIGALRM, stalled)
try:
try:
while sent < size and (data := os.read(fd, min(PIECE, size - sent))):
sha.update(data)
sent += len(data)
view = memoryview(data)
while view:
left = deadline - time.monotonic()
if left <= 0:
raise Stalled
signal.alarm(max(1, min(stall, math.ceil(left)))) # each piece that moves starts the wait again
n = os.write(out, view[:WRITE])
wrote += n
view = view[n:]
finally:
signal.alarm(0) # an alarm already on its way still arrives here, inside the `except Stalled` below
except Stalled:
if wrote == sent and not (sent < size and os.fstat(fd).st_size > sent):
pass # the last alarm came after the last write: everything read was sent
elif time.monotonic() >= deadline:
raise Refused(f'the desktop did not take {size} bytes within {GRACE + size / MIN_RATE:.0f} seconds '
f'(at least {MIN_RATE >> 10} KiB a second): stopped', 1) from None
else:
raise Refused(f'the desktop took less than {WRITE >> 10} KiB in {stall} seconds: stopped', 1) from None
finally:
signal.signal(signal.SIGALRM, old)
return sha.hexdigest(), sent
Loading
Loading