diff --git a/.gitignore b/.gitignore index 4ed1fbf..9c8efdc 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,8 @@ target/ .spike-*-id .control-id .DS_Store + +.openshell-separated/ +.control-separated-id +.builder-separated-id +__pycache__/ diff --git a/README.md b/README.md index 1f8cfed..b977491 100644 --- a/README.md +++ b/README.md @@ -1,77 +1,159 @@ -# OpenShell on E2B +# OpenShell on E2B: separate supervisor and worker + +An [NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell) compute driver that creates **two private E2B microVMs per task**. The shared gateway and compute driver stay outside both task VMs. OpenShell owns policy, program identity, request authorization and audit; E2B owns VM isolation, provisioning and the worker's external network fence. + +This is a live demo integration pinned to OpenShell **v0.1.2**, with the limitations below. + +```text + SHARED CONTROL PLANE + Laptop CLI ── mTLS ──► OpenShell gateway ↔ E2B compute driver ──► E2B API + Policies, sessions, lifecycle │ + ▲ │ creates/deletes + │ authenticated gateway channel │ both per task + ┌────────────────┴───────────────┐ ┌─────────┴────────────────────┐ + │ Sandbox A · supervisor │ │ Sandbox B · worker │ + │ 1 vCPU / 1 GiB │ │ 2 vCPU / 2 GiB │ + │ OpenShell supervisor + proxy │◄═══►│ OpenShell runtime + agent │ + │ Task auth + worker traffic key │ │ Per-session TLS identity │ + │ Allowed outbound requests ─────┼─► │ E2B host-enforced deny-all │ + │ internet │ outbound, including root │ + └────────────────────────────────┘ └──────────────────────────────┘ + Separate guest kernels; task pair destroyed together +``` + +A initiates authenticated WSS tunnels through E2B's port proxy to B and to the shared gateway. OpenShell's TLS session channel runs inside the worker tunnel. An agent's DNS/TCP/HTTP requests travel from B over that channel to A; A applies OpenShell policy and opens the approved upstream connection. **B never needs a direct internet connection.** This uses the existing public E2B HTTPS transport with private ingress tokens, not private VPC peering. + +## Run + +Requires Node 22+, `E2B_API_KEY` in `.env`, and the templates built in that E2B team. The templates are already built in the local demo account. + +```bash +npm ci +npm run setup # OpenShell CLI + wstunnel, if .bin/ is missing +npm run demo # connects control automatically; demonstrates a pair; deletes the pair +npm run demo:verify # also kills A and verifies the worker fails closed +``` + +The demo prints all three VM IDs and checks: + +- No supervisor process runs in the shared control plane. +- Every task has a distinct supervisor A and worker B. +- E2B management credentials stay in control; the worker has no supervisor auth bundle. +- Default-deny policy blocks an unlisted destination. +- Direct HTTPS works from A but fails from B **even as root outside the agent namespace**. +- A live OpenShell policy permits `curl GET`, denies `curl POST`, and denies `wget` for the same host. +- OpenShell audit records reach the gateway. +- With `demo:verify`, killing A suspends the agent in B and leaves B's external egress blocked. +- Deletion removes both task VMs while the shared control plane remains available. + +`dist/separated-demo-verification.json` and `dist/separated-demo-fault-verification.json` record the assertions and VM IDs from the latest successful standard and failure-drill runs. `npm run demo -- --keep` retains a live task for inspection instead of deleting it. + +For a two-terminal presentation: + +```bash +# Terminal 1: keep the CLI tunnel open; --new creates a fresh shared control VM. +npm run connect -- --new + +# Terminal 2: +npm run demo +npm run watch # live gateway/driver logs plus each separate supervisor's logs +``` -An external [NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell) compute driver that runs each agent in its own private E2B microVM. OpenShell confines the agent (network policy per program, HTTP-level rules, audit log); E2B provides the machines. +Local state is isolated from the original demo: `.control-separated-id`, `.openshell-separated/`, and localhost port `17691`. Use `XDG_CONFIG_HOME=$PWD/.openshell-separated .bin/openshell …` for manual CLI commands. Closing the laptop tunnel does not stop the control plane or active pairs. Reconnecting reuses the gateway tunnel identity. -**Status: working demo, not production.** See [Limits](#limits). +```bash +npm run demo:down # deletes this control plane's task VMs and the control VM +``` + +## Native OpenShell dashboard demo + +```bash +cd /Users/matt/e2b/openshell-e2b +npm run demo:dashboard +``` -![demo](media/demo.gif) +The upper pane is the unmodified OpenShell v0.1.2 `openshell term` UI. The lower-left pane controls the walkthrough; the lower-right pane samples the worker's real heartbeat and kernel process state through E2B. The launcher connects to the gateway automatically and isolates its state under `dist/dashboard/`. -Recorded with [VHS](https://github.com/charmbracelet/vhs) from `media/demo.tape` against a live E2B control plane (`vhs media/demo.tape`). +Run these commands in the lower-left pane, pausing to explain each result: +```bash +demo create # one logical OpenShell sandbox → supervisor A + worker B +demo links # show task name and all three E2B IDs +demo fence # default deny; root HTTPS succeeds in A and fails in B +ui policy # from the initial dashboard, open the first sandbox +demo policy # allow /usr/bin/curl read-only access to api.github.com +ui logs # from policy, open native logs filtered to sandbox events +demo requests # curl GET 200, curl POST 403, wget denied +ui audit # scroll up one page to inspect the request decisions +ui follow # resume following all events +demo sever # destroy this task's supervisor; observe the worker freeze +demo cleanup # remove the task pair; verify the shared gateway survives ``` -laptop E2B control box E2B agent box - (private) (private, per sandbox) -┌───────────────┐ tunnel 1 ┌──────────────────────────┐ ┌──────────────────────┐ -│ openshell CLI │═════════►│ openshell-gateway │ │ netns: loopback only │ -└───────────────┘ (mTLS) │ (NVIDIA) │ │ openshell-sandbox │ - │ ↕ unix socket │ │ (NVIDIA, patched) │ - │ openshell-driver-e2b │ │ uid 1500, 0 caps, │ - │ (this repo) ─► E2B API │ tunnel 2 │ no_new_privs │ - │ openshell-supervisor │═════════►│ └ the agent │ - │ (NVIDIA, patched) │ └──────────────────────┘ - │ policy + egress proxy │ - │ ─► internet │ - └──────────────────────────┘ + +For manual navigation, click the upper pane. On the dashboard, Tab switches panels and Enter opens a selected sandbox. In a sandbox, `l` opens logs; `s` cycles the log source; `f` follows; Page Up scrolls; Enter expands a selected event; Esc returns. Click the lower-left pane before running the next command. The `ui` shortcuts assume the sequence above and that the demo is the first sandbox; with other tasks present, select the printed task name manually. + +Keep your signed-in **local browser** open to E2B Console → Sandboxes, sorted by Started at descending. After `demo create`, inspect the two new rows: `openshell-supervisor-separated` and `openshell-workload-separated`. Their metadata has the same `openshell.ai/sandbox-name` and different `openshell.ai/role` values. Open a row for E2B CPU/memory monitoring. Return to the native OpenShell view for policy and audit, then show worker B surviving supervisor loss. The Console template filter selects a single template; use the unfiltered list to show both roles together. + +Use a large terminal (about 140 columns × 35 rows or larger). Ctrl-B, D exits the presentation and triggers cleanup of only its task pair. The shared control VM remains available. Before presenting, check the control VM has enough time remaining: this prototype has a one-hour lifetime. To start fresh, stop the existing `npm run connect` process in its terminal, then run `npm run connect -- --new` in that terminal and leave it open. + +```bash +npm run demo:record:dashboard +# → media/openshell-dashboard-demo.mp4, 45 seconds, 1080p ``` -- **Tunnels** go through E2B's HTTPS port address: E2B traffic token + random secret path + E2B certificate verification + OpenShell TLS inside, with a 10 s heartbeat. -- **No control-plane credentials in the agent box.** The E2B API key, gateway JWTs and E2B traffic tokens stay in the control box. The agent box holds only what its own side of the session needs: the per-session TLS key (read by the runtime before the agent starts) and the gateway's public keys. -- **The driver** is Rust on NVIDIA's own crates (pinned to `v0.1.2`) for the boundary protocol. E2B calls go through `driver/e2b-helper.mjs` (E2B's official JS SDK), because E2B has no Rust SDK. +Recording requires VHS, tmux, FFmpeg and Menlo. The tape executes the live scenario, waits for actual results, and exports only after all 14 assertions plus supervisor GET-allow and POST-deny audit records pass. The capture is accelerated; network and provisioning wait times vary. The edited `media/openshell-dashboard-console-demo.mp4` adds a six-second cutaway captured from the signed-in local Chrome Console for the same task. Console captures are manual and are not part of the unattended recorder. To pause a recording for this cutaway, use `DASHBOARD_CAPTURE=1 npm run demo:record:dashboard`; after `dist/dashboard/console-ready` appears, capture the list and run `touch dist/dashboard/console-captured` from another terminal within three minutes. -## Run the demo +Evidence is saved in `dist/dashboard/state.json` and `dist/dashboard/supervisor-events.jsonl`. The separate `demo:studio` view below keeps the raw supervisor feed visible alongside the worker. -Needs `E2B_API_KEY` in `.env` and Node 22+. +## Three-pane live demo and video ```bash -npm install -npm run setup # downloads the openshell CLI + wstunnel into .bin/ -npm run connect -- --new # terminal 1: fresh control box + tunnel (~12 s); keep it running -npm run demo # terminal 2: create → isolation → policy → GET 200 / POST 403 → audit → delete +npm run demo:studio # interactive: walkthrough, supervisor audit, worker heartbeat +npm run demo:record # live run → media/separated-supervisor-demo.mp4 (45 seconds) ``` -Rebuild from source (only after changing the driver or the OpenShell version): +The left pane runs `studio create`, `studio fence`, `studio policy`, `studio requests`, `studio sever`, and `studio cleanup`. The upper-right pane streams audit records directly from supervisor A. The lower-right pane reads the agent heartbeat and kernel process state directly from worker B. `studio sever` destroys only this demo's supervisor VM: the worker survives, its agent freezes, and a root-level HTTPS probe remains blocked by E2B's network fence. + +The recorder requires VHS, tmux, gum, FFmpeg and the Menlo font. It connects the gateway automatically, records the real commands, and accelerates the capture into a 45-second 1080p MP4. Export requires 14 live assertions plus GET-allow and POST-deny records from the supervisor feed. The full-speed capture, assertion report and source audit events are saved under `dist/studio/`. Detach with Ctrl-B, D to close the interactive studio; the launcher cleans up its task pair and any laptop tunnel it started. The shared control VM remains available. No other task pairs are removed. + +## Build and test ```bash -npm run template:builder # once: 8 vCPU Rust builder template -npm run build:binaries # patched NVIDIA binaries + driver → dist/v0.1.2/ (~5 min in E2B) -npm run template:workload # agent box template -npm run template:control # control box template +npm run template:builder # once per E2B team +npm run build:binaries # builds current working-tree sources and runs Rust tests +npm run template:workload +npm run template:supervisor +npm run template:control +npm run typecheck +npm run demo:verify ``` -`build:binaries` backports the eMudhra fix to NVIDIA v0.1.2 with rustls-webpki 0.103.16 and webpki-roots / webpki-root-certs 1.0.10 before compiling. +The templates are `openshell-control-separated`, `openshell-supervisor-separated`, and `openshell-workload-separated`. Original demo template aliases are not changed. The build backports NVIDIA's eMudhra trust fix using rustls-webpki 0.103.16 and webpki-roots / webpki-root-certs 1.0.10. + +For incremental development, `build:binaries -- --keep-builder` retains the compiler VM; `build:binaries -- --reuse --driver-only --keep-builder` rebuilds the driver there. A failed build retains the compiler VM for diagnosis until its one-hour timeout. It contains source and compiler outputs, not the E2B management key. -## Layout +## Implementation boundaries -| Path | What | +| Component | Responsibility | |---|---| -| `driver/src/boundary.rs` | Builds OpenShell's `BoundaryConfig` + `SandboxRuntimeDescriptor` and the outer-fence evidence | -| `driver/src/lifecycle.rs` | Create / delete: E2B box, fence check, papers, runtime, tunnel, supervisor | -| `driver/src/service.rs` | The `ComputeDriver` gRPC service | -| `templates/launch-sandbox.sh` | The fence: netns with only loopback, PID ns, uid 1500, zero caps, policy DNS | -| `patches/landlock-abi2.patch` | **Demo-only** patch, see below | -| `scripts/connect.ts` | One-command control plane + CLI tunnel | -| `scripts/demo.sh` | The demo | -| `scripts/watch.ts` | `npm run watch`: live, color-coded control-plane log (driver, gateway, supervisors) | -| `media/demo.tape` | VHS script for the recording | - -## Limits - -- **Landlock v2 patch.** E2B's guest kernel is Linux 6.1 (Landlock ABI v2). OpenShell v0.1.x requires ABI v3 (kernel ≥ 6.2) so read-only paths can't be truncated. This build accepts v2, so **read-only paths are not protected against truncate**. The real fix is an E2B kernel ≥ 6.2. -- Stop/start (pause/resume) is not implemented. Driver state is in memory only. -- One client certificate is shared by the CLI and supervisors. -- The fence evidence records the template name (`openshell-workload`), not an immutable build ID. -- Create and delete run as driver-owned tasks (safe if the gateway cancels a request); a box whose rollback fails is retried until it's gone. Health monitoring and owner-checked state directories are in. -- Boxes are created with a 1-hour timeout (the maximum on some E2B plans). The control box dies an hour after creation: run `npm run connect -- --new` shortly before use. +| `driver/src/lifecycle.rs` | Create the pair, configure worker isolation, provision task credentials in A, start the supervisor; roll back both on failure | +| `driver/src/boundary.rs` | NVIDIA boundary descriptors, per-session TLS, workload identity and fence evidence | +| `driver/src/health.rs` | Check both VMs and A's process tree; report failed readiness; retry failed rollback deletes | +| `driver/e2b-helper.mjs` | E2B SDK operations; private ingress for both roles, internet disabled for workers | +| `templates/launch-supervisor.py` | A's supervisor and two tunnel clients; stop the group if any process exits | +| `templates/launch-sandbox.sh` | B's OpenShell runtime, isolated namespace, uid 1500 and dropped capabilities | +| `scripts/demo-separated.ts` | Real E2B/OpenShell assertions, failure drill and pair cleanup | + +The worker keeps OpenShell's Landlock, seccomp, namespace and executable-identity mechanisms. E2B's VM boundary and host networking add independent isolation; this implementation does **not** silently translate filesystem or syscall policy into template scoping. E2B Secrets and request transforms are not integrated in this version. + +## Demo limitations and production work + +- **Landlock ABI v2 patch:** the current worker kernel is Linux 6.1. The existing demo patch accepts ABI v2, which does not protect read-only files against truncate. This is not full OpenShell filesystem-policy equivalence. +- **Shared gateway client certificate:** supervisors still use the demo's CLI mTLS certificate. Per-task launch JWTs and worker TLS sessions are distinct, but A remains trusted with broad gateway access. Production needs scoped gateway authorization and separate client identities; VM separation alone does not solve that. +- **Control-plane persistence:** driver inventory is in memory. Restarting the driver reaps its metadata-tagged task VMs instead of adopting sessions. Stop/start, pause/resume and durable reconciliation are not implemented. +- **Lifetimes:** task VMs and the demo control VM have a one-hour TTL. The logical control plane is shared across tasks, but this demo does not provision an indefinitely available production gateway. Create a fresh control plane shortly before presenting. +- **Network transport:** WSS through E2B's ingress proxy adds hops. Private sandbox-to-sandbox transport and narrower supervisor egress are future improvements. The external worker fence is configured at create time; the boundary evidence records this configuration rather than a platform-signed attestation. +- **Templates:** descriptors currently reference template aliases rather than immutable build IDs. ## License diff --git a/driver/e2b-helper.mjs b/driver/e2b-helper.mjs index 9819167..20b109d 100644 --- a/driver/e2b-helper.mjs +++ b/driver/e2b-helper.mjs @@ -20,7 +20,7 @@ // extend {sandboxId, timeoutMs} → {} // // Every sandbox is created private (allowPublicTraffic: false): its URLs -// require the traffic access token, which only the driver holds. +// require the traffic access token, which the driver delegates only to the paired supervisor. import { Sandbox, CommandExitError } from 'e2b' const TUNNEL_PORT = 9000 @@ -32,10 +32,11 @@ async function readStdin() { } const ops = { - async create({ template, metadata = {}, timeoutMs = 60 * 60_000 }) { + async create({ template, metadata = {}, timeoutMs = 60 * 60_000, allowInternetAccess = false }) { const sbx = await Sandbox.create(template, { metadata, timeoutMs, + allowInternetAccess, network: { allowPublicTraffic: false }, }) return { sandboxId: sbx.sandboxId, trafficAccessToken: sbx.trafficAccessToken, host9000: sbx.getHost(TUNNEL_PORT) } diff --git a/driver/src/boundary.rs b/driver/src/boundary.rs index 29a6c5f..ec4bdab 100644 --- a/driver/src/boundary.rs +++ b/driver/src/boundary.rs @@ -9,7 +9,7 @@ //! ┌───────────────────────┴───────────────────────┐ //! ▼ ▼ //! BoundaryConfig → bootstrap.json SandboxRuntimeDescriptor → runtime-descriptor.json -//! read by openshell-sandbox (agent box) read by openshell-supervisor (control box) +//! read by openshell-sandbox (agent box) read by openshell-supervisor (separate supervisor box) //! "listen on control.sock with this TLS cert, "connect to 127.0.0.1:, trust only //! accept only this gateway's keys" this session's CA, expect this fence" //! @@ -43,6 +43,7 @@ struct E2bOuterFenceEvidence<'a> { network_namespace: &'static str, fenced_interfaces: &'a [String], public_traffic: bool, + host_egress: &'static str, } impl E2bOuterFenceEvidence<'_> { @@ -85,7 +86,7 @@ pub struct E2bBoundarySpec { pub e2b_sandbox_id: String, pub template: String, pub fenced_interfaces: Vec, - /// Local end of tunnel 2 in the control box; the supervisor connects here. + /// Local end of tunnel 2 in the supervisor box; the supervisor connects here. pub supervisor_connect: SocketAddr, pub supervisor_tls: SandboxTlsClientConfig, pub workload_identity: ResolvedWorkloadIdentity, @@ -109,6 +110,7 @@ impl E2bBoundarySpec { network_namespace: "os", fenced_interfaces: &self.fenced_interfaces, public_traffic: false, + host_egress: "E2B allowInternetAccess=false", } .project(&self.generation)?; diff --git a/driver/src/e2b.rs b/driver/src/e2b.rs index 0049e28..5bb2459 100644 --- a/driver/src/e2b.rs +++ b/driver/src/e2b.rs @@ -18,7 +18,7 @@ pub struct E2b { pub helper: PathBuf, } -#[derive(Debug, Deserialize)] +#[derive(Deserialize)] #[serde(rename_all = "camelCase")] pub struct Created { pub sandbox_id: String, @@ -69,10 +69,15 @@ impl E2b { } /// Create a private sandbox from `template`, tagged with `metadata`. - pub async fn create(&self, template: &str, metadata: Value) -> Result { + pub async fn create( + &self, + template: &str, + metadata: Value, + allow_internet: bool, + ) -> Result { self.call( "create", - json!({ "template": template, "metadata": metadata, "timeoutMs": 3_600_000 }), + json!({ "template": template, "metadata": metadata, "timeoutMs": 3_600_000, "allowInternetAccess": allow_internet }), ) .await } diff --git a/driver/src/health.rs b/driver/src/health.rs index b6e9fef..f4823ef 100644 --- a/driver/src/health.rs +++ b/driver/src/health.rs @@ -1,126 +1,72 @@ -//! Health monitor: notices when a running sandbox breaks and tells the gateway. -//! -//! every 5 s did the supervisor or the tunnel client exit? (local, cheap) -//! every 30 s does the E2B box still exist? (one E2B API call for all) -//! retry killing boxes whose create rollback failed -//! -//! A problem flips the sandbox's Ready condition to False with a reason, and -//! the change goes out on the watch stream. Without this, a sandbox whose -//! tunnel died would keep reporting Ready. - -use crate::lifecycle::{self, Running}; +//! Monitor both task microVMs and the remote supervisor process tree. +//! Snapshot before network calls; never hold the sandbox table lock over I/O. +use crate::lifecycle; use crate::service::Shared; use openshell_core::proto::compute::v1::{ WatchSandboxesEvent, WatchSandboxesSandboxEvent, watch_sandboxes_event::Payload, }; -use std::collections::HashSet; use std::time::Duration; use tracing::{info, warn}; -const PROCESS_CHECK: Duration = Duration::from_secs(5); -/// Check E2B every Nth process check (6 × 5 s = 30 s). -const E2B_CHECK_EVERY: u32 = 6; -/// Names for `Running::children`, in the order lifecycle.rs starts them. -const CHILD_NAMES: [&str; 2] = ["tunnel", "supervisor"]; - pub fn spawn(shared: Shared) { tokio::spawn(async move { - let mut tick: u32 = 0; loop { - tokio::time::sleep(PROCESS_CHECK).await; - tick = tick.wrapping_add(1); - let box_check = if tick.is_multiple_of(E2B_CHECK_EVERY) { - retry_leaked(&shared).await; - check_boxes(&shared).await - } else { - None - }; - let mut table = shared.sandboxes.lock().await; - for running in table.values_mut() { - if running.deleting { - continue; // the delete task owns this one now + tokio::time::sleep(Duration::from_secs(10)).await; + let leaked: Vec<_> = shared.leaked.lock().await.iter().cloned().collect(); + for id in leaked { + if shared.cfg.e2b.kill(&id).await.is_ok() { + shared.leaked.lock().await.remove(&id); + info!(e2b = %id, "health: rollback cleaned up"); + } + } + let snapshot: Vec<_> = shared + .sandboxes + .lock() + .await + .iter() + .filter(|(_, r)| !r.deleting) + .map(|(id, r)| (id.clone(), r.e2b_id.clone(), r.supervisor_id.clone())) + .collect(); + if snapshot.is_empty() { + continue; + } + let alive = match shared.cfg.e2b.list_ids(&shared.cfg.owner).await { + Ok(alive) => alive, + Err(error) => { + warn!(%error, "health: E2B list unavailable"); + continue; } - if let Some((reason, message)) = problem(running, box_check.as_ref()) - && lifecycle::mark_not_ready(running, reason, &message) - { - warn!(sandbox = %running.sandbox.name, reason, %message, "health: sandbox not ready"); - let _ = shared.events.send(WatchSandboxesEvent { - payload: Some(Payload::Sandbox(WatchSandboxesSandboxEvent { - sandbox: Some(running.sandbox.clone()), - })), - }); + }; + for (id, worker, supervisor) in snapshot { + let problem = if !alive.contains(&worker) || !alive.contains(&supervisor) { + Some(( + "E2bSandboxGone", + "A member of the task pair expired or was killed", + )) + } else { + match shared.cfg.e2b.run(&supervisor, + "test -f /run/openshell-task/ready && test ! -f /run/openshell-task/failed && kill -0 $(cat /run/openshell-task/launcher.pid) 2>/dev/null", "root").await { + Ok(result) if result.exit_code != 0 => Some(("SupervisorExited", "Supervisor or authenticated transport exited")), + // A transient SDK transport failure is not proof of a dead process. + _ => None, + } + }; + if let Some((reason, message)) = problem { + let mut table = shared.sandboxes.lock().await; + if let Some(running) = table.get_mut(&id) + && !running.deleting + && running.supervisor_id == supervisor + && lifecycle::mark_not_ready(running, reason, message) + { + warn!(sandbox = %running.sandbox.name, reason, message, "health: sandbox not ready"); + let _ = shared.events.send(WatchSandboxesEvent { + payload: Some(Payload::Sandbox(WatchSandboxesSandboxEvent { + sandbox: Some(running.sandbox.clone()), + })), + }); + } } } - drop(table); } }); } - -/// Boxes we knew about BEFORE asking E2B, and the boxes E2B reports alive. -/// Only boxes in the first set can be judged, so a sandbox created while the -/// query was in flight is never mistaken for a dead one. -struct BoxCheck { - known_before: HashSet, - alive: HashSet, -} - -async fn check_boxes(shared: &Shared) -> Option { - let known_before: HashSet = shared - .sandboxes - .lock() - .await - .values() - .map(|r| r.e2b_id.clone()) - .collect(); - if known_before.is_empty() { - return None; - } - // The network call happens outside the lock. - match shared.cfg.e2b.list_ids(&shared.cfg.owner).await { - Ok(alive) => Some(BoxCheck { - known_before, - alive, - }), - Err(e) => { - warn!(error = %e, "health: E2B list failed; skipping box check"); - None - } - } -} - -async fn retry_leaked(shared: &Shared) { - let leaked: Vec = shared.leaked.lock().await.iter().cloned().collect(); - for box_id in leaked { - if shared.cfg.e2b.kill(&box_id).await.is_ok() { - shared.leaked.lock().await.remove(&box_id); - info!(e2b = %box_id, "health: leaked E2B box killed"); - } - } -} - -/// The first thing wrong with this sandbox, if anything. -fn problem(running: &mut Running, box_check: Option<&BoxCheck>) -> Option<(&'static str, String)> { - for (child, name) in running.children.iter_mut().zip(CHILD_NAMES) { - if let Ok(Some(status)) = child.try_wait() { - let reason = if name == "supervisor" { - "SupervisorExited" - } else { - "TunnelExited" - }; - return Some((reason, format!("{name} process exited ({status})"))); - } - } - if let Some(check) = box_check - && check.known_before.contains(&running.e2b_id) - && !check.alive.contains(&running.e2b_id) - { - return Some(( - "E2bSandboxGone", - format!( - "E2B box {} no longer exists (expired or killed)", - running.e2b_id - ), - )); - } - None -} diff --git a/driver/src/lifecycle.rs b/driver/src/lifecycle.rs index 3ab5f6e..26efd55 100644 --- a/driver/src/lifecycle.rs +++ b/driver/src/lifecycle.rs @@ -1,24 +1,7 @@ -//! Create and delete sandboxes. This is where the pieces meet. -//! -//! CREATE, step by step (numbers match the comments below): -//! -//! gateway ── CreateSandbox(spec + `launch_authentication`) ──► driver -//! 1 decode the gateway's signed launch authentication -//! 2 create a private E2B box from the openshell-workload template -//! 3 build the fence (netns with only loopback) and check it -//! 4 make this session's TLS cert and both security papers (NVIDIA code) -//! 5 write bootstrap.json + cert + key into the box, owned by uid 1500 -//! 6 start openshell-sandbox inside the fence -//! 7 start socat + wstunnel in the box (tunnel 2, server side) -//! 8 start the wstunnel client here (tunnel 2, local end on 127.0.0.1:) -//! 9 start openshell-supervisor here, pointed at that port -//! gateway ◄── supervisor connects ── sandbox becomes Ready -//! -//! Secrets and where they live: -//! gateway/sandbox JWTs ........ control box only (supervisor auth bundle, 0600) -//! E2B API key ................. control box only (driver environment) -//! traffic access token ........ control box only (wstunnel client header) -//! agent box ................... session TLS key + gateway PUBLIC keys. No tokens. +//! One task = a supervisor microVM plus a worker microVM. +//! The shared gateway/driver owns both lifecycles; no task processes run here. +//! E2B denies worker egress outside the guest. `OpenShell` keeps its runtime +//! fence and session-authenticated channel inside that boundary. use crate::boundary::{self, E2bBoundarySpec}; use crate::e2b::E2b; @@ -31,9 +14,9 @@ use openshell_sandbox_backend::boundary_protocol::{ }; use rand::RngCore; use std::collections::HashMap; -use std::net::{SocketAddr, TcpListener}; +use std::net::SocketAddr; use std::path::{Path, PathBuf}; -use tokio::process::{Child, Command}; + use tracing::{info, warn}; const WORKLOAD_UID: u32 = 1500; @@ -41,28 +24,28 @@ const WORKLOAD_UID: u32 = 1500; const TUNNEL_PORT: u16 = 9000; /// socat inside the agent box: TCP 127.0.0.1:7000 → control.sock. const SOCAT_PORT: u16 = 7000; +const DIR: &str = "/run/openshell-task"; /// Everything the driver needs to know about its surroundings. #[derive(Clone, Debug)] pub struct Config { pub template: String, pub e2b: E2b, - pub supervisor_bin: PathBuf, - pub wstunnel_bin: PathBuf, + pub supervisor_template: String, + pub gateway_tunnel: PathBuf, pub gateway_endpoint: String, /// Client cert the supervisor uses to reach the gateway with mutual TLS. pub gateway_ca: PathBuf, pub gateway_cert: PathBuf, pub gateway_key: PathBuf, - pub state_dir: PathBuf, pub owner: String, } -/// A running sandbox: its `OpenShell` record plus the local helper processes. +/// Both microVMs belong to exactly one `OpenShell` sandbox. pub struct Running { pub sandbox: DriverSandbox, pub e2b_id: String, - pub children: Vec, + pub supervisor_id: String, /// A delete is in progress; the record stays visible until E2B confirms. pub deleting: bool, } @@ -71,14 +54,14 @@ pub struct Running { /// the caller can keep retrying that kill instead of losing track of it. pub struct CreateError { pub message: String, - pub leaked_box: Option, + pub leaked_boxes: Vec, } impl From for CreateError { fn from(message: String) -> Self { Self { message, - leaked_box: None, + leaked_boxes: Vec::new(), } } } @@ -114,11 +97,6 @@ fn status(message: &str, ready: bool, e2b_id: &str, reason: &str) -> DriverSandb } } -fn free_local_port() -> std::io::Result { - // Ask the OS for a free port, then release it for wstunnel to bind. - TcpListener::bind("127.0.0.1:0")?.local_addr() -} - fn random_hex(bytes: usize) -> String { let mut buf = vec![0u8; bytes]; rand::rng().fill_bytes(&mut buf); @@ -168,31 +146,6 @@ pub fn private_dir(path: &Path) -> Result<(), String> { } } -async fn write_private(path: &Path, bytes: &[u8]) -> Result<(), String> { - use std::os::unix::fs::OpenOptionsExt; - let shown = path.display().to_string(); - let path = path.to_path_buf(); - let bytes = bytes.to_vec(); - tokio::task::spawn_blocking(move || { - use std::io::Write; - // 0600 from the moment the file exists, and never follow a symlink - // someone planted where a secret should go (O_NOFOLLOW). - let mut f = std::fs::OpenOptions::new() - .create(true) - .truncate(true) - .write(true) - .mode(0o600) - .custom_flags(libc::O_NOFOLLOW) - .open(&path)?; - // An existing file keeps its old mode on open; force 0600 either way. - f.set_permissions(std::os::unix::fs::PermissionsExt::from_mode(0o600))?; - f.write_all(&bytes) - }) - .await - .map_err(|e| e.to_string())? - .map_err(|e| format!("write {shown}: {e}")) -} - pub async fn create(cfg: &Config, sandbox: &DriverSandbox) -> Result { let spec = sandbox .spec @@ -208,48 +161,70 @@ pub async fn create(cfg: &Config, sandbox: &DriverSandbox) -> Result Ok(Running { - sandbox: DriverSandbox { - status: Some(status( - "OpenShell runtime and supervisor started", - true, - &e2b_id, - "BackendReady", - )), - ..sandbox.clone() - }, - e2b_id, - children, - deleting: false, - }), - Err(message) => { - warn!(e2b = %e2b_id, error = %message, "create failed, deleting E2B box"); - let leaked_box = match cfg.e2b.kill(&e2b_id).await { - Ok(()) => None, - Err(kill_error) => { - tracing::error!(e2b = %e2b_id, error = %kill_error, "rollback failed; will retry the kill"); - Some(e2b_id) - } - }; - Err(CreateError { - message, - leaked_box, - }) + let metadata = |role: &str| { + serde_json::json!({ + "openshell.ai/managed-by": "openshell-e2b", + "openshell.ai/driver-owner": cfg.owner, + "openshell.ai/sandbox-id": sandbox.id, + "openshell.ai/sandbox-name": sandbox.name, + "openshell.ai/role": role, + }) + }; + // Create B first, then A. Every failure rolls back everything allocated. + let worker = cfg + .e2b + .create(&cfg.template, metadata("worker"), false) + .await?; + let supervisor = match cfg + .e2b + .create(&cfg.supervisor_template, metadata("supervisor"), true) + .await + { + Ok(created) => created, + Err(message) => return Err(rollback(cfg, message, &[&worker.sandbox_id]).await), + }; + info!(sandbox = %sandbox.name, worker = %worker.sandbox_id, supervisor = %supervisor.sandbox_id, "E2B task pair created"); + if let Err(message) = provision_and_start( + cfg, + sandbox, + &auth, + session_id, + &generation, + &worker, + &supervisor, + ) + .await + { + return Err(rollback(cfg, message, &[&supervisor.sandbox_id, &worker.sandbox_id]).await); + } + Ok(Running { + sandbox: DriverSandbox { + status: Some(status( + "Separate supervisor and worker started", + true, + &worker.sandbox_id, + "BackendReady", + )), + ..sandbox.clone() + }, + e2b_id: worker.sandbox_id, + supervisor_id: supervisor.sandbox_id, + deleting: false, + }) +} + +async fn rollback(cfg: &Config, message: String, ids: &[&str]) -> CreateError { + let mut leaked_boxes = Vec::new(); + for id in ids { + if let Err(error) = cfg.e2b.kill(id).await { + warn!(e2b = %id, %error, "rollback failed; health monitor will retry"); + leaked_boxes.push((*id).to_owned()); } } + CreateError { + message, + leaked_boxes, + } } async fn provision_and_start( @@ -259,7 +234,8 @@ async fn provision_and_start( session_id: openshell_core::SandboxSessionId, generation: &str, boxed: &crate::e2b::Created, -) -> Result, String> { + supervisor_box: &crate::e2b::Created, +) -> Result<(), String> { let e2b_id = boxed.sandbox_id.as_str(); // 3. Build the fence now (idempotent; launch-sandbox.sh checks it again) @@ -292,7 +268,7 @@ async fn provision_and_start( }) }) .collect::, String>>()?; - let supervisor_connect = free_local_port().map_err(|e| e.to_string())?; + let supervisor_connect = SocketAddr::from(([127, 0, 0, 1], 17000)); let identity = ResolvedWorkloadIdentity::new( WORKLOAD_UID, WORKLOAD_UID, @@ -379,83 +355,106 @@ async fn provision_and_start( ) .await?; - // Local state for this sandbox: supervisor papers (0600) and sockets. - let dir = cfg.state_dir.join(&sandbox.id); - private_dir(&dir)?; - private_dir(&dir.join("proxy-tls"))?; + // All task-specific supervisor state is written into A, never the control VM. + let supervisor_id = &supervisor_box.sandbox_id; + cfg.e2b + .run_ok( + supervisor_id, + "install -d -m 0700 /run/openshell-task /run/openshell-task/proxy-tls /sandbox", + "root", + ) + .await?; let descriptor = papers .runtime_descriptor .backend_descriptor() .map_err(|e| e.to_string())?; - let descriptor_path = dir.join("runtime-descriptor.json"); - write_private(&descriptor_path, &descriptor.payload).await?; - let auth_path = dir.join("auth.json"); let auth_bundle = serde_json::to_vec(&auth.supervisor).map_err(|e| e.to_string())?; - write_private(&auth_path, &auth_bundle).await?; - - // 8. Tunnel 2, local end. The traffic token rides in a header only we have. - // 10 s heartbeat: E2B's proxy drops silent connections after about a minute. - let wstunnel = Command::new(&cfg.wstunnel_bin) - .args([ - "client", - "-L", - &format!("tcp://{supervisor_connect}:127.0.0.1:{SOCAT_PORT}"), - "--http-upgrade-path-prefix", - &path_secret, - "-H", - &format!("e2b-traffic-access-token: {}", boxed.traffic_access_token), - "--websocket-ping-frequency", - "10s", - // Verify E2B's TLS certificate. wstunnel skips this by default, - // which would let an interceptor capture the token and path. - "--tls-verify-certificate", - &format!("wss://{}", boxed.host9000), - ]) - .stdout(std::fs::File::create(dir.join("wstunnel.log")).map_err(|e| e.to_string())?) - .stderr(std::fs::File::create(dir.join("wstunnel.err.log")).map_err(|e| e.to_string())?) - .kill_on_drop(true) - .spawn() - .map_err(|e| format!("start wstunnel client: {e}"))?; - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - - // 9. NVIDIA's supervisor: policy engine + egress proxy for this sandbox. + for (name, bytes) in [ + ("runtime-descriptor.json", descriptor.payload), + ("auth.json", auth_bundle), + ( + "gateway-ca.crt", + tokio::fs::read(&cfg.gateway_ca) + .await + .map_err(|e| e.to_string())?, + ), + ( + "gateway-client.crt", + tokio::fs::read(&cfg.gateway_cert) + .await + .map_err(|e| e.to_string())?, + ), + ( + "gateway-client.key", + tokio::fs::read(&cfg.gateway_key) + .await + .map_err(|e| e.to_string())?, + ), + ] { + cfg.e2b + .write( + supervisor_id, + &format!("{DIR}/{name}"), + &bytes, + "root", + Some(0o600), + ) + .await?; + } + let gateway: serde_json::Value = serde_json::from_slice( + &tokio::fs::read(&cfg.gateway_tunnel) + .await + .map_err(|e| e.to_string())?, + ) + .map_err(|e| format!("gateway tunnel config: {e}"))?; let main_process_spec = env::MainProcessConfig::encode_driver_spec(sandbox.spec.as_ref()) .map_err(|e| e.to_string())?; - let supervisor = Command::new(&cfg.supervisor_bin) - .arg("--backend-descriptor-file") - .arg(&descriptor_path) - .arg("--auth-bundle-file") - .arg(&auth_path) - .args(["--workdir", "/sandbox"]) - .env_clear() - .env("PATH", "/usr/local/bin:/usr/bin:/bin") - .env( + let environment = HashMap::from([ + ("PATH", "/usr/local/bin:/usr/bin:/bin".to_owned()), + ( env::ADMITTED_ISOLATION_BACKEND, - openshell_sandbox_backend::BACKEND_NAME, - ) - .env(env::MAIN_PROCESS_SPEC, main_process_spec) - .env(env::ENDPOINT, &cfg.gateway_endpoint) - .env(env::SANDBOX_ID, &sandbox.id) - .env(env::SANDBOX, &sandbox.name) - .env(env::SSH_SOCKET_PATH, dir.join("ssh.sock")) - .env(env::PROXY_TLS_DIR, dir.join("proxy-tls")) - .env( + openshell_sandbox_backend::BACKEND_NAME.to_owned(), + ), + (env::MAIN_PROCESS_SPEC, main_process_spec), + (env::ENDPOINT, cfg.gateway_endpoint.clone()), + (env::SANDBOX_ID, sandbox.id.clone()), + (env::SANDBOX, sandbox.name.clone()), + (env::SSH_SOCKET_PATH, format!("{DIR}/ssh.sock")), + (env::PROXY_TLS_DIR, format!("{DIR}/proxy-tls")), + ( env::NETWORK_RUNTIME_CAPABILITIES, - env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY, + env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY.to_owned(), + ), + (env::LOG_LEVEL, "info".to_owned()), + (env::TELEMETRY_ENABLED, "false".to_owned()), + (env::TLS_CA, format!("{DIR}/gateway-ca.crt")), + (env::TLS_CERT, format!("{DIR}/gateway-client.crt")), + (env::TLS_KEY, format!("{DIR}/gateway-client.key")), + ]); + let config = serde_json::json!({ + "worker": {"host": boxed.host9000, "token": boxed.traffic_access_token, "path": path_secret}, + "gateway": gateway, + "environment": environment, + }); + cfg.e2b + .write( + supervisor_id, + &format!("{DIR}/config.json"), + &serde_json::to_vec(&config).map_err(|e| e.to_string())?, + "root", + Some(0o600), ) - .env(env::LOG_LEVEL, "info") - .env(env::TELEMETRY_ENABLED, "false") - .env(env::TLS_CA, &cfg.gateway_ca) - .env(env::TLS_CERT, &cfg.gateway_cert) - .env(env::TLS_KEY, &cfg.gateway_key) - .stdout(std::fs::File::create(dir.join("supervisor.log")).map_err(|e| e.to_string())?) - .stderr(std::fs::File::create(dir.join("supervisor.err.log")).map_err(|e| e.to_string())?) - .kill_on_drop(true) - .spawn() - .map_err(|e| format!("start supervisor: {e}"))?; - info!(sandbox = %sandbox.name, e2b = %e2b_id, port = %supervisor_connect, "supervisor started"); - - Ok(vec![wstunnel, supervisor]) + .await?; + cfg.e2b + .run_background( + supervisor_id, + "exec /opt/openshell/launch-supervisor.py > /run/openshell-task/launcher.log 2>&1", + "root", + ) + .await?; + cfg.e2b.run_ok(supervisor_id, "for i in $(seq 100); do test -f /run/openshell-task/ready && exit 0; test -f /run/openshell-task/failed && exit 1; sleep 0.2; done; exit 1", "root").await?; + info!(sandbox = %sandbox.name, worker = %e2b_id, supervisor = %supervisor_id, "separate supervisor started"); + Ok(()) } /// User environment for the agent, minus names `OpenShell` reserves for itself @@ -474,20 +473,20 @@ fn child_environment(sandbox: &DriverSandbox) -> HashMap { vars } -/// Stop the local processes and kill the E2B box. Returns an error if E2B -/// didn't confirm, so the caller keeps the record and the delete can be retried. +/// Attempt both deletes even if one fails. Retain the record for safe retry. pub async fn delete( cfg: &Config, sandbox_id: &str, - e2b_id: &str, - mut children: Vec, + worker: &str, + supervisor: &str, ) -> Result<(), String> { - for child in &mut children { - let _ = child.kill().await; + let results: [Result<(), String>; 2] = + tokio::join!(cfg.e2b.kill(supervisor), cfg.e2b.kill(worker)).into(); + let failures: Vec = results.into_iter().filter_map(Result::err).collect(); + if !failures.is_empty() { + return Err(failures.join("; ")); } - cfg.e2b.kill(e2b_id).await?; - let _ = tokio::fs::remove_dir_all(cfg.state_dir.join(sandbox_id)).await; - info!(sandbox_id, e2b = %e2b_id, "deleted"); + info!(sandbox_id, worker, supervisor, "task pair deleted"); Ok(()) } @@ -575,4 +574,74 @@ mod tests { let _ = std::fs::remove_file(&link); let _ = std::fs::remove_dir_all(&target); } + fn fake_config( + name: &str, + ) -> Result<(super::Config, std::path::PathBuf), Box> { + let dir = scratch(name); + std::fs::create_dir(&dir)?; + let helper = dir.join("helper.py"); + std::fs::write( + &helper, + r#" +import json, pathlib, sys +req = json.load(sys.stdin) +root = pathlib.Path(__file__).parent +with (root / 'calls.jsonl').open('a') as f: + f.write(json.dumps({'op': sys.argv[1], **req}) + '\n') +if req.get('sandboxId') == 'unreachable': + print(json.dumps({'ok': False, 'error': 'injected API failure'})) +else: + print(json.dumps({'ok': True, 'killed': True})) +"#, + )?; + let cfg = super::Config { + template: "worker".into(), + supervisor_template: "supervisor".into(), + e2b: crate::e2b::E2b { + node: "python3".into(), + helper, + }, + gateway_tunnel: dir.join("gateway.json"), + gateway_endpoint: "https://localhost:17670".into(), + gateway_ca: dir.join("ca"), + gateway_cert: dir.join("cert"), + gateway_key: dir.join("key"), + owner: "test-owner".into(), + }; + Ok((cfg, dir)) + } + + #[tokio::test] + async fn delete_attempts_worker_even_when_supervisor_delete_fails() + -> Result<(), Box> { + let (cfg, dir) = fake_config("delete-pair")?; + assert!( + super::delete(&cfg, "task", "worker", "unreachable") + .await + .is_err() + ); + let calls = std::fs::read_to_string(dir.join("calls.jsonl"))?; + assert_eq!(calls.lines().count(), 2); + assert!(calls.contains("worker") && calls.contains("unreachable")); + std::fs::remove_dir_all(dir)?; + Ok(()) + } + + #[tokio::test] + async fn rollback_tracks_only_failed_deletes_for_retry() + -> Result<(), Box> { + let (cfg, dir) = fake_config("rollback-pair")?; + let result = + super::rollback(&cfg, "provision failed".into(), &["unreachable", "worker"]).await; + assert_eq!(result.message, "provision failed"); + assert_eq!(result.leaked_boxes, vec!["unreachable"]); + assert_eq!( + std::fs::read_to_string(dir.join("calls.jsonl"))? + .lines() + .count(), + 2 + ); + std::fs::remove_dir_all(dir)?; + Ok(()) + } } diff --git a/driver/src/main.rs b/driver/src/main.rs index 2840593..16f610b 100644 --- a/driver/src/main.rs +++ b/driver/src/main.rs @@ -61,11 +61,11 @@ struct Args { #[arg(long, env = "OPENSHELL_E2B_NODE", default_value = "node")] node: PathBuf, - /// NVIDIA's supervisor (patched build) and the tunnel client, on this machine. - #[arg(long, env = "OPENSHELL_E2B_SUPERVISOR")] - supervisor: PathBuf, - #[arg(long, env = "OPENSHELL_E2B_WSTUNNEL")] - wstunnel: PathBuf, + /// Dedicated per-task supervisor template and authenticated gateway tunnel. + #[arg(long, default_value = "openshell-supervisor-separated")] + supervisor_template: String, + #[arg(long)] + gateway_tunnel: PathBuf, /// How supervisors reach the gateway, and the client certificate they use. #[arg( @@ -81,7 +81,7 @@ struct Args { #[arg(long, env = "OPENSHELL_E2B_GATEWAY_KEY")] gateway_key: PathBuf, - /// Per-sandbox local state (supervisor papers, logs). + /// Private control-plane state directory. #[arg( long, env = "OPENSHELL_E2B_STATE_DIR", @@ -113,23 +113,32 @@ async fn main() -> Result<()> { // Serve NVIDIA's ComputeDriver gRPC interface (defined in their // compute_driver.proto) with our implementation in service.rs. // Runs until SIGINT or SIGTERM. On either, the server stops and the - // sandbox table is dropped, which kills local supervisors and tunnels. + // shutdown kills both members of every tracked task pair. let config = lifecycle::Config { template: args.template, e2b: e2b::E2b { node: args.node, helper: args.helper, }, - supervisor_bin: args.supervisor, - wstunnel_bin: args.wstunnel, + supervisor_template: args.supervisor_template, + gateway_tunnel: args.gateway_tunnel, gateway_endpoint: args.gateway_endpoint, gateway_ca: args.gateway_ca, gateway_cert: args.gateway_cert, gateway_key: args.gateway_key, - state_dir: args.state_dir, owner: args.owner, }; + // In-memory demo state cannot adopt old sessions. Reap only this control + // plane's task VMs on restart; metadata prevents touching another gateway. + for id in config + .e2b + .list_ids(&config.owner) + .await + .map_err(|e| miette::miette!(e))? + { + config.e2b.kill(&id).await.map_err(|e| miette::miette!(e))?; + } let driver = service::E2bDriver::new(config); let shared = driver.shared(); let server = tonic::transport::Server::builder() @@ -151,18 +160,17 @@ async fn main() -> Result<()> { } } -/// Stop every local helper process and kill every E2B box this driver owns. +/// Kill both members of every task pair this driver owns. /// State is in memory only, so boxes left behind would be orphans. async fn shutdown(shared: service::Shared) { let running: Vec<_> = shared.sandboxes.lock().await.drain().collect(); let leaked: Vec<_> = shared.leaked.lock().await.drain().collect(); let mut kills = Vec::new(); - for (_, mut sandbox) in running { - for child in &mut sandbox.children { - let _ = child.start_kill(); + for (_, sandbox) in running { + for id in [sandbox.supervisor_id, sandbox.e2b_id] { + let e2b = shared.cfg.e2b.clone(); + kills.push(tokio::spawn(async move { e2b.kill(&id).await })); } - let e2b = shared.cfg.e2b.clone(); - kills.push(tokio::spawn(async move { e2b.kill(&sandbox.e2b_id).await })); } for box_id in leaked { let e2b = shared.cfg.e2b.clone(); diff --git a/driver/src/service.rs b/driver/src/service.rs index dc78e41..18ef197 100644 --- a/driver/src/service.rs +++ b/driver/src/service.rs @@ -245,9 +245,7 @@ impl ComputeDriver for E2bDriver { Ok(runtime_identity) } Err(error) => { - if let Some(box_id) = error.leaked_box { - shared.leaked.lock().await.insert(box_id); - } + shared.leaked.lock().await.extend(error.leaked_boxes); Err(error.message) } }; @@ -286,7 +284,7 @@ impl ComputeDriver for E2bDriver { let id = request.into_inner().sandbox_id; // Mark the record as deleting (it stays in Get/List/Watch) and take // what the cleanup needs. A second delete meanwhile is told to wait. - let (e2b_id, children) = { + let (e2b_id, supervisor_id) = { let mut table = self.sandboxes.lock().await; let Some(running) = table.get_mut(&id) else { return Ok(Response::new(DeleteSandboxResponse { deleted: false })); @@ -302,17 +300,14 @@ impl ComputeDriver for E2bDriver { .announce(Payload::Sandbox(WatchSandboxesSandboxEvent { sandbox: Some(running.sandbox.clone()), })); - let parts = ( - running.e2b_id.clone(), - std::mem::take(&mut running.children), - ); + let parts = (running.e2b_id.clone(), running.supervisor_id.clone()); drop(table); // announced under the lock; released right after parts }; // Driver-owned task: cancellation of this request can't abandon it. let shared = self.shared.clone(); let task = tokio::spawn(async move { - let result = lifecycle::delete(&shared.cfg, &id, &e2b_id, children).await; + let result = lifecycle::delete(&shared.cfg, &id, &e2b_id, &supervisor_id).await; let mut table = shared.sandboxes.lock().await; match &result { // Only now, after E2B confirmed, the record goes away. diff --git a/media/dashboard-start.sh b/media/dashboard-start.sh new file mode 100755 index 0000000..9b00e72 --- /dev/null +++ b/media/dashboard-start.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")/.." +export STUDIO_DIR=dist/dashboard STUDIO_NAME_PREFIX=gh-demo STUDIO_COMPACT=1 STUDIO_DEMO_DELAY=2500 +SOCKET=openshell-dashboard +tmux -L "$SOCKET" -f media/tmux.conf new-session -d -s dashboard -x "$(tput cols)" -y "$(tput lines)" 'env XDG_CONFIG_HOME="$PWD/.openshell-separated" .bin/openshell term --theme dark' +tmux -L "$SOCKET" split-window -v -t dashboard:0.0 -l 10 'bash --rcfile media/dashboard.bashrc -i' +tmux -L "$SOCKET" split-window -h -t dashboard:0.1 -l 39% 'node --import tsx --env-file=.env scripts/studio.ts worker-feed' +tmux -L "$SOCKET" set-option -g mouse on +tmux -L "$SOCKET" select-pane -t dashboard:0.0 -T 'OPENSHELL · native policy and audit dashboard' +tmux -L "$SOCKET" select-pane -t dashboard:0.1 -T 'DEMO CONTROLS · laptop' +tmux -L "$SOCKET" select-pane -t dashboard:0.2 -T 'WORKER B · independent live observation' +tmux -L "$SOCKET" select-pane -t dashboard:0.1 +exec tmux -L "$SOCKET" attach -t dashboard diff --git a/media/dashboard.bashrc b/media/dashboard.bashrc new file mode 100644 index 0000000..39f3b63 --- /dev/null +++ b/media/dashboard.bashrc @@ -0,0 +1,36 @@ +# Repo-local interactive helpers; no shell profile changes. +export STUDIO_DIR=dist/dashboard STUDIO_NAME_PREFIX=gh-demo STUDIO_COMPACT=1 STUDIO_DEMO_DELAY=2500 +demo() { + clear + if [ "${1:-}" = links ]; then + node --input-type=module -e 'import {readFileSync} from "node:fs"; const s=JSON.parse(readFileSync("dist/dashboard/state.json")); console.log(`Task: ${s.name}\nA / supervisor: ${s.supervisor}\nB / worker: ${s.worker}\nControl: ${s.control}\n\nE2B Console → Sandboxes → sort Started at newest first.\nThe two latest rows are the task pair. Inspect their role metadata.`);' + elif [ "${1:-}" = ready ]; then + touch dist/dashboard/console-captured + else + node --import tsx --env-file=.env scripts/studio.ts "$@" + fi +} +# The view commands assume the starting view named in README.md. +ui() { + case "$1" in + policy) tmux -L openshell-dashboard send-keys -t dashboard:0.0 Tab Tab Enter ;; + logs) tmux -L openshell-dashboard send-keys -t dashboard:0.0 l s s ;; + audit) tmux -L openshell-dashboard send-keys -t dashboard:0.0 PageUp Down Down Down Down Down ;; + follow) tmux -L openshell-dashboard send-keys -t dashboard:0.0 s G ;; + home) tmux -L openshell-dashboard send-keys -t dashboard:0.0 Escape; sleep 0.15; tmux -L openshell-dashboard send-keys -t dashboard:0.0 Escape ;; + esac +} +console_capture() { + if [ "${DASHBOARD_CAPTURE:-0}" = 1 ]; then + touch dist/dashboard/console-ready + while [ ! -f dist/dashboard/console-captured ]; do sleep 1; done + fi + echo CONSOLE_READY +} +PS1='\[\e[38;5;117m\]demo\[\e[0m\] $ ' +set +m +clear +printf '\033[1;94mOpenShell + E2B · supervisor and worker in separate VMs\033[0m\n\n' +printf 'demo create → fence → policy → requests → sever → cleanup\n' +printf 'Click the native UI above: Tab selects a panel; Enter opens a sandbox.\n' +printf 'Inside a sandbox: [l] logs [Esc] policy/back. Ctrl-B, D exits.\n' diff --git a/media/dashboard.tape b/media/dashboard.tape new file mode 100644 index 0000000..b88c4f2 --- /dev/null +++ b/media/dashboard.tape @@ -0,0 +1,78 @@ +Output dist/dashboard/full-speed.mp4 +Require tmux +Set Shell "bash" +Set FontFamily "Menlo" +Set FontSize 20 +Set LineHeight 1.1 +Set Width 1920 +Set Height 1080 +Set Padding 18 +Set Margin 12 +Set MarginFill "#11111b" +Set BorderRadius 10 +Set WindowBar Colorful +Set Theme "Catppuccin Mocha" +Set TypingSpeed 12ms +Set WaitTimeout 180s +Set Framerate 30 +Hide +Type "bash media/dashboard-start.sh" +Enter +Sleep 3s +Show +Sleep 3s +Type "demo create" +Enter +Wait+Screen /PAIR READY/ +Sleep 3s +Type "demo fence" +Enter +Wait+Screen /FENCE VERIFIED/ +Sleep 3s +# Set DASHBOARD_CAPTURE=1 to pause for an authenticated Console cutaway. +# Continue with: touch dist/dashboard/console-captured +Hide +Type "console_capture" +Enter +Wait+Screen /CONSOLE_READY/ +Type "ui policy" +Enter +Sleep 2s +Show +Type "demo policy" +Enter +Wait+Screen /POLICY ACTIVE/ +Sleep 5s +Hide +Type "ui logs" +Enter +Sleep 1s +Show +Type "demo requests" +Enter +Wait+Screen /REQUESTS VERIFIED/ +Hide +Type "ui audit" +Enter +Sleep 1s +Show +Sleep 7s +Hide +Type "ui follow" +Enter +Sleep 1s +Show +Type "demo sever" +Enter +Wait+Screen /FAIL-CLOSED VERIFIED/ +Sleep 7s +Type "demo cleanup" +Enter +Wait+Screen /CLEANUP VERIFIED/ +Sleep 3s +Hide +Type "ui home" +Enter +Sleep 1s +Show +Sleep 3s diff --git a/media/openshell-dashboard-console-demo.mp4 b/media/openshell-dashboard-console-demo.mp4 new file mode 100644 index 0000000..d25f12a Binary files /dev/null and b/media/openshell-dashboard-console-demo.mp4 differ diff --git a/media/openshell-dashboard-demo.mp4 b/media/openshell-dashboard-demo.mp4 new file mode 100644 index 0000000..72d6379 Binary files /dev/null and b/media/openshell-dashboard-demo.mp4 differ diff --git a/media/separated-supervisor-demo.mp4 b/media/separated-supervisor-demo.mp4 new file mode 100644 index 0000000..56f3400 Binary files /dev/null and b/media/separated-supervisor-demo.mp4 differ diff --git a/media/separated-supervisor.tape b/media/separated-supervisor.tape new file mode 100644 index 0000000..3091999 --- /dev/null +++ b/media/separated-supervisor.tape @@ -0,0 +1,86 @@ +# Run via npm run demo:record. The wrapper connects, verifies, and cleans up. +Output dist/studio/full-speed.mp4 +Require tmux +Require gum +Set Shell "bash" +Set FontFamily "Menlo" +Set FontSize 22 +Set LineHeight 1.15 +Set Width 1920 +Set Height 1080 +Set Padding 20 +Set Margin 20 +Set MarginFill "#11111b" +Set BorderRadius 12 +Set WindowBar Colorful +Set Theme "Catppuccin Mocha" +Set TypingSpeed 15ms +Set WaitTimeout 120s +Set Framerate 30 + +Hide +Type "bash media/studio-start.sh" +Enter +Sleep 3s +Type `card "OpenShell + E2B" "The supervisor gets its own microVM." "The agent gets another." "" "Live policy decisions above right." "Live worker heartbeat below right."` +Enter +Sleep 500ms +Show +Sleep 4s + +Type "studio create" +Enter +Wait+Screen /PAIR READY/ +Sleep 3s + +Hide +Type `card "Two independent boundaries" "OpenShell policy in A. E2B network fence around B."` +Enter +Sleep 400ms +Show +Type "studio fence" +Enter +Wait+Screen /FENCE VERIFIED/ +Sleep 4s + +Hide +Type `card "Change policy. Keep the worker running." "Allow curl to read GitHub. Watch A reload the rule."` +Enter +Sleep 400ms +Show +Type "studio policy" +Enter +Wait+Screen /POLICY ACTIVE/ +Sleep 3s +Type "studio requests" +Enter +Wait+Screen /REQUESTS VERIFIED/ +Sleep 5s + +Hide +Type `card "Now pull the supervisor's plug." "The worker VM survives. Does the agent keep running?"` +Enter +Sleep 400ms +Show +Sleep 2s +Type "studio sever" +Enter +Wait+Screen /FAIL-CLOSED VERIFIED/ +Sleep 6s + +Hide +Type `card "Separate machines. One task lifecycle." "Delete the pair; keep the shared gateway."` +Enter +Sleep 400ms +Show +Type "studio cleanup" +Enter +Wait+Screen /CLEANUP VERIFIED/ +Sleep 3s + +Hide +Type `card "The supervisor is outside the worker." "✓ policy decisions in a separate microVM" "✓ worker root cannot bypass the network fence" "✓ supervisor loss freezes the agent" "" "Live demo · OpenShell v0.1.2" "Landlock v2: read-only file truncation is not blocked." "Shared gateway certificate remains a demo limitation."` +Enter +Sleep 400ms +Show +Sleep 5s diff --git a/media/studio-start.sh b/media/studio-start.sh new file mode 100755 index 0000000..66537cc --- /dev/null +++ b/media/studio-start.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")/.." +SOCKET=openshell-studio +tmux -L "$SOCKET" -f media/tmux.conf new-session -d -s studio -x "$(tput cols)" -y "$(tput lines)" "bash --rcfile media/studio.bashrc -i" +tmux -L "$SOCKET" split-window -h -t studio:0.0 -l 43% "node --import tsx --env-file=.env scripts/studio.ts supervisor-feed" +tmux -L "$SOCKET" split-window -v -t studio:0.1 -l 45% "node --import tsx --env-file=.env scripts/studio.ts worker-feed" +tmux -L "$SOCKET" select-pane -t studio:0.0 -T "LIVE WALKTHROUGH · laptop" +tmux -L "$SOCKET" select-pane -t studio:0.1 -T "A / SUPERVISOR · remote audit" +tmux -L "$SOCKET" select-pane -t studio:0.2 -T "B / WORKER · live kernel state" +tmux -L "$SOCKET" select-pane -t studio:0.0 +exec tmux -L "$SOCKET" attach -t studio diff --git a/media/studio.bashrc b/media/studio.bashrc new file mode 100644 index 0000000..28681fe --- /dev/null +++ b/media/studio.bashrc @@ -0,0 +1,10 @@ +# Project-local shell for the recorded walkthrough. +studio() { node --import tsx --env-file=.env scripts/studio.ts "$@"; } +card() { media/card.sh "$@"; } +PS1='\[\e[38;5;117m\]demo\[\e[0m\] $ ' +set +m +clear +card "OpenShell · two private E2B microVMs" \ + "A makes the decisions. B runs the agent." \ + "" "Run: studio create → fence → policy → requests → sever → cleanup" \ + "All commands and side panels use live E2B sandboxes." diff --git a/package-lock.json b/package-lock.json index 734b71f..e66a926 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8,6 +8,10 @@ "dependencies": { "e2b": "^2.51.0", "tsx": "^4.23.15" + }, + "devDependencies": { + "@types/node": "26.6.5", + "typescript": "7.0.2" } }, "node_modules/@bufbuild/protobuf": { @@ -463,6 +467,356 @@ "node": ">=18.0.0" } }, + "node_modules/@types/node": { + "version": "26.6.5", + "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.5.tgz", + "integrity": "sha512-uMzsv9stDdofIZBx4UGh4Z9ZgXnGC9seRDscilBcTHAeOFk67MZg/k9bWVfO+bk4qZmG+l52hRnLwUiH2LnJTg==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~8.9.0" + } + }, + "node_modules/@typescript/typescript-aix-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-aix-ppc64/-/typescript-aix-ppc64-7.0.2.tgz", + "integrity": "sha512-MTKKkWB7p/0E9xi1d1tHtZ5PiLkGEMIq88pK2CubZjOsLtYTLqhgIgi6zepFa+9GHZ6h05NMCkQxGKiPXMxXtQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-arm64/-/typescript-darwin-arm64-7.0.2.tgz", + "integrity": "sha512-gowzar9MwS/aRWp6f3a4KUqzRjAZjOsmGNCM6LcTgXum+dBfgsBVMN+AgvOCCbguXyick6LJhpBszxMebJ8syA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-x64/-/typescript-darwin-x64-7.0.2.tgz", + "integrity": "sha512-SZ9xZInqApNlNGc9s0W1VSsktYSOe9cFqNOIqmN1Gs8SmkjKZYFt017G4VwPxASInODuAdbTW7sXiFUf893RgA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-arm64/-/typescript-freebsd-arm64-7.0.2.tgz", + "integrity": "sha512-W5NH4y/J0plIIS5b2xvTEkU7JFxyqdMAOgf+Ilhl0vHQXKO5dZoxd+C/jEtq56c4F3wk71RB4BMRQ2XdI+bwYQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-x64/-/typescript-freebsd-x64-7.0.2.tgz", + "integrity": "sha512-UMGDx5sTpzNw3WiPebH7l90IWfJggEd+egHt/q6p7/Cm3zqoV7VxkGXt+3DxPIw8CcmvAB0j3sVVfbhX+M4Tpw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm/-/typescript-linux-arm-7.0.2.tgz", + "integrity": "sha512-gffT3xPz9sR7j/YJExkyPntrI0P2EP9XbOyWzth2/Gs0RstK+90RBcO0ncXoXy/beYll1SXw846Nf2zdnEz0QQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm64/-/typescript-linux-arm64-7.0.2.tgz", + "integrity": "sha512-Qh4eU4/y3yDjnfjjyPYihMj5/ODIlmt+Bzu17OI+fiSRDW57QmU5SiN63exPRNJPKUzcc1INa1NXdrJ+MqHjUQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-loong64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-loong64/-/typescript-linux-loong64-7.0.2.tgz", + "integrity": "sha512-uEHck9i8hoAzXPiYRib1O7miOnz23SxIeVl6F4LXox+qov1K35jHcEW6VHKvZI+pyvl7fZEP4MCU5LYvIq1GuQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-mips64el": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-mips64el/-/typescript-linux-mips64el-7.0.2.tgz", + "integrity": "sha512-R4KvAMnE43W5Qeqb0Ly56O3mWMWIAgsMyz36DCaycd5nbg/9kzm0liw3JocfRqyJY0KPmzFjbswozXyW0DnIYA==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-ppc64/-/typescript-linux-ppc64-7.0.2.tgz", + "integrity": "sha512-DORx5b3sd/4S7eayxm4FQv+A7CrkUIGRaHiwI8oiHTAI1fAPWhF4J0vAlkC8biAlHSVVwxMQ3tjZ2/DVbnQiiA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-riscv64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-riscv64/-/typescript-linux-riscv64-7.0.2.tgz", + "integrity": "sha512-wf0jqEDOjrPRnKwYRyyJDRo11KMbvMFrU+q4zqKyChODBzvlkbhNQfKvLxQCcwTpdDaXSHZTVuh0JoCrKCUMHQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-s390x": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-s390x/-/typescript-linux-s390x-7.0.2.tgz", + "integrity": "sha512-IkwJc3L7yhytWd/ewjyxNDfOmswCm9GWMJT/ue/dU4aZNbwZeYAetq42VyLmsmSjvoX7z74X6ZaYCtzAr0EuGw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-x64/-/typescript-linux-x64-7.0.2.tgz", + "integrity": "sha512-EYdf2cNg7rgCWJnxCdJ+F3V39O8ihb37eHAu1LK8oAFizgTQbPOK7zHHXbPt8rX24COqODXeI3sIf0fCXG7H/A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-arm64/-/typescript-netbsd-arm64-7.0.2.tgz", + "integrity": "sha512-+polYF4MF04aPpO5FTkHran9yUQDSXqy5GiSDKpsll5jy3l3+g9QLhpf39T+ePtefhXLOGrLl0QIjkQP6VnelA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-x64/-/typescript-netbsd-x64-7.0.2.tgz", + "integrity": "sha512-8YIT0EHM/3dq10ZOVF/A7pc/YSMtbcecct4rWtexrnSCHOPcpC2KTLXfTCR6vDpnSiY12heNb1GiN/wu+T/FyA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-arm64/-/typescript-openbsd-arm64-7.0.2.tgz", + "integrity": "sha512-APT8+ClYnuYm1u9+kgGXoMj2VzWzcymwh2gNSQVySHfkRDGOTVkoWLjCmOQSaO+PoqQ57B0flRp9SA+7GnnkzQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-x64/-/typescript-openbsd-x64-7.0.2.tgz", + "integrity": "sha512-yX7s+Q0Dln0Dt9tEzZsAjXXR/+ytBM7AlglaqyeMPxQszJ1JhlJdZ6jLA+IzldHtflX81em7lDao1xXu+aRRkg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-sunos-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-sunos-x64/-/typescript-sunos-x64-7.0.2.tgz", + "integrity": "sha512-dLJDGaLZ1D4HPQn62u1n8mBDkJREwMsAkCdkwd4Ieqw+x3TUyTsqY0YiBCtE6H6OzzgGk3iuZ3vFWRS+E8/d1g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-arm64/-/typescript-win32-arm64-7.0.2.tgz", + "integrity": "sha512-Gyl1Vy6OsWesLzmq+EP0Fb7b4Nid5232AvcA2SFcdYreldpNtYFFofPjnt62y9hQy7VTaZp65ICJjuAQRaVcIQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-x64/-/typescript-win32-x64-7.0.2.tgz", + "integrity": "sha512-0BQ3HkAHHlKLSp1qRvf3SUhGpGsDuhB/jgFw75guyqbxJqEaS0Cw/VFO8i2nHglJUzQCRtMMR/IBAKE3ETMC4g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" + } + }, "node_modules/balanced-match": { "version": "4.0.4", "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", @@ -737,6 +1091,41 @@ "fsevents": "~2.3.3" } }, + "node_modules/typescript": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz", + "integrity": "sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc" + }, + "engines": { + "node": ">=16.20.0" + }, + "optionalDependencies": { + "@typescript/typescript-aix-ppc64": "7.0.2", + "@typescript/typescript-darwin-arm64": "7.0.2", + "@typescript/typescript-darwin-x64": "7.0.2", + "@typescript/typescript-freebsd-arm64": "7.0.2", + "@typescript/typescript-freebsd-x64": "7.0.2", + "@typescript/typescript-linux-arm": "7.0.2", + "@typescript/typescript-linux-arm64": "7.0.2", + "@typescript/typescript-linux-loong64": "7.0.2", + "@typescript/typescript-linux-mips64el": "7.0.2", + "@typescript/typescript-linux-ppc64": "7.0.2", + "@typescript/typescript-linux-riscv64": "7.0.2", + "@typescript/typescript-linux-s390x": "7.0.2", + "@typescript/typescript-linux-x64": "7.0.2", + "@typescript/typescript-netbsd-arm64": "7.0.2", + "@typescript/typescript-netbsd-x64": "7.0.2", + "@typescript/typescript-openbsd-arm64": "7.0.2", + "@typescript/typescript-openbsd-x64": "7.0.2", + "@typescript/typescript-sunos-x64": "7.0.2", + "@typescript/typescript-win32-arm64": "7.0.2", + "@typescript/typescript-win32-x64": "7.0.2" + } + }, "node_modules/undici": { "version": "7.30.0", "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.0.tgz", @@ -746,6 +1135,13 @@ "node": ">=20.18.1" } }, + "node_modules/undici-types": { + "version": "8.9.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz", + "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==", + "dev": true, + "license": "MIT" + }, "node_modules/undici8": { "name": "undici", "version": "8.10.2", diff --git a/package.json b/package.json index 74b421b..ddaa1a7 100644 --- a/package.json +++ b/package.json @@ -11,12 +11,25 @@ "sbx": "tsx --env-file=.env scripts/sbx.ts", "spike:tunnel": "tsx --env-file=.env spikes/tunnel.ts", "template:control": "tsx --env-file=.env templates/control.ts", - "demo": "scripts/demo.sh", + "demo": "tsx --env-file=.env scripts/demo-separated.ts", "fleet": "scripts/fleet.sh", - "watch": "tsx --env-file=.env scripts/watch.ts" + "watch": "tsx --env-file=.env scripts/watch.ts", + "template:supervisor": "tsx --env-file=.env templates/supervisor.ts", + "demo:verify": "tsx --env-file=.env scripts/demo-separated.ts --fault", + "typecheck": "tsc --noEmit", + "demo:down": "tsx --env-file=.env scripts/demo-down.ts", + "studio": "tsx --env-file=.env scripts/studio.ts", + "demo:studio": "bash scripts/studio.sh", + "demo:record": "bash scripts/studio.sh --record", + "demo:dashboard": "bash scripts/dashboard.sh", + "demo:record:dashboard": "bash scripts/dashboard.sh --record" }, "dependencies": { "e2b": "^2.51.0", "tsx": "^4.23.15" + }, + "devDependencies": { + "@types/node": "26.6.5", + "typescript": "7.0.2" } } diff --git a/scripts/build-binaries.ts b/scripts/build-binaries.ts index 04055d9..458efdd 100644 --- a/scripts/build-binaries.ts +++ b/scripts/build-binaries.ts @@ -1,7 +1,7 @@ // Build every Linux binary the demo needs, in an E2B builder box, into dist//: // // openshell-sandbox NVIDIA's runtime, with patches/landlock-abi2.patch (agent boxes) -// openshell-supervisor NVIDIA's supervisor, same patch (control box) +// openshell-supervisor NVIDIA's supervisor, same patch (supervisor box) // openshell-driver-e2b our driver, from ./driver (control box) // // Usage: npm run build:binaries @@ -12,12 +12,19 @@ import { Sandbox } from 'e2b' const TAG = process.env.OPENSHELL_TAG ?? 'v0.1.2' const NVIDIA_BINS = ['openshell-sandbox', 'openshell-supervisor'] const log = (...a: unknown[]) => console.log(new Date().toISOString().slice(11, 19), ...a) -const stream = { onStdout: (d: string) => process.stdout.write(d), onStderr: (d: string) => process.stderr.write(d) } +const stream = { onStdout: (d: string) => { process.stdout.write(d) }, onStderr: (d: string) => { process.stderr.write(d) } } async function main() { - const sbx = await Sandbox.create('openshell-builder', { timeoutMs: 60 * 60_000, network: { allowPublicTraffic: false } }) + const reuse = process.argv.includes('--reuse') + const driverOnly = process.argv.includes('--driver-only') + const sbx = reuse + ? await Sandbox.connect((await readFile('.builder-separated-id', 'utf8')).trim(), { timeoutMs: 60 * 60_000 }) + : await Sandbox.create('openshell-builder', { timeoutMs: 60 * 60_000, network: { allowPublicTraffic: false } }) + await writeFile('.builder-separated-id', sbx.sandboxId) + let succeeded = false log('builder', sbx.sandboxId) try { + if (!reuse && !driverOnly) { // NVIDIA's code at the pinned tag, with the demo patch applied. await sbx.files.write('/tmp/landlock-abi2.patch', await readFile('patches/landlock-abi2.patch', 'utf8')) await sbx.commands.run( @@ -34,8 +41,10 @@ async function main() { { timeoutMs: 180_000, ...stream }, ) - // Our driver source (committed files only, no target/ or secrets). - const tar = execFileSync('git', ['archive', 'HEAD', 'driver']) + } + // Build the working branch, including edits not yet committed. Explicit + // source paths exclude credentials, recordings and local build outputs. + const tar = execFileSync('tar', ['-cf', '-', 'driver/Cargo.toml', 'driver/Cargo.lock', 'driver/src']) await sbx.files.write('/tmp/driver.tar', tar.buffer.slice(tar.byteOffset, tar.byteOffset + tar.byteLength) as ArrayBuffer) await sbx.commands.run('mkdir -p /tmp/src && tar -xf /tmp/driver.tar -C /tmp/src', { timeoutMs: 60_000 }) @@ -43,16 +52,16 @@ async function main() { // Both builds in one command, in parallel; the box has 8 cores. await sbx.commands.run( // (wait by PID: job specs like %1 need job control, absent in non-interactive shells) - `(cd /tmp/os && cargo build --release ${NVIDIA_BINS.map((b) => `--bin ${b}`).join(' ')}) > /tmp/os.log 2>&1 & p1=$!; ` + - '(cd /tmp/src/driver && cargo build --release) > /tmp/driver.log 2>&1 & p2=$!; ' + - 'wait $p1; a=$?; wait $p2; b=$?; grep -hE "^error|Finished" /tmp/os.log /tmp/driver.log; exit $((a | b))', + (driverOnly ? 'true > /tmp/os.log & p1=$!; ' : `(cd /tmp/os && cargo build --release ${NVIDIA_BINS.map((b) => `--bin ${b}`).join(' ')}) > /tmp/os.log 2>&1 & p1=$!; `) + + '(cd /tmp/src/driver && cargo test --release && cargo build --release) > /tmp/driver.log 2>&1 & p2=$!; ' + + 'wait $p1; a=$?; wait $p2; b=$?; grep -hE "^error|Finished|test result" /tmp/os.log /tmp/driver.log; exit $((a | b))', { timeoutMs: 50 * 60_000, ...stream }, ) log(`builds took ${Math.round((Date.now() - t0) / 1000)}s`) await mkdir(`dist/${TAG}`, { recursive: true }) const outputs = [ - ...NVIDIA_BINS.map((b) => [`/tmp/os/target/release/${b}`, b]), + ...(driverOnly ? [] : NVIDIA_BINS).map((b) => [`/tmp/os/target/release/${b}`, b]), ['/tmp/src/driver/target/release/openshell-driver-e2b', 'openshell-driver-e2b'], ] for (const [remote, name] of outputs) { @@ -60,9 +69,14 @@ async function main() { await writeFile(`dist/${TAG}/${name}`, bytes, { mode: 0o755 }) log(`dist/${TAG}/${name}`, `${(bytes.length / 1e6).toFixed(1)} MB`) } + succeeded = true } finally { - await sbx.kill() - log('builder killed') + if (succeeded && !process.argv.includes('--keep-builder')) { + await sbx.kill() + log('builder killed') + } else { + log('builder retained for diagnosis/rebuild:', sbx.sandboxId, '(1-hour TTL)') + } } } diff --git a/scripts/connect.ts b/scripts/connect.ts index d2350a9..95eb891 100644 --- a/scripts/connect.ts +++ b/scripts/connect.ts @@ -1,15 +1,15 @@ // connect.ts: bring up (or reconnect to) the OpenShell control plane on E2B // and link your laptop's `openshell` CLI to it. One command: // -// npm run connect reuse the control box in .control-id, or create one +// npm run connect reuse the control box in .control-separated-id, or create one // npm run connect -- --new always create a fresh control box // -// laptop E2B control sandbox (private, openshell-control template) +// laptop E2B control sandbox (private, openshell-control-separated template) // ┌───────────────────────────┐ ┌──────────────────────────────────────────────┐ // │ openshell CLI │ │ wstunnel server :9000 (secret path only) │ -// │ → https://127.0.0.1:17690 │ → 127.0.0.1:17670 openshell-gateway │ -// │ wstunnel client :17690 ═══╪══ wss:// ════════╪══► ↕ unix socket │ -// │ + e2b-traffic-access-token header │ openshell-driver-e2b → E2B API (agent boxes) │ +// │ → https://127.0.0.1:17691 │ → 127.0.0.1:17670 openshell-gateway │ +// │ wstunnel client :17691 ═══╪══ wss:// ════════╪══► ↕ unix socket │ +// │ + e2b-traffic-access-token header │ openshell-driver-e2b → E2B API (task pairs) │ // └───────────────────────────┘ └──────────────────────────────────────────────┘ // // Layers of protection (defence in depth; the inner mTLS is what protects the session): @@ -29,12 +29,12 @@ import { Sandbox } from 'e2b' const GATEWAY_PORT = 17670 // gateway inside the control box (localhost only) const TUNNEL_PORT = 9000 // the only port we use through E2B's proxy -const LOCAL_PORT = 17690 // where the CLI connects on the laptop -const GATEWAY_NAME = 'e2b' -const ROOT = '/opt/openshell-e2b' // baked into the openshell-control template +const LOCAL_PORT = 17691 // where the CLI connects on the laptop +const GATEWAY_NAME = 'e2b-separated' +const ROOT = '/opt/openshell-e2b' // baked into the openshell-control-separated template const STATE = '/home/user/openshell' // per-box: certificates, config, logs -const CONFIG = `${process.cwd()}/.openshell` // isolated CLI config on the laptop (gitignored) -const ID_FILE = '.control-id' +const CONFIG = `${process.cwd()}/.openshell-separated` // isolated CLI config on the laptop (gitignored) +const ID_FILE = '.control-separated-id' const log = (...a: unknown[]) => console.log(new Date().toISOString().slice(11, 19), ...a) @@ -51,7 +51,7 @@ async function controlBox(): Promise { } } } - const sbx = await Sandbox.create('openshell-control', { + const sbx = await Sandbox.create('openshell-control-separated', { timeoutMs: 60 * 60_000, network: { allowPublicTraffic: false }, metadata: { 'openshell.ai/role': 'control-plane' }, @@ -131,14 +131,27 @@ async function main() { await sleep(1000) log(`${name}: started`) } + // Persist the transport path: reconnecting a laptop must not invalidate + // the gateway channel of existing task supervisors. + const pathFile = `${STATE}/gateway-path` + const pathSecret = (await exists(pathFile)) + ? (await sbx.files.read(pathFile)).trim() + : randomBytes(16).toString('hex') + await sbx.files.write(pathFile, pathSecret) + await sbx.files.write(`${STATE}/gateway-tunnel.json`, JSON.stringify({ + host: sbx.getHost(TUNNEL_PORT), token, path: pathSecret, + })) + await sbx.commands.run(`chmod 600 ${pathFile} ${STATE}/gateway-tunnel.json`) const restart = process.env.RESTART === '1' - // The driver reads the E2B API key from driver.env. Agent boxes never get it. + await daemon('wstunnel', + `${ROOT}/bin/wstunnel server ws://0.0.0.0:${TUNNEL_PORT} --restrict-to 127.0.0.1:${GATEWAY_PORT} ` + + `--restrict-http-upgrade-path-prefix ${pathSecret}`, restart) await daemon('driver', `sh -c 'set -a; . ${STATE}/driver.env; set +a; exec ${ROOT}/bin/openshell-driver-e2b ` + - `--bind-socket ${STATE}/run/e2b.sock --template openshell-workload ` + + `--bind-socket ${STATE}/run/e2b.sock --template openshell-workload-separated ` + + `--supervisor-template openshell-supervisor-separated --owner ${sbx.sandboxId} ` + `--helper ${ROOT}/helper/e2b-helper.mjs --node ${ROOT}/node/bin/node ` + - `--supervisor ${ROOT}/bin/openshell-supervisor --wstunnel ${ROOT}/bin/wstunnel ` + - `--state-dir ${STATE}/sandboxes ` + + `--gateway-tunnel ${STATE}/gateway-tunnel.json --state-dir ${STATE}/sandboxes ` + `--gateway-ca ${STATE}/pki/ca.crt --gateway-cert ${STATE}/pki/client/tls.crt --gateway-key ${STATE}/pki/client/tls.key'`, restart) await daemon('gateway', @@ -146,12 +159,14 @@ async function main() { `--tls-cert ${STATE}/pki/server/tls.crt --tls-key ${STATE}/pki/server/tls.key --tls-client-ca ${STATE}/pki/ca.crt ` + `--enable-mtls-auth true --compute-driver e2b --compute-driver-socket ${STATE}/run/e2b.sock`, restart) - // Always restart the tunnel server: it must use this run's fresh secret path. - const pathSecret = randomBytes(16).toString('hex') - await daemon('wstunnel', - `${ROOT}/bin/wstunnel server ws://0.0.0.0:${TUNNEL_PORT} --restrict-to 127.0.0.1:${GATEWAY_PORT} ` + - `--restrict-http-upgrade-path-prefix ${pathSecret}`, - true) + + // `gateway add --local` requires the client certificates to exist already. + const mtls = `${CONFIG}/openshell/gateways/${GATEWAY_NAME}/mtls` + await mkdir(mtls, { recursive: true, mode: 0o700 }) + for (const [remote, local] of [['pki/ca.crt', 'ca.crt'], ['pki/client/tls.crt', 'tls.crt'], ['pki/client/tls.key', 'tls.key']]) { + await writeFile(`${mtls}/${local}`, await sbx.files.read(`${STATE}/${remote}`)) + await chmod(`${mtls}/${local}`, 0o600) + } // Register the gateway with the isolated CLI config, unless it already is. const cli = (args: string[]) => @@ -162,16 +177,6 @@ async function main() { log(`CLI: registered gateway '${GATEWAY_NAME}'`) } - // The CLI's client certificate. These files ARE the login: whoever holds - // tls.key can talk to the gateway. Written after `gateway add --local`, - // which copies in the certs of any local OpenShell gateway on this laptop. - const mtls = `${CONFIG}/openshell/gateways/${GATEWAY_NAME}/mtls` - await mkdir(mtls, { recursive: true, mode: 0o700 }) - for (const [remote, local] of [['pki/ca.crt', 'ca.crt'], ['pki/client/tls.crt', 'tls.crt'], ['pki/client/tls.key', 'tls.key']]) { - await writeFile(`${mtls}/${local}`, await sbx.files.read(`${STATE}/${remote}`)) - await chmod(`${mtls}/${local}`, 0o600) - } - // Open the tunnel on the laptop. 10 s heartbeat: spike A showed E2B's // proxy drops connections that stay silent for about a minute. const client = spawn(`${process.cwd()}/.bin/wstunnel`, [ @@ -182,7 +187,11 @@ async function main() { // Verify E2B's TLS certificate; wstunnel skips it by default. '--tls-verify-certificate', `wss://${sbx.getHost(TUNNEL_PORT)}`, - ], { stdio: ['ignore', 'ignore', 'pipe'] }) + ], { + stdio: ['ignore', 'ignore', 'pipe'], + // wstunnel expects a boolean, while many terminal tools set NO_COLOR=1. + env: { ...process.env, NO_COLOR: 'true' }, + }) // Surface tunnel errors (bind failures, auth rejections) instead of hiding them. client.stderr?.on('data', (d) => { const line = String(d).trim() diff --git a/scripts/dashboard.sh b/scripts/dashboard.sh new file mode 100755 index 0000000..7b29a81 --- /dev/null +++ b/scripts/dashboard.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Native OpenShell UI plus live worker telemetry. Owns only its demo task pair. +set -euo pipefail +cd "$(dirname "$0")/.." +export STUDIO_DIR=dist/dashboard STUDIO_NAME_PREFIX=gh-demo STUDIO_COMPACT=1 STUDIO_DEMO_DELAY=2500 +mkdir -p "$STUDIO_DIR" +if tmux -L openshell-dashboard has-session -t dashboard 2>/dev/null; then + echo 'Attach to the existing demo: tmux -L openshell-dashboard attach -t dashboard' >&2 + exit 1 +fi +node --import tsx --env-file=.env scripts/studio.ts reset +rm -f "$STUDIO_DIR/console-ready" "$STUDIO_DIR/console-captured" +connector='' +feed='' +finish() { + result=$? + if [ -f "$STUDIO_DIR/state.json" ]; then + if ! node --import tsx --env-file=.env scripts/studio.ts cleanup >"$STUDIO_DIR/cleanup.log" 2>&1; then + cat "$STUDIO_DIR/cleanup.log" >&2; result=1 + fi + fi + tmux -L openshell-dashboard kill-session -t dashboard 2>/dev/null || true + [ -z "$feed" ] || kill "$feed" 2>/dev/null || true + [ -z "$connector" ] || kill "$connector" 2>/dev/null || true + exit "$result" +} +trap finish EXIT +if ! XDG_CONFIG_HOME="$PWD/.openshell-separated" .bin/openshell status >/dev/null 2>&1; then + node --import tsx --env-file=.env scripts/connect.ts >"$STUDIO_DIR/connect.log" 2>&1 & + connector=$! + ready=0 + for _ in $(seq 90); do + if XDG_CONFIG_HOME="$PWD/.openshell-separated" .bin/openshell status >/dev/null 2>&1; then ready=1; break; fi + kill -0 "$connector" 2>/dev/null || { cat "$STUDIO_DIR/connect.log" >&2; exit 1; } + sleep 1 + done + [ "$ready" = 1 ] || { echo 'Gateway connection timed out.' >&2; exit 1; } +fi +node --import tsx --env-file=.env scripts/studio.ts supervisor-feed >"$STUDIO_DIR/supervisor-feed.log" 2>&1 & +feed=$! +if [ "${1:-}" = '--record' ]; then + vhs media/dashboard.tape + node --input-type=module <<'NODE' +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +const state = JSON.parse(readFileSync('dist/dashboard/state.json', 'utf8')); +assert.equal(state.phase, 'deleted'); +assert.equal(state.checks.length, 14); +assert(state.frozenHeartbeat); +const audit = readFileSync('dist/dashboard/supervisor-events.jsonl', 'utf8'); +assert.match(audit, /HTTP:GET.*ALLOWED/); +assert.match(audit, /HTTP:POST.*DENIED/); +NODE + duration=$(ffprobe -v error -show_entries format=duration -of csv=p=0 dist/dashboard/full-speed.mp4) + rate=$(awk -v duration="$duration" 'BEGIN { printf "%.9f", 45 / duration }') + ffmpeg -y -hide_banner -loglevel error -i dist/dashboard/full-speed.mp4 -vf "setpts=$rate*PTS" -r 30 -c:v libx264 -preset fast -crf 18 -an -movflags +faststart media/openshell-dashboard-demo.mp4 + echo 'Recorded media/openshell-dashboard-demo.mp4 · 45 seconds · live capture, accelerated' +else + bash media/dashboard-start.sh +fi diff --git a/scripts/demo-down.ts b/scripts/demo-down.ts new file mode 100644 index 0000000..db658a4 --- /dev/null +++ b/scripts/demo-down.ts @@ -0,0 +1,15 @@ +// Delete only the task pairs owned by this demo control plane, then control. +import { readFile, rm } from 'node:fs/promises' +import { Sandbox } from 'e2b' +const file = '.control-separated-id' +const id = (await readFile(file, 'utf8')).trim() +const pages = Sandbox.list({ query: { metadata: { 'openshell.ai/driver-owner': id } } }) +while (pages.hasNext) { + for (const item of await pages.nextItems()) { + await Sandbox.kill(item.sandboxId) + console.log('Deleted', item.metadata['openshell.ai/role'], item.sandboxId) + } +} +await Sandbox.kill(id) +await rm(file) +console.log('Deleted shared control plane', id) diff --git a/scripts/demo-separated.ts b/scripts/demo-separated.ts new file mode 100644 index 0000000..f24ee63 --- /dev/null +++ b/scripts/demo-separated.ts @@ -0,0 +1,195 @@ +// Real E2B/OpenShell integration demo. Creates and removes a task pair. +import assert from 'node:assert/strict' +import { execFile, spawn, type ChildProcess } from 'node:child_process' +import { promisify } from 'node:util' +import { mkdtemp, readFile, writeFile, rm, mkdir } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { setTimeout as sleep } from 'node:timers/promises' +import { Sandbox, CommandExitError } from 'e2b' + +const exec = promisify(execFile) +const env = { ...process.env, XDG_CONFIG_HOME: join(process.cwd(), '.openshell-separated'), OPENSHELL_COLOR: 'never' } +const fault = process.argv.includes('--fault') +const keep = process.argv.includes('--keep') +assert(!(fault && keep), '--fault and --keep cannot be combined') +const name = `separated-${Date.now().toString(36)}` +const results: { check: string; passed: true }[] = [] +const pass = (check: string) => { results.push({ check, passed: true }); console.log(` ✓ ${check}`) } +const step = (text: string) => console.log(`\n\x1b[1m${text}\x1b[0m`) +const os = async (args: string[], timeout = 60_000) => { + const command = exec('.bin/openshell', args, { env, timeout, maxBuffer: 4 * 1024 * 1024 }) + // OpenShell reads piped stdin before exec; terminate it for noninteractive calls. + command.child.stdin?.end() + return (await command).stdout +} +const inside = (cmd: string) => os(['sandbox', 'exec', '-n', name, '--no-tty', '--', 'sh', '-c', cmd], 30_000) +async function inventory(owner: string) { + const pages = Sandbox.list({ query: { metadata: { 'openshell.ai/driver-owner': owner } } }) + const items = [] + while (pages.hasNext) items.push(...await pages.nextItems()) + return items +} +async function run(sbx: Sandbox, cmd: string) { + try { return await sbx.commands.run(cmd, { user: 'root', timeoutMs: 20_000 }) } + catch (e) { if (e instanceof CommandExitError) return e; throw e } +} +async function main() { + let connector: ChildProcess | undefined + let creator: ChildProcess | undefined + let controlId = '' + let ids: string[] = [] + let completed = false + const dir = await mkdtemp(join(tmpdir(), 'openshell-separated-')) + let connectLog = '' + let createLog = '' + try { + step('1. Shared control plane — gateway + E2B compute driver') + try { await os(['sandbox', 'list'], 5_000) } + catch { + connector = spawn(process.execPath, ['--import', 'tsx', '--env-file=.env', 'scripts/connect.ts'], { env: process.env, stdio: ['ignore', 'pipe', 'pipe'] }) + connector.stdout?.on('data', d => { connectLog += d.toString(); process.stdout.write(d) }) + connector.stderr?.on('data', d => { connectLog += d.toString() }) + let online = false + for (let i = 0; i < 90; i++) { + if (connector.exitCode !== null) throw new Error(`Control setup exited: ${connectLog}`) + try { await os(['sandbox', 'list'], 2_000); online = true; break } catch { await sleep(1_000) } + } + assert(online, 'Control plane did not become ready') + } + controlId = (await readFile('.control-separated-id', 'utf8')).trim() + console.log(` Control: ${controlId} (shared; remains running after this task)`) + const control = await Sandbox.connect(controlId) + const controlProcesses = await run(control, 'pgrep -x openshell-super || true') + assert.equal(controlProcesses.stdout.trim(), '', 'Supervisor must not run in the control VM') + pass('No supervisor process in the control plane') + + step(`2. Create task ${name} — two independent E2B microVMs`) + const started = Date.now() + creator = spawn('.bin/openshell', ['sandbox', 'create', '--name', name, '--no-tty', '--', 'sleep', 'infinity'], { env, stdio: ['ignore', 'pipe', 'pipe'] }) + creator.stdout?.on('data', d => { createLog += d.toString() }) + creator.stderr?.on('data', d => { createLog += d.toString() }) + let ready = false + for (let i = 0; i < 180; i++) { + const list = await os(['sandbox', 'list'], 10_000) + if (list.split('\n').some(l => l.startsWith(name + ' ') && /\bReady\b/.test(l))) { ready = true; break } + if (creator.exitCode !== null && creator.exitCode !== 0) throw new Error(`Create failed: ${createLog}`) + await sleep(1_000) + } + assert(ready, `Task did not become Ready: ${createLog}`) + const pair = (await inventory(controlId)).filter(s => s.metadata['openshell.ai/sandbox-name'] === name) + assert.equal(pair.length, 2, 'A task must own exactly two VMs') + const a = pair.find(s => s.metadata['openshell.ai/role'] === 'supervisor')! + const b = pair.find(s => s.metadata['openshell.ai/role'] === 'worker')! + assert(a && b && a.sandboxId !== b.sandboxId && a.sandboxId !== controlId && b.sandboxId !== controlId) + ids = [a.sandboxId, b.sandboxId] + console.log(` A · supervisor/proxy: ${a.sandboxId}\n B · worker/agent: ${b.sandboxId}`) + console.log(` Ready in ${((Date.now() - started) / 1000).toFixed(1)}s (create request → gateway Ready)`) + pass('Gateway Ready with a distinct supervisor/worker pair') + const supervisor = await Sandbox.connect(a.sandboxId) + const worker = await Sandbox.connect(b.sandboxId) + assert.equal((await run(supervisor, 'test -f /run/openshell-task/ready && kill -0 $(cat /run/openshell-task/supervisor.pid)')).exitCode, 0) + assert.equal((await run(worker, 'test ! -e /opt/openshell/openshell-supervisor && test ! -e /run/openshell-task/auth.json')).exitCode, 0) + for (const sbx of [supervisor, worker]) { + assert.equal((await run(sbx, 'test -z "$E2B_API_KEY" && test ! -f /home/user/openshell/driver.env')).exitCode, 0) + } + pass('Supervisor and session credentials live in A; E2B management key stays in control') + console.log((await inside('printf " Agent uid="; id -u; grep -E "^(CapEff|NoNewPrivs)" /proc/self/status')).trim()) + + step('3. Two enforcement layers — OpenShell policy + E2B worker egress fence') + const denied = await inside('if curl -sS --max-time 5 https://api.github.com/zen >/dev/null 2>&1; then echo UNEXPECTED; else echo DEFAULT_DENY; fi') + assert.match(denied, /DEFAULT_DENY/) + pass('Unlisted destination denied before any network policy is added') + const probe = "curl --noproxy '*' -sS --connect-timeout 3 --max-time 5 -o /dev/null -w '%{http_code}' https://1.1.1.1/cdn-cgi/trace" + const aProbe = await run(supervisor, probe) + assert.equal(aProbe.exitCode, 0, 'External probe must be reachable from the supervisor') + const bProbe = await run(worker, probe) + assert.notEqual(bProbe.exitCode, 0, 'Worker root must not have direct egress') + assert.equal(bProbe.stdout, '000') + pass('Direct HTTPS works from A; blocked from B even as root outside the agent namespace') + + step('4. Apply an OpenShell policy live — GitHub read-only, curl only') + const base = await os(['policy', 'get', name, '--base']) + const split = base.indexOf('\n---\n') + assert(split >= 0, 'Expected policy YAML document') + const policyPath = join(dir, 'policy.yaml') + await writeFile(policyPath, base.slice(split + 5) + ` +network_policies: + github: + name: github-readonly + endpoints: + - host: api.github.com + port: 443 + protocol: rest + enforcement: enforce + access: read-only + binaries: + - path: /usr/bin/curl +`) + console.log((await os(['policy', 'set', name, '--policy', policyPath, '--wait'])).trim()) + const get = (await inside('curl -sS -m 15 -o /dev/null -w "%{http_code}" https://api.github.com/zen')).trim() + assert.equal(get, '200'); pass('curl GET api.github.com/zen → HTTP 200') + const post = (await inside('curl -sS -m 15 -o /dev/null -w "%{http_code}" -X POST -d "{}" https://api.github.com/gists')).trim() + assert.equal(post, '403'); pass('curl POST api.github.com/gists → HTTP 403 (blocked by OpenShell)') + const wget = await inside('command -v wget >/dev/null || exit 9; if wget -q -T 5 -t 1 -O /dev/null https://api.github.com/zen; then echo UNEXPECTED; else echo PROGRAM_DENIED; fi') + assert.match(wget, /PROGRAM_DENIED/); pass('wget denied for the same host: program identity enforced') + const audit = await os(['logs', name]) + const auditLines = audit.split('\n').filter(l => /HTTP:|NET:OPEN/.test(l)) + assert(auditLines.some(l => /DENIED/.test(l)), 'Expected deny audit record') + console.log(auditLines.slice(-4).map(l => ' ' + l.slice(0, 180)).join('\n')) + pass('OpenShell audit records retained across the separate supervisor channel') + + if (fault) { + step('5. Failure drill — stop A; B must not gain direct access') + await supervisor.kill() + // The runtime owns fail-closed agent suspension. Check its kernel-visible + // process state from envd, outside the agent PID/network namespace. + let frozen = false + for (let i = 0; i < 30; i++) { + const states = await run(worker, "ps -u 1500 -o stat=,comm=") + if (states.stdout.split('\n').some(l => /^T/.test(l.trim()) && /sleep/.test(l))) { frozen = true; break } + await sleep(1_000) + } + assert(frozen, 'OpenShell must freeze the agent after supervisor channel loss') + pass('Worker runtime suspended the agent when its supervisor disappeared') + const after = await run(worker, probe) + assert.notEqual(after.exitCode, 0); assert.equal(after.stdout, '000') + pass('Worker root egress remains denied after supervisor loss') + } + if (keep) { + console.log(`\nTask retained for inspection: ${name}\nDelete: XDG_CONFIG_HOME=$PWD/.openshell-separated .bin/openshell sandbox delete ${name}`) + } else { + step(`${fault ? '6' : '5'}. Delete task — both microVMs removed, control plane stays up`) + await os(['sandbox', 'delete', name]) + let remaining: string[] = ids + for (let i = 0; i < 20; i++) { + const live = await inventory(controlId) + remaining = ids.filter(id => live.some(s => s.sandboxId === id)) + if (!remaining.length) break + await sleep(1_000) + } + assert.equal(remaining.length, 0, 'Task deletion must remove BOTH VMs') + pass('Supervisor and worker deleted together') + await os(['sandbox', 'list']); pass('Shared control plane still available') + } + completed = true + await mkdir('dist', { recursive: true }) + const report = fault ? 'dist/separated-demo-fault-verification.json' : 'dist/separated-demo-verification.json' + await writeFile(report, JSON.stringify({ verifiedAt: new Date().toISOString(), name, controlId, supervisorId: a.sandboxId, workerId: b.sandboxId, fault, kept: keep, checks: results }, null, 2) + '\n') + console.log(`\nPASS — ${results.length} live checks. Report: ${report}`) + } finally { + if (!completed || !keep) { + await os(['sandbox', 'delete', name], 30_000).catch(() => {}) + // On partial create, discover only this demo's pair for fallback cleanup. + if (controlId) { + const left = await inventory(controlId).catch(() => []) + for (const s of left.filter(s => s.metadata['openshell.ai/sandbox-name'] === name)) await Sandbox.kill(s.sandboxId) + } + } + creator?.kill() + await rm(dir, { recursive: true, force: true }) + if (!keep || !completed) connector?.kill() + else if (connector) console.log('Laptop tunnel remains open for inspection. Ctrl-C closes the tunnel; task expires in one hour.') + } +} +main().catch(error => { console.error(`\nFAIL: ${error.message}`); process.exitCode = 1 }) diff --git a/scripts/demo.sh b/scripts/demo.sh index 9976460..768391a 100755 --- a/scripts/demo.sh +++ b/scripts/demo.sh @@ -11,7 +11,7 @@ # 7. delete set -euo pipefail cd "$(dirname "$0")/.." -export XDG_CONFIG_HOME="$PWD/.openshell" +export XDG_CONFIG_HOME="$PWD/.openshell-separated" os() { .bin/openshell "$@"; } in_sandbox() { os sandbox exec -n "$NAME" --no-tty -- sh -c "$1"; } step() { printf '\n\033[1m== %s\033[0m\n' "$*"; } diff --git a/scripts/fleet.sh b/scripts/fleet.sh index 92199d5..85535f4 100755 --- a/scripts/fleet.sh +++ b/scripts/fleet.sh @@ -6,14 +6,14 @@ # npm run fleet -- 5 5 workloads # KEEP=1 npm run fleet leave them running (then: openshell sandbox delete ) # -# 1. create N sandboxes in parallel → N private E2B microVMs, one control box -# 2. each is its own machine → one E2B box per sandbox, same fence +# 1. create N sandboxes in parallel → 2N private E2B microVMs, one control box +# 2. each is its own machine → one supervisor + worker pair per task, same fence # 3. per-sandbox policy → only the first may read api.github.com # 4. enforcement per sandbox → first GET 200, the rest denied # 5. delete set -euo pipefail cd "$(dirname "$0")/.." -export XDG_CONFIG_HOME="$PWD/.openshell" +export XDG_CONFIG_HOME="$PWD/.openshell-separated" os() { .bin/openshell "$@"; } in_sandbox() { os sandbox exec -n "$1" --no-tty -- sh -c "$2"; } step() { printf '\n\033[1m== %s\033[0m\n' "$*"; } @@ -44,7 +44,7 @@ done echo "all $COUNT Ready after $(( $(date +%s) - start ))s" os sandbox list | grep -E "^NAME|^$PREFIX-" -step "2. each workload is its own E2B box, same fence" +step "2. each workload gets its own supervisor and worker VMs" # The fence hides the E2B sandbox id from the workload, so ask the E2B API: # the driver tags every box with its OpenShell sandbox name. npx tsx --env-file=.env scripts/sbx.ts ls | grep " $PREFIX-" | awk '{print $5, "-> E2B box", $1}' | sort diff --git a/scripts/sbx.ts b/scripts/sbx.ts index a32cc8e..43e8a6c 100644 --- a/scripts/sbx.ts +++ b/scripts/sbx.ts @@ -16,8 +16,8 @@ if (cmd === 'create') { try { await sbx.commands.run(command.join(' '), { timeoutMs: 0, - onStdout: (d) => process.stdout.write(d), - onStderr: (d) => process.stderr.write(d), + onStdout: (d) => { process.stdout.write(d) }, + onStderr: (d) => { process.stderr.write(d) }, }) } catch (e) { if (e instanceof CommandExitError) process.exit(e.exitCode) diff --git a/scripts/studio.sh b/scripts/studio.sh new file mode 100644 index 0000000..d770fc9 --- /dev/null +++ b/scripts/studio.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Reproducible three-pane demo; only this studio task is cleaned up on exit. +set -euo pipefail +cd "$(dirname "$0")/.." +if tmux -L openshell-studio has-session -t studio 2>/dev/null; then + echo 'A studio session is already open. Attach with: tmux -L openshell-studio attach' >&2 + exit 1 +fi +mkdir -p dist/studio +node --import tsx --env-file=.env scripts/studio.ts reset +connector='' +finish() { + result=$? + if [ -f dist/studio/state.json ]; then + if ! node --import tsx --env-file=.env scripts/studio.ts cleanup >dist/studio/cleanup.log 2>&1; then + cat dist/studio/cleanup.log >&2 + result=1 + fi + fi + tmux -L openshell-studio kill-session -t studio 2>/dev/null || true + if [ -n "$connector" ]; then kill "$connector" 2>/dev/null || true; fi + exit "$result" +} +trap finish EXIT +if ! XDG_CONFIG_HOME="$PWD/.openshell-separated" .bin/openshell status >/dev/null 2>&1; then + node --import tsx --env-file=.env scripts/connect.ts >dist/studio/connect.log 2>&1 & + connector=$! + ready=0 + for _ in $(seq 90); do + if XDG_CONFIG_HOME="$PWD/.openshell-separated" .bin/openshell status >/dev/null 2>&1; then ready=1; break; fi + kill -0 "$connector" 2>/dev/null || { cat dist/studio/connect.log >&2; exit 1; } + sleep 1 + done + [ "$ready" = 1 ] || { echo 'Gateway connection timed out.' >&2; exit 1; } +fi +if [ "${1:-}" = '--record' ]; then + vhs media/separated-supervisor.tape + # Tape waits stop on real results. Refuse to publish an incomplete run. + node --input-type=module <<'NODE' +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +const state = JSON.parse(readFileSync('dist/studio/state.json', 'utf8')); +assert.equal(state.phase, 'deleted'); +assert.equal(state.checks.length, 14); +assert(state.frozenHeartbeat); +const audit = readFileSync('dist/studio/supervisor-events.jsonl', 'utf8'); +assert.match(audit, /HTTP:GET.*ALLOWED/); +assert.match(audit, /HTTP:POST.*DENIED/); +NODE + duration=$(ffprobe -v error -show_entries format=duration -of csv=p=0 dist/studio/full-speed.mp4) + rate=$(awk -v duration="$duration" 'BEGIN { printf "%.9f", 45 / duration }') + ffmpeg -y -hide_banner -loglevel error -i dist/studio/full-speed.mp4 -vf "setpts=$rate*PTS" -r 30 -c:v libx264 -preset fast -crf 18 -an -movflags +faststart media/separated-supervisor-demo.mp4 + echo 'Recorded: media/separated-supervisor-demo.mp4 (45 seconds, 1080p; accelerated live capture)' +else + bash media/studio-start.sh +fi diff --git a/scripts/studio.ts b/scripts/studio.ts new file mode 100644 index 0000000..fedabf9 --- /dev/null +++ b/scripts/studio.ts @@ -0,0 +1,274 @@ +// Live, chapter-driven demo. Every displayed result comes from OpenShell or E2B. +import assert from 'node:assert/strict' +import { execFile, spawn } from 'node:child_process' +import { promisify } from 'node:util' +import { appendFile, mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises' +import { setTimeout as sleep } from 'node:timers/promises' +import { Sandbox, CommandExitError, SandboxNotFoundError } from 'e2b' + +const dir = process.env.STUDIO_DIR || 'dist/studio' +const file = `${dir}/state.json` +const exec = promisify(execFile) +const env = { ...process.env, XDG_CONFIG_HOME: `${process.cwd()}/.openshell-separated`, OPENSHELL_COLOR: 'never' } +const c = { reset: '\x1b[0m', dim: '\x1b[90m', blue: '\x1b[94m', green: '\x1b[92m', red: '\x1b[91m', amber: '\x1b[93m', bold: '\x1b[1m' } +type State = { + name: string; control: string; supervisor?: string; worker?: string + phase: 'creating' | 'ready' | 'cutting' | 'frozen' | 'deleted' + checks: string[]; rootFence?: boolean; frozenHeartbeat?: string; createdAt: string +} +const state = async () => JSON.parse(await readFile(file, 'utf8')) as State +const save = async (s: State) => { + await writeFile(`${file}.tmp`, JSON.stringify(s, null, 2) + '\n') + await rename(`${file}.tmp`, file) +} +const line = (message: string, color = c.green) => console.log(`${color}${message}${c.reset}`) +const pass = (s: State, message: string) => { s.checks.push(message); line(` ✓ ${message}`) } +const os = async (args: string[], timeout = 30_000) => { + const command = exec('.bin/openshell', args, { env, timeout, maxBuffer: 4 * 1024 * 1024 }) + command.child.stdin?.end() + return (await command).stdout +} +const inside = (s: State, cmd: string) => os(['sandbox', 'exec', '-n', s.name, '--no-tty', '--', 'sh', '-c', cmd]) +async function run(sbx: Sandbox, cmd: string) { + try { return await sbx.commands.run(cmd, { user: 'root', timeoutMs: 15_000 }) } + catch (error) { if (error instanceof CommandExitError) return error; throw error } +} +async function pair(s: State) { + const pages = Sandbox.list({ query: { metadata: { 'openshell.ai/driver-owner': s.control, 'openshell.ai/sandbox-name': s.name } } }) + const items = [] + while (pages.hasNext) items.push(...await pages.nextItems()) + return items +} +const probe = "curl --noproxy '*' -sS --connect-timeout 3 --max-time 5 -o /dev/null -w '%{http_code}' https://1.1.1.1/cdn-cgi/trace" +const sample = 'cat /tmp/agent-heartbeat; ps -u 1500 -o stat=,comm=' + +async function create() { + const previous = await state().catch(() => undefined) + assert(!previous || previous.phase === 'deleted', 'A studio task already exists. Run studio cleanup first.') + await os(['status']) + const s: State = { name: `${process.env.STUDIO_NAME_PREFIX || 'studio'}-${Date.now().toString(36)}`, control: (await readFile('.control-separated-id', 'utf8')).trim(), phase: 'creating', checks: [], createdAt: new Date().toISOString() } + await save(s) + await writeFile(`${dir}/supervisor-events.jsonl`, '') + console.log('Creating two private E2B microVMs…') + const control = await Sandbox.connect(s.control) + assert.equal((await run(control, 'pgrep -x openshell-super || true')).stdout.trim(), '') + pass(s, 'Shared control plane: gateway + driver only') + const creator = spawn('.bin/openshell', ['sandbox', 'create', '--name', s.name, '--no-tty', '--', 'sh', '-c', 'n=0; while :; do n=$((n+1)); printf "%s\\n" "$n" > /tmp/agent-heartbeat; sleep 0.5; done'], { env, stdio: ['ignore', 'ignore', 'pipe'] }) + let errors = '' + creator.stderr.on('data', chunk => { errors += String(chunk) }) + try { + let ready = false + for (let i = 0; i < 90; i++) { + const list = await os(['sandbox', 'list']) + if (list.split('\n').some(row => row.startsWith(s.name + ' ') && /\bReady\b/.test(row))) { ready = true; break } + assert(creator.exitCode === null || creator.exitCode === 0, `Create failed: ${errors}`) + await sleep(1_000) + } + assert(ready, 'Pair did not become Ready') + const items = await pair(s) + assert.equal(items.length, 2) + s.supervisor = items.find(item => item.metadata['openshell.ai/role'] === 'supervisor')?.sandboxId + s.worker = items.find(item => item.metadata['openshell.ai/role'] === 'worker')?.sandboxId + assert(s.supervisor && s.worker && new Set([s.control, s.supervisor, s.worker]).size === 3) + pass(s, 'Supervisor A and worker B are distinct microVMs') + const worker = await Sandbox.connect(s.worker) + assert.equal((await run(worker, 'test ! -e /run/openshell-task/auth.json && test ! -e /opt/openshell/openshell-supervisor')).exitCode, 0) + const identity = await inside(s, 'id -u; grep -E "^(CapEff|NoNewPrivs)" /proc/self/status') + assert.match(identity, /^1500\n/); assert.match(identity, /CapEff:\s+0+/); assert.match(identity, /NoNewPrivs:\s+1/) + pass(s, 'Worker: uid 1500, zero capabilities, no supervisor auth') + s.phase = 'ready' + await save(s) + console.log(`\n A / supervisor ${s.supervisor}\n B / worker ${s.worker}`) + console.log('\n agent in B ── TLS channel ──► policy in A ──► internet') + line('\nPAIR READY', c.blue) + } finally { creator.kill() } +} + +async function fence(s: State) { + assert.equal(s.phase, 'ready') + const denied = await inside(s, 'if curl -sS -m 5 https://api.github.com/zen >/dev/null 2>&1; then echo UNEXPECTED; else echo DEFAULT_DENY; fi') + assert.match(denied, /DEFAULT_DENY/) + pass(s, 'OpenShell blocks GitHub before policy is added') + const [a, b] = await Promise.all([Sandbox.connect(s.supervisor!), Sandbox.connect(s.worker!)]) + const [allowed, blocked] = await Promise.all([run(a, probe), run(b, probe)]) + assert.equal(allowed.exitCode, 0); assert.equal(allowed.stdout, '200') + assert.notEqual(blocked.exitCode, 0); assert.equal(blocked.stdout, '000') + console.log('\n Same direct HTTPS probe, as root, outside agent namespaces:') + line(' A / supervisor HTTP 200 upstream reachable') + line(' B / worker HTTP 000 E2B egress fence', c.red) + pass(s, 'Worker root cannot bypass the external network fence') + s.rootFence = true + await save(s) + line('\nFENCE VERIFIED', c.blue) +} + +async function policy(s: State) { + assert.equal(s.phase, 'ready') + const worker = await Sandbox.connect(s.worker!) + const before = Number((await run(worker, 'cat /tmp/agent-heartbeat')).stdout.trim()) + assert(Number.isFinite(before) && before > 0, 'Expected a running agent heartbeat') + const base = await os(['policy', 'get', s.name, '--base']) + const split = base.indexOf('\n---\n') + assert(split >= 0) + await writeFile(`${dir}/policy.yaml`, base.slice(split + 5) + '\nnetwork_policies:\n github:\n name: github-readonly\n endpoints:\n - host: api.github.com\n port: 443\n protocol: rest\n enforcement: enforce\n access: read-only\n binaries:\n - path: /usr/bin/curl\n') + console.log(' program /usr/bin/curl\n host api.github.com:443\n access read-only\n') + console.log((await os(['policy', 'set', s.name, '--policy', `${dir}/policy.yaml`, '--wait'])).trim()) + await sleep(600) + const after = Number((await run(worker, 'cat /tmp/agent-heartbeat')).stdout.trim()) + assert(after > before, 'Agent heartbeat must advance across the policy update') + pass(s, 'Policy updated live; worker keeps running') + await save(s) + line('\nPOLICY ACTIVE', c.blue) +} + +async function requests(s: State) { + assert.equal(s.phase, 'ready') + const get = (await inside(s, 'curl -sS -m 15 -o /dev/null -w "%{http_code}" https://api.github.com/zen')).trim() + assert.equal(get, '200'); pass(s, 'curl GET /zen → HTTP 200') + if (process.env.STUDIO_DEMO_DELAY) await sleep(Number(process.env.STUDIO_DEMO_DELAY)) + const post = (await inside(s, 'curl -sS -m 15 -o /dev/null -w "%{http_code}" -X POST -d "{}" https://api.github.com/gists')).trim() + assert.equal(post, '403'); pass(s, 'curl POST /gists → HTTP 403') + if (process.env.STUDIO_DEMO_DELAY) await sleep(Number(process.env.STUDIO_DEMO_DELAY)) + const wget = await inside(s, 'command -v wget >/dev/null || exit 9; if wget -q -T 5 -t 1 -O /dev/null https://api.github.com/zen; then echo UNEXPECTED; else echo DENIED; fi') + assert.match(wget, /DENIED/); pass(s, 'wget GET /zen → program denied') + const audit = await os(['logs', s.name]) + assert.match(audit, /HTTP:GET .*ALLOWED/); assert.match(audit, /HTTP:POST .*DENIED/) + pass(s, 'Allow / deny audit arrives from separate supervisor A') + await save(s) + line('\nREQUESTS VERIFIED', c.blue) +} + +async function sever(s: State) { + assert.equal(s.phase, 'ready') + s.phase = 'cutting'; await save(s) + line('Terminating supervisor sandbox A…', c.amber) + await Sandbox.kill(s.supervisor!) + const worker = await Sandbox.connect(s.worker!) + let frozen = '' + for (let i = 0; i < 30; i++) { + const observation = await run(worker, sample) + if (observation.stdout.split('\n').some(row => /^T\S*\s+(sh|sleep)$/.test(row.trim()))) { frozen = observation.stdout.split('\n')[0].trim(); break } + await sleep(1_000) + } + assert(frozen, 'Expected stopped agent process in worker B') + await sleep(1_200) + assert.equal((await run(worker, 'cat /tmp/agent-heartbeat')).stdout.trim(), frozen, 'Agent heartbeat must stop') + pass(s, `Worker is still alive; agent FROZEN at heartbeat ${frozen}`) + s.frozenHeartbeat = frozen; s.phase = 'frozen'; await save(s) + const blocked = await run(worker, probe) + assert.notEqual(blocked.exitCode, 0); assert.equal(blocked.stdout, '000') + pass(s, 'Root egress remains blocked after supervisor loss') + await save(s) + line('\nFAIL-CLOSED VERIFIED', c.red) +} + +async function cleanup(s: State) { + if (s.phase === 'deleted') return + await os(['sandbox', 'delete', s.name]).catch(() => {}) + // List is eventually consistent after deletion. Wait for the inventory and + // confirm each known VM is absent through the individual lookup as well. + for (let i = 0; i < 60; i++) { + const left = await pair(s) + if (!left.length) break + if (i === 10 || i === 30) for (const item of left) await Sandbox.kill(item.sandboxId) + await sleep(1_000) + } + assert.equal((await pair(s)).length, 0, 'Both task VMs must be gone') + for (const id of [s.supervisor, s.worker].filter(Boolean)) { + await assert.rejects(Sandbox.getInfo(id!), SandboxNotFoundError) + } + pass(s, 'Both task microVMs removed') + await os(['status']); pass(s, 'Shared control plane stays available') + s.phase = 'deleted'; await save(s) + line(`\nCLEANUP VERIFIED · ${s.checks.length} live assertions`, c.blue) +} + +function auditLine(raw: string): string | undefined { + const text = raw.replace(/\x1b\[[0-9;]*m/g, '') + if (/Policy reloaded successfully/.test(text)) return `${c.blue}POLICY UPDATED · live, no restart` + if (/Isolation boundary attached/.test(text)) return `${c.blue}TLS channel attached to worker B` + if (/Isolation boundary enforcement confirmed/.test(text)) return `${c.green}Worker isolation confirmed` + let match = text.match(/HTTP:(\w+) .*?(ALLOWED|DENIED).*?http:\/\/[^/]+(\/\S*)/) + if (match) return `${match[2] === 'ALLOWED' ? c.green : c.red}${match[2].padEnd(7)} ${match[1].padEnd(5)} ${match[3]}` + match = text.match(/NET:OPEN .*?(ALLOWED|DENIED) \/usr\/bin\/(\w+)\(\d+\) -> (\S+)/) + if (match) return `${match[1] === 'ALLOWED' ? c.green : c.red}${match[1].padEnd(7)} ${match[2].padEnd(5)} ${match[3]}` + return undefined +} + +async function supervisorFeed() { + line('Waiting for supervisor A…', c.dim) + let current = ''; let handle: { kill(): Promise } | undefined; let offline = false + const stop = async () => { await handle?.kill().catch(() => {}); process.exit(0) } + process.on('SIGTERM', stop); process.on('SIGINT', stop) + while (true) { + const s = await state().catch(() => undefined) + if (s?.supervisor && s.phase !== 'deleted' && current !== s.supervisor) { + current = s.supervisor; offline = false + console.log('\x1b[2J\x1b[H') + line(`A / ${s.supervisor}`, c.blue) + console.log('Separate guest kernel · policy + proxy\n') + const a = await Sandbox.connect(s.supervisor) + let partial = '' + handle = await a.commands.run('exec tail -n +1 -F /run/openshell-task/supervisor.log', { + user: 'root', background: true, timeoutMs: 0, + onStdout: chunk => { + const lines = (partial + chunk).split('\n'); partial = lines.pop() ?? '' + for (const raw of lines) { + const formatted = auditLine(raw) + if (formatted) { + console.log(formatted + c.reset) + void appendFile(`${dir}/supervisor-events.jsonl`, JSON.stringify({ observedAt: new Date().toISOString(), raw: raw.replace(/\x1b\[[0-9;]*m/g, '') }) + '\n') + } + } + }, + }) + } + // Phase is set only after the SDK kill and kernel-state assertion succeed. + if (s?.phase === 'frozen' && !offline) { offline = true; line('\n■ SUPERVISOR VM TERMINATED', c.red); console.log('Worker B remains isolated.'); } + if (s?.phase === 'deleted' && current && !offline) { offline = true; line('\n■ SUPERVISOR VM REMOVED', c.dim) } + await sleep(400) + } +} + +async function workerFeed() { + let worker: Sandbox | undefined; let current = ''; let previous = ''; let last = '' + while (true) { + const s = await state().catch(() => undefined) + let screen = `${c.blue}Waiting for worker B…${c.reset}` + if (s?.worker) { + if (current !== s.worker) { current = s.worker; worker = await Sandbox.connect(s.worker); previous = '' } + if (s.phase === 'deleted') screen = `${c.blue}B / ${s.worker}${c.reset}\n\n${c.green}■ WORKER VM REMOVED\n\nTask pair cleaned up.${c.reset}` + else { + try { + const result = await run(worker!, sample) + assert.equal(result.exitCode, 0) + const beat = result.stdout.split('\n')[0].trim() + const stopped = result.stdout.split('\n').some(row => /^T\S*\s+(sh|sleep)$/.test(row.trim())) + screen = `${c.blue}B / ${s.worker}${c.reset}\nSeparate guest kernel · runtime + agent\n\n${stopped ? c.red : c.green}${c.bold}${stopped ? '■ AGENT FROZEN' : '● AGENT RUNNING'}${c.reset}\n\nHeartbeat ${beat.padStart(5, '0')} ${stopped ? 'STOPPED' : beat !== previous ? '▂ ▄ ▆ █ ▆ ▄' : '▁ ▂ ▄ ▆ ▄ ▂'}\nKernel ${stopped ? 'T = stopped by runtime' : 'agent process active'}\nIdentity uid 1500 · zero caps\nRoot egress ${s.rootFence ? 'DENIED · verified HTTPS probe' : 'probe pending'}\n\n${c.dim}Heartbeat and process state read live from B.${c.reset}` + previous = beat + if (process.env.STUDIO_COMPACT === '1') screen = `${c.blue}B / ${s.worker}${c.reset}\n${stopped ? c.red : c.green}${c.bold}${stopped ? '■ AGENT FROZEN' : '● AGENT RUNNING'}${c.reset}\nHeartbeat ${beat.padStart(5, '0')} ${stopped ? 'STOPPED' : '▂ ▄ ▆ █ ▆ ▄'}\nKernel ${stopped ? 'T = stopped by runtime' : 'agent process active'}\nIdentity uid 1500 · zero caps\nRoot HTTPS ${s.rootFence ? 'DENIED · verified' : 'probe pending'}\n${c.dim}Read live from the worker VM${c.reset}` + } catch { screen = `${c.amber}Worker observation unavailable; checking lifecycle…${c.reset}` } + } + } + if (screen !== last) { process.stdout.write('\x1b[2J\x1b[H' + screen + '\n'); last = screen } + await sleep(700) + } +} + +await mkdir(dir, { recursive: true }) +const command = process.argv[2] +try { + if (command === 'supervisor-feed') await supervisorFeed() + else if (command === 'worker-feed') await workerFeed() + else if (command === 'create') await create() + else if (command === 'reset') { + const s = await state().catch(() => undefined) + assert(!s || s.phase === 'deleted', 'Run studio cleanup before reset') + await rm(file, { force: true }) + } + else { + const actions = { fence, policy, requests, sever, cleanup } + assert(command in actions, 'Usage: studio create|fence|policy|requests|sever|cleanup') + await actions[command as keyof typeof actions](await state()) + } +} catch (error) { console.error(`STUDIO FAILED: ${(error as Error).message}`); process.exitCode = 1 } diff --git a/scripts/watch.ts b/scripts/watch.ts index 503b9e7..1e0c91f 100644 --- a/scripts/watch.ts +++ b/scripts/watch.ts @@ -1,8 +1,8 @@ -// watch.ts: live, color-coded view of what happens inside the control box. +// Live logs from the shared control plane and each separate supervisor VM. // // npm run watch (while `npm run connect` is running in another terminal) // -// Streams the gateway, driver and every supervisor log from the control box +// Streams gateway/driver logs from control and supervisor logs from sandbox A // and keeps only the lines that tell the story: sandbox lifecycle, supervisor // sessions, policy loads, and every ALLOWED / DENIED network decision. import { readFile } from 'node:fs/promises' @@ -13,7 +13,7 @@ const color = { gateway: '\x1b[35m', driver: '\x1b[33m', supervisor: '\x1b[36m', // Lines worth showing, and how to shorten them. const INTERESTING = [ - /E2B box created|health:|supervisor started|deleted|create failed|starting E2B compute driver/, // driver + /E2B task pair created|health:|supervisor started|deleted|create failed|starting E2B compute driver/, // driver /CreateSandbox|DeleteSandbox|supervisor session: accepted|Compute driver connected|relay opened/, // gateway /Isolation boundary (attached|enforcement confirmed|agent started)|Policy reloaded|initial policy|ALLOWED|DENIED/, // supervisor ] @@ -32,9 +32,9 @@ const COMPACT = process.argv.includes('--compact') const g = '\x1b[32m', r = '\x1b[31m', y = '\x1b[33m', m = '\x1b[35m', c = '\x1b[36m', d = '\x1b[2m', b = '\x1b[1m', x = '\x1b[0m' const short = (id: string) => id.slice(0, 8) const COMPACT_RULES: [RegExp, (m: RegExpMatchArray) => string][] = [ - [/E2B box created .*e2b=(\w+)/, (k) => `${y}driver ${x}▸ E2B microVM created ${d}${short(k[1])}${x}`], + [/E2B task pair created .*worker=(\w+) supervisor=(\w+)/, (k) => `${y}driver ${x}▸ worker ${short(k[1])} + supervisor ${short(k[2])}`], [/create failed|fence check failed/, () => `${r}driver ▸ create failed (see driver log)${x}`], - [/supervisor started/, () => `${y}driver ${x}▸ tunnel 2 up, supervisor started`], + [/supervisor started/, () => `${y}driver ${x}▸ separate supervisor started`], [/CreateSandbox request completed/, () => `${m}gateway ${x}▸ CreateSandbox done`], [/Isolation boundary attached/, () => `${c}superv. ${x}▸ attached to fenced runtime ${d}(TLS 1.3)${x}`], [/Isolation boundary enforcement confirmed/, () => `${c}superv. ${x}▸ isolation confirmed`], @@ -45,7 +45,7 @@ const COMPACT_RULES: [RegExp, (m: RegExpMatchArray) => string][] = [ [/NET:OPEN .*DENIED \/usr\/bin\/(\w+)\(\d+\) -> (\S+)/, (k) => `${r}${b}DENY ${x}${r} net ${k[1]} → ${k[2]}${x}`], [/HTTP:(\w+) .*ALLOWED \w+ http:\/\/[^/]+(\/\S*)/, (k) => `${g}${b}ALLOW${x}${g} http ${k[1]} ${k[2]}${x}`], [/HTTP:(\w+) .*DENIED \w+ http:\/\/[^/]+(\/\S*)/, (k) => `${r}${b}DENY ${x}${r} http ${k[1]} ${k[2]} ${d}(read-only rule)${x}`], - [/lifecycle: deleted/, () => `${y}driver ${x}▸ E2B microVM deleted`], + [/task pair deleted/, () => `${y}driver ${x}▸ supervisor + worker deleted`], [/health: sandbox not ready .*reason="?(\w+)/, (k) => `${r}health ▸ not ready: ${k[1]}${x}`], ] @@ -66,55 +66,58 @@ function paint(source: string, line: string): string { } async function main() { - const id = (await readFile('.control-id', 'utf8')).trim() - const sbx = await Sandbox.connect(id) - console.log(`${color.dim}watching control box ${id.slice(0, 10)}… (Ctrl-C to stop)${color.reset}\n`) - // Each log gets its own labelled stream ("source|line"). Supervisor logs are - // picked up as sandboxes appear; ones that already existed start from now, - // new ones from their first line. - // A per-run marker file, so a watcher left over from an earlier run can't - // claim this run's logs. The trap stops only this script's own tails. - // (Never `kill 0`: every E2B command shares envd's process group, so that - // would also kill envd, the gateway and the driver.) - const run = `/tmp/watch-${process.pid}-${Date.now()}` - const script = [ - `cd ${STATE} && : > ${run} && trap 'kill $(jobs -p) 2>/dev/null' EXIT`, - `(tail -n 0 -F gateway.log | sed -u 's/^/gateway|/') &`, - `(tail -n 0 -F driver.log | sed -u 's/^/driver|/') &`, - 'from=0', - 'while true; do', - ' for f in sandboxes/*/supervisor.err.log; do', - ` [ -e "$f" ] || continue; grep -qxF "$f" ${run} && continue; echo "$f" >> ${run}`, - ' if [ "$from" = 0 ]; then n=0; else n=+1; fi', - ` (tail -n $n -F "$f" | sed -u 's/^/supervisor|/') &`, - ' done; from=1; sleep 2', - 'done', - ].join('\n') - let partial = '' - const handle = await sbx.commands.run(script, { - background: true, - timeoutMs: 0, - onStdout: (chunk) => { - const lines = (partial + chunk).split('\n') - partial = lines.pop() ?? '' // keep an incomplete last line for the next chunk - for (const raw of lines) { - const bar = raw.indexOf('|') - const source = raw.slice(0, bar) - const line = raw.slice(bar + 1) - if (COMPACT) { - const out = compact(line) - if (out) console.log(out) - } else if (INTERESTING.some((re) => re.test(line))) { - console.log(paint(source, tidy(line))) + const id = (await readFile('.control-separated-id', 'utf8')).trim() + const control = await Sandbox.connect(id) + console.log(`${color.dim}control ${id}; discovering per-task supervisor VMs…${color.reset}\n`) + const handles = new Map }>() + let stopped = false + const follow = async (sbx: Sandbox, source: string, path: string, key: string) => { + let partial = '' + const handle = await sbx.commands.run(`exec tail -n ${source === 'supervisor' ? '+1' : '0'} -F ${path}`, { + user: 'root', background: true, timeoutMs: 0, + onStdout: chunk => { + const lines = (partial + chunk).split('\n') + partial = lines.pop() ?? '' + for (const line of lines) { + if (COMPACT) { + const out = compact(line) + if (out) console.log(out) + } else if (INTERESTING.some(re => re.test(line))) { + console.log(paint(source, tidy(line))) + } } - } - }, - }) - // Stop the remote loop when we stop, instead of leaving it running in the box. - const stop = async () => { await handle.kill().catch(() => {}); process.exit(0) } + }, + }) + if (stopped) { await handle.kill().catch(() => {}); return } + handles.set(key, handle) + } + await follow(control, 'gateway', `${STATE}/gateway.log`, 'gateway') + await follow(control, 'driver', `${STATE}/driver.log`, 'driver') + const seen = new Set() + const stop = async () => { + stopped = true + await Promise.allSettled([...handles.values()].map(h => h.kill())) + process.exit(0) + } process.on('SIGINT', stop) process.on('SIGTERM', stop) - await handle.wait() + while (!stopped) { + const pages = Sandbox.list({ query: { metadata: { + 'openshell.ai/driver-owner': id, 'openshell.ai/role': 'supervisor', + } } }) + try { + while (pages.hasNext) { + for (const item of await pages.nextItems()) { + if (seen.has(item.sandboxId)) continue + const sbx = await Sandbox.connect(item.sandboxId) + console.log(`${color.supervisor}supervisor A ${item.sandboxId} · ${item.metadata['openshell.ai/sandbox-name']}${color.reset}`) + await follow(sbx, 'supervisor', '/run/openshell-task/supervisor.log', item.sandboxId) + seen.add(item.sandboxId) + } + } + } catch (error) { if (!stopped) console.error('Log discovery:', (error as Error).message) } + await new Promise(resolve => setTimeout(resolve, 2_000)) + } } main().catch((e) => { console.error(e.message ?? e); process.exit(1) }) diff --git a/templates/control.ts b/templates/control.ts index 2b7dcd9..c165bf1 100644 --- a/templates/control.ts +++ b/templates/control.ts @@ -1,7 +1,6 @@ // openshell-control: the E2B template for the trusted control box. // // /opt/openshell-e2b/bin/openshell-gateway NVIDIA gateway, stock release -// /opt/openshell-e2b/bin/openshell-supervisor NVIDIA supervisor, demo patch (dist/) // /opt/openshell-e2b/bin/openshell-driver-e2b our driver (dist/) // /opt/openshell-e2b/bin/wstunnel tunnel client + server // /opt/openshell-e2b/node/ Node 24 (the E2B SDK needs >= 22) @@ -22,7 +21,6 @@ const ROOT = '/opt/openshell-e2b' export const template = Template({ fileContextPath: process.cwd() }) .fromBaseImage() .aptInstall(['curl', 'ca-certificates', 'xz-utils', 'jq']) - .copy(`dist/${TAG}/openshell-supervisor`, `${ROOT}/bin/openshell-supervisor`, { mode: 0o755, user: 'root' }) .copy(`dist/${TAG}/openshell-driver-e2b`, `${ROOT}/bin/openshell-driver-e2b`, { mode: 0o755, user: 'root' }) .copy('driver/e2b-helper.mjs', `${ROOT}/helper/e2b-helper.mjs`, { mode: 0o644, user: 'root' }) .runCmd([ @@ -33,7 +31,7 @@ export const template = Template({ fileContextPath: process.cwd() }) `cd ${ROOT}/helper && echo '{"type":"module","private":true}' > package.json && PATH=${ROOT}/node/bin:$PATH npm install --silent e2b@2.51.0`, ], { user: 'root' }) -Template.build(template, 'openshell-control', { +Template.build(template, 'openshell-control-separated', { cpuCount: 2, memoryMB: 2048, onBuildLogs: defaultBuildLogger(), diff --git a/templates/launch-sandbox.sh b/templates/launch-sandbox.sh index 4dac810..ac5ed82 100644 --- a/templates/launch-sandbox.sh +++ b/templates/launch-sandbox.sh @@ -4,7 +4,7 @@ # Runs as root inside the agent's E2B sandbox. Called by the driver after it # has written /.openshell/channel/sandbox/{bootstrap.json,server.crt,server.key}. # -# E2B sandbox (root namespace, has internet) +# E2B sandbox (host-enforced deny-all egress, including root namespace) # └── network namespace "os" ← only a loopback interface: no route out # └── PID namespace ← openshell-sandbox is PID 1, sees nothing else # └── openshell-sandbox ← uid/gid 1500, zero capabilities, no_new_privs diff --git a/templates/launch-supervisor.py b/templates/launch-supervisor.py new file mode 100644 index 0000000..3c831e3 --- /dev/null +++ b/templates/launch-supervisor.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +"""Run one task's supervisor and its two authenticated transports in sandbox A. + +Only an allowlisted environment reaches subprocesses. If any child exits, +terminate all of them; OpenShell's worker channel then fails closed. +""" +import json +import os +from pathlib import Path +import signal +import socket +import subprocess +import time + +STATE = Path('/run/openshell-task') +BIN = '/opt/openshell' +children = [] + + +def stop(*_): + raise SystemExit(1) + + +def start(name, args, env): + with (STATE / (name + '.log')).open('ab') as log: + child = subprocess.Popen(args, env=env, stdout=log, stderr=log) + children.append((name, child)) + (STATE / (name + '.pid')).write_text(str(child.pid)) + return child + + +def tunnel(name, spec, port, destination): + # No E2B management credential is installed in this VM. + return start(name, [BIN + '/wstunnel', 'client', '-L', + f'tcp://127.0.0.1:{port}:127.0.0.1:{destination}', + '--http-upgrade-path-prefix', spec['path'], + '-H', 'e2b-traffic-access-token: ' + spec['token'], + '--websocket-ping-frequency', '10s', '--tls-verify-certificate', + 'wss://' + spec['host']], {'PATH': '/usr/bin:/bin', 'NO_COLOR': 'true'}) + + +def main(): + os.umask(0o077) + signal.signal(signal.SIGTERM, stop) + signal.signal(signal.SIGINT, stop) + (STATE / 'launcher.pid').write_text(str(os.getpid())) + config = json.loads((STATE / 'config.json').read_text()) + tunnel('worker-tunnel', config['worker'], 17000, 7000) + tunnel('gateway-tunnel', config['gateway'], 17670, 17670) + # A listener must exist before OpenShell tries to connect. The gateway + # itself decides Ready only after the supervisor authenticates/admission. + for port in (17000, 17670): + for _ in range(100): + if any(p.poll() is not None for _, p in children): + raise RuntimeError('transport exited during startup') + try: + with socket.create_connection(('127.0.0.1', port), timeout=.2): + break + except OSError: + time.sleep(.1) + else: + raise RuntimeError('transport did not start') + start('supervisor', [BIN + '/openshell-supervisor', + '--backend-descriptor-file', str(STATE / 'runtime-descriptor.json'), + '--auth-bundle-file', str(STATE / 'auth.json'), '--workdir', '/sandbox'], + config['environment']) + time.sleep(2) + while True: + for name, child in children: + if child.poll() is not None: + raise RuntimeError(f'{name} exited ({child.returncode})') + (STATE / 'ready').touch() + time.sleep(.5) + + +if __name__ == '__main__': + try: + main() + except BaseException as error: + (STATE / 'failed').write_text(str(error)) + raise + finally: + (STATE / 'ready').unlink(missing_ok=True) + for _, child in children: + if child.poll() is None: + child.terminate() + for _, child in children: + try: + child.wait(timeout=3) + except subprocess.TimeoutExpired: + child.kill() + child.wait() diff --git a/templates/supervisor.ts b/templates/supervisor.ts new file mode 100644 index 0000000..c36ec5d --- /dev/null +++ b/templates/supervisor.ts @@ -0,0 +1,13 @@ +// Per-task sandbox A: policy engine + proxy. No compute driver or E2B API key. +import { Template, defaultBuildLogger } from 'e2b' +const TAG = process.env.OPENSHELL_TAG ?? 'v0.1.2' +const WSTUNNEL = 'https://github.com/erebe/wstunnel/releases/download/v11.0.0/wstunnel_11.0.0_linux_amd64.tar.gz' +const template = Template({ fileContextPath: process.cwd() }) + .fromBaseImage() + .aptInstall(['curl', 'ca-certificates', 'python3']) + .copy(`dist/${TAG}/openshell-supervisor`, '/opt/openshell/openshell-supervisor', { mode: 0o755, user: 'root' }) + .copy('templates/launch-supervisor.py', '/opt/openshell/launch-supervisor.py', { mode: 0o755, user: 'root' }) + .runCmd(`curl -fsSL ${WSTUNNEL} | tar xz -C /opt/openshell wstunnel && chmod 755 /opt/openshell/wstunnel`, { user: 'root' }) +await Template.build(template, 'openshell-supervisor-separated', { + cpuCount: 1, memoryMB: 1024, onBuildLogs: defaultBuildLogger(), +}) diff --git a/templates/workload.ts b/templates/workload.ts index 8155a92..6f5e69e 100644 --- a/templates/workload.ts +++ b/templates/workload.ts @@ -27,7 +27,7 @@ export const template = Template({ fileContextPath: process.cwd() }) 'install -d -m 0700 -o 1500 -g 1500 /.openshell/channel/sandbox /sandbox', ], { user: 'root' }) -Template.build(template, 'openshell-workload', { +Template.build(template, 'openshell-workload-separated', { cpuCount: 2, memoryMB: 2048, onBuildLogs: defaultBuildLogger(), diff --git a/tsconfig.json b/tsconfig.json new file mode 100644 index 0000000..89b81c9 --- /dev/null +++ b/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "strict": true, + "noEmit": true, + "skipLibCheck": true, + "types": ["node"] + }, + "include": ["scripts/**/*.ts", "templates/**/*.ts"] +} diff --git a/verification/separated-supervisor.md b/verification/separated-supervisor.md new file mode 100644 index 0000000..9f997ca --- /dev/null +++ b/verification/separated-supervisor.md @@ -0,0 +1,58 @@ +# Separate-supervisor integration verification + +Live E2B runs on 2026-10-23, using OpenShell v0.1.2 with the repository’s Landlock ABI v2 and eMudhra trust patches. + +## Results + +| Run | Assertions | Supervisor A | Worker B | +|---|---:|---|---| +| Standard demo, automatic reconnect | 11 passed | `iutm2avrh9u2gvqx4hpm5` | `i1x2e2rec0pk62mlf96ag` | +| Supervisor-loss failure drill | 13 passed | `if5br8r1jbihwakk3qcu5` | `i89op0hwp2qcdhwswlwrk` | + +Both used shared control plane `iveay7fm552ml9lnnpzkb`. All task VMs and the build VM were deleted after verification. The shared control plane remained available. + +Observed create-to-gateway-Ready times were 10.5 seconds for the standard run and 10.7 seconds for the failure drill. These are individual demo measurements, not a platform benchmark. + +## Live assertions + +- No supervisor process in the control plane. +- Gateway Ready with a distinct supervisor/worker pair. +- Supervisor and session credentials live in A; E2B management key stays in control. +- Unlisted destination denied before any network policy is added. +- Direct HTTPS works from A; blocked from B even as root outside the agent namespace. +- curl GET api.github.com/zen → HTTP 200. +- curl POST api.github.com/gists → HTTP 403 (blocked by OpenShell). +- wget denied for the same host: program identity enforced. +- OpenShell audit records retained across the separate supervisor channel. +- Worker runtime suspended the agent when its supervisor disappeared. +- Worker root egress remains denied after supervisor loss. +- Supervisor and worker deleted together. +- Shared control plane still available. + +The live log viewer also discovered sandbox A automatically and streamed its policy load, HTTP allow/deny and program-identity audit events alongside the shared gateway and driver logs. + +## Three-pane studio recording + +The [45-second recording](../media/separated-supervisor-demo.mp4) was captured from a live task pair on 2026-10-23. Supervisor A was `izp5c9p5qf0ddihua8d8d`; worker B was `i8yxqv6k05hd4r4t41yqy`. The 63.72-second capture is accelerated to approximately 1.42× playback at 1920×1080. + +All 14 studio assertions passed, including a heartbeat that continued through the policy update, actual GET-allow and POST-deny audit events from A, and an agent frozen at heartbeat 79 after A was terminated. B's kernel reported the agent stopped; its heartbeat remained unchanged; a fresh root HTTPS probe still failed. Both task VMs were removed and the existing shared control plane remained available. The captured results are in [studio-recording.json](studio-recording.json). + +Reproduce the recording with `npm run demo:record`, or open the interactive three-pane layout with `npm run demo:studio`. + +## Build checks + +- Rust release build succeeded in an E2B Linux builder. +- Five Rust tests passed, including partial-delete failure and rollback retry bookkeeping. +- `cargo clippy --release -- -D warnings` passed. +- `npm run typecheck` passed. +- Python supervisor launcher parsed successfully and ran in the live supervisor VM. + +## E2B templates + +| Role | Template ID | Verified build ID | +|---|---|---| +| Control | `es2xarqheh9gwbg9qh82` | `3e82ac42-b208-4483-916d-90cd7f993570` | +| Supervisor A | `pbk6uy8y2vbofhm3mruw` | `87980c14-b6d3-4c4f-a494-015ae04f45e7` | +| Worker B | `z56j2f9wjzdta7x7zn4o` | `362c5a2a-32a7-4e53-b0a7-610fe98da037` | + +Reproduce with `npm run demo:verify`. The [README](../README.md) describes setup, transport, credential boundaries and the remaining production limitations. diff --git a/verification/studio-recording.json b/verification/studio-recording.json new file mode 100644 index 0000000..eaae3a2 --- /dev/null +++ b/verification/studio-recording.json @@ -0,0 +1,33 @@ +{ + "verifiedAt": "2026-10-23T23:52:51.775Z", + "video": "media/separated-supervisor-demo.mp4", + "width": 1920, + "height": 1080, + "durationSeconds": 45.066667, + "sourceDurationSeconds": 63.72, + "name": "studio-mvlmheoz", + "control": "iveay7fm552ml9lnnpzkb", + "phase": "deleted", + "checks": [ + "Shared control plane: gateway + driver only", + "Supervisor A and worker B are distinct microVMs", + "Worker: uid 1500, zero capabilities, no supervisor auth", + "OpenShell blocks GitHub before policy is added", + "Worker root cannot bypass the external network fence", + "Policy updated live; worker keeps running", + "curl GET /zen → HTTP 200", + "curl POST /gists → HTTP 403", + "wget GET /zen → program denied", + "Allow / deny audit arrives from separate supervisor A", + "Worker is still alive; agent FROZEN at heartbeat 79", + "Root egress remains blocked after supervisor loss", + "Both task microVMs removed", + "Shared control plane stays available" + ], + "createdAt": "2026-10-23T23:50:15.971Z", + "supervisor": "izp5c9p5qf0ddihua8d8d", + "worker": "i8yxqv6k05hd4r4t41yqy", + "rootFence": true, + "frozenHeartbeat": "79", + "supervisorAuditRecords": 9 +}