Skip to content

Add xArm6 robot learning workshop (teleop, ACT training, deploy) - #5493

Draft
Nick Hehr (HipsterBrown) wants to merge 28 commits into
mainfrom
xarm-act-stacking-workshop
Draft

Nick Hehr (HipsterBrown) wants to merge 28 commits into
mainfrom
xarm-act-stacking-workshop

Conversation

@HipsterBrown

@HipsterBrown Nick Hehr (HipsterBrown) commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR adds a new sequential workshop, Teach an xArm6 to Stack Blocks, for an upcoming hackathon. It walks through end-to-end robot learning. A person teleoperates an xArm6 follower with an SO-ARM101 leader, and Viam records each demonstration. Participants train an ACT (Action Chunking with Transformers) policy with LeRobot on a cloud GPU, then deploy it on the machine. The leader and follower are different arms on purpose, to show mixed hardware on one platform.

flowchart LR
    T["Teleop<br/>SO-ARM101 → xArm6<br/>(Cartesian teleop)"] --> R["Record<br/>capture control sensor<br/>+ capture-control webapp"]
    R --> C["Curate<br/>review:keep →<br/>sequence dataset"]
    C --> Tr["Train<br/>custom ACT script<br/>(cloud GPU)"]
    Tr --> D["Deploy<br/>viam-labs:vla<br/>policy + controller"]
Loading
 docs/tutorials/
 ├── pick-and-place/
 ├── so-arm101-palletizing/
+└── xarm-act-stacking/
+    ├── _index.md                        # overview, hardware, plan-your-day table
+    ├── _phase-template.md               # headless
+    ├── 01-platform-mental-model.md
+    ├── 02-configure-the-cell.md         # fragment, teleop mapping, home poses
+    ├── 03-record-demonstrations.md      # capture control + webapp
+    ├── 04-review-and-build-a-dataset.md # keep/discard, sequence dataset
+    ├── 05-train-a-policy.md             # milestone one
+    ├── 06-deploy-the-policy.md          # milestone two
+    ├── 07-evaluate-and-iterate.md       # optional
+    └── 08-wrap-up.md
 .htmltest.yml                           # ignore pending companion repo

The pages use the existing workshop format: front matter registers the workshop, and the pages use the checkpoint and workshop-nav shortcodes. This PR changes no layout or shortcode files. Every page is draft: true until the dependencies below land. The capability tags are all existing ones, so docs-health needs no change.

Evidence

  • Before: /tutorials/ shows two workshop cards (pick-and-place, so-arm101-palletizing).
    After: a draft build (hugo --buildDrafts) shows a third card. The nav chain renders overview → Phases 1–7 → wrap-up, each page shows "Phase N of 7", and no phase pages leak into /tutorials/all/. A production build excludes the draft pages.
  • prettier 3.2.5, markdownlint, vale, scripts/frontmatter_rules.py, and make build-prod all pass. Local vale was 3.26.0, not the 3.12.0 that CI pins, so CI's run is the source of truth.
  • Each page passed a spec-compliance and quality review against its sources: the vla-inference-service, smolvla-so101-training, and capture-control-webapp READMEs and code, and the teleop-cartesian doc. A final pass reviewed the whole branch for cross-page consistency and safety.

Before publishing (TODO(dep) markers)

Blocking milestone two:

  • viam:ufactory:gripper needs a {"set": N, "wait": false} option on DoCommand. Today set polls until the jaw settles, up to 10 seconds, which stalls the 10 Hz control loop (06:266). Add optional non-blocking gripper "set" DoCommand viam-modules/viam-ufactory-xarm#145 adds it. Once it's released, add "write_args": {"wait": false} to the Phase 6 controller's gripper config and remove the "arm freezes at a gripper change" troubleshooting row.
  • An ACT checkpoint loads and runs through viam-labs:vla:policy. Verified on vla 0.2.1 on an Intel NUC: the policy reached ready, its specs were correct, and one infer returned 100 actions in 0.22 s on the CPU.

Teleop module extraction:

  • Module ID, model name, and attributes (02:193), and a docs link (02:239).
  • What rejected counts (02:269).
  • Recheck the minimum viam-server version, currently v0.119.0 (_index:103).

ACT training script and companion repo (viam-devrel/xarm-act-stacking):

  • Publish the training script. It's public as devrel:act-policy 0.1.0, a general ACT training script, with the Robot policy task type. Its arm, gripper, and cameras arguments are required.
  • Make the companion repo public. It's private for now.
  • The script name, the container, the argument names, and the defaults: steps 50,000 and image_size 0. These are measured on an A100, at 0.081 s per step. Phase 5 passes the xArm arguments explicitly.
  • The script builds its input features from the dataset, so the feature keys follow the camera names.
  • Mat zone count (07:27).
  • Public fragment ID, if one is published (02:49).
  • cell.json must set the teleop gripper_invert: true and leader_tool_offset_mm, because Phase 2 tells readers they're already set.

Version pins:

  • viam-labs:vla 0.2.1, which matches the training container's LeRobot commit.
  • viam:capture-control (03:27).

Verify on hardware:

  • xArm gripper open and closed values, about 840 and 2 (05:54, 06:143).
  • state_dim and action_dim of 7 on an xArm checkpoint (06:135). They were 6 and 6 on the 5-joint SO-101 checkpoint, as expected.
  • xArm6 joint_limits_degs (06:192).
  • Measured actions_per_chunk (06:214). On a NUC's CPU, the floor is 3.
  • RealSense color-only attribute (02:118).
  • Cameras with different resolutions. The script expects them to work, but the validation dataset's cameras were both 640x480.

Platform checks:

  • The capture override keeps the gripper's additional_params.docommand_input (03:94).
  • Saving a pose to a switch defined in a fragment works (02:256).
  • The "All sequences" dialog can filter by tag (04:76).
  • The hosted webapp URL is live (03:125).

Also:

  • Replace the ASSET placeholders with photos, screenshots, and diagrams.
  • Do a full dry run on real hardware to set the time estimates.
  • Set draft: false.

Merge Danger

Door: two-way

All pages are drafts, so merging publishes nothing. Reverting is a plain revert.

Blast Radius: none

The only change outside the workshop directory is one IgnoreURLs entry in .htmltest.yml, for the companion repo that doesn't exist yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
@viambot viambot added the safe to build This pull request is marked safe to build from a trusted zone label Nov 3, 2026
@netlify

netlify Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for viam-docs ready!

Name Link
🔨 Latest commit 4d680f4
🔍 Latest deploy log https://app.netlify.com/projects/viam-docs/deploys/6aeba6490cd10ba44d23ffa5
😎 Deploy Preview https://deploy-preview-5493--viam-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
1 paths audited
Performance: 31 (🔴 down 3 from production)
Accessibility: 100 (no change from production)
Best Practices: 100 (no change from production)
SEO: 92 (no change from production)
PWA: 60 (no change from production)
View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify project configuration.

Phase 5: name the act-xarm script and the container, document the
script's defaults (which match the workshop, so no arguments are needed),
and replace the broken cameras quoting with + separators. Add the
clock-camera failure and the measured A100 training time.

Phase 6: pin viam-labs:vla 0.2.1, note that its setup needs git and
curl, close the ACT-loads and input-features TODOs, correct the
n_action_steps note, and add a measured CPU latency for the floor.

Phase 7 and overview: fill in the steps and image_size defaults, and
list git and curl as prerequisites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb
The training script has nothing xArm-specific beyond its defaults, so it
is now published as devrel:act-policy with the Robot policy task type.
Its arm, gripper, and cameras arguments are required and its gripper
defaults follow a percentage gripper, so Phase 5 passes the xArm values
explicitly again, in the app table and the CLI command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FAJrDL3amKcRmphzeVnyb

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to build This pull request is marked safe to build from a trusted zone

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants