Skip to content

gofer: do not stat cached regular files during PrepareSave by default - #16474

Closed
copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl1007708632
Closed

copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl1007708632

Conversation

@copybara-service

Copy link
Copy Markdown

gofer: do not stat cached regular files during PrepareSave by default

The problem ? , refer #16208

When you run runsc checkpoint, gVisor pauses the container to save its state.

On shared mounts, gVisor was calling stat on every single cached file one by one. Each stat call sends a message to the host and waits for an answer.

This metadata was only collected to check file sizes and edit times on restore. But runsc never turns on this check by default.

If a container has thousands of files (like node_modules or Python packages), the container was frozen for seconds just doing useless stat calls.


My suggestion

We skip these stat calls during save by default.

We add an option called ValidateMetadata (set to false by default). Now, gVisor only calls stat if the user explicitly asks for it with --validate-metadata.


Code change

In pkg/sentry/fsimpl/gofer/save_restore.go:

- if !d.fs.cachedMetadataAuthoritative() {
+ if opts.ValidateMetadata && !d.fs.cachedMetadataAuthoritative() {
      if err := d.inode.updateMetadata(ctx); err != nil {
          return err
      }
  }

Stats

Cached Files Host stat Calls (Before) Host stat Calls (Now) Pause Downtime (Before) Pause Downtime (Now)
500 files 500 calls 0 ~50 ms – 150 ms < 1 ms
1,000 files 1,000 calls 0 ~100 ms – 300 ms < 1 ms
5,000 files 5,000 calls 0 ~500 ms – 1,500 ms < 1 ms
10,000 files 10,000 calls 0 1.0 s – 3.0 s ~2 ms

Added unit tests in pkg/sentry/vfs/save_restore_test.go to test that options pass correctly.
Added test and benchmark in pkg/sentry/fsimpl/gofer/gofer_test.go

fixes #16208

FUTURE_COPYBARA_INTEGRATE_REVIEW=#16346 from DARSHANR007:issue-16208/gofer-preparesave-stat fced933

@copybara-service copybara-service Bot added the exported Issue was exported automatically label Oct 27, 2026
@copybara-service
copybara-service Bot force-pushed the test/cl1007708632 branch 3 times, most recently from 6806251 to 06772f8 Compare October 27, 2026 03:56
##  The problem ? , refer #16208

When you run `runsc checkpoint`, gVisor pauses the container to save its state.

On shared mounts, gVisor was calling `stat` on **every single cached file** one by one. Each `stat` call sends a message to the host and waits for an answer.

This metadata was only collected to check file sizes and edit times on restore. But `runsc` **never turns on this check by default**.

If a container has thousands of files (like `node_modules` or Python packages), the container was frozen for seconds just doing useless `stat` calls.

---

## My suggestion

We skip these `stat` calls during save by default.

We add an option called `ValidateMetadata` (set to `false` by default). Now, gVisor only calls `stat` if the user explicitly asks for it with `--validate-metadata`.

---

## Code change

In `pkg/sentry/fsimpl/gofer/save_restore.go`:

```diff
- if !d.fs.cachedMetadataAuthoritative() {
+ if opts.ValidateMetadata && !d.fs.cachedMetadataAuthoritative() {
      if err := d.inode.updateMetadata(ctx); err != nil {
          return err
      }
  }
```

---

## Stats

| **Cached Files** | **Host `stat` Calls (Before)** | **Host `stat` Calls (Now)** | **Pause Downtime (Before)** | **Pause Downtime (Now)** |
| :--------------- | :----------------------------- | :-------------------------- | :-------------------------- | :----------------------- |
| **500 files**    | 500 calls                      | 0                           | ~50 ms – 150 ms             | < 1 ms                   |
| **1,000 files**  | 1,000 calls                    | 0                           | ~100 ms – 300 ms            | < 1 ms                   |
| **5,000 files**  | 5,000 calls                    | 0                           | ~500 ms – 1,500 ms          | < 1 ms                   |
| **10,000 files** | 10,000 calls                   | 0                           | 1.0 s – 3.0 s               | ~2 ms                    |

Added unit tests in ``pkg/sentry/vfs/save_restore_test.go`` to test that options pass correctly.
Added test and benchmark in ``pkg/sentry/fsimpl/gofer/gofer_test.go``

fixes #16208

FUTURE_COPYBARA_INTEGRATE_REVIEW=#16346 from DARSHANR007:issue-16208/gofer-preparesave-stat fced933
PiperOrigin-RevId: 1007708632
@copybara-service copybara-service Bot closed this Nov 2, 2026
@copybara-service
copybara-service Bot deleted the test/cl1007708632 branch November 2, 2026 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exported Issue was exported automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimization: Don't stat every cached gofer file during checkpoint when restore does not validate metadata

1 participant