Linux: fs.writeFile() can result in empty reads for external watchers due to truncation

VSCode Version: 0.10.11 Commit f291f4a
OS Version: Linux 4.2.0-34-generic broken selection feedback when selecting bold text in markdown files #39~14.04.1 (Ubuntu)

Steps to Reproduce:

Open known good partial (dust template), e.g, the menuItem.toggle.dust.emacs attached
Add and delete a space, then save.
Re-run dust rendering

The output will be missing several elements. Given the dust template I'm using, I believe it may be a failure in the eq test. BUT I know this isn't a Dust.js problem because it works if I save it from emacs:

Steps to fix:

Open partial saved above, e.g., the menuItem.toggle.dust.code attached, in a good editor, e.g., emacs.
Add and delete a space, then save
Re-run dust rendering.

The output will be as expected. I was able to toggle back & forth between the two editors and consistently reproduce the results. What emacs saved worked fine. What code saved did not.

I've attached the two partials. I'm guessing it's a unicode/text format issue. None of my diff tools showed any difference, but CLEARLY dust.js is treating one differently from the other.

menuItem.toggle.dust.zip

FWIW, here is the template in those attached files:

{@eq key=widget value="true"}
{:else}
<div class="text"><span>{text}</span></div>
<div class="toggleBox">
    <div class="toggleRight"> 
        <div class="text"><span>{true.text}</span></div>
        <img class="toggle" src="../../images/menu/toggle_right.png">
    </div>
    <div class="toggleLeft">
        <div class="text"><span>{false.text}</span></div>
        <img class="toggle" src="../../images/menu/toggle_left.png">
    </div>
</div>
{/eq}

Joe Andrieu (@jandrieu)

Joe Andrieu (@jandrieu) I can confirm the two files are byte-to-byte equal.

They also do not contain any special characters (all char codes are <=127), so I do not think this is an encoding issue.

I use the following code to check the two files:

var fs = require('fs');

var bytes1 = fs.readFileSync('menuItem.toggle.dust.code');
var bytes2 = fs.readFileSync('menuItem.toggle.dust.emacs');

// Check equality
if (bytes1.length !== bytes2.length) {
    console.log('different lengths');
    process.exit(0);
}
for (var i = 0; i < bytes1.length; i++) {
    if (bytes1[i] !== bytes2[i]) {
        console.log('different byte at offset ' + i);
        process.exit(0);
    }
}

console.log('byte-to-byte equality');

// Look for non-ASCII chars
var asString = bytes1.toString('utf8');
for (var i = 0; i < asString.length; i++) {
    var chCode = asString.charCodeAt(i);

    if (chCode >= 0 && chCode <= 127) {
        // nothing special
        continue;
    }
    console.log('found non-ASCII char with chCode: ' + chCode);
}
console.log('finished looking for non-ASCII chars');

And I get the following output:

byte-to-byte equality
finished looking for non-ASCII chars

Can you please provide more context? Do you run the dust rendering manually after saving the file or does it run on file change events.

Perhaps the problem is transient (i.e. the file is available on disk with 0 bytes for a short moment and then it gets filled up with content). If the dust rendering runs on file changes, perhaps it reads at a time when the file is empty (content not yet flushed by VSCode).

Come to think of it, I think I have experienced something similar while selfhosting on linux with VSCode. Daniel Imms (@Tyriar) Martin Aeschlimann (@aeschli) Does the gulp watch sometime report "File xyz.ts is not a module" for you?

Benjamin Pasero (@bpasero) How do we save files to disk on Linux?

alexandrudima

alexandrudima using fs.writeFile: https://github.com/Microsoft/vscode/blob/master/src/vs/workbench/services/files/node/fileService.ts#L265

But there is a case where we encode the contents of the file if the encoding is not UTF 8.

Maybe emacs goes for the old file rename trick when saving (to get an atomic save):

write to a tmp file
maybe remove destination file
rename tmp file as destination file

alexandrudima

alexandrudima I don't think I've seen that before.

Daniel Imms (@Tyriar)

Daniel Imms (@Tyriar) This is what I often see while selfhosting (gulp watch) on Ubuntu. It happens most often when I do git operations and sometimes when I type (with autosave enabled).

If I then do another change in the file, it gets picked up. I always thought it's the fault of our gulp watch scripts, but maybe it's a timing thing and gulp ends up reading the file as empty.

*** Error: /home/alexdima-qwerty/Desktop/src/vscode/src/vs/editor/browser/viewParts/linesDecorations/linesDecorations.ts(11,48): File '/home/alexdima-qwerty/Desktop/src/vscode/src/vs/editor/browser/viewParts/glyphMargin/glyphMargin.ts' is not a module.

alexandrudima

alexandrudima ah, I tend to stop gulp watch when I do a destructive git command and I also don't use autosave.

I was able to reproduce the problem. The error is that the file is being emptied and saved, then saved again with the new content. As expected it has nothing to do with dust. Fortunately, I'm using fs.watch, so it was easy to isolate. Here's the output of running the attached watcher.js script on the attached watched.txt file as I edit the latter (just adding/deleting spaces so sadly, the edits aren't that obvious in the output).

watched.zip

First, editing and saving (twice) with VSCode:

joe@dev-frontend:~/Software/codeTest$ ./watcher.js 
watching for changes
watched.txt changed

watched.txt changed
This is arbitrary text.  
watched.txt changed

watched.txt changed
This is arbitrary text.

Then, when editing and saving (four times) with emacs

watching for changes
watched.txt changed
This is arbitrary text. 
watched.txt changed
This is arbitrary text. 
watched.txt changed
This is arbitrary text.  
watched.txt changed
This is arbitrary text.

Joe Andrieu (@jandrieu)

Joe Andrieu (@jandrieu) I can reproduce with VS Code and other editors that use node for writing the file. It looks like the basic file watching facility in node.js on Linux reports 2 change events when writing to a file (see nodejs/node#3042) and the first time you read from the file it is likely empty and only after gets the content filled in.

Are you able to switch to a better file watching library for your project? VS Code itself is using chokidar (https://github.com/paulmillr/chokidar) and I think they do a good job in normalizing duplicate file events like this one.

Ok. If you want the gory details, go visit that other issue. I tests a few alternatives and found it isn't just node.js' fault.

The short version is that if VSCode wants clean file watch events, we'll need to do the tmp file / rename trick--or something similar. It isn't so much a bug in node as it is a limitation of the operating system, one that we'll have to work around if we want clean behavior.

I searched for a problem with gulp.watch sometimes failing to read sass files, and it seems a common error that is resolved by people using Sublime, to set its property atomic_save to true.
That is using the write tmp file, rename -trick. However, atomic save seems to generate lots of other unwanted side effects.

Note: I got this problem with VSCode 1.6.1 and gulp.watch on Windows, so not only Linux.

Duplicate of #56361

Navigation Menu

Repository navigation

Linux: fs.writeFile() can result in empty reads for external watchers due to truncation #4727

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Linux: fs.writeFile() can result in empty reads for external watchers due to truncation #4727

Description

Activity

alexdima commented on Mar 29, 2016

bpasero commented on Mar 29, 2016

alexdima commented on Mar 29, 2016

Tyriar commented on Mar 29, 2016

alexdima commented on Mar 29, 2016

Tyriar commented on Mar 29, 2016

jandrieu commented on Mar 30, 2016

bpasero commented on Mar 30, 2016

10 remaining items

jandrieu commented on Apr 10, 2016

janneromell commented on Oct 25, 2016

bpasero commented on Sep 5, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions