typescript/ast/scanner: scan() never advances past # not followed by an identifier start (7.1.0-beta) #65075

🔎 Search Terms

createScanner, scan, PrivateIdentifier, #, infinite loop, scanner does not advance, typescript/ast/scanner

🕗 Version & Regression Information

• Reproduces in typescript@7.1.0-beta (typescript/ast/scanner). The JS scanner is used through the new API entry points.
• Does not reproduce in typescript@5.9.2 (ts.createScanner).
• Node 22.22.3, Linux x64.

⏯ Playground Link

No response

💻 Code

// node repro.mjs
import { createScanner } from "typescript/ast/scanner";
import { SyntaxKind } from "typescript/ast";

for (const text of ["a # b", "# ", "#1"]) {
  const scanner = createScanner(/*skipTrivia*/ true, /*languageVariant*/ undefined, text);
  const tokens = [];
  for (let i = 0; i < 6; i++) {
    const kind = scanner.scan();
    tokens.push(`${SyntaxKind[kind]}@${scanner.getTokenStart()}-${scanner.getTokenEnd()}`);
    if (kind === SyntaxKind.EndOfFile) break;
  }
  console.log(JSON.stringify(text), tokens.join(", "));
}

🙁 Actual behavior

When # is not followed by an identifier start, scan() returns PrivateIdentifier with an empty range and never advances, so it never reaches EndOfFile:

"a # b" Identifier@0-1, PrivateIdentifier@2-2, PrivateIdentifier@2-2, PrivateIdentifier@2-2, PrivateIdentifier@2-2, PrivateIdentifier@2-2
"# "    PrivateIdentifier@0-0, PrivateIdentifier@0-0, ...
"#1"    PrivateIdentifier@0-0, PrivateIdentifier@0-0, ...

Any loop of the form while (scanner.scan() !== SyntaxKind.EndOfFile) hangs on such input.

🙂 Expected behavior

The scanner consumes the # and moves on, as typescript@5.9.2 does:

"a # b" Identifier@0, PrivateIdentifier@2, Identifier@4, EndOfFileToken@5
"# "    PrivateIdentifier@0, EndOfFileToken@2
"#1"    PrivateIdentifier@0, NumericLiteral@1, EndOfFileToken@2

(5.9 also reports an "Invalid character" error for the lone #.) At minimum, every scan() call should advance the position or return EndOfFile.

Additional information about the issue

Input like this is common in tooling that scans arbitrary text, such as comments, markdown-ish strings, and partial edits.