While the tokenizer will gracefully decode most encoded characters: ``` ⇒ node > var Tokenizer = require( 'simple-html-tokenizer' ); undefined > Tokenizer.tokenize( '&' )[ 0 ].chars === '&' true ``` It doesn't handle characters whose encodings exceed 16 bits (e.g. emoji): ``` ⇒ node > var Tokenizer = require( 'simple-html-tokenizer' ); undefined > Tokenizer.tokenize( '😅' )[ 0 ].chars === '😅' false ``` It may be that `EntityParser` should use `String.fromCodePoint` in place of `String.fromCharCode` instead, or an equivalent polyfill? Related: - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/fromCodePoint - https://thekevinscott.com/emojis-in-javascript/
Activity
rwjblue commentedon Nov 20, 2018
Seems reasonable to me...
@krisselden what do you think?
krisselden commentedon Nov 21, 2018
Yes
cvx commentedon Oct 16, 2026
I believe #115 closes this issue