Repository navigation
fix(tts): speak <sub>/<sup> content instead of discarding it - #3779
whosawhatsis wants to merge 3 commits into
Conversation
|
Thank you very much! |
| ttsQueue.push(current); | ||
| } | ||
|
|
||
| if (pendingSubSupSeparator) { |
There was a problem hiding this comment.
Could we just remove the pendingSubSupSeparator global state and instead test lowerTagName === "sup" || lowerTagName === "sup" here?
(see above const lowerTagName = textNode.parentElement?.tagName?.toLowerCase();)
| // length here, and the leading one is subtracted back out (with the result | ||
| // clamped into the node) when converting an utterance offset to a DOM offset. | ||
| // eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
| const isSUBSUP = !isRUBY && (txtNode as any).__SUBSUP && !isOnlyWhiteSpace(txtNode.nodeValue); |
There was a problem hiding this comment.
I think that the !isRUBY condition is superfluous here, as the TextNode cannot have both ruby/rb and sup/sub parent Elements.
| const nodeLen = txtNode.nodeValue.length; | ||
|
|
||
| const l = isRUBY ? 0 : isOnlyWhiteSpace(txtNode.nodeValue) ? 1 : txtNode.nodeValue.length; | ||
| const l = isRUBY ? 0 : isOnlyWhiteSpace(txtNode.nodeValue) ? 1 : (sep * 2) + nodeLen; |
There was a problem hiding this comment.
I appreciate that "readability" is subjective, but for future code reviewers I think it would be better if the existing isRUBY ternary was expanded with isSUBSUP (this is perhaps even more obvious further below, where the proposed Min/Max statements are hard to correlate intuitively with the isSUBSUP case vs. the general case):
I think this captures the intent more obviously, at a quick glance:
const l = isRUBY ?
0 :
isOnlyWhiteSpace(txtNode.nodeValue) ?
1 :
isSUBSUP ?
(txtNode.nodeValue.length + 2) :
txtNode.nodeValue.length
| rangeStartNode = txtNode; | ||
| rangeStartOffset = isRUBY ? 0 : l - (acc - charIndexAdjusted); | ||
| rangeStartOffset = isRUBY ? 0 : | ||
| Math.min(Math.max(l - (acc - charIndexAdjusted) - sep, 0), nodeLen); |
There was a problem hiding this comment.
May I propose this simpler cascading expression which offers the advantage of not altering the existing implementation (i.e. the non-isSUBSUP case):
rangeStartOffset =
isRUBY ?
0 :
isSUBSUP ?
Math.min(Math.max(l - 1 - (acc - charIndexAdjusted), 0), txtNode.nodeValue.length) :
(l - (acc - charIndexAdjusted));
| rangeEndNode = txtNode; | ||
| rangeEndOffset = isRUBY ? (txtNode.nodeValue.length - 1) : l - (acc - charIndexEnd); | ||
| rangeEndOffset = isRUBY ? (nodeLen - 1) : | ||
| Math.min(Math.max(l - (acc - charIndexEnd) - sep, 0), nodeLen); |
There was a problem hiding this comment.
rangeEndOffset =
isRUBY ?
(txtNode.nodeValue.length - 1) :
isSUBSUP ?
Math.min(Math.max(l - 1 - (acc - charIndexEnd), 0), txtNode.nodeValue.length) :
(l - (acc - charIndexEnd));
<sub> and <sup> were listed in _doNotProcessDeepChildTagNames, so their text was never collected into a ttsQueueItem and never spoken. Unlike the other non-descended elements on that list (img, svg, math), they had no alternative text path either, so the content was silently lost with no way for an author to supply a spoken form. In scientific prose this changes the statement rather than just its phrasing: "pi r squared" was read as "pi r". - remove "sup"/"sub" from _doNotProcessDeepChildTagNames and add !isSubSup to the deep-dive condition, so the text joins the surrounding utterance and word-level highlighting keeps working - add an aria-label/title override, following the linkNeedsDeepDive idiom, so authors can supply better phrasing (<sup aria-label="squared">2</sup>) - add "noteref" to _skippables, so footnote reference markers remain skipped via the mechanism intended for it (and under the user's existing skippability setting) rather than as a side effect of the tag blacklist
Speaking sup/sub content is not sufficient on its own: combineTextNodes() joins
raw nodeValues with no separator, so a subscript or superscript and the text
around it arrive at the speech engine as a single token.
10<sup>23</sup> -> "1023" spoken "one thousand twenty-three"
<em>R</em><sub>o</sub> -> "Ro" one syllable, indistinguishable from R_i
<em>t</em><sub>o</sub> -> "to" the ordinary word
10<sub>23</sub>45 -> "102345" a wholly invented number
<sub>θ</sub>x -> "θx" dropped entirely - speech engines discard a
token mixing Greek and latin script
Measured on a physics textbook: of 172 prose sub/sup sites, 8 fused into a false
number and 40 into a false word. This is very likely why sup/sub were on
_doNotProcessDeepChildTagNames in the first place - a blunt guard against
producing garbage - so simply removing them from that list is not enough.
Text inside sup/sub is now fenced by SUBSUP_SEPARATOR on both sides when the
utterance string is assembled. Both sides are needed: a leading separator alone
still lets the subscript fuse with whatever follows it. The separators exist
only in that string and never in the DOM, so rendering, CFIs and locators are
untouched.
This follows the existing __RUBY convention: a text node is tagged, and both
combineTextNodes() and the two offset walkers in readaloud.ts agree on the
length that node contributes. The walkers subtract the leading separator back
out and clamp into the node when converting an utterance offset to a DOM
offset, so word- and sentence-level highlighting stay aligned - verified by
mapping every utterance offset back to its DOM character across fusion,
non-fusion and nested-element fixtures.
- drop the pendingSubSupSeparator module state; tag the text node from the DOM
instead, in the same place and shape as the existing __RUBY tagging
- use closest("sup, sub") rather than a direct parent-tag test, so inline markup
inside a sup/sub still gets fenced: <sup><a href="#fn1">1</a></sup> and
<sup><em>n</em></sup> read as fused with a direct parent test
- __RUBY and __SUBSUP are mutually exclusive by construction, so drop the
redundant !isRUBY guard
- keep the existing cascading-ternary shape in both offset walkers and leave the
non-isSUBSUP path byte-identical to develop: only the isSUBSUP branch clamps,
where previously the clamp applied to every text node
Reported-by: Daniel Weck <daniel.weck@gmail.com>
59e3ccb to
5d7cfad
Compare
|
Thank you for the detailed review — all five comments are applied and the branch is rebased onto The cascading-ternary shape (comments 3–5): you caught an actual regression, not just a readability issue. My version wrapped The
if (textNode.nodeValue.trim().length && textNode.parentElement?.closest("sup, sub")) {
(textNode as any).__SUBSUP = true;
}Happy to switch to the direct test if you would rather keep it strictly parallel to Comment 2 ( Comment 2's premise, verified: a text node cannot have both a After the pass:
No rush on this given 3.5.0 — just flagging that the branch is updated so you are not reviewing the old diff. |
Fixes #
Part 1 of 2. # is stacked on this branch and adds a
ttsSubSupEnabledpreference (default on) as an opt-out for the behaviour change noted at the bottom.
This PR stands alone if you would rather not take the preference.
What
<sub>and<sup>text was never spoken by read-aloud. In scientific prose thisdoes not merely sound awkward, it changes the statement:
πr<sup>2</sup>was readas "pi r",
v<sub>1</sub>as "v",SO<sub>4</sub><sup>2-</sup>as "SO".Three changes in
src/r2-xxx-js/r2-navigator-js/electron/renderer/common/dom-text-utils.ts:"sup"/"sub"removed from_doNotProcessDeepChildTagNames, with!isSubSupadded to the deep-dive condition. The elements are now descendedinto like any other inline element, which keeps their text in the surrounding
utterance —
v<sub>1</sub>reads as one phrase, and thetextNodesarray stayspopulated so word-level highlighting keeps working. (Pushing a separate queue
item, as
imgdoes, would have fragmented every sentence containing an index.)An
aria-label/titleoverride, following the existinglinkNeedsDeepDiveidiom, so authors can supply better phrasing:<sup aria-label="squared">2</sup>. Consistent with themathbranch a fewlines below, which prefers
alttextand falls back totextContent."noteref"added to_skippables. This is what makes (1) safe: footnotereference markers were previously suppressed only as a side effect of the tag
blacklist.
_skippableslisted note bodies (footnote,endnote,note,rearnote) but never the reference.computeEpubTypesalready resolvesepub:type="noteref"and normalisesrole="doc-noteref"→noteref, so onlythe list entry was missing. Correctly-marked references stay silent, now via the
mechanism intended for it and under the user's existing skippability control.
Why not just add an alt-text fallback and leave the blacklist alone
That would have left every existing book silent, since the fallback only fires when
the attribute is present — the fix would have required every publisher to re-mark
their content. The
mathbranch already establishes the better pattern in this samefunction: attribute if provided, content otherwise.
Verification
Confirmed by ear in a running build (this branch on
develop), on a real EPUB, forboth fusion classes:
…two atoms of oxygen (written O<sub>2</sub>), we have 6.0214076 * 10<sup>23</sup>……relative sizes of r, R<sub>o</sub>, and R<sub>i</sub>… R<sub>o</sub> = 0.05 m, R<sub>i</sub> = 0.04 mA small test EPUB covering all seventeen cases is linked from # (with the
script that generates it). Running
generateTtsQueue()over that file fromdevelopand from this branch gives:
v₁ v₂R_o R_it_o10²³H₂Ox₁²aria-label="squared"epub:type="noteref"role="doc-noteref"<sup>markerdisplay:noneTests 10, 11, 13 and 14 are the regression cases and are unaffected. Test 12 is the
single behaviour change, covered by the preference in #.
Plus:
tsc --noEmit -p tsconfig.json— exit 0, zero diagnosticseslinton the changed file — clean, identical to baselinegenerateTtsQueue()bundled from this branch and fromdevelop, run against jsdom fixtures, comparingcombinedText. At the defaultsetting (skippability on):
π<em>r</em><sup>2</sup>"the area is πr exactly""the area is πr2 exactly"<em>v</em><sub>1</sub> … <sub>2</sub>"compare v with v""compare v1 with v2"SO<sub>4</sub><sup>2-</sup>"the SO ion""the SO42- ion"μ<sub>s</sub>"use μ for static""use μs for static"<sup aria-label="squared">2</sup>"the area is r here""the area is r","squared","here"<sup><a epub:type="noteref">1</a></sup>"as shown earlier"<sup><a role="doc-noteref">1</a></sup>"as shown earlier"<sup><a href="#fn1">1</a></sup>"as shown earlier""as shown1 earlier"<sup style="display:none">9</sup>"visible text"I can contribute the fixture harness as a jest test if you want it in-tree; it
needs
jest-environment-jsdom, which isn't currently a devDependency, so I left itout of this PR rather than expand the dependency set unasked.
Why removing the tags from the list is not sufficient on its own (commit 3)
combineTextNodes()joins rawnodeValues with no separator, so a base and itssub/superscript reach the speech engine as one token:
develop10<sup>23</sup><em>R</em><sub>o</sub><em>t</em><sub>o</sub><em>x</em><sup>2</sup>Measured on a 12-chapter physics textbook: of 172 prose
sub/supsites, 8 fusedinto a false number and 40 into a false word. That is 28% of sites producing
confidently wrong output — which is very likely why
sup/subwere put on_doNotProcessDeepChildTagNamesto begin with.This is not platform-specific. Measured on all three desktop platforms, each
through the Web Speech API so it is that platform's real engine — macOS Samantha,
espeak-ng via speech-dispatcher on Linux, SAPI
Speech_OneCoreunder Chrome on Windows:we have 1023 atomswe have one thousand twenty three atomswe have 10 23 atomscompare Ro with Ri/compare R o with R ithe time to matters/the time t o mattersOn every engine
1023matches the spelled-out number rather than the split digits, andthe fused letters run shorter than the separated ones — the digits are parsed as one
number and the letters as one syllable everywhere, so the separator is needed on all
three. It reads as a blunt guard against
garbage rather than an oversight, so this PR would not be defensible without
addressing it.
So text inside
sup/subnow contributes a leadingSUBSUP_SEPARATORwhen theutterance string is assembled. The separator exists only in that string, never in
the DOM, so rendering, CFIs and locators are untouched.
Why a plain space, and not a zero-width character
U+FEFF is the obvious alternative, and on macOS it works exactly as well — timing
synthesised speech,
10<FEFF>23and10 23come out within 4 ms of each other inevery case I tried. I still went with a space, for two reasons.
It cannot be discarded. U+FEFF is
Default_Ignorable_Code_Point, so an engine thatstrips it before tokenising is doing what the character's definition invites, not
misbehaving. Were an engine to do that, this fix would silently do nothing there and
the failure would look exactly like the bug it was meant to fix. A space has no such
licence anywhere in the chain.
And the separator is visible to the user in a place where visibility helps. The
utterance string is what the read-aloud panel displays, and that panel shows what is
about to be spoken, not what the book looks like — the sub/sup distinction is already
gone from it.
H 2 Otruthfully shows the separation being applied;H2Owould renderidentically whether the subscript were separated, fused, or dropped altogether, which is
precisely the ambiguity that made this bug hard to notice in the first place.
This follows the existing
__RUBYconvention exactly: tag the text node, and havecombineTextNodes()and the two offset walkers inreadaloud.tsagree on the lengththat node contributes. The walkers subtract the separator back out when converting an
utterance offset into a DOM offset, so word- and sentence-level highlighting stay
aligned. I verified that by replicating the walker and mapping every utterance
offset back to its DOM character across fusion, non-fusion and nested-element
fixtures — 0 mismatches.
Note this makes the utterance say "ten, twenty-three" rather than "ten to the
twenty-third". Correct pronunciation of an exponent needs semantics, which is what
the
aria-labeloverride in commit 2 is for; the separator's job is only to stop thereader inventing a different number.
Behaviour change / release note
Note references marked up as a bare
<sup>with noepub:typeorrolewill nowhave their marker spoken. Correctly-marked references are unaffected at the default
setting, since
"noteref"is now in_skippablesand skippability defaults to on(
mediaOverlaysEnableSkippability: truedrivesttsSkippabilityEnable).# adds a
ttsSubSupEnabledpreference (default on) as an explicitopt-out for that case; switching it off reproduces current
developoutputbyte-for-byte on every fixture above.
The
aria-labeloverride emits its own queue item, so it splits the sentence intothree utterances — the same as the existing
img/link/pagebreak overrides, but moreaudible inline. Change 2 can be dropped independently of 1 and 3 if you would
rather not have that.