Could we also use the Guided Navigation representation for read aloud?

[HadrienGardeur]

As I'm spending time working on roles and exploring what's in EPUB and ARIA, I'm starting to wonder if we shouldn't just think about read aloud as a specific use case for guided navigation.

By definition, when using the read aloud feature of a reading app, the user is literally guided throughout the content of a publication, with similar concerns for skippability and escapability.

What if we could just have a single representation with shared roles for both? This would require extending roles to structural HTML elements and some ARIA roles as well, but IMO this would make a lot of sense.

To support read aloud, we would generate a Guided Navigation objects where:

• we extract relevant semantics from the HTML
• would use the proposed object representation for text

<section role="doc-chapter" epub:type="chapter">
  <h1>Title of the chapter</h1>
  <blockquote lang="fr">Citation pour commencer cette section de la publication</blockquote>
</section>

"guided": [
  {
    "role": ["section", "chapter"],
    "children": [
      {
        "role": ["header"],
        "level": 1,
        "text": {
          "plain": "Title of the chapter",
          "language": "en"
        }
      },
      {
        "role": ["blockquote"],
        "text": {
          "plain": "Citation pour commencer cette section de la publication",
          "language": "fr"
        }
      }
    ]
  }
] 

Any thoughts on this @mickael-menu @danielweck @qnga or @chocolatkey?

[HadrienGardeur]

I've opened a separate discussion to discuss useful text fragments for this use case:

• Support for CSS selectors and positions in `textref` #6

[mickael-menu]

I would not replace the Content Iterator API with a Guided Navigation document as they serve different purposes. However, that could make sense to have a step converting the Content Iterator into a Guided Navigation document before submitting it to a navigator. To keep in mind when implementing the Media Overlays, to see if we could leverage the guided orchestration with Read aloud too.

That raises some concerns too:

• Can we represent an utterance that contains one word in a different language? E.g. "Des fancy-fair à la fraise".
• The Read aloud implementation relies on the Content Iterator being streamable and computed on-the-fly. Can we do the same thing with a Guided Navigation document, with pagination maybe?

[HadrienGardeur]

However, that could make sense to have a step converting the Content Iterator into a Guided Navigation document before submitting it to a navigator.

That's what I had in mind as well. But to do so, we'll need to make sure that the Content Iterator API can provide what we'll need. In the case of footnotes/endnotes, this might even mean parsing other XHTML resources for example.

Can we represent an utterance that contains one word in a different language? E.g. "Des fancy-fair à la fraise".

Yes, that's supported by SSML.

<speak xml:lang="fr">Des <voice xml:lang="en">fancy-fair</voice> à la fraise.</speak>

I'm not too happy with the approach of dividing such a sentence into multiple utterances, it really breaks the naturalness of the better TTS voices available on most platforms now.
I think it's something that we'll need to revisit as well.

The Read aloud implementation relies on the Content Iterator being streamable and computed on-the-fly. Can we do the same thing with a Guided Navigation document, with pagination maybe?

Up to a certain point.
In order to support escapability, you need to escape potentially several levels higher than where you currently are, which could be difficult if you paginate over a single resource.
There's also the case of FXL and PDF where you might want to calculate the Guided Navigation Document over multiple pages to get a better representation.

[qnga]

However, that could make sense to have a step converting the Content Iterator into a Guided Navigation document before submitting it to a navigator.

What would be the point of using a Content Iterator first? On mobile I don't see any. On the Web, it depends on the place where we want to generate the Guided navigation document. If it can be generated on the server-side (why not?) there's no need for a Content Iterator either on the Web.

I guess it will still be possible to share some code between the Content Iterator and the generation of the Guided navigation document, with the advantage of relaxing some constraints such as the lack of access to other XHTML resources.

[HadrienGardeur]

I guess it will still be possible to share some code between the Content Iterator and the generation of the Guided navigation document, with the advantage of relaxing some constraints such as the lack of access to other XHTML resources.

I think that's hard to say right now because we're not done documenting roles and the logic for extracting them yet. But you're absolutely right, that we could end up with requirements that are not compatible with the Content Iterator.

[mickael-menu]

What would be the point of using a Content Iterator first?

The Content Iterator is the basic building block to get the content of the publication in a format agnostic model. From there, you can have a Guided Navigation builder that takes the raw Content elements and build proper utterances, tokenizing the sentences, converting Image elements to utterances when they have an alt description, setting up the skippability groups, etc. Basically what the TtsUtteranceIterator is doing right now.

If you don't use the Content Iterator, you will need to reimplement the Guided Navigation builder for all the publication formats.

Now it's too early to tell if that means extracting most of the content iteration code into a new shared API between the Content Service and the Guided Navigation builder. But I would first try working with what's available with the Content Iterator API. Maybe like Hadrien said some requirements will not be compatible with the Content Iterator. In which case we can evolve this API to cover more use cases, or extract some shared code.

[llemeurfr]

As a general comment, I would refrain from trying to re-invent a document format (HTML) as an object serialisation syntax (json).

[HadrienGardeur]

We're definitely not going down that road because we're excluding most HTML elements from this list of roles, but it's important to convey the semantics of some structural elements and/or specific media.

For example if we have an object with just plain text and either the blockquote or pullquote roles, this could trigger a specific behaviour for read aloud where we prepend and append additional utterances to indicate: "quote" and "end of quote", like you'd normally do it verbally.

[HadrienGardeur]

Almost three months later after this initial discussion, I'm more convinced than ever that this is the right thing to do.

I'll start with a pretty straightforward set of examples:

• headers
• single language paragraph
• paragraph with a language switch in the middle
• image with an alternate description
• pagebreak
• list
• table
• footnote/endnote

I've started a separate Markdown document specifically for that and will most likely open separate discussions to cover specific cases.

[gautierchomel]

The markdown document address changed from Index to Read me