emojis from docx

[steven-r]

Hi all, I've created a publishing pipeline for my novel and I'm currently fighting with emojis (example: ❤️😏📖👌😒). Emojis are dropped from word while processing and there's no chance to have them available in filters or output.

An example document is attached. It creates empty output with the following command:

pandoc emoji-test.docx --from=docx  --to=native --output /tmp/emoji.native

This is native output:

[ Para [ Str "\65039" ] ]

I'm using the lua emoji package as a workaround by entering `:wink:' ...

Any help would be appreciated.

[rnwst]

Unicode code point 65039 (U+FE0F) is Variation Selector 16, which is used to select colour variants of emojis (rather than monochrome versions). Perhaps pandoc isn't parsing emojis correctly.

You said you attached an example document, but I can't see any attachments.

[steven-r]

here we go ...
emoji-test.docx

[jgm]

Here's the XML in your docx:

  <w:body>
    <w:p w14:paraId="39905791" w14:textId="01C8BB87" w:rsidR="009D449F" w:rsidRDefault="009D449F">
      <w:r>
        <w:rPr>
          <mc:AlternateContent>
            <mc:Choice Requires="w16se"/>
            <mc:Fallback>
              <w:rFonts w:ascii="Segoe UI Emoji" w:eastAsia="Segoe UI Emoji" w:hAnsi="Segoe UI Emoji" w:cs="Segoe UI Emoji"/>
            </mc:Fallback>
          </mc:AlternateContent>
        </w:rPr>
        <mc:AlternateContent>
          <mc:Choice Requires="w16se">
            <w16se:symEx w16se:font="Segoe UI Emoji" w16se:char="2764"/>
          </mc:Choice>
          <mc:Fallback>
            <w:t>❤
            </w:t>
          </mc:Fallback>
        </mc:AlternateContent>
      </w:r>
      <w:r>
        <w:t>️
        </w:t>
      </w:r>
      <w:r>
        <w:rPr>
          <mc:AlternateContent>
            <mc:Choice Requires="w16se"/>
            <mc:Fallback>
              <w:rFonts w:ascii="Segoe UI Emoji" w:eastAsia="Segoe UI Emoji" w:hAnsi="Segoe UI Emoji" w:cs="Segoe UI Emoji"/>
            </mc:Fallback>
          </mc:AlternateContent>
        </w:rPr>
        <mc:AlternateContent>
          <mc:Choice Requires="w16se">
            <w16se:symEx w16se:font="Segoe UI Emoji" w16se:char="1F60F"/>
          </mc:Choice>
          <mc:Fallback>
            <w:t>😏
            </w:t>
          </mc:Fallback>
        </mc:AlternateContent>
      </w:r>
      <w:r>
        <w:rPr>
          <mc:AlternateContent>
            <mc:Choice Requires="w16se"/>
            <mc:Fallback>
              <w:rFonts w:ascii="Segoe UI Emoji" w:eastAsia="Segoe UI Emoji" w:hAnsi="Segoe UI Emoji" w:cs="Segoe UI Emoji"/>
            </mc:Fallback>
          </mc:AlternateContent>
        </w:rPr>
        <mc:AlternateContent>
          <mc:Choice Requires="w16se">
            <w16se:symEx w16se:font="Segoe UI Emoji" w16se:char="1F4D6"/>
          </mc:Choice>
          <mc:Fallback>
            <w:t>📖
            </w:t>
          </mc:Fallback>
        </mc:AlternateContent>
      </w:r>
      <w:r>
        <w:rPr>
          <mc:AlternateContent>
            <mc:Choice Requires="w16se"/>
            <mc:Fallback>
              <w:rFonts w:ascii="Segoe UI Emoji" w:eastAsia="Segoe UI Emoji" w:hAnsi="Segoe UI Emoji" w:cs="Segoe UI Emoji"/>
            </mc:Fallback>
          </mc:AlternateContent>
        </w:rPr>
        <mc:AlternateContent>
          <mc:Choice Requires="w16se">
            <w16se:symEx w16se:font="Segoe UI Emoji" w16se:char="1F44C"/>
          </mc:Choice>
          <mc:Fallback>
            <w:t>👌
            </w:t>
          </mc:Fallback>
        </mc:AlternateContent>
      </w:r>
      <w:r>
        <w:rPr>
          <mc:AlternateContent>
            <mc:Choice Requires="w16se"/>
            <mc:Fallback>
              <w:rFonts w:ascii="Segoe UI Emoji" w:eastAsia="Segoe UI Emoji" w:hAnsi="Segoe UI Emoji" w:cs="Segoe UI Emoji"/>
            </mc:Fallback>
          </mc:AlternateContent>
        </w:rPr>
        <mc:AlternateContent>
          <mc:Choice Requires="w16se">
            <w16se:symEx w16se:font="Segoe UI Emoji" w16se:char="1F612"/>
          </mc:Choice>
          <mc:Fallback>
            <w:t>😒
            </w:t>
          </mc:Fallback>
        </mc:AlternateContent>
      </w:r>
    </w:p>
    <w:sectPr w:rsidR="009D449F">
      <w:pgSz w:w="11906" w:h="16838"/>
      <w:pgMar w:top="1417" w:right="1417" w:bottom="1134" w:left="1417" w:header="708" w:footer="708" w:gutter="0"/>
      <w:cols w:space="708"/>
      <w:docGrid w:linePitch="360"/>
    </w:sectPr>
  </w:body>
<```
I suspect it's the AlternateContent stuff that is causing trouble here.

[jgm]

It looks like walkDocument is getting rid of most of the content.
I didn't write this part of the code, so I'm not entirely clear why it's there:

[jgm]

This comes from commit 2e8ecb3
I think there must be a better approach.

[steven-r]

So the issue ist that for AlternateContent the choices need to be checked if the Requires namespace is supported by the reader?
If not supported, take the Fallback?

[steven-r]

Did a short verification and the problem is resolved with 536d548.