Commonmark_x: Issue with generating gfm_auto_identifiers

[lamyergeier]

What I want to do? I want pandoc to add unique gfm identifiers automatically to each header (with commonmark_x reader and writer)? I need this as a potential solution to Tagging each header in a markdown file · jgm/pandoc · Discussion #9875, as I plan to use these identifiers as names of array to define the tags for the header in yaml block.

---

Info

List of enabled extensions with commonmark_x

$ pandoc --list-extensions=commonmark_x | grep -v -E '^-'  
+alerts
+attributes
+bracketed_spans
+definition_lists
+emoji
+fancy_lists
+fenced_divs
+footnotes
+gfm_auto_identifiers
+implicit_header_references
+pipe_tables
+raw_attribute
+raw_html
+smart
+strikeout
+subscript
+superscript
+task_lists
+tex_math_dollars
+yaml_metadata_block

List of disabled extensions with commonmark_x

$ pandoc --list-extensions=commonmark_x | grep -v -E '^\+' 
-ascii_identifiers
-autolink_bare_uris
-east_asian_line_breaks
-hard_line_breaks
-implicit_figures
-rebase_relative_paths
-sourcepos
-tex_math_gfm
-wikilinks_title_after_pipe
-wikilinks_title_before_pipe

Observation

• By default gfm_auto_identifiers is enabled and ascii_identifiers is disabled.

Conclusion

• gfm identifier would get automatically added if it is missing.

Experiment

Sample Data

$ \cat Sample.markdown                                                                        

# Introduction ö ❤️ [^1]

[^1]: [introduction]

[introduction]: introductory text

Experiment 1

$ pandoc -f commonmark_x -t commonmark_x+ascii_identifiers Sample.markdown                      

# Introduction ö ❤️ [^1] {#introduction-ö-️-}

\[introduction\]: introductory text

[^1]: \[introduction\]

Experiment 2

$  pandoc -f commonmark_x -t commonmark_x-gfm_auto_identifiers+ascii_identifiers Sample.markdown

# Introduction ö ❤️ [^1] {#introduction-ö-️-}

\[introduction\]: introductory text

[^1]: \[introduction\]

Observation

• Experiment 1 and 2 - The identifier is neither ascii nor gfm (as the emoji is missing from identifier!)

Questions

1. Why both experiment don't output ascii identifier when it is enabled in the writer explicitly?
2. Why Experiment 2 works? auto_identifiers are not defined for commonmark_x and gfm_auto_identifiers was disabled.
3. What is the correct method to make writer add gfm_auto_identifier which has only ascii characters ` to commonmark_x output?
4. Is gfm_auto_identifiers superior to auto_identifiers as auto_identifiers is not available in commonmark_x?

[jgm]

There are several separate issues here which we should try to untangle:

1. You claim that the algorithm for gfm auto identifiers should not omit emojis. This may have been something that changed since the code was written, because there is explicit code to remove emojis if gfm or ascii auto identifiers are selected. Can you point to documentation about this algorithm?
2. The identifiers are created in the readers, not the writers, so if you want ascii identifiers you should put the extension on commonmark_x in -f not -t. Then you will get an ascii-only identifier. [EDITED: this works.]
3. You can specify ascii_identifiers in the writer, but the only effect this has is to help pandoc decide whether to omit the explicit identifier. (It will do that if it matches the identifier that would be generated automatically with the selected options.) Identifiers are never generated in the writers. So, note the difference:

% pandoc -f commonmark_x+ascii_identifiers -t commonmark_x+ascii_identifiers
# Introduction ö ❤️ 
^D
# Introduction ö ❤️
% pandoc -f commonmark_x+ascii_identifiers -t commonmark_x                 
# Introduction ö ❤️ 
^D
# Introduction ö ❤️ {#introduction-o-}

In the second case an explicit identifier is generated because the generated one (with this selection of options) would be different from the one the reader produces.

[lamyergeier]

From: https://pandoc.org/MANUAL.html

Extension: gfm_auto_identifiers

Changes the algorithm used by auto_identifiers to conform to GitHub’s method. Spaces are converted to dashes (-), uppercase characters to lowercase characters, and punctuation characters other than - and _ are removed. Emojis are replaced by their names.

---

So, I was expecting the following:

{#introduction-o-heart}

[jgm]

Thank you. There were some bugs here; I've pushed fixes and they'll be in the next release.