Changes to tags backend to expand macros having undesirable side effects

[james-ball-qualcomm]

In PR #131, it has made some changes to the tagged text that aren't good. Here's the diff from my unit level test comparing the old and the new:

CHECKING BUILT TAGS AGAINST EXPECTED TAGS
diff tests/norm-rule/expected/test-norm-tags.json build/test-norm-tags.json && echo "diff PASSED" || (echo "diff FAILED"; exit 1)
5,6c5,6
<     "norm:clarification-without-text": "isn't clear enough.",
<     "norm:clarification-with-text": "also isn't clear enough.",
---
>     "norm:clarification-without-text": "isn&#8217;t clear enough.",
>     "norm:clarification-with-text": "also isn&#8217;t clear enough.",
15,16c15,16
<     "norm:bold": "ABC is a network - Bold is removed by tags backend so I don't see it",
<     "norm:italics": "Let's have fun today - Italics is removed by tags backend so I don't see it",
---
>     "norm:bold": "*ABC* is a n*et*work - Bold is removed by tags backend so I don&#8217;t see it",
>     "norm:italics": "Let&#8217;s have _fun_ to_day_ - Italics is removed by tags backend so I don&#8217;t see it",
19c19
<     "norm:inline-underline": "ABC [.underline]#inside tag# GHI",
---
>     "norm:inline-underline": "ABC inside tag GHI",
26,32c26,32
<     "norm:macro_call": "200 {ge} 100",
<     "norm:hyperlink1": "ABC &lt;&lt;norm:superscript&gt;&gt; DEF",
<     "norm:hyperlink2": "DEF &lt;&lt;norm:superscript,custom text&gt;&gt; GHI",
<     "norm:hyperlink3": "ABC &lt;&lt;non-norm-anchor&gt;&gt; DEF",
<     "norm:hyperlink4": "DEF &lt;&lt;non-norm-anchor,custom text&gt;&gt; GHI",
<     "norm:hyperlink5": "GHI &lt;&lt;norm:superscript&gt;&gt; and &lt;&lt;norm:subscript&gt;&gt; JKL",
<     "norm:hyperlink6": "JKL &lt;&lt;norm:superscript ,hello&gt;&gt; and &lt;&lt;norm:subscript, goodbye&gt;&gt; MNO",
---
>     "norm:macro_call": "200 &#8805; 100",
>     "norm:hyperlink1": "ABC  DEF",
>     "norm:hyperlink2": "DEF custom text GHI",
>     "norm:hyperlink3": "ABC  DEF",
>     "norm:hyperlink4": "DEF custom text GHI",
>     "norm:hyperlink5": "GHI  and  JKL",
>     "norm:hyperlink6": "JKL hello and goodbye MNO",
diff FAILED

In summary, here's what I see that is not good:

• In my test.adoc, I have "[#norm:bold]#ABC is a network" but it gets converted to "ABC is a network" (lost the double **). Same problem exists for double __.
• [.underline] is removed
• Hyperlinks such as <> (actually <<foo>>) are removed completely

Overall, this is a net negative so we should probably back it out (yes, I prematurely merged the PR).

Can you figure out how to just replace macros and not have all these other changes?

[james-ball-qualcomm]

Created PR #170 to revert PR #131.

[Timmmm]

Damn that's annoying. I'll give it another go.

[Timmmm]

Ok I think I've figured out how to deal with all of that except getting n*et*work instead of n**et**work. Unfortunately the parser loses the information about whether the et should use * or **, and reconstructing that information is non-trivial.

So we could either:

1. Keep it like it is (formatting information gets lost).
2. Keep it like n*et*work which is wrong but maybe not that bad (it renders as literal *s if you render it as Asciidoc).

Which would you prefer?

[james-ball-qualcomm]

I think I'd prefer adding a new command line option to my create_normative_rules Ruby script to pass it the name of the adoc file with the macros (nicely changed in recent ISA manual to be in src/symbols.adoc). Then I can parse it and apply the macro changes myself since there doesn't seem to be a suitable facility in AsciiDoctor to use.

[james-ball-qualcomm]

Here's the contents of src/symbols.adoc right now (easy to parse):

// Macros that can be used in place of unicode characters to make writing them
// easier, e.g. "X {ge} Y".
:le: ≤
:ge: ≥
:ne: ≠
:approx: ≈
:inf: ∞
:circ: ○
:bullet: ∙
:times: ×
:check: ✓
:vertical-ellipsis: ⋮
:ellipsis: ⋯
:prime: ′
:interpunct: ·
:pm: ±
:rightarrow: →

[Timmmm]

No I mean we can do the attribute substitution fine. It's the "quoted" notes (bold, italic, superscript, etc.) that are awkward.

I'll make a PR for just the attribute substitution for now.

[james-ball-qualcomm]

What do you mean by "attribute substitution"? Do you mean macro substitution? And I'm not following your use of the term "quoted" notes and don't know why they are awkward.

I don't see any PR for the attribute substitution but look forward to it if it will just handle the macro substitution.

[Timmmm]

What do you mean by "attribute substitution"? Do you mean macro substitution?

Like {le} - they're actually attributes. Macros are things like image:... or https:....

And I'm not following your use of the term "quoted"

Asciidoc calls the basic formatting things like bold and italic "quoted inline" nodes. Terrible name if you ask me.