How to remove wikilinks and handle citatios?

[putzwasser]

Description

Hi,

I'm doing my first steps with quarto. I am using obsidian and, thus, have wiki links in the documents I want to process using quarto. Wiki links I use are:

• [[link to a obsidian note]]
• [[link to a obsidian note|some alias]]

Additionally, I create markdown literature notes for the sources I'll quote. I link these in my documents, too.

My literature notes are titled after the bibliography cite key: /Obsidian/Vault/Literature-Notes/@citekey.md

In a text I link these like this:

Some text [[@citekey]]

For quarto I'd use an additional set of brackets:

Some text [[[@citekey]]]

So, basically, I want to get rid of the wiki links (mostly the [[/]]) before quarto/Pandoc does anything.

As quarto doesn't support wiki links OOTB I tried to create a Lua filter to remove the wiki links:

function Str(str)
	-- Regex pattern explanation:
	-- %[%[ : Matches the opening double brackets `[[`
	--     ([^%|%]]+) : Captures the link text. Matching ends if a `|` or `]` occurs.
	--                  This is the main link text in case there is no alias.
	--     %|? : Optionally matches the pipe `|` character that separates the link text from the alias.
	--     ([^%]]*) : Optionally captures the alias text following the `|`, up to the closing `]]`.
	--                This group will be empty if there is no `|` with a following alias.
	-- %]%] : Matches the closing double brackets `]]`
	local pattern = "%[%[([^%|%]]+)%|?([^%]]*)%]%]"

	str.text = str.text:gsub(pattern, function(main, alias)
			if alias == "" then
					return main
			else
					return alias
			end
	end)

	return str
end

The Lua script itself seems to work correctly (see further below).

But in the quarto/Pandoc context this doesn't really work. So, please help me fix that.

Minimal working example

1. Create a new project
2. Create the filter file remove-wikilinks.lua, paste the contents from above and add it to _quarto.yaml

   filters:
     - remove-wikilinks.lua
   

3. Edit intro.qmd

# Introduction

This is a book created from [[markdown]] and executable code.

See [[@knuth84]] for additional discussion of [[my link to literate programming|literate programming]].

See Knuth [-[[@knuth84]]] also for something else.

5. Build the document.

Output

The output is:

This is completely off.

Please help me understand why the output isn't as I expected it to be.

1. Why aren't the wiki links substituted correctly?
2. Why is the citation off?

I'm having a hard time understanding this, because the Lua code seems to be working:

Lua Code Snippet

https://www.jdoodle.com/ia/12JH

Code Snippet's Output

The above snippet creates the output I'd like quarto to process:

# Introduction

This is a book created from markdown and executable code.

See @knuth84 for additional discussion of literate programming.

See Knuth [-@knuth84] also for something else.

[putzwasser]

I think, I learned that Str isn't the right object to work with. I read about custom readers.

I created this custom reader:

function Reader(input, args)
	-- process wiki links
	local pattern = "%[%[([^%|%#%]]+)%#?[^%]]*%|?([^%]]*)%]%]"
	input = tostring(input):gsub(pattern, function(main, alias)
		if alias == "" then
				return main
		else
				return alias
		end
	end)

	-- Removes comments
	input = input:gsub("%%%%[^%%]+%%%%", "")

	local blocks = pandoc.read(input, 'markdown').blocks

	return pandoc.Pandoc(blocks)
end

This works "fine" (in a limited test) with pandoc:

# Introduction

%% Comment %%


%%
Multiline Comment
%%


This is a book %%inline comment%% created from [[markdown]] and executable code.

See [[@knuth84]] for additional discussion of [[my link to literate programming|literate programming]].

See Knuth [-[[@knuth84]]] also for something else.

# Introduction (more wiki link syntax)

This is a book %%inline comment%% created from [[markdown#heading|markdown]] and [[executable#otherheading]] code.

See [[@knuth84]] for [[additional#^somesection]] discussion of [[my link to literate programming#^section|literate programming]].

See Knuth [-[[@knuth84]], 54] also for something else.

Renders via pandoc

pandoc -f readobsidian.lua -t pdf intro.qmd -o output.pdf -C --bibliography=references.bib

To this:

This looks great so far.

The problem is: I can't use the reader with quarto:

project:
  type: book

book:
  title: "Test"
  author: "Norah Jones"
  date: "17.5.2024"
  chapters:
    - index.qmd
    - intro.qmd
    - summary.qmd
    - references.qmd

bibliography: references.bib

format:
  pdf:
    documentclass: scrreprt
    from: "/tmp/test/readobsidian.lua"

Output

$ quarto render
[1/4] index.qmd
[2/4] intro.qmd
[3/4] summary.qmd
[4/4] references.qmd

pandoc 
  to: latex
  from: /tmp/test/readobsidian.lua
  output-file: index.tex
  standalone: true
  toc: true
  number-sections: true
  top-level-division: chapter
  pdf-engine: xelatex
  variables:
    graphics: true
    tables: true
  default-image-extension: pdf
  
metadata
  crossref:
    chapters: true
  documentclass: scrreprt
  papersize: letter
  classoption:
    - DIV=11
  header-includes:
    - '\KOMAoption{captions}{tableheading}'
  block-headings: true
  bibliography:
    - references.bib
  title: Test
  author: Norah Jones
  date: 17.5.2024
  
Error running Lua:
/opt/quarto/share/pandoc/datadir/_format.lua:175: Warning: Invalid format /tmp/test/readobsidian.lua. Assuming 'markdown'.
stack traceback:
	/opt/quarto/share/pandoc/datadir/_format.lua:175: in function '_format.parse_format'
	/opt/quarto/share/pandoc/datadir/readqmd.lua:135: in function 'readqmd.readqmd'
	/opt/quarto/share/filters/qmd-reader.lua:13: in function 'Reader'

If I make from a relative path from: "readobsidian.lua", I get this error:

Error running Lua:
Unknown input format readobsidian
stack traceback:
	/opt/quarto/share/pandoc/datadir/readqmd.lua:157: in function 'readqmd.readqmd'
	/opt/quarto/share/filters/qmd-reader.lua:13: in function 'Reader'

[mcanouil]

Using the native format can help developing Lua filter (which I don't think is a reader as it comes too late).

---
title: "Quarto Playground"
format: native
references:
  - type: article-journal
    id: WatsonCrick1953
    author:
      - family: Watson
        given: J. D.
      - family: Crick
        given: F. H. C.
    title: 'Molecular structure of nucleic acids: a structure for
      deoxyribose nucleic acid'
---

# Introduction

This is a book created from [[markdown]] and executable code.

See [[@WatsonCrick1953]] for additional discussion of [[my link to literate programming|literate programming]].

See Knuth [-[[@WatsonCrick1953]]] also for something else.

You can see that [[ and ]] are parts of the string they are next to.
The stricky bit is really for citation, as [@...] has meaning.

, Str "[-["
, Cite
    [ Citation
        { citationId = "WatsonCrick1953"
        , citationPrefix = []
        , citationSuffix = []
        , citationMode = NormalCitation
        , citationNoteNum = 2
        , citationHash = 0
        }
    ]
    [ Str "(Watson"
    , Space
    , Str "and"
    , Space
    , Str "Crick,"
    , Space
    , Str "n.d.)"
    ]
, Str "]]"

Here you would need to check the three elements to rewrite them as you want.

Alternatively, you could do your search and replace in a pre-render script: https://quarto.org/docs/projects/scripts.html#pre-and-post-render (it accepts Lua)

[mcanouil]

FYI, you might want to look at the other threads about Obsidian: https://github.com/quarto-dev/quarto-cli/discussions?discussions_q=obsidian

[putzwasser]

Using the native format can help developing Lua filter (which I don't think is a reader as it comes too late).

Yeah, a filter comes too late. It is also inefficient to modify the AST in this case, IMHO. A regex replace before producing the AST is the way to go, I think.

That is why I tried a reader. My test with pandoc itself is promising. It creates the result that I want.

My problem is that I can't get it to work in the quarto context, due to this error:

Error running Lua:
Unknown input format readobsidian
stack traceback:
/opt/quarto/share/pandoc/datadir/readqmd.lua:157: in function 'readqmd.readqmd'
/opt/quarto/share/filters/qmd-reader.lua:13: in function 'Reader'

Any ideas how to resolve that?

Alternatively, you could do your search and replace in a pre-render script: https://quarto.org/docs/projects/scripts.html#pre-and-post-render (it accepts Lua)

I saw this, but I'd need an example to start working with it. The docs are to thin, there, for me, unfortunately.

Because I don't want to modify/edit the source .md file in a script. That would destroy the purpose 😉 ...

FYI, you might want to look at the other threads about Obsidian: https://github.com/quarto-dev/quarto-cli/discussions?discussions_q=obsidian

Thanks. I already read a couple, but none seemed fitting.

[mcanouil]

If you don't want to change the file on disk, then pre-render scripts are not what you want.

I am not sure about the issue when using a Reader, but I don't think you can actually use several of them in a row.

@cscheid might have some wisdom to share here.

[cderv]

As of today, Quarto can't work with a custom Lua reader. Only Pandoc formats as documented in their Manual (name + extensions).

For now, we don't have a way to allow a custom reader from the user's side, and AFAIK we don't document it. It seems we don't disallow it either, and we probably shouldn't allow some lua path in from key in YAML.

Here are some context on the situation:

Quarto already uses a custom reader for Pandoc as we need to handle some "Quarto Markdown" specificity.

Our internal custom reader will always be used - no matter the value of the from key.

quarto-cli/src/command/render/pandoc.ts

Lines 851 to 855 in 76d4ae6

	// set up the custom .qmd reader
	if (allDefaults.from) {
	formatFilterParams["user-defined-from"] = allDefaults.from;
	}
	allDefaults.from = resourcePath("filters/qmd-reader.lua");

The original from format is used as the flavor format to use in our Reader for a pandoc.read() call (in readqmd.lua)

So for now, you can't tweak Quarto processing with a custom Lua reader.

You would probably need to pre-processing step to convert your obsidian .md using Pandoc to produce another .md, and use your custom Lua reader at this step. Then you could use Quarto to convert.

Or you need to find another way to process the content for now.

I don't know if it would be safe enough and how we could allow custom reader. This is quite advanced and not on our roadmap for now.
Though I see how powerful it could be as extension mechanism.