How can one extract works cited in project-manuscripts from a larger .bib-file and write them to a local project-bound .bib-file?

[Gewerd-Strauss]

Description

Hello,

this is probably an odd question.
I am still setting up a quarto project, code will mostly be written in R, with some parts in ahk for my own convenience. I will require to provide citations for third-party code used, as well as any regular citation within the manuscript itself.
The project itself is deployed to a GH-page, and is written in a mix of .md- and .qmd-syntax, depending on whether or not code execution is required. The full details are not required.

For R-Code, the grateful-package is really neat for collecting citations for all packages loaded within a project. This package seemingly works for Quarto-projects as well, as a pre-render script (located in projectroot/scripts/pre-render/compile_bibs.R):

print(getwd())
a <- quarto::quarto_inspect()
root <- a$dir
setwd(root)
refs <- grateful::scan_packages(pkgs="All",dependencies = T)
table <- grateful::cite_packages(output=c("table"),out.dir  = stringr::str_c(root,"\\.pandoc"),pkgs = "All",cite.tidyverse = T,bib.file = "packages",dependencies = F)

So far I always cited from my personal reference lib, and didn't have to provide a bib-file. Therefore, I could reference my large main-lib bib-file by an absolute path in the manuscript's bibliography-frontmatter.
However, part of the thesis scope is that the project must have no outside dependencies that other people can't access - and that obviously includes the bib-file which contains papers cited. Citations are formatted in standard pandoc format, e.g. [@altschul2001blast].

A quick overview on the project structure:

root/
    .pandoc/
        X.csl
        packages.bib (autogenerated by pre-render script mentioned above)
        references.bib   // this is the desired output location of the file I am trying to generate and then reference to in the manuscript's YAML
    report/
        manuscript.qmd   
    scripts/
        pre-render/
            compile_bibs.R

I am now trying - and failing - to figure out how/if I can implement this for citations within non-code files (e.g. .qmd/.md-files). So I would have to scan the quarto-project for citations, extract them from an external bib-file and import them into a separate bib-file within the scope of the project. This file would then be referenced in the YAML and is used during rendering.

Is something like this possible? I've done a somewhat deep search for how to do this, but so far I could not find anything that is of any substantial help. I am aware this is only partially a quarto-related question at its core. I would need a solution written in R for proper implementation into the project's workflow.

I am open for any suggestions.

Thank you,
Sincerely,
~Gw

[mcanouil]

Yes and you can write your own script as a pre-render script, including R script, see https://quarto.org/docs/projects/scripts.html#pre-and-post-render.

Note that I find this approach a bit cumbersome, i.e., your described workflow is:

1. Use citation from a bibliography manager.
2. Render/scan (via project script)
3. Make a bib file
4. Render/scan (via project script)

Why not export the needed citations from your bibliography manager such as Zotero and use the bib file directly?

[Gewerd-Strauss]

Why not export .... directly?

In principal I would prefer that, but it would require some major maintenance work on my end.
My zotero is set up on a somewhat large library with 1k+ references last I checked. This means that after finishing a manuscript I would have to manually track down all cites in the project, manually search for them in zotero and manually add them to a collection, which I could then export into the quarto-project to be used internally. Sure that's possible when actually wrapping up at the end. But when regularly compiling a work-in-progress project, this would take too much work during the writing-process.

Not to mention the general issues arising from doing a critical maintenance step manually.

For R, I have unfortunately yet to find (a) package(s) allowing me to identify and extract cites from a bibliography based on a list of cite-keys obtained from scanning the source files.

[mcanouil]

In my opinion, maintaining a bibliography is not really up to Quarto.
Even Rmarkdown or Jupyter don't handle this for the users.

Note that even if you have a large library, there is nothing blocking you from creating a collection in which you place the articles you need as you go.
Then at the end you can export the collection.

[mcanouil]

I don't know how you insert your bibliography, but Quarto VSCode visual editor (I just discovered that because I do not user the Visual Editor) does a pretty good job at doing what you want/need, i.e., it creates the Bib file from my Zptero library.
Note that because the Visual Editor code is shared between RStudio IDE and the VSCode extension, I strongly believe you can do the same in the RStudio IDE.

Screen.Recording.2024-01-14.at.19.29.06.mov

If you want to go the LUA filter road, see https://tex.stackexchange.com/questions/392070/pandoc-markdown-create-self-contained-bib-file-from-cited-references
There are very likely other threads on the Internet you can use.

[Gewerd-Strauss]

I will take a look at the lua-road in that case. For reasons I am not going into beyond "I find both incredibly inefficient for writing manuscripts", I don't use either the VSC- nor RStudio interface to quarto, so those sensible solutions don't apply to me.

I'll close this with the summarised answer from cderv below.

Thank you.

[cderv]

I want to share this as its own answer what was already mentioned in another thread in this discussion.

The Visual Editor for Quarto document (available in VS CODE and RStudio IDE) brings support for Zotero Citation which includes adding to the bibliography file for the project

This enables a workflow where:

• You can centralize all your citations in Zotero
• Include them very easily while adding to a document
• And when you include them, write them in a local bib file to your project

So it does not really answer the extraction of the bibliography items already in Qmd file, but it could be quite helpful for future projects IMO.

Links to doc:

• https://quarto.org/docs/visual-editor/vscode/#zotero-citations
• https://quarto.org/docs/visual-editor/technical.html#citations