Accept local, web, and cloud URI as inputs and outputs

[liquidcarbon]

Description

Papermill is able to run and output notebooks from and to cloud URIs, but it does not handle format conversion (only .ipynb -> .ipynb)

It would be great if Quarto could do things like
file://path/to/notebook.ipynb -> s3://path/to/output.html
https://path/to/notebook.qmd -> gs://path/to/output.html

http(s) would normally be read-only, unless you have a publishing mechanism defined; for others, it would look for cloud credentials in the usual places.

From https://github.com/nteract/papermill :

Supported Name Handlers

Papermill supports the following name handlers for input and output paths during execution:

Local file system: local

HTTP, HTTPS protocol: http://, https://

Amazon Web Services: AWS S3 s3://

Azure: Azure DataLake Store,
Azure Blob Store adl://, abs://
Google Cloud: Google Cloud Storage gs://

[mcanouil]

Unlikely to happen because this make your project no longer self contained which is the main idea behind Quarto.

If you want something like that, I suggest you define pre and post render scripts as defined in Quarto's documentation.
The pre-render script could download online file into whatever environment you want, then Quarto render the file, then you can publish the rendered file to whatever location/host as long as you follow the location/host documentation.

[liquidcarbon]

Thanks - the pre/post render scripts was news to me, definitely an option to consider! https://quarto.org/docs/projects/scripts.html

Can you pelase speak a bit more to how the "self-contained" idea is compromised by this? The execution and rendering still remain in one place, just the inputs and outputs are elsewhere. Is it very different from using CDNs to pull necessary CSS/JS?

[mcanouil]

Here you are making the assumption or implying that Quarto is only about single file projects.
It is not.
Allowing online file as input would also need to be allowed in multi file projects.
That is not a small change.
Also this means cache/freeze would no longer be available to such documents generated from online input.

Your comparisons with CDN, only applies to HTML documents which does not necessarily have those.
You are missing all the other formats.

[cscheid]

We'll keep the request in mind. What additional value would you get out of a wrapper script that did curl -O URL; quarto render file.qmd? Do you mind explaining where this would improve your workflow?

[liquidcarbon]

@cscheid if I'm honest, I just stumbled into this functionality on papermill and thought "this would be nice in Quarto". Indeed, doing what you suggest would work for the simple use cases that come to mind at this time. As @mcanouil explains, it gets more complicated with projects.

Up to now we've been talking about just publishing HTML. I've also been thinking about facilitating complete reproduction of someone else's work from a published .qmd file (sourced from someone's repo, or a publication's supplementary info). Scenarios where instead of jupyter: python3 pointer to a local kernel you've got things that enable you to precisely recreate execution environment, such as:

(in approx. order of increasing complexity)

• environment definitions, via e.g. requirements.txt or poetry.lock
• remote kernels (pointing to ec2ip:port or similar, with running Jupyter/R)
• container registry links
• resource allocation

Then you compare original and published HTML outputs and call it good. 🍻
If they don't match, article is rejected. 😏

Curious to hear your thoughts on extending Quarto in that direction?

[cscheid]

These are all things we would like to support at some point in the future, but are not high in our priority list. Specifically, quarto-native solutions need to support the knitr ecosystem together with the Jupyter ecosystem, and so our designs are necessarily more complex than might appear at first.

[liquidcarbon]

Yes, I appreciate that any endeavors have to fit into both ecosystems and that makes it more difficult, and all the more amazing that you chose to create a tool that serves both. :)

Thanks for your input here and your work generally!