Python scripts for easily handling variables and outputs

[LucunJi]

I found it better to put lengthy command in some files, and forward the variables using arguments.
Being able to store the configurations and logs elsewhere is another plus.
Although it is an overkill for customizing PDF export command, these facilitating Python scripts may be the help of someone with similar needs. > Codes here <

I will make this a pypi package for easier access if people think it useful :-)

The variable names are stored as string constants and can be concatenated with arguments.
These scripts can also generate a handy usage message like this

usage: md2pdf.py [-h] --sourcepath {{file_path:absolute}} --sourcetitle {{title}} --sourcefolder {{folder_path:absolute}},

so the argument list can be directly pasted in Shellcommand configuration.
Such arguments are parsed and stored in a typed way like a dataclass.

There is also a function to setup a logger suitable for Shellcommands which prints ERROR and CRITICAL to stderr, and INFO to stdout and other debug information elsewhere.

[Taitava]

Wow! 😎 Looks a pretty cool thing to me! 🌞

I didn't try to run it yet, but I looked at the Python code. I'll need to explore it more at a better time 🙂.

Does it support custom variables? I.e. variables starting with {{_ instead of {{.

P.S. I noticed the following "special" variables there:

OUTPUT = Variable("output")
PASSTHROUGH = Variable("passthrough")

• {{output}} is only available in output wrappers, i.e. after a shell command is executed, and some output is received. There's no way to use it as input for a shell command 🙂 .
• {{passthrough}} is only available in debug mode and all it does is that it takes an argument, escapes special characters in it and returns it. I'm using it for testing the escaping system, but I guess there's not much real-world usages for it 🙂 .

Maybe these two could be taken out, or put a remark on them in the code? 👍

Thank you for this! I appreciate it! 🥇

I'll try it out and come back when I have some more feedback. 🙂

[Taitava]

Pondering: Query variables from Obsidian in realtime?

Your work made me start pondering a question: What if we could skip reading variables in the shell command part altogether? E.g. md2pdf.py --sourcepath {{file_path:absolute}} --sourcetitle {{title}} --sourcefolder {{folder_path:absolute}} would become just md2pdf.py. As the shell command is launched from Obsidian, a parent process - subprocess relationship is formed. If I'm not mistaken, there should be some way for the subprocess (i.e. a Python script in this case) to communicate with the parent (i.e. Obsidian/this plugin) while the subprocess is still running. Would it be possible for the subprocess to send a message to the parent, querying variable data? It could query for whatever variables, and the Shell commands plugin would respond to the query. E.g. subprocess queries for {{title}} and receives My note title. Or e.g. query for {{yaml_value:aYAMLPropertyNameHere}} to get A YAML value.

Benefits:

• Cleaner shell command for calling the python script: data retrieval is done in behind the scenes.
• For long/eternal processes (e.g. daemons), the process could ask for fresh, new data from Obsidian whenever it needs it. Ask for {{title}} now to retrieve My note title, ask it again after some time to get Another note title as the user has carried on to view another note.

This requires me to explore what possibilities Node.js's child_process offers for communication between parent and subprocesses while the subprocess is still running. Perhaps it could use custom output streams and custom input streams? Or something else?

[LucunJi]

What a coincidence, I came up with exactly the same idea yesterday! From a Linux user's perspective, I can use stdout/stdin, unnamed pipes, named pipes (FIFO) or sockets for communication. Actually, a stdout/stdin pair should be enough

General Workflow

1. To make a query, subprocess sends a string "{{!variable:args}}" and wait for input
2. Then Shell Commands sends the data in an unescaped string, terminated with something such as EOF (I forgot most of inter-process communication, sorry)
3. The subprocess reads the input plus knowing that the input terminates. Then it continues its routine.
4. If the subprocess sends a string in some other format, then it should be a normal output message. Shell Commands can handle it normally. (no need to do this if not using stdout/stderr)

Pros & Cons

+ No need to worry about how to escape and when to escape strings
+ Short and clean script: only a executable file
+ We can easily run script in most programming languages

- When the script grows to some extent, it might be better to write a new extension

---

Putting the growing code in a separate file blurs the boundary between write a Shell Command and writing a new Obsidian extension.
I think it's okay to put code into a separate file just for better syntax highlighting, string formatting (when I use python), debugging and version-controlling.
But when the custom variables increases and the functionality becomes more complicated, I might consider to leave the comfort zone and learn to write an extension.

[Taitava]

Sorry, it took me some time to read and answer your message 🙂. And I didn't try your script yet.

stdout

I like the simplicity of your idea about using the normal stdout/stdin pair for the communication. I do understand that outputting a variable name (e.g. {{title}}) is easy to distinguish from other output that should be shown to a user, e.g. Hello from a shell command., and so it could be doable. However, I might still want to consider using an unnamed stream for outputting variable names. This is how I see it:

• From a (Python or whatever) script's point of view, having separate streams for normal output and for querying variables makes a clear distinction on what is being done. It can avoid confusion when reading the script file's code. stdout would be solely for output meant to be seen by a user, while printing to a custom stream would indicate clearly that the aim is to do something else than show output to a user, i.e. to query for variables.
• In case the variable query has a syntax error, e.g. the query contains {title} instead of {{title}}, the SC plugin should be able to tell the script that there's a problem in a query. If stdout would be used, SC would just assume {title} should be handled as normal output (e.g. by showing it to a user) because it doesn't match the layout of a parseable variable. (I need to figure out some way to include a success/failure flag in the response message, maybe use numbers in the response e.g. {title} --> 1:Syntax error. and {{title}} --> 0:My note title. There can also be other errors even if the syntax is correct, e.g. a file pane not being open when {{title}} is queried for.)
• I had a third point on my mind, too, but I forgot it.
• However, the plugin could still offer a choice to users:
  • Use a custom stream/pipe. (Which one actually is an official name for the term, stream or pipe?)
  • Use stdout - but can only be selected, if output channel for stdout is set to ignore. So I would not allow mixing variable queries and normal output in the same stream. At least not in the beginning.

stdin

So far, I don't have anything against using stdin for transfering the query response.

• Since SC 0.18.0 (released in January this year), it's possible to use stdin for passing variable values to shell commands. It's not realtime, the values are put to stdin at the beginning of the executed process. However, if it's used for more than one variable, linebreaks should be avoided in the values, or you should have some other way to disquish different values from each other.
• When a script queries for variables, it should make sure no initial content is present in stdin to avoid mixing up initial variable values with queried variables values.
• Optionally, a custom stream/pipe could be used here, too.

---

Putting the growing code in a separate file blurs the boundary between write a Shell Command and writing a new Obsidian extension.

True. Absolutely true! It's good to keep in mind that at some point, a dedicated plugin might be better suited for a task at hand, rather than trying to "bend" a general purpose tool like SC to do highly complicated stuff. As the developer of this plugin, I like to see how far this plugin can go in providing advanced flexibility for integrating Obsidian with other programs. But a user should not grow too big ties with this plugin, as it does have it's limits.