[Node] Arbitrary embedding .meta info into a tiddler file, i.e. single-file tiddlers of any type

[CyberFoxar]

Idea for arbitrary-embedding metadata info inside a tiddler.

The gist

Would it be possible to have the information that's currently in a .meta file embedded in the tiddler itself, without resorting to a .json or .tid tiddler ?

Why

I am in the process of migrating a good bunch of my markdown notes from dendron (think obsidian, but more open) to tiddlywiki.

One feature I like about dendron is the use of YAML Frontmatter, which is a block of yaml (or anything, really), delimited by fences, that defines meta-information about the rest of the document. It works more or less exactly how a .tid file is constructed, except it's a tad more versatile.

Anyhow, my issue stems from the fact that I'd love to reuse/keep the frontmatter that is already in my markdown notes, without having to convert them to json/tid or live with a bunch of .meta files. I have a personal gripe with the use of .meta files for text-based content, as I feel this will create undue clutter in my notes as well as risk me losing even more info next time I try switching systems.

How?

I have taken a look as to if a plugin might fix that, but it seems to be a root issue with how the nodeJS filesystem code is implemented, taking a look at this line and this one from /core/modules/utils/filesystem.js it seems this behavior is not (currently) overridable via plugins.

A tentative fix would be to either have a hook to change the behavior at this place, or allow plugins to inject themselves there. Say, allow a plugin to hook into filesystem.js.saveTiddlerToFile to handle either the actual writing of the file, or pass on the "compiled" binary data to the call to fs.writeFile. This would also require to patch boot.js to handle loading tiddler with meta in them, specifically patching /boot/boot.js.[$tw.loadTiddlersFromSpecification, $tw.loadTiddlersFromFile] to be able to read and load meta-in-tiddlers.

This would enable a plugin developer like me to use any arbitrary file format to serialize (and deserialize) single-file tiddlers in the filesystem. It'd allow me to call a plugin such as remark-frontmatter to parse frontmatter of text files and use that as the metadata for the tiddler, but it would also enable more enterprising people to use any kind of field for meta info. One could imagine storing the metadata inside of the EXIF data of a picture (see XMP:Description) or the "comment" field of JPEG files.

Similar subjects

The subject of tiddlywiki's handling of .meta files has been lightly touched on in the past, see this discussion on Discourse of someone whom a frontmatter-enabled tiddlywiki would've helped.

Interest for storing and accessing metadata within EXIF has been mentionned here in the Discourse.

I am not sure if there are more specific examples.

Offer for help

I am really liking TiddlyWiki so far, and I'd be glad to help / contribute to the above feature. I am still learning the ropes a bit, but I was already looking into patching this behavior into my own version of tiddlywiki, but I figure that some of you guys might have a better idea of how best it should be done.

[joshuafontany]

Great idea. I'd also need to look through the specifics of both the utils filesystem code & the filesystem adaptor plugin.

I think we would also need to hook into the boot code that loads tiddlers & tracks filesystem paths. Possibly add an MIME type there as well.

That all seems like we could abstract it better to support custom metadata.

[Jermolene]

Thanks @CyberFoxar. As you note, this has been discussed a few times in the past which is a clear sign that it is an enduring issue.

In TiddlyWiki terms, I think the proposal is to introduce a new file format for representing tiddlers with embedded YAML metadata, and then to use that format in the Node.js configuration in preference to .tid and .json files.

There are a couple of issues:

• YAML is quite an unwieldy specification and therefore a sizeable and complex dependency. The smallest YAML library I can find is 40KB
• Users regularly report being surprised and confused by some of YAMLs behaviour (eg for example the Norway problem – also see https://noyaml.com/)
• It is not clear what TW should do with the front matter beyond the string fields that TiddlyWiki supports

So, where I am at the moment is that I think we should indeed introduce a new file format that is similar to .tid but allows for multiline fields and a few other niceties we can borrow from YAML. I don't see a clean way to make it backwards compatible, so it would be a matter of a new file extension.

Could you give some examples of the metadata you'd like to use, and the effect you'd expect them to have?

[Jermolene]

Thanks @CyberFoxar that all makes sense.

Some background: boot.js has to know how to read all the different tiddler file types. It has the abstraction of a tiddlerdeserialiser module that knows how to read tiddlers from a particular file type:

https://github.com/Jermolene/TiddlyWiki5/blob/a081e58273da9b7974eac1e860193e371a32e69c/boot/boot.js#L1599-L1745

The mapping of file extension to file type is done here:

https://github.com/Jermolene/TiddlyWiki5/blob/a081e58273da9b7974eac1e860193e371a32e69c/boot/boot.js#L2444-L2492

In terms of writing files of the new format, the serialiser module can be packaged as a plugin. I think the logic here would need to be adjusted:

https://github.com/Jermolene/TiddlyWiki5/blob/a081e58273da9b7974eac1e860193e371a32e69c/core/modules/utils/filesystem.js#L217-L286

The core changes needed to satisfy your proposal would make a good grounding for subsequently changing the default file format.

[CyberFoxar]

Thanks !

Great to see that I'm not too rusty on my Node-fu. I had arrived more-or-less at the same bits of code you pointed out. Glad to see I didn't totally miss the mark (helps that the code is impeccably documented).

I might try and give a shot at implementing my proposal as PR, though I couldn't find a lot of documentation on how to setup a dev environment that uses NodeJS. The setup info on the dev site expects a single-file wiki, and running using node tiddlywiki.js edition/tw5.com --listen gives errors about missing plugins. I'll try editions/server and others, see if it changes x)

I'll try fiddling a bit with the various options and try and take care to document the process to contribute there too.

[pmario]

node tiddlywiki.js edition/tw5.com-server --listen

Should do the trick. New tiddlers created in the wiki will land in the edtion/tw5.com/tiddlers directory. Have a closer look at the tiddlywiki.info file in the tw5.com-server directory

[Jermolene]

Hi @CyberFoxar the scripts in the /bin give you some examples that might be helpful. When I'm testing the Node.js client-server configuration locally, I usually just run ./bin/serve.sh.

[joshuafontany]

I use VSCode, and base much of the setup on the scripts in the ./bin files. Here is my
D:\code\GitHub\TiddlyWiki5\.vscode\launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "Save Images TW5.com",
            "skipFiles": [
                "<node_internals>/**"
            ],
            "program": "${workspaceFolder}/tiddlywiki.js",
            "args": [ 
                "${workspaceFolder}/editions/tw5.com-server",
                "--save",
                "[!is[system]is[image]]",
                "[encodeuricomponent[]addprefix[tiddlers/]]"
            ], 
        },        {
            "type": "node",
            "request": "launch",
            "name": "Launch Caching Server",
            "skipFiles": [
                "<node_internals>/**"
            ],
            "program": "${workspaceFolder}/tiddlywiki.js",
            "args": [ "${workspaceFolder}/editions/tw5.com-server", "--listen", "use-browser-cache=yes" ], 
        },
        {
            "type": "node",
            "request": "launch",
            "name": "Launch MWS",
            "skipFiles": [
                "<node_internals>/**"
            ],
            "program": "${workspaceFolder}/tiddlywiki.js",
            "args": [ "${workspaceFolder}/editions/multiwikiserver", "--mws-listen" ], 
        }
    ]
}

These start "F5 Start Debugging" sessions in the "Debug Console", and are kicked off in the "Run and Debug (Ctrl + Shift D)" Panel-tab on the left side. You can set break-points, use debugger; lines as permanent break-points to inspect variables, etc.

[pmario]

This would sidestep nicely the issues with settling on an actual filetype and the restrictions this might entail, as well as broaden the range of what a tiddler actually is, without bloating too much the boot process, though plugins that do the actual decoding/encoding might slow down the saving/reading themselves.

I'm OK with your example of listing JPG-files and extract the EXIF data as TW meta data. But I'm against the possibility to handle arbitrary binary files with unknown content for security reasons.

[CyberFoxar]

That's a valid concern, but I'm not sure it's a big issue.
One thing that might alleviate the security concerns is that the "core" of tw would not really handle binary files (at least, not more that it is actually doing), just have the ability to defer their handling to a plugin.

This, in turn, means that you'd need a specific plugin to handle a specific type of binary file. This "delegate" the security issues to the plugin developper, and the wiki administrator. It's not great, but the separation of concerns still is there.

I'm trying to think of scenarios where this might be an issue. I can imagine trying to inject a tiddler with arbitrary binary content in, but it wouldn't be written to disk unless a proper handler for the type is registered. The responsibility would be on the plugin maker to make sure the file written can be read correctly after... Maybe one could try and inject a nefarious payload, but that wouldn't be executed, since the only thing interacting with the file is tiddlywiki and the plugin. Maybe someone could try to create buggy invalid files ? But it's going through nodeJS' fs library anyway.

There's also the idea that you're not giving away rights to write to the filesystem either. The proposed api would be that tiddlywiki simply ask for the data to write and the filename to use. This would then be passed to nodeJS, which does some sanity checks on the written binary data. The plugin wouldn't even really have the right to choose where the file is saved, just how it should be.

It is, however, entierly possible that I'm missing something.

[joshuafontany]

@CyberFoxar I think abstracting this away from being directly inside the boot.js code is a good idea. The issue with conceptualizing this as a node.js tiddlywiki5 plugin is that THOSE are currently handled as "files on disk" and thus you need to go through the boot process to "load tiddlers" BEFORE any "tiddler stored plugin" is available to use by the system.

If you'd like to sync on this at some point, let me know.

[CyberFoxar]

Yeah, that's an issue with the approach. I believe it could work, though, because boot.js does load some modules before loading the rest of the plugins (like the tiddlerdesrializer module). I wonder how well that works, though, so some fiddling and experimentation is required before a definite answer.

Another approach might be to just have a config/fileinfo that says "refrain from loading tiddlers here at boot and wait for plugin XYZ" and then, once plugins are loaded, have some logic to "re-discover" those tiddler and add them post-boot. I'm not sure how well this approach would work, since it would mean that we'd add potentially a few hundred of tiddler to the store post-boot...

[CyberFoxar]

Alright, an update.
I've somewhat decided how I'm going to try and implement this.

My goal is to create a new kind of plugin, that I'm tentatively naming tiddlerLoader, with the API described below.

Proposed Mechanism: deferred tiddler loading

I'm trying to work out how a deferred loading of Tiddlers might work. The goal is to add a new, optional property to tiddlywiki.files.directory and tiddlywiki.files.tiddlers. Something like:

{
    "directories": [
        {
            "isDeferred": true // {boolean}
            "deferredContentType": "bin/test-tiddler" // {string} - defines which plugin to defer to
            // Rest of the fields as-is
        }
    ]
}

This optional property would signal to boot.js that files in here should not be processed further, the fileinfo be saved for reuse later, and that the plugin registered for deferredContentType should be called upon later, once plugins are actually loaded.

I'm a bit iffy on the details, but I believe that registering another StartupTask to handle the deferred loading might work. It'll involve reading through the deferred directories (maybe via a hashmap of some kind) and executing the relevant plugins on them. If, after we've done so there are still deferred directories (i.e. we have nested deferred directories), register another startup task to do the same.

This would basically be a lite version of the content of $tw.boot.execStartup and would load the new tiddlers as well as register the new plugins.

Tiddlers loaded via this method need to be marked in some way (so that tiddlerLoader can be invoked in place of the normal way) , I suggest adding a new field to $tw.Tiddler, something like deferredContentType. This would enable patching filesystem.js to discriminate when saving a file.

TiddlerLoader API proposal

My JS is a bit rusty, but that's what I'm working with for now. Please do tell me improvements !

The most obvious one, that I'm on the fence about is whether or not we should try to make the API more flexible, to allow passing a bit more info that just the filename to the plugin. Maybe $tw.config.contentTypeInfo[type] or something similar ?

/*\
title: Proposed API for deferred tiddler loader module
type: application/javascript
module-type: tiddlerLoader

Example API
\*/

(function(){
    /**
     * When given binary data, tries to deserialize it into a tiddler.
     * 
     * @param {string} filename - original filename, does not include path
     * @param {Buffer} fileBuffer - file as read by NodeJS
     * @returns {$tw.Tiddler} - decoded Tiddler object to be added to the store
     */
    function loadTiddlerFromBinary(filename, fileBuffer){
        throw Error("not implemented")
    }

    /**
     * When given a Tiddler, binarize it however we like and gives
     * back a temporary object holding the data.
     * 
     * @param {string} filename 
     * @param {$tw.Tiddler} tiddler - tiddler to be binarized
     * @returns {
     *  {
     *      buffer: Buffer,
     *      filename: string
     *      fileOptions: {fs.WriteFileOptions | undefined}
     *  }
     * }
     */
    function makeBinaryFromTiddler(filename, tiddler){
        throw Error("not implemented")
    }

    TiddlerLoaderPlugin.prototype.load = loadTiddlerFromBinary
    TiddlerLoaderPlugin.prototype.save = makeBinaryFromTiddler

    function TiddlerLoaderPlugin(){
        // init stuff
    }

    exports["bin/test-tiddler"] = TiddlerLoaderPlugin
})()

[CyberFoxar]

More news ! I've made a PR ( #8201 ) to implement the proposed feature !

[linonetwo]

@CyberFoxar We discussed this before #7534

And I think your #8201 is not adding yaml metadata support, instead, it is trying to solve #3060 (comment) ? Isn't it?

[CyberFoxar]

To be honest, it's been a long while since I've looked at the code, much less touched it since then, so take it with a grain of salt.

The PR itself does indeed touch on the #3060 issue, but only partly.

It would allow for a plugin implementation for #7534 rather than needing a "big" core change for it.

My PR is not adding yaml metadata support, but its implementation would enable it to be done via a plugin. This would sidestep any issues with embedding a yaml parser inside tiddlywiki's core, and enables simpler parsers like the demo one in the PR or a yaml-frontmatter one. One could even imagine storing tiddlers in some kind of hybrid binary format, maybe something like a GZipped JSON, were you need to decompress/recompress the file before parsing the inner json as a proper tiddler.

It might be able to solve the #3060 issue via a plugin too. One could imagine a plugin that's watching the filesystem and reloading tiddlers as changes are made.

I remember, however, that the PR is incomplete. I kind of remember that my implementation had a bunch of issues with tiddlywiki's systems, but not what they were. Feel free to take over the PR or take inspiration from it.