[Feature] Search/index .md files recursively

[jorgeEF]

Hello!

First of all, congratulations @dullage on Flatnotes — it's a really nice and useful app. Thank you so much for sharing it with the world!

I have a small (and hopefully not too silly) feature request:
Could you allow Flatnotes to search for .md files recursively within the /data folder?

Why?
I totally get and appreciate the idea of not enforcing note organization, and it works well within Flatnotes. But in my case, it conflicts a bit with one of the things I love most about Markdown: having my notes remain app-independent.

I’ve been organizing my Markdown notes in folders for years. I mostly write them using Ghostwriter on my PC and Markor on my Android phone, syncing everything with Syncthing. This setup allows me to keep things neatly categorized and quickly searchable across devices.

Sadly, Flatnotes doesn’t currently search .md files recursively in the /data path. That means I’d need to flatten my entire folder structure and move all notes directly into /data to use Flatnotes, which would really mess with how other apps interact with my notes.

Would this change interfere with Flatnotes’ design or performance?
I believe it wouldn’t negatively impact your current workflow or the minimalist approach Flatnotes promotes. Instead, it would simply give users the option to organize notes into folders if they prefer, making Flatnotes even more flexible without compromising its simplicity.

I hope you’ll consider it. For me, this feature would open up a new way to access my notes via web browser, and I imagine it could be helpful for other users too, making Flatnotes even more appealing.

Thanks again for your work on this awesome app!

Cheers!

[ctmbl]

+1 love that suggestion!

[dullage]

This comes up quite a bit. There was even a PR for this a while back, which has a long history of conversation on this topic.

I completely get the need, but haven't yet found a clean solution that I'm comfortable with. The key issues are:

• Duplicate filenames across subdirectories.
• Performance (having to recursively explore all subdirectories looking for a note/notes). Imagine navigating to /note/My%20Note. Currently, flatnotes knows exactly where the file is and can just open it. With subdirectories, it has to recursively explore until it finds it.
• Where to create new notes.

One suggestion has been to use special characters in the note title to dictate the path e.g. sub1/sub2/My Note but this is pretty horrible from a UI and a UX perspective. It'd be an ugly hack at best.

Ideally, if a hacky solution is found, it'd be one that isn't part of flatnotes itself. For example, it'd be great if the data directory could be mounted to the docker container in a way that does this flattening without flatnotes being involved. Maybe something is possible using a custom docker volume driver or even using inotify to automatically create symlinks in a flat directory.

[ctmbl]

Hi @dullage
While I'm not is the desperate need of this feature it seems like an interesting challenge to me.

I've read the previous PR thread about this and totally get why you weren't comfortable with the solutions proposed there.
I'll try to give my own answer to your questions

Performance

When we can agree than recursively looking for notes is not ideal nor optimal I don't think it will be a performance issue. Are we talking of a handful of milliseconds here?
I can't assume the order of magnitude of the number of notes managed by users of flatnotes but after a quick and dirty profiling I've done I enumerated an heavily nested repository of ~90.000 files in 3.6s which would give ~40ms for 1000 notes (this only gives an idea of course, nothing more)
So is performance really an issue here?

Where to create new notes

The more sensible answer would be at the root of /data, flatnotes would only support "reading" nested directory not diretcly dealing with them, otherwise I would totally agree it goes against one of the very principle of flatnotes

Duplicate filenames across subdirectories

This is the more difficult one in my opinion.
It can be separated into two different issues if I got the problem right: existing duplicates in nested directories and creating a note from flatnotes that exists not at the root of /data but in another place.

As said in the previous PR thread the 2nd case isn't an issue as it is already gracefully handled by flatnotes (at least in the context of the PR and if we trust the author, but that doesn't seem unreasonable)

The first issue of existing duplicates (same name, different directory) is the real blocker to this feature in my opinion.
It could however be dealt with by simply ignoring the existence of duplicates:

• when searching notes, eliminate duplicates (this could add a little time overhead to the search maybe)
• when opening / editing a note, only open and edit the first one find (by a depth-first or breadth-first tree search this would be implicit I think)

However I'm not familiar with flatnotes internal code so it might exist other issues/places in the code I didn't consider, if this is the case please let me konw!

---

That being said, one of your prime concern was that the code responsible for dealing with nested directory would be external to flatnotes.
With this in mind I sincerely think the best idea that rose from #70 is your idea of "bring-your-own-storage", being able to provide a class or plugin that expose an API to list, get and read notes.

Could you outline the API of such a plugin? And maybe what would need to be refactor in flatnotes to isolate these components?
Also how would you plan to supply this plugin code? Ideally it would need to be as simple as an environment variable (holding a path to python file for example) to not break the really nice docker/docker compose deployment workflow.

Hope that you'll like these suggestions 😉

[dullage]

Thanks for putting these thoughts together @ctmbl, and sorry for the late reply. Ultimately, I can't see an elegant solution to this problem. There are certainly some inelegant solutions, but I wouldn't want to introduce these to the flatnotes codebase.

As you say, a "bring-your-own-storage" option may be the only option here. I made some progress towards this, but nothing was finalised, and unfortunately it's not something I'll be able to pick up in the near term.

I still feel like there might be a non-flatnotes specific solution here, e.g. some kind of file system similar to MergerFS.

[alexindigo]

Hey @dullage, this looks like the cornerstone feature for quite a few folks.

I'm curious to understand, if you just treat any subfolder as part of the file name, e.g. you're ok with spaces in the name"My Note.md", so why not to have "My/Note.md"?

If the concern that it may get "ugly" with long winded "sub1/sub2/sub3/sub4/note4.md" – it will be user's decision and trade-off.
And it could be as similarly ugly with having spaces in the long filename.

The main argument for me, is that markdown supposed to be app independent, and having ability to put it into arbitrary folder structure is essential for this.

As a useful benefit, it will also allow to pull in notes from different sources, for example, I count mount (and auto sync) notes from my computer into data/dad, and from my wife's computer into data/mom, and we still edit them in our Obsidian apps, while make them available to read in the wiki for the whole family.

Please consider this option, it will open up your project to many new use cases and users.

[jorgeEF]

I just stoped using flatnotes as it doesn't fit my use case because it forces a unique way to work with the notes in a single folder.

[alexindigo]

Here #321 I added prototype changes, to show case that my proposal doesn't require a lot (or architectural) changes, and won't change things for users that use it without slashes in the names.

[oxidand]

I'll subscribe to this thread, because this is the only thing stopping me from using flatnotes. It's by far the most suitable web notes solution I could find, if only it could read notes in subfolders (I'm completely OK that new notes will be created in the root.

Maybe having something like vaults (Obsidian terminology), if I could use one flatnotes instance to switch between multiple folders with flat notes. All I need is a way to separate my work notes from my personal ones.