Real-Time Scanning

[MrDaisyBates]

I'm guessing the real-time scanning only detects changes to the markdown file if it was done inside of Obsidian?

I'm a bit of an outlier. I tend to use Tangent to type, create, etc my notes, and I primarily use Obsidian just to interact with my notes (bases, graph view, etc).

Tangent is pretty "pure" in the fact that any changes to files (regardless of where they come from), they just show up in Tangent in real time. So, I was kind of surprised when (while Obsidian was open), I tampered with something that was on my to do list (in Tangent), and then I tried to mark it completed in Task Board, and it prompted me with a conflict. Kind of "git" styled.

I see there's a scan upon opening of Obsidian option, but I suppose the way the plugin is made, a true real-time experience is out of the question? I'm likely to create these conflicts frequently as I tend to keep Tangent and Obsidian open simultaneously and dance back and forth between them depending on what I'm doing.

[tu2-atmanand]

Hi @MrDaisyBates !

Thank you for bringing this up. Because, recently even I was thinking on this topic and also I think I havent spoke on this topic before, but an overview is present in the docs : https://tu2-atmanand.github.io/task-board-docs/docs/Features/Auto_Scanning_Files/#real-time-scanning

Now, your concern is valid here. While designing this plugin, I keep a note of this case. Becuase Obsidian works on simple text files which lies on your system. The data is not stored inside any database or custom file formats like other note-taking apps. So, for any user its very easy to edit the files outside Obsidian. And this will likely going to create a lot of conflicts when a plugin within Obsidian environment is trying to make operations on the same file simultaneously. For example, this kind of situation is very likely during sync operation, specially if user is using an external sync service like apple cloud, dropbox, etc.

As you have correctly mentioned, to handle this case, I implemented the "Auto scan the vault on Obsidian startup" feature. But this feature only works when you open the Obsidian application after the changes was made to the note in the vault.

Possible Solutions

Now, to address the case which you have mentioned. That is, when the Obsidian application is open and you are making changes the files from the vault through an external application. I have two solutions for this.

But these below solutions will only work, if Obsidian itself will be able to detect the file changes through this external application.

Can you please test this :

• Open the note inside Obsidian.
• Now edit the same note through your external application. In your case its Tangent.
• Now switch back to Obsidian and check whether the content has been updated or not.
• Also, you may open Obsidian and Tangent side-by-side in your screen and check whether the data is updating in real-time inside Obsidian Live Editor/Reading Mode/Source Mode.

All plugins depends on the APIs provided by Obsidian, so if Obsidian itself can able to detect these changes in real-time, then it will be very easy for Task Board to detect them. Please confirm this, to get better understanding.

An indicator approach

An easiest solution will be and I think this will be implemented anyways in the future is to provide an indicator on the "Refresh" button whenever a file has been updated, either through Obsidian or through any external service.

Advantages :

• Task Board will be working in its optimized form as before, it wont be scanning the updated note neither will refreshing the Task Board view continuously whenever the file will be updating.

Disadvantages :

• User will need to do an extra step of clicking on the "Refresh" button to get the latest changes and refresh the view, before making any changes.

A True Real-Time Scanning approach

Now, just to give an overview of why Task Board's Real-time scanning was designed the way it is now, is because, whenever a user is making any changes to the note, a trigger is fired continuously. For example, if user is typing something in the Obsidian editor, a trigger is fired every 1 second to indicate that a particular file has been updated. The problem is, Task Board cannot know, for sure, when user has stopped making the changes to the note. So Task Board also has to depend on this trigger to update its cache. And, if say, we make Task Board to work in "True Real-time" mode, it will continuously be scanning the file every one second and if the view is opened, it will be continuously be refreshing the view every 1 second. Now, even though it looks very minimal processing operations and with latest hardware, these operations looks insignificant. But still these operations are consuming power. And I have this ADHD (or I dont know if I am too optimist), it doesnt feel right to me. I feel like these are unnecessary power consumptions and a better approach can be used.

Hence, I came up with this case where, even though user will be typing continuously in the note, Task Board will scan the file and update the view only after user has switched their focus from that editor. But, I also have noticed that (rarely), sometimes these event of user switching the focus from the editor to other tabs is not detected properly and it misses to update the view, which can lead to conflicts, but this is very rare. (I guess the indicator will be able to help users to take a note of this in future)

Coming back to this "True Real-time" scanning mode approach, then its simple to implement, but I will still need to think more on a better approach. You will also see this approach being used by the CardBoard plugin. Let me know if CardBoard plugin is able to scan the changes made through external application in real-time. This information will help to get more understanding on this mechanism of Obsidian.

Advantages :

• User will get the changes on the board in real-time.

Disadvantages :

• More resource consumption in background when user is editing their notes.

[MrDaisyBates]

Hello again, @tu2-atmanand !

Yes, so, I basically did exactly as you suggested.

I:

1. Opened a note in a tab in Obsidian and Tangent
2. I opened Task Board in another tab in Obsidian and activated that tab
3. I added a task to the note in Tangent
4. The item did not immediately appear in the undated column
5. I switched to the note tab in Obsidian (still open, of course), and it was populated with the new task
6. Switching back to Task Board, it now appeared in the undated column

I'm certainly not a programmer at this point in my life (though I've been pondering over the years how essential of a skill it is and I'd like to make time to learn some basics in the future. Maybe next year).

Rationally, I have a general sense as to why this situation is so complicated even if I don't fully grasp the challenge of implementation.

In my mind, I almost imagine a long term solution being kind of like git in the fact that, unless designated otherwise, any changes within the Obsidian vault folder would start being "tracked" automatically in this case, and if a task's text formatting appeared, it would populate it into Task Board.

I assume this is something that maybe you are restricted from doing for some reason (maybe there's a privilege restriction, or maybe there's huge performance restrictions, etc. Again, it's over my head, for sure).

I am very sympathetic about the ethics/morality of purposely making an incredibly inefficient plugin that is scanning every 1 second as you described. That's kind of why I'm talking from a perspective I understand, even if it's limited in this case.

Git is def the thing that keeps coming to mind for me because it's the closest thing I can think of. I'm guessing git tracking a folder is implemented how you described? Git is so old and I'd assume it'd have a more effective and efficient approach than that, but I can't know.

I will also take your advice and try the CardBoard plugin and see what result I get in that context and I'll update you on the status following. It may be a few hours before I get a chance to try, but soon enough. I saw the plugin when I found yours but I hadn't tried it (I mainly avoided it because it appeared as if it hadn't been maintained for a while).

Anyways, thanks again for your consideration on this matter. I'm optimistic we can find a solution that works for my case as well as is far more efficient than your worst case scenario of scanning every second, for example.

[tu2-atmanand]

I will also take your advice and try the CardBoard plugin and see what result I get in that context and I'll update you on the status following. It may be a few hours before I get a chance to try, but soon enough. I saw the plugin when I found yours but I hadn't tried it (I mainly avoided it because it appeared as if it hadn't been maintained for a while).

Sure!
But I only have mentioned it just to get a more insights on the behind the scenes part. I will also not recommend using CardBoard for daily use, since its development has been halted, so anything can break at any time.

Git is def the thing that keeps coming to mind for me because it's the closest thing I can think of. I'm guessing git tracking a folder is implemented how you described? Git is so old and I'd assume it'd have a more effective and efficient approach than that, but I can't know.

I am not sure if you are referring the Git application or the Git plugin within Obsidian. But just to give an overview :

Git Application

Git application is very powerful and very optimized. Its way powerful then Obsidian when it comes to optimized file changes tracking, recording and storing the history. The reason for this is, the tech-stack used here. Git application is build using C/C++ and Obsidian is build using Electron technology.

When you build something with languages like C/C++ you are talking to the hardware/system very efficiently. But when you make a software using frameworks like Electron. Then Electron framework runs on a browser engine. So basically when you run Obsidian you are technically running a browser engine in the background. This is the same engine on which most of the browsers are built-on, for example Firefox and all. This the reason why you can open websites in Obsidian. And now why Obsidian is less powerful then Git application? The reason is, there is a middleman between Obsidian and your file-system/hardware, which is the Electron engine, along with a lot of other things. So, application like Obsidian cannot interact with your file-system as efficiently as applications like Git. And sometimes, Obsidian might have few restriction compared to Git application, since there are lot of middlemen here.

Git plugin

Plugins in Obsidian have again further restriction compared, as you have mentioned. So, if any plugin wants to do something directly to the file-system, that might work on few operating system, but might not work on other, like mobile phones. And as a plugin developer you might need to think from all perspectives. Hence, a plugin developer will need to depend on the APIs provided by Obsidian, so more middlemen when interacting with the file-system.

Git plugin specifically is not doing anything special, its simply reading the changes detected by the Git application, so its completely dependent on Git application itself.

Now these information might not be that relevant to our discussion and I can be wrong in few places. But since you mentioned that you are interested in programming, so thought of sharing my knowledge I gained over the years.

But, since you have mentioned about Git, I actually have few approaches which can be inspired by how Git application works. Like only detecting which files where updated when Obsidian was closed last time by reading their last updated time, etc. I will think more on these approaches and will share later after refining.

[tu2-atmanand]

Do you have any idea what Bases is doing? Like, how they're handling that?

I havent used it extensively yet, just have experimented a little bit. Bases are nothing but a way to visualize the notes in your vault. Actually, the reason why its so smooth is because, Obsidian is maintaining a cache of all its file metadata. And whenever any file is changed, this cache is updated as well, inturn updating the base view. And, why Bases are more optimized than Task Board is because, Bases cannot see the data from your notes, it just reads the meta-data of your notes. So these scanning operations are very lighweight. Whereas for Task Board to detect if there is any inline-tasks inside the note, it will need to read and scan the whole data of the note. This is the main problem we are trying to solve here. Git application and Bases plugin only reads the meta-data but Task Board reads the complete file content.

Actually, for Task Board also its not a big thing to implement a "True Real-time" scanning mode. A user will not going to notice any performance difference at all. The plugin will work as its working now. The only reason why I didnt implemented this mode is because of my concern of the plugin using a tiny bit of extra energy out of your system and i was feeling uncomfortable that, whenever I am typing something inside my note, something is running in the background processing the file again and again. Which is a specially a case on mobile devices, laptops. But its still going to be very insignificant. I guess, I can implement this mode, just to see if there will be any difference at all. Since, I didnt noticed any such performance difference when I was using CardBoard a long back.

[MrDaisyBates]

No, for sure. I'm quite happy to give CardBoard a try. I just got home from what I was doing and will probably snoop into that now.

Git, I actually mean git the version management stuff that software developers and stuff use (not github specifically...that's just an implementation of git). https://git-scm.com/about#free-and-open-source

I'm not very qualified to talk about git. I assumed you were probably more qualified than me, but I'm assuming you're talking about git the application in this case. I def don't mean the plugin. But git is a technology used to track and monitor changes to notes/files within a directory, and can be used to change the state of the entire directory or individual files back to any "committed" (tracked) previous state. I believe it's incredibly efficient, lightweight, etc and is stellar at detecting even minute changes and allowing you decide which you want to keep, etc (handling merges and whatnot).

I know basically nothing about Electron, too. I'm sorry if it's too limiting for my suggestion here. I just know that git is very effective at immediately detecting changes and is very lightweight. I actually manage my note folder using git (via GitKraken) which is why I know even what I do.

But yes, anywho. I'm speaking from a level of ignorance in these areas and I apologize if they're not all that useful. They were just things that came to mind that seem to parse very large amounts of data/changes without being too wasteful about it.

I'll now go checkout the CardBoard plugin and see what kind of responses I get out of it when it comes to the real-time changes when a note is closed in Obsidian but open in Tangent. Brb.

[MrDaisyBates]

Okay, @tu2-atmanand . I'm back.

Yes, I installed the CardBoard Plugin, changed the filter list to a deny list (and it was blank, so "deny nothing"), and I created an "undated" column on a new board.

I made sure the note I was interested in tampering with was not open in Obsidian. I went to Tangent and I added a new task. As soon as I was done typing it, I moved my cursor to a different line (never hit enter), and I clicked over to Obsidian and it was already on the list.

I then opened both applications side by side. I'm on Fedora 43 if it matters, but while Tangent was the active application, I could type and modify the task all I wanted and nothing would happen in Obsidian. Whenever I activated Obsidian, however, with absolutely no noticeable latency, the task updated to however I'd modified it in Tangent while away from Obsidian.

So, if their plugin is open source, you might consider snooping through their code and seeing if it's obvious how they handled it. I have absolutely no idea what's going on behind the scenes and if it is or isn't efficient, but it does appear to at least be effective.

[tu2-atmanand]

I'm not very qualified to talk about git. I assumed you were probably more qualified than me, but I'm assuming you're talking about git the application in this case. I def don't mean the plugin. But git is a technology used to track and monitor changes to notes/files within a directory, and can be used to change the state of the entire directory or individual files back to any "committed" (tracked) previous state. I believe it's incredibly efficient, lightweight, etc and is stellar at detecting even minute changes and allowing you decide which you want to keep, etc (handling merges and whatnot).

Yes, this is the same technology I was referring to when I said Git application, which we have to separately install in our system. I think GitKraken is high modified version of the vanilla git application. But under the hood its using the Git technology for sure.

Honestly speaking, even I dont know how Git works and its behind the scene algorithm. Its kind of legendary if we talk about how its designed. But during my last reply I wasnt actually talking about its ability to detect the changes in the content, this is very complex algorithm they are using. I was only referring to how its detecting any file changes, which is basically, the system is telling the Git technology that, "hey this file has been updated by the user...", then git runs its algorithm to find the changes done to the content by the user.

But now I have understood you better, you were sharing the idea where, how Task Board can detect the changes done to the file content and find whether there is any inline-task was added or not. This is actually another topic (and pretty complex one, IMO) for discussion.

With my knowledge of Git, I think there is no Git interface which an Obsidian plugin can use to detect the file content changes. But still Ill need to research on this.

[MrDaisyBates]

I know I wasn't explicitly asked to, @tu2-atmanand , but I'm in regular contact with the developer of Tangent, so I've actually reached out to him directly and he seemed welcoming to being able to potentially have his brain picked on the topic. I've stated the general nature of the issue in the Discord and I've also linked our discussion. I'm not sure if he will prefer to come here directly himself or if I'll just relay it from the Discord. He's quite familiar with the context and he's incredibly experienced, which is why I asked him in the first place. I'm sure you don't mind and you're more than welcoming of it. I just wanted to give a headsup so it's not out of the blue.

[MrDaisyBates]

After talking to Tangent's creator, while he's not an Obsidian Plugin developer, he seems quite confident that this command would do the job at detecting changes to any file, even if it was changed via Tangent. My understanding is it also wouldn't require it to scan the vault every second, for example, as you were originally concerned about.

https://docs.obsidian.md/Reference/TypeScript+API/Vault/on('modify')

[tu2-atmanand]

That's really kind!

Okay, so let me implement a "True Real Time" mode. So, now user can able to switch between :

• Real Time (This is the "True Real Time" mode we are referring to in this discussion)
• Optimized real time (This is the current real time mode present in the plugin in the current version)
• Manual (This is basically when user turns off the "Real-time scanning" setting)

Ofcourse, Ill release this "True Real Time" mode in the testing phase first (beta release) for proper testing. After that it can be released for normal use. And users can decide whether they want to switch to this mode. The "Optimized real time" mode will be the default one as it is in the current version right now.

And, I am pretty sure, in todays even mid-range systems, this new mode will be as smooth Bases and all, so there shouldn't be any problem at all.

[tu2-atmanand]

Created a ticket to track the development : #619

[tu2-atmanand]

Also, the reason why I am researching on finding an optimized approach for detecting file updates when user is continuously typing inside a note which is triggering the 'file modified' event is because. Few months back, this topic caught my attention : https://forum.obsidian.md/t/disable-auto-save-or-change-frequency/14230

[tu2-atmanand]

Alright!

Now, just to revise everything again, here is a problem we are trying to solve :

If a user is continuously typing inside a note, may it be within Obsidian or may it be an external application like tangent. The thing which matters is when this data is being stored to the disk. For example, below is the demo, with both Obsidian and Tangent software :

With Obsidian

So, on the left you may see the events triggered whenever the file is being saved to the disk. What this means is, Obsidian saves your changes every 2 seconds to the disk. The software is designed this way, so user never looses their data. But, the problem here is that, all the plugins relies on this event to find out if any file has been updated or not. And since, Obsidian is triggering this event, even when the user is continuously typing in the note. As a plugin developer, I found this approach very inefficient. But as a user this is a very powerful approach, since user will never going to loose their data. In events like the system suddenly shuts down and all.

With Tangent

My first impression of this software, I am really impressed. Because, this software has used the same approach which I used while developing Task Board. Because, as you can see Tangent software is not saving your changes to the disk in real-time. It only saves them, when user changes their focus from either that file to another file or switches to a different application itself. Thats why in this case, the event is only triggered when I switched to a different window after making this changes.

Further thoughts

So, its definitely possible to upgrade Task Board with this third mode, where, when user will make any changes in external applications and switches to Task Board, their changes will be instantly refreshed.

But to show what a "True Real Time" mode is, which is how the CardBoard plugin works by default. Here is an Gif :

Here you may see that, whenever a user is making changes to their note, their whole focus should be inside the note, but since Obsidian triggers the event every two seconds, the CardBoard view is getting refreshed every 2 seconds. According to me, this is both inefficient as well as unnecessary, since if user is typing in their notes, their whole focus is in the editor, so the CardBoard view need not have to update in real-time. It should only refresh after user has stopped typing or basically when user switches to the CardBoard tab. And the reason why this is inefficient is because, even todays mid-range systems wont show any performance lag with this real-time operations, still there is a power consumption going on. A "True Real Time" mode in Task Board will look something like this.

But, now that I have used Tangent, I will also make sure that :

• Even in "Optimized Real Time" mode, when user will switch from external application to Task Board, it should refresh its view only that time, since this will happen only once and not every 2 seconds, so its actually optimized anyways. And in the current version itself, this should have happened, but I think Task Board missed to detect this change and missed to refresh the view. And this was the main reason why @MrDaisyBates got the content conflict screen. So, will need to optimize this as well.

[taylorhadden]

Hi there! Developer of Tangent here. Thanks for the kind words!

The solution here may be "debouncing", where the reaction to a change is delayed every time a new change comes in. Say you update three or four seconds after the last change event. Every new change event would cancel the last pending update and queue a new one. This is what Tangent does for autosave (in addition to saving as soon as you switch focus).

As a side note, Tangent does use views similar to what you show as "true real time" for its card views. It responds not to file changes, but to changes in the shared editor state. This is much, much less work than fully re-parsing and re-rendering on each keypress.

@MrDaisyBates is one of our most active users in the Tangent community. He's great at being a squeaky wheel, and Tangent has benefited greatly.

[tu2-atmanand]

First of all, Thank you very much for tuning in.

It's an honor to have you in the conversation, not only because of your expertise, but I have huge respect for open-source developers/contributors. And after coming across Tangent, its really nice to see such a responsive and high quality open source software project in active development. After using it I realized about the features like focused typing is so nice and useful (I might be saying the feature name wrong, as I first time saw it). I haven't used it extensively yet. So, Ill be missing most of its other features. But, my first impressions were really good, very intuitive software. Over time, will get to learn more about it.

Regarding your suggestion of debouncing. I actually had this in mind, but wasnt really sure whether it will really going to solve our original issue. Definitely it will reduce the operations of parsing and UI refreshing, but if the debounce time is more, than the last user change will take some time to show up. For this I guess, I can use the focus change eventListener and that time do the last UI refresh. But debouncing will surely reduce the operations. So, I guess, I will surely give it a try. Until I implement it, I wont really see what it brings to the table. (Also, I am not an experienced programmer, so I hope, I will implement this debounce feature of React properly)

[MrDaisyBates]

Thanks for the kind words and popping in, @taylorhadden ! Love and appreciate you, and I love and appreciate Tangent! It's a must in my day to day life.

And thanks for all the thought and consideration for how to iterate and improve your plugin, @tu2-atmanand ! I'm sure Taylor would tell you that his core vision for Tangent was only partially defined when he started it, what, 5+ years ago at this point? It's become more clear on an iterative basis, month to month and year to year. Sometimes we have no idea where we end up, and it just takes that courage to keep taking one step at a time. Appreciate your commitment to your craft/work. It's the kind that you look back five years later and see just how far you've come.

Merry Christmas and Happy Holidays to you two and your loved ones. <3

[taylorhadden]

Five years this spring!

[tu2-atmanand]

Development Update

Okay, So I have implemented the feature I have explained in the following ticket : #619

Here is how its going to work...

Setting

User can select the mode in which Task Board should work for scanning their updated/modified files.

Rea-time mode

This is a true real-time scanning mode, just like how its in the CardBoard plugin. And to be honest, because there are so many additional features implemented in Task Board, I thought this feature will consume a little more CPU while user is contineously typing in the note. But, its exactly same CPU usage as it is in the CardBoard plugin.

Again, I personally will not going to use this mode ever. It looks good and all. But practically speaking, I dont think, users will ever want it, even if they are on their multi-monitor setup. Or I dont know, maybe it will have a very important use-case if users are syncing in real-time, etc.. For now, I have simply implemented it as a feature and also in an attempt to implement/solve the functionality discussed by @MrDaisyBates in this discussion topic.

But there is a good news, this functionality of "refreshing the board when changes have been made from external source/application", is now possible with the default "Balanced" mode itself, as explained below...

Balanced mode

This is basically the default mode, which is available in older version with the name "Real-time scanning" setting. The name earlier was a little confusing, as it wasnt exactly a "real-time" mode earlier. In this mode, the plugin only detects which files has been modified. And when user switches their focus from one tab to another or from one window to another, it proceeds with the scanning process for these files and updates the content on the board. Thats, why its a balanced version, where nothing is running in the background when user is typing in their note and the board only refreshes when user is actually focused on the board.

In this mode itself, I implemented an additional event listener, where, when user will make any changes in any external software => the file will be updated on the disk => Obsidian will then pick up this changes => And as soon as Obsidian will reads this changes it will notify Task Board => Task Board will scan the files and update the content on the Board.

So, in this mode itself, I hope I have implemented the requested feature. But, there are various other cases like some files updates in background, maybe in case of external syncing or something and if Obsidian didnt pick this changes. Then this will be the limitation of Obsidian and a plugin wont able to do anything in here.

I am also brainstorming on handling part of this situation, but not sure if a plugin will able to do this. Read the forum topic here : https://forum.obsidian.md/t/fr-maintain-a-registry-of-all-the-files-which-has-been-modified-when-obsidian-was-closed/110122?u=tu2_atmanand

Manual mode

In this mode, Task Board will only know which files have been updated by the user and it will keep a note of those files. But it will not do any background work, until user manually clicks on the refresh button and fetches the latest changes.

In the Balanced mode and Manual mode, I have provide an indicator on the refresh button, which tells the user that, some files has been modified and they should click on this button before making any changes through the board. I have kept this feature in the Balanced mode because sometimes, Obsidian or Task Board doesnt able to capture the event when user switches from one tab to another or from one window to another. In this case, the indicator will be a sign that some modified changes has not been scanned yet. Will improve this feature moving forward.

Testing with Tangent

This has been tested in both "Real-time" and "Balanced" mode.

As soon as user will switch away from the Tangent application or when the Tangent application will write the changes to the disk, these changes are picked up by Obsidian and are notified to the Task Board. And rest scanning happens as usual. So latest changes will be present on the Board.

[MrDaisyBates]

Thanks for the hard work!

[MrDaisyBates]

Sorry for my delay in response, @tu2-atmanand ! I just wanted to say I appreciate your effort and contributions to your project here! I tend to take the month of January off, so I've been doing my own thing. But now I'm back.

I've not gotten a ton of hands on with these new changes yet, but at first glance (on the "Balanced" mode), it does seem a lot more responsive to changes that I make in Tangent. The scan that happens 10 seconds after booting up Obsidian is also a very great remedy for the functionality somewhat breaking when modified from an external editor, as well.

Anywho. I really appreciate it and I'm excited to get my hands dirty with it. I think it's a great step in the right direction. Thanks again!

[tu2-atmanand]

Auto detect modified files when Obsidian was inactive

While working on this "Scanning modes" feature, I realized there was another very serious problem which I should handle now. (And which still going to need a lot of attention in the future.).

For now, I have handled this case as explained in the following ticket : #641 .

This looks related to the current discussion hence sharing it here.