Data Manipulation Query Support

[sheeley]

Dataview currently solves, in an amazing fashion, for the R in data CRUD. As I've been writing both DQL and JS, I've regularly wanted CUD capabilities. My general use case/pattern in these situations is:

1. Identify a set of notes that I'd like created, updated, or deleted
2. Either
   a. use Dataview to generate... subpar CLI scripts like echo ${template} > "${noteName}.md" for each file I'd like created
   b. click each link, update as fast as possible manually, wish for an as-seen-on-tv-simple tool
3. assuming automation (2a), run for a single or small number of notes for validation
4. F it, ship it

I think it'd be a valuable addition to Dataview to:

1. Add a live-query experience like Inline Query for Live Preview #728 that can be used to identify a set of notes that would be modified
2. Add syntax for modification/creation of:
   a. frontmatter
   b. metadata
   c. non-frontmatter content.
   d. full content
   e. create/modify dates (keep existing or set explicit)
3. Add previews for:
   a. full file delta
   b. frontmatter only
   c. text only
   d. parsed metadata only
   e. files to be deleted
4. Confirm before running (dialog or key combo - return vs cmd&return, for example)
5. Consider supporting:
   a. automated snapshotting integrations, if possible
   b. events to notify other plugins
   c. plaintext log of changes to be made, machine/human readable for automated undo/understanding context

I'm currently thinking about something like this:

(create | delete | update) 
(set (frontmatter | tag|content)(key)=(expression|value))
(set date (create|modified)=date)
where (...)
annotate (comment)

Experience

I'd expect something like:

Query:
<textarea>
Results:
<paginated list of notes, showing action and enabling preview of delta on hover>
<input clear> <input submit>

This should be accessible via both DQL and JS, and both should support pre-populated queries/actions.

Examples

Create Notes

I capture a number of quotes and create "homepages" for authors that don't have them weekly or so.

create
set content=file("template/author")
where (pseudo: find non-existing notes that are referenced as `author`)

Update Notes

I accidentally created a number of notes with person: true rather than record: person and wanted to fix.

update 
set frontmatter record=person
remove frontmatter person
where person

When using Bear, I had a tag that I prefixed with an underscore to force its sorting. In Obsidian, I don't need this anymore, so:

update set tag #_tag=#tag
where #_tag

Then there are just some times where I may want to execute sweeping change and easily visualize it.

update set content=replace(content, "peanut butter", "almond butter")
where record="recipe"

Maybe I do want to set the create or modification date:

update set tag #2021=#2022
set date modified
where #active

Or I want to backdate some creations:

create
set content="Buy Apple stock!"
set date create,modified=1991-01-01

I've finished a project and can delete its notes, capturing the file list in finished-project just in case.

delete
where #finishedProjectNotes
annotate finished-project

Additionally, it probably makes sense to support tasks as a first-class citizen. Perhaps that means it would be worth supporting create|delete|update page|task or similar.

Feedback on any of the above is certainly welcomed. I've got a very clear set of examples I keep running into, but I expect other perspectives will differ, particularly with changes as impactful as content updates or deletes.

I've been otherwise occupied and unable to contribute for a bit, @blacksmithgu, but if this sounds interesting to you, I'd be interested in helping to implement.

[blacksmithgu]

I broadly agree with the value in update queries as the next logical step in Dataview, though I'm mostly concerned with a few things we should iron out:

1. Safe implementation - once we go from "mostly just reading data" to "reading and writing data", we introduce the danger of data loss due to error or bug. We should consider either a backup mechanism OR some other way of ensuring that changes can be easily reverted.
2. Clean updates - Metadata can appear either (1) implicitly, (2) in frontmatter, or (3) via relatively complex inline fields. We'll need to figure out the mechanics for how to safely and efficiently update implicit and inline field metadata (frontmatter is pretty easy, since we have existing YAML libraries to do this, though we need to make sure it changes as little as possible).

The syntax I think seems fine, though we should start by adding mutable JavaScript APIs for testing purposes before we implement the general DQL API.

[sheeley]

Totally agree about safety. I was thinking about two phases:

• a preview like VS Code provides when doing a multi-file search and replace
• a begin; update; commit transactional experience.
  I think they're actually complimentary, particularly if it's possible to trigger snapshotting at begin and commit.

Your point about metadata is also very accurate. I'd suggest that operations can affect frontmatter, metadata, content, or raw.

• frontmatter: metadata specified in frontmatter
• metadata: metadata anywhere within note
• content: non-frontmatter
• raw: whole note