Standard Src Field Mapping for all pods

[Harshita-mindfire]

This is the discussion for RFC Standard Src Field Mapping for all pods.

[hikchoi]

Thank you for starting the thread.

Reposting a relevant discussion point that came up in #2410 :

---

this isn't actually an error though. when we export tags, the pod should read all tags. or if its not, its something we should discuss the behavior for in [[Standard Src Field Mapping for All Pods|dendron://private/task.2022.01.23.standard-src-field-mapping-for-all-pods]]

I would argue that it should not.

The existence of frontmatter tags gives an affordance that says: "here is where you should put all the tags".

Then the counter question as a user would be, "why would tags in the body be mapped when I have explicitly set my tags in the frontmatter?

Wouldn't it be better to explicitly state in the frontmatter that I want these tags to be used in the source field mapping rather than looking at all tags in the NoteProps?

Consider this case:

I have a task note. It's a fictional task to refactor my notes. I jot down some ideas for how to organize tags for my personal notes.
I have set some tag to categorize this task in frontmatter.

Some of my sentences in this note will look like this:

• #scope.recipe is too big of a catchall. I should make this better
• The number of notes with #scope.undefined is getting out of hands. I need to give them a proper scope!
• These notes should not be in #scope.finance: [...]

Should these use of hashtags be mapped to Airtable?

It's a made up case, but it illustrates my point:

As-is, the success of export depends on the integrity of all the tags used in the note body, if any of the tag is entirely or partially part of the source field mapping defined by the pod. This limitation (or the lack of) is steering the user to write a note in a certain way that is not obvious, but the feature that is forcing the user to do so has no part in aiding the creation of note content. There is both immediate cost (non-zero cost writing / making sure it can export) and deferred cost (surprises down the line when I realize these exported notes do not look like what I expected to be in the destination) in the export pod behaving like so.

With the export pod defining source field mapping, I am basically now blocked from using those tags in my body if I don't want this to happen.

This goes for both valid, existing tags and non-existent ones as well.

• If it's a tag that is srcFieldMapped, and a singleTag field, it will hit an error because I already have one in frontmatter and just mentioned it again in the body.

  • e.g. try adding this sentence to any of our tasks: Maybe we can do X here, but this would be out of scope. Let's create a new task to follow up. as a #sprint.next item.. It won't if this task note itself already has a sprint tag in frontmatter.

• If it's a tag that doesn't exist yet, you can't even mention them if it's under any tag hierarchy that has been source field mapped. because Airtable doesn't know about these tags and refusing them.

One could argue that I can just escape the hashtag or just not make it a tag (just use wikilinks). That is indeed one solution, but it's not zero-cost. and it leaves the users wondering. It's preventing me from writing notes in a certain way.

[hikchoi]

@kevinslin 's response to this:

---

I think there is value in supporting both cases. The way we can support that with frontmatter mapping is by limiting the scope in the "to" filed

# pull tags from anywhere in the note
Scope: {to: "all", type: "multiSelect", filter: "tags.scope.*"}
# pull tags from the frontmatter
Scope: {to: "fm.tags", type: "multiSelect", filter: "tags.scope.*"}
# pull tags from the body
Scope: {to: "body", type: "multiSelect", filter: "tags.scope.*"}
# pull tags from a specific section in the body
Scope: {to: "body#section1", type: "multiSelect", filter: "tags.scope.*"}

[Harshita-mindfire]

I have proposed the idea of having a scope property instead of making changes to the existing to field in the RFC.

Scope: { to: tags, type: multiSelect, filter: "tags.size.*", scope: all },

This is because in the below mapping, it is not very clear that we are trying to map with all tags present in the note.

pull tags from anywhere in the note
Scope: {to: "all", type: "multiSelect", filter: "tags.scope.*"}

[hikchoi]

I think the dissonance I had was this:

my immediate understanding of source field mapping, was that they are the frontmatter of this note, that will be mapped to the pod destination.

In reality, source field mapping covers the entire NoteProps, and frontmatter was only part of the story.

I agree with what @kevinslin suggested (which is also covered in the RFC. The linked issue was created without prior knowledge to this RFC), but I think my misunderstanding highlights that this can be confusing.

Especially for our end users who have no knowledge of what the NoteProps object looks like internally.

I think it would be important to provide a clear documentation of this moving forward as this would be a standard for all of our built in pods as well as a base for creating custom pods that could be contributed by our users.

[kevinslin]

type field enumeration

type: type of this field in destination

can we have an enumeration of all types?

using filters vs filter

filter: pattern to filter values of to field.

should this be filters instead to apply multiple filters?

more generic value transformation

statusSymbols: status symbols used in the frontmatter.

This is very specific for todos. what we are doing is remapping values. Can we make this more generic? For example, we can take a clean parameter that will perform a series of actions on the value. for now, we can support just remap.

Status: {to: status, type: string, clean: [
    {action: remap, data:{x : 'CLOSED', w: 'OPEN'}}
]},

another motivating example. when we upload a scope, we use #scope.workspace. we might want to strip the prefix. a hypothetical action

Size: {to: tags, type: singleSelect, filter: "tags.scope.*", clean: [
    {action: stripPrefix, data:{prefix: "scope."}}
]},

scope limiting for links

Additionally, for mapping tags, we can have the scope property to limit the scope.

This shouldn't just be for tags but also links

scope limiting for single select

Additionally, for mapping tags, we can have the scope property to limit the scope.
This should be valid for only multiSelect type.
scope can have the following values:
fm: Frontmatter of Note
body: Note body

why only multiselect? for single select, i could also have a single tag in the body.

scope limiting to section in body

something we want to support is also mapping to a section within a document. so the following config

Scope: {to: tags, type: multiSelect, filter: "tags.size.*", scope: section#header-1},

applied to the following note

## Header 1

#scope.workspace

## Header2

#scope.basics

would extract #scope.workspace

• NOTE: this isn't implemented today but would like this to be noted in the spec

[jonathanyeung]

can we have an enumeration of all types?

Is this possible though? How can we enumerate on all field types across all potential services?

[kevinslin]

we wouldn't. we should have some primitives (eg. string, number, booolean). these types affect how dendron sends data to the service (we will format dates differently then strings), not how the service receives them.

we can have an escape hatch of object which can be custom defined

[kevinslin]

json schema lookup

something we called out when starting is to model this with json schema in mind and support similar validation rules. . at the minimum, we should support the following:

• declare if a field is required
• declare if a field accepts null values
• declare if a field accepts blank values

can we apply [[Learn and Adapt|dendron://dendron.handbook/handbook.company.values#learn-and-adapt]] from existing mapping languages? in this rfc, we should have a sectio on prior art (eg. json schema, https://www.prisma.io/, etc)

additional considerations

from our use of airtable, some additional parameters we want to support based on our usage with airtable:

• if a value is blank, do we want to sync? should support the following (eg. skipOnEmpty, skipOnNull, deleteOnEmpty)
  • eg. if owner is empty, don't upload the field (vs uploading an empty string)

[Harshita-mindfire]

json schema lookup
something we called out when starting is to model this with json schema in mind and support similar validation rules. . at the minimum, we should support the following:

declare if a field is required
declare if a field accepts null values
declare if a field accepts blank values

I hope I'm understanding it correctly, are we talking about a JSON schema for srcFieldMapping, like we validate other config options with AJV? If so, can we support a JSON schema for this? There will be random key-value pairs that are very specific to the service.

[kevinslin]

So not talking about explicitly using json schema for this but rather, to learn from the json schema spec in the design of this feature. this RFC should cover:

• how to define mappings (eg. field from dendron to field to external service)
• how to define constraints (eg. fields that are required vs optional)
• how to define behavior (eg. fields that are empty)

[kevinslin]

currently, we are missing constraints and behavior and we also don't enumerate over mapping types

[SeriousBug]

Minor question, what if there's a required or skipOnEmpty that you are trying to map? I think the field mappings should be nested into another key so they don't clash with the options.

[jonathanyeung]

Rather than have them at the same level of the map keys, let's put required, skipOnEmpty, default to be on the per-property level. That way we can specify the behavior for each property, and avoid the problem @SeriousBug mentioned.

[kevinslin]

agree on having the requirements be per-property. that being said, do we know the rationale of why json schema has required be a standalone key?

[kevinslin]

The type field can have following values:

can we have a description of what the types mean? especially singleSelect, multiSelect, linkedRecord

clean: array of actions to perform on the value.

1. can we specify what the built-in actions and how they apply?
2. if you have multiple actions, what is the order?
3. does a default value go through cleaning?

---

other comments:

• examples, some use filter whereas others use filters

[kevinslin]

can we also enumerate valid scopes?

[Harshita-mindfire]

Sure. Was thinking whether we'd want to enumerate clean actions as well? Currently we just support remap.