Optimization Challenge! Pulling related tasks from everywhere into a master page for a tag (person/project/place)

[ces3001]

I have written a dataviewjs that pulls tasks from that page, as well as from tasks across my Obsidian vault that are tagged with “myTag” (that is, the current page’s first alias), or that come from pages tagged with “myTag", or tasks from specific pages listed with includetasksfrom:: [[note1]],[[note2]]. It works great, it allows me to put tasks anywhere in my notes and consolidate them easily... except it’s slow, causing Obsidian to be unusable for several seconds when the “master” note is opened.

I feel contributing this to the community would be helpful to many.

Can someone better at this than me help optimize this code?
Thank you!

const ver = "0.7.2"
// to include in other pages: 
// ```dataviewjs
// await dv.view("scripts/tasklist", {});
// ```
// parameters are optional (defaults shown): {tasksFromThisPage:true, taggedTasksFromAnywhere:true, tasksFromTaggedPages:false, tasksFromIncludedPages:true}
// 2022-08-01 0.7.2 Added capacity to look at [includetasksfrom:: [[link]],[[link2]]] so that links update in page when destination file is renamed. Previously, was static string in the frontmatter.

// defaults
let tasksFromThisPage = true
let taggedTasksFromAnywhere = true
let tasksFromTaggedPages = false
let tasksFromIncludedPages = true
// override any defaults from parameters passed
if (typeof input !== "undefined") {
if (typeof input.tasksFromThisPage !== "undefined") { tasksFromThisPage = input.tasksFromThisPage }
if (typeof input.taggedTasksFromAnywhere !== "undefined") { taggedTasksFromAnywhere = input.taggedTasksFromAnywhere }
if (typeof input.tasksFromTaggedPages !== "undefined") { tasksFromTaggedPages = input.tasksFromTaggedPages }
if (typeof input.tasksFromIncludedPages !== "undefined") { tasksFromIncludedPages = input.tasksFromIncludedPages }
}

let myTag = dv.current().aliases[0] // assume first alias is a tag the defines this person or project
let taskList = dv.array([])
// Tasks from this page
if (tasksFromThisPage) { 
  taskList = taskList.concat( 
	dv.current().file.tasks.where(t => !t.completed &&
		dv.date(t.defer) <= dv.date('today')))}

// tasks from other pages listed in includetasksfrom
if (tasksFromIncludedPages && dv.current().includetasksfrom) {
  taskList = taskList.concat(
   dv.pages()
	.where(p => dv.current().includetasksfrom.some(n => (p.file.name == n) || (String(p.file.link) == String(n))))
	.file.tasks.where(t => !t.completed && 
		 dv.date(t.defer) <= dv.date('today')))}

// tasks marked with this tag from other pages
if (taggedTasksFromAnywhere) {
  taskList = taskList.concat(  
   dv.pages()
	.where(p => p.file.name != dv.current().file.name)
	.file.tasks.where(t => !t.completed && 
		dv.current().aliases.some(alias => t.text.includes(alias)) &&
		dv.date(t.defer) <= dv.date('today')))} 

// tasks from any page which contains this tag, except if tagged #multiproject
if (tasksFromTaggedPages) { 
  taskList = taskList.concat( 
   dv.pages(myTag + " and -#multiproject")
	.sort(p => p.file.mtime, "desc")
	.file.tasks
		.where(t => !t.completed &&
			dv.date(t.defer) <= dv.date('today')))}

// print them out, make sure no dups, and sort by appearance in note
dv.taskList(taskList.distinct(t => t.text).sort(t => t.line))

[ces3001]

The script requires the main page to have an alias (first one) in the form of a tag. (Thanks to Tag Wrangler for the idea of creating Tag Pages— note: Tag Wrangler not needed for this script to work).

---
aliases: [“#MyTag”,”other alias”]
---

[AB1908]

Likely, dv.pages().where(p => p.file.name != dv.current().file.name) is the most expensive part of your code. May want to approach the idea from another direction.

[ces3001]

Thanks for the insight. Actually that line seems superfluous if we are including tasks from current page (which I generally do), and since dups are removed later. I commented it out, and it makes a small difference.

I think the main bottleneck is around this section below, where I reference particular pages I want to include tasks from. Removing [includetasksfrom:: speeds up task retrieval significantly (and could still use optimization on the rest of the code). Obviously, I’d like to keep this feature.

// tasks from other pages listed in includetasksfrom
if (tasksFromIncludedPages && dv.current().includetasksfrom) {
  taskList = taskList.concat(
   dv.pages()
	.where(p => dv.current().includetasksfrom.some(n => (p.file.name == n) || (String(p.file.link) == String(n))))
	.file.tasks.where(t => !t.completed && 
		 dv.date(t.defer) <= dv.date('today')))}

[blacksmithgu]

I think the main issue is you are not making use of the dataview index. For example, in this section:

if (tasksFromIncludedPages && dv.current().includetasksfrom) {
  taskList = taskList.concat(
   dv.pages()
	.where(p => dv.current().includetasksfrom.some(n => (p.file.name == n) || (String(p.file.link) == String(n))))
	.file.tasks.where(t => !t.completed && 
		 dv.date(t.defer) <= dv.date('today')))}

You are filtering over EVERY page to find the pages that are in your list. You should not do this - instead, iterate over the pages in the list directly:

if (tasksFromIncludedPages && dv.current().includetasksfrom) {
  for (let page of dv.current().includetasksfrom) {
    taskList = taskList.concat(dv.page(page).file.tasks.where(t => !t.completed && dv.date(t.defer) <= dv.date('today')));
  }
}

This should be a decent performance boost. The "tasks from anywhere" query may also be slow, but this is more Dataview's fault - I need to set up additional task indices to make finding task files faster.

[ces3001]

Amazing. So simple it escaped me! Thank you. However, I do get an error Evaluation Error: Error: dv.page only handles string and link paths; was provided type ‘object’ when attempting the following piece of code:

let taskList = dv.array([])
if (dv.current().includetasksfrom) {
  for (let page of dv.current().includetasksfrom) {
   taskList = taskList.concat(dv.page(page).file.tasks);
  }
}

It seems dv.page(“[[link]]”) doesn’t work, only dv.page(“link”). I attempted to remove the [[ and ]], but that doesn’t do it because the links also have a |alias embedded in them. Could parse that out too (something like dv.page(String(page).replace(/[\[\]]/g,"").replace(/\|(.+)/g,””))), but it seems like there must be a cleaner solution that deals with this, and allows either inline and frontmatter fields to work.

[ces3001]

So this code works with inline includetasksfrom:: [[link1]], [[link2]] (and it’s frontmatter equivalent includetasksfrom: ["[[link1]]”,"[[link2]]”]) but it fails if there is only one link (i.e. includetasksfrom:: [[link1]]) — I get includetasksfrom not iterable error. Is there a universal code that works with frontmatter, inline, both single links as well as lists? If we need eliminate frontmatter, fine.

if (tasksFromIncludedPages && dv.current().includetasksfrom) {
  for (let page of dv.current().includetasksfrom) {
    taskList = taskList.concat(dv.page(String(page).replace(/[\[\]]/g,"").replace(/\|(.+)/g,""))
    	.file.tasks.where(t => !t.completed && dv.date(t.defer) <= dv.date('today')));
  }
}

[ces3001]

Solution:

if (tasksFromIncludedPages && dv.current().includetasksfrom) {
  for (let page of dv.array(dv.current().includetasksfrom)) {
    taskList = taskList.concat(dv.page(String(page).replace(/[\[\]]/g,"").replace(/\|(.+)/g,""))
    	.file.tasks.where(t => !t.completed && dv.date(t.defer) <= dv.date('today')));
  }
}

[blacksmithgu]

Yes, dv.array() is the correct approach.

[ces3001]

Awesome. Thanks for monitoring these threads and creating an awesome contribution. Will be donating your way asap.

[ces3001]

updated to version 0.9.2.

const ver = "0.9.2"
// to include in pages (simplest): 
// ```dataviewjs
// await dv.view("scripts/tasklist");
// ```
// to include in pages (with all options):
// ```dataviewjs
// await dv.view("scripts/tasklist", {thePage: "Optional page name", tasksFromThisPage:true, taggedTasksFromAnywhere:true, tasksFromTaggedPages:true, tasksFromIncludedPages:true});
// ```
// parameters are optional (defaults to current page, or true if not included)
//  See https://github.com/blacksmithgu/obsidian-dataview/discussions/1509#discussioncomment-4900741 
// 0.9.2	2023-02-17 - Minor cosmetic touches when being passed a file
// 0.9.1	2023-02-16 - Fix a bug when there are no aliases specified
// 0.9		2023-02-07 - Added the parameter to specify a page (thePage) to pull the tasks from
// 0.8.6	2023-01-08 - Added the feature to ignore pages with #ignoretasks tag (similar to ignoring pages with #multiproject) in the tasksFromTaggedPages section
// 0.8.5	2022-12-22 - Bug fix myAliases
// 0.8.4	2022-12-16 - Output "No tasks" if no tasks.
// 0.8.3	2022-12-03 - bug fixes: if not a tag in aliases, or no aliases (so that we can still use this in any page, not just projects or people)
// 0.8.2	2022-11-18 - bug fixes: tasksFromIncludedPages can now handle none, single, list
// 0.8.1	2022-11-17 - bug fixes: tasksFromIncludedPages, can now handle inline and frontmatter fields. 
// 			syntax in frontmatter (note outer brackets and links in quotes)
//				includeTasksFrom: ["[[link1]]","[[link2]]"]
//			  syntax inline (note double "::", and no outer brackets or quotes)
//				includeTasksFrom:: [[link1]],[[link2]]
// 0.8		2022-10-31 - Updated tasksFromIncludedPages per https://github.com/blacksmithgu/obsidian-dataview/discussions/1509#discussioncomment-3985539
// 0.7.3	2022-10-26 - Changed defaults, and added optimization notes and changed behavior taggedTasksFromAnywhere to include current page in an optimization compromise.
//						Currently, tasksFromIncludedPages only works from frontmatter, and is really slow. I thought I had gotten it to work with inline syntax, but no longer works. 
// 0.7.2	2022-08-01 - Added capacity to look at [includeTasksFrom:: [[link]],[[link2]]] so that links update in page when destination file is renamed. Previously, was static string in the frontmatter.

// defaults
let debug = false
let thePage = dv.current()
let tasksFromThisPage = true
let taggedTasksFromAnywhere = true
let tasksFromTaggedPages = true
let tasksFromIncludedPages = true 
// override any defaults from parameters passed
if (typeof input !== "undefined") {
if (typeof input.thePage !== "undefined") { 
	thePage = dv.page(input.thePage)
	dv.span("*From* " + thePage.file.link + "*:* \n")
}
if (typeof input.debug !== "undefined") { debug = input.debug }
if (typeof input.tasksFromThisPage !== "undefined") { tasksFromThisPage = input.tasksFromThisPage }
if (typeof input.taggedTasksFromAnywhere !== "undefined") { taggedTasksFromAnywhere = input.taggedTasksFromAnywhere }
if (typeof input.tasksFromTaggedPages !== "undefined") { tasksFromTaggedPages = input.tasksFromTaggedPages }
if (typeof input.tasksFromIncludedPages !== "undefined") { tasksFromIncludedPages = input.tasksFromIncludedPages }
}

if (debug) { dv.span(">[!DEBUG]\n") }

let myAliases = ""
let myTag = ""
try{
	myAliases = thePage.aliases
	if (myAliases.length > 0) { 
		myTag = myAliases[0] // assume first alias is a tag the defines this person or project
		if (myTag[0] != "#") {
			myTag = "" 
		}
	} else {
		myAliases = ""
	}
} catch(e) { 
	myAliases = ""
	myTag = ""
}
if (debug) { dv.span("> myAliases:" + myAliases + " myTag:" + myTag + "\n") }

let taskList = dv.array([])

// Tasks from this page
// optimization: fast
if (tasksFromThisPage) { 
try{
  taskList = taskList.concat( 
	thePage.file.tasks.where(t => !t.completed &&
		dv.date(t.defer) <= dv.date('today')))}
finally{}
}

// tasks from other pages listed in includeTasksFrom
// 2022-10-31 https://github.com/blacksmithgu/obsidian-dataview/discussions/1509 optimization
if (tasksFromIncludedPages && thePage.includeTasksFrom) {
  for (let page of dv.array(thePage.includeTasksFrom)) {
    taskList = taskList.concat(dv.page(String(page).replace(/[\[\]]/g,"").replace(/\|(.+)/g,""))
    	.file.tasks.where(t => !t.completed && dv.date(t.defer) <= dv.date('today')));
  }
}

// tasks marked with this tag or alias from other pages
// optimization: tolerable
if (taggedTasksFromAnywhere && myAliases != "") {
  taskList = taskList.concat(  
   dv.pages()
	//.where(p => p.file.name != thePage.file.name) // speed optimization compromise: comment out to speed things up (we thus don't exclude current page)
	.file.tasks.where(t => !t.completed && 
		thePage.aliases.some(alias => t.text.includes(alias)) &&
		dv.date(t.defer) <= dv.date('today')))} 

// tasks from any page which contains this tag, except if tagged #multiproject or #ignoretasks
// optimization: fast
if (tasksFromTaggedPages && myTag != "") {  
  taskList = taskList.concat( 
   dv.pages(myTag + " and (-#multiproject and -#ignoretasks)")
	.sort(p => p.file.mtime, "desc")
	.file.tasks
		.where(t => !t.completed &&
			dv.date(t.defer) <= dv.date('today')))}

// print them out, make sure no dups, and sort by appearance in note
if (taskList.length > 0) {
	dv.taskList(taskList.distinct(t => t.text).sort(t => t.line)) }
else {
	dv.span("*No tasks*")
}