File Attachments Proposal

[ryneeverett]

It would be handy if it were possible to attach files to a task. Personally, I would mostly use it for text files, pdf's, and images.

I found #786 which alludes to this but no more recent discussion. One of the main benefits and challenges which doesn't seem to be considered there is synchronization.

While storing files separately from the database is the most performant solution, I doubt the additional complication of the synchronization and storage system would be worth it. Files could be stored as blobs if we set a reasonable size limitation (1gb?). The first problem I'm aware of is that I think we'd need some additional logic in taskchampion to batch large operations that are too big to reliably fit on the wire and then reassemble them on the other end.

From a user interface perspective, I would see this working a lot like annotations, except a file path is given rather than a string and it is stored with a timestamp.

task attach <filter> <name (if omitted use file name)> <path (if omitted open xdg file chooser)>
task open <filter> <name> <application <if omitted try xdg-open)>

[djmitche]

I suppose there are two use-cases for this, which we would need to distinguish: a small text file containing notes on the task, or arbitrary (potentially binary) files. For the former, we could do a reasonable job of "synchronizing" them, using operations like "insert string S at position N" and "delete K characters at position N". This is pretty standard OT stuff, although most of the research assumes that you get to see the user's editing behavior (think etherpad or google docs). We could probably adapt some kind of character-by-character diff approach to turning edits into operations.

For the latter, we would want to do wholesale replacement of the object, with latest-write-wins semantics. Trying to model replacement of one PDF with another as byte-stream edits would be pretty inefficient.

For latest-write-wins, I think we could just us some kind of content-addressable storage approach, storing only the content hashes in the operation stream. So maybe an attached file looks like "att_rfp.pdf": "441c1e3a8ddb7b12c5d48a9c724df0e1c326621b00b43ebb2d227aa2c5650a64faf4fc946534356b4da90294fac60dbf1a7bf2d21bcd35e3412a4c08b69fee72". Updates to a file would just change the hash as a sync operation. The files themselves would be uploaded, encrypted, to the server as a separate API. This is trivial for object-storage (AWS, GCP) and pretty easy to add for taskchampion-sync-server. The server could apply its own size maximum. We would need to think about garbage collection of old objects, but that's not insurmountable.

The downside is, users will be tempted to task 1 attach notes.txt and edit those notes as the task proceeds -- and the last-write-wins semantics will eventually surprise them.

[djmitche]

Hyrum's Law suggests someone will use it for notes, whether we intend to or not!

Locally, they could be stored in the database -- although SQLite's performance with 1GB files might be poor, so we could consider storing them as files on the fs too. My comment was only talking about sync.

As for compression -- yes, compression is always warranted and done before encryption. That might be counter-productive for already-compressed files, but I think modern compression algorithms handle that case with pretty minimal overhead.

[ryneeverett]

Hyrum's Law suggests someone will use it for notes, whether we intend to or not!

IMO that person can't complain about last-write-wins conflict resolution.

But also, I think the client can make it clear that the user is replacing and not editing. I would envision writing the file to /tmp as read-only when they open it. They would have to save it elsewhere and re-attach it to "modify" it.

My comment was only talking about sync.

My comment was responding to these lines:

The files themselves would be uploaded, encrypted, to the server as a separate API... We would need to think about garbage collection of old objects, but that's not insurmountable.

It seems like the separate API and garbage collection considerations wouldn't be necessary if we treated files like normal data attributes. The only additional consideration would be handling larger operations than we're currently equipped for.

[djmitche]

Good point about making the semantics clear.

If the files are normal task attributes, that's potentially a lot of garbage -- every task attach would generate an operation of that size, and even if that file is overwritten the operation creating it would remain in the operation stream.

But for "reasonably-sized" files this might be fine. Maybe to help ensure that we should have some kind of max-file-size config, where users can increase it if they want to but hopefully with the awareness that things might get slow and/or big.

[ryneeverett]

If the files are normal task attributes, that's potentially a lot of garbage -- every task attach would generate an operation of that size, and even if that file is overwritten the operation creating it would remain in the operation stream.

By encouraging large task attributes we're magnifying some aspects of taskwarrior that might already be problematic:

• storage usage is doubled by storing all data both in operations and in the working set
• we might be lacking a satisfactory system for purging old data (Seems like there have been a number of issues opened about this in the last couple years but I don't recall to what extent they are resolved.)

Let me know if I've misinterpreted you.

[ryneeverett]

An advantage of the separate API for files is that there could be an option to synchronize files lazily which would be ideal for low-storage devices. (phones)

[djmitche]

Could attachments just be URLs, maybe with some convenience support for things like GitHub Gists, Google docs, Pastebins, etc.?

[ryneeverett]

I think this idea is basically to adopt some additional features from taskopen into taskwarrior but maybe add some upload support as well.

I think this route is tantamount to saying, "taskwarrior's sweet spot is small strings, if you need files use something else". There's merit to doing one thing well and avoiding bloat.

Personally, I don't currently have a publicly exposed (so it would be available to all devices on which I use taskwarrior) network service to store random files nor a third party service I like. But I might consider getting one (or creating a private network specifically for this purpose) if it had an integration that enhanced taskwarrior.