Suggestions for the use of notmuchfs with dovecot (IMAP)

[maertsen]

Hi Tim,

Thanks for writing notmuchfs! I've been trying to use it as fallback for imap clients in combination with my main use of mutt-kz/alot. In ISSUES you state the question "An IMAP or POP server could run on top of notmuchfs easily. What would make this more useful?".

I've run into some things which could be improved, in particular for the combination notmuchfs + dovecot. Allthough I lack the C skills to contribute code in the short term, I wanted to share my thoughts with you.

usability

• do something sensible with the dovecot control and index files. Currently, I instruct dovecot to put those outside the Maildir / in memory, e.g.

mail_location = maildir:~/Maildir:INDEX=MEMORY:CONTROL=/var/dovecot/%u

but it would be more sensible to allow passthrough for both. I don't fully understand why this doesn't work in the first place, but I guess it has something to do with dovecot trying to obtain locks, which triggers unsupported errors in FUSE.
Documentation for these dovecot features can be found at: http://wiki2.dovecot.org/MailLocation

• perpetuate the size lie to the Maildir S= flag in the filename of individual mail messages, which dovecot indexes. Currently this triggers errors in dovecot, causing it to assume its index is corrupt:

Feb 25 16:14:46 hostname dovecot: imap(user): Error: Cached message size smaller than expected (6209 < 7233)
Feb 25 16:14:50 hostname dovecot: imap(user): Error: Corrupted index cache file (in-memory index).cache: Broken physical size for mail UID 50707
Feb 25 16:14:50 hostname dovecot: imap(user): Error: read(/home/user/Maildir/.spam/cur/#home#user#Maildir.real#.6-in-black#cur#1333623703.M275836P2052.hostname,S=6209:2,S) failed: Input/output error (FETCH for mailbox spam UID 50707)

See also: http://wiki2.dovecot.org/MailboxFormat/Maildir

improvements

1. allow the creation of tags, e.g. by moving messages to some special. Sample use-case: moving a message from inbox (such as illustrated in your example) to a folder with tag:needsreply, which in turn would add that tag.

[maertsen]

Hi Tim,

I've done some more thinking about my ideal notmuch filesystem API geared towards remote/IMAP access and have created a design document. Your input is most welcome!

[tsto]

Thanks for your feedback. I don't have a dovecot test environment, but I'm guessing that
locking doesn't work not because flock() or lock() doesn't work, but because mknod() isn't implemented (so dovecot can not create its lock file). This is quite invasive to implement (Where would the file actually be stored? getattr(), readdir(), unlink(), read(), write() etc. would all have to be aware of it, etc.)

Perpetuating the size lie into the 'S' flag would be straightforward though.

Adding/removing tags by moving pseudo-files to directories with different names would be tricky to understand because the directory names are arbitrary notmuch queries, not just 'tag specifications'. A cleaner way (and one that would fit with more mail clients) would be to support changes to the fake X-Label header, and translate them back to tag add/removes. But this is also problematic (but possible) with immutable messages, that's why I didn't originally implement it.

[CyberShadow]

One problem with using Dovecot over notmuchfs is that it's not currently possible to do inotify on FUSE. This means that there is no way to detect changes in the notmuch database and propagate them instantly to IMAP servers and then to IMAP clients (which presumably use IMAP IDLE). Thus, clients using IDLE will not know when something changed (incl. when new mail was received); Dovecot would need to be configured to use periodic rescanning, or the clients would need to use polling instead of IDLE.