December 2019: OpenSMTPD and filters work, articles and goodies

[poolpOrg]

[ngortheone]

implicit local parts of the rules is what I disliked and I am absolutely excited to see them go away!

[terminar]

Hi Gilles,
thank you for the great work with OpenSMTPD and especially the new filter API.
I had to switch our really old mailservers from qmail+vpopmail to something new and had some issues with postfix. Then i looked at OpenSMTPD and you blog entries, tried it, mixed it up with dovecot+rspamd and i loved it!

I was able to port most of our special configuration/functionality to the new setup but "mail queue archiving" was missing (and using dovecot + sieve was not near to the smtpd enough as needed).

Soooo i read your latest smtp-filters man page, took the "filter-rspamd" sourcecode, stripped it nearly completely, used it as opensmtpd-go-filter-template and - created a mail queue archive filter.

• Home: https://teruna.dev/cyberphoria/filter-archive
• Mirror: https://github.com/terminar/filter-archive

It's working for me - so i'm fine with it - but there are some open questions and maybe legal issues i want to fix first of all because I took your sourcecode (filter-rspamd) as start template. I'm not sure what's the legal situation is here for the LICENSE + sourcecode. Yes, the code will be completely open and free.

Open questions:

• Do i have to put everything under your copyright?
• Do you want to be mentioned as first or second author in the LICENSE + headers? (you are currently only mentioned in the README.md)
• as a maintainer of OpenSMTPD, can you take some time for a small code review if i made some horrible bugs or misunderstanding of how the OpenSMTPD filters should be used - and give me some feedback?
• one specific OpenSMTPD question which also concern filters: I'm using email-adresses as smtp auth login (meaning user@domain.tld) but in the debug logs and in the "link-auth" report event, i am only getting the user part, not the "full" authorization username. Did i made a configuration mistake or is this intended?

Thank you for the awesome work!

[poolpOrg]

Hi @terminar,

thank you for the great work with OpenSMTPD and especially the new filter API.
I had to switch our really old mailservers from qmail+vpopmail to something new and had some issues with postfix. Then i looked at OpenSMTPD and you blog entries, tried it, mixed it up with dovecot+rspamd and i loved it!

yay :-)

Open questions:

• Do i have to put everything under your copyright?

Only files which borrow logic I wrote should bear my name and retain the ISC license, files not borrowing my code don't have to be under my copyright nor under the ISC license.

You're under no obligation to publish your source, but if you do the ISC license should always appear on files borrowing my code. If you intend to publish your source with a different license, the files that are covered by my license should display both mine and yours. Your license will then only cover parts that are not already licensed ISC (you might want to document which parts these are in your project).

Hope this clears things ?

• Do you want to be mentioned as first or second author in the LICENSE + headers? (you are currently only mentioned in the README.md)

Quite frankly I don't really care

• as a maintainer of OpenSMTPD, can you take some time for a small code review if i made some horrible bugs or misunderstanding of how the OpenSMTPD filters should be used - and give me some feedback?

Did a fast read because I'm not too comfortable with Go yet, no horrible bugs were seen.

You understood the API good however one thing is kind of incorrect, you're using the data filtering event where you should be using the tx-data reporting event as you're not really taking any decision (you always proceed) but just gathering information.

• one specific OpenSMTPD question which also concern filters: I'm using email-adresses as smtp auth login (meaning user@domain.tld) but in the debug logs and in the "link-auth" report event, i am only getting the user part, not the "full" authorization username. Did i made a configuration mistake or is this intended?

It should work, that's what I use myself:

report|0.5|1577495177.655938|smtp-in|link-auth|a5e3831343a80fac|gilles@poolp.org|pass

Thank you for the awesome work!

Thanks !

[terminar]

Only files which borrow logic I wrote should bear my name and retain the ISC license, files not borrowing my code don't have to be under my copyright nor under the ISC license.

I fixed it in the filter-archive.go file.

Hope this clears things ?

Yes, thanks!

Quite frankly I don't really care
Nevertheless i fixed it ;)

Did a fast read because I'm not too comfortable with Go yet, no horrible bugs were seen.

Thanks, i think that's enough for now. If there are some bugs i'll see them with crashes.

Just to mention it, i had an idea:
I am thinking about moving the code into a more generic "opensmtpd-filter" golang package (which capsulates the whole session management, "version < 0.5" check and some logic to allow registerEvent(), writeResult(), ...) also for other filter-developers. That has also to do with filter-rspamd.
One thing that i don't 'like' about filter-rspamd is the in-memory caching of the messages in the s.tx structs. If i understand correct that means if e.g. multiple 20mb messages are received in parallel from different servers, filter-rspamd will buffer them in the s.tx struct, send them to rspamd and then, answer (and free the memory). That MAY fill the main memory of the server really fast and in an uncontrollable manner - but maybe i am just thinking about a theoretical situation. What i really wanted to say: Such buffering to-file, to memory can also be a part of a more generic opensmtpd-filter golang package (just selecting which buffer type as filter developer) and concentrating on the plugin itself. The existing more complex golang filters may benefit from that idea:

• filter-rspamd
• filter-checksenderdomain
• filter-senderscore
• filter-greylist

filter-checksenderdomain seems to be too small to have a benefit and filter-eventlog is C code (but may be ported really easy to golang with less code because also most of the needed code still exists in filter-archive and filter-rspamd).

IF someone is interested into something like that, I will support such package (with e.g. writing code and using it in filter-archive). Otherwise i'm waiting until i'm in the need of it for myself (maybe because writing another filter/report addon and don't want to duplicate code and logic).

You understood the API good however one thing is kind of incorrect, you're using the data filtering event where you should be using the tx-data reporting event as you're not really taking any decision (you always proceed) but just gathering information.

Fixed, thanks! I haven't realized the event in the documentation.

[...]-----------------------------------------------------------------[...]
I moved the other questions to single issues

• Question regarding config|subsystem|smtp-out + smtp-in (missing in 0.4?) OpenSMTPD/OpenSMTPD#1021
• Feedback to new filter API + questions regarding tx-data/tx-commit/tx-rollback/tx-reset OpenSMTPD/OpenSMTPD#1022

and removed the fixed link-auth question.

[jdelic]

@terminar

Just to mention it, i had an idea:
I am thinking about moving the code into a more generic "opensmtpd-filter" golang package (which capsulates the whole session management, "version < 0.5" check and some logic to allow registerEvent(), writeResult(), ...) also for other filter-developers. That has also to do with filter-rspamd.

Could you take a look at opensmtpd-filters-go, I'm interested in feedback and it might serve some of your use-cases already. (Still does in-memory caching though)

[terminar]

Could you take a look at opensmtpd-filters-go, I'm interested in feedback and it might serve some of your use-cases already. (Still does in-memory caching though)

@jdelic Looks interesting and it seems that i'm not the only one who is thinking about this!
But give me some time to look at the code and understand it in depth. I'll comment it in your repository as a new issue.