enable mw api emailuser backend

[jsnshrmn]

provides features required for mass emailing for survey mediawiki API.

Rationale

In T412427, we found that we were unable to send high volume mail via WMF cloud VPS SMTP infrastructure. So, we're going to try using mediawiki instead.

Phabricator Ticket

https://phabricator.wikimedia.org/T409420

How Has This Been Tested?

• Unit tests have been added for the management command and user selection logic
• No unit tests have been added for the backend api client
• I have done quite a bit of manual testing

You can test run the command logic and djmail message tracking by passing through the default djmail backend (which should be console email for local dev).

For example:
python manage.py survey_active_users 000001 en fr de ja ru es --backend djmail.backends.default.EmailBackend --batch_size 10

You may omit the backend to send it through the api, but in mediawiki you'll need to add and confirm email addresses for users with wp_usernames that match users in your TWLight. If you setup something like mailhog in mediawiki to handle mail delivery, you'll be able to see the survey messages accumulate there. It's kind of a pain.

Screenshots of your changes (if appropriate):

N/A

Types of changes

What types of changes does your code introduce? Add an x in all the boxes that apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Minor change (fix a typo, add a translation tag, add section to README, etc.)

[katydidnot]

I was unable to confirm the email address in my local mediawiki environment to test the API, any tips?

[jsnshrmn]

I was unable to confirm the email address in my local mediawiki environment to test the API, any tips?

I recommend setting up mail handling for mediawiki
https://www.mediawiki.org/wiki/MediaWiki-Docker/Configuration_recipes/Mail_handling

[suecarmol]

Made a first pass of the code and everything looks good. I left some questions that might need to be addressed in this PR.

The command python manage.py survey_active_users 000001 en fr de ja ru es --backend djmail.backends.default.EmailBackend --batch_size 10 seemed to hang in my local environment, but python manage.py survey_active_users 000001 en fr de ja ru es --backend djmail.backends.default.EmailBackend --batch_size 10 --staff_test seems to be working fine.

[jsnshrmn]

Made a first pass of the code and everything looks good. I left some questions that might need to be addressed in this PR.

The command python manage.py survey_active_users 000001 en fr de ja ru es --backend djmail.backends.default.EmailBackend --batch_size 10 seemed to hang in my local environment, but python manage.py survey_active_users 000001 en fr de ja ru es --backend djmail.backends.default.EmailBackend --batch_size 10 --staff_test seems to be working fine.

I changed the strategy for checking for existing emails; before I was searching based on the user profile language, but realized that we would miss messages if the user changed their language preference. What I didn't do was reload a prod-sized user set and retest. It didn't hang on my machine, but it did take a really, really long time. The staff test works against a much smaller subset of users. I'll just switch back to the original strategy, which runs much faster, and should only result in a few possible edge cases of duplicate emails.

[jsnshrmn]

Made a first pass of the code and everything looks good. I left some questions that might need to be addressed in this PR.
The command python manage.py survey_active_users 000001 en fr de ja ru es --backend djmail.backends.default.EmailBackend --batch_size 10 seemed to hang in my local environment, but python manage.py survey_active_users 000001 en fr de ja ru es --backend djmail.backends.default.EmailBackend --batch_size 10 --staff_test seems to be working fine.

I changed the strategy for checking for existing emails; before I was searching based on the user profile language, but realized that we would miss messages if the user changed their language preference. What I didn't do was reload a prod-sized user set and retest. It didn't hang on my machine, but it did take a really, really long time. The staff test works against a much smaller subset of users. I'll just switch back to the original strategy, which runs much faster, and should only result in a few possible edge cases of duplicate emails.

I benchmarked several approaches with a large number of users and we now have the best of both worlds; the user query now happens quickly and we still search for all current translations of the subject line.

[jsnshrmn]

@suecarmol @DHardy-WMF @katydidnot I added in support for the qualification exemptions: I did some checks and found that they had a small impact on the recipient count. Should make testing a little easier!

[jsnshrmn]

@DHardy-WMF if you have a local db setup that supports maxlag, I'd appreciate at least some minimal manual testing on this. This function is completely untested.