shift opendkim, nginx, dovecot, postfix, mtail deployers to their respective directories

[hpk42]

does not change but re-arrange codes to make "deployers.py" less of a monster, and keep all main service related deployment in their respective directory.

[cliffmccarthy]

The rearranged code looks right to me.

[missytake]

A test doesn't complete (mail_domain=staging2.testrun.org):

CHATMAIL_DOMAIN2=staging-ipv4.testrun.org pytest cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py::TestEndToEndDeltaChat::test_securejoin -s --timeout=60
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.12.3, pytest-8.3.4, pluggy-1.5.0 -- /home/user/code/pyinfra-sysadmin/servers/nine.testrun.org/chatmail/venv/bin/python3
cachedir: .pytest_cache
Deltachat core=v2.23.0 sqlite=3.46.1 journal_mode=wal
rootdir: /home/user/code/pyinfra-sysadmin/servers/nine.testrun.org/chatmail/cmdeploy/src/cmdeploy/tests
configfile: pytest.ini
plugins: cmdeploy-0.2, xdist-3.6.1, typeguard-4.4.1, timeout-2.4.0, deltachat-2.23.0, deltachat-rpc-client-1.152.2
timeout: 60.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                                                                                                            

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py::TestEndToEndDeltaChat::test_securejoin WARN: could not use preconfigured keys
[acsetup] 0.112 started configure on <Account path=/tmp/pytest-of-user/pytest-256/test_securejoin0/ac1/dc.db>
WARN: could not use preconfigured keys
[acsetup] 0.213 started configure on <Account path=/tmp/pytest-of-user/pytest-256/test_securejoin0/ac2/dc.db>
bringing accounts online
wait_all_configured finds accounts= {<Account path=/tmp/pytest-of-user/pytest-256/test_securejoin0/ac1/dc.db>: 'CONFIGURING', <Account path=/tmp/pytest-of-user/pytest-256/test_securejoin0/ac2/dc.db>: 'CONFIGURING'}
ac1 waiting for inbox IDLE to become ready
ac2 waiting for inbox IDLE to become ready
finished, account2state {<Account path=/tmp/pytest-of-user/pytest-256/test_securejoin0/ac1/dc.db>: 'IDLEREADY', <Account path=/tmp/pytest-of-user/pytest-256/test_securejoin0/ac2/dc.db>: 'IDLEREADY'}
all accounts online

========== ac1: create QR code and let ac2 scan it, starting the securejoin ==========

========== ac2: start QR-code based setup contact protocol ==========
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of events (125853370443456) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/user/code/pyinfra-sysadmin/servers/nine.testrun.org/chatmail/venv/lib/python3.12/site-packages/deltachat/events.py", line 259, in run
    event = lib.dc_get_next_event(event_emitter)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of events (125853390390976) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/user/code/pyinfra-sysadmin/servers/nine.testrun.org/chatmail/venv/lib/python3.12/site-packages/deltachat/events.py", line 259, in run
    event = lib.dc_get_next_event(event_emitter)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
FAILED

================================================================================================================= FAILURES ==================================================================================================================
___________________________________________________________________________________________________ TestEndToEndDeltaChat.test_securejoin ___________________________________________________________________________________________________

self = <tests.online.test_2_deltachat.TestEndToEndDeltaChat object at 0x727686ae4800>, cmfactory = <deltachat.testplugin.ACFactory object at 0x727686ae4d40>, lp = <deltachat.testplugin.lp.<locals>.Printer object at 0x727686ae6c90>
maildomain2 = 'staging-ipv4.testrun.org'

    def test_securejoin(self, cmfactory, lp, maildomain2):
        ac1 = cmfactory.new_online_configuring_account(cache=False)
        cmfactory.switch_maildomain(maildomain2)
        ac2 = cmfactory.new_online_configuring_account(cache=False)
        cmfactory.bring_accounts_online()
    
        lp.sec("ac1: create QR code and let ac2 scan it, starting the securejoin")
        qr = ac1.get_setup_contact_qr()
    
        lp.sec("ac2: start QR-code based setup contact protocol")
        ch = ac2.qr_setup_contact(qr)
        assert ch.id >= 10
>       ac1._evtracker.wait_securejoin_inviter_progress(1000)

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py:122: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.12/site-packages/deltachat/events.py:173: in wait_securejoin_inviter_progress
    event = self.get_matching("DC_EVENT_SECUREJOIN_INVITER_PROGRESS")
venv/lib/python3.12/site-packages/deltachat/events.py:118: in get_matching
    for ev in self.iter_events(timeout=timeout, check_error=check_error):
venv/lib/python3.12/site-packages/deltachat/events.py:114: in iter_events
    yield self.get(timeout=timeout, check_error=check_error)
venv/lib/python3.12/site-packages/deltachat/events.py:107: in get
    ev = self._event_queue.get(timeout=timeout)
/usr/lib/python3.12/queue.py:171: in get
    self.not_empty.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Condition(<unlocked _thread.lock object at 0x727686af8cc0>, 0)>, timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout (>60.0s) from pytest-timeout.

/usr/lib/python3.12/threading.py:355: Failed
======================================================================================================= 1 failed in 60.62s (0:01:00) ========================================================================================================

This makes CI timeout after 30 minutes.

[missytake]

Looks like the staging machines have failing DKIM. This log output happens in nine's filtermail-incoming.service when I run CHATMAIL_DOMAIN2=nine.testrun.org pytest cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py::TestEndToEndDeltaChat::test_securejoin -s --timeout=60:

Nov 15 20:13:41 nine filtermail[1775383]: handle_DATA before-queue
Nov 15 20:13:41 nine filtermail[1775383]: Processing DATA message from [email protected]
Nov 15 20:13:41 nine filtermail[1775383]: Incoming: Filtering encrypted mail.
Nov 15 20:13:41 nine filtermail[1775383]: re-injecting the mail that passed checks
Nov 15 20:13:41 nine filtermail[1775383]: ('127.0.0.1', 60916) SMTP session exception
Nov 15 20:13:41 nine filtermail[1775383]: Traceback (most recent call last):
Nov 15 20:13:41 nine filtermail[1775383]:   File "/usr/local/lib/chatmaild/venv/lib/python3.11/site-packages/aiosmtpd/smtp.py", line 756, in _handle_client
Nov 15 20:13:41 nine filtermail[1775383]:     await method(arg)
Nov 15 20:13:41 nine filtermail[1775383]:   File "/usr/local/lib/chatmaild/venv/lib/python3.11/site-packages/aiosmtpd/smtp.py", line 1519, in smtp_DATA
Nov 15 20:13:41 nine filtermail[1775383]:     status = await self._call_handler_hook('DATA')
Nov 15 20:13:41 nine filtermail[1775383]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 15 20:13:41 nine filtermail[1775383]:   File "/usr/local/lib/chatmaild/venv/lib/python3.11/site-packages/aiosmtpd/smtp.py", line 473, in _call_handler_hook
Nov 15 20:13:41 nine filtermail[1775383]:     status = await hook(self, self.session, self.envelope, *args)
Nov 15 20:13:41 nine filtermail[1775383]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 15 20:13:41 nine filtermail[1775383]:   File "/usr/local/lib/chatmaild/venv/lib/python3.11/site-packages/chatmaild/filtermail.py", line 301, in handle_DATA
Nov 15 20:13:41 nine filtermail[1775383]:     return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
Nov 15 20:13:41 nine filtermail[1775383]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 15 20:13:41 nine filtermail[1775383]:   File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Nov 15 20:13:41 nine filtermail[1775383]:     result = self.fn(*self.args, **self.kwargs)
Nov 15 20:13:41 nine filtermail[1775383]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 15 20:13:41 nine filtermail[1775383]:   File "/usr/local/lib/chatmaild/venv/lib/python3.11/site-packages/chatmaild/filtermail.py", line 317, in sync_handle_DATA
Nov 15 20:13:41 nine filtermail[1775383]:     client.sendmail(
Nov 15 20:13:41 nine filtermail[1775383]:   File "/usr/lib/python3.11/smtplib.py", line 908, in sendmail
Nov 15 20:13:41 nine filtermail[1775383]:     raise SMTPDataError(code, resp)
Nov 15 20:13:41 nine filtermail[1775383]: smtplib.SMTPDataError: (554, b'5.7.1 No valid DKIM signature found')

[missytake]

Fixed it :)

[cliffmccarthy]

Thanks, @missytake! When I reviewed this previously, I overlooked the fact that the default argument to _configure_opendkim() was being overridden with a different (correct) value by the caller.