Dependency version conflict when using synapse-s3-storage-provider

[markeeisner]

Describe the bug
When running the playbook with no changes to any vars it now fails with a boto3 / synapse-s3-storage-provider version mismatch.

TASK [custom/matrix-synapse : Ensure customized Docker image for Synapse is built] ************************************* fatal: [matrix.example.org]: FAILED! => {"changed": false, "msg": "Error building localhost/matrixdotorg/synapse - code: 1, message: The command '/bin/sh -c pip install 'boto3<1.36.0' 'botocore<1.36.0' synapse-s3-storage-provider==1.6.0' returned a non-zero code: 1, logs: ['Step 1/2 : FROM ghcr.io/element-hq/synapse:v1.139.2', '', ' ---> cd7f0b5ddfeb', \"Step 2/2 : RUN pip install 'boto3<1.36.0' 'botocore<1.36.0' synapse-s3-storage-provider==1.6.0\", '', ' ---> Running in 5744da1d2d51', 'Collecting boto3<1.36.0', ' Downloading boto3-1.35.99-py3-none-any.whl.metadata (6.7 kB)', 'Collecting botocore<1.36.0', ' Downloading botocore-1.35.99-py3-none-any.whl.metadata (5.7 kB)', 'Collecting synapse-s3-storage-provider==1.6.0', ' Downloading synapse_s3_storage_provider-1.6.0-py3-none-any.whl.metadata (4.9 kB)', 'INFO: pip is looking at multiple versions of synapse-s3-storage-provider to determine which version is compatible with other requirements. This could take a while.', '\\x1b[91mERROR: Cannot install boto3<1.36.0 and synapse-s3-storage-provider==1.6.0 because these package versions have conflicting dependencies.', '\\x1b[0m', '', 'The conflict is caused by:', ' The user requested boto3<1.36.0', ' synapse-s3-storage-provider 1.6.0 depends on boto3<2.0 and >=1.36.0', '', 'To fix this you could try to:', \"1. loosen the range of package versions you've specified\", '2. remove package versions to allow pip to attempt to solve the dependency conflict', '', '\\x1b[91m', '[notice] A new release of pip is available: 25.0.1 -> 25.2', '[notice] To update, run: pip install --upgrade pip', '\\x1b[0m', '\\x1b[91mERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts', '\\x1b[0m', ' ---> Removed intermediate container 5744da1d2d51']"}

To Reproduce
My vars.yml file looks like this:

matrix_domain:

matrix_homeserver_implementation: synapse

matrix_homeserver_generic_secret_key:

postgres_connection_password:

matrix_playbook_docker_installation_enabled: true

matrix_synapse_ext_password_provider_ldap_enabled: true
matrix_synapse_ext_password_provider_ldap_uri:
  - "ldaps://"
matrix_synapse_ext_password_provider_ldap_start_tls: true
matrix_synapse_ext_password_provider_ldap_base:
matrix_synapse_ext_password_provider_ldap_attributes_uid: "uid"
matrix_synapse_ext_password_provider_ldap_attributes_mail: "mail"
matrix_synapse_ext_password_provider_ldap_attributes_name: "cn"
matrix_synapse_ext_password_provider_ldap_bind_dn:
matrix_synapse_ext_password_provider_ldap_bind_password:
matrix_synapse_ext_password_provider_ldap_filter:

matrix_appservice_double_puppet_enabled: true

matrix_mautrix_whatsapp_enabled: true

matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_api_id:
matrix_mautrix_telegram_api_hash:
matrix_mautrix_telegram_configuration_extension_yaml: |
  telegram:
    device_info:
      device_model: GooglePixel 6
      system_version: SDK 32
      app_version: 8.7.4 (26367)
      lang_code: en
      system_lang_code: en

backup_borg_enabled: true
backup_borg_location_repositories:
  - ssh://
backup_borg_storage_encryption_passphrase:
backup_borg_ssh_key_private: |
  -----BEGIN OPENSSH PRIVATE KEY-----
  -----END OPENSSH PRIVATE KEY-----
backup_borg_configuration_extension_yaml: |
  storage:
    umask: 22

matrix_synapse_admin_enabled: true

exim_relay_sender_address:
exim_relay_relay_use: true
exim_relay_relay_host_name:
exim_relay_relay_host_port: 587
exim_relay_relay_auth: true
exim_relay_relay_auth_username: 
exim_relay_relay_auth_password: 

matrix_playbook_reverse_proxy_type: playbook-managed-traefik

matrix_synapse_ext_synapse_s3_storage_provider_enabled: true
matrix_synapse_ext_synapse_s3_storage_provider_config_bucket: matrix
matrix_synapse_ext_synapse_s3_storage_provider_config_region_name:
matrix_synapse_ext_synapse_s3_storage_provider_config_endpoint_url: 
matrix_synapse_ext_synapse_s3_storage_provider_config_access_key_id: 
matrix_synapse_ext_synapse_s3_storage_provider_config_secret_access_key: 
matrix_synapse_ext_synapse_s3_storage_provider_config_storage_class: STANDARD

matrix_sliding_sync_enabled: true

jitsi_enabled: true
jitsi_hostname: 
jitsi_enable_auth: true
jitsi_enable_guests: true
jitsi_auth_type: ldap
jitsi_ldap_url: 
jitsi_ldap_base:
jitsi_ldap_binddn:
jitsi_ldap_bindpw:
jitsi_ldap_filter: "uid=%u"
jitsi_ldap_group_dn:
jitsi_ldap_auth_method: "bind"
jitsi_ldap_version: "3"
jitsi_ldap_use_tls: true
jitsi_ldap_tls_ciphers: ""
jitsi_ldap_tls_check_peer: false
jitsi_prosody_max_participants: 6
jitsi_timezone: Europe/London
jitsi_web_config_resolution_width_ideal_and_max: 1920
jitsi_web_config_resolution_height_ideal_and_max: 1080
jitsi_web_custom_config_extension: |
  config.requireDisplayName = true;
  config.startAudioOnly = true;

ntfy_enabled: true
ntfy_hostname:
ntfy_web_root: app
ntfy_credentials:
  - username:
    password:
    admin: true
ntfy_attachment_enabled: true
ntfy_attachment_expiry_duration: "16h"

matrix_rtc_enabled: true
matrix_static_files_container_labels_base_domain_enabled: true

Run the playbook with just setup-all -K without making changes to vars.yml since the last time the playbook ran successfully.

Expected behavior
Running playbook should function without a dependency version mismatch when no changes to vars.yml have been made since the last successful run.

Matrix Server:

• OS: Almalinux 10
• Architecture: x86-64

Additional context
It seems commit 2d05db5 bumped the synapse-s3-storage-provider version to 1.6.0. This version requires boto3 version >=1.36.0 which conflicts with a workaround for upstream changes which broke non-AWS s3 connections. See #3964 for the workaround pinning boto to <1.36.0. It appears that the upstream issues are not fixed so disabling the workaround from #3964 is likely to result in a broken S3 storage provider.

As it stands enabling synapse-s3-storage-provider with an otherwise default vars.yml will result in failure due to this dependency conflict.

Adding matrix_synapse_ext_synapse_s3_storage_provider_version: 1.5.0 to vars.yml allows the playbook to run successfully.

[spantaleev]

We don't nee the boto workaround for s3-storage-provider=1.6.0, but... s3-storage-provider=1.6.0 requires Synapse v1.140.0, which hasn't been released yet.

Ref: matrix-org/synapse-s3-storage-provider#134 (comment)

So... I've reverted us to s3-storage-provider=1.5.0.