Import uploads related process takes up many GB of RAM

[svandragt]

Description

After having ALS 21 running for 2 hours, my computer runs out of memory (32GB RAM + 32GB swap). Because I'm on linux, and because of #819 the memory limits for docker containers when not specified are set to the system RAM limit. However, this masks the underlying issue, which is that mc mirror --watch --overwrite -a local/s3-myproject /content uses up 19GB of RAM.

I traced this call back to

altis-local-server/inc/composer/class-docker-compose-generator.php

Line 561 in 6e91746

'entrypoint' => "/bin/sh -c \"mc mirror --watch --overwrite -a local/{$this->bucket_name} /content\"",

As a workaround I will attempt to avoid importing uploads or reducing the uploads folder. However the content/ folder is 683MB with 11427 files so I suspect something else is going on.

Steps to reproduce:

1. start my project
2. go into a series of calls and other activies
3. Find computer starts to stutter due to memory problems
4. restart the project containers and see the problem disappears

mc not using more than a few hundred MB of ram.

$ uname -a
Linux sander-x670 6.8.0-59-generic #61~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 17:03:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ neofetch           
         eeeeeeeeeeeeeeeee            sander@sander-x670 
      eeeeeeeeeeeeeeeeeeeeeee         ------------------ 
    eeeee  eeeeeeeeeeee   eeeee       OS: elementary OS 7.1 Horus x86_64 (based on Ubuntu 22.04.3 LTS)
  eeee   eeeee       eee     eeee     Host: MS-7D67 1.0 
 eeee   eeee          eee     eeee    Kernel: 6.8.0-59-generic 
eee    eee            eee       eee   Uptime: 2 hours, 27 mins 
eee   eee            eee        eee   Packages: 4056 (dpkg), 99 (flatpak), 13 (snap) 
ee    eee           eeee       eeee   Shell: zsh 5.8.1 
ee    eee         eeeee      eeeeee   Resolution: 1920x1080, 1920x1080 
ee    eee       eeeee      eeeee ee   DE: Pantheon 
eee   eeee   eeeeee      eeeee  eee   WM: Mutter(Gala) 
eee    eeeeeeeeee     eeeeee    eee   Theme: io.elementary.stylesheet.cocoa [GTK3] 
 eeeeeeeeeeeeeeeeeeeeeeee    eeeee    Icons: elementary [GTK3] 
  eeeeeeee eeeeeeeeeeee      eeee     Terminal: io.elementary.t 
    eeeee                 eeeee       CPU: AMD Ryzen 9 7900X3D (24) @ 5.660GHz 
      eeeeeee         eeeeeee         GPU: AMD ATI Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT 
         eeeeeeeeeeeeeeeee            Memory: 7199MiB / 31765MiB 

                                                                                                                      

Acceptance criteria:

• Investigate the bug described above
• Check if https://github.com/humanmade/product-dev/issues/1402 fixes the issue
• Issue describe in description no longer occurs

For Altis Team Use

Ready for Work Checklist

Is this ticket ready to be worked on? See
the Play Book Definition of Ready

• Is the title clear?
• Is the description clear and detailed enough?
• Are acceptance criteria listed?
• Have any dependencies been identified? (Optional)
• Have any documentation/playbook changes been identified? (Optional)
• Is an estimate or time box assigned?
• Is a priority label assigned?
• Is this ticket added to a milestone?
• Is this ticket added to an epic? (Optional)

Completion Checklist

Is this ticket done? See
the Play Book Definition of Done

• Has the acceptance criteria been met?
• Is the documentation updated (including README)?
• Do any code/documentation changes meet project standards?
• Are automatic tests in place to verify the fix or new functionality?
  • Or are manual tests documented (at least on this ticket)?
• Are any Playbook/Handbook pages updated?
• Has a new module release (patch/minor) been created/scheduled?
• Have the appropriate backport labels been added to the PR?
• Is there a roll-out (and roll-back) plan if required?

[svandragt]

[wisyhambolu]

Please add your planning poker estimate with Zenhub @vladislavhmn

[jerico]

We are moving away from MinIO including the mc command used here. We will be using aws-cli to sync and import-uploads. #886