Codemagic build not publishing in s3 time out error

[mohan-balakrishnan]

We are facing issue in Codemagic from yesterday. While publishing to S3 it took more than 25 mins. As the default build timeout exceeds it shows error. Please do the needful.

[icarusdust]

Hi @mohan-balakrishnan Have you tried increasing the max build duration ? How long does it usually take to complete that step ?

[mohan-balakrishnan]

Usually build will complete in 10 mints. Now it is not working. Even I have set build minutes to 30 minutes

[icarusdust]

@mohan-balakrishnan we are looking into it and will keep you updated asap.

[mikhail-tokarev]

Hi @mohan-balakrishnan

sorry it took longer than we expected :( can you run another build and check upload time now?

[mohan-balakrishnan]

Yes, i triggered another build it is taking more time. still same issue. my build is not publishing to s3.

[remarkov]

Hi @mohan-balakrishnan,

I am trying to debug the issue at the moment and unfortunately I am not able to reproduce the issue with my test environment, tried on a couple of different M1 hosts already and see the upload speed to S3 is 1-2.5 MiB/s. Would you please post a link to any of your failed / timed out builds? Also, would it be ok if I try to rerun one of such builds on your behalf? That would help a lot to investigate the issue and find a fix.

[remarkov]

@mohan-balakrishnan never mind, I finally managed to reproduce the issue so I will continue my investigation and will post an update here as soon as I find something.

[mohan-balakrishnan]

@remarkov - Thanks for your update and support.

[remarkov]

@mohan-balakrishnan so far I discovered a very frustrating thing. the issue does not affect any certain buckets nor users nor file types. the file size is what matters only! and that is why I was not able to reproduce it initially, I was trying with a 600 MB test file and got a decent transfer speeds. but once I tried with a 8 MB file I've got the same very poor network performance. more to that, I tried the same test on my local M1 laptop and got the same issue! trying the same on Intel machine the 8 MB files uploads in 2-3 seconds.
so far my conclusion is that arm version of aws cli tools has some bug related to transfer speed per thread. we get the decent upload speeds for bigger files because the default chunk size is exactly 8 MB and if we try to upload at least 80 MB file we will use 10 threads. smaller files use less threads thus the low speed.
currently I'm trying to find a workaround that we may use so I'll post an update here if we implement something.

[mohan-balakrishnan]

Hi @remarkov,

Great!!. finally issue has been identified. Our application build size is 16 MB. when we re-run build it is taking 30 mints sometimes and build is getting deployed. some times build gets failed. Looking for quick workaround

[remarkov]

Unfortunately whatever I tried gave me the same result on M1, even when I tried to install the x86 version and run it using Rosetta.
I just filed an issue in AWS CLI repo, hopefully they will come up with some fix or at least a workaround. For now I am out of ideas what else can we try.

[mohan-balakrishnan]

Hi @remarkov ,

I am using aws-cli/2.7.18 in this small file or large file it works well.

[remarkov]

@mohan-balakrishnan are you on M1 machine? I just tried 2.7.18 on M1 host and got the same slow upload speed. if you are on M1, could you please post the full output of aws --version command? also, what version of macOS are you using?

[mohan-balakrishnan]

@remarkov, I missed to say that I'm using windows machine. Let me try with mac too.

[remarkov]

@mohan-balakrishnan Ah, windows is a different story, even macOS but running on Intel CPU works fine, just M1 hosts are somehow affected.

[mohan-balakrishnan]

@remarkov - Any solution or workaround for this.

[remarkov]

@mohan-balakrishnan further investigation showed that this is a weird limitation of the virtualization platform we use under the hood to run our builder machines. Unfortunately to the moment we were not able to find any workaround whatever we tried. We keep looking for a working solution but I can't give any ETA for now.

[remarkov]

@mohan-balakrishnan sorry for the delay, here is an update. The underlying issue seems to be in the Apple virtualization framework itself so the chances it will be properly fixed any time soon are rather low.
But the good news is that it seems there is a good workaround which solves the problem. It worked in our internal testing environment and we already applied it in production so please go ahead and give it a try.

[remarkov]

@mohan-balakrishnan sorry, I was too quick to post the good news, I noticed a bug in the new configuration and had to rollback the change. Anyway, hopefully this will be fixed pretty soon and most likely you will be able to test the fix. I'll post another update here once everything is live.

[gregiOS]

Any update there?

[mohan-balakrishnan]

@remarkov - any updates on this ?

[gregiOS]

https://github.com/aws/aws-cli/issues/7481 AWS was informed, but they are not super active there

[gregiOS]

Hello, there is AWS repo issue that is being investigated by them https://github.com/aws/aws-cli/issues/7481
Can you guys follow up with that?

[remarkov]

Hi @mohan-balakrishnan ,
First of all, I'm sorry I did not reply to you earlier! Unfortunately there's not much of a progress so far. As you can see, I was able to narrow down the problem to aws-cli and it seems their devs are currently investigating the issue which is luckily confirmed by other users who face the same. Unfortunately, we were not able to find any workaround so we just patiently waiting for the aws-cli team to find and fix the issue on their side.

Hi @gregiOS ,
Thanks for the link, but as you can see, I was the one who opened that issue so we keep an eye on the progress and really hope the devs will be able to fix it soon.

[dtrdic]

This discussion is being closed due to no activity in the last 14 days. You can reopen the discussion at any time if needed.

[dtrdic]

Hey @mohan-balakrishnan @gregiOS just wanted to follow up on this issue..
It was discovered that uploads to AWS from with an Apple Virtualization Framework VM would not be slow if we pinned the S3 endpoint IPs to 3.5.0.0/19 via /etc/hosts (This range is advertised for S3 here https://ip-ranges.amazonaws.com/ip-ranges.json)

Here's a PoC (bucket_name should be changed to the target bucket, i.e. "github-cloud" for GitHub LFS):

bucket_name="YOUR_BUCKET_NAME"
rr_rw_soa="$(host -t SOA s3-r-w.us-east-1.amazonaws.com | grep -o '[^ ]\{1,\}\.awsdns-[^ ]\{1,\}')"
rr_rw_a="$(timeout 2s bash -c "while ! host s3-r-w.us-east-1.amazonaws.com '${rr_rw_soa}' | grep -o '3\.5\.[0-9]\{1,\}\.[0-9]\{1,\}'; do continue; done")"

rr_w_soa="$(host -t SOA s3-w.us-east-1.amazonaws.com | grep -o '[^ ]\{1,\}\.awsdns-[^ ]\{1,\}')"
rr_w_a="$(timeout 2s bash -c "while ! host s3-w.us-east-1.amazonaws.com '${rr_w_soa}' | grep -o '3\.5\.[0-9]\{1,\}\.[0-9]\{1,\}'; do continue; done")"

sudo tee -a /etc/hosts <<EOD
$(head -n 1 <<<"${rr_rw_a}") "${bucket_name}.s3.us-east-1.amazonaws.com"
$(head -n 1 <<<"${rr_w_a}") "${bucket_name}.s3.amazonaws.com"

EOD

In our testing, a 10MB git-lfs push took 6 seconds with this fix, and > 90 seconds without it.

We're not sure what exactly is causing such a significant performance difference between the IPs in 3.5.0.0/19 and any other valid S3 service IPs, but it does seem to suggest the problem isn't entirely related to the TCP configuration on the guest and host.

[dtrdic]

I'm tentatively closing the discussion since we haven't heard back from you. You can open the discussion again by replying to this message or opening a new discussion.