CI: reduce pressure on build infrastructure #1106
Conversation
lumag
requested review from
mwasilew,
ndechesne,
ricardosalveti,
sbanerjee-quic and
vkraleti
as code owners
ndechesne
left a comment
ndechesne
left a comment
There was a problem hiding this comment.
I like this change.. let's see what the other have to say. It does break users though, we need to be cautious, and we need to update the README as well.
|
Removing the target from the yml is not good in my opinion because it will force the user to choose which image they want and this may not be easy for everyone. |
|
We could consider splitting the machines more evenly between the compile_warm_up & compile steps to reduce parallel load. Currently, the split is 2:7. Eventually, we can group 4–5 machines into a batch and run builds in multiple batches. |
I'm not sure, why choosing an image is not easy. I think that specifying the image in the distro is counter-intuitive as it steers the user towards a particular image. |
That would increase the amount of cache misses in the warm_up stage, so it's not logical |
Because this forces the user to know the images from a predefined list and this makes it difficult for beginners to use. I think we should be the ones to make this choice and thus make the use of the layer simpler for new users. For someone experienced, changing the image is trivial, but I don't believe it is like that for everyone. |
In that case warm_up stage can remain as. However, I believe the compile step should be split into multiple batches. Since the number of machines is expected to grow over time, reducing the job count from 35 to 28 (as proposed in this PR) may only be a temporary relief. If the additional 7 jobs are already becoming a bottleneck, we might face the same issue again soon. |
If we have a problem with the CI runners and the number of jobs we create we should fix our infrastructure instead of limiting ourselves. I have yet to see a problem related to the number of jobs we create. |
|
@doanac is working on identifying the correct fix, anything we do to remove the amount of jobs will only be temporary. |
Beginners still have to pass a correct set fo YAML files, which is not that trivial. I think we can document both, the set of YAMLs and the default image. |
|
This pull request has been marked as stale due to 30 days of inactivity. To prevent automatic closure in 5 days, remove the stale label or add a comment. You can reopen a closed pull request at any time. |
Test run workflowTest jobs for commit 52d6f1d
|
Test run workflowTest jobs for commit 52d6f1d
|
Test run workflowTest jobs for commit 808f949
|
CI is having stability issues, most likely caused by the huge number of jobs trying to build and upload artifacts in parallel. Reduce the number of jobs being by merging two qcom-distro jobs into a single one and using special jobs for qcom-armv7a builds (since we don't support proprietary builds on qcom-armv7a). Signed-off-by: Dmitry Baryshkov <[email protected]>
Test run workflowTest jobs for commit 5e3a4ce
|
Test run workflowTest jobs for commit 5e3a4ce
|
Test run workflowTest jobs for commit 5e3a4ce
|
6 participants
CI is having stability issues, most likely caused by the huge number of jobs trying to build and upload artifacts in parallel. Reduce the number of jobs being by merging two qcom-distro jobs into a single one and using special KAS file for qcom-armv7a / qcom-distro builds.