Introduce a Timed-Out state for builds

[shiv-tyagi]

The builds can sometimes get stuck due to several intermittent issues and we keep showing them as Running. We should introduce a new timed-out state and move such builds to that state if they do not finish within a specified time.

This image shows several builds are marked as running but are actually stuck.

[neo-0007]

Hi @shiv-tyagi , I was not able to reproduce this locally but analyzing the current stuck builds i see that it always gets stuck in the Cloning build source... step, Currently in such cases does the build process gets stopped or keeps on running ? . Your idea to introduce a new TIMED-OUT state seems nice but is it only for the UI or will it cancel the build process entirely ?

I don't have much experience with Flask, so I want to make sure I understand the expected behavior before implementing this.

[lakhmanisahil]

Hello @shiv-tyagi ,

I’d like to pick this up. My plan is to add a TIMED_OUT build state and mark builds that remain in RUNNING beyond a fixed duration as timed out, then update the UI accordingly.

Before starting, I wanted to ask if there’s a preferred timeout value and whether you’d prefer this handled via a periodic cleanup task or during build status queries.

Also, just for reference, I’m relatively new to Flask, so I wanted to confirm the preferred approach before making changes. Thanks!

[shiv-tyagi]

Yes, the build should be aborted after the specified time if it is already running. We can start with a timeout of 15 minutes after a build starts running.

About how to handle it, I have not thought about it properly yet. There is a progress updater in the code... which is already a periodic task. So the frontend logic can go somewhere in there. And a similar timout logic in the subprocess call which runs a build in the bulider module as well. I will leave it to you to figure out further and submit a PR.

Since, both of you want to work on the PR, I would not want to disappoint any one of you by assigning it to the other person. So whoever submits a PR first can take this.

There are plenty of more issues across repos in ardupilot. So please feel free to pick up some another issue and ask your doubts on discord.

[lakhmanisahil]

Hey @shiv-tyagi, I tried to reproduce this locally by following the steps mentioned here: https://github.com/ArduPilot/CustomBuild?tab=readme-ov-file#running-locally-without-docker-on-ubuntu, but I was unable to do so in my local environment—the status remains Pending. Am I proceeding in the right direction?

[shiv-tyagi]

@lakhmanisahil I would suggest you to run locally WITH docker.

Setting up things without docker can be difficult as the application has a lot of moving parts.

[lakhmanisahil]

Hello @shiv-tyagi,

I am currently working on this feature and plan to submit a PR soon. Before finalizing it, I had a couple of clarifications:

1. While testing different timeout values (e.g., 1–2 minutes), I noticed that the build is correctly marked as timed out, but the underlying build process still appears to continue/start.
   Should a timeout fully abort/terminate the build process, or is it sufficient for the build to be only marked as timed out from the system’s perspective?

2. Also, when I restart the app or run docker compose up --build after making changes, I still see older builds listed.
   Is this expected behavior, or should previous builds be cleared/invalidated on restart or rebuild?

I have attached screenshots above for reference.
Based on the current behavior, I wanted to confirm whether I am heading in the right direction with this implementation, or if a different approach was intended for this feature.

[shiv-tyagi]

Yes @lakhmanisahil, you are heading in correct direction.

1. The build, if it is already going on, should be aborted after the timeout. There is a progress updater component in the build manager module. You would need to introduce a new state there and based on a timeout you would need to move to that state. Define the timeout value somewhere in the code and pass it to the progress updator. That same value would also need to go in the builder module and that would also terminate a long running build after that time. For now the workflow will be separate and independent at both places, the progress updater and the builder. We can think of unifying it in future.

2. Yes this is expected behaviour. We persist the builds across restarts.

[lakhmanisahil]

Hi @shiv-tyagi,

I’ve opened a PR to address this issue: #204

Would appreciate your review and any suggestions for improvement. Thanks!