[RESOLVED] Ongoing incident

[helinanever]

We’re investigating an issue with running builds on macOS machines. Your builds may stay queued until the issue is resolved.

[helinanever]

We've identified the issue and are working on recovering the macOS machines. The next update will be in 30 minutes.

[helinanever]

The issue is related to our NAS storage. We are currently upgrading the firmware to restore the connection and will need a little more time to fully resolve the issue.

[helinanever]

Unfortunately, the firmware upgrade did not resolve the issue and we were unable to restore the connection to NAS. We’re currently attempting an alternative solution to restore the macOS build machines. We'll update you as soon as there is new information. We apologize for the inconvenience caused.

[prasunsultania]

it is working for us now :)

[arturoszulc]

For us not yet

[HoussemBenAbderrahmen]

not yet also

[mikhail-tokarev]

it takes some time to start all queued builds, but we can see that macOS builds are restored now.

[HoussemBenAbderrahmen]

it works thanks for the quick effort

[mikhail-tokarev]

Incident report

On Friday, August 11, 2023, at 12:00 UTC we lost connection to our NAS server after an automatically applied security patch overwrote critical configuration files. We immediately started working on restoring the proper configuration of the NAS server.

Unfortunately, the first attempts to restore the NAS server configuration were unsuccessful. At that moment, we started working on a backup plan. A patch was deployed at 14:20 UTC after which our macOS build machines became available again for user builds but with one limitation – using or updating the stored build cache was not possible.

We were able to properly fix the NAS server configuration and restore the cache functionality on Monday, August 14 at 09:00 UTC. We also updated our system configuration to make sure such scenarios never happen again. Our macOS build machines now work as expected with full functionality available to our users. However, we will continue monitoring to ensure that everything remains in good working order.

Thank you for your patience and understanding!

[WernerRaathAtPolymorph]

I seem to still be facing this issue.
Build ID: 64dd096d52bd43d172ead175

[mikhail-tokarev]

hey
all services are operating as expected. Please make sure you refreshed the page, cancel and start a new build. If issue still persists for you file a new issue please, probably something specific to your build configuration.