CI Remote Cache Error.

[ysiraichi]

🐛 Bug

The remote cache seems not to be working.
There are a couple of my PRs whose CI can't build due to a timeout.

For reference, when remote caching was working, it took PyTorch/XLA less than 400s to build (link).
However, without the remote cache, it's taking 1700s. (link).

In the beginning of PyTorch/XLA build, we can see the following error message:

    ERROR: Failed to query remote execution capabilities: UNAVAILABLE: Credentials failed to obtain metadata

Although we could increase the timeout to around 80 minutes, I don't think that's a good solution.

cc @bhavya01 @zhanyong-wan @qihqi

[ysiraichi]

One thing we could do is use both the remote and disk cache (cached by GitHub cache action).
Then, I believe we would avoid problems like this.

[zhanyong-wan]

I haven't used GitHub cache action before. We can try it.

I used to be able to finish the build within ~30m without the remote cache. Any idea what caused the regression?

In any case, we should make the build work without remote cache, as contributors who don't have write permission to this repo cannot use the cache.

[ysiraichi]

I used to be able to finish the build within ~30m without the remote cache. Any idea what caused the regression?

Build of PyTorch and PyTorch/XLA?
PyTorch took around 9 minutes.
PyTorch/XLA took around 28 minutes.

Now that I looked at it, we were less than 5 minutes short.
So, probably, 50 minutes timeout would be enough.

In any case, we should make the build work without remote cache, as contributors who don't have write permission to this repo cannot use the cache.

Got it.
I will try to set it up.