flaky test: ITMetricsTest.testListMetrics

[ldetmer]

[ERROR] com.google.cloud.logging.it.ITMetricsTest.testListMetrics -- Time elapsed: 8.848 s <<< ERROR!
com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Control requests' and limit 'Control requests per minute per user' of service 'logging.googleapis.com' for consumer 'project_number:1016721519174'.
at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:216)
at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:210)
at com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction.apply(ApiFutures.java:396)
at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:241)
at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:229)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:139)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1299)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1060)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:810)
at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:202)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1299)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1060)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:810)
at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:92)
at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:74)
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1130)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1299)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1060)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:810)
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:568)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:569)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

[cindy-peng]

Hi @ldetmer, I believe this is a flaky test caused by quota limit. Oftentimes it can be resolved by re-run the tests. Could you retry running the tests?

[zhumin8]

failing this test repeatedly in #1746

update: rebuilding each failing job works. May be hitting quota limit because of tests running at roughly the same time

[cindy-peng]

Looks like a transient quota issue. All tests have been successful since February: https://btx.cloud.google.com/invocations;page_size=100;offset=200;p=250644331956?q=target:github%2Fjava-logging%2Fgoogle-cloud-logging%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.logging.it.ITMetricsTest

Closing this for now as tests have been passing. Please feel free to re-open this if we run into this issue again.