TPU: Too many pings

[dhruvrnaik]

Getting this error running a Dall-E model based on minDALLE implementation.
Not sure what this is about, how can I debug this?

Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
E0414 18:11:50.603413912 4155781 chttp2_transport.cc:1098]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
E0414 18:11:50.603461253 4157566 chttp2_transport.cc:1098]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
E0414 18:11:50.603493868 4155381 chttp2_transport.cc:1098]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
E0414 18:11:50.603487762 4157164 chttp2_transport.cc:1098]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
E0414 18:11:50.603608799 4154787 chttp2_transport.cc:1098]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
E0414 18:11:50.603642585 4153786 chttp2_transport.cc:1098]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
2022-04-14 18:11:57.726255: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:157] RPC failed with status = "UNAVAILABLE: Too many pings" and grpc_error_string = "{"created":"@1649959912.230381522","description":"Error received from peer ipv4:127.0.0.1:51011","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Too many pings","grpc_status":14}", maybe retrying the RPC
2022-04-14 18:11:57.726263: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:157] RPC failed with status = "UNAVAILABLE: Too many pings" and grpc_error_string = "{"created":"@1649959912.875546981","description":"Error received from peer ipv4:127.0.0.1:51011","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Too many pings","grpc_status":14}", maybe retrying the RPC
2022-04-14 18:11:57.726295: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:157] RPC failed with status = "UNAVAILABLE: Too many pings" and grpc_error_string = "{"created":"@1649959912.875473886","description":"Error received from peer ipv4:127.0.0.1:51011","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Too many pings","grpc_status":14}", maybe retrying the RPC
2022-04-14 18:11:57.726148: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:157] RPC failed with status = "UNAVAILABLE: Too many pings" and grpc_error_string = "{"created":"@1649959912.875794503","description":"Error received from peer ipv4:127.0.0.1:51011","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Too many pings","grpc_status":14}", maybe retrying the RPC
2022-04-14 18:11:57.726217: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:157] RPC failed with status = "UNAVAILABLE: Too many pings" and grpc_error_string = "{"created":"@1649959912.875416577","description":"Error received from peer ipv4:127.0.0.1:51011","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Too many pings","grpc_status":14}", maybe retrying the RPC
2022-04-14 18:11:57.726314: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:157] RPC failed with status = "UNAVAILABLE: Too many pings" and grpc_error_string = "{"created":"@1649959912.875727708","description":"Error received from peer ipv4:127.0.0.1:51011","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Too many pings","grpc_status":14}", maybe retrying the RPC
2022-04-14 18:11:57.726363: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:157] RPC failed with status = "UNAVAILABLE: Too many pings" and grpc_error_string = "{"created":"@1649959912.875692677","description":"Error received from peer ipv4:127.0.0.1:51011","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Too many pings","grpc_status":14}", maybe retrying the RPC
2022-04-14 18:14:07.371928: E tensorflow/core/distributed_runtime/master_session.cc:564] RunStep still blocked after 60 seconds. Failed with error status: CANCELLED: RunManyGraphs
2022-04-14 18:14:10.624736: E tensorflow/core/distributed_runtime/master_session.cc:569] - No response from RunGraph call to worker: /job:localservice/replica:0/task:0
2022-04-14 18:14:10.624800: E tensorflow/core/distributed_runtime/master_session.cc:564] RunStep still blocked after 60 seconds. Failed with error status: CANCELLED: RunManyGraphs
2022-04-14 18:14:10.624900: E tensorflow/core/distributed_runtime/master_session.cc:569] - No response from RunGraph call to worker: /job:localservice/replica:0/task:0
2022-04-14 18:14:10.624920: E tensorflow/core/distributed_runtime/master_session.cc:564] RunStep still blocked after 60 seconds. Failed with error status: CANCELLED: RunManyGraphs
2022-04-14 18:14:10.625004: E tensorflow/core/distributed_runtime/master_session.cc:569] - No response from RunGraph call to worker: /job:localservice/replica:0/task:0
2022-04-14 18:14:11.146997: E tensorflow/core/distributed_runtime/master_session.cc:564] RunStep still blocked after 60 seconds. Failed with error status: CANCELLED: RunManyGraphs
2022-04-14 18:14:11.183945: E tensorflow/core/distributed_runtime/master_session.cc:569] - No response from RunGraph call to worker: /job:localservice/replica:0/task:0
2022-04-14 18:14:11.184042: E tensorflow/core/distributed_runtime/master_session.cc:564] RunStep still blocked after 60 seconds. Failed with error status: CANCELLED: RunManyGraphs
2022-04-14 18:14:11.184134: E tensorflow/core/distributed_runtime/master_session.cc:569] - No response from RunGraph call to worker: /job:localservice/replica:0/task:0
2022-04-14 18:14:11.184260: E tensorflow/core/distributed_runtime/master_session.cc:564] RunStep still blocked after 60 seconds. Failed with error status: CANCELLED: RunManyGraphs
2022-04-14 18:14:11.184336: E tensorflow/core/distributed_runtime/master_session.cc:569] - No response from RunGraph call to worker: /job:localservice/replica:0/task:0
2022-04-14 18:14:11.184359: E tensorflow/core/distributed_runtime/master_session.cc:564] RunStep still blocked after 60 seconds. Failed with error status: CANCELLED: RunManyGraphs
2022-04-14 18:14:11.184416: E tensorflow/core/distributed_runtime/master_session.cc:569] - No response from RunGraph call to worker: /job:localservice/replica:0/task:0