Failure to save network stats

[darkrift]

This is using Bazel 9, not sure if relevant though

ERROR: The Build Event Protocol upload failed: Not retrying publishBuildEvents, no more attempts left: status='Status{code=UNKNOWN, description=Failed to process build event batch: Failed to commit build event batch: Failed to save individual event: Failed to save build event of type *proto.BuildEventId_BuildMetrics: Failed to save network metrics: Failed to save system network stats: Failed to save system network stats to database: failed to encode args[1]: unable to encode 0xffffffffe2919000 into binary format for int8 (OID 20): 18446744073215774720 is greater than maximum value for int64, cause=null}' UNKNOWN: UNKNOWN: Failed to process build event batch: Failed to commit build event batch: Failed to save individual event: Failed to save build event of type *proto.BuildEventId_BuildMetrics: Failed to save network metrics: Failed to save system network stats: Failed to save system network stats to database: failed to encode args[1]: unable to encode 0xffffffffe2919000 into binary format for int8 (OID 20): 18446744073215774720 is greater than maximum value for int64 UNKNOWN: UNKNOWN: Failed to process build event batch: Failed to commit build event batch: Failed to save individual event: Failed to save build event of type *proto.BuildEventId_BuildMetrics: Failed to save network metrics: Failed to save system network stats: Failed to save system network stats to database: failed to encode args[1]: unable to encode 0xffffffffe2919000 into binary format for int8 (OID 20): 18446744073215774720 is greater than maximum value for int64

[isakstenstrom]

Hmm interesting.

From my testing, args[1] corresponds to the systemNetworkStats.BytesRecv field. That error appears to say that the host machine during the invocation received almost 16 exabytes worth of data? What kind of monster machines are you running on??:) Alternatively it could also be -0.5GB, but negative bytes are not much better.

The build event protocol specifies that the bytes_recv field has the type uint64. No matter if Bazel tries to send a extremely large number or a negative number, I think this is a bug in Bazel.

Do you have a json file with the BES for this invocation? I assume that this is a very hard to replicate error, but I would very much like to see what Bazel is sending in this case.

[darkrift]

{"id":{"buildMetrics":{}},"lastMessage":true,"buildMetrics":{"actionSummary":{"actionsCreated":"2123","actionsExecuted":"1745","actionsCreatedNotIncludingAspects":"2123","actionData":[{"mnemonic":"CopyFile","actionsExecuted":"1095","firstStartedMs":"1771619876820","lastEndedMs":"1771619892316","systemTime":"0.004s","userTime":"0.001s","actionsCreated":"39685"},{"mnemonic":"CppCompile","actionsExecuted":"487","firstStartedMs":"1771619876805","lastEndedMs":"1771619892316","actionsCreated":"86481"},{"mnemonic":"GoCompilePkg","actionsExecuted":"40","firstStartedMs":"1771619879500","lastEndedMs":"1771619892316","actionsCreated":"7129"},{"mnemonic":"CppArchive","actionsExecuted":"20","firstStartedMs":"1771619876814","lastEndedMs":"1771619885224","actionsCreated":"7596"},{"mnemonic":"CreateCompileJar","actionsExecuted":"17","firstStartedMs":"1771619876805","lastEndedMs":"1771619880408","actionsCreated":"68"},{"mnemonic":"CopyToDirectory","actionsExecuted":"15","firstStartedMs":"1771619876813","lastEndedMs":"1771619891897","actionsCreated":"1083"},{"mnemonic":"CppLink","actionsExecuted":"15","firstStartedMs":"1771619881415","lastEndedMs":"1771619888852","actionsCreated":"527"},{"mnemonic":"RunNogo","actionsExecuted":"15","firstStartedMs":"1771619884459","lastEndedMs":"1771619892316","actionsCreated":"4661"},{"mnemonic":"Javac","actionsExecuted":"9","firstStartedMs":"1771619884041","lastEndedMs":"1771619890041","actionsCreated":"39"},{"mnemonic":"CcStage0Compile","actionsExecuted":"5","firstStartedMs":"1771619876814","lastEndedMs":"1771619883950","actionsCreated":"395"},{"mnemonic":"CppTransitiveArchive","actionsExecuted":"5","firstStartedMs":"1771619876805","lastEndedMs":"1771619892312","actionsCreated":"330"},{"mnemonic":"StampJarManifest","actionsExecuted":"4","firstStartedMs":"1771619876803","lastEndedMs":"1771619888578","actionsCreated":"68"},{"mnemonic":"ValidateNogo","actionsExecuted":"3","firstStartedMs":"1771619890038","lastEndedMs":"1771619892312","actionsCreated":"488"},{"mnemonic":"Turbine","actionsExecuted":"2","firstStartedMs":"1771619884340","lastEndedMs":"1771619887369","actionsCreated":"24"},{"mnemonic":"GoStdlib","actionsExecuted":"2","firstStartedMs":"1771619880435","lastEndedMs":"1771619884457","actionsCreated":"40"},{"mnemonic":"Genrule","actionsExecuted":"2","firstStartedMs":"1771619876901","lastEndedMs":"1771619878471","actionsCreated":"245"},{"mnemonic":"ExecutableSymlink","actionsExecuted":"2","firstStartedMs":"1771619876852","lastEndedMs":"1771619878440","actionsCreated":"2195"},{"mnemonic":"JavaToolchainCompileBootClasspath","actionsExecuted":"1","firstStartedMs":"1771619878422","lastEndedMs":"1771619880650","actionsCreated":"6"},{"mnemonic":"Ijar","actionsExecuted":"1","firstStartedMs":"1771619880650","lastEndedMs":"1771619884041","actionsCreated":"6"},{"mnemonic":"JavaToolchainCompileClasses","actionsExecuted":"1","firstStartedMs":"1771619876802","lastEndedMs":"1771619878421","actionsCreated":"6"}],"runnerCount":[{"name":"total","count":1745},{"name":"disk cache hit","count":150,"execKind":"Remote"},{"name":"remote cache hit","count":34,"execKind":"Remote"},{"name":"internal","count":403},{"name":"darwin-sandbox","count":1,"execKind":"Local"},{"name":"remote","count":1157,"execKind":"Remote"}],"actionCacheStatistics":{"sizeInBytes":"10558612","saveTimeInMs":"38","hits":298,"misses":1745,"missDetails":[{},{"reason":"DIFFERENT_DEPS"},{"reason":"DIFFERENT_ENVIRONMENT"},{"reason":"DIFFERENT_FILES"},{"reason":"CORRUPTED_CACHE_ENTRY"},{"reason":"NOT_CACHED","count":1259},{"reason":"UNCONDITIONAL_EXECUTION","count":1},{"reason":"DIGEST_MISMATCH","count":485}],"cacheCheckSemaphoreWaitTimeInMs":"67552"}},"memoryMetrics":{"garbageMetrics":[{"type":"CodeHeap \u0027non-profiled nmethods\u0027","garbageCollected":"2614912"},{"type":"CodeHeap \u0027profiled nmethods\u0027","garbageCollected":"5887616"},{"type":"Compressed Class Space","garbageCollected":"2112"},{"type":"G1 Eden Space","garbageCollected":"6899630080"},{"type":"G1 Old Gen","garbageCollected":"67203880"},{"type":"G1 Survivor Space","garbageCollected":"124387080"},{"type":"Metaspace","garbageCollected":"7192"}]},"targetMetrics":{"targetsConfigured":"1095","targetsConfiguredNotIncludingAspects":"1095"},"packageMetrics":{"packagesLoaded":"1"},"timingMetrics":{"cpuTimeInMs":"40266","wallTimeInMs":"21383","analysisPhaseTimeInMs":"1434","executionPhaseTimeInMs":"15582","actionsExecutionStartInMs":"5749","criticalPathTime":"15.495441292s"},"cumulativeMetrics":{"numAnalyses":13,"numBuilds":12},"artifactMetrics":{"sourceArtifactsRead":{"sizeInBytes":"24826709","count":2277},"outputArtifactsSeen":{"sizeInBytes":"1254374304","count":7635},"outputArtifactsFromActionCache":{"sizeInBytes":"327099840","count":785},"topLevelArtifacts":{}},"buildGraphMetrics":{"actionLookupValueCount":116046,"actionCount":35793,"outputArtifactCount":46751,"postInvocationSkyframeNodeCount":807172,"actionLookupValueCountNotIncludingAspects":115171,"actionCountNotIncludingAspects":35792,"inputFileConfiguredTargetCount":29465,"outputFileConfiguredTargetCount":15873,"otherConfiguredTargetCount":5445,"dirtiedValues":[{"skyfunctionName":"DIRECTORY_LISTING","count":"174"},{"skyfunctionName":"BUILD_CONFIGURATION","count":"136"},{"skyfunctionName":"PLATFORM","count":"4"},{"skyfunctionName":"TARGET_PATTERN","count":"1"},{"skyfunctionName":"SINGLE_TOOLCHAIN_RESOLUTION","count":"339"},{"skyfunctionName":"PACKAGE","count":"1"},{"skyfunctionName":"TOOLCHAIN_RESOLUTION","count":"651"},{"skyfunctionName":"ARTIFACT_NESTED_SET","count":"2046"},{"skyfunctionName":"REGISTERED_TOOLCHAINS","count":"135"},{"skyfunctionName":"PACKAGE_LOOKUP","count":"176"},{"skyfunctionName":"ARTIFACT","count":"28"},{"skyfunctionName":"BASELINE_OPTIONS","count":"2"},{"skyfunctionName":"ACTION_EXECUTION","count":"4740"},{"skyfunctionName":"GLOBS","count":"1"},{"skyfunctionName":"BUILD_CONFIGURATION_KEY","count":"3"},{"skyfunctionName":"FILE","count":"702"},{"skyfunctionName":"CONFIGURED_TARGET","count":"574013"},{"skyfunctionName":"REGISTERED_EXECUTION_PLATFORMS","count":"136"},{"skyfunctionName":"TARGET_COMPLETION","count":"65"},{"skyfunctionName":"ASPECT","count":"875"}],"changedValues":[{"skyfunctionName":"PRECOMPUTED","count":"1"},{"skyfunctionName":"ERROR_TRANSIENCE","count":"1"},{"skyfunctionName":"FILE_STATE","count":"2"}],"builtValues":[{"skyfunctionName":"TARGET_PATTERN","count":"1"},{"skyfunctionName":"PACKAGE","count":"1"},{"skyfunctionName":"ARTIFACT_NESTED_SET","count":"956"},{"skyfunctionName":"ARTIFACT","count":"2301"},{"skyfunctionName":"FILE_STATE","count":"6"},{"skyfunctionName":"ACTION_EXECUTION","count":"1506"},{"skyfunctionName":"GLOBS","count":"1"},{"skyfunctionName":"FILE","count":"160"},{"skyfunctionName":"CONFIGURED_TARGET","count":"1249"}],"cleanedValues":[{"skyfunctionName":"ACTION_EXECUTION","count":"138"},{"skyfunctionName":"REGISTERED_EXECUTION_PLATFORMS","count":"134"},{"skyfunctionName":"PLATFORM","count":"3"},{"skyfunctionName":"PACKAGE_LOOKUP","count":"49"},{"skyfunctionName":"SINGLE_TOOLCHAIN_RESOLUTION","count":"329"},{"skyfunctionName":"TOOLCHAIN_RESOLUTION","count":"642"}],"evaluatedValues":[{"skyfunctionName":"PLATFORM","count":"5"},{"skyfunctionName":"TARGET_PATTERN","count":"1"},{"skyfunctionName":"SINGLE_TOOLCHAIN_RESOLUTION","count":"329"},{"skyfunctionName":"PACKAGE","count":"1"},{"skyfunctionName":"TOOLCHAIN_RESOLUTION","count":"1098"},{"skyfunctionName":"ARTIFACT_NESTED_SET","count":"4199"},{"skyfunctionName":"PACKAGE_LOOKUP","count":"98"},{"skyfunctionName":"ARTIFACT","count":"2347"},{"skyfunctionName":"TEST_COMPLETION","count":"1"},{"skyfunctionName":"FILE_STATE","count":"6"},{"skyfunctionName":"ACTION_EXECUTION","count":"4949"},{"skyfunctionName":"BUILD_DRIVER","count":"2"},{"skyfunctionName":"GLOBS","count":"1"},{"skyfunctionName":"FILE","count":"180"},{"skyfunctionName":"CONFIGURED_TARGET","count":"1443"},{"skyfunctionName":"REGISTERED_EXECUTION_PLATFORMS","count":"267"},{"skyfunctionName":"TARGET_COMPLETION","count":"1"}]},"workerMetrics":[{"processId":41390,"mnemonic":"Javac","isMultiplex":true,"isMeasurable":true,"workerStats":[{"collectTimeInMs":"1771619892245","workerMemoryInKb":8336,"lastActionStartTimeInMs":"1771611418866","priorWorkerMemoryInKb":8288}],"workerIds":[1],"workerKeyHash":"2130855249","workerStatus":"ALIVE","actionsExecuted":"1","priorActionsExecuted":"1"}],"networkMetrics":{"systemNetworkStats":{"bytesSent":"18446744072900581376","bytesRecv":"18446744070989369344","packetsSent":"8214046","packetsRecv":"13168333","peakBytesSentPerSec":"1797491877","peakBytesRecvPerSec":"5182434579","peakPacketsSentPerSec":"13451685","peakPacketsRecvPerSec":"20210899"}},"workerPoolMetrics":{"workerPoolStats":[{"hash":2130855249,"mnemonic":"Javac","aliveCount":"1"}]},"dynamicExecutionMetrics":{},"remoteAnalysisCacheStatistics":{}}}

[meroton-benjamin]

That is indeed a bazel bug, it is claiming that it sent exabytes of data, you should file a bug report to bazel as well.

We normalize the data which is why the problem shows up 18446744072900581376 can not be represented as int64, I assume bazel is trying to send over a negative value (-0.5GB which also doesn't make any sense but atleast is a reasonable number to get) but the protocol demands that it should only send positive integers.

    "networkMetrics": {
      "systemNetworkStats": {
        "bytesSent": "18446744072900581376",
        "bytesRecv": "18446744070989369344",
        "packetsSent": "8214046",
        "packetsRecv": "13168333",
        "peakBytesSentPerSec": "1797491877",
        "peakBytesRecvPerSec": "5182434579",
        "peakPacketsSentPerSec": "13451685",
        "peakPacketsRecvPerSec": "20210899"
      }
    },

Maybe we should just treat the values as signed in this case instead of returning an error?

[darkrift]

Should we raise that to Bazel then instead ? I have not seen that with Bazel 8

[meroton-benjamin]

I think so, if you can reliably reproduce the issue you might even be able to bisect where the issue has been added in bazel 9. But just being able to add a reference to a bug report may be useful in case we decide to have a workaround in bb-portal for the issue.

[meroton-benjamin]

Did you raise a ticket to bazel? I'd like to try and reproduce it and reference the ticket in the workaround. My thought is that we'd just allow the value to be negative, it doesn't make much sense but if that's what bazel emits then that's what bazel emits.

[darkrift]

Haven't had time to make a reproducer, but it happens when a test fails, so this might be pretty easy to reproduce

[darkrift]

@meroton-benjamin both fields are declared as a uint64 though, not int64: https://github.com/bazelbuild/bazel/blob/ed96aaa336ceaeb406c7b251c4291697daea66e2/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1298

[darkrift]

I opened a ticket (bazelbuild/bazel#28813), but I think bb-portal has a problem with those fields independent of Bazel sending huge values, they are within the protobuf defined spec