This repository was archived by the owner on Nov 21, 2025. It is now read-only.
Drop telemetry.log files #42
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's Changing
I'm proposing dropping the Zeek
telemetrylog files from the sample dataWhy
To get the Zed "perf-compare" automation running healthy again.
Details
The shaper update in brimdata/super#5212 indirectly caused the problem. Those changes reflected the dropping of two fields from the telemetry logs as of Zeek v7.0.0. Since the sample data was most recently generated from Zeek v6.2.0 (#41) and hence still had those fields, the Zeek reference shaper now started flagging the presence of these "extra" fields via Zed
errorvalues (i.e., to inform the user that they may want to add them to the shaper or crop them). Known issue brimdata/super#5107 causes a panic when one attempts to output those Zederrorvalues in Zeek TSV format, and perf-compare attempts to output in many supported formats including Zeek TSV, and so that's why perf-compare failed.There were other ways I could have gone about fixing this, such as regenerating the data set again from pcap now using Zeek v7.0.0 so those extra fields would no longer be in the telemetry log files. However, the telemetry logs only showed up starting with the Zeek v6.2.0 changes in #41, and the fact the Zeek people are actively dropping fields between releases tells me these logs may keep changing in the near future anyway. Therefore in the interest of getting back up & running as quickly as possible, I'm proposing we just drop 'em for now.