Use file content heuristics to decide file reader. #1962

Dimi1010 · 2025-09-12T09:40:58Z

The PR adds heuristics based on the file content that is more robust than deciding based on the file extension.

The new decision model scans the start of the file for its magic number signature. It then compares it to the signatures of supported file types [1] and constructs a reader instance based on the result.

A new function createReader has been added due to changes in the public API of the factory.

If no reader is available to read the file: nullptr is returned. (previously returned PcapFileDeviceReader)
If the file could not be opened or does not exist: std::runtime_error is thrown. (previously returned PcapFileDeviceReader)
The new function returns std::unique_ptr<IFileDeviceReader> instead of IFileDeviceReader*.

…sed on the magic number.

…le-selection

… tied to it.

…ics detection method.

codecov · 2025-09-12T09:59:59Z

Codecov Report

❌ Patch coverage is 86.59218% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.45%. Comparing base (f2ba2a6) to head (c7cab2b).
⚠️ Report is 2 commits behind head on dev.

Files with missing lines	Patch %	Lines
Pcap++/src/PcapFileDevice.cpp	83.03%	15 Missing and 4 partials ⚠️
Tests/Pcap++Test/Tests/FileTests.cpp	91.80%	1 Missing and 4 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##              dev    #1962    +/-   ##
========================================
  Coverage   83.45%   83.45%            
========================================
  Files         311      311            
  Lines       55000    55163   +163     
  Branches    11789    11935   +146     
========================================
+ Hits        45899    46036   +137     
- Misses       7822     8248   +426     
+ Partials     1279      879   -400

Flag	Coverage Δ
alpine320	`75.92% <65.47%> (-0.03%)`	⬇️
fedora42	`75.88% <66.27%> (-0.01%)`	⬇️
macos-13	`81.59% <78.84%> (-0.02%)`	⬇️
macos-14	`81.59% <78.84%> (-0.02%)`	⬇️
macos-15	`81.60% <78.57%> (-0.02%)`	⬇️
mingw32	`70.65% <65.57%> (-0.01%)`	⬇️
mingw64	`70.63% <65.57%> (+0.09%)`	⬆️
npcap	`?`
rhel94	`75.88% <65.47%> (-0.03%)`	⬇️
ubuntu2004	`60.17% <44.44%> (-0.03%)`	⬇️
ubuntu2004-zstd	`60.27% <44.44%> (-0.01%)`	⬇️
ubuntu2204	`75.82% <65.47%> (-0.03%)`	⬇️
ubuntu2204-icpx	`60.63% <50.58%> (-0.08%)`	⬇️
ubuntu2404	`75.92% <65.47%> (-0.03%)`	⬇️
ubuntu2404-arm64	`75.61% <65.47%> (-0.02%)`	⬇️
unittest	`83.45% <86.59%> (+<0.01%)`	⬆️
windows-2022	`85.44% <72.72%> (+0.11%)`	⬆️
windows-2025	`85.47% <72.72%> (+0.06%)`	⬆️
winpcap	`85.47% <72.72%> (-0.14%)`	⬇️
xdp	`53.40% <0.00%> (-0.17%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Pcap++/src/PcapFileDevice.cpp

seladb · 2025-09-15T07:42:24Z

Pcap++/src/PcapFileDevice.cpp

+		enum class CaptureFileFormat
+		{
+			Unknown,
+			Pcap,
+			PcapNG,
+			Snoop,
+		};


ditto: this can be an enum inside of CaptureFileFormatDetector

I suppose. It has internal linkage so it doesn't really matter.

But then we would end up with really long case names: CaptureFileFormatDetector::FileFormat::Pcap?

I guess it's fine? It's all internal anyway...

Pcap++/src/PcapFileDevice.cpp

Tests/Pcap++Test/Tests/FileTests.cpp

seladb · 2025-09-15T08:08:40Z

Tests/Pcap++Test/Tests/FileTests.cpp

-	PTF_ASSERT_NOT_NULL(dynamic_cast<pcpp::PcapNgFileReaderDevice*>(genericReader));
-	PTF_ASSERT_TRUE(genericReader->open());
+	// ------- IFileReaderDevice::createReader() Factory
+	// TODO: Move to a separate unit test.


We should add the following to get more coverage:

Open a snoop file

Open a file that is not any of the options

Open pcap files with different magic numbers

Assuming we add a version check for snoop and pcap file: create temp files with bogus data that has the magic number but wrong versions

3d713ab adds the following tests:

Pcap, PcapNG, Zst file with correct content + extension

Pcap, PcanNG file with correct content + wrong extension

Bogus content file with correct extension (pcap, pcapng, zst)

Bogus content file with wrong extension (txt)

Haven't found a snoop file to add. Do we have any?

Open pcap files with different magic numbers

Do you mean Pcap content that has just its magic number changed? Because IMO it is reasonable to consider that invalid format and fail as regular bogus data.

Assuming we add a version check for snoop and pcap file: create temp files with bogus data that has the magic number but wrong versions

Pending on #1962 (comment) .

Pcap++/src/PcapFileDevice.cpp

Move it out if it needs to be reused somewhere.

Libpcap supports reading this format since 0.9.1. The heuristics detection will identify such magic number as pcap and leave final support decision to the pcap backend infrastructure.

seladb · 2025-09-21T08:10:16Z

@Dimi1010 some CI tests fail...

…le-selection

Dimi1010 · 2025-09-26T08:18:56Z

@Dimi1010 some CI tests fail...

@seladb I think I found the issue. According to ChatGPT the Winpcap's wpcap.dll does not actually support nanosecond precision files [1], due to being based on libpcap 1.0.x. Nanosecond precision pcaps were added in libpcap 1.5.

The NPcap sdk and drivers provide their own wpcap.dll where they do support nanosec precision.

The tests we have on nanosecond precision pcap files ("nanosecs.pcap") currently are flawed. We don't use a static base base to compare to. We write a packet with the PcapFileWriterDevice configured to write nanoseconds precision and then attempt to read that with PcapFileReaderDevice. The issue with that is that the writer has 0 error reporting if its configured to write nanosecond precision but it can't due to PCAP_TSTAMP_PRECISION_NANO not being defined and just uses pcap_open_dead which uses default precision.

Due to the above the current nanosecond test in the master branch passes with Winpcap, even though it doesn't actually write / read a nanosecond precision pcap.

The new tests use a static copy of nanoseconds.pcap generated from Npcap for base image to compare to. Due to that, the a PcapFileWriterDevice with the Winpcap backend fails to read that and fails the test due to open() failure.

Edit:
[1] - Confirmed that by looking at the Winpcap DLL API. It does not have functions like pcap_open_dead_with_tstamp_precision, etc.

seladb · 2025-09-30T08:56:55Z

@Dimi1010 some CI tests fail...

@seladb I think I found the issue. According to ChatGPT the Winpcap's wpcap.dll does not actually support nanosecond precision files [1], due to being based on libpcap 1.0.x. Nanosecond precision pcaps were added in libpcap 1.5.

The NPcap sdk and drivers provide their own wpcap.dll where they do support nanosec precision.

The tests we have on nanosecond precision pcap files ("nanosecs.pcap") currently are flawed. We don't use a static base base to compare to. We write a packet with the PcapFileWriterDevice configured to write nanoseconds precision and then attempt to read that with PcapFileReaderDevice. The issue with that is that the writer has 0 error reporting if its configured to write nanosecond precision but it can't due to PCAP_TSTAMP_PRECISION_NANO not being defined and just uses pcap_open_dead which uses default precision.

Due to the above the current nanosecond test in the master branch passes with Winpcap, even though it doesn't actually write / read a nanosecond precision pcap.

The new tests use a static copy of nanoseconds.pcap generated from Npcap for base image to compare to. Due to that, the a PcapFileWriterDevice with the Winpcap backend fails to read that and fails the test due to open() failure.

Edit: [1] - Confirmed that by looking at the Winpcap DLL API. It does not have functions like pcap_open_dead_with_tstamp_precision, etc.

Maybe we can remove pcap files that have nanoseconds precision from this PR and replace them with other pcap files to make the tests pass?

Dimi1010 · 2025-09-30T09:03:25Z

Maybe we can remove pcap files that have nanoseconds precision from this PR and replace them with other pcap files to make the tests pass?

Then, we don't have coverage for the magic number for nanosecond pcap.

See #1977. It is a fix for the underlying issue.

…le-selection # Conflicts: # Pcap++/src/PcapFileDevice.cpp

Nanoseconds are skipped if they aren't supported.

…td compression is unsupported, the format detector will return Unknown.

Dimi1010 · 2025-10-02T16:45:35Z

Pcap++/src/PcapFileDevice.cpp

+
+				// PcapNG backend can support ZstdCompressed Pcap files, so we assume an archive is compressed PcapNG.
+				// If Zstd is not supported, we cannot read the file anyway, so we return Unknown.
+				if (isPcapNgFile(content) || (checkZstdSupport() && isZstdArchive(content)))


Decided to return Unknown instead of PcapNG, if Zst is not supported in the binary.
This will in turn, return nullptr for reader device. I think that is better than returning a PcapNGDevice that can't be opened.

Maybe we can also do something similar for PcapReaderDevice's nanosecond format files, if the environment can't read them?

Any thoughts @seladb ?

Yeah, I agree that it's the correct behavior 👍

seladb · 2025-10-03T07:21:12Z

Pcap++/src/PcapFileDevice.cpp

+
+				// PcapNG backend can support ZstdCompressed Pcap files, so we assume an archive is compressed PcapNG.
+				// If Zstd is not supported, we cannot read the file anyway, so we return Unknown.
+				if (isPcapNgFile(content) || (checkZstdSupport() && isZstdArchive(content)))


Yeah, I agree that it's the correct behavior 👍

Tests/Pcap++Test/TestDefinition.h

seladb · 2025-10-03T07:34:50Z

Tests/Pcap++Test/TestDefinition.h

+PTF_TEST_CASE(TestIFileReaderDeviceFactory_Pcap_MicroPrecision);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_Pcap_NanoPrecision);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_PcapNG);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_PcapNG_ZST);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_PcapNG_ZST_Unsupported);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_Invalid);


Maybe add another test for the snoop format?

seladb · 2025-10-03T07:38:53Z

Tests/Pcap++Test/TestDefinition.h

+PTF_TEST_CASE(TestIFileReaderDeviceFactory_Pcap_MicroPrecision);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_Pcap_NanoPrecision);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_PcapNG);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_PcapNG_ZST);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_PcapNG_ZST_Unsupported);
+PTF_TEST_CASE(TestIFileReaderDeviceFactory_Invalid);


Can we shorten the test names to fit into 35 characters? The current output doesn't look good:

TestLoggerMultiThread : PASSED TestIFileReaderDeviceFactory_Pcap_MicroPrecision: PASSED TestIFileReaderDeviceFactory_Pcap_NanoPrecision: PASSED TestIFileReaderDeviceFactory_PcapNG: PASSED TestIFileReaderDeviceFactory_PcapNG_ZST: PASSED TestIFileReaderDeviceFactory_Invalid: PASSED TestPcapFileReadWrite : PASSED TestPcapFileMicroPrecision : PASSED TestPcapFileNanoPrecision : PASSED

Do we have a reason to stick to 35 chars output? Perhaps we can up it to something like 50~80?
I can reduce the names, but it would hurt the specificity of the test name.

Somewhat shortened them, but also increased the width to 45.
Should be fine now?

4f52f59 & 41fe188

I asked ChatGPT, here is its suggestions:

TestIFileReaderDeviceFactory_Pcap_MicroPrecision → TestReaderFactory_Pcap_Micro TestIFileReaderDeviceFactory_Pcap_NanoPrecision → TestReaderFactory_Pcap_Nano TestIFileReaderDeviceFactory_PcapNG → TestReaderFactory_PcapNG TestIFileReaderDeviceFactory_PcapNG_ZST → TestReaderFactory_PcapNG_ZST TestIFileReaderDeviceFactory_Invalid → TestReaderFactory_Invalid

Tests/Pcap++Test/Tests/FileTests.cpp

seladb · 2025-10-03T07:45:47Z

Tests/Pcap++Test/Tests/FileTests.cpp

 	}
 };

+PTF_TEST_CASE(TestIFileReaderDeviceFactory_Pcap_MicroPrecision)


In addition to test a real pcap file, maybe we can add syntethic files that have a different magic number to test all options?
We don't have to put them in PcapExample/file_heuristics, instead we can create vectors with the content std::vector<uint8_t> and save them to temp files

Hmm, what would be the purpose? Just to test that it returns nullptr?
Doesn't TestIFileReaderDeviceFactory_Invalid already handle that?

I mean the other possible magic numbers of a valid pcap file. Since it's not easy to find such pcap files, we can generate synthetic files that are not actually valid, but will look valid for the sake of the test

seladb · 2025-10-03T07:49:13Z

Tests/Pcap++Test/Tests/FileTests.cpp

+	constexpr const char* PCAPNG_ZST_FILE_PATH = "PcapExamples/file_heuristics/pcapng-example.pcapng.zst";
+	constexpr const char* PCAPNG_ZSTD_FILE_PATH = "PcapExamples/file_heuristics/pcapng-example.pcapng.zstd";


The extension doesn't matter to determine if it's a Zstd file, so we probably don't need both files

Fair enough. I will remove the zstd file. I think the .zst is the more common extension?

Edit: #1962 (comment)

seladb · 2025-10-03T07:50:31Z

Tests/Pcap++Test/Tests/FileTests.cpp

+	// Garbage data, correct extension
+	constexpr const char* PCAP_BOGUS_FILE_PATH = "PcapExamples/file_heuristics/bogus-content.pcap";
+	constexpr const char* PCAPNG_BOGUS_FILE_PATH = "PcapExamples/file_heuristics/bogus-content.pcapng";
+	constexpr const char* PCAPNG_ZST_BOGUS_FILE_PATH = "PcapExamples/file_heuristics/bogus-content.zst";
+
+	// Garbage data
+	constexpr const char* BOGUS_FILE_PATH = "PcapExamples/file_heuristics/bogus-content.txt";


Ditto: since the extension doesn't matter, do we need all 4 files?

Technically this verifies that the extension doesn't matter? I guess this can be said about #1962 (comment) too.

I don't think it's really needed...

Dimi1010 added 4 commits September 12, 2025 12:03

Added heuristics file content detector that determines the content ba…

02de760

…sed on the magic number.

Merge remote-tracking branch 'upstream/dev' into feature/heuristic-fi…

d2b6339

…le-selection

Moved stream checkpoint outside format detector as it is not directly…

685dd9f

… tied to it.

Added a new factory function createReader that uses the new heurist…

40dee69

…ics detection method.

Dimi1010 added the enhancement label Sep 12, 2025

Add <algorithm> include.

f1e3e18

Dimi1010 added 2 commits September 12, 2025 13:17

Added unit tests.

8da1790

Deprecated old factory function.

3ad51e2

Dimi1010 added the API deprecation Pull requests that deprecate parts of the public interface. label Sep 12, 2025

Dimi1010 added 3 commits September 12, 2025 14:08

Add byte-swapped zstd magic number.

15c2000

Lint

17af8d4

Move enum closer to first usage.

46418ec

Dimi1010 marked this pull request as ready for review September 12, 2025 11:36

Dimi1010 requested a review from seladb as a code owner September 12, 2025 11:36

Dimi1010 requested review from clementperon, tigercosmos and egecetin September 12, 2025 11:36

tigercosmos approved these changes Sep 12, 2025

View reviewed changes

seladb reviewed Sep 15, 2025

View reviewed changes

Dimi1010 added 4 commits September 15, 2025 15:45

Added unit tests for file reader device factory.

3d713ab

Revert indentation.

a2391ec

Fixed StreamCheckpoint to restore original stream state.

ea328d7

Merge branch 'dev' into feature/heuristic-file-selection

db86c3e

Dimi1010 commented Sep 19, 2025

View reviewed changes

Pcap++/src/PcapFileDevice.cpp Outdated Show resolved Hide resolved

Dimi1010 added 3 commits September 20, 2025 12:59

Merge branch 'dev' into feature/heuristic-file-selection

4aed9bd

Moved isStreamSeekable helper to inside CaptureFileFormatDetector.

a83ae2b

Move it out if it needs to be reused somewhere.

Added pcap magic number for Alexey Kuznetzov's modified pcap format.

916e872

Libpcap supports reading this format since 0.9.1. The heuristics detection will identify such magic number as pcap and leave final support decision to the pcap backend infrastructure.

Merge remote-tracking branch 'upstream/dev' into feature/heuristic-fi…

022529f

…le-selection

Split the unit test into multiple smaller tests.

169fcd2

Dimi1010 mentioned this pull request Sep 26, 2025

PcapFileWriter does not actually write nanosecond precision files if used with Winpcap backend. #1975

Closed

Dimi1010 added 8 commits October 2, 2025 15:15

Merge branch 'dev' into feature/heuristic-file-selection

db8c848

Merge remote-tracking branch 'upstream/dev' into feature/heuristic-fi…

3e74912

…le-selection # Conflicts: # Pcap++/src/PcapFileDevice.cpp

Added helper to indicate if ZstSupport is enabled for PcapNg devices.

f1613c4

Split pcap microsecond and nanosecond file heuristics tests.

bc2bacd

Nanoseconds are skipped if they aren't supported.

Skipping Zst test case if zst is not supported.

58ac45d

Due to file heuristics returning PcapNG format on Zstd archive, if Zs…

3b4b5ad

…td compression is unsupported, the format detector will return Unknown.

Lint

18379b4

Added invalid device factory to pcap tag.

8a4f6f8

Dimi1010 commented Oct 2, 2025

View reviewed changes

Updated static zst archives to be actual archives.

7776e0e

Dimi1010 requested a review from seladb October 2, 2025 17:16

seladb reviewed Oct 3, 2025

View reviewed changes

Dimi1010 added 5 commits October 3, 2025 11:22

Centralized PTF test name width under a macro.

4f52f59

Add Pcap++Test header files to test sources for IDE tooling.

88ebfff

Fixed test output formatting.

41fe188

Lint

c8ae4f8

Typo fix.

c7cab2b

		constexpr const char* PCAPNG_ZST_FILE_PATH = "PcapExamples/file_heuristics/pcapng-example.pcapng.zst";
		constexpr const char* PCAPNG_ZSTD_FILE_PATH = "PcapExamples/file_heuristics/pcapng-example.pcapng.zstd";

Use file content heuristics to decide file reader. #1962

Are you sure you want to change the base?

Use file content heuristics to decide file reader. #1962

Uh oh!

Conversation

Dimi1010 commented Sep 12, 2025

Uh oh!

codecov bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dimi1010 Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

seladb commented Sep 21, 2025

Uh oh!

Dimi1010 commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seladb commented Sep 30, 2025

Uh oh!

Dimi1010 commented Sep 30, 2025

Uh oh!

Dimi1010 Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dimi1010 Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dimi1010 Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Sep 12, 2025 •

edited

Loading

Dimi1010 Sep 15, 2025 •

edited

Loading

Dimi1010 commented Sep 26, 2025 •

edited

Loading

Dimi1010 Oct 2, 2025 •

edited

Loading

Dimi1010 Oct 3, 2025 •

edited

Loading

Dimi1010 Oct 3, 2025 •

edited

Loading