Address more extraction edge cases; improve naming and consistency

[egibs]

This PR addresses another subtle bug when extracting .tar.gz archives that are actually tar archives as well as other invalid .gz files and also adds better symlink handling.

I noticed this failure:

unable to process /tmp/malcontent-821299582/included-source/felix-ebpf-gpl.tar.gz: extract to temp: failed to extract /tmp/malcontent-821299582/included-source/felix-ebpf-gpl.tar.gz: failed to create gzip reader: gzip: invalid header

And inspected the file locally:

$ file felix-ebpf-gpl.tar.gz 
felix-ebpf-gpl.tar.gz: POSIX tar archive (GNU)

To fix this, I moved the .gz case statement below the .tar.gz and .tgz case statement so that it wouldn't preempt ExtractTar and also added a check to validate that a given file is actually a valid gzip file (and using the defaut tar reader if not).

With these changes, the file extracts and scans correctly:

🔎 Scanning "~/Downloads/calico-fips/included-source/felix-ebpf-gpl.tar.gz"
├─ 🟡 ~/Downloads/calico-fips/included-source/felix-ebpf-gpl.tar.gz ∴ /felix/bpf-gpl/bin/from_nat_info.o [MEDIUM]
│     ≡ command & control [MEDIUM]
│       🟡 addr/ip — mentions an IP and port:
│          client_ip, ctx_port, dst_port, get_port, host_ip, inner_ip, intf_ip, local_port, nat_ip, nat_port, orig_ip, orig_port, o…
│     ≡ credential [MEDIUM]
│       🟡 sniffer/bpf — BPF (Berkeley Packet Filter): bpf
│     ≡ discovery [MEDIUM]
│       🟡 multiple — collects system and network information: ip_addr, ipv46_addr, ipv4_addr, ipv6_addr
│     ≡ filesystem [MEDIUM]
│       🟡 path/relative — references and possibly executes relative path:
│          ./arp, ./bpf, ./conntrack_types, ./counters, ./failsafe, ./fib, ./include, ./jump, ./metadata, ./nat_lookup, ./nat_types…
│     ≡ networking [MEDIUM]
│       🟡 ip/addr — mentions an 'IP address': ip_addr
│       🟡 ip/host_port — connects to an arbitrary host:port: host(revnat->port, host_to_ctx_port
│       🔵 ip/icmp — ICMP (Internet Control Message Protocol), aka ping
│       🟡 socket/raw — send raw and/or malformed IP packets: IPPROTO_RAW
│       🔵 socket/send — send a message to a socket
│     ≡ persistence [LOW]
│       🔵 kernel_module/symbol_lookup — bpf: BPF
│
├─ 🟡 ~/Downloads/calico-fips/included-source/felix-ebpf-gpl.tar.gz ∴ /felix/bpf-gpl/bin/to_lo_fib_info_co-re.o [MEDIUM]
│     ≡ command & control [MEDIUM]
│       🟡 addr/ip — mentions an IP and port:
│          client_ip, ctx_port, dst_port, func_ip, get_port, host_ip, inner_ip, intf_ip, local_port, nat_ip, nat_port, orig_ip, ori…
│     ≡ credential [MEDIUM]
│       🟡 sniffer/bpf — BPF (Berkeley Packet Filter): bpf
│     ≡ discovery [MEDIUM]
│       🟡 multiple — collects system and network information: ip_addr, ipv46_addr, ipv4_addr, ipv6_addr
│     ≡ filesystem [MEDIUM]
│       🟡 path/relative — references and possibly executes relative path:
│          ./arp, ./bpf, ./conntrack_types, ./counters, ./failsafe, ./fib, ./ifstate, ./include, ./jump, ./nat_lookup, ./nat_types,…
│     ≡ networking [MEDIUM]
│       🟡 ip/addr — mentions an 'IP address': ip_addr
│       🟡 ip/host_port — connects to an arbitrary host:port: host source port, host(revnat->port, host_to_ctx_port
│       🔵 ip/icmp — ICMP (Internet Control Message Protocol), aka ping
│       🟡 ip/syncookie — references SYN cookies, used to resist DoS attacks: syncookie
│       🟡 socket/raw — send raw and/or malformed IP packets: IPPROTO_RAW
│       🔵 socket/send — send a message to a socket
│     ≡ persistence [LOW]
│       🔵 kernel_module/symbol_lookup — bpf: BPF
│
├─ 🔵 ~/Downloads/calico-fips/included-source/felix-ebpf-gpl.tar.gz ∴ /felix/bpf-gpl/include/libbpf/include/uapi/linux/if_xdp.h [LOW]
│     ≡ anti-static [LOW]
│       🔵 obfuscation/bitwise — uses bitwise math: 1 << 0, 1 << 1, 1 << 2, 1 << 3
│     ≡ networking [LOW]
│       🔵 socket/send — send a message to a socket: sendto
│
...

[egibs]

Uncovered a few more edge cases while working on this, namely:

• .7.gz files packaged in Debian that are actually just ASCII files
• symlink handling within tar archives wasn't working as expected with nonexistent files

[stevebeattie]

I noticed this failure:

unable to process /tmp/malcontent-821299582/included-source/felix-ebpf-gpl.tar.gz: extract to temp: failed to extract /tmp/malcontent-821299582/included-source/felix-ebpf-gpl.tar.gz: failed to create gzip reader: gzip: invalid header

And inspected the file locally:

$ file felix-ebpf-gpl.tar.gz 
felix-ebpf-gpl.tar.gz: POSIX tar archive (GNU)

FYI, this specific instance looks to be a bug in the wolfi package for calico, in the build here https://github.com/wolfi-dev/os/blob/4e9ab93174c17cae8e9defb012b4a731621be1af/calico-3.29.yaml#L206-L212 ; it's trying to replicate the gpl ebpf tarball build from the upstream build process at https://github.com/projectcalico/calico/blob/810091a6acf98ecc891c5bc3c07374a786d68743/node/Makefile#L286-L290 but it dropped the xz compression without adding gzip compression, despite the target rename.

• .7.gz files packaged in Debian that are actually just ASCII files

Do you have specific examples? The only example I could find on a running Ubuntu system was /var/lib/dpkg/alternatives/builtins.7.gz which is where the current configuration info is stored for /etc/alternatives/builtins.7.gz (I can understand the intent of this alternatives link, but it doesn't seem to be used by anything but the bash packages).

Overall, though probably outside the scope of this PR, I'd like to see such files be flagged by malcontent, as they are pretty rarely legit, they probably are either:

• bugs in the upstream or the downstream packaging (e.g. calico),
• the extremely rare legit case (alternatives of compressed files, test cases exercising these kinds of issues),
• or something trying to look like something it's not, which is a bit suspicious.

Thanks.

[egibs]

Do you have specific examples?

The example you found was the one I came across. TBD if there are others in other distros.

I like the idea of flagging files for further review but the changes in this PR will at least avoid trying to extract archives that aren't valid.

[stevebeattie]

Thanks for making these changes, this looks much better.