csv adapter repeats header

[twiggler]

We got a report from a user that the following keeps repeating headers:

target-query -f mft ./2020JimmyWilson.E01 | rdump -C > broken-csv.mft.csv

When using the default_formatter, the mft plugin cycles through multiple record descriptors.

Because the csv adapter compares the record descriptor with the previous descriptor to determine if the
header should be written, it writes multiple headers.

According to the grammar in RFC 4180 a csv file consists of a single optional header followed by one or more records, where each record has the same number of fields.

Maybe it makes more sense to let the header be the union of all possible record fields. But perhaps these are not known in advance.

[yunzheng]

I'm not fully familiar with the MFT plugin, but it seems it currently outputs two different RecordDescriptors:

# <RecordDescriptor filesystem/ntfs/mft/std, hash=9c129434>
RecordDescriptor("filesystem/ntfs/mft/std", [
    ("string", "hostname"),
    ("string", "domain"),
    ("datetime", "ts"),
    ("string", "ts_type"),
    ("uint32", "segment"),
    ("path", "path"),
    ("string", "owner"),
    ("filesize", "filesize"),
    ("boolean", "resident"),
    ("boolean", "inuse"),
    ("string", "volume_uuid"),
    ("string", "_source"),
    ("string", "_classification"),
    ("datetime", "_generated"),
    ("varint", "_version"),
])

# <RecordDescriptor filesystem/ntfs/mft/filename, hash=cd256725>
RecordDescriptor("filesystem/ntfs/mft/filename", [
    ("string", "hostname"),
    ("string", "domain"),
    ("datetime", "ts"),
    ("string", "ts_type"),
    ("uint32", "filename_index"),
    ("uint32", "segment"),
    ("path", "path"),
    ("string", "owner"),
    ("filesize", "filesize"),
    ("boolean", "resident"),
    ("boolean", "inuse"),
    ("boolean", "ads"),
    ("string", "volume_uuid"),
    ("string", "_source"),
    ("string", "_classification"),
    ("datetime", "_generated"),
    ("varint", "_version"),
])

CSV output does not work very will with mixed RecordDescriptors as it can also appear interleaved. It would even be better if the flow.record CSV writer would raise an error if it encounters a different RecordDescriptor.

Maybe an option could be added to CSV so it can dump it to a directory, eg:

$ rdump mft.records --csv-directory /tmp

Which would then dump it to a directory containing CSV data per descriptor, eg:

/tmp/filesystem/ntfs/mft/filename.csv
/tmp/filesystem/ntfs/mft/std.csv

As a current workaround, you could dump it based on the Descriptor name:

$ target-query -f mft ./2020JimmyWilson.E01 | rdump --csv -s 'r._desc.name == "filesystem/ntfs/mft/filename"' > mft_filename.csv
$ target-query -f mft ./2020JimmyWilson.E01 | rdump --csv -s 'r._desc.name == "filesystem/ntfs/mft/std"' > mft_std.csv

Somewhat related, the DuckDB and SQLite writer will create a table per record descriptor, it will also update the columns if there are dynamic fields per RecordDescriptor. (see #72 which was the issue before we created the DuckDB and SQLite adapter to support this).

tl;dr, CSV is not very well suited for mixed record descriptors. But ideas and suggestions are welcome.