Merge pull request #564 from onekey-sec/multi-volume

feat(processing): add multi-file handler support

Author: martonilles

docs/api.md

@@ -23,6 +23,16 @@ hide:
       show_root_heading: true
       show_source: true
 
+::: unblob.models.DirectoryHandler
+    handler: python
+    options:
+      members:
+        - get_dependencies
+        - calculate_multifile
+        - extract
+      show_root_heading: true
+      show_source: true
+
 ::: unblob.models.Extractor
     handler: python
     options:
@@ -31,3 +41,12 @@ hide:
         - extract
       show_root_heading: true
       show_source: true
+
+::: unblob.models.DirectoryExtractor
+    handler: python
+    options:
+      members:
+        - get_dependencies
+        - extract
+      show_root_heading: true
+      show_source: true

docs/development.md

@@ -196,6 +196,58 @@ If you need to parse structure using different endianness, the class exposes two
     If your format allows it, we strongly recommend you to inherit from the
     StructHandler given that it will be strongly typed and less prone to errors.
 
+### DirectoryHandler class
+
+`DirectoryHandler` is a specialized handler responsible for identifying multi-file formats
+located in a directory or in a subtree. The abstract class is located in
+[unblob/models.py](https://github.com/onekey-sec/unblob/blob/main/unblob/models.py):
+
+```python
+class DirectoryHandler(abc.ABC):
+    """A directory type handler is responsible for searching, validating and "unblobbing" files from multiple files in a directory."""
+
+    NAME: str
+
+    EXTRACTOR: DirectoryExtractor
+
+    PATTERN: DirectoryPattern
+
+    @classmethod
+    def get_dependencies(cls):
+        """Return external command dependencies needed for this handler to work."""
+        if cls.EXTRACTOR:
+            return cls.EXTRACTOR.get_dependencies()
+        return []
+
+    @abc.abstractmethod
+    def calculate_multifile(self, file: Path) -> Optional[MultiFile]:
+        """Calculate the MultiFile in a directory, using a file matched by the pattern as a starting point."""
+
+    def extract(self, paths: List[Path], outdir: Path):
+        if self.EXTRACTOR is None:
+            logger.debug("Skipping file: no extractor.", paths=paths)
+            raise ExtractError
+
+        # We only extract every blob once, it's a mistake to extract the same blob again
+        outdir.mkdir(parents=True, exist_ok=False)
+
+        self.EXTRACTOR.extract(paths, outdir)
+```
+
+- `NAME`: a unique name for this handler
+- `PATTERN`: A `DirectoryPattern` used to identify a starting/main file of the given format.
+- `EXTRACTOR`: a [DirectoryExtractor](extractors.md).
+- `get_dependencies()`: returns the extractor dependencies. This helps unblob keep
+  track of [third party dependencies](extractors.md).
+- `calculate_multifile()`: this is the method that needs to be overridden in your
+  handler. It receives a `file` Path object identified by the `PATTERN` in the directory.
+  This is where you implement the logic to compute and return the `MultiFile` file set.
+
+Any files that are being processed as part of a `MultiFile` set would be skipped from `Chunk`
+detection.
+
+Any file that is part of multiple `MultiFile` is a collision and results in a processing error.
+
 ### Example Handler implementation
 
 Let's imagine that we have a custom file format that always starts with the
@@ -367,6 +419,44 @@ PATTERNS = [
 ]
 ```
 
+### DirectoryPatterns
+
+The `DirectoryHandler` uses these patterns to identify the starting/main file of a given
+multi-file format. There are currently two main types: `Glob` and `SingleFile`
+
+#### Glob
+
+The `Glob` object can use traditional globbing to detect files in a directory. This could be used when
+the file could have a varying part. There are cases where multiple multi-file set could be in a single
+directory. The job of the `DirectoryPattern` is to recognize the main file for each set.
+
+Here is an example on `Glob`:
+
+```python
+PATTERN = Glob("*.7z.001")
+```
+
+This example identify the first volume of a multi-volume sevenzip archive. Notice that this could pick
+up all first volumes in a given directory. (NB: Detecting the other volumes of a given set is the
+responsibility of the `DirectoryHandler.calculate_multifile` function. Do not write a `Glob` which picks
+up all the files of a multi-file set as that would result in errors.)
+
+
+#### SingleFile
+
+The `SingleFile` object can be used to identify a single file with a known name. (Obviously only use this if the
+main file name is well-known and does not have a varying part. It also means that only a single multi-file set
+can be detected in a given directory.)
+
+Here is an example on `SingleFile`:
+
+```python
+PATTERN = SingleFile("meta-data.json")
+```
+
+This would pick up the file `meta-data.json` and pass it to the `DirectoryHandler`. The handler still has to
+verify the file and has to find the additional files.
+
 ## Writing extractors
 
 !!! Recommendation
@@ -412,6 +502,32 @@ Two methods are exposed by this class:
 - `extract()`: you must override this function. This is where you'll perform the
   extraction of `inpath` content into `outdir` extraction directory
 
+### DirectoryExtractor class
+
+The `DirectoryExtractor` interface is defined in
+[unblob/models.py](https://github.com/onekey-sec/unblob/blob/main/unblob/models.py):
+
+```python
+class DirectoryExtractor(abc.ABC):
+    def get_dependencies(self) -> List[str]:
+        """Return the external command dependencies."""
+        return []
+
+    @abc.abstractmethod
+    def extract(self, paths: List[Path], outdir: Path):
+        """Extract from a multi file path list.
+
+        Raises ExtractError on failure.
+        """
+```
+
+Two methods are exposed by this class:
+
+- `get_dependencies()`: you should override it if your custom extractor relies on
+  external dependencies such as command line tools
+- `extract()`: you must override this function. This is where you'll perform the
+  extraction of `paths` files into `outdir` extraction directory
+
 ### Example Extractor
 
 Extractors are quite complex beasts, so rather than trying to come up with a
@@ -451,3 +567,9 @@ Learn from us so you can avoid them in the future 🙂
   back.
 - Watch out for [negative seeking](https://github.com/onekey-sec/unblob/pull/280)
 - Make sure you get your types right! signedness can [get in the way](https://github.com/onekey-sec/unblob/pull/130).
+- Try to use as specific as possible patterns to identify data in Handlers to avoid false-positive matches
+  and extra processing in the Handler.
+- Try to avoid using overlapping patterns, as patterns that match on the same data could easily collide. Hyperscan
+  does not guarantee priority between patterns matching on the same data. (Hyperscan reports matches ordered by the
+  pattern match end offset. In case multiple pattern match on the same end offset the matching order depends on the
+  pattern registration order which is undefined in unblob.)

docs/glossary.md

@@ -38,3 +38,8 @@ the recursion depth is reached. Beyond that level, no further extraction will
 happen.  
 For example, if a `tar.gz` contains a `zip` and a text file, the
 recursion depth will be **3**: 1. gzip layer, 2. tar, 3. zip and text file.
+
+#### MultiFile
+
+A set of files that were identified by a `DirectoryHandler` representing a format
+which consists of multiple files. `MultiFile` is extracted using a `DirectoryExtractor`

docs/index.md

@@ -33,7 +33,8 @@ extraction of arbitrary firmware.
 Specialized tools that can extract information from those firmware images already
 exist, but we were carving for something smarter that could identify both
 **start-offset** and **end-offset** of a specific chunk
-(e.g. filesystem, compression stream, archive, ...).
+(e.g. filesystem, compression stream, archive, ...) as well as handle formats
+split across multiple files.
 
 We **stick to the format standard** as much as possible when deriving these
 offsets, and we clearly define what we want out of identified chunks (e.g., not
@@ -98,6 +99,15 @@ unblob identifies known and unknown chunks of data within a file:
 
 ![unblob_architecture.webp](unblob_architecture.webp)
 
+unblob also supports special formats where data is split across multiple files
+like multi-volume archives or data & meta-data formats:
+
+- Special **DirectoryHandler** is responsible to identify the files that make up
+a multi files set.
+
+- Identified MultiFile sets are not carved, but rather directly extracted using
+special **DirectoryExtractor**.
+
 ## Used technologies
 
 - unblob is written in [Python](https://www.python.org/).

tests/conftest.py

@@ -11,5 +11,5 @@
 
 @pytest.fixture
 def task_result():
-    task = Task(path=Path("/nonexistent"), depth=0, chunk_id="")
+    task = Task(path=Path("/nonexistent"), depth=0, blob_id="")
     return TaskResult(task)

tests/integration/archive/sevenzip/__input__/multi-volume.tgz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c17f34d380545b1f139b52486f48b1852a9e74c2079a8e0338b0b7600a720fd6
+size 10240

tests/integration/archive/sevenzip/__output__/multi-volume.tgz_extract/mv.7z.001

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a89ae76a6a3624af5eef2bbebe3ac0ac9916d66f3a65b055a4931f28065fd55e
+size 100

tests/integration/archive/sevenzip/__output__/multi-volume.tgz_extract/mv.7z.002

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3af8f79806dc5059ce0c746906f6d4c1f4e0206abd5e1742f8a8215bf6ebae0
+size 81

tests/integration/archive/sevenzip/__output__/multi-volume.tgz_extract/mv.7z_extract/file1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47d741b6059c6d7e99be25ce46fb9ba099cfd6515de1ef7681f93479d25996a4
+size 9

tests/integration/archive/sevenzip/__output__/multi-volume.tgz_extract/mv.7z_extract/file2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0763097d2327a89fb7fc6a1fad40f87d2261dcdd6c09e65ee00b200a0128e1c
+size 9