This repo contains code for generating and applying Azure Device Update Diffs. There are two main code-bases here. The code for Diff Generation and is located at src/managed/DiffGen, while the code for Diff Application is at src/diffs.

Diff Generation

The Diff Generation code-base is written in C# in .NET 6.0, but leverages some native code.

Components

The following represent the main components for Diff Generation:

1. ArchiveUtility

   Source Code: src/managed/DiffGen/ArchiveUtility

   This module is written in C# using .NET 6.0; it contains basic building blocks used for parsing and describing archive files. The module includes a basic object model to describe Archives in terms of Chunks, Payloads and Recipes and an ArchiveLoader which is an extensible mechanism to load Archive files.

2. DiffGeneration

   Source Code: src/managed/DiffGen/DiffGeneration

   This module is written in C# using .NET 6.0; it contains various methods to compare and transform items in archives. The goal of this module is to create a diff between a source and target archive by leveraging the information from their object models and utilize 3rd party compression (zstd, zlib, bsdiff) in a targeted way. When the work has been done, the code will then using PINVOKEs to callin to the C-API exports provided by the Diff Application code to serialize the object model into a binary form.

3. archive implementations

   Source Code: src/managed/DiffGen/archives

   There are several archives implementing a few different archive types which are used for updates. Each of them are separate implementations of IArchive from the ArchiveUtility module. Each implementation allows for adding new archive types to ArchiveLoader, so we can extend the behavior. Example supported archive types: cpio, tar, SWUpdate, ext4 These modules are written in C# using .NET 6.0.

4. demos

   Source Code: src/managed/DiffGen/demos

   There are a few demos used for Diff Generation. These are used to test/demonstrate features. These demos are written in C# using .NET 6.0.

5. tests

   Source Code: src/managed/DiffGen/tests

   There are a few tests used to test unit test behaviors related to diff generation. These tests are written in C# using .NET 6.0.

6. DiffGenTool

   Source Code: src/managed/DiffGen/tools/DiffGenTool

The Diff Generation solution depends on several pieces of native code.
1. DumpExtFs

   Source Code: src/tools/dumpextfs

   This tool is written in C++ using C++17 and is build in Linux using some scripts in the folder. It can be build for Linux or Cross-compiled to win32 using Mingw. This tool links with assets from the e2fsprogs project with some slight modifications - a git .patch files is included. The tool is used to process ext4 files by the managed ext4 archive module. This tool has a simple usage - simply supply a ext4 archive to the tool and it will output an JSON to stdout, or write to a file by supplying an additional parameter.

   This module uses two libraries from e2fsprogs.

2. zstd_compress_file Source Code: src

3. Diff API

   Source Code: src/diffs/api

   This is the C API module that implements diff creation and application. It wraps the rest of the native code which is written in C++. We provide a C API wrapper to enable easy PInvokes for the C# code.

Diff API

The Diff API code-base is written in C++ using C++17.

Open Source Dependencies

The code uses a few open source libraries:
1. BSDiff
2. Bzip2 (via BSDiff)
3. libgcrypt (linux only)
4. ligpg-error (linux only)
5. ms-gsl
6. ZLIB
7. ZSTD

Components

The following represent the major components of Diff Application:
1. diff lib

   Source Code: src/diffs

   This module is written in C++ using C++17. This is the main module for implementing diffs. This code has objects to represent the diff, archive_items (chunks, blobs, payload), and recipes; these exists in parallel to the objects that are present in the DiffGeneration code-base, as they represent the same ideas. The a diff is composed of a set of one or more chunks, which themselves contain recipes, which may refer to more archive_items; the diff ends up with a tree-like structure. While applying the diff we must traverse the tree of archive_items and recipes and write them to the desired location. To accomplish this we create an apply_context object that can be mutated as it visits the tree; the mutations will allow us to read/write data for the diff application appropriately without having to modify any state on the diff items.

2. diff api

   Source Code: src/diffs/api

   The code compiles with default C options for gcc and Visaul Studio, but should be compatible with C11 and C17. A C API to expose apply and create functionality from the C++ code in the diff lib. This allows both for PINVOKE from C# code and also calling from applications written in C. The headers for the code here expose a simple session model for the API with create/close APIs and other APIs for interacting with the handle to apply or create diffs.

3. hash_utility

   Source Code: src/hash_utility

   This module is written in C++ using C++17. A module to support hashing bytes in a cross-platform manner. It has #ifdefs for WIN32 to use bcrypt and otherwise uses mhash for Linux. We want to isolate any code that has to use #ifdefs from the rest of the code.

4. io_utility

   Source Code: src/io_utility

   This module is written in C++ using C++17. A module to support reading and writing from streams. The streams/objects here are not compatible with boost or C++ streams, but instead are specific to this implementation. Streaming to/from files is supported as well as streaming with various compression types (zstd, zlib, bsdiff).

4) applydiff

Source Code: src/tools/applydiff

This tool is written in C. The code compiles with default C options for gcc and Visaul Studio, but should be compatible with C11 and C17. This tool is a simple wrapper around the diff api. It is used as a proof of concept for the API and also a testing tool in development.

Building in Linux: To build the Diff Generation code in Linux, use the dotnet SDK: sudo apt-get install -y dotnet-sdk-6.0 dotnet buildbuild/diff-generation.sln To clean the build output, use: dotnet clean build/diff-generation.sln And to run the unit tests on Linux: dotnet test src/managed/DiffGen/tests/UnitTests/UnitTests.csproj To build the native C++ tools, preferably use CMAKE at the AzureDeviceUpdateDiffs/src/out folder.