Skip to content
This repository was archived by the owner on Apr 8, 2020. It is now read-only.

Design documentation

Jump to bottom
Jonas Reinwald edited this page Feb 20, 2018 · 21 revisions

Structure (FIXME remove subpoints from structure once done)

Abstract

This project was develop by Julian Raufelder and Jonas Reinwald as an assignment in the context of the lecture PiR (Programmieren in Rust - Programming in Rust), held by Professor Mächtel at the University of Applied Science in Constance. The goal was to give students the chance to get a better understanding of a new programming language (Rust) and to help them understand how to collaboratively work on an open source team project.

This repository aims to provide a cross-plattform backup solution for android phones with a small (and in the future hopefully growing) but robust set of features for the rust community. Meanwhile, the goal of this wiki page is to provide insight into the design thoughts, the development process and the difficulties we were having along the way.

A packaged version of this code can be found on Crates.io.

Project Idea

Design / Structure

The crate has several features which can be invoked either programmatically or by specifying the respective keywords when using the cli tool. This also reveals the first notable design decision: We split our work into a module / library part and a cli part. The cli part is a thin wrapper around our module - it basically only handles user interaction on the console (parsing params, printing help, ...) and uses the module part itself to carry out any real work. There are a few reasons for providing not just the module but also the cli wrapper:

  • Users can start using the crate quickly and test out features before integrating it into their own work
  • People without programming experience (i.e. people wo would not be able to use the module part of the crate) should be able to benefit from our work
  • Implementing it as a thin wrapper is a good test for our module API and gives us valuable feedback (shortcomings, awkward usage) about it

The module part is where most of the work went into, it currently (v0.5.0) has the following sub-modules / features:

  • adb_command: Building and executing calls to adb, the tool used to interact with the android phone(s).
  • backup: Building and executing backup commands (full backup with apps and optionally their apks, shared storaage, and system apps or just individual apps) and getting a list of apps installed on a phone is done here. Uses adb_command.
  • database
    • management: Handles connections with device specific databases and the insertion / retrieval of data.
    • migration: Used to ensure our databases are in a valid state (have the correct version) and to migrate databases from an older version to the current one.
  • devices: Generates a list of all attached android devices and information about them. Uses adb_command.
  • file_transfer: Can pull or push single files from / to a device. Uses adb_command.
  • lib: Central API interface of the module. Uses all other sub-modules listed here and evaluates their return values / potential errors.
  • logging: Configuration for the logging mechanism. Used at the initialization of the module.
  • restore: Counterpart to backup, restores android backups to the device. Used adb_command.

Dependencies

  • clap: Used for specifying and handling all command line parameters.
  • failure:
  • fern: A logging framework which we have configured to log to a file when being used as a dependency of another rust project and to the console (stdout) when being invoked from our cli tool.
  • rusqlite: Rustqlite offers bindings for SQLite (with an instance of a native SQLite client already bundled within). We use it to store complete backups (and in the future single files) in a SQLite database for easier access and centralized safekeeping of all data.

Tests & CI

Especially a backup suite have to to be as reliable as possible. A high test coverage is essential to proof the quality of the code. With this mindset we started developing adbackup, put on Travis (Linux & macOS) and Appveyour (Windows) for continuous integration to build the program and to execute our tests on every commit.
The main idea was it to split our tests into three different categories:

  • unit: directly in the source file. Used to test unit based the functions
  • integration: Tests are placed in a separate file. Test the functionality of adbackup from inside but with a greater view than an unit e.g. call the backup command and check the result.
  • blackbox: Also here, all tests are placed in a separate file. Using rusts Command API to execute the possible commands of adbackup and check the results.

Most of the integration- and blackbox- and also allot of unit-test require the presence of a android device. We already used in other projects an android emulator for executing tests, local and on Travis but only while building an android application.
While starting developing adbackup, we come across a problem using travis and Appveyor in combination with such an emulator:

At the moment it is not possible to start an android emulator on appveyor in general and on Travis only, if the project target is Android. On the other side, travis doesn't support multy targets right now so if we choose android, we can not build rust applications. It is realy sad that it is not possible with this setting, to execute all adbackup commands and check the results withoud having an Android device on the server side.

We decided to write unit tests for functions, where no emulator is involved or where we can mock the response/data from the device. For all other tests that depend on an android devices, we created an issue to periodically check, if Travis and Appveyor will fix the limitations and as soon as that happens, we will tighten up the missing tests: https://github.com/DonatJR/adbackup/issues/20

Deployment

adbackup is available on crates.io and as soon as we have an first release version 1.0.0, we will host the binaries for each version and operation system in this repository on github: releases

Difficulties and Limitations

General

Experimental branch

A good backup solution has to provide incremental backups. We wanted to implement this feature from the get-go, but a few rather unfortunate issues encountered during development prevented us from doing so:

  1. The backup created by android phones / adb is encrypted by a encryption mechanism for which we need a respective decryption method. We could not find a fitting rust crypto crate which handles this singlehandedly (FIXME better explanation) (but the ring crate might soon be able to). This forced us to rely on a third party java tool, the Android Backup Extractor, to handle the decryption of the backup, which in turn meant potential users would have to download or build an executable of the extractor and also download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files if they were not yet running Java 9.
  2. Once decrypted an android backup is basically a tar-archive. To extract these archives we wanted to use the tar-rs create. Unfortunately, the android archives could contain files with file names which are not valid for every operating system adbackup is intended to run on (e.g. : on Windows). This results in a few non-extractable files for every backup and in turn could lead to corrupted data when restoring the (incomplete) backup if these files were essential for certain apps or the entire system to run. To circumvent this in the future we either need to find a crate which can replace / map invalid file name characters when extracting files from archives or request / implement this feature in the tar-rs crate.

On its own, the first issue would be bad enough, but combined with the second, more severe point we could not in good conscience publish the crate with the incremental backup feature enabled. As a compromise and because we didn't just want to delete code which could become useful in the near future we decided to create an experimental-features-branch (https://github.com/DonatJR/adbackup/tree/experimental-features).

We intend to run this branch in parallel with develop and host all as of yet unstable features / code snippets on or under it.

Conclusion

Structure

Clone this wiki locally