Add compatibility with new DEX module (#94)

Also,
- rewrite a lot of the code
- add tests
- add travisci integration
- clean up rules

Author: CalebFenton

.dockerignore

@@ -0,0 +1,8 @@
+test-data
+research
+
+venv
+
+.idea
+.pytest_cache
+apkid.egg-info

.gitignore

@@ -1,9 +1,10 @@
 # precompiled rules included in dist but not source
 apkid/rules/rules.yarc
 
-# Apks and dex files for testing
+# APKs and DEX files for local testing
 test-data
 research
+
 ### VirtualEnv template
 # Virtualenv
 # http://iamzed.com/2009/05/07/a-primer-on-virtualenv/
@@ -318,6 +319,7 @@ nosetests.xml
 coverage.xml
 *,cover
 .hypothesis/
+.pytest_cache/
 
 # Translations
 *.mo

.travis.yml

@@ -0,0 +1,18 @@
+language: python
+dist: trusty
+cache:
+  pip: true
+#  directories:
+#    - $HOME/.cache/samples
+python:
+  - "2.7"
+  - "3.6"
+install:
+  - git clone --recursive https://github.com/rednaga/yara-python-1 yara-python
+  - cd yara-python
+  - CFLAGS="-std=gnu99" python setup.py build --enable-dex install
+  - cd ../
+  - pip install -e .
+script:
+  - ./prep-release.py | true
+  - pytest

docker/Dockerfile Dockerfiledocker/Dockerfile renamed to Dockerfile

@@ -1,11 +1,12 @@
 FROM python:2.7-slim
 
 RUN apt-get update -qq && apt-get install -y git build-essential gcc pandoc
-
 RUN pip install --upgrade pip
-RUN git clone https://github.com/rednaga/yara-python.git
+
+RUN git clone --recursive https://github.com/rednaga/yara-python-1.git yara-python
 WORKDIR yara-python
-RUN python setup.py install
+RUN CFLAGS="-std=gnu99" python setup.py build --enable-dex install
+
 RUN mkdir /apkid
 COPY ./ /apkid/
 WORKDIR /apkid

README.md

@@ -1,68 +1,77 @@
 # APKiD
 
+[![Build Status](https://travis-ci.org/rednaga/APKiD.svg?branch=master)](https://travis-ci.org/rednaga/APKiD)
+
 APKiD gives you information about how an APK was made. It identifies many compilers, packers, obfuscators, and other weird stuff. It's _PEiD_ for Android.
 
-For more information on what this tool can be used for, check out:
+For more information on what this tool can be used for check out:
 
 * [Android Compiler Fingerprinting](http://hitcon.org/2016/CMT/slide/day1-r0-e-1.pdf)
 * [Detecting Pirated and Malicious Android Apps with APKiD](http://rednaga.io/2016/07/31/detecting_pirated_and_malicious_android_apps_with_apkid/)
 
 # Installing
 
-The _yara-python_ clone and compile steps here are temporarily necessary because we must point directly to our modified version of a _Yara_ branch which includes our DEX Yara module. This step is nessecary until (if?) the original maintainers of _Yara_ merge our module into the master branch. When this happens, we will undate the instructions here. After the _yara-python_ fork is compiled, you can use `pip` to the most currently published `APKiD` package.
+Unfortunately, you can't just `pip install` APKiD since it depends on RedNaga's custom fork of [yara-python](https://github.com/rednaga/yara-python-1).
+
+First, install our yara-python fork:
 
 ```bash
-git clone --recursive https://github.com/rednaga/yara-python
+git clone --recursive https://github.com/rednaga/yara-python-1 yara-python
 cd yara-python
-python setup.py install
+python setup.py build --enable-dex install
+```
+
+Then, you can install apkid normally:
+```bash
 pip install apkid
 ```
 
-## Docker install
+This extra step is necessary until yara-python is updated with a version of Yara which includes the new, experimental DEX module.
+
+## Docker
 
-In an attempt to reduce the support ticket we receive from the above instructions being hard to follow, there is
-a docker file and script which can be used for processing files quickly. This also serves as a proof that the above
-instructions _do_ work! This usage, of course, requires that you have docker correctly installed on your machine. However the following instructions should "just work" if you have docker and git install on a machine:
+If installing is too complicated, you can just use [Docker](https://www.docker.com/community-edition)! Of course, this usage requires that you have git and docker installed on your machine.
+
+Here's how to use Docker:
 
 ```bash
 git clone https://github.com/rednaga/APKiD
 cd APKiD/
-docker-compose build
-cd docker/
-./apkid.sh ~/reverse/targets/android/example/example.apk
-[+] APKiD 1.0.0 :: from RedNaga :: rednaga.io
+docker build . -t rednaga:apkid
+docker/apkid.sh ~/reverse/targets/android/example/example.apk
+[+] APKiD 1.2.1 :: from RedNaga :: rednaga.io
 [*] example.apk!classes.dex
  |-> compiler : dx
 ```
 
 # Usage
 
 ```
-usage: apkid [-h] [-j] [-t TIMEOUT] [-o DIR] [FILE [FILE ...]]
+usage: apkid [-h] [-j] [-t TIMEOUT] [-o DIR] [-q] [FILE [FILE ...]]
 
-APKiD - Android Application Identifier v1.0.0
+APKiD - Android Application Identifier v1.2.1
 
 positional arguments:
-  FILE                  apk, dex, or directory
+  FILE                           apk, dex, or directory
 
 optional arguments:
-  -h, --help            show this help message and exit
-  -j, --json            output results in JSON format
-  -t TIMEOUT, --timeout TIMEOUT
-                        Yara scan timeout (in seconds)
-  -o DIR, --output-dir DIR
-                        write individual JSON results to this directory
+  -h, --help                     show this help message and exit
+  -j, --json                     output scan results in JSON format
+  -t TIMEOUT, --timeout TIMEOUT  Yara scan timeout (in seconds)
+  -o DIR, --output-dir DIR       write individual results to this directory (implies --json)
+  -q, --quiet                    suppress extraneous output
 ```
 
 # Submitting New Packers / Compilers / Obfuscators
 
 If you come across an APK or DEX which APKiD does not recognize, please open a GitHub issue and tell us:
+
 * what you think it is
 * the file hash (either MD5, SHA1, SHA256)
 
-We are open to any type of concept you might have for "something interesting" to detect, so do not limit yourself solely to packers, compilers or obfuscators. If there is an interesting anti disassembler, anti vm, anti* trick, please make an issue.
+We are open to any type of concept you might have for "something interesting" to detect, so do not limit yourself solely to packers, compilers or obfuscators. If there is an interesting anti-disassembler, anti-vm, anti-* trick, please make an issue.
 
-You're also welcome to submit pull requests. Just be sure to include a file hash so we can check the rule.
+Pull requests are welcome. If you're submitting a new rule, be sure to include a file hash of the APK / DEX so we can check the rule.
 
 # License
 
@@ -72,15 +81,15 @@ Depending on your needs, you must choose one of them and follow its policies. A
 
 # Hacking
 
-First you will need to install the specific version of _yara-python_ the project depends on (more information about this in the _Installing_ section):
+First, you'll need to install our fork of _yara-python_:
 
 ```bash
-git clone --recursive https://github.com/rednaga/yara-python
+git clone --recursive https://github.com/rednaga/yara-python-1 yara-python
 cd yara-python
-python setup.py install
+python setup.py build --enable-dex install
 ```
 
-Then, clone this repo, compile the rules, and install the package in editable mode:
+Then, clone this repository, compile the rules, and install the package in editable mode:
 
 ```bash
 git clone https://github.com/rednaga/APKiD
@@ -94,3 +103,20 @@ If the above doesn't work, due to permission errors dependent on your local mach
 ```bash
 pip install -e .[dev] --user
 ```
+
+If you update any of the rules, be sure to run `prep-release.py` to recompile them.
+
+# For Maintainers
+
+This section is for package maintainers.
+
+To update the PyPI package:
+
+```bash
+./prep-release.py readme
+rm dist/*
+python setup.py sdist bdist_wheel
+twine upload --repository-url https://upload.pypi.org/legacy/ dist/*
+```
+
+For more information see [Packaging Projects](https://packaging.python.org/tutorials/packaging-projects/).