Merge pull request #36 from yaronyg/master

Upgraded binaries to Tor 2.5.10 and Android to SDK 21

Author: yaronyg

.gitignore

@@ -21,3 +21,6 @@ local.properties
 
 # Proguard folder generated by Eclipse
 proguard/
+
+# intellij
+.idea/

README.md

@@ -10,32 +10,131 @@ __How__: We are really just a thin Java wrapper around the Tor OP binaries and j
 
 __Who__: This work is part of the [Thali Project](http://www.thaliproject.org/mediawiki/index.php?title=Main_Page) and is being actively developed by Yaron Y. Goland assigned to the Microsoft Open Technologies Hub. We absolutely need your help! Please see the FAQ below if you would like to help!
 
-To use the library:
+# How do I use this library?
+For now you get to build this yourself. Eventually, when it has enough testing, I might consider publishing it to some maven repository.
 
-1. Install local Maven
-2. Clone the repo.
-3. In your gradle.properties file (I usually keep mine in the .gradle sub-directory of my home directory, gradle knows to look there) put in the following properties:
+1. Install local maven (seriously, if you don't do this, nothing else will work)
+2. Clone this repo
+3. Install the JDK and if you are using Android, the Android SDK as well
+4. Navigate to the 'universal' sub-directory and run 'gradlew install'
 
+## Android
+If you are going to use this library with Android then go to the 'android' sub-directory and run 'gradlew install'.
+
+Now everything should be built and installed into your local maven.
+
+In your Android project add the following to dependencies in build.gradle:
+```groovy
+    compile 'com.msopentech.thali:ThaliOnionProxyAndroid:0.0.2'
+    compile 'org.slf4j:slf4j-android:1.7.7'
 ```
-systemProp.MAVEN_UPLOAD_REPO_URL=System.getProperty('user.home') + '.m2/repository'
-systemProp.MAVEN_UPLOAD_VERSION=0.0.0
+
+Also add mavenLocal() to your repositories in build.gradle (remember, this is the root level repositories, NOT repositories under buildscript).
+
+While this code doesn't do much, using it is kind of a pain because of all the ceremony in Java land. So if you are going to host a Tor hidden service on your device or if you want to open connections to the Internet via Tor then there are a variety of things you need to know. My recommendation is that you open android/src/androidTest/java/com/msopentech/thali/toronionproxy/TorOnionProxySmokeTest.java and look up the method testHiddenServiceRecycleTime and start reading.
+
+But everyone wants sample code so here is some sample code that will start a hidden service and open a socket to it.
+
+```Java
+String fileStorageLocation = "torfiles";
+OnionProxyManager onionProxyManager =
+        new AndroidOnionProxyManager(this.getApplicationContext(), fileStorageLocation);
+int totalSecondsPerTorStartup = 4 * 60;
+int totalTriesPerTorStartup = 5;
+
+// Start the Tor Onion Proxy
+if (onionProxyManager.startWithRepeat(totalSecondsPerTorStartup, totalTriesPerTorStartup) == false) {
+    Log.e("TorTest", "Couldn't start Tor!");
+    return;
+}
+
+// Start a hidden service listener
+int hiddenServicePort = 80;
+int localPort = 9343;
+String onionAddress = onionProxyManager.publishHiddenService(hiddenServicePort, localPort);
+
+// It can taken anywhere from 30 seconds to a few minutes for Tor to start properly routing
+// requests to to a hidden service. So you generally want to try to test connect to it a
+// few times. But after the previous call the Tor Onion Proxy will route any requests
+// to the returned onionAddress and hiddenServicePort to 127.0.0.1:localPort. So, for example,
+// you could just pass localPort into the NanoHTTPD constructor and have a HTTP server listening
+// to that port.
+
+// Connect via the TOR network
+// In this case we are trying to connect to the hidden service but any IP/DNS address and port can be
+// used here.
+Socket clientSocket =
+        Utilities.socks4aSocketConnection(onionAddress, hiddenServicePort, "127.0.0.1", localPort);
+
+// Now the socket is open but note that it can take some time before the Tor network has everything
+// connected and connection requests can fail for spurious reasons (especially when connecting to
+// hidden services) so have lots of retry logic.
 ```
 
-These are the values you need to publish to your local maven.
+## Java
+If you are going to use this library with Java then go to the 'java' sub-directory and run 'gradlew install'.
+
+I would also recommend running 'gradlew test'. This will make sure you are properly set up and will copy your test files which I recommend reading.
 
-4. Then run './gradlew uploadArchives' on Universal. 
-5. Then run './gradlew build' on either the Android or Java project depending on your needs. The Android project contains an ARM binary. The Java project contains binaries for Linux, Mac and Windows. 
+Now everything should be build and installed into your local maven.
 
-Note that the Java project requires an additional JAR file that you can find in Universal/libs. Alternatively if you use Maven and uploadArchives then we will install the JAR file in maven local and you can just refer to it that way. For example, one of our Java projects has the following in its gradle
+Now go to your build.gradle file (or equivalent) and make sure you add:
+```groovy
+apply plugin: 'maven'
+```
 
+Then go to your repositories and add:
 ```groovy
-    compile 'com.msopentech.thali:ThaliOnionProxyJava:0.0.0'
-    compile 'com.msopentech.thali:BriarJtorctl:0.0.0'
+mavenLocal()
 ```
 
-Those using Android don't need to worry about the additional JAR since it gets packaged automatically with the Android AAR.
+Then go to dependencies and add in:
+```groovy
+compile 'com.msopentech.thali:ThaliOnionProxyJava:0.0.2'
+compile 'com.msopentech.thali:BriarJtorctl:0.0.2'
+compile 'org.slf4j:slf4j-simple:1.7.7'
+```
 
-The main class of interest is the OnionProxyManager. You can create this on Android using the AndroidOnionProxyManager and in Java using the (no points for guessing) JavaOnionProxyManager. See the OnionProxyManager class in universal for details on supported methods and such.
+As discussed above, the code in this library is pretty trivial. But using it is hard because of the complexities of Tor and Java. For those in Java land please go to java\src\test\java\com\msopentech\thali\toronionproxy\TorOnionProxySmokeTest and check out testHiddenServiceRecycleTime().
+ 
+But here is some sample code to get you started.
+
+```Java
+String fileStorageLocation = "torfiles";
+OnionProxyManager onionProxyManager = new JavaOnionProxyManager(
+        new JavaOnionProxyContext(
+                Files.createTempDirectory(fileStorageLocation).toFile()));
+
+int totalSecondsPerTorStartup = 4 * 60;
+int totalTriesPerTorStartup = 5;
+
+// Start the Tor Onion Proxy
+if (onionProxyManager.startWithRepeat(totalSecondsPerTorStartup, totalTriesPerTorStartup) == false) {
+    return;
+}
+
+// Start a hidden service listener
+int hiddenServicePort = 80;
+int localPort = 9343;
+String onionAddress = onionProxyManager.publishHiddenService(hiddenServicePort, localPort);
+
+// It can taken anywhere from 30 seconds to a few minutes for Tor to start properly routing
+// requests to to a hidden service. So you generally want to try to test connect to it a
+// few times. But after the previous call the Tor Onion Proxy will route any requests
+// to the returned onionAddress and hiddenServicePort to 127.0.0.1:localPort. So, for example,
+// you could just pass localPort into the NanoHTTPD constructor and have a HTTP server listening
+// to that port.
+
+// Connect via the TOR network
+// In this case we are trying to connect to the hidden service but any IP/DNS address and port can be
+// used here.
+Socket clientSocket =
+        Utilities.socks4aSocketConnection(onionAddress, hiddenServicePort, "127.0.0.1", localPort);
+
+// Now the socket is open but note that it can take some time before the Tor network has everything
+// connected and connection requests can fail for spurious reasons (especially when connecting to
+// hidden services) so have lots of retry logic.        
+```
 
 # Acknowledgements
 A huge thanks to Michael Rogers and the Briar project. This project started by literally copying their code (yes, I asked first) which handled things in Android and then expanding it to deal with Java. We are also using Briar's fork of JTorCtl until their patches are accepted by the Guardian Project.
@@ -53,10 +152,13 @@ On top of universal are the java and android projects. They contain code specifi
 Note however that shared files like the jtorctl-briar, geoip and torrc are kept in Universal and we use a gradle task to copy them into the android and java projects.
 
 One further complication are tests. Hard experience has taught that putting tests into universal doesn't work well because it means we have to write custom wrappers for each test in android and java in order to run them. So instead the tests live primarily in the android project and we use a gradle task to copy them over to the Java project. This lets us share identical tests but it means that all edits to tests have to happen in the android project. Any changes made to shared test code in the java project will be lost. This should not be an issue for anyone but a dev actually working on Tor_Onion_Proxy_Library, to users its irrelevant.
+
 ## What is the maturity of the code in this project?
-Well the release version is currently 0.0.0 so that should say something. This is an alpha. We have (literally) one test. Obviously we need a heck of a lot more coverage. But we have run that test and it does actually work which means that the Tor OP is being run and is available.
+Well the release version is currently 0.0.2 so that should say something. This is an alpha. We have (literally) one test. Obviously we need a heck of a lot more coverage. But we have run that test and it does actually work which means that the Tor OP is being run and is available.
+
 ## Can I run multiple programs next to each other that use this library?
 Yes, they won't interfere with each other. We use dynamic ports for both the control and socks channel. 
+
 ## Can I help with the project?
 ABSOLUTELY! You will need to sign a [Contributor License Agreement](https://cla.msopentech.com/) before submitting your pull request. To complete the Contributor License Agreement (CLA), you will need to submit a request via the form and then electronically sign the Contributor License Agreement when you receive the email containing the link to the document. This needs to only be done once for any Microsoft Open Technologies OSS project. 
 
@@ -67,14 +169,27 @@ git config user.email YourAlias@YourDomain
 ```
 
 What we most need help with right now is test coverage. But we also have a bunch of features we would like to add. See our issues for a list.
+
 ## Where does jtorctl-briar.jar come from?
 This is a fork of jtorctl with some fixes from Briar. So we got it out of Briar's depot. The plan is that jtorctl is supposed to accept Briar's changes and then we will start to build jtorctl ourselves from the Tor depot directly.
+
 ## Where did the binaries for the Tor OP come from?
-I grabbed the ARM binary from Briar's depot but I believe they got it from Guardian. We really need to start building that ourselves.
+### Android
+The ARM binary for Android came from the OrBot distribution available at https://guardianproject.info/releases/. I take the latest PIE release qualify APK, unzip it and go to res/raw and then decompress tor.mp3 and go into bin and copy out the tor executable file and put it into android/src/main/assets
+
+### Windows
+I download the Expert Bundle for Windows from https://www.torproject.org/download/download.html.en and took tor.exe, libeay32.dll, libevent-2-0-5.dll, libgcc_s_sjlj-1.dll and ssleay32.dll from the Tor directory. I then need to zip them all together into a file called tor.zip and stick them into java/src/main/resources/native/windows/x86.
+
+### Linux
+I download the 32 bit Tor Browser Bundle for Linux from https://www.torproject.org/download/download.html.en and then unzipped and untared it and navigated to tor-browser_en-US\Browser\TorBrowser\Tor\ and copied out the tor and libevent-2.0.so.5 files. Note that for stupid reasons I really should fix I currently need to zip these files together into a file called tor.zip before sticking them into java/src/main/resources/native/linux/x86.
+
+I then do the same thing but this time with the 64 bit download and put the results into the x64 linux sub-directory.
+
+### OS/X
+I download the OS/X Tor Browser bundle from https://www.torproject.org/download/download.html.en and using 7Zip opened my way into the dmg file inside of 0.unknown partition\TorBrowser.app\TorBrowser\Tor and copied out tor.real and libevent-2.0.5.dylib. And, as with Linux, I then need to zip those two files together into a file called tor.zip and put that into java/src/main/resources/native/osx/x64.
 
-I grabbed the Windows tor.exe file from Tor's expert bundle. I had to install it and then went into the install files, found tor.exe and extracted it.
-I grabbed the OS/X and Linux (both 32 and 64 bit) from the Tor Browser Bundles for those platforms. Note that the Mac version consists of the tor.real and libevent-2.0.5.dylib files. The Linux versions consists of the tor and libevent-2.0.so.5 files.
+## Where did the geoip and geoip6 files come from?
+I took them from the Data/Tor directory of the Windows Expert Bundle (see previous question).
 
-And yes, we really should just build this all ourselves.
-## Where did the geoip file come from?
-I'm pretty sure it came from the Windows Expert Bundle.
+## Why does the Android code require minSdkVersion 16?!?!?! Why so high?
+The issue is the tor executable that I get from Guardian. To run on Lollipop the executable has to be PIE. But PIE support only started with SDK 16. So if I'm going to ship a PIE executable I have to set minSdkVersion to 16. But!!!!! Guardian actually also builds a non-PIE version of the executable. So if you have a use case that requires support for an SDK less than 16 PLEASE PLEASE PLEASE send mail to the [Thali Mailing list](https://pairlist10.pair.net/mailman/listinfo/thali-talk). We absolutely can fix this. We just haven't had a good reason to. So please give us one!

android/build.gradle

@@ -13,35 +13,34 @@ See the Apache 2 License for the specific language governing permissions and lim
 
 buildscript {
     repositories {
+    	mavenLocal()
         mavenCentral()
     }
     dependencies {
-        classpath 'com.android.tools.build:gradle:0.10.+'
-        classpath 'com.github.dcendents:android-maven-plugin:1.1'
+        classpath 'com.android.tools.build:gradle:1.0.1'
+        classpath 'com.github.dcendents:android-maven-plugin:1.2'
     }
 }
 
-apply plugin: 'android-library'
+apply plugin: 'com.android.library'
 apply plugin: 'android-maven'
 
-version = System.getProperty('MAVEN_UPLOAD_VERSION')
+version = "0.0.2"
 group = 'com.msopentech.thali'
 archivesBaseName = 'ThaliOnionProxyAndroid'
 
 repositories {
     mavenLocal()
     mavenCentral()
-    maven { url "http://thaliartifactory.cloudapp.net/artifactory/libs-snapshot" }
-    maven { url "http://thaliartifactory.cloudapp.net/artifactory/libs-release" }
 }
 
 android {
-    compileSdkVersion 19
-    buildToolsVersion "19.0.1"
+    compileSdkVersion 21
+    buildToolsVersion "21.1.2"
 
     defaultConfig {
-        minSdkVersion 1
-        targetSdkVersion 19
+        minSdkVersion 16 // Please see the readme if you need Sdk support below 16
+        targetSdkVersion 21
     }
 
     // Needed to work around https://code.google.com/p/android/issues/detail?id=61573#c14
@@ -57,31 +56,11 @@ android {
 dependencies {
     compile fileTree(dir: '../universal/libs', include: '*.jar')
     compile 'org.slf4j:slf4j-api:1.7.7'
-    compile 'com.msopentech.thali:ThaliOnionProxyUniversal:' + System.getProperty('MAVEN_UPLOAD_VERSION')
+    compile 'com.msopentech.thali:ThaliOnionProxyUniversal:' + version
 
     androidTestCompile 'org.slf4j:slf4j-android:1.7.7'
 }
 
-uploadArchives {
-    repositories {
-        mavenDeployer {
-            repository(url: System.getProperty('MAVEN_UPLOAD_REPO_URL')) {
-                authentication(userName: System.getProperty('MAVEN_UPLOAD_USERNAME'), password: System.getProperty('MAVEN_UPLOAD_PASSWORD')) 
-            }
-
-            pom.project {
-                licenses {
-                    license {
-                        name 'Apache 2.0'
-                        url 'https://thali.codeplex.com/SourceControl/latest#license.txt'
-                        distribution 'repo'
-                    }
-                }
-            }
-        }
-    }
-}
-
 task sourcesJar(type: Jar) {
     classifier = 'sources'
     from android.sourceSets.main.java.srcDirs

android/gradle/wrapper/gradle-wrapper.properties

@@ -3,4 +3,4 @@ distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionUrl=http\://services.gradle.org/distributions/gradle-1.12-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-2.2.1-all.zip