Merge pull request #257 from sourcerer-io/develop

feat: add markdown and qml langs support, add java 10 support

Author: anatolystansler

README.md

@@ -1,37 +1,43 @@
-
 <h1 style="font-weight:normal">
   <a href="https://sourcerer.io"><img src=https://user-images.githubusercontent.com/20287615/34189346-d426d4c2-e4ef-11e7-9da4-cc76a1ed111d.png alt="Sourcerer Logo", width=50></a>
   &nbsp; sourcerer.io
 </h1>
-  A visual profile for software engineers. Start with your public GitHub at <a href="https://sourcerer.io/start">https://sourcerer.io/start</a>. Screenshot below.
+A visual profile for software engineers.  
+<br>
+  
+Start with your public GitHub at <a href="https://sourcerer.io/start">https://sourcerer.io/start</a>.
 <br>
 
-What is it?
-===========
+
+Screenshot
+==========
 
 ![profile-sliced-sized](https://user-images.githubusercontent.com/20287615/34188912-b3aaf8ba-e4ed-11e7-861c-2adf13a921ac.png)
 <br>
 
-Once you feed it some git repos, you will get a beautiful profile that will help
-you learn things about yourself, connect to others, and become a better
-engineer. Example profiles: <https://sourcerer.io/sergey>,
-<https://sourcerer.io/ddeveloperr>, <https://sourcerer.io/osantana>, <https://sourcerer.io/adnanrahic>.
+What is it?
+===========
+
+Once you feed [Sourcerer](https://sourcerer.io/) some git repos, you will get a beautiful profile that will help you learn things about yourself, connect to others, and become a better
+engineer. 
 
-Profile is the first step. Some of the things on our roadmap:
-news that is relevant to your code, engineers to follow and learn from,
-technology and libraries you should know about, projects
-that could use your help.
+Example profiles:<br> 
+<https://sourcerer.io/sergey><br>
+<https://sourcerer.io/ddeveloperr><br>
+<https://sourcerer.io/wanghuaili><br>
+<https://sourcerer.io/adnanrahic><br>
 
-Both open source and closed source repos are fine. The easiest way to get started with open source repos is to go to
-<https://sourcerer.io/start>, and select GitHub. For closed source repos, you can
- use this Sourcerer app. You will still need an account, which you can get at <https://sourcerer.io/>
+Both open source and closed source repos are fine. The easiest way to get started is with your open source repos. Go to <https://sourcerer.io/start>, and select *Build with GitHub* and watch your profile build. For closed source repos, you will need to use this app. If you already created an account using GitHub, you would have received an email with credentials for the app.  If not, You will need a new account, which you can get at <https://sourcerer.io/start>, and select *Build with app*.
 
-Sourcerer app does not upload source code anywhere, and it never will. The app looks at repos locally on your
-machine, and then sends stats to sourcerer.io. The best way to verify is to
-look at the code. [src/main/proto/sourcerer.proto](https://github.com/sourcerer-io/sourcerer-app/blob/develop/src/main/proto/sourcerer.proto)
+The Sourcerer app does **NOT** upload source code anywhere, and it **NEVER** will. The app looks at repos locally on your machine, and then sends stats to sourcerer.io. The best way to verify is to look at the code. [src/main/proto/sourcerer.proto](https://github.com/sourcerer-io/sourcerer-app/blob/develop/src/main/proto/sourcerer.proto)
 is a good start as it describes the client-server protocol.
 
-You will need an account, sign up on <https://sourcerer.io/>
+Creating your profile is just the first step for us at Sourcerer. Some of the things on our roadmap include:
+* News that is relevant to your code
+* Engineers to follow and learn from
+* Technology and libraries you should know about
+* Projects that could use your help
+
 
 Requirements
 ============
@@ -75,7 +81,17 @@ $ gradle build -Penv=production
 $ java -jar build/libs/sourcerer-app.jar
 ```
 
-Contribution
+Contributing
 ============
 
-We welcome contributions to Sourcerer App by the community. Check the [Contribution guide](https://github.com/sourcerer-io/sourcerer-app/blob/master/CONTRIBUTING.md) for more information.
+We love contributions!  Check out the [Contribution guide](https://github.com/sourcerer-io/sourcerer-app/blob/master/CONTRIBUTING.md) for more information.
+
+Some handy links:<br>
+* [Sourcerer Site](https://sourcerer.io/)
+* [Sourcerer Blog](https://blog.sourcerer.io)
+* [Follow Sourcerer on Twitter](https://twitter.com/sourcerer_io)
+* [Follow Sourcerer on Facebook](https://www.facebook.com/sourcerer.io/)
+
+License
+=======
+Sourcerer is under the MIT license. See the [LICENSE](https://github.com/sourcerer-io/sourcerer-app/blob/develop/LICENSE.md) for more information.

build.gradle

@@ -1,7 +1,7 @@
 // Copyright 2017 Sourcerer Inc. All Rights Reserved.
 
 buildscript {
-    ext.kotlin_version = '1.1.51'
+    ext.kotlin_version = '1.2.31'
     repositories {
         mavenCentral()
         jcenter()
@@ -95,17 +95,17 @@ dependencies {
     compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"
     compile "org.jetbrains.kotlin:kotlin-reflect:$kotlin_version"
     compile "com.beust:jcommander:1.72"
-    compile 'com.google.protobuf:protobuf-java:3.4.0'
+    compile 'com.google.protobuf:protobuf-java:3.5.1'
     compile 'commons-codec:commons-codec:1.5'
-    compile 'com.fasterxml.jackson.core:jackson-databind:2.8.9'
-    compile 'com.fasterxml.jackson.module:jackson-module-kotlin:2.8.9'
-    compile 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.8.9'
-    compile 'io.reactivex.rxjava2:rxjava:2.1.6'
-    compile 'com.github.kittinunf.fuel:fuel:1.11.0'
-    compile 'com.github.kittinunf.fuel:fuel-rxjava:1.11.0'
+    compile 'com.fasterxml.jackson.core:jackson-databind:2.9.5'
+    compile 'com.fasterxml.jackson.module:jackson-module-kotlin:2.9.5'
+    compile 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.5'
+    compile 'io.reactivex.rxjava2:rxjava:2.1.12'
+    compile 'com.github.kittinunf.fuel:fuel:1.12.1'
+    compile 'com.github.kittinunf.fuel:fuel-rxjava:1.12.1'
     compile 'org.eclipse.jgit:org.eclipse.jgit:4.9.0.201710071750-r'
     compile 'org.slf4j:slf4j-nop:1.7.2'
-    compile 'io.sentry:sentry:1.6.0'
+    compile 'io.sentry:sentry:1.7.3'
 
     testCompile 'org.jetbrains.kotlin:kotlin-test'
     testCompile 'org.jetbrains.spek:spek-api:1.1.5'
@@ -115,7 +115,7 @@ dependencies {
 
 protobuf {
     protoc {
-        artifact = "com.google.protobuf:protoc:3.4.0"
+        artifact = "com.google.protobuf:protoc:3.5.1"
     }
 }
 

src/install/install

@@ -45,7 +45,9 @@ fi
 VERSION=0
 if which java > /dev/null ; then
 	VERSION_STR=\$(java -version 2>&1 | awk -F '"' '/version/ {print \$2}')
-	if [[ "\$VERSION_STR" > "9." ]]; then  # Format: 9.0.1
+	if [[ "\$VERSION_STR" = "10" || "\$VERSION_STR" > "10" ]]; then  # Format: 10
+    	VERSION=10
+	elif [[ "\$VERSION_STR" > "9." ]]; then  # Format: 9.0.1
 	    VERSION=9
 	elif [[ "\$VERSION_STR" > "1.8." ]]; then  # Format: 1.8.1
 	    VERSION=8
@@ -65,8 +67,8 @@ if [ \$VERSION = 0 ] ; then
   exit 1
 fi
 
-# Java 9 requires additional parameters.
-if [ \$VERSION = 9 ] ; then
+# Java 9 and 10 requires additional parameters.
+if [[ \$VERSION = 9 || \$VERSION = 10 ]] ; then
   java --add-modules java.activation --add-opens java.base/java.nio=ALL-UNNAMED -jar $JAR_DIR/sourcerer.jar "\$@"
 else
   java -jar $JAR_DIR/sourcerer.jar "\$@"

src/main/kotlin/app/Logger.kt

@@ -225,8 +225,8 @@ object Logger {
         }
     }
 
-    val isDebug: Boolean
-        inline get() = LEVEL >= DEBUG
+    val isTrace: Boolean
+        inline get() = LEVEL >= TRACE
 
     @kotlin.PublishedApi
     internal fun addBreadcrumb(message: String, level: Breadcrumb.Level) {

src/main/kotlin/app/extractors/CommonExtractor.kt

@@ -37,6 +37,7 @@ class CommonExtractor : ExtractorInterface {
             reversedMap["lisp"] = listOf("lisp", "lsp", "l")
             reversedMap["lua"] = listOf("lua")
             reversedMap["makefile"] = listOf("makefile")
+            reversedMap["markdown"] = listOf("md", "markdown")
             reversedMap["matlab"] = listOf("m", "mlx")
             reversedMap["maven"] = listOf("pom")
             reversedMap["ocaml"] = listOf("ml", "mli")
@@ -47,6 +48,7 @@ class CommonExtractor : ExtractorInterface {
             reversedMap["processing"] = listOf("pde")
             reversedMap["prolog"] = listOf("P")
             reversedMap["puppet"] = listOf("pp")
+            reversedMap["qml"] = listOf("qml")
             reversedMap["r"] = listOf("r", "R")
             reversedMap["rust"] = listOf("rs")
             reversedMap["sas"] = listOf("sas")

src/main/kotlin/app/extractors/ExtractorInterface.kt

@@ -203,6 +203,15 @@ interface ExtractorInterface {
             probabilities[libraries.indexOf(it)] >= 0.2 * maxProbability
         }
 
+        // For languages with small number of libraries(e.g. less than 20).
+        // When found high probability then there won't be two or more
+        // values with high probability. And vise versa,
+        // if several values with high probability are found
+        // then the prediction is unsure. So we don't take them into account.
+        if (libraries.size < 20 && selectedCategories.size > 1) {
+            return emptyList()
+        }
+
         if (maxProbabilityCategory == languageLabel) {
             return emptyList()
         }

src/main/kotlin/app/hashers/CommitCrawler.kt

@@ -135,18 +135,18 @@ object CommitCrawler {
             // Smart casts are not yet supported for a mutable variable captured
             // in an inline lambda, see
             // https://youtrack.jetbrains.com/issue/KT-7186.
-            if (Logger.isDebug) {
+            if (Logger.isTrace) {
                 val commitName = commit.name
                 val commitMsg = commit.shortMessage
-                Logger.debug { "commit: $commitName; '$commitMsg'" }
+                Logger.trace { "commit: $commitName; '$commitMsg'" }
                 if (parentCommit != null) {
                     val parentCommitName = parentCommit.name
                     val parentCommitMsg = parentCommit.shortMessage
-                    Logger.debug { "parent commit: $parentCommitName; " +
+                    Logger.trace { "parent commit: $parentCommitName; " +
                         "'$parentCommitMsg'" }
                 }
                 else {
-                    Logger.debug { "parent commit: null" }
+                    Logger.trace { "parent commit: null" }
                 }
             }
 

src/test/kotlin/test/tests/extractors/ExtractorTest.kt

@@ -112,7 +112,7 @@ class ExtractorTest : Spek({
         }
 
         it("ruby extractor extracts rails") {
-            val line = "class Article < ApplicationRecord"
+            val line = "class Article < ActiveRecord::Base"
             assertExtractsLineLibraries("rails",
                     line, RubyExtractor())
         }