Merge pull request #305 from sourcerer-io/develop

feat: new tech classfiers

Author: astansler

.gitattributes

@@ -0,0 +1 @@
+src/test/resources/samples/* linguist-vendored

README.md

@@ -1,71 +1,101 @@
 <h1 style="font-weight:normal">
-  <a href="https://sourcerer.io"><img src=https://user-images.githubusercontent.com/20287615/34189346-d426d4c2-e4ef-11e7-9da4-cc76a1ed111d.png alt="Sourcerer Logo", width=50></a>
-  &nbsp; sourcerer.io
+  <a href="https://sourcerer.io">
+    <img src=https://user-images.githubusercontent.com/20287615/34189346-d426d4c2-e4ef-11e7-9da4-cc76a1ed111d.png alt="Sourcerer" width=35>
+  </a>
+  &nbsp;sourcerer.io&nbsp;
+  <a href="https://sourcerer.io/start"><img src=https://img.shields.io/badge/sourcerer-start%20now-brightgreen.svg?colorA=087c08></a>
+  <a href="https://github.com/sourcerer-io/sourcerer-app/releases"><img src=https://img.shields.io/github/release/sourcerer-io/sourcerer-app.svg?colorB=58839b></a>
+  <a href="https://github.com/sourcerer-io/sourcerer-app/blob/master/LICENSE.md"><img src=https://img.shields.io/github/license/sourcerer-io/sourcerer-app.svg?colorB=ff0000></a>
 </h1>
-A visual profile for software engineers.  
-<br>
-  
-Start with your public GitHub at <a href="https://sourcerer.io/start">https://sourcerer.io/start</a>.
-<br>
-
-
-Screenshot
-==========
 
-![profile-sliced-sized](https://user-images.githubusercontent.com/20287615/38911373-9f700892-4283-11e8-853b-1ca1fb9f5fa0.png)
+A visual profile for software engineers.
 <br>
 
-What is it?
-===========
-
-Once you feed [Sourcerer](https://sourcerer.io/) some git repos, you will get a beautiful profile that will help you learn things about yourself, connect to others, and become a better
-engineer. 
-
-Example profiles:<br> 
-<https://sourcerer.io/sergey><br>
-<https://sourcerer.io/ddeveloperr><br>
-<https://sourcerer.io/wanghuaili><br>
-<https://sourcerer.io/adnanrahic><br>
-
-Both open source and closed source repos are fine. The easiest way to get started is with your open source repos. Go to <https://sourcerer.io/start>, and select *Build with GitHub* and watch your profile build. For closed source repos, you will need to use this app. If you already created an account using GitHub, you would have received an email with credentials for the app.  If not, You will need a new account, which you can get at <https://sourcerer.io/start>, and select *Build with app*.
+<p align="center">
+  <img alt="sergey" src="https://user-images.githubusercontent.com/20287615/42118899-cc2c0b94-7bbb-11e8-86a8-96dcd044b5e2.gif">
+</p>
 
-The Sourcerer app does **NOT** upload source code anywhere, and it **NEVER** will. The app looks at repos locally on your machine, and then sends stats to sourcerer.io. The best way to verify is to look at the code. [src/main/proto/sourcerer.proto](https://github.com/sourcerer-io/sourcerer-app/blob/develop/src/main/proto/sourcerer.proto)
-is a good start as it describes the client-server protocol.
+Features
+========
+* Profile creation with a single click
+* Support of 100 languages (even exotic ones like COBOL)
+* Detection of more than [1,000 libraries](https://github.com/sourcerer-io/awesome-libraries) in code with per-line statistics
+* Visual presentation your development experience
+* *Finally!* Summary of all repositories you've contributed to :tada:
+* Interesting facts about yourself
 
 Creating your profile is just the first step for us at Sourcerer. Some of the things on our roadmap include:
-* News that is relevant to your code
 * Engineers to follow and learn from
 * Technology and libraries you should know about
 * Projects that could use your help
 
+Get started
+===========
+The easiest way to get started is with your open source repos. Go to [sourcerer.io/start](https://sourcerer.io/start), and select *Build with GitHub* and watch your profile build. 
+
+For closed source repos, you will need to use this app. If you already created an account using GitHub, you would have received an email with credentials for the app. If not, You will need a new account, which you can get at [sourcerer.io/join](https://sourcerer.io/join>).
+
+Showcase
+========
+<center>
+  <table>
+    <tr>
+      <td><a href="https://sourcerer.io/chdemko"><img width="120" alt="chdemko" src="https://user-images.githubusercontent.com/20287615/42243607-c7f6c40c-7ec6-11e8-9f8e-d4450d1d92d1.png"></a></td>
+      <td><a href="https://sourcerer.io/chendaniely"><img width="120" alt="chendaniely" src="https://user-images.githubusercontent.com/20287615/42243623-d1bd479a-7ec6-11e8-983c-00945c926dc6.png"></a></td>
+      <td><a href="https://sourcerer.io/lauragift21"><img width="120" alt="lauragift21" src="https://user-images.githubusercontent.com/20287615/42243624-d1d24cd0-7ec6-11e8-84c0-3a1aad54a774.png"></a></td>
+      <td><a href="https://sourcerer.io/maracuja-juice"><img width="120" alt="maracuja-juice" src="https://user-images.githubusercontent.com/20287615/42243626-d1eb19c2-7ec6-11e8-8e94-fa02d8f11894.png"></a></td>
+    </tr>
+    <tr>
+      <td><a href="https://sourcerer.io/marisbotero"><img width="120" alt="marisbotero" src="https://user-images.githubusercontent.com/20287615/42243627-d203bb58-7ec6-11e8-945e-49b878f07436.png"></a></td>
+      <td><a href="https://sourcerer.io/nordes"><img width="120" alt="nordes" src="https://user-images.githubusercontent.com/20287615/42243628-d21df464-7ec6-11e8-9147-31b99ea3465d.png"></a></td>
+      <td><a href="https://sourcerer.io/ppapadeas"><img width="120" alt="ppapadeas" src="https://user-images.githubusercontent.com/20287615/42243629-d23b27e6-7ec6-11e8-92c1-0c3edc2f3dba.png"></a></td>
+      <td><a href="https://sourcerer.io/praharshjain"><img width="120" alt="praharshjain" src="https://user-images.githubusercontent.com/20287615/42243630-d2562abe-7ec6-11e8-8ad3-fd6ca3ddd413.png"></a></td>
+    </tr>
+  </table>
+</center>
 
 Requirements
 ============
+* Web browser
 
-* Linux or macOS
-* Java 8+ Platform ([JRE](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html) for Linux or [JDK](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) for macOS)
-* Git repositories with master branch with at least one commit
-* Account on <https://sourcerer.io/>
+or
 
-Install/uninstall
-=================
+* Linux or macOS or Windows
+* Java 8+ Platform ([JRE](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html) for Linux and Windows or [JDK](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) for macOS)
 
+Usage
+=====
 To install sourcerer run the following command:
 
 ```
 curl -s https://sourcerer.io/app/install | bash
 ```
 
-To remove sourcerer from your machine:
+To run wizard use `sourcerer` command
 
-```
-sourcerer --uninstall
-```
+Internals
+=========
+The app looks at repos locally on your machine, and then sends stats to sourcerer.io. The best way to verify is to look at the code. Protobuf messages declared in [src/main/proto/sourcerer.proto](https://github.com/sourcerer-io/sourcerer-app/blob/develop/src/main/proto/sourcerer.proto) is a good start as it describes the client-server protocol.
+The Sourcerer app does **NOT** upload source code anywhere, and it **NEVER** will.
+
+FAQ
+===
+### How can I process private repos?
+We process only public repos using GitHub OAuth. To process private repos you need to run sourcerer app locally. See [Get started](#get-started) for instructions. Sourcerer app sends only statistical information to our servers and never sends code.
+
+### Why do you need GitHub permissions?
+We use emails to identify commit authorship, read orgs access to get list of public repositories that you've contributed to. You also need to grant access to read this public information from an organization.
+
+### Other questions
+See [sourcerer.io/faq](https://sourcerer.io/faq).
+
+Contributing
+============
+We love contributions! Check out the [Contribution guide](https://github.com/sourcerer-io/sourcerer-app/blob/master/CONTRIBUTING.md) for more information. Simplest and really helpfull for the community would be contribution meta information to our [supported libraries list](https://github.com/sourcerer-io/awesome-libraries). If you an author of a library you show definitely add yours to the list or you can help to someone whose work you use.
 
 Build
 =====
-
-To build and run this application, you'll need latest versions of Git, Gradle and JDK installed on your computer. From your command line:
+To build and run this application locally, you'll need latest versions of Git, Gradle and JDK installed on your computer. From your command line:
 
 ```
 # Clone this repository
@@ -81,17 +111,13 @@ $ gradle build -Penv=production
 $ java -jar build/libs/sourcerer-app.jar
 ```
 
-Contributing
-============
-
-We love contributions!  Check out the [Contribution guide](https://github.com/sourcerer-io/sourcerer-app/blob/master/CONTRIBUTING.md) for more information.
+License
+=======
+Sourcerer is under the MIT license. See the [LICENSE](https://github.com/sourcerer-io/sourcerer-app/blob/develop/LICENSE.md) for more information.
 
-Some handy links:<br>
+Links
+=====
 * [Sourcerer Site](https://sourcerer.io/)
 * [Sourcerer Blog](https://blog.sourcerer.io)
 * [Follow Sourcerer on Twitter](https://twitter.com/sourcerer_io)
 * [Follow Sourcerer on Facebook](https://www.facebook.com/sourcerer.io/)
-
-License
-=======
-Sourcerer is under the MIT license. See the [LICENSE](https://github.com/sourcerer-io/sourcerer-app/blob/develop/LICENSE.md) for more information.

build.gradle

@@ -36,8 +36,8 @@ buildConfig {
     buildConfigField 'String', 'PROFILE_URL', 'https://sourcerer.io/'
 
     // App version.
-    buildConfigField 'int', 'VERSION_CODE', '10'
-    buildConfigField 'String', 'VERSION', '0.2.5'
+    buildConfigField 'int', 'VERSION_CODE', '12'
+    buildConfigField 'String', 'VERSION', '0.3.0'
 
     // Logging.
     buildConfigField 'String', 'ENV', project.hasProperty('env') ? env : 'production'
@@ -53,7 +53,7 @@ buildConfig {
     buildConfigField 'boolean', 'IS_GA_ENABLED', 'true'
 
     // Models storage path.
-    buildConfigField 'String', 'LIBRARY_MODELS_URL', 'https://storage.googleapis.com/sourcerer-app/library-models/v1/'
+    buildConfigField 'String', 'LIBRARY_MODELS_URL', 'https://storage.googleapis.com/sourcerer-app/library-models/v2/'
 
     // Hashing.
     buildConfigField 'boolean', 'COMMIT_HASHER_ENABLED', project.hasProperty('commit-hasher-enabled') ? project.property('commit-hasher-enabled').toString() : 'true'
@@ -107,6 +107,7 @@ dependencies {
     compile 'org.slf4j:slf4j-nop:1.7.2'
     compile 'io.sentry:sentry:1.7.3'
 
+    testCompile 'commons-io:commons-io:2.6'
     testCompile 'org.jetbrains.kotlin:kotlin-test'
     testCompile 'org.jetbrains.spek:spek-api:1.1.5'
     testCompile 'org.junit.platform:junit-platform-runner:1.0.1'

do.sh

@@ -28,61 +28,61 @@ GRADLE_VERSION=4.2.0
 
 # run only inside build container
 build_jar_inside() {
-  if [ "$NAMESPACE" == "sandbox" ]; then
+  if [ "${NAMESPACE}" == "sandbox" ]; then
     API="https://sandbox.sourcerer/api/commit"
     LOG="debug"
-  elif [ "$NAMESPACE" == "staging" ]; then
+  elif [ "${NAMESPACE}" == "staging" ]; then
     API="https://staging.sourcerer/api/commit"
     LOG="info"
-  elif [ "$NAMESPACE" == "local" ]; then
+  elif [ "${NAMESPACE}" == "local" ]; then
     API="http://localhost:3181"
     LOG="debug"
   else
     API="https://sourcerer.io/api/commit"
     LOG="info"
   fi
-  gradle -Penv=$NAMESPACE -Plog=$LOG -Papi=$API build
+  gradle -Penv=${NAMESPACE} -Plog=${LOG} -Papi=${API} build
 }
 
 build_jar() {
-  docker run -i -v $VOLUME:/home/gradle/app --workdir=/home/gradle/app \
-    -e LOG=$LOG -e NAMESPACE=$NAMESPACE \
-    gradle:$GRADLE_VERSION \
+  docker run -i -v ${VOLUME}:/home/gradle/app --workdir=/home/gradle/app \
+    -e LOG=${LOG} -e NAMESPACE=${NAMESPACE} \
+    gradle:${GRADLE_VERSION} \
     ./do.sh build_jar_inside
 }
 
 build_prod_inside() {
-  docker build -t $REPO_NAME .
+  docker build -t ${REPO_NAME} .
 }
 
 deploy() {
   source ./deploy/${NAMESPACE}_env.sh
   envsubst < ./deploy/sourcerer-app.yaml > /tmp/deploy.yaml
-  kubectl --namespace=$NAMESPACE apply -f /tmp/deploy.yaml
+  kubectl --namespace=${NAMESPACE} apply -f /tmp/deploy.yaml
 }
 
 ######################
 
 run_jar() {
-  docker run -i -v $VOLUME:/app --workdir=/app gradle:$GRADLE_VERSION \
+  docker run -i -v ${VOLUME}:/app --workdir=/app gradle:${GRADLE_VERSION} \
     java -jar build/libs/app.jar
 }
 
 run_prod() {
-  docker run -i -p $PORT:80 $REPO_NAME
+  docker run -i -p ${PORT}:80 ${REPO_NAME}
 }
 
 push() {
-  gcloud docker -- push $REPO_NAME
+  gcloud docker -- push ${REPO_NAME}
 }
 
 #---------------------#
 #----- Execution -----#
 #---------------------#
 
-fn_exists $COMMAND
+fn_exists ${COMMAND}
 if [ $? -eq 0 ]; then
-  $COMMAND $ARGUMENTS
+  ${COMMAND} ${ARGUMENTS}
 else
   echo "Command not found"
 fi

src/main/kotlin/app/extractors/CExtractor.kt

@@ -4,27 +4,15 @@
 
 package app.extractors
 
-import app.model.CommitStats
-import app.model.DiffFile
-
 class CExtractor : ExtractorInterface {
     companion object {
-        const val LANGUAGE_NAME = "c"
-        val FILE_EXTS = listOf("c")
-        val evaluator by lazy {
-            ExtractorInterface.getLibraryClassifier(LANGUAGE_NAME)
-        }
+        const val LANGUAGE_NAME = Lang.C
         val importRegex = Regex("""^([^\n]*#include)\s[^\n]*""")
         val commentRegex = Regex("""^([^\n]*//)[^\n]*""")
         val extractImportRegex =
             Regex("""#include\s+["<](\w+)[/\w+]*\.\w+[">]""")
     }
 
-    override fun extract(files: List<DiffFile>): List<CommitStats> {
-        files.map { file -> file.language = LANGUAGE_NAME }
-        return super.extract(files)
-    }
-
     override fun extractImports(fileContent: List<String>): List<String> {
         val imports = mutableSetOf<String>()
 
@@ -45,9 +33,13 @@ class CExtractor : ExtractorInterface {
         return super.tokenize(newLine)
     }
 
-    override fun getLineLibraries(line: String,
-                                  fileLibraries: List<String>): List<String> {
+    override fun mapImportToIndex(import: String, lang: String,
+                                  startsWith: Boolean): String? {
+        // TODO(lyaronskaya): Add C to libraries.
+        return super.mapImportToIndex(import, Lang.CPP, startsWith = true)
+    }
 
-        return super.getLineLibraries(line, fileLibraries, evaluator, LANGUAGE_NAME)
+    override fun getLanguageName(): String? {
+        return LANGUAGE_NAME
     }
 }

src/main/kotlin/app/extractors/CSharpExtractor.kt

@@ -4,57 +4,39 @@
 
 package app.extractors
 
-import app.model.CommitStats
-import app.model.DiffFile
-
 class CSharpExtractor : ExtractorInterface {
     companion object {
-        val LANGUAGE_NAME = "csharp"
-        val FILE_EXTS = listOf("cs")
-        val LIBRARIES = ExtractorInterface.getLibraries("cs")
-        val evaluator by lazy {
-            ExtractorInterface.getLibraryClassifier(LANGUAGE_NAME)
-        }
+        const val LANGUAGE_NAME = Lang.CSHARP
         val importRegex = Regex("""^.*using\s+(\w+[.\w+]*)""")
         val commentRegex = Regex("""^([^\n]*//)[^\n]*""")
         val extractImportRegex = Regex("""using\s+(\w+[.\w+]*)""")
     }
 
-    override fun extract(files: List<DiffFile>): List<CommitStats> {
-        files.map { file -> file.language = LANGUAGE_NAME }
-        return super.extract(files)
-    }
-
     override fun extractImports(fileContent: List<String>): List<String> {
         val imports = mutableSetOf<String>()
 
         fileContent.forEach {
             val res = extractImportRegex.find(it)
             if (res != null) {
-                val importedName = res.groupValues[1]
-                LIBRARIES.forEach { library ->
-                    if (importedName.startsWith(library)) {
-                        imports.add(library)
-                    }
-                }
+                imports.add(res.groupValues[1])
             }
         }
 
         return imports.toList()
     }
 
     override fun tokenize(line: String): List<String> {
-        val importRegex = Regex("""^.*using\s+(\w+[.\w+]*)""")
-        val commentRegex = Regex("""^([^\n]*//)[^\n]*""")
         var newLine = importRegex.replace(line, "")
         newLine = commentRegex.replace(newLine, "")
         return super.tokenize(newLine)
     }
 
-    override fun getLineLibraries(line: String,
-                                  fileLibraries: List<String>): List<String> {
+    override fun mapImportToIndex(import: String, lang: String,
+                                  startsWith: Boolean): String? {
+        return super.mapImportToIndex(import, lang, startsWith = true)
+    }
 
-        return super.getLineLibraries(line, fileLibraries, evaluator,
-            LANGUAGE_NAME)
+    override fun getLanguageName(): String? {
+        return LANGUAGE_NAME
     }
 }