Add documentation for the similarity notebook

Signed-off-by: m09 <[email protected]>

Author: m09

notebooks/Project and Developer Similarity.ipynb

@@ -61,7 +61,7 @@
     "\n",
     "full_run = Run(\"similarity\", \"full\")\n",
     "limited_run = Run(\"similarity\", \"limited\")\n",
-    "run = full_run"
+    "run = limited_run"
    ]
   },
   {
@@ -75,7 +75,17 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We start by the bulk of the preprocessing: extracting identifiers with [gitbase](http://docs.bigartm.org/en/stable/index.html). Since gitbase exposes any codebase as a relational database, we can extract what we wish with a SQL query:"
+    "We start by the bulk of the preprocessing: extracting identifiers from code with [`gitbase`](http://docs.bigartm.org/en/stable/index.html). `gitbase` exposes git repositories as SQL databases with the following schema:\n",
+    "\n",
+    "![`gitbase` schema](img/gitbase-schema.png)\n",
+    "\n",
+    "Ok, it's probably hard to read. It's likely possible to use `Right click` > `View image` to explore it, but we'll extract the tables we're going to use below to make our life easy:\n",
+    "\n",
+    "![tables](img/tables.png)\n",
+    "\n",
+    "Using those 3 tables, we can get identifiers with the `uast_extract(blob, key) text array` [`gitbase` function](https://docs.sourced.tech/gitbase/using-gitbase/functions), that leverages [Babelfish](https://doc.bblf.sh/). The nice point about using Babelfish is that since it exposes the same API for different languages, we write a query once, and we get a preprocessing that works readily for plenty of languages: c#, c++, c, cuda, opencl, metal, bash, shell, go, java, javascript, jsx, php, python, ruby and typescript.\n",
+    "\n",
+    "All we have to do now is to write the query!"
    ]
   },
   {
@@ -86,6 +96,7 @@
    "source": [
     "from bz2 import open as bz2_open\n",
     "from json import dumps as json_dumps, loads as json_loads\n",
+    "from pprint import pprint\n",
     "\n",
     "from utils import SUPPORTED_LANGUAGES, query_gitbase\n",
     "\n",
@@ -115,18 +126,23 @@
     "        \",\".join(\"'%s'\" % language for language in SUPPORTED_LANGUAGES),\n",
     "        \"LIMIT %d\" % limit if limit > 0 else \"\"\n",
     "    )\n",
-    "\n",
+    "    print(\"Extracting identifiers with the following gitbase query:\")\n",
+    "    print(sql)\n",
+    "    print(\"First extracted rows:\")\n",
     "    with bz2_open(identifiers_path, \"wt\", encoding=\"utf8\") as fh:\n",
+    "        shown = 0\n",
     "        for row in query_gitbase(sql):\n",
     "            if row[\"identifiers\"] is None:\n",
     "                continue\n",
-    "            # for key, value in row.items():\n",
-    "            #     row[key] = value.decode(\"utf8\", \"replace\")\n",
     "            row[\"identifiers\"] = json_loads(row[\"identifiers\"])\n",
+    "            while shown < 10:\n",
+    "                shown += 1\n",
+    "                print(\"Row %d:\" % shown)\n",
+    "                pprint(row)\n",
     "            fh.write(\"%s\\n\" % json_dumps(row))\n",
     "\n",
     "\n",
-    "extract_identifiers(run.path(Files.IDENTIFIERS))"
+    "extract_identifiers(run.path(Files.IDENTIFIERS), 1000)"
    ]
   },
   {
@@ -139,7 +155,9 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "outputs": [],
    "source": [
     "from bz2 import open as bz2_open\n",
@@ -158,12 +176,23 @@
     "            open(counter_path, \"wb\") as fh_counter:\n",
     "        identifiers_counter = Counter()\n",
     "        token_parser = TokenParser()\n",
+    "        shown = set()\n",
+    "        print(\"10 first splits:\")\n",
     "        for row_str in fh_identifiers:\n",
     "            row = json_loads(row_str)\n",
     "            identifiers = row.pop(\"identifiers\")\n",
     "            split_identifiers = []\n",
     "            for identifier in identifiers:\n",
-    "                split_identifiers.extend(token_parser(identifier))\n",
+    "                split_identifier = list(token_parser(identifier))\n",
+    "                split_identifiers.extend(split_identifier)\n",
+    "                if (len(shown) < 10\n",
+    "                        and identifier not in shown\n",
+    "                        and len(split_identifier) > 1):\n",
+    "                    shown.add(identifier)\n",
+    "                    print(\"Splitting %s into (%s)\" % (\n",
+    "                        identifier,\n",
+    "                        \", \".join(split_identifier)\n",
+    "                    ))\n",
     "            identifiers_counter.update(split_identifiers)\n",
     "            row[\"split_identifiers\"] = split_identifiers\n",
     "            fh_split_identifiers.write(\"%s\\n\" % json_dumps(row))\n",
@@ -265,6 +294,18 @@
     "The preprocessing is over! We now create the input dataset, in the VW format (see https://bigartm.readthedocs.io/en/stable/tutorials/datasets.html). We replace spaces in `file_path` to avoid creating false identifiers (VW would consider the latter parts of a path containing spaces to be identifiers)."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def build_file_id(repository_id: str, lang: str, file_path: str):\n",
+    "    return \"%s//%s//%s\" % (repository_id,\n",
+    "                           lang,\n",
+    "                           file_path.replace(\" \", \"_\"))"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -277,19 +318,26 @@
     "\n",
     "def build_vw_dataset(filtered_identifiers_path: str,\n",
     "                     vw_dataset_path: str):\n",
+    "    !rm -rf vw_dataset_path\n",
     "    with bz2_open(filtered_identifiers_path, \"rt\", encoding=\"utf8\") as fh_filtered_identifiers, \\\n",
     "            open(vw_dataset_path, \"w\") as fh_vw:\n",
+    "        shown = 0\n",
+    "        print(\"Showing first 10 lines:\")\n",
     "        for row_str in fh_filtered_identifiers:\n",
     "            counter = Counter()\n",
     "            row = json_loads(row_str)\n",
     "            counter.update(row[\"split_identifiers\"])\n",
-    "            fh_vw.write(\"%s//%s//%s %s\\n\" % (\n",
-    "                row[\"repository_id\"],\n",
-    "                row[\"lang\"],\n",
-    "                row[\"file_path\"].replace(\" \", \"_\"),\n",
+    "            line = \"%s %s\" % (\n",
+    "                build_file_id(row[\"repository_id\"],\n",
+    "                              row[\"lang\"],\n",
+    "                              row[\"file_path\"]),\n",
     "                \" \".join(\"%s:%d\" % (identifier, count)\n",
     "                     for identifier, count in counter.items())\n",
-    "            ))\n",
+    "            )\n",
+    "            if shown < 10:\n",
+    "                shown +=1\n",
+    "                print(\"Line %d: %s\" % (shown, line))\n",
+    "            fh_vw.write(\"%s\\n\" % line)\n",
     "\n",
     "\n",
     "build_vw_dataset(run.path(Files.FILTERED_IDENTIFIERS),\n",
@@ -376,7 +424,23 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This topic model is probably quite good already, but since Bigartm is a powerful library, we can improve it even further by making it sparser: documents and topics will be sharper, they will contain less words and topics and will focus on the most important ones."
+    "This topic model is probably quite good already, but since Bigartm is a powerful library, we can improve it even further by making it sparser. Sparse topics for a document means that it will have mainly a few topics with high weight and other topics with weight 0.\n",
+    "\n",
+    "Example of non-sparse documents:\n",
+    "\n",
+    "|           | Backend | Logging | Machine Learning | Data Processing |\n",
+    "|-----------|---------|---------|------------------|-----------------|\n",
+    "| server.py | 0.8     | 0.1     | 0.06             | 0.04            |\n",
+    "| utils.py  | 0.1     | 0.7     | 0.08             | 0.12            |\n",
+    "\n",
+    "Same documents but with sparse topics:\n",
+    "\n",
+    "|           | Backend | Logging | Machine Learning | Data Processing |\n",
+    "|-----------|---------|---------|------------------|-----------------|\n",
+    "| server.py | 0.85    | 0.15    | 0                | 0               |\n",
+    "| utils.py  | 0.12    | 0.88    | 0                | 0               |\n",
+    "\n",
+    "This makes understanding the documents and topics easier: they contain less non-zero entries and are more focused on the most important stuff."
    ]
   },
   {
@@ -438,9 +502,10 @@
     "from bz2 import open as bz2_open\n",
     "from json import loads as json_loads\n",
     "from pickle import dump as pickle_dump, load as pickle_load\n",
+    "from typing import Optional\n",
     "\n",
     "from numpy import ones as numpy_ones\n",
-    "from pandas import read_csv as pandas_read_csv\n",
+    "from pandas import DataFrame, read_csv as pandas_read_csv\n",
     "from pyLDAvis import prepare as pyldavis_prepare\n",
     "\n",
     "\n",
@@ -449,17 +514,41 @@
     "                          common_counter_path: str,\n",
     "                          filtered_identifiers_path: str,\n",
     "                          pyldavis_data_path: str):\n",
-    "    topics_identifiers_df = pandas_read_csv(artm_topics_identifiers_path, delimiter=\";\")\n",
-    "    files_topics_df = pandas_read_csv(artm_files_topics_path, delimiter=\";\")\n",
-    "    topic_term_dists = topics_identifiers_df.iloc[:, 2:].values.T\n",
-    "    doc_topic_dists = files_topics_df.iloc[:, 2:].values\n",
-    "    doc_topic_dists /= doc_topic_dists.sum(axis=1, keepdims=1)\n",
-    "    for i, row in enumerate(doc_topic_dists):\n",
+    "\n",
+    "    def clean_artm_df(df: DataFrame,\n",
+    "                      to_delete: str,\n",
+    "                      transpose_name: Optional[str] = None):\n",
+    "        del df[to_delete]\n",
+    "        if transpose_name is not None:\n",
+    "            df = df.T\n",
+    "            df.index.name = transpose_name\n",
+    "        return df\n",
+    "\n",
+    "    # We exchange rows and columns (transpose, .T) to have the topics as rows\n",
+    "    # and the identifiers as columns\n",
+    "    topics_identifiers_df = pandas_read_csv(\n",
+    "        artm_topics_identifiers_path,\n",
+    "        delimiter=\";\",\n",
+    "        index_col=\"token\")\n",
+    "    topics_identifiers_df = clean_artm_df(topics_identifiers_df, \"class_id\", \"topic\")\n",
+    "    print(\"Start of the topics × identifiers dataframe:\")\n",
+    "    display(topics_identifiers_df.head())\n",
+    "\n",
+    "    files_topics_df = pandas_read_csv(\n",
+    "        artm_files_topics_path,\n",
+    "        delimiter=\";\",\n",
+    "        index_col=\"title\")\n",
+    "    clean_artm_df(files_topics_df, \"id\")\n",
+    "    print(\"Start of the files × topics dataframe:\")\n",
+    "    display(files_topics_df.head())\n",
+    "\n",
+    "    files_topics_df /= files_topics_df.sum(axis=1)[:, None]\n",
+    "    filler = (numpy_ones((files_topics_df.shape[1],))\n",
+    "              / files_topics_df.shape[1])\n",
+    "    for i, row in files_topics_df.iterrows():\n",
     "        if not (0.9 < row.sum() < 1.1):\n",
-    "            doc_topic_dists[i] = (numpy_ones((doc_topic_dists.shape[1],))\n",
-    "                                  / doc_topic_dists.shape[1])\n",
-    "    doc_index = files_topics_df[\"title\"].values\n",
-    "    vocab = topics_identifiers_df[\"token\"].values\n",
+    "            files_topics_df.loc[i, :] = filler\n",
+    "    vocab = topics_identifiers_df.columns\n",
     "    with bz2_open(filtered_identifiers_path, \"rt\", encoding=\"utf8\") as fh_rj, \\\n",
     "            open(common_counter_path, \"rb\") as fh_rp:\n",
     "        common_identifiers_counter = pickle_load(fh_rp)\n",
@@ -473,11 +562,11 @@
     "                    row[\"file_path\"].replace(\" \", \"_\"))\n",
     "            ] = len(row[\"split_identifiers\"])\n",
     "        term_frequency = [common_identifiers_counter[t] for t in vocab]\n",
-    "        doc_lengths = [doc_lengths_index[doc] for doc in doc_index]\n",
+    "        doc_lengths = [doc_lengths_index[doc] for doc in files_topics_df.index]\n",
     "\n",
     "    with open(pyldavis_data_path, \"wb\") as fh:\n",
-    "        pyldavis_data = pyldavis_prepare(topic_term_dists=topic_term_dists, \n",
-    "                                         doc_topic_dists=doc_topic_dists,\n",
+    "        pyldavis_data = pyldavis_prepare(topic_term_dists=topics_identifiers_df.values, \n",
+    "                                         doc_topic_dists=files_topics_df.values,\n",
     "                                         doc_lengths=doc_lengths,\n",
     "                                         vocab=vocab,\n",
     "                                         term_frequency=term_frequency,\n",
@@ -523,7 +612,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Projects search"
+    "## Projects topics"
    ]
   },
   {
@@ -548,7 +637,6 @@
     "\n",
     "from numpy import sum as np_sum, vectorize\n",
     "from pandas import read_csv as pandas_read_csv\n",
-    "from sklearn.metrics.pairwise import cosine_similarity\n",
     "\n",
     "\n",
     "def build_projects_topics(artm_files_topics_path: str,\n",
@@ -568,6 +656,15 @@
     "                      run.path(Files.REPOS_TOPICS))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Developers topics\n",
+    "\n",
+    "Now that we've computed file and project topics, let's compute topics for developers: we'll weight the topics of each file depending on how many lines each developers wrote in it. That'll give us a topic distribution for each developer!"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -621,9 +718,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from pandas import DataFrame, read_csv as pandas_read_csv\n",
     "from pickle import dump as pickle_dump, load as pickle_load\n",
-    "from tqdm import tqdm_notebook as tqdm\n",
+    "\n",
+    "from pandas import DataFrame, read_csv as pandas_read_csv\n",
+    "from tqdm.notebook import tqdm\n",
     "\n",
     "\n",
     "def build_authors_topics(contributions_path: str,\n",
@@ -643,9 +741,11 @@
     "                                          for i in range(files_topics_df.shape[1])])\n",
     "    for file_id, counter in tqdm(contribs):\n",
     "        file_topics = files_topics_df.loc[file_id]\n",
+    "        if len(file_topics.shape) > 1:\n",
+    "            file_topics = file_topics.iloc[0, :].squeeze()\n",
     "        total = sum(counter.values())\n",
     "        for author, lines in counter.items():\n",
-    "            authors_topics_df.loc[author] += file_topics * lines / total\n",
+    "            authors_topics_df.loc[author, :] += file_topics * lines / total\n",
     "    authors_topics_df /= authors_topics_df.sum(axis=1)[:, None]\n",
     "    authors_topics_df.dropna(inplace=True)\n",
     "    with open(authors_topics_path, \"wb\") as fh:\n",
@@ -657,6 +757,19 @@
     "                     run.path(Files.AUTHORS_TOPICS))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Projects and developers search\n",
+    "\n",
+    "We have topics for projects and developers, great!\n",
+    "\n",
+    "Now we can compare them: we just have to define how far a given set of topics is from another and we're good to go. In the following cell we're using cosine similarity. It's widely used for that prupose and works quite well. Plus, `scikit-learn` has it already implemented if you don't want to write the (few) lines it requires :)\n",
+    "\n",
+    "With that defined, we can compare devs to devs, devs to projects, projects to devs and projects to projects! Let's go."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -665,6 +778,10 @@
    "source": [
     "from pickle import load as pickle_load\n",
     "\n",
+    "from numpy import sum as vectorize\n",
+    "from pandas import read_csv as pandas_read_csv\n",
+    "from sklearn.metrics.pairwise import cosine_similarity\n",
+    "\n",
     "\n",
     "def build_comparison_functions(artm_topics_identifiers_path: str,\n",
     "                               repos_topics_path: str,\n",