Merge pull request #113 from ContextLab/revision-3

Nature Communications revision 3

Author: paxtonfitzpatrick

README.md

@@ -17,14 +17,16 @@ analyses via [Docker](https://www.docker.com/).
 
 ## Table of Contents
 
-- [Repo Organization](#repo-organization)
-- [Installing Docker](#installing-docker)
-- [Running the Analyses](#running-the-analyses)
-  - [Option 1: `launch_notebooks.sh`](#option-1-launch_notebookssh)
-  - [Option 2: Manual setup](#option-2-manual-setup)
-  - [Additional info](#additional-info)
-- [Running the Experiment](#running-the-experiment)
-- [Useful docs](#useful-docs)
+- [Text embedding models yield high-resolution insights into conceptual knowledge from short multiple-choice quizzes](#text-embedding-models-yield-high-resolution-insights-into-conceptual-knowledge-from-short-multiple-choice-quizzes)
+  - [Table of Contents](#table-of-contents)
+  - [Repo Organization](#repo-organization)
+  - [Installing Docker](#installing-docker)
+  - [Running the Analyses](#running-the-analyses)
+    - [Option 1: `launch_notebooks.sh`](#option-1-launch_notebookssh)
+    - [Option 2: Manual setup](#option-2-manual-setup)
+    - [Additional information](#additional-information)
+  - [Running the Experiment](#running-the-experiment)
+  - [Useful additional documentation](#useful-additional-documentation)
 
 
 ## Repo Organization
@@ -35,7 +37,7 @@ The repository is organized as follows:
 .
 ├── code : all analysis code used in the paper
 │   ├─ notebooks : Jupyter notebooks for running analyses
-│   └─── khan_helpers : Python package with helper code for analyses
+│   └─ khan_helpers : Python package with helper code for analyses
 ├── data : all data analyzed in the paper
 │   ├── embeddings : 2D UMAP embeddings for knowledge maps
 │   ├── models : trained models
@@ -91,7 +93,7 @@ The script will:
 2. Build the image from [`Dockerfile-analyses`](docker/Dockerfile-analyses), if
    it doesn't already exist
 3. Create and run a container from the image, if one doesn't already exist
-4. Launch a [Jupyter notebook](https://jupyter.org/) server inside the 
+4. Launch a [Jupyter notebook](https://jupyter.org/) server inside the
    container, if one isn't already running
 5. Open the notebook app in your default browser
 6. Attach stdout to the notebook server logs in the container
@@ -129,10 +131,10 @@ analysis environment following the steps below
 
 ### Option 2: Manual setup
 
-1. After [installing Docker](#installing-docker), launch the desktop app or 
+1. After [installing Docker](#installing-docker), launch the desktop app or
    start the daemon from the command line.
-2. _From the repository's root directory_, build the "`khan`" image from the 
-   [Dockerfile-analyses](docker/Dockerfile-analyses) file in the 
+2. _From the repository's root directory_, build the "`khan`" image from the
+   [Dockerfile-analyses](docker/Dockerfile-analyses) file in the
    [docker](docker) directory:
 
    ```sh
@@ -145,17 +147,17 @@ analysis environment following the steps below
    docker run -it -p 8888:8888 --name Khan -v $PWD:/mnt khan
    ```
 
-   The command above binds port 8888 in the continer to port 8888 on the host so 
-   we can access the Jupyter notebook server from a web browser, and bind-mounts 
-   the repository to the container's `/mnt` directory so we can read and write 
+   The command above binds port 8888 in the continer to port 8888 on the host so
+   we can access the Jupyter notebook server from a web browser, and bind-mounts
+   the repository to the container's `/mnt` directory so we can read and write
    files from inside it.
 
-4. The notebook server will launch automatically when the container is run. Copy 
-   and paste the 3rd link that appears (the one starting with 
+4. The notebook server will launch automatically when the container is run. Copy
+   and paste the 3rd link that appears (the one starting with
    `http://127.0.0.1:8888`) into a web browser to access the notebook app.
 
-5. You can then open any notebook in [`code/notebooks/`](code/notebooks) and 
-   run the code inside it. When finished, return to the terminal and press 
+5. You can then open any notebook in [`code/notebooks/`](code/notebooks) and
+   run the code inside it. When finished, return to the terminal and press
    **Control+c** to stop the notebook server and exit the container.
 
 6. To launch the container and notebooks any time after this initial setup, run:
@@ -167,38 +169,38 @@ analysis environment following the steps below
 
 ### Additional information
 
-- You can launch an interactive `bash` shell inside the container to explore or 
-  modify its contents with `docker exec -it Khan bash`. To exit the container, 
+- You can launch an interactive `bash` shell inside the container to explore or
+  modify its contents with `docker exec -it Khan bash`. To exit the container,
   either press **Control+d** or type `exit`. **Note** that:
-  - if the container isn't already running, you must start it first with 
+  - if the container isn't already running, you must start it first with
     `docker start Khan`
-  - When you enter the container this way (rather than with `docker attach` or 
-    the [`launch_notebooks.sh`](launch_notebooks.sh) script), the container 
-    isn't automatically stopped when you exit it. To stop the container after 
+  - When you enter the container this way (rather than with `docker attach` or
+    the [`launch_notebooks.sh`](launch_notebooks.sh) script), the container
+    isn't automatically stopped when you exit it. To stop the container after
     exiting, use `docker stop Khan`
-- You can get the URL of the running notebook server from inside the container 
+- You can get the URL of the running notebook server from inside the container
   with `jupyter notebook list`
-- The container pre-installs some nifty 
-  [extensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/index.html) 
-  for customizing the Jupyter Notebook interface. If you want to enable any of 
+- The container pre-installs some nifty
+  [extensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/index.html)
+  for customizing the Jupyter Notebook interface. If you want to enable any of
   them:
   - Open the notebook application in a browser and click on the "**Nbextensions**" tab.  (To
     launch the notebook application, start the notebook server as described above and visit
     the server's address in your web browser.)
-  - Uncheck the "_disable configuration for nbextensions without explicit 
-    compatibility ..._" box (nearly all of them _are_ compatible, we're just 
+  - Uncheck the "_disable configuration for nbextensions without explicit
+    compatibility ..._" box (nearly all of them _are_ compatible, we're just
     using a newer version of Jupyter notebooks).
-  - Click on any of the listed extensions to see a description and further 
+  - Click on any of the listed extensions to see a description and further
     options, and check the box next to its name to enable it.
   - Refresh any running notebooks for changes to take effect.
 
 
 ## Running the Experiment
 
-1. After [installing Docker](#installing-docker), launch the desktop app or 
+1. After [installing Docker](#installing-docker), launch the desktop app or
    start the daemon from the command line.
-2. _From the repository's root directory_, build the "`khan-exp`" image from the 
-   [Dockerfile-experiment](docker/Dockerfile-experiment) file in the 
+2. _From the repository's root directory_, build the "`khan-exp`" image from the
+   [Dockerfile-experiment](docker/Dockerfile-experiment) file in the
    [docker](docker) directory:
 
    ```sh
@@ -211,26 +213,26 @@ analysis environment following the steps below
    docker run -it -p 22363:22363 -v "$PWD/exp:/exp" --name Khan-exp khan-exp
    ```
 
-   The command above bind-mounts the container to the repository's [`exp/`](exp) 
-   directory so the psiTurk server can read and run the experiment code, and 
-   binds port 22363 between the container and host so the server can be accessed 
+   The command above bind-mounts the container to the repository's [`exp/`](exp)
+   directory so the psiTurk server can read and run the experiment code, and
+   binds port 22363 between the container and host so the server can be accessed
    from a web browser.
 
-   **Note**: the port published by the container must match the port listed in 
+   **Note**: the port published by the container must match the port listed in
    [`exp/config.txt`](exp/config.txt).
 
-4. Your shell prompt (`$PS1`) should now start with `root@`, indicating that 
-   you're now running a `bash` shell from _inside_ the container. To start the 
+4. Your shell prompt (`$PS1`) should now start with `root@`, indicating that
+   you're now running a `bash` shell from _inside_ the container. To start the
    psiTurk experiment server, run:
 
    ```sh
    psiturk server on
    ```
 
-   When you see "_`Now serving on http://0.0.0.0:22363`_," the experiment server 
-   is ready. Starting the server for the first time will also create 
-   `exp/server.log`, a logfile for the experiment server, and 
-   `exp/efficient-learning-khan.db`, a SQLite database to hold raw experiment 
+   When you see "_`Now serving on http://0.0.0.0:22363`_," the experiment server
+   is ready. Starting the server for the first time will also create
+   `exp/server.log`, a logfile for the experiment server, and
+   `exp/efficient-learning-khan.db`, a SQLite database to hold raw experiment
    data.
 
 5. Generate a link to the experiment in "debug mode":
@@ -239,18 +241,18 @@ analysis environment following the steps below
    psiturk debug -p
    ```
 
-   This will output a URL in the format 
-   `http://0.0.0.0:22363/ad?assignmentId=debug<XXXXXX>&hitId=debug<YYYYYY>&workerId=debug<ZZZZZZ>&mode=debug`, 
-   where `<XXXXXX>` and `<ZZZZZZ>` will form a unique identifier for the run 
-   (i.e., a participant's unique ID). In debug mode, the experiment will behave 
-   normally and data will still be saved properly, but psiTurk will not try to 
-   connect to [Amazon Mechanical Turk](https://www.mturk.com/)'s servers. This 
-   is useful because it enables the experiment to be run locally without the 
+   This will output a URL in the format
+   `http://0.0.0.0:22363/ad?assignmentId=debug<XXXXXX>&hitId=debug<YYYYYY>&workerId=debug<ZZZZZZ>&mode=debug`,
+   where `<XXXXXX>` and `<ZZZZZZ>` will form a unique identifier for the run
+   (i.e., a participant's unique ID). In debug mode, the experiment will behave
+   normally and data will still be saved properly, but psiTurk will not try to
+   connect to [Amazon Mechanical Turk](https://www.mturk.com/)'s servers. This
+   is useful because it enables the experiment to be run locally without the
    user having to create AWS & MTurk accounts, supply access keys, etc.
 
-6. Copy and paste the URL into a web browser, and follow the on-screen 
-   instructions to progress through the experiment. **Note**: the experiment 
-   will not work in Google Chrome. Recommended browsers include Safari and 
+6. Copy and paste the URL into a web browser, and follow the on-screen
+   instructions to progress through the experiment. **Note**: the experiment
+   will not work in Google Chrome. Recommended browsers include Safari and
    Firefox.
 
 7. When finished, return to the terminal and shut down the experiment server:

code/notebooks/README.md

@@ -20,6 +20,6 @@ analyses are described or figures appear in the paper.
 - `2_topic-word-distributions-fig.ipynb` – code to generate Supplementary Figure 1
 - `3_topic-weight-fig.ipynb` – code to generate Supplementary Figure 2 and compute related stats
 - `4_lecture-question-correlation-peaks.ipynb` – code to generate Supplementary Figures 3 & 4
-- `5_model-comparison-word-overlap.ipynb` – code to generate Supplementary Figure 5
-- `6_model-comparison-BERT.ipynb` – code to generate Supplementary Figure 6
-- `7_individual-maps.ipynb` – code to generate Supplementary Figures 7–11
+- `5_individual-maps.ipynb` – code to generate Supplementary Figures 5–9
+- `6_model-comparison-BERT.ipynb` – code to generate Supplementary Figure 10
+- `7_model-comparison-word-overlap.ipynb` – code to generate Supplementary Figure 11