tbp.tbs_sensorimotor_intelligence

This repository contains code to replicate experiments from our paper, "Thousand Brains Systems: Sensorimotor Intelligence for Rapid, Robust Learning and Inference".

Experiments make use of either the Monty framework, or Pytorch in the case of deep learning models. Instructions for environment setup and experiment execution can be found in the README files within monty/ (here) and pytorch/ (here). Information on licensing can also be found in the respective directories.

The basic structure of this repository is as follows:

• monty/: Contains all the code needed to run experiments using tbp.monty, the open-source, sensorimotor learning system presented in our paper. This directory includes a README file with instructions for setting up a Python environment and executing the experiments.
• pytorch/: Contains the vision transformer (ViT) model used in the paper and the experiments performed with it. Like with monty/, this directory contains a README file with instructions for setting up an appropriate Python environment and running the ViT-based experiments.
• scripts/: Contains analysis and plotting scripts used to generate figures.

Filesystem Setup

Experiments and plotting scripts access three main directories -- DMC_ROOT_DIR, DMC_ANALYSIS_DIR, and MONTY_DATA -- each of which can be modified by setting the corresponding environment variable. (Note that DMC is shorthand for Demonstrating Monty's Capabilities, the name of this project we have been using internally.)

• DMC_ROOT_DIR holds the output generated by running experiments. It defaults to ~/tbp/results/dmc/, and it has the following subdirectories:

  • pretrained_models/: Models generated by running Monty's pretraining experiments. Required to run evaluation experiments using tbp.monty.
  • results/: Output generated by running evaluation experiments. Required to run plotting scripts.
  • view_finder_images/: Intermediate data generated by Monty that was used to train and evaluate the ViT model. Required to run pytorch/ViT experiments but not Monty experiments or plotting scripts.
  • visualizations/: Output from a set of auxilliary experiments used to capture targeted information for figures visualizations only. Required to run plotting scripts.

  Any of the above subdirectories may either be downloaded or recomputed locally.

• DMC_ANALYSIS_DIR holds the output generated by plotting scripts, and it defaults to ~/tbp/results/dmc_analysis. Its subdirectories are matched to plotting scripts (e.g., scripts/fig4.py stores its output to DMC_ANALYSIS_DIR/fig4/).

• MONTY_DATA is used only by tbp.monty, and it contains the YCB object dataset. It defaults to ~/tbp/data.

Downloading Datasets

Any or all of the datasets used in this paper can be downloaded using download.py, and the datasets you'll need will depend on your specific use case. For example, if you intend to reproduce all Monty-based results from scratch, including pretraining the models, you'd only need the YCB object dataset. Using download.py, you'd run

$ python download.py ycb

On the other hand, if you only want to regenerate figures using precomputed data without running any experiments, run

$ python download.py pretrained_models results visualizations

The following table should help you decide which datasets you'll need based on your intended use.

	ycb	pretrained_models	results	view_finder_images	visualizations
Run Monty Pretraining Experiments	✓
Run Monty Evaluation Experiments	✓	✓
Run Plotting Scripts		✓	✓		✓
Run ViT Experiments				✓

Not that we don't provide the pretrained models for the ViT due to their size. To obtain those, follow the instructions in the pytorch README. The scripts provided there will pretrain a ViT from scratch and fine-tune a model downloaded from HuggingFace.

Running Experiments and/or Reproducing Figures

To run Monty experiments, follow instructions in the monty README. There you will create tbs_sensorimotor_intelligence, the Python environment used to run experiments using the Monty framework.

To run experiments using the ViT model, follow the instructions in the pytorch README. Experiments is this directory are run with a separate Python environment (vit).

Once you've generated or downloaded the experiment results, you can reproduce all figures and tables individually by running the scripts located in the scripts/ directory. Each script contains a module-level docstring with detailed descriptions about required datasets and which figure panels each function generates. All scripts, including those used to analyze ViT results, should be run with tbs_sensorimotor_intelligence (the Python environment set up from for running Monty experiments).