Shaokai/multi video (#39)

* updates

* fixed get acceleration

* several updates

* updates

* WIP

* replaced hardcoded identifier with identifier class

* lots of improvements

* fixed one more conflict

* some fixes to web app

* Fixed typos

* Added yaml dependency

* Added init in behavior analysis folder

* Added pycocotools into dependency

* reformatted test functions

* black and isort

* Added demo data test and added back video files

* added test demo test

* Added tables into dependency

* try adjusting the paths

* removed pycocotools dependency to see whether it helps

* see if pining numpy and table version in workflow helps

* edited the pytest.yml instead

* tables 3.8 in test

* Separated installation from readme

* Added some useful prints

* no need to print meta info again

* added self debug into code execution as default

* cleaned output

* Fixed a bug for keypoint_info being none

* Update README.md

* Update README.md

* Delete INSTALL.md

- this is redundant with information in main docs now

* Update README.md

* Update and rename amadesuGPT-cpu.yml to amadesuGPT.yml (#40)

* Update and rename amadesuGPT-cpu.yml to amadesuGPT.yml

* Delete conda/amadesuGPT-gpu.yml

* Delete conda/amadesuGPT-minimal.yml

* Update amadesuGPT.yml

- amadeusgpt should be a dependency

---------

Co-authored-by: Mackenzie Mathis <[email protected]>

Author: yeshaokai

.github/workflows/pytest.yml

@@ -35,6 +35,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
+        pip install pytest numpy==1.23.5 tables==3.8.0
         pip install pytest
         pip install pytest-timeout
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

.github/workflows/test_and_deploy.yaml

@@ -65,8 +65,9 @@ jobs:
           python-version: "3.10"
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
+          python -m pip install --upgrade pip          
           pip install -U setuptools setuptools_scm wheel twine build
+
       - name: Build package
         run: |
           python -m build .

README.md

@@ -22,102 +22,88 @@
 - We use large language models (LLMs) to bridge natural language and behavior analysis.
 - This work is published at **NeurIPS2023!** Read the paper, [AmadeusGPT: a natural language interface for interactive animal behavioral analysis]([https://www.google.com/search?q=amadeusgpt+openreview&sca_esv=590699485&rlz=1C5CHFA_enCH1059CH1060&ei=K1N6ZaHdKvmrur8PosOOkAo&ved=0ahUKEwjhnta83I2DAxX5le4BHaKhA6IQ4dUDCBE&uact=5&oq=amadeusgpt+openreview&gs_lp=Egxnd3Mtd2l6LXNlcnAiFWFtYWRldXNncHQgb3BlbnJldmlldzIHECEYoAEYCjIHECEYoAEYCki2HVDfAliOHHACeACQAQGYAYMDoAHaGaoBCDEuMTEuMS40uAEDyAEA-AEBwgIFECEYqwLCAggQABiABBiiBMICCBAAGIkFGKIE4gMEGAEgQYgGAQ&sclient=gws-wiz-serp#:~:text=AmadeusGPT%3A%20a%20natural,net%20%E2%80%BA%20pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/1456560769bbc38e4f8c5055048ea712-Paper-Conference.pdf)) by [Shaokai Ye](https://github.com/yeshaokai), [Jessy Lauer](https://github.com/jeylau), [Mu Zhou](https://github.com/zhoumu53), [Alexander Mathis](https://github.com/AlexEMG) & [Mackenzie W. Mathis](https://github.com/MMathisLab).
 - Like this project? Please consider giving us a star ⭐️!
-  
-## Install & Run AmadeusGPT🎻
-
-### Install with `pypi`
 
-- AmadeusGPT is a Python package hosted on `pypi`. You can create a virtual env (conda, etc, see below*) and run:
-```python
-pip install 'amadeusgpt[streamlit]'
-```
-Note that in order to access our demo video and keypoint files, we recommend to install from the source.
+## What is AmadeusGPT? 
 
-### Install from the source
+**Developed by part of the same team that brought you [DeepLabCut](https://www.mackenziemathislab.org/deeplabcut), AmadeusGPT is a natural language interface that turns natural language descriptions of behaviors into machine-executable code.** The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of animal behavior and technical machine learning knowledge, so we wanted to ease this jump.
+In short, we provide a "code-free" interface for you to analysis video data of animals. If you are a [DeepLabCut](https://www.mackenziemathislab.org/deeplabcut) user, this means you could upload your videos and .h5 keypoint files and then ask questions, such as "How much time does the mouse spend in the middle of the open field?". 
+In our original work (NeurIPS 2023) we used GPT3.5 and GPT4 as part of our agent. We continue to support the latest OpenAI models, and are continuing to actively develop this project. 
+  
+## Get started: install AmadeusGPT🎻
 
-**Make sure you edit those installation scripts to point to your own conda path before you run them**
+### [1] Set up a conda environment:
 
-#### Minimal installation
-**Recommended for:** Running AmadeusGPT without GPUs. This setup is lightweight and is limited to processing movie files and keypoint outputs (.h5) from DeepLabCut.
+Conda is an easy-to-use Python interface that supports launching [Jupyter Notebooks](https://jupyter.org/). If you are completely new to this, we recommend checking out the [docs here for getting conda installed](https://deeplabcut.github.io/DeepLabCut/docs/beginner-guides/beginners-guide.html#beginner-user-guide). Otherwise, proceed to use one of [our supplied conda files](https://github.com/AdaptiveMotorControlLab/AmadeusGPT/tree/main/conda). As you will see we have minimal dependencies to get started, and [here is a simple step-by-step guide](https://deeplabcut.github.io/DeepLabCut/docs/installation.html#step-2-build-an-env-using-our-conda-file) you can reference for setting it up (or see [BONUS](README.md#bonus---customized-your-conda-env) below). Here is the quick start command:
 
 ```bash
-# Install the minimal environment
-bash install_minimal.sh
-
-# Activate the conda environment
-conda activate amadeusgpt-minimal
+conda env create -f amadeusGPT.yml
 ```
+To note, some modules AmadeusGPT can use benefit from GPU support, therefore we recommend also having an NVIDIA GPU available and installing CUDA. 
 
-#### GPU installation
-**Recommended for:** Users on Linux with GPUs. Support for Windows and MacOS will be added in the future.
+### [2] You will need an openAI key:
 
-```bash
-# Install the gpu environment
-bash install_gpu.sh
+**Why OpenAI API Key is needed** AmadeusGPT relies on API calls of OpenAI (we will add more LLM options in the future) for language understanding and code writing. Sign up for a [openAI API key](https://platform.openai.com/account/api-keys) [here](https://platform.openai.com/account/api-keys).
+
+Then, you can add this into your environment by passing the following in the terminal after you launched your conda env:
 
-# Activate the conda environment
-conda activate amadeusgpt-gpu
+```bash
+export OPENAI_API_KEY='your API key' 
 ```
 
-#### CPU installation
-**Recommended for:** MacOS / Linux users working with very small video files.
+Or inside a python script or Jupyter Notebook, add this if you did not pass at the terminal stage:
 
-```bash
-# Install the cpu environment
-bash install_cpu.sh
 
-# Activate the conda environment
-conda activate amadeusgpt-cpu
+```python
+import os
+os.environ["OPENAI_API_KEY"] = 'your api key' 
 ```
 
+### [3] 🪄 That's it! Now you have AmadeusGPT installed! 
 
-### Setup OpenAI API Key to use AmadeusGPT
-
-**Why OpenAI API Key is needed** AmadeusGPT relies on API calls of OpenAI (we will add more options in the future) for language understanding and code writing.
+See below on how to get started!
 
-You can either add this into your environment by following:
 
-```bash
-export OPENAI_API_KEY='your API key' 
-```
+## Get started: run AmadeusGPT🎻
 
-Or inside your python script or jupyter notebook, add this line in the beginning of the file
+We provide a StreamLit App, or you can use AmadeusGPT in any python interface, such as Jupyter notebooks. For this we suggest getting started from our demos:
 
+### Try AmadeusGPT with an example Jupyter Notebook
+You can git clone (or download) this repo to grab a copy and go. We provide example notebooks [here](notebook)!
 
-```python
-import os
-os.environ["OPENAI_API_KEY"] = 'your api key' 
-```
+### Here are a few demos that could fuel your own work, so please check them out!
 
-- Please note that you need an [openAI API key](https://platform.openai.com/account/api-keys), which you can easily create [here](https://platform.openai.com/account/api-keys).
+1) [Draw a region of interest (ROI) and ask, "when is the animal in the ROI?"](notebook/EPM_demo.ipynb)
+2) [Use a DeepLabCut SuperAnimal pose model to do video inference](notebook/custom_mouse_demo.ipynb) - (make sure you use a GPU if you don't have corresponding DeepLabCut keypoint files already!
+3) [Write you own integration modules and use them](notebook/Horse_demo.ipynb). Bonus: [source code](amadeusgpt/integration_modules). Make sure you delete the cached modules_embedding.pickle if you add new modules!
+4) [Multi-Animal social interactions](notebook/MABe_demo.ipynb)
+5) [Reuse the task program generated by LLM and run it on different videos](notebook/MABe_demo.ipynb)
+7) You can ask one query across multiple videos. Put your keypoint files and video files (pairs) in the same folder and specify the `data_folder` as shown in this [Demo](notebook/custom_mouse_video.ipynb). Make sure your video file and keypoint file follows the normal DeepLabCut convention, i.e., `prefix.mp4` `prefix*.h5`.
 
 
 
-### Try AmadeusGPT with local web app
+### Try AmadeusGPT with a local WebApp
+- You will need to git clone this repo and have a copy locally. Then in your env run `pip install 'amadeusGPT[streamlit]'`
+- Then you can open the terminal and within the directory run:
 ```python
 make app
 ```
 
-### Do I need to provide keypoint files?
-- If you only provide the raw video file, we use SuperAnimal models [SuperAnimal models](https://www.nature.com/articles/s41467-024-48792-2) to predict your video. This is only supported with cpu or gpu installation. While we highly recommend gpu installation, we are working on faster, light-weight superanimal models to work on your CPU.
-- If you already have keypoint file corresponding to the video file, look up how we set-up the config file in the notebooks.  Right now we only support keypoint output from DeepLabCut. Other keypoint formats can be added upon feature requests.
-
 
-### Try AmadeusGPT with our example notebooks
-We provide example notebooks at [Notebooks](notebook)
-
-### Notebook as use-case demo
-
-1) Draw ROI and ask when is the animal in the ROI.  [Demo](notebook/EPM_demo.ipynb)
-2) Obtain the binary mask for retrieved masks (for further sampling such as neural data pairing) [Demo](notebook/EPM_demo.ipynb)
-3) Use SuperAnimal video inference (make sure you use gpu or cpu installation) if you don't have corresponding DeepLabCut keypoint file [Demo](notebook/custom_mouse_demo.ipynb)
-4) Write you own integration modules and use them [Demo](notebook/Horse_demo.ipynb) [Source code](amadeusgpt/integration_modules). Make sure you delete the cached modules_embedding.pickle if you add new modules
-5) Multi-Animal social interaction.  [Demo](notebook/MABe_demo.ipynb)
-6) Reuse the task program generated by LLM and run it on different videos [Demo](notebook/MABe_demo.ipynb)
+## [BONUS - customized your conda env] 
+If you want to set up your own env, 
 
+```bash
+conda create -n amadesuGPT python=3.10
+```
+the key dependencies that need installed are:
+```python
+pip install notebook
+conda install hdf5
+pip install amadeusgpt
+```
 ## Citation
 
-  If you use ideas or code from this project in your work, please cite us  using the following BibTeX entry. 🙏
+  If you use ideas or code from this project in your work, please cite us using the following BibTeX entry. 🙏
 
 ```
 @article{ye2023amadeusGPT,
@@ -133,10 +119,17 @@ We provide example notebooks at [Notebooks](notebook)
 ## License 
 
 AmadeusGPT is license under the Apache-2.0 license.
-  -  🚨 Please note several key dependencies have their own licensing. Please carefully check the license information for [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut) (LGPL-3.0 license), [SAM](https://github.com/facebookresearch/segment-anything) (Apache-2.0 license), [CEBRA](https://github.com/AdaptiveMotorControlLab/CEBRA) (Non-Commercial), etc.
+  -  🚨 Please note several key dependencies have their own licensing. Please carefully check the license information for [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut) (LGPL-3.0 license), [SAM](https://github.com/facebookresearch/segment-anything) (Apache-2.0 license), etc.
+
+## FAQ:
 
+### Do I need to provide keypoint files or is video-only enough to get starte?
+- If you only provide a video file, we use SuperAnimal models [SuperAnimal models](https://www.nature.com/articles/s41467-024-48792-2) to predict which animal is in your video. While we highly recommend GPU installation, we are working on faster, light-weight SuperAnimal models to work on your CPU.
+- If you already have keypoint file corresponding to the video file, look how we set-up the config file in the Notebooks.  Right now we only support keypoint output from DeepLabCut.
 
 ## News
+- July 2024 0.1.0 is released! This is a major code update ...
+- June 2024 as part of the CZI EOSS, The Kavli Foundation now supports this work! ✨
 - 🤩 Dec 2023, code released!
 - 🔥 Our work was accepted to NeuRIPS2023
 - 🧙‍♀️ Open-source code coming in the fall of 2023

amadeusgpt/__init__.py

@@ -7,7 +7,6 @@
 
 from matplotlib import pyplot as plt
 
-from amadeusgpt.implementation import AnimalBehaviorAnalysis
 from amadeusgpt.integration_modules import *
 from amadeusgpt.main import AMADEUS
 from amadeusgpt.version import VERSION, __version__

amadeusgpt/analysis_objects/analysis_factory.py

@@ -1,9 +1,11 @@
-from amadeusgpt.analysis_objects.object import Object
-from numpy import ndarray
-from scipy.spatial import ConvexHull
-from typing import List, Dict, Any
+from typing import Any, Dict, List
+
 import matplotlib.path as mpath
 import numpy as np
+from numpy import ndarray
+from scipy.spatial import ConvexHull
+
+from amadeusgpt.analysis_objects.object import Object
 
 
 class Animal(Object):
@@ -92,11 +94,16 @@ def get_path(self, ind):
         codes, verts = zip(*path_data)
         return mpath.Path(verts, codes)
 
-    def get_keypoints(self) -> ndarray:
+    def get_keypoints(self, average_keypoints=False) -> ndarray:
         # the shape should be (n_frames, n_keypoints, 2)
         # extending to 3D?
-        assert len(self.keypoints.shape) == 3, f"keypoints shape is {self.keypoints.shape}"
-        return self.keypoints
+        assert (
+            len(self.keypoints.shape) == 3
+        ), f"keypoints shape is {self.keypoints.shape}"
+        if not average_keypoints:
+            return self.keypoints
+        else:
+            return np.nanmean(self.keypoints, axis=1)
 
     def get_center(self):
         """
@@ -166,13 +173,13 @@ def get_acceleration(self) -> ndarray:
         )
         assert len(accelerations.shape) == 3
         return accelerations
-    
+
     def get_acceleration_mag(self) -> ndarray:
         """
-        Returns the magnitude of the acceleration vector    
+        Returns the magnitude of the acceleration vector
         """
         accelerations = self.get_acceleration()
-        acceleration_mag = np.linalg.norm(accelerations, axis=-1)  
+        acceleration_mag = np.linalg.norm(accelerations, axis=-1)
         acceleration_mag = np.expand_dims(acceleration_mag, axis=-1)
         assert len(acceleration_mag.shape) == 3
         return acceleration_mag
@@ -226,23 +233,24 @@ def calc_head_cs(self):
         return mouse_cs
 
 
-
 if __name__ == "__main__":
     # unit testing the shape of kinematics data
     # acceleration, acceleration_mag, velocity, speed, and keypoints
 
     from amadeusgpt.config import Config
     from amadeusgpt.main import AMADEUS
-    config = Config("/Users/shaokaiye/AmadeusGPT-dev/amadeusgpt/configs/MausHaus_template.yaml")
+
+    config = Config(
+        "/Users/shaokaiye/AmadeusGPT-dev/amadeusgpt/configs/MausHaus_template.yaml"
+    )
     amadeus = AMADEUS(config)
     analysis = amadeus.get_analysis()
     # get an instance of animal
     animal = analysis.animal_manager.get_animals()[0]
 
-    print ("velocity shape", animal.get_velocity().shape)
-    print ("speed shape", animal.get_speed().shape)
-    print ("acceleration shape", animal.get_acceleration().shape)
-    print ("acceleration_mag shape", animal.get_acceleration_mag().shape)
+    print("velocity shape", animal.get_velocity().shape)
+    print("speed shape", animal.get_speed().shape)
+    print("acceleration shape", animal.get_acceleration().shape)
+    print("acceleration_mag shape", animal.get_acceleration_mag().shape)
 
     print(animal.query_states("acceleration_mag").shape)
-