Merge pull request #273 from keenon/main

Update prod branch with latest changes

Author: AlbertoCasasOrtiz

.gitignore

@@ -7,4 +7,7 @@ biomechanics_net.egg-info/
 .mypy_cache/
 __pycache__
 test_dataset.db
-TODO
+TODO
+.DS_Store
+/test_data
+.idea/

LICENSE.txt

@@ -0,0 +1,49 @@
+###########################
+License:
+GPL v3.0 - https://opensource.org/license/gpl-3-0/
+with the following modification:
+###########################
+
+Any software licensed under the terms of the GNU General Public License (GPL) 
+also requires that any data generated or processed using the software must be 
+publicly shared in accordance with the Data-Sharing Agreement between 
+Stanford University and you. By downloading, installing, or using the software, 
+you agree to be bound by the terms of the GPL license and the Data-Sharing Agreement.
+
+###########################
+Data-Sharing Agreement
+###########################
+
+This Data-Sharing Agreement ("Agreement") is entered into by and between Stanford University ("Provider") and you ("Recipient") as a condition of receiving a license to use this software.
+
+WHEREAS, Provider has developed and licensed certain software ("Software") that can be used to process motion capture data; and
+WHEREAS, Recipient desires to use the Software to process motion capture data, and Provider has agreed to grant Recipient a license to use the Software on the condition that any data generated or processed using the Software is shared publicly;
+
+NOW, THEREFORE, the parties agree as follows:
+
+Definitions.
+(a) "Data" means any motion capture data generated or processed using the Software.
+(b) "Publicly Share" means to make Data available to the public, free of charge and without restriction, through a publicly accessible platform or repository.
+
+License to Use Software.
+(a) Provider hereby grants Recipient a non-exclusive, non-transferable, revocable license to use the Software for the sole purpose of generating and processing Data.
+(b) Recipient agrees to use the Software only in accordance with the terms of the GNU General Public License (GPL).
+
+Data-Sharing Obligations.
+(a) Recipient agrees to Publicly Share all Data generated or processed using the Software.
+(b) Recipient agrees to provide attribution to Provider and the Software in any publication, presentation, or other public use of the Data.
+(c) Recipient agrees to provide Provider with a copy of all Data generated or processed using the Software.
+
+Termination.
+(a) Either party may terminate this Agreement upon written notice if the other party breaches any of its obligations under this Agreement.
+(b) Termination of this Agreement shall not affect Recipient's obligations to Publicly Share any Data generated or processed using the Software prior to termination.
+
+Limitation of Liability.
+(a) Provider makes no representations or warranties with respect to the Software or Data, and disclaims all implied warranties, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non-infringement.
+(b) Provider shall not be liable for any direct, indirect, special, incidental, or consequential damages arising out of or in connection with the use of the Software or Data.
+
+Miscellaneous.
+(a) This Agreement constitutes the entire agreement between the parties and supersedes all prior or contemporaneous agreements or understandings, whether written or oral.
+(b) This Agreement shall be governed by and construed in accordance with the laws of [YOUR STATE/COUNTRY], without giving effect to any conflict of law principles.
+(c) This Agreement may not be amended or modified except in writing signed by both parties.
+(d) This Agreement shall be binding upon and inure to the benefit of the parties and their respective successors and assigns.

README.md

@@ -1,33 +1,50 @@
-### BiomechanicsNet
+### AddBiomechanics
 
-This is an open effort to assemble a large dataset of human motion, recorded from several modalities (kinematics, GRF, EMG, and IMU). We compose this dataset out of dozens of heterogeneous motion capture datasets published in both biomechanics and computer graphics. While there are dozens of these datasets, few contain everything we’d like. For example, [Mahmood et. al](https://amass.is.tue.mpg.de/) is very large and covers many motion types, but contains only skeleton kinematics (no GRF, EMG, or accelerometers). [Carmago et. al](http://www.epic.gatech.edu/opensource-biomechanics-camargo-et-al/) contains EMG, motion capture, and GRF data for motions on uneven surfaces, but only for the lower-half of the body. [Lencioni et. al](https://springernature.figshare.com/collections/Human_kinematic_kinetic_and_EMG_data_during_level_walking_toe_heel-walking_stairs_ascending_descending/4494755) contains EMG, motion capture, and GRF data for the whole body, but only for walking up and down stairs. There are dozens more datasets, each with its own idiosyncrasies.
+[![DOI](https://zenodo.org/badge/398424759.svg)](https://zenodo.org/badge/latestdoi/398424759)
 
-Our goal in this project is to provide a standard format for modality-sparse human motion data, and loaders for as many datasets as we can. We standardize on the [Rajagopal Human Body Model](https://simtk.org/projects/full_body), as implemented in the [Nimble Physics Engine](https://nimblephysics.org).
+This is an open effort to assemble a large dataset of human motion. We're hoping to faccilitate this by providing easy-to-use tools that can automatically process motion capture data, and prepare it for biomechanical analysis. We're also working to provide large aggregate datasets in standard formats, along with tools to easily handle the data, at some point in the near future.
 
-Licenses-permitting, we plan to make pre-translated aggregate datasets available for public download.
+### Getting Set Up (for Stanford Developers)
 
-## Getting Set Up For Development (frontend)
+*A note for non-Stanford devs: these instructions probably won't help you!* We share the AddBiomechanics source code so that researchers can fully understand the methods. We highly encourage you to use the web application rather than building the code from source. Note that we are a small team and are not able to support individuals wishing to build from source. You're welcome to try, but it's probably going to be harder than you hope, and we're sorry about that. Part of the complexity here is that the cloud application is built to interface directly with a web of different AWS resources, each of which has its own (currently undocumented) IAM setup, which are provisioned and continually maintained by our team for the public instance of AddBiomechanics. If you are trying to run your own independent instance to avoid sharing data, even if we gave you the permissions files referenced in these instructions, your code would by default talk to our AWS resources, and effectively just join our cluster. If you want it to talk to your own resources, we cannot offer support debugging your setup to get everything to work.
 
-1. Install the Amplify CLI `npm install -g @aws-amplify/cli` (may require `sudo`, depending on your setup)
-2. From inside the `frontend` folder, run `amplify configure`, and follow the instructions to create a new IAM user for your computer (in the 'us-west-2' region)
-3. From inside the `frontend` folder, run `amplify init`
+## Getting Set Up for Development (frontend)
+
+1. Download the [aws-exports-dev.js](https://drive.google.com/file/d/1IBr3Fm-8rYeGudyWLvIEGPkdzdpR0I90/view?usp=sharing) file, rename it `aws-exports.js` and put it into the `frontend/src` folder.
+2. Run `yarn start` to launch the app!
+
+## Notes (frontend)
+
+Note: the above instructions will cause your local frontend to target the dev servers, if you would rather interact with production servers, download the [aws-exports-prod.js](https://drive.google.com/file/d/1VZVgHHwSP-xmJW-qZeQ6U92FYWoU36aP/view?usp=sharing) file, rename it `aws-exports.js` and put it into the `frontend/src` folder.
+
+Because the app is designed to be served as a static single page application (see the wiki for details) running it locally with the appropriate `aws-exports.js` will behave exactly the same as viewing it from [dev.addbiomechanics.org](https://dev.addbiomechanics.org) (dev servers) or [app.addbiomechanics.org](https://app.addbiomechanics.org) (prod servers)
+
+## Getting Set Up For Deployment (frontend)
+
+1. Log in with the AddBiomechanics AWS root account on your local `aws` CLI.
+2. Install the Amplify CLI `npm install -g @aws-amplify/cli` (may require `sudo`, depending on your setup)
+3. From inside the `frontend` folder, run `amplify configure`, and follow the instructions to create a new IAM user for your computer (in the 'us-west-2' region)
+4. From inside the `frontend` folder, run `amplify init`
     a. When asked "Do you want to use an existing environment?" say YES
     b. Choose the environment "dev"
     c. Choose anything you like for your default editor
     d. Select the authentication method "AWS profile", and select the profile you created in step 2
-4. Run `yarn start` to launch the app!
-## Getting Set Up For Development (server)
+5. Run `yarn start` to launch the app!
+## Getting Set Up For Development (server processing algorithm)
+
+The core algorithm for processing data exists in `server/engine/engine.py`. To test changes `engine.py`:
 
-1. Download (credentials)[https://drive.google.com/file/d/1okCCdvqaZh20gc4TG152o7yJV9_vnBtf/view?usp=sharing] into `.devcontainer/.aws/credentials` and `server/.aws/credentials`.
-2. Download (server_credentials.csv)[https://drive.google.com/file/d/1e1GrwpOm0viZhNGkw_lDNPa_cfYhJ3r3/view?usp=sharing] into `.devcontainer/server_credentials.csv` and `server/server_credentials.csv`.
-3. Open this project in VSCode, and then use Ctrl+Shift+P and get to the command "Remote-Containers: Open Folder in Container...". Re-open this folder in a Docker container.
-4. Using a VSCode Terminal, navigate to `frontend` and execute `yarn start` to begin serving a live frontend
+1. Run `pip3 install -r /engine/requirements.txt`
+2. Download the [`test_engine.sh` script](https://drive.google.com/file/d/1n-9KSv-wZevuVNwShb1Ur36MRAZlnNhv/view?usp=share_link), place it in this directory
+3. Download the [test_data/ folder](https://drive.google.com/drive/folders/1jGfgM1m13ksqLZByKUEoUwsy22OVtEza?usp=share_link) (ask Keenon for access to this), place it in this directory
+4. Run `./test_engine.sh` to test out your changes to `engine.py` on existing data. Change the line `TEST_NAME="opencap_test"` to different run against other folder names you find in `test_data/` (careful, don't include the `_original` part or you'll overwrite your input data by accident)
 
 ## Hosting a Processing Server
 
 1. Got into the `server` folder
-2. Run `docker build -f Dockerfile.dev .` (to run a dev server) or `docker build -f Dockerfile.prod .` (to run a prod server) to build the Docker container to run the server. It's important that you rebuild the Docker container each time you boot a new server, since that sets it up with its own PubSub connection.
-3. Run the docker container you just built! That's your server. Leave it running as a process.
+2. Download the `server_credentials.csv` file, which Keenon can give you a link to
+3. Run `docker build -f Dockerfile.dev .` (to run a dev server) or `docker build -f Dockerfile.prod .` (to run a prod server) to build the Docker container to run the server. It's important that you rebuild the Docker container each time you boot a new server, since that sets it up with its own PubSub connection.
+4. Run the docker container you just built! That's your server. Leave it running as a process.
 
 ## Switching between Dev and Prod
 By default, the main branch is pointed at the dev servers. We keep the current prod version on the `prod` branch.

analytics/.gitignore

@@ -0,0 +1,2 @@
+emails.txt
+users.json

analytics/get_dev_bucket_size.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+aws s3 ls s3://biomechanics-uploads161949-dev --recursive --human-readable --summarize

analytics/get_num_users.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+aws cognito-idp list-users --user-pool-id us-west-2_vRDVX9u35 > users.json
+node parse_users.js

analytics/get_prod_bucket_size.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+aws s3 ls s3://biomechanics-uploads83039-prod --recursive --human-readable --summarize

analytics/parse_users.js

@@ -0,0 +1,31 @@
+const fs = require('fs');
+
+try {
+  const rawText = fs.readFileSync('./users.json', 'utf8');
+  const data = JSON.parse(rawText);
+  let emails = [];
+  let text = "";
+  for (const user of data.Users) {
+    for (const attr of user.Attributes) {
+      if (attr.Name === 'email') {
+         emails.push(attr.Value);
+         text += attr.Value + '\n';
+      }
+    }
+  }
+  console.log(emails.length);
+  console.log(emails);
+  let domains = [];
+  for (const email of emails) {
+    const domain = email.split('@')[1];
+    if (!domains.includes(domain)) {
+        domains.push(domain);
+        console.log(domain);
+    }
+  }
+  console.log('Domains: '+domains.length);
+  fs.writeFileSync('./emails.txt', text);
+} catch (err) {
+  console.error(err);
+}
+

blender/.gitignore

@@ -0,0 +1,5 @@
+data
+*.csv
+*.blend1
+dropped_trials.txt
+Geometry/

blender/README.md

@@ -0,0 +1,12 @@
+# Blender Support
+
+These are scripts to run from inside Blender to load animations, and produce high quality renders.
+
+## Dev Installation
+
+You can setup PyCharm to use your bundled Blender Python interpreter. On Mac, that's located at:
+
+`/Applications/Blender.app/Contents/Resources/3.3/python/bin/python3.10`
+
+Then, you can install the appropriate version of [fake-bpy-model](https://github.com/nutti/fake-bpy-module) to match 
+your Blender.