Full extraction notebook

.gitignore

@@ -37,7 +37,8 @@ ipython_config.py
 .LSOverride
 
 # Icon must end with two \r
-Icon
+Icon
+
 
 # Thumbnails
 ._*
@@ -140,6 +141,8 @@ docs/_build/
 target/
 
 # Jupyter Notebook
+notebooks/full_extraction/rtk
+notebooks/full_extraction/rolls_data
 
 # IPython
 

notebooks/full_extraction/README.md

@@ -0,0 +1,60 @@
+# RTK Full Extraction Pipeline
+
+## Installation
+0. (Optional) Copy this folder to somewhere you'll remember it
+1. Ensure python is installed (highly recommend running from within VSCode)
+2. Create a virtual environment: 
+```bash
+python3 -m venv rtk # Creates a virtual environment called "rtk"
+source rtk/bin/activate # Enters the virtual environment
+pip install -r requirements.txt # Installs the required packages
+```
+3. Install RTKLIB (see below)
+- MacOS users will have to modify some Makefiles and build from source
+- Linux users will have to build from source
+- Windows users can install a prebuilt version (google RTKLIB)
+
+
+## How to use
+1. Download the Hammerschlag Hall Basestaion Data
+2. Open this folder with VSCode
+3. Open `rtk_extraction.ipynb` and edit the fields in the first code section:
+    - BUGGIES: List of rovers that you want to do data analysis on
+    - DATE: Date you want to do data analysis for (format: YYYY-MM-DD)
+    - CLEANUP: If true, deletes all the temp files created during analysis
+    - DEBUG: Adds extra print statements if getting unexpected outputs
+4. Run the first 3 code sections. They will create a bunch of directories to
+   ensure data can be batch processed easily.
+5. Move rover ubx files (firmware uses extension .BIN) to the `raw_ubx` folder
+   corresponding to the date in the corresponding rover directories.
+6. If zipped, extract the basestation data and move it into the basestation_data
+   directory corresponding to the date.
+7. Attempt to run all cells in the notebook. If unsuccessful, ask for help.
+
+## Known issues
+- API for map of buggy course is broken so displaying shows on a white background
+
+## Special Installation Steps
+
+### Windows Users
+1. Download latest version from RTKLIB (https://rtklib.com)
+2. Extract the Zip File to somewhere on your hard drive, and copy file path (Either Ctrl + Shift + C, or double click and select "Copy as Path")
+3. Open Settings and Search for "Edit Environment Variables for Your Account" ![image](https://github.com/user-attachments/assets/4c383597-7cd0-479c-8b86-4668df919594)
+4. Once opened, select "Path" and press "Edit"![image](https://github.com/user-attachments/assets/aed27548-ff0c-4958-994c-b4c7b088093e)
+5. Afterwards, select "New" and paste the file path of the RTKLIB into the section ![image](https://github.com/user-attachments/assets/5f62413b-6954-48cb-9fb8-9761a17d6ada)
+
+
+
+### (MacOS users) Installing from source
+1. Have cmake & make installed (probably need homewbrew - https://brew.sh/)
+2. Download [this release of RTKLIB] (https://github.com/rtklibexplorer/RTKLIB/releases/tag/b34k)
+3. Edit the file src/options.c to include `#define _DARWIN_C_SOURCE` on line 31 (above `#define _POSIX_C_SOURCE 199506)
+4. Run the following sequence of commands:
+```shell
+cd app/consapp/convbin/gcc
+make
+sudo make install # Will ask for your password
+cd ../../rnx2rtkp/gcc
+make
+sudo make install # Might ask for your password
+```

notebooks/full_extraction/constants.py

@@ -0,0 +1,33 @@
+PARSE_CSV_HEADER = ["timestamp", # TODO change this to "timestamp_str"
+                    "latitude",
+                    "longitude",
+                    "altitude",
+                    "fix-type",
+                    "satellites",
+                    "sdn",
+                    "sde",
+                    "sdu",
+                    "sdne",
+                    "sdeu",
+                    "sdun",
+                    "age",
+                    "ratio"
+]
+
+CALC_CSV_HEADER = [
+    "timestamp", # UTC time (float)
+    "timestamp_str",
+    "fix_type",
+    "latitude",
+    "latitude_processed",
+    "longitude",
+    "longitude_processed",
+    "altitude", # meters
+    "velocity", # 2d vector
+    "speed", # scalar (abs val of vector)
+    "acceleration_vector",
+    "acceleration_magnitude",
+    "horizontal_accuracy",
+    "vertical_accuracy",
+    "speed_accuracy"
+ ]

notebooks/full_extraction/requirements.txt

@@ -0,0 +1,56 @@
+appnope==0.1.4
+asttokens==2.4.1
+comm==0.2.2
+contourpy==1.3.0
+cycler==0.12.1
+debugpy==1.8.6
+decorator==5.1.1
+executing==2.1.0
+fonttools==4.54.1
+geographiclib==2.0
+geopy==2.4.1
+ipykernel==6.29.5
+ipympl==0.9.4
+ipython==8.28.0
+ipython-genutils==0.2.0
+ipywidgets==8.1.5
+jedi==0.19.1
+jupyter_client==8.6.3
+jupyter_core==5.7.2
+jupyterlab_widgets==3.0.13
+kiwisolver==1.4.7
+matplotlib==3.9.2
+matplotlib-inline==0.1.7
+nest-asyncio==1.6.0
+numpy==2.1.1
+packaging==24.1
+pandas==2.2.3
+parso==0.8.4
+pexpect==4.9.0
+pillow==10.4.0
+platformdirs==4.3.6
+plotly==5.24.1
+prompt_toolkit==3.0.48
+psutil==6.0.0
+ptyprocess==0.7.0
+pure_eval==0.2.3
+Pygments==2.18.0
+pynmeagps==1.0.42
+pyparsing==3.1.4
+pyrtcm==1.1.1
+python-dateutil==2.9.0.post0
+pytz==2024.2
+pyubx2==1.2.46
+pyzmq==26.2.0
+scipy==1.14.1
+seaborn==0.13.2
+six==1.16.0
+stack-data==0.6.3
+tenacity==9.0.0
+tornado==6.4.1
+tqdm==4.66.5
+traitlets==5.14.3
+tzdata==2024.2
+utm==0.7.0
+wcwidth==0.2.13
+widgetsnbextension==4.0.13