Updated code for SatIQ paper

Author: jsmailes

Dockerfile

@@ -1,5 +1,4 @@
-#FROM tensorflow/tensorflow:latest
-FROM tensorflow/tensorflow:latest-gpu
+FROM tensorflow/tensorflow:2.10.1-gpu
 
 # avoid questions when installing things in apt-get
 ARG DEBIAN_FRONTEND=noninteractive
@@ -21,4 +20,6 @@ RUN pip install notebook
 
 RUN pip install seaborn
 
+RUN pip install --upgrade "protobuf<=3.20.1"
+
 WORKDIR /

README.md

@@ -3,9 +3,20 @@
 This repository contains all the data collection, model training, and analysis code for the `SatIQ` system, described in the paper "Watch This Space: Securing Satellite Communication through Resilient Transmitter Fingerprinting".
 This system can be used to authenticate Iridium satellite transmitters using high sample rate message headers.
 
-Additional materials:
-- Paper (arXiv preprint): https://arxiv.org/abs/2305.06947
-- Full dataset (Iridium message headers): https://zenodo.org/record/8220494
+> [!NOTE]
+> This version of the repository contains the code used for the paper "SatIQ: Extensible and Stable Satellite Authentication using Hardware Fingerprinting".
+> The code used for "Watch This Space: Securing Satellite Communication through Resilient Transmitter Fingerprinting" can be found [here](https://github.com/ssloxford/SatIQ/tree/v1.0.2).
+
+Additional materials (SatIQ):
+- "SatIQ" paper: https://www.cs.ox.ac.uk/files/14805/main.pdf
+- Full dataset (UK): https://doi.org/10.7910/DVN/P5FUAW
+- Full dataset (Germany): https://doi.org/10.7910/DVN/RXWV1M
+- Full dataset (Switzerland): https://doi.org/10.7910/DVN/OSSJ68
+- Trained model weights: https://doi.org/10.7910/DVN/GANMDZ
+
+Additional materials (Watch This Space):
+- "Watch This Space" paper: https://arxiv.org/abs/2305.06947
+- Full dataset: https://zenodo.org/record/8220494
 - Trained model weights: https://zenodo.org/record/8298532
 
 When using this code, please cite the following paper: "Watch This Space: Securing Satellite Communication through Resilient Transmitter Fingerprinting".
@@ -60,9 +71,24 @@ notebook
 A GPU is recommended (with all necessary drivers installed), and a moderate amount of RAM will be required to run the data preproccessing and model training.
 
 
-### Downloading Data
+### Downloading Data (SatIQ)
+
+The full dataset for "SatIQ" is stored on the Harvard Dataverse at the following URL: https://dataverse.harvard.edu/dataverse/satiq.
+
+This includes three datasets for each of the three locations (UK, Germany, Switzerland), and trained model weights.
+
+These can be downloaded from the site directly, but the following script may be preferable due to the large file size:
+```bash
+TODO https://eamonnbell.webspace.durham.ac.uk/2023/03/07/bulk-downloading-from-dataverse/
+```
+
+> [!WARNING]
+> The files are very large (approximately 1TB total).
+> Ensure you have enough disk space before downloading.
+
+### Downloading Data (Watch This Space)
 
-The full dataset is stored on Zenodo at the following URL: https://zenodo.org/record/8220494
+The full dataset for "Watch This Space" is stored on Zenodo at the following URL: https://zenodo.org/record/8220494.
 
 These can be downloaded from the site directly, but the following script may be preferable due to the large file size:
 ```bash
@@ -91,6 +117,37 @@ done
 See the instructions below on processing the resulting files for use.
 
 
+### Directory Structure
+
+The training and analysis scripts expect the repository to be laid out as follows:
+```
+SatIQ
+├── ...
+└── data
+    ├── models
+    │   ├── downsample
+    │   │   └── ...
+    │   └── ...
+    ├── tfrecord
+    │   ├── ...
+    │   ├── germany
+    │   │   └── ...
+    │   ├── switzerland
+    │   │   └── ...
+    │   └── uk-switzerland
+    └── test
+        ├── embeddings
+        │   └── ...
+        └── labels
+            └── ...
+```
+
+Any downloaded model/loss files with `downsample` in the name should be placed in `data/models/downsample`, and any other model files should be placed in `data/models`.
+
+The `uk-switzerland` directory can be populated using `preprocessing/dataset-combine.sh`, and the `embeddings` and `labels` directories using `preprocessing/generate-embeddings.py`.
+These are described in greater detail below.
+
+
 ## Usage
 
 ### TensorFlow Container
@@ -108,6 +165,7 @@ If your machine has no GPUs:
 
 The `util` directory contains the main data processing and model code:
 - `data.py` contains utilities for data loading and preprocessing.
+- `processing.py` contains utilities for processing and analysis of results.
 - `models.py` contains the main model code.
 - `model_utils.py` contains various helper classes and functions used during model construction and training.
 
@@ -128,6 +186,8 @@ Data will be stored in `data/db.sqlite3`.
 If a different SDR is used, the `iridium_extractor` configuration may need to be altered.
 Change the `docker-compose.yml` to ensure the device is mounted in the container, and modify `iridium_extractor/iridium_extractor.py` to use the new device as a source.
 
+The `autorun.sh` and `restart.sh` scripts are provided for convenience, in order to automate the process of stopping the container and moving the resulting database files to a permanent storage location.
+
 
 ### Data Preprocessing
 
@@ -136,13 +196,25 @@ It is recommended to run these scripts from within the TensorFlow container desc
 
 > [!NOTE]
 > Converting databases to NumPy files and filtering is only necessary if you are doing your own data collection.
-> If the provided dataset on Zenodo is used, only the `np-to-tfrecord.py` script is needed.
+> If the "SatIQ" dataset is used, no preprocessing is required.
+> If the "Watch This Space" dataset is used, only the `np-to-tfrecord.py` script is required.
 
 > [!IMPORTANT]
 > Please note that these scripts load the full datasets into memory, and will consume large amounts of RAM.
 > It is recommended that you run them on a machine with at least 128GB of RAM.
 
 
+#### db-to-tfrecord.py
+
+This script extracts database files and processes them directly into TFRecord files, optionally adding weather data if provided.
+This should be used preferentially over the legacy scripts described below.
+To run this script, use the command-line arguments as directed by the script itself.
+
+```bash
+python3 db-to-tfrecord.py --help
+```
+
+
 #### db-to-np-multiple.py
 
 This script extracts the database files into NumPy files.
@@ -203,7 +275,7 @@ Be careful using this option, as it creates a much larger number of files, and t
 > This script in particular will use a large amount of RAM, since it loads the entire dataset into memory at once.
 > Processing may be done in batches by using the `--max-files` and `--skip-files` command-line arguments, or the script below.
 
-##### np-to-tfrecord-parallel.sh
+#### np-to-tfrecord-parallel.sh
 
 This script can run multiple instances of `np-to-tfrecord.py` in parallel, allowing preprocessing to be sped up and/or less RAM to be used.
 
@@ -235,8 +307,30 @@ python3 sqlite3-compress.py <INPUT PATH> <OUTPUT PATH>
 ```
 
 
+#### dataset-combine.sh
+
+This script builds the combined UK-Switzerland dataset out of the two separate datasets by linking files.
+This script has no configuration options and is used as follows:
+```bash
+./dataset-combine.sh
+```
+
+
+#### generate-embeddings.py
+
+This script takes a trained model (or multiple models) and generates the embeddings of the dataset produced by that model, to enable faster analysis.
+To use this script, modify `data_dir`, `model_dir`, and `output_dir` to point to the relevant input/output directories, and ensure `model_names` contains the correct names of the models from which the embeddings should be generated.
+The script should then be run with no arguments
+```bash
+python3 generate-embeddings.py
+```
+
+
 #### Noise
 
+> [!NOTE]
+> These scripts are only used for "Watch This Space", and should not be used with the "SatIQ" data.
+
 The `noise` directory contains modified versions of the above scripts that filter the dataset to remove the messages with the highest noise.
 Use in the same way as above.
 
@@ -245,16 +339,20 @@ Ensure that all the requisite directories have been created before these scripts
 
 ### Model Training
 
-The script for training the `SatIQ` model is found in `code/training/ae-triplet-conv.py`.
+The scripts for model training can be found in the `training` directory.
 Ensure that data is placed in the `data` directory before running.
+The `ae-triplet-conv-dataset-slices.py` script is used to train models from "SatIQ", and `ae-triplet-conv` to train models from "Watch This Space".
+Additionally, `train-___.sh` scripts are provided as examples for training multiple models sequentially under different configurations.
 
 Adjust the arguments at the top of the script to ensure the data and output directories are set correctly (these should be fine if running inside the TensorFlow Docker container), then run the script with no arguments:
 ```bash
-python3 ae-triplet-conv.py
+python3 ae-triplet-conv-dataset-slices.py
 ```
 
-This will take a long time.
-The checkpoints should appear in `data/models`.
+Additional command-line arguments can be used to adjust characteristics of the model.
+
+Training will take a long time.
+The checkpoints will appear in `data/models`.
 
 
 ### Analysis
@@ -267,9 +365,16 @@ See [Setup](#Setup) for requirements to run outside docker.
 
 Note that these also require a large amount of RAM, and a GPU is recommended in order to run the models.
 
-The `plots-data.ipynb` notebook contains plots relating to the raw samples.
+The `satiq-data.ipynb` notebook contains plots relating to the raw samples.
+
+The `satiq-models.ipynb` notebook contains all the analysis of the trained models.
 
-The `plots-models.ipynb` notebook contains all the analysis of the trained models.
+> [!NOTE]
+> The `past-ai-___.pdf` plots require access to the dataset from the paper "PAST-AI: Physical-layer authentication of satellite transmitters via deep learning".
+> Please contact the authors of this paper for access if needed.
+
+> [!NOTE]
+> The `wts-data.ipynb` and `wts-models.ipynb` are also included for legacy purposes -- these are the equivalent analysis scripts from "Watch This Space".
 
 
 ## Contribute