You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+95-2Lines changed: 95 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,6 +3,22 @@
3
3
# Collaborative-Coding-Exam
4
4
Repository for final evaluation in the FYS-8805 Reproducible Research and Collaborative coding course
5
5
6
+
## **Table of Contents**
7
+
1.[Project Description](#project-description)
8
+
2.[Installation](#installation)
9
+
3.[Usage](#usage)
10
+
4.[Results](#results)
11
+
5.[Citing](#citing)
12
+
13
+
## Project Description
14
+
This project involves collaborative work on a digit classification task, where each participant works on distinct but interconnected components within a shared codebase. <br>
15
+
The main goal is to develop and train digit classification models collaboratively, with a focus on leveraging shared resources and learning efficient experimentation practices.
16
+
### Key Aspects of the Project:
17
+
-**Individual and Joint Tasks:** Each participant has separate tasks, such as implementing a digit classification dataset, a neural network model, and an evaluation metric. However, all models and datasets must be compatible, as we can only train and evaluate using partners' models and datasets.
18
+
-**Shared Environment:** Alongside working on our individual tasks, we collaborate on joint tasks like the main file, and training and evaluation loops. Additionally, we utilize a shared Weights and Biases environment for experiment management.
19
+
-**Documentation and Package Management:** To ensure proper documentation and ease of use, we set up Sphinx documentation and made the repository pip-installable
20
+
-**High-Performance Computing:** A key learning objective of this project is to gain experience with running experiments on high-performance computing (HPC) resources. To this end, we trained all models on a cluster
<br> Replace placeholders with your desired values:
55
+
56
+
-`<MODEL_NAME>`: You can choose from different models ( `"MagnusModel", "ChristianModel", "SolveigModel", "JanModel", "JohanModel"`).
57
+
58
+
59
+
-`<DATASET_NAME>`: The following datasets are supported (`"svhn", "usps_0-6", "usps_7-9", "mnist_0-3", "mnist_4-9"`)
60
+
61
+
62
+
-`<METRIC_1> ... <METRIC_N>`: Specify one or more evaluation metrics (`"entropy", "f1", "recall", "precision", "accuracy"`)
63
+
64
+
65
+
-`<RESULTS_DIRECTORY>`: Folder where all model outputs, logs, and checkpoints are saved
66
+
67
+
68
+
-`<RUN_NAME>`: Name for WANDB project
69
+
29
70
30
-
### Running on a k8s cluster
71
+
-`<DEVICE>`: `"cuda", "cpu", "mps"`
72
+
73
+
74
+
## Running on a k8s cluster
31
75
32
76
In your job manifest, include:
33
77
@@ -46,3 +90,52 @@ to pull the latest build, or check the [packages](https://github.com/SFI-Visual-
46
90
47
91
> [!NOTE]
48
92
> The container is build for a `linux/amd64` architecture to properly build Cuda 12. For other architectures please build the docker image locally.
93
+
94
+
95
+
## Results
96
+
### JanModel & MNIST_0-3
97
+
This section reports the results from using the model "JanModel" and the dataset MNIST_0-3 which contains MNIST digits from 0 to 3 (Four classes total).
98
+
For this experiment we use all five available metrics, and train for a total of 20 epochs.
99
+
100
+
We achieve a great fit on the data. Below are the results for the described run:
101
+
102
+
| Dataset Split | Loss | Entropy | Accuracy | Precision | Recall | F1 |
In addition to the overall task, I was assigned the implementation of a multi-layer perceptron model, a dataset loader for a subset of the MNIST dataset, and an accuracy metric.
6
+
7
+
## Network Implementation In-Depth
8
+
For the network part, I was tasked with making a simple MLP network model for image classification tasks. The model consists of two hidden layers with 100 neurons each followed by a leaky-relu activation. This implementation involves creating a custom class that inherits from the PyTorch `nn.Module` class. This allows our class to have two methods: the `__init__` method and a `forward` method. When we create an instance of the class, we can call the instance like a function, which will run the `forward` method.
9
+
10
+
The network is initialized with the following parameters:
11
+
*`image_shape`
12
+
*`num_classes`
13
+
14
+
The `image_shape` argument provides the shape of the input image (channels, height, width) which is used to correctly initialize the input size of the first layer. The `num_classes` argument defines the number of output neurons, corresponding to the number of classes in the dataset.
15
+
16
+
The forward method in this class processes the input as follows:
17
+
1. Flattens the input image.
18
+
2. Passes the flattened input through the first fully connected layer (`fc1`).
19
+
3. Applies a LeakyReLU activation function.
20
+
4. Passes the result through the second fully connected layer (`fc2`).
21
+
5. Applies another LeakyReLU activation function.
22
+
6. Passes the result through the output layer (`out`).
23
+
24
+
## MNIST Dataset In-Depth
25
+
For the dataset part, I was tasked with creating a custom dataset class for loading a subset of the MNIST dataset containing digits 0 to 3. This involved creating a class that inherits from the PyTorch `Dataset` class.
26
+
27
+
The class is initialized with the following parameters:
28
+
*`data_path`
29
+
*`sample_ids`
30
+
*`train` (optional, default is False)
31
+
*`transform` (optional, default is None)
32
+
*`nr_channels` (optional, default is 1)
33
+
34
+
The `data_path` argument stores the path to the four binary files containing MNIST dataset. The verification of presence of these files and their download, if necessary, is facilitated by the `Downloader`class. The `sample_ids` parameter contains the indices of images and their respective labels that are to be loaded from MNIST dataset. Filtering and random splitting of these indices is performed within the `load_data`function. `train`is a boolean flag indicating whether to load data from training (for training and validation splits) or from testing (test split) part of the MNIST dataset. `transform` is a callable created with `torch.compose()` to be applied on the images. `nr_channels` is not used in this dataset, only included for compatibility with other functions.
35
+
36
+
The class has two main methods:
37
+
*`__len__`: Returns the number of samples in the dataset.
38
+
*`__getitem__`: Retrieves the image and label at the specified index.
39
+
40
+
## Accuracy Metric In-Depth
41
+
For the metric part, I was tasked with creating an accuracy metric class. The `Accuracy` class computes the accuracy of a model's predictions. The class is initialized with the following parameters:
42
+
*`num_classes`
43
+
*`macro_averaging` (optional, default is False)
44
+
45
+
The `num_classes` argument specifies the number of classes in the classification task. The `macro_averaging`argument is a boolean flag specifying whether to compute the accuracy using micro or macro averaging.
46
+
47
+
The class has the following methods:
48
+
*`forward`: Stores the true and predicted labels computed on a batch level.
49
+
*`_macro_acc`: Computes the macro-averaged accuracy on stored values.
50
+
*`_micro_acc`: Computes the micro-averaged accuracy on stored values.
51
+
*`__returnmetric__`: Returns the computed accuracy based on the averaging method for all stored predictions.
52
+
*`__reset__`: Resets the stored true and predicted labels.
53
+
54
+
The `forward` method takes the true labels and predicted labels as input and stores them. The `_macro_acc` method computes the macro-average accuracy by averaging the accuracy for each class. The `_micro_acc` method computes the micro-average accuracy by calculating the overall accuracy. The `__returnmetric__` method returns the computed accuracy based on the averaging method. The `__reset__` method resets the stored true and predicted labels to prepare for the next epoch.
Copy file name to clipboardExpand all lines: doc/Magnus_page.md
+51-1Lines changed: 51 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,9 +21,28 @@ Each input is flattened over the channel, height and width channels. Then they a
21
21
22
22
23
23
## SVHN Dataset In-Depth
24
+
The dataloader I was tasked with making is to load the well-known SVHN dataset. This is a RGB dataset with real-life digits taken from house numbers. The class inherits from the torch Dataset class, and has four methods:
25
+
*__init__ : initialized the instance of the class
26
+
*_create_h5py: Creates the h5 object containing data from the downloaded .mat files for ease of use
27
+
*__len__ : Method needed in use of the DataLoader class. Returns length of the dataset
28
+
*__getitem__ : Method needed in use of the DataLoader class. Loads a image - label pair, applies any defined image transformations, and returns both image and label.
24
29
25
30
26
31
32
+
The __init__ method takes in a few arguments.
33
+
* data_path (Path): Path where either the data is downloaded or where it is to be downloaded to.
34
+
* train (bool): Which set to use. If true we use the training set of SVHN, and if false we use the test set of SVHN.
35
+
* transform: The transform functions to be applied to the returned image.
36
+
* nr_channels: How many channels to use. Can be either 1 or 3, corresponding to either greyscale or RGB images respectively.
37
+
38
+
In the init we check for the existence of the SVHN dataset. If it does not exist, then we run the _create_h5py method which will be explained later. Then the labels are loaded into memory as they are needed for the __len__ method among other things.
39
+
40
+
The _create_h5py method downloads a given SVHN set (train or test). We also change the label 10 to 0, as the SVHN dataset starts at index 1, with 10 representing images with the digit zero. After the download, we create two .h5 files. One with the labels and one with the images.
41
+
42
+
Lastly, in __getitem__ we take index (number between 0 and length of label array). We retrive load the image h5 file, and retrive the row corresponding to the index.
43
+
We then convert the image to an Pillow Image object, then apply the defined transforms before returning the image and label.
44
+
45
+
27
46
28
47
## Entropy Metric In-Depth
29
48
The EntropyPrediction class' main job is to take some inputs from the MetricWrapper class and store the batchwise Shannon Entropy metric of those inputs. The class has four methods with the following jobs:
@@ -41,4 +60,35 @@ With permission I've used the scipy implementation to calculate entropy here. We
41
60
42
61
Next we have the __returnmetric__ method which is used to retrive the stored metric. This returns the mean over all stored values. Effectively, this will return the average Shannon Entropy of the dataset.
43
62
44
-
Lastly we have the __reset__ method which simply emptied the variable which stores the entropy values to prepare it for the next epoch.
63
+
Lastly we have the __reset__ method which simply emptied the variable which stores the entropy values to prepare it for the next epoch.
64
+
65
+
## More on implementation choices
66
+
It should be noted that a lot of our decisions came from a top-down perspective. Many of our classes have design choices to accomendate the wrappers which handle the initialization and dataflow of the different metrics, dataloaders, and models.
67
+
All in all, we've made sure you don't really need to interact with the code outside setting up the correct arguments for the run, which is great for consistency.
68
+
69
+
70
+
## Challenges
71
+
### Running someone elses code
72
+
This section answers the question on what I found easy / difficult running another persons code.
73
+
74
+
I found it quite easy to run others code. We had quite good tests, and once every test passed, I only had one error with the F1 score not handeling an unexpected edgecase. To fix this I raised an issue, and it was fixed shortly after.
75
+
76
+
One thing I did find a bit difficult was when people would change integral parts of the common code such as wrappers or loader functions (usually for the better), but did not raise an issue or notify about the change. It did cause some moments of questions, but in the end we sorted it out through weekly meetings where we agreed on design choices and how to handle loading of the different modules.
77
+
78
+
The issues mentioned above also lead to a week or so where there was always a test failing, and the person whos' code was failing did not have time to work on it for a few days.
79
+
80
+
### Someone running my code
81
+
This section answers the question on what I found easy / difficult having someone run my code.
82
+
83
+
I did not experience that anyone had issues with my code. After I fixed all issues and tests related to my code, it seems to have run fine, and no issues have been raised to my awareness about this.
84
+
85
+
86
+
## Tools
87
+
This section answers the question of which tools from the course I used during the home-exam.
88
+
89
+
For this exam I used quite a few tools from the course.
90
+
I've never used pytest and test functions while writing code. This was quite fun to learn how to use, and having github actions also run the same tests was a great addition.
91
+
92
+
Github actions we used for quite a few things. We checked for code formatting, documentation generation and run the code tests.
93
+
94
+
Using sphinx for documentation was also a great tool. Turns out it's possible to write the doc-string in such a way that it automatically generates the documentation for you. This has helped reduce the workload with documentation a lot, and makes writing proper docstrings worthwile.
0 commit comments