In this assignment you will practice writing backpropagation code, and training Neural Networks and Convolutional Neural Networks. The goals of this assignment are as follows:
- understand Neural Networks and how they are arranged in layered architectures
- understand and be able to implement (vectorized) backpropagation
- implement various update rules used to optimize Neural Networks
- implement batch normalization for training deep networks
- implement dropout to regularize networks
- effectively cross-validate and find the best hyperparameters for Neural Network architecture
- understand the architecture of Convolutional Neural Networks
- gain an understanding of how a modern deep learning library (PyTorch) works and gain practical experience using it to train models.
2024-09-20: Due to multiple difficulties with local setups, it's now highly recommended to do HW01 on Google Colab instead, where setup is much easier. Instructions for setup in this Ed post.
But, if you'd like to do it locally on your own machine, here are the steps.
Make sure your machine is set up with the assignment dependencies. If you're unfamiliar with Docker or any other part of the setup process, we highly recommend reading the setup document, which gives a high-level description of what the commands below (and their parameters) are doing.
The preferred approach for installing all the assignment dependencies is to use Anaconda, which is a Python distribution that includes many of the most popular Python packages for science, math, engineering and data analysis. Once you install Anaconda you can run the following command inside the homework directory to install the required packages for this homework:
conda env create -f cs182_hw1.yamlOnce you have all the packages installed, run the following command every time to activate the environment when you work on the homework.
conda activate cs182_hw1We've designed a Docker container that has all requisite packages installed + should work for Windows and Mac.
To use this approach, install Docker Desktop for your OS. After launching, either create an account or log in with your Github account. Then, go on the left sidebar and click on Images > Hub > Search > verityw/datac182. After downloading the hw1 image (which may take some time, on account of the file being ~10 GB), you should be able to start up a container by opening a terminal in Docker Desktop (bottom right corner) and running:
docker run -it -p 8888:8888 -v /PATH/TO/HW/REPO:/contained verityw/datac182:hw1 bashPlease make sure to replace /PATH/TO/HW/REPO in the above command with the actual path to the homework repo on your local machine!!! If you get some sort of permission error, you'll need to enable file-sharing for mounting directories in Docker Desktop.
This will also connect port 8888 on your host machine to 8888 in the container (you can change either of these if needed). Additionally, it mounts your homework repo to a directory inside the container called /contained (any files in your homework repo will be accessible both inside and outside the container). These are the only persistent files: make sure anything you want to save goes in that mounted directory, since all other files will be deleted when you stop the Docker container!
This allows you to enter the Docker container through an interactive terminal, which you can use as a normal terminal. You can use this to run e.g., the data download script (if you get a permission denied error, run chmod u+r+x get_datasets.sh before running the script again). Likewise, you should be able to compile the Cython stuff (if it says that it's trying to call gcc but can't find it, just run apt-get update and apt-get gcc . If it says something about assigning double to int, check the HW1 announcement thread for a fix / re-pull the repo).
Finally, after entering your homework repo in the container run:
jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token=''This will start the Jupyter server, which you'll be able to use on your local machine by going to localhost:8888 in your browser. You'll be able to run the notebooks for the homework this way!
This assignment is provided pre-setup with a VirtualBox image. Installation Instructions:
- Follow the instructions here to install VirtualBox if it is not already installed.
- Download the VirtualBox image here
- Load the VirtualBox image using the instructions here
- Start the VM. The username and password are both cs182. Required packages are pre-installed and the cs182_hw1 environment activated by default.
- Download the assignment code onto the VM yourself.
Once you have the starter code, you will need to download the CIFAR-10 dataset. Run the following from this repo:
cd deeplearning/datasets
./get_datasets.shIf running on Windows (where you can't run bash scripts), please run the above inside the Docker interactive terminal.
Convolutional Neural Networks require a very
efficient implementation. We have implemented of the functionality using
Cython; you will need to compile the Cython extension
before you can run the code. From the deeplearning directory, run the following
command:
python setup.py build_ext --inplaceAfter you download data, you should start the IPython notebook server from the homework 1 directory with the following command:
jupyter notebookThis opens a brower tab for you to work.
If you are unfamiliar with IPython, you should read our IPython tutorial.
Once you are done working run the collect_submission.sh script;
this will produce a file called assignment1.zip.
Upload this file to Gradescope.
The Gradescope will run an autograder on the files you submit. It is very unlikely but still possible that your implementation might fail to pass some test cases due to randomness.
If you think your code is correct, you can simply rerun the autograder to check check whether it is really due to randomness.
The IPython notebook FullyConnectedNets.ipynb will introduce you to our
modular layer design, and then use those layers to implement fully-connected
networks of arbitrary depth. To optimize these models you will implement several
popular update rules.
In the IPython notebook BatchNormalization.ipynb you will implement batch
normalization, and use it to train deep fully-connected networks.
The IPython notebook Dropout.ipynb will help you implement Dropout and explore
its effects on model generalization.
In the IPython Notebook ConvolutionalNetworks.ipynb you will implement several
new layers that are commonly used in convolutional networks as well as implement
a small convolutional network.
Now that you've implemented and gained an understanding for many key components of a basic deep learning library, it is time to move on to a modern deep learning library: Pytorch. Here, we will walk you through the key concepts of PyTorch, and you will use it to experiment and train a model on CIFAR10.
If you would like to access GPUs for faster training, we recommend you use Google Colab (https://colab.research.google.com/). If you use Colab for this notebook, make sure to manually download the completed notebook and place it in the assignment directory before submitting. Also remember to download required output file and place it into submission_logs/ directory.
I'm getting some weird issue about Cython assigning float to int
This is because Cython (which compiles Python to C) thinks that the code you provided is in Python 3 instead of Python 2. This is a problem because Python 2 converts int / int to int, while Python 3 makes it a float. By default, it should set to Python 2 (even if the actual Python version you're using is 3) -- I'm not sure why it isn't doing that for you. You can force it to use Python 2 by going into deeplearning/setup.py and looking for the part:
setup(
ext_modules=cythonize(extensions),
)This is what compiles im2col_cython.pyx into C. You can add compiler_directives={'language_level' : "2"} inside the cythonize() to force it to use Python 2, see here.
Cython gives an about a file called longintrepr.h not being found
Please lower your Python version to < 3.11 (we suggest 3.8 or 3.9). This should not be an issue for anyone using our Conda yaml file (or the Docker), since it sets the Python version to one of those versions.
I get an error "AMD-V is disabled in the BIOS" or "Intel-VT is disabled in the BIOS" or similar
Solution: See this link
The virtual machine won't boot
Solutions:
- Try increasing the number of allocated CPUs: Under Settings→System→Processor
- Try increasing the amount of allocated memory: