This repository contains the Django database and web API code. For information about contributing to Autograder.io, see our contributing guide.

Server Dev Setup

This tutorial will walk you through setting up your local machine for modifying and testing the server code.

System Requirements

OS: A currently supported Ubuntu LTS

It may be possible to run the server tests on OSX. If you decide to try this, you're on your own. Newer versions of Ubuntu are usually ok, but some steps may differ.

Clone and Checkout

git clone [email protected]:eecs-autograder/autograder-server.git
cd autograder-server
git checkout develop

Install Docker Community Edition

Ubuntu: https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1

Install Postgres

Postgres is required to run the test suite. The simplest way to set this up is to run an official Postgres docker container. Invoke the script below to start a postgres container for development:

bash dev_scripts/start_postgres_dev.sh

Install Redis Server

sudo apt-get install redis-server

Install Python 3.10

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.10 python3.10-distutils python3.10-venv
curl https://bootstrap.pypa.io/get-pip.py | sudo python3.10
python3.10 -m pip install pip==24.0

python3.10 -m venv venv
source venv/bin/activate

pip install pip-tools wheel
pip-sync requirements.txt requirements-dev.txt

You can then run pipenv shell to start a shell in the virtual environment, or if you prefer you can prefix the python commands below with pipenv run.

If you run into errors installing psycopg, please refer to https://www.psycopg.org/docs/install.html#build-prerequisites for troubleshooting tips.

Updating Packages

This section contains some useful reference commands for pip-tools.

# Note: After running any of these variants of `pip-compile`, you should
# run `pip-sync requirements.txt requirements-dev.txt`
# (or `./install_pip_packages.sh`, which is an alias script for this command).

# Update one non-dev (listed in requirements.in) package
pip-compile -P <package name>

# Update one dev (listed in requirements-dev.in) package
pip-compile requirements-dev.in -P <package name>

# Update all non-dev packages
pip-compile --upgrade

# Update all dev packages
pip-compile requirements-dev.in --upgrade

To install a new non-dev package, add it to requirements.in and then run pip-compile and ./install_pip_packages.sh.

To install a new dev package, add it to requirements-dev.in and then run pip-compile requirements-dev.in and ./install_pip_packages.sh.

Generate Secrets

Run the following command to generate Django and GPG secrets.

python3.10 generate_secrets.py

Running the Unit Tests

To run the tests (takes 15-60 minutes):

./manage.py test

To exclude the slowest test cases from the run:

./manage.py test --exclude-tag slow

Updating schema.yml and Rendering the Schema

This project uses DRF's schema generation as a starting point for discovering API operations and their URL params. We extend this functionality in autograder/rest_api/schema.py to add Model Schema generation and to fill in endpoint details.

To update schema.yml, run:

bash dev_scripts/generate_schema.sh

If you are running the full development stack, you may skip the next step.

To render and serve the API using Swagger UI, run (requires Docker):

# Run from the project root directory, i.e. "autograder-server"
docker run -d --name ag_schema -p 8080:8080 --env-file schema/env -v $(pwd)/schema:/root swaggerapi/swagger-ui

Then navigate to localhost:8080 in your browser. To change the port, change 8080:8080 to <your port>:8080, e.g. 9001:8080.

Coding Standards

In addition to the items listed here, all source code must follow our Python coding standards.

Use these import aliases for commonly-used modules: - import autograder.core.models as ag_models - NOTE: Do NOT import autograder.core.models from modules inside that package. - import autograder.core.fields as ag_fields - import autograder.rest_api.permissions as ag_permissions - import autograder.rest_api.serializers as ag_serializers - import autograder.core.utils as core_ut - import autograder.handgrading.models as hg_models

Linters

You can run all of the required linter checks by invoking:

bash lint.sh

In addition to running pycodestyle, pydocstyle, and mypy, this will also check that migrations & API schema are up to date and that the API schema is valid.

Note that validating the API schema requires Node 16 (newer versions may work as well). You can install Node 16 with NVM.

Profiling

You can use django-debug-toolbar to profile API requests.

1. Follow the dev stack setup tutorial for the autograder-full-stack repo.
2. Populate the database with benchmark data. Benchmark scripts should be added to autograder-server/benchmarks and should include:
   • Instructions on how to run them on the development stack.
   • Results from the last time they were run (and specifying what machine).
3. Visit the API URL in your browser with the query parameter debug=true appended. For example: https://localhost:<port>/api/users/current/?debug=true

Versioning & Branches

This package uses calendar versioning following Python conventions, with version numbers of the form yyyy.mm.X, where X is for minor versions. For example: 2024.08.0 corresponds to August 2024. We also make use of pre-release tags such as .devX. Since we don't build this module as a Python package, we zero-pad the month to be consistent with branch names.

Development & Release Branches: Protocols and Workflow

"develop" branch

Use feature branches for all changes, and make a pull request against the develop branch.

"release-*" branches

Name release branches as release-YYYY.MM.x, replacing YYYY with the full year and MM with the zero padded month (e.g., release-2024.08.x).

Do NOT merge or rebase directly between the develop and release branches. Once a release branch is created, it should only be updated with bugfix- or (rarely) feature-style branches. Squash-and-merge for this type of PRs. After the squashed branch is merged into a release branch, cherry-pick the squashed commit on top of develop and open a pull request to merge the changes into develop.

The version of README.md (this file) on the develop branch is the source of truth. Update this file on release branches just before publishing a release. If instructions differ across releases, include both, and label which version the instructions apply to.

Publishing a release

To create a github release, trigger a workflow_dispatch event on the release branch. Pass the version number as input.

CI will update the version number, lint and test the module, tag the release, and create a GitHub release.