diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml new file mode 100644 index 0000000..f0a6ced --- /dev/null +++ b/.github/workflows/pre-commit.yml @@ -0,0 +1,15 @@ +name: pre-commit + +on: + push: + branches: + - main + pull_request: + types: [ opened, synchronize, reopened, ready_for_review ] + workflow_dispatch: +jobs: + pre-commit: + uses: vortexntnu/vortex-ci/.github/workflows/reusable-pre-commit.yml@main + with: + ros_distro: 'humble' + config_path: '.pre-commit-config.yaml' diff --git a/.gitignore b/.gitignore index 452c82c..8fd2a3d 100644 --- a/.gitignore +++ b/.gitignore @@ -174,6 +174,15 @@ log/ # data data/ +*.out +*/*.out + +.vscode/ +log/ + +# Ignore all experiment result folders +object_detection_training/*/ +unet_roboflow_training/*/ # Training outputs and Roboflow datasets results/ diff --git a/README.md b/README.md index cc869f8..d8b130a 100644 --- a/README.md +++ b/README.md @@ -12,15 +12,3 @@ Make sure that you have the following installed on your system: - [Python](https://www.python.org/) - [pip](https://pip.pypa.io/en/stable/installation/) - [venv](https://docs.python.org/3/library/venv.html) - ---- - -## :test_tube: Testing - -The automated tests are stored in the `tests` directory. Make sure to [install `pytest`](https://docs.pytest.org/en/7.1.x/getting-started.html). The tests can be ran by using the command `pytest .`. The commands will run the test define the the `test_*.py` files. Make sure that you are in the `YOLO-detect-buoys` directory. The naming convention of testing files is comprised of the word `test_` followed by the name of the file to which the test is related: `test_.py`. An example of that is `tests/test_utils.py` which is a python test file tah tests the functions `utils`. - ---- - -## :rotating_light: Linting - -Linting improves code quality by ensuring the the codebase does not contain bad-practices. Make sure to [install PyLint](https://pypi.org/project/pylint/) globally in order to use CLI. Use the command `pylint $(git ls-files '*.py')` in order to lint the files tracked by git. PyLint is used as the Python linter in this project. Make sure to install the [PyLint extension](https://pypi.org/project/pylint/) for VSCode or [PyLint plugin](https://plugins.jetbrains.com/plugin/11084-pylint) for JetBrains products like Pycharm. The rules are saved in the `.pylintrc` files. diff --git a/YOLO-detect-buoys/README.md b/YOLO-detect-buoys/README.md deleted file mode 100644 index 9ab9610..0000000 --- a/YOLO-detect-buoys/README.md +++ /dev/null @@ -1,65 +0,0 @@ -README.md - -# YOLO detect buoys - -[![codecov](https://codecov.io/github/vortexntnu/vortex-image-processing/graph/badge.svg?token=yS64SRLzUs)](https://codecov.io/github/vortexntnu/vortex-image-processing) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/pylint-dev/pylint) - -## This is the README file for the project by Vortex that uses YOLO to detect buoys. - -## Getting Started: - -### :package: Prerequisites: - -Make sure that you have the following installed on your system: - -- [Python](https://www.python.org/) -- [pip](https://pip.pypa.io/en/stable/installation/) -- [venv](https://docs.python.org/3/library/venv.html) - -### :runner: Run the application: - -- Create a virtual environment by running `python -m venv venv`. -- Download the project dependencies by running `pip install -r requirements.txt`. -- Activate the virtual environment by running `source venv/bin/activate` (for Unix-based systems) or `venv\Scripts\activate` (for Windows). -- Make sure to create `.env` file and add `ROBOFLOW_API_KEY` variable equal to your token (you can obtain the token by following the instructions in the next section). -- To specify which dataset to read from Roboflow make to add the following environment variables to the `.env` file: - - `ROBOFLOW_PROJECT_ID`: Id of the Roboflow project (_buoy-detection-qzjg1_) - - `WORKSPACE`: if the grouping of the available datasets for our organization (_vortexbuoytrainingset_) - - `ROBOFLOW_PROJECT_VERSION`: the number that indicates which version of the dataset pull be pulled (_1_) - - `DATASET_FORMAT`: the format in which the dataset will be received (_yolov8_) -- Run the application by executing the `main.py` file. - ---- - -## :robot: Roboflow - -The data is stored in the [Roboflow](https://roboflow.com/) service. In order to get the data you have to first obtain the token which could give you permissions to access through Roboflow APIs. After obtaining the Roboflow token, you have get the necessary information which would allow Roboflow to find and fetch the required dataset. First make to get the Roboflow token as described in the following section. - -### :key: Get Token: - -The steps to get the token: - -#### Go to settings: - -Go to the upper right side of the Roboflow main page. Press on the your name to open the manu. Select the settings `Settings` option in the menu: - -![Menu up right](assets_docs/gifs-docs/open_settings.gif) - -#### Find the Token under your workspace: - -Go to the side manu, and choose the name of the dataset, which in this case is `Vortexbouytrainingset` (markets as red in the image) and choose `Roboflow API`, as demonstrated by: -![Side menu](assets_docs/gifs-docs/get_api_key.gif) - -Then, in the appeared page, copy the string under the `Private API Key (for inference and REST API)`. - ---- - -## :test_tube: Testing - -The automated tests are stored in the `tests` directory. Make sure to [install `pytest`](https://docs.pytest.org/en/7.1.x/getting-started.html). The tests can be ran by using the command `pytest .`. The commands will run the test define the the `test_*.py` files. Make sure that you are in the `YOLO-detect-buoys` directory. The naming convention of testing files is comprised of the word `test_` followed by the name of the file to which the test is related: `test_.py`. An example of that is `tests/test_utils.py` which is a python test file tah tests the functions `utils`. - ---- - -## :rotating_light: Linting - -Linting improves code quality by ensuring the the codebase does not contain bad-practices. Make sure to [install PyLint](https://pypi.org/project/pylint/) globally in order to use CLI. Use the command `pylint $(git ls-files '*.py')` in order to lint the files tracked by git. PyLint is used as the Python linter in this project. Make sure to install the [PyLint extension](https://pypi.org/project/pylint/) for VSCode or [PyLint plugin](https://plugins.jetbrains.com/plugin/11084-pylint) for JetBrains products like Pycharm. The rules are saved in the `.pylintrc` files. diff --git a/YOLO-detect-buoys/__main__.py b/YOLO-detect-buoys/__main__.py deleted file mode 100644 index 67c0dfe..0000000 --- a/YOLO-detect-buoys/__main__.py +++ /dev/null @@ -1,24 +0,0 @@ -"""The main entry point for YOLO detector buoys.""" - -from os import getcwd, path - -from ultralytics import YOLO -from utils import get_data, get_device, process_video - -if "__main__" == __name__: - device = get_device() - model = YOLO("yolov8n.pt") - - file_path = path.abspath(getcwd()) - dataset = get_data() - - result = model.train( - data=dataset.location + "\\data.yaml", epochs=50, imgsz=640, device=device - ) - - model.val() - - process_video("https://youtu.be/4WGpIOwkLA4?feature=shared", model) - -# References: -# https://docs.ultralytics.com/quickstart/#install-ultralytics diff --git a/YOLO-detect-buoys/assets_docs/gifs-docs/get_api_key.gif b/YOLO-detect-buoys/assets_docs/gifs-docs/get_api_key.gif deleted file mode 100644 index 0c21238..0000000 Binary files a/YOLO-detect-buoys/assets_docs/gifs-docs/get_api_key.gif and /dev/null differ diff --git a/YOLO-detect-buoys/assets_docs/gifs-docs/open_settings.gif b/YOLO-detect-buoys/assets_docs/gifs-docs/open_settings.gif deleted file mode 100644 index c17ca44..0000000 Binary files a/YOLO-detect-buoys/assets_docs/gifs-docs/open_settings.gif and /dev/null differ diff --git a/YOLO-detect-buoys/assets_docs/images/settings-side-menu.png b/YOLO-detect-buoys/assets_docs/images/settings-side-menu.png deleted file mode 100644 index 7eb8096..0000000 Binary files a/YOLO-detect-buoys/assets_docs/images/settings-side-menu.png and /dev/null differ diff --git a/YOLO-detect-buoys/assets_docs/images/up_right_menu.png b/YOLO-detect-buoys/assets_docs/images/up_right_menu.png deleted file mode 100644 index 930c99a..0000000 Binary files a/YOLO-detect-buoys/assets_docs/images/up_right_menu.png and /dev/null differ diff --git a/YOLO-detect-buoys/tests/__init__.py b/YOLO-detect-buoys/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/YOLO-detect-buoys/tests/test_utils.py b/YOLO-detect-buoys/tests/test_utils.py deleted file mode 100644 index e0268b0..0000000 --- a/YOLO-detect-buoys/tests/test_utils.py +++ /dev/null @@ -1,36 +0,0 @@ -"""Module for the utility functions.""" - -from os import getcwd, path - -from dotenv import load_dotenv -from torch import cuda, device -from utils import get_data, get_device - - -def test_get_device() -> None: - """Test the get_device function. - - This makes sure that if the function is changed, the tests verifies that - the function returns the expected device. - """ - expected_device = ( - device(device="cuda") if cuda.is_available() else device(device="cpu") - ) - assert get_device() == expected_device - - -def test_get_data() -> None: - """Test the get_data function. - - This makes sure that if the function is changed, the tests verifies that - the function returns the expected dataset location. - """ - load_dotenv() - - dataset = get_data() - - expected_location = path.join(getcwd(), "data") - - assert path.normpath(dataset.location) == path.normpath(expected_location), ( - "The dataset location is not correct." - ) diff --git a/YOLO-detect-buoys/utils/__init__.py b/YOLO-detect-buoys/utils/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/YOLO-detect-buoys/utils/get_data.py b/YOLO-detect-buoys/utils/get_data.py deleted file mode 100644 index 58e3aab..0000000 --- a/YOLO-detect-buoys/utils/get_data.py +++ /dev/null @@ -1,38 +0,0 @@ -# Path: YOLO-detect-buoys/utils/get_data.py - -"""Get data from Roboflow.""" - -from os import getenv - -from dotenv import load_dotenv -from roboflow import Roboflow -from roboflow.core.dataset import Dataset - - -def get_data_roboflow() -> Dataset: - """Get data from Roboflow. - - Do not expose the API key in the code. - """ - # import environment variables - - load_dotenv() - - rf = Roboflow(api_key=getenv("ROBOFLOW_API_KEY")) - project = rf.workspace(the_workspace=getenv("WORKSPACE")).project( - project_id=getenv("ROBOFLOW_PROJECT_ID") - ) - version = project.version(version_number=int(getenv("ROBOFLOW_PROJECT_VERSION"))) - dataset = version.download( - model_format=getenv("DATASET_FORMAT"), overwrite=True, location="./data" - ) - - return dataset - - -def get_data(*args, **kwargs): - """Get data from Roboflow. - - Interface function for any method of getting data. - """ - return get_data_roboflow(*args, **kwargs) diff --git a/YOLO-detect-buoys/utils/get_device.py b/YOLO-detect-buoys/utils/get_device.py deleted file mode 100644 index 9852d02..0000000 --- a/YOLO-detect-buoys/utils/get_device.py +++ /dev/null @@ -1,14 +0,0 @@ -"""This module provides a utility function to get the device to use for training and inference.""" - -import torch - - -def get_device() -> torch.device: - """Get the device to use for training and inference. - - https://stackoverflow.com/questions/48152674/how-do-i-check-if-pytorch-is-using-the-gpu - - Returns: - torch.device: The device to use for training and inference. - """ - return torch.device(device="cuda" if torch.cuda.is_available() else "cpu") diff --git a/YOLO-detect-buoys/utils/process_video.py b/YOLO-detect-buoys/utils/process_video.py deleted file mode 100644 index 3fd6388..0000000 --- a/YOLO-detect-buoys/utils/process_video.py +++ /dev/null @@ -1,86 +0,0 @@ -"""The function to use YOLOv8 to process a video.""" - -from collections import defaultdict - -from cv2 import VideoCapture, destroyAllWindows, imshow, polylines, waitKey -from numpy import hstack, int32 -from pafy import new -from ultralytics import YOLO - - -def process_video(url: str, model: YOLO) -> None: - """Process a video with YOLOv8 tracking. - - Args: - url (str): The URL of the video to process. - model (YOLO): The YOLOv8 model to use for tracking. - - Returns: - None: This function saves the processed video locally. - - Example: - process_video("https://youtu.be/4WGpIOwkLA4?feature=shared", model) - """ - video = new(url) - best = video.getbest(preftype="mp4") - - cap = VideoCapture(best.url) - - # Store the track history - track_history = defaultdict(lambda: []) - - # Loop through the video frames - while cap.isOpened(): - # Read a frame from the video - success, frame = cap.read() - - if success: - # Run YOLOv8 tracking on the frame, persisting tracks between frames - # https://docs.ultralytics.com/modes/track/ - results = model.track(frame, persist=True) - - # makes sure the boxes and track IDs are not None - if results[0].boxes is not None and results[0].boxes.id is not None: - # Get the boxes and track IDs - boxes = results[0].boxes.xywh.cpu() - track_ids = results[0].boxes.id.int().cpu().tolist() - - # Visualize the results on the frame - annotated_frame = results[0].plot() - - # Plot the tracks - for box, track_id in zip(boxes, track_ids): - x_val, y_val, _, _ = box - track = track_history[track_id] - track.append((float(x_val), float(y_val))) # x, y center point - if len(track) > 30: # retain 90 tracks for 90 frames - track.pop(0) - - # Draw the tracking lines - points = ( - hstack(track).astype(int32).reshape((-1, 1, 2)) - ) # finally this works! 😵‍💫 - - polylines( - annotated_frame, - [points], - isClosed=False, - color=(230, 230, 230), - thickness=10, - ) - else: - annotated_frame = frame - - # Display the annotated frame - imshow("YOLOv8 Tracking", annotated_frame) - - # Break the loop if 'q' is pressed - if waitKey(1) & 0xFF == ord("q"): - break - else: - # Break the loop if the end of the video is reached - break - - # Release the video capture object and close the display window - cap.release() - destroyAllWindows() diff --git a/YOLO-detect-buoys/yolo-detection-buoys.ipynb b/YOLO-detect-buoys/yolo-detection-buoys.ipynb deleted file mode 100644 index 914f04e..0000000 --- a/YOLO-detect-buoys/yolo-detection-buoys.ipynb +++ /dev/null @@ -1,190 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "import cv2\n", - "from ultralytics import YOLO" - ] - }, - { - "cell_type": "code", - "execution_count": 42, - "metadata": {}, - "outputs": [], - "source": [ - "model = YOLO(\"./yolov8n.pt\")" - ] - }, - { - "cell_type": "code", - "execution_count": 32, - "metadata": {}, - "outputs": [], - "source": [ - "def predict(chosen_model, img, classes=[], conf=0.5):\n", - " if classes:\n", - " results = chosen_model.predict(img, classes=classes, conf=conf)\n", - " else:\n", - " results = chosen_model.predict(img, conf=conf)\n", - "\n", - " return results\n", - "\n", - "\n", - "def predict_and_detect(chosen_model, img, classes=[], conf=0.5):\n", - " results = predict(chosen_model, img, classes, conf=conf)\n", - "\n", - " print(results)\n", - " for result in results:\n", - " for box in result.boxes:\n", - " cv2.rectangle(\n", - " img,\n", - " (int(box.xyxy[0][0]), int(box.xyxy[0][1])),\n", - " (int(box.xyxy[0][2]), int(box.xyxy[0][3])),\n", - " (255, 0, 0),\n", - " 2,\n", - " )\n", - " cv2.putText(\n", - " img,\n", - " f\"{result.names[int(box.cls[0])]}\",\n", - " (int(box.xyxy[0][0]), int(box.xyxy[0][1]) - 10),\n", - " cv2.FONT_HERSHEY_PLAIN,\n", - " 1,\n", - " (255, 0, 0),\n", - " 1,\n", - " )\n", - " return img, results" - ] - }, - { - "cell_type": "code", - "execution_count": 40, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n", - "0: 640x640 (no detections), 32.0ms\n", - "Speed: 6.5ms preprocess, 32.0ms inference, 2.0ms postprocess per image at shape (1, 3, 640, 640)\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "ultralytics.engine.results.Results object with attributes:\n", - "\n", - "boxes: ultralytics.engine.results.Boxes object\n", - "keypoints: None\n", - "masks: None\n", - "names: {0: 'person', 1: 'bicycle', 2: 'car', 3: 'motorcycle', 4: 'airplane', 5: 'bus', 6: 'train', 7: 'truck', 8: 'boat', 9: 'traffic light', 10: 'fire hydrant', 11: 'stop sign', 12: 'parking meter', 13: 'bench', 14: 'bird', 15: 'cat', 16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe', 24: 'backpack', 25: 'umbrella', 26: 'handbag', 27: 'tie', 28: 'suitcase', 29: 'frisbee', 30: 'skis', 31: 'snowboard', 32: 'sports ball', 33: 'kite', 34: 'baseball bat', 35: 'baseball glove', 36: 'skateboard', 37: 'surfboard', 38: 'tennis racket', 39: 'bottle', 40: 'wine glass', 41: 'cup', 42: 'fork', 43: 'knife', 44: 'spoon', 45: 'bowl', 46: 'banana', 47: 'apple', 48: 'sandwich', 49: 'orange', 50: 'broccoli', 51: 'carrot', 52: 'hot dog', 53: 'pizza', 54: 'donut', 55: 'cake', 56: 'chair', 57: 'couch', 58: 'potted plant', 59: 'bed', 60: 'dining table', 61: 'toilet', 62: 'tv', 63: 'laptop', 64: 'mouse', 65: 'remote', 66: 'keyboard', 67: 'cell phone', 68: 'microwave', 69: 'oven', 70: 'toaster', 71: 'sink', 72: 'refrigerator', 73: 'book', 74: 'clock', 75: 'vase', 76: 'scissors', 77: 'teddy bear', 78: 'hair drier', 79: 'toothbrush'}\n", - "orig_img: array([[[207, 177, 148],\n", - " [210, 180, 151],\n", - " [199, 170, 143],\n", - " ...,\n", - " [188, 164, 128],\n", - " [182, 156, 119],\n", - " [204, 177, 140]],\n", - "\n", - " [[205, 178, 152],\n", - " [205, 178, 152],\n", - " [195, 168, 142],\n", - " ...,\n", - " [192, 169, 131],\n", - " [203, 177, 140],\n", - " [196, 169, 132]],\n", - "\n", - " [[189, 165, 145],\n", - " [192, 171, 150],\n", - " [181, 160, 139],\n", - " ...,\n", - " [199, 176, 138],\n", - " [204, 178, 141],\n", - " [193, 167, 127]],\n", - "\n", - " ...,\n", - "\n", - " [[106, 96, 96],\n", - " [115, 105, 105],\n", - " [122, 112, 112],\n", - " ...,\n", - " [113, 109, 114],\n", - " [114, 110, 115],\n", - " [149, 145, 150]],\n", - "\n", - " [[119, 109, 109],\n", - " [133, 123, 123],\n", - " [142, 132, 132],\n", - " ...,\n", - " [129, 125, 130],\n", - " [122, 118, 123],\n", - " [142, 138, 143]],\n", - "\n", - " [[131, 121, 121],\n", - " [129, 119, 119],\n", - " [123, 113, 113],\n", - " ...,\n", - " [128, 124, 129],\n", - " [111, 107, 112],\n", - " [112, 108, 113]]], dtype=uint8)\n", - "orig_shape: (640, 640)\n", - "path: 'image0.jpg'\n", - "probs: None\n", - "save_dir: None\n", - "speed: {'preprocess': 6.5021514892578125, 'inference': 31.9974422454834, 'postprocess': 2.001523971557617}\n" - ] - } - ], - "source": [ - "image = cv2.imread(\n", - " \"C:\\\\Users\\Yauhen\\\\vortex-image-processing\\\\data\\\\test\\images\\\\20230611_112730_jpg.rf.a539b08490107377fbdca22df90bd5ee.jpg\"\n", - ")\n", - "print(model.predict(image)[0])" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "-1" - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "venv", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.4" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/docs/Makefile b/docs/Makefile deleted file mode 100644 index d4bb2cb..0000000 --- a/docs/Makefile +++ /dev/null @@ -1,20 +0,0 @@ -# Minimal makefile for Sphinx documentation -# - -# You can set these variables from the command line, and also -# from the environment for the first two. -SPHINXOPTS ?= -SPHINXBUILD ?= sphinx-build -SOURCEDIR = . -BUILDDIR = _build - -# Put it first so that "make" without argument is like "make help". -help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -.PHONY: help Makefile - -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/conf.py b/docs/conf.py deleted file mode 100644 index b213ea2..0000000 --- a/docs/conf.py +++ /dev/null @@ -1,35 +0,0 @@ -# pylint: skip-file -import os -import sys - -# Add project directory to sys.path for autodoc to find your modules -sys.path.insert(0, os.path.abspath("../YOLO-detect_buoys")) - -# Project information -project = "vortex image build" -copyright = "2024, Yauhen Yavorski" -author = "Yauhen Yavorski" - -# General configuration -extensions = [ - "sphinx.ext.autodoc", # Automatically document docstrings - "sphinx.ext.napoleon", # Support for Google-style and NumPy-style docstrings - "sphinx.ext.viewcode", # Link to source code in the docs - "sphinx.ext.autosummary", # Generate summaries of functions/classes - "myst_parser", # Support for Markdown files (optional) -] - -# Automatically generate autosummary pages -autosummary_generate = True - -# Napoleon settings (optional) -napoleon_google_docstring = True -napoleon_numpy_docstring = True - -# Templates and exclude patterns -templates_path = ["_templates"] -exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] - -# Options for HTML output (theme and static paths) -html_theme = "sphinx_rtd_theme" # Change to ReadTheDocs theme -html_static_path = ["_static"] diff --git a/docs/index.rst b/docs/index.rst deleted file mode 100644 index 4f40c02..0000000 --- a/docs/index.rst +++ /dev/null @@ -1,17 +0,0 @@ -.. vortex image build documentation master file, created by - sphinx-quickstart on Fri Sep 27 13:37:08 2024. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -vortex image build documentation -================================ - -.. toctree:: - :maxdepth: 2 - - ../YOLO-detect-buoys/* - -Indices and tables -================== -* :ref:`genindex` -* :ref:`modindex` diff --git a/docs/make.bat b/docs/make.bat deleted file mode 100644 index 954237b..0000000 --- a/docs/make.bat +++ /dev/null @@ -1,35 +0,0 @@ -@ECHO OFF - -pushd %~dp0 - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set SOURCEDIR=. -set BUILDDIR=_build - -%SPHINXBUILD% >NUL 2>NUL -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.https://www.sphinx-doc.org/ - exit /b 1 -) - -if "%1" == "" goto help - -%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% -goto end - -:help -%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% - -:end -popd diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index 298b8ab..0000000 Binary files a/requirements.txt and /dev/null differ diff --git a/yolo_object_detection_roboflow_training/Job.slurm b/yolo_object_detection_roboflow_training/Job.slurm new file mode 100644 index 0000000..e1a2dbe --- /dev/null +++ b/yolo_object_detection_roboflow_training/Job.slurm @@ -0,0 +1,40 @@ +#!/bin/bash +#SBATCH --partition=GPUQ +#SBATCH --account=studiegrupper-vortex +#SBATCH --job-name=vortex-obj-detect-train +#SBATCH --time=1:00:00 +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --gres=gpu:a100:1 +#SBATCH --constraint="gpu40g|gpu80g|gpu32g" +#SBATCH --mem=32G +#SBATCH --output=%x_%j.out + +set -euo pipefail + +export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}" + +# Helps reduce fragmentation for some training runs +export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" + +module purge +module --ignore_cache load foss/2022a +module --ignore_cache load Python/3.10.4-GCCcore-11.3.0 + +VENV_BASE="${SLURM_TMPDIR:-/tmp}" +if ! VENV_DIR="$(mktemp -d -p "$VENV_BASE" venv-objdetect-XXXXXXXX)"; then + echo "Error: Failed to create temporary virtualenv directory under '$VENV_BASE'." >&2 + exit 1 +fi + +python3 -m venv "$VENV_DIR" +source "$VENV_DIR/bin/activate" + +python -m pip install --upgrade pip +python -m pip install --no-cache-dir -r requirements.txt + +srun python train.py + +deactivate +rm -rf "$VENV_DIR" diff --git a/yolo_object_detection_roboflow_training/README.md b/yolo_object_detection_roboflow_training/README.md new file mode 100644 index 0000000..60dcc07 --- /dev/null +++ b/yolo_object_detection_roboflow_training/README.md @@ -0,0 +1,43 @@ +# Object detection training +Scripts for training an object detection model using YOLOv8 on Roboflow datasets. + +## Environment variables +Training requires a Roboflow API key. +Set it **once per shell session** before submitting the Slurm job: +```bash +export ROBOFLOW_API_KEY= +``` + +You can verify it is set with: +```bash +echo $ROBOFLOW_API_KEY +``` +**Do not commit your API key to Git.** + +## Running the SLURM job +1. SSH into the cluster (IDUN) +2. Navigate to the project directory +3. Submit the job: +```bash +sbatch --export=ALL,ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY Job.slurm +``` +After submission, Slurm will print a job ID. + +## Monitoring the job +- Check job status: +```bash +squeue -j +``` +- To list all your jobs: +```bash +squeue -u $USER +``` +- Follow the job output: +```bash +tail -f $(ls -t vortex-obj-detect-train_*.out | head -n 1) +``` + +## Canceling a job +```bash +scancel +``` diff --git a/yolo_object_detection_roboflow_training/requirements.txt b/yolo_object_detection_roboflow_training/requirements.txt new file mode 100644 index 0000000..ab07fc6 --- /dev/null +++ b/yolo_object_detection_roboflow_training/requirements.txt @@ -0,0 +1,12 @@ +huggingface_hub==0.23.3 +numpy==1.26.4 +onnx>=1.12.0 +onnxruntime>=1.16.0 +opencv_contrib_python==4.9.0.80 +opencv_python==4.9.0.80 +pafy==0.5.5 +protobuf==4.24.0 +python-dotenv==1.0.1 +roboflow==1.1.24 +torch==2.2.1 +ultralytics==8.0.196 diff --git a/yolo_object_detection_roboflow_training/train.py b/yolo_object_detection_roboflow_training/train.py new file mode 100644 index 0000000..d34fe6c --- /dev/null +++ b/yolo_object_detection_roboflow_training/train.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +"""Train a YOLOv8 object detection model using a Roboflow dataset.""" + +import os +from pathlib import Path + +import torch +from roboflow import Roboflow +from ultralytics import YOLO + +ROBOFLOW_WORKSPACE_NAME = "hei-qp1ee" +ROBOFLOW_PROJECT_NAME = "simulatorvalve-4rcbu" +ROBOFLOW_PROJECT_VERSION = "1" + +# Use a pretrained YOLOv8m model to avoid training from scratch. +# TODO: Test different YOLOv8 model sizes. +MODEL_NAME = "yolov8m" + +# Specify the GPU device index to use for training. +DEVICE = 0 + + +def main() -> None: + if not torch.cuda.is_available(): + raise RuntimeError("CUDA is required but not available.") + + print("CUDA version:", torch.version.cuda) + print("cuDNN version:", torch.backends.cudnn.version()) + print("Using GPU:", torch.cuda.get_device_name(0)) + + # The Roboflow API key is loaded from the environment to avoid hard-coding secrets. + try: + roboflow_api_key = os.environ["ROBOFLOW_API_KEY"] + except KeyError as e: + raise RuntimeError( + "ROBOFLOW_API_KEY must be set as an environment variable" + ) from e + + rf = Roboflow(api_key=roboflow_api_key) + project = rf.workspace(ROBOFLOW_WORKSPACE_NAME).project(ROBOFLOW_PROJECT_NAME) + + dataset = project.version(ROBOFLOW_PROJECT_VERSION).download("yolov8") + data_yaml_path = Path(dataset.location) / "data.yaml" + + experiment_name = ( + f"{ROBOFLOW_PROJECT_NAME}_v{ROBOFLOW_PROJECT_VERSION}_{MODEL_NAME}" + ) + + model = YOLO(f"{MODEL_NAME}.pt") + + model.train( + data=str(data_yaml_path), + epochs=200, + imgsz=640, + batch=16, + device=DEVICE, + workers=8, + project=Path("results") / "yolov8", + name=experiment_name, + ) + + # Load the best checkpoint produced during training. + best_weights_path = ( + Path("results") / "yolov8" / experiment_name / "weights" / "best.pt" + ) + if not best_weights_path.exists(): + raise FileNotFoundError( + f"Trained model checkpoint not found at {best_weights_path}. " + "Ensure training completed successfully before validation and export." + ) + + trained_model = YOLO(str(best_weights_path)) + + metrics = trained_model.val(data=str(data_yaml_path)) + print("Validation metrics:", metrics) + + # Export formats for deployment + for fmt in ["onnx"]: + trained_model.export(format=fmt, device=DEVICE) + + +if __name__ == "__main__": + main() diff --git a/yolo_segmentation_training/Job.slurm b/yolo_segmentation_training/Job.slurm index 2ca1c76..cb431be 100644 --- a/yolo_segmentation_training/Job.slurm +++ b/yolo_segmentation_training/Job.slurm @@ -37,4 +37,4 @@ python -m pip install --no-cache-dir -r requirements.txt srun python train.py deactivate -rm -rf "$VENV_DIR" \ No newline at end of file +rm -rf "$VENV_DIR" diff --git a/yolo_segmentation_training/config.yaml b/yolo_segmentation_training/config.yaml index dfb0b12..9b56e85 100644 --- a/yolo_segmentation_training/config.yaml +++ b/yolo_segmentation_training/config.yaml @@ -3,7 +3,7 @@ project_id: "pipe-line-instance-seg-rxqdn" workspace_id: "hei-qp1ee" version: "1" # Dataset version number (exclude 'v') # TODO: test and compare pretrained models and look at accuracy vs. speed tradeoff -# Currently v8 nano model because it is the smalles and fastest to train. Consider trying s or m versions to capture more complexity +# Currently v8 nano model because it is the smallest and fastest to train. Consider trying s or m versions to capture more complexity model_type: "yolov8n-seg.pt" epochs: 5 patience: 2 diff --git a/yolo_segmentation_training/requirements.txt b/yolo_segmentation_training/requirements.txt index c6873a7..ab07fc6 100644 --- a/yolo_segmentation_training/requirements.txt +++ b/yolo_segmentation_training/requirements.txt @@ -9,4 +9,4 @@ protobuf==4.24.0 python-dotenv==1.0.1 roboflow==1.1.24 torch==2.2.1 -ultralytics==8.0.196 \ No newline at end of file +ultralytics==8.0.196 diff --git a/yolo_segmentation_training/train.py b/yolo_segmentation_training/train.py index b0066ae..f0f3a2f 100644 --- a/yolo_segmentation_training/train.py +++ b/yolo_segmentation_training/train.py @@ -1,9 +1,9 @@ import os +from datetime import datetime, timezone from pathlib import Path import torch import yaml -from datetime import datetime, timezone from dotenv import load_dotenv from roboflow import Roboflow from ultralytics import YOLO @@ -48,11 +48,9 @@ rf = Roboflow(api_key=ROBOFLOW_API_KEY) project = rf.workspace(WORKSPACE_ID).project(PROJECT_ID) version = project.version(VERSION) - + dataset = version.download( - DATASET_FORMAT, - location=str(dataset_dir), - overwrite=True + DATASET_FORMAT, location=str(dataset_dir), overwrite=True ) else: print("Dataset already exists. Skipping download.")