This project, the Machine Learning Testbed Platform, was initially conceived and developed through an experimental process leveraging advanced AI models (Gemini Flash 2.0 and 2.5). The objective was to explore the capabilities of AI in bootstrapping a functional software project from inception to a foundational state within a constrained timeframe (three weeks) and resource environment (free tier). The successful outcome demonstrates the potential for AI assistance in accelerating initial development phases.
It is important to note that no code in this project was written by a human; the entire codebase was generated by the AI.
The platform is designed to provide a flexible and extensible environment for conducting Machine Learning experiments. It facilitates the core workflows of model training, validation, and testing within a structured framework.
The primary goal is to establish a robust and adaptable foundation for ML experimentation. The platform is being built incrementally, focusing on core functionalities first, with subsequent layers adding more advanced features and capabilities. The long-term vision is a platform capable of supporting diverse ML tasks and research initiatives.
The project is structured to promote modularity and maintainability, built with Python and PyTorch, managed with Poetry, and configurable via YAML. Key components include:
- Data Handling (
src/data): Modules for processing and managing datasets, including loaders for common benchmarks like CIFAR10, CIFAR100, and MNIST. Features a caching mechanism for transformed data. - Training Engine (
src/engine): Contains the core logic for ML experiments. This includes configurable loss functions (src/engine/losses/), various optimizers (Adam, SGD, RMSprop insrc/engine/optimizers/), and a robust training and evaluation loop (src/engine/trainer.py) with metric computation (src/engine/metrics.py). - Models (
src/models): Houses implementations of machine learning models, starting with foundational structures like aSimpleNN. - Testing (
tests): A comprehensive test suite covering various modules to ensure the reliability and correctness of the platform's components. - Documentation (
docs): Comprehensive guides covering project setup, usage, development guidelines, and user stories.
The project development has followed an iterative approach, with implementation steps tracked comprehensively in the git commit history. The codebase is largely covered by automated tests to ensure stability and correctness as features are added and refined.
This project was developed over three weeks using AI assistance, primarily with Gemini Flash models (2.0 and 2.5) via the Roo Code tool. The process involved significant trial and error, requiring multiple restarts from scratch when the AI generated overly complex or unrecoverable code.
An iterative development strategy proved essential. Defining clear guiding documents, such as the Developer Guide, User Stories, and User Stories Status, was crucial for keeping the AI on track and maintaining code quality. However, this approach necessitated constant review and validation of the AI's output at each step.
Challenges were encountered when the AI struggled to fix failing tests, sometimes requiring manual intervention to rollback or delete problematic tests and then regenerate them with the AI.
A key factor in the project's feasibility was the use of free-tier AI models, resulting in zero development cost. The transition from Cline to Roo Code was made due to Roo Code's effective handling of Google API throttling via its rate limiter, which was a significant issue with Cline.
It is worth noting that all documentation within this directory was generated by the AI during the project's development.
The docs/ directory contains several documents providing detailed information about the project:
Developer Guide: Guidelines and standards for contributing to the project, including testing requirements and commit message conventions.Project Description: An overview of the project's goals, incremental development approach, key features, and technical stack.Project Setup Description: Instructions for setting up the project environment.User Guide: Detailed instructions on configuring and running the training script, including available parameters and an example configuration.User Stories: The complete list of user stories guiding the incremental development.User Stories Status: Tracks the completion status of the defined user stories.
Refer to the documentation in the docs/ directory for detailed instructions on setting up the project and getting started with running experiments.
Contributions are welcome! Please refer to the developer guide in the docs/ directory for guidelines on how to contribute.
This project is licensed under the MIT License - see the LICENSE.md file for details.