Restructure Repository for Improved Separation of Concerns

[achrefbenammar404]

Issue: Restructure Repository for Improved Separation of Concerns

Description

The current repository structure lacks clear separation between the services for model development (training, testing, etc.) and the services used for predictions within the API application. This organization makes it challenging to manage, maintain, and scale the project, especially as both development and production environments evolve.

Proposed Solution

1. Segregate Model Development and API Services:

   • Create a dedicated model_dev directory to house all components related to model development, including:
     • Data preprocessing for training
     • Model training scripts and configuration
     • Testing and evaluation scripts
     • Experiment tracking (if applicable)
   • Create an api_app directory for all production-related services, including:
     • The API application code and endpoint definitions
     • Model inference and prediction services
     • Any utilities specifically for running the model in production (e.g., loading and managing model instances)

2. Define Clear Interfaces Between Components:

   • Establish clear, documented interfaces for passing the trained model from model_dev to api_app.
   • Use a standardized model export format to allow easy model loading within the API application.

3. Update Documentation:

   • Provide a clear directory structure in the README, specifying the role of each folder.
   • Include instructions for both developers working on model training and engineers focusing on API deployment.

Proposed Directory Structure

The new structure could look something like this:

/repo-root
|-- model_dev/
|   |-- preprocessing/
|   |-- training/
|   |-- evaluation/
|   `-- utils/
|
|-- api_app/
|   |-- prediction_service/
|   |-- api/
|   |-- utils/
|   `-- config/
|
|-- README.md
|-- requirements.txt
`-- other_project_files/

Tasks

• Create model_dev directory with subfolders for training, preprocessing, and evaluation.
• Move all model development files to the model_dev directory.
• Create api_app directory with subfolders for the API and prediction services.
• Move all API-related and prediction service files to api_app.
• Update import paths and adjust configurations to match the new structure.
• Update the README with the new directory structure and purpose of each folder.
• Test both model training and API service to ensure functionality is intact.

Expected Outcome

• A cleaner, more organized repository structure with clear separation between model development and API services.
• Easier navigation and maintenance for both model developers and API engineers.
• A scalable structure to accommodate future development in either model training or API service.

Additional Notes

• Ensure backward compatibility for any existing deployment workflows.
• Test integration points where api_app loads the trained model from model_dev.