Prerelease 1.0.0

[neomatamune]

Major Code Reorganization and Library Structure Improvement

This pull request represents a significant reorganization of the CosmoTech Acceleration Library (CoAL) codebase to improve structure, maintainability, and usability. The changes focus on creating a more coherent architecture, improving documentation, and enhancing test coverage.

Key Changes

1. Package Structure Reorganization

• Renamed and Restructured Modules:

  • Moved from CosmoTech_Acceleration_Library to a more standard Python package structure with cosmotech.coal as the main namespace
  • Reorganized CLI commands from coal/cli to a dedicated csm_data module
  • Created clear separation between core library functionality and CLI tools

• Improved Module Organization:

  • Cloud service integrations (AWS, Azure) are now in dedicated modules
  • Database connectors (PostgreSQL, SingleStore) have their own modules
  • Store implementations are better organized with clear interfaces

2. Enhanced Documentation

• Comprehensive README: Updated with clear explanations of all major components

• New Tutorials: Added detailed tutorials for:

  • Contributing to CoAL
  • Using the CosmoTech API
  • Working with the datastore
  • Using csm-data CLI commands

• API Documentation: Improved documentation for all public APIs

• Pull Request Guidelines: Added clear guidelines for contributors

3. Testing Infrastructure

• Comprehensive Test Suite: Added extensive unit tests for all components

• Test Coverage Tools:

  • Added find_untested_functions.py to identify code without tests
  • Added generate_test_files.py to scaffold test files
  • Added GitHub workflow to check for untested functions

• Coverage Requirements: Established minimum coverage requirements

4. Development Workflow Improvements

• Pre-commit Hooks: Added configuration for code quality checks
• Black Formatting: Standardized code formatting across the codebase
• Linting: Updated GitHub workflows for linting

5. Feature Enhancements

• CosmoTech API Integration: Improved and expanded API client functionality

  • Better authentication handling
  • Enhanced Twin Data Layer support
  • Improved runner and dataset operations

• Cloud Service Support:

  • Enhanced Azure Data Explorer (ADX) integration
  • Added S3 bucket operations
  • Improved Azure Blob storage support

• Data Management:

  • Enhanced store implementations (CSV, Pandas, PyArrow)
  • Improved PostgreSQL and SingleStore integration

6. Internationalization

• Added comprehensive translation support for both English and French
• Structured translation files for all user-facing components

Technical Details

Removed Components

• Removed legacy Modelops module in favor of the new architecture
• Removed outdated scenario download functionality
• Cleaned up unused test data and samples

Added Components

• New AWS S3 integration module
• Enhanced Azure ADX functionality
• Improved CosmoTech API client with better authentication
• New dataset converters and download utilities
• Comprehensive runner operations support

Migration Notes

This PR represents a significant restructuring, but maintains backward compatibility where possible. Users should:

1. Update import statements to use the new module structure
2. Replace any usage of removed components with their new equivalents
3. Review the updated documentation for new features and improvements

Testing and Validation

• All unit tests pass with the new structure
• Test coverage has been significantly improved
• Manual testing has been performed for key functionality

Documentation Updates

• README.md has been completely updated
• New CONTRIBUTING.md guidelines have been added
• Comprehensive tutorials have been created
• API documentation has been updated

This PR addresses the need for a more maintainable and well-structured codebase, making it easier for both users and contributors to work with the CosmoTech Acceleration Library.