Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
d92c65d
Project Scaffolding and Development Infrastructure: Add development c…
neomatamune Mar 24, 2025
75ef822
Core Project Structure Reorganization: Remove old structure and updat…
neomatamune Mar 24, 2025
da1fd47
AWS Integration: Add S3 functionality with tests
neomatamune Mar 24, 2025
ed7a4d9
Azure Storage Integration: Add blob, functions, and storage functiona…
neomatamune Mar 24, 2025
c74997f
Azure Data Explorer (ADX) Integration: Add authentication, ingestion,…
neomatamune Mar 24, 2025
b7f20ac
CosmoTech API Core Updates: Update connection, workspace, and run fun…
neomatamune Mar 24, 2025
1388953
CosmoTech API Extensions: Add run_data, run_template, and parameters …
neomatamune Mar 24, 2025
3917438
Dataset Functionality: Add dataset converters, download capabilities,…
neomatamune Mar 24, 2025
f9c51e7
Runner Module Implementation: Add runner functionality with data, dat…
neomatamune Mar 24, 2025
27db0ba
Twin Data Layer Tests: Add comprehensive tests for twin data layer fu…
neomatamune Mar 24, 2025
9481a51
PostgreSQL Integration: Add PostgreSQL runner and store functionality…
neomatamune Mar 24, 2025
95e9b38
SingleStore Integration: Add SingleStore store functionality with tests
neomatamune Mar 24, 2025
a8eee1e
Store Module Enhancements: Update CSV, Pandas, PyArrow, and native Py…
neomatamune Mar 24, 2025
0ce8edf
CSM Engine Updates: Update CSM engine functionality with tests
neomatamune Mar 24, 2025
67cf729
Utility Module Updates: Update API and PostgreSQL utilities with tests
neomatamune Mar 24, 2025
1aa55f3
CSM Data Module: Implement CSM data module structure and command impl…
neomatamune Mar 24, 2025
0501add
Orchestrator Plugin Templates: Update plugin templates and reorganize…
neomatamune Mar 24, 2025
76780d7
Translation and Internationalization: Add translation infrastructure …
neomatamune Mar 24, 2025
5d05276
Documentation and Tutorials: Update documentation and add comprehensi…
neomatamune Mar 24, 2025
afdea2d
Cleanup: Remove deprecated data samples and update configuration files
neomatamune Mar 24, 2025
99b36b2
Final Cleanup: Remove old CLI structure and runner.py
neomatamune Mar 24, 2025
1eb993f
Apply black formating
neomatamune Mar 24, 2025
54edfb1
Remove comitted vscode configuration + add sonarqube config
neomatamune Apr 2, 2025
2632723
wip commit : adx store ingestion
neomatamune Apr 10, 2025
e6a6aee
Updates for adx ingestion speed and on error rollback
neomatamune Apr 18, 2025
409d682
Reorganized adx_send_data code
neomatamune Apr 22, 2025
75a849f
Separate content of csm-data and coal for adx_send_data
neomatamune Apr 22, 2025
d949d8a
remove ADXQueriesWrapper
neomatamune Apr 22, 2025
9e0c3ba
Add more logging
neomatamune Apr 23, 2025
2818492
Only kept en-US i18n for now, separated the big file in smaller ones …
neomatamune Apr 23, 2025
24c9041
correct typo
neomatamune Apr 23, 2025
e39e2b9
Moved to cleaner log translation organization
neomatamune Apr 24, 2025
e03e5ff
Add remaining translations
neomatamune Apr 24, 2025
448f38e
remove unused parameter
neomatamune Apr 24, 2025
1fbd8c7
Update version to 1.0.0
neomatamune Apr 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
18 changes: 18 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[run]
source = cosmotech.coal
omit =
*/tests/*
*/site-packages/*
*/__pycache__/*

[report]
exclude_lines =
pragma: no cover
def __repr__
precision = 2
skip_covered = true

[html]
directory = coverage_html_report
skip_covered = false
skip_empty = true
31 changes: 31 additions & 0 deletions .github/workflows/check_untested_functions.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Check Untested Functions

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
check-untested-functions:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[test]"

- name: Run untested functions check
run: |
python find_untested_functions.py
# This is informational only for now, will not fail the build
# In the future, we can make this fail the build by adding:
# exit $(python find_untested_functions.py | grep "Total untested functions:" | awk '{print $4}')
24 changes: 24 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: Lint

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.dev.txt
- name: Check code formatting with Black
run: |
black --check .
35 changes: 0 additions & 35 deletions .github/workflows/linter.yml

This file was deleted.

28 changes: 28 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Run Tests

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
test:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[test]"

- name: Run tests with coverage
run: |
pytest tests/unit/coal/ --cov=cosmotech.coal --cov-report=term
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -131,4 +131,8 @@ dmypy.json
tmp/
.idea/

generated/
generated/
.vscode/

# sonarcube folders
.scannerwork/
27 changes: 27 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-added-large-files

- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black

- repo: local
hooks:
- id: pytest-check
name: pytest-check
entry: pytest tests/unit/coal/ --cov=cosmotech.coal --cov-report=term-missing --cov-fail-under=90
language: system
pass_filenames: false
always_run: true
- id: check-untested-functions
name: check-untested-functions
entry: python find_untested_functions.py
language: system
pass_filenames: false
always_run: true
111 changes: 111 additions & 0 deletions BLACK_SETUP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Black Code Formatter Setup

This project uses [Black](https://github.com/psf/black) for code formatting to ensure consistent code style across the codebase.

## Configuration

Black is configured in the `pyproject.toml` file with the following settings:

```toml
[tool.black]
line-length = 120
target-version = ["py311"]
include = '\.pyi?$'
exclude = '''
/(
\.git
| \.hg
| \.mypy_cache
| \.tox
| \.venv
| _build
| buck-out
| build
| dist
| generated
| __pycache__
)/
'''
```

## Installation

Install Black and other development tools:

```bash
pip install -r requirements.dev.txt
```

Or install Black directly:

```bash
pip install black==23.3.0
```

## Usage

### Manual Usage

Run Black on your codebase:

```bash
# Format all Python files in the project
python -m black .

# Format a specific directory
python -m black cosmotech/coal/

# Check if files would be reformatted without actually changing them
python -m black --check .

# Show diff of changes without writing files
python -m black --diff .
```

### Pre-commit Hooks

This project uses pre-commit hooks to automatically run Black before each commit. To set up pre-commit:

1. Install pre-commit:

```bash
pip install pre-commit
```

2. Install the git hooks:

```bash
pre-commit install
```

Now Black will run automatically on the files you've changed when you commit.

### VSCode Integration

If you use VSCode, the included settings in `.vscode/settings.json` will automatically format your code with Black when you save a file. Make sure you have the Python extension installed.

## CI Integration

To enforce Black formatting in your CI pipeline, add a step to run Black in check mode:

```yaml
- name: Check code formatting with Black
run: |
python -m pip install black==23.3.0
python -m black --check .
```

## Migrating Existing Code

When first applying Black to an existing codebase, you might want to:

1. Run Black with `--diff` first to see the changes
2. Consider running it on one module at a time
3. Make the formatting change in a separate commit from functional changes

## Benefits of Using Black

- Eliminates debates about formatting
- Consistent code style across the entire codebase
- Minimal configuration needed
- Widely adopted in the Python community
Loading