Skip to content

Commit b5018f4

Browse files
BenjaminWattsBenjaminWatts
committed
feat: comprehensive model improvements with enums, mixins, and type safety
Major improvements to generated models: ENUMS (22 types, 210 values): - Added DatasetEnum, PsrtypeEnum, FueltypeEnum, BusinesstypeEnum, and 18 more - Full IDE autocomplete for all valid field values - Type-safe validation prevents invalid values FIELD MIXINS (18 types): - Eliminated 629 repeated field definitions (52.9% reduction) - SettlementFields, BmUnitFields, DocumentFields provide fields + methods - PublishTimeFields, StartTimeFields, DatasetFields, QuantityFields, etc. - Fields now inherited, not repeated across models METHOD MIXINS (29 types): - 141+ helper methods for common operations - Conversions: MW/GW, £/MWh to £/kWh, C/F - Classifications: renewable/fossil, transmission/interconnector - Calculations: spread, utilization, duration REQUIRED FIELDS: - 894 fields (64%) now properly required (was 0%) - Overrides OpenAPI spec's 'everything optional' approach - Core fields like dataset, document_id, publish_time, quantity are required SNAKE_CASE NAMING: - All fields use Pythonic snake_case (publish_time, document_id) - Proper aliases maintain API compatibility (alias='publishTime') - populate_by_name=True allows both formats VALIDATION: - Settlement periods validate short/long days (UK clock changes) - Time ranges ensure logical ordering (time_to > time_from) - Level ranges ensure consistency (level_to >= level_from) - Flow direction validates Up/Down only - Frequency validates 47-53 Hz range SUCCINCT IMPORTS: - Module-level imports: from elexon_bmrs import enums, validators, field_mixins - Aliases for convenience in generated code CODE REDUCTION: - 629 field definitions eliminated (52.9%) - ~2,500 lines of methods eliminated - Total: ~3,100+ lines eliminated - File size: 2,292 lines (was ~3,500+) COVERAGE: - 140 models (50%) use mixins - 47 total mixin types - 22 enum types - 894 required fields Files added: - elexon_bmrs/enums.py - 22 enum types - elexon_bmrs/validators.py - 29 method mixins - elexon_bmrs/field_mixins.py - 18 field mixins - tools/generate_enums.py - enum generator - test_all_improvements.py - comprehensive test suite Files modified: - tools/generate_models.py - enhanced with enum/mixin/snake_case support - elexon_bmrs/generated_models.py - regenerated with all improvements - elexon_bmrs/__init__.py - export enums Breaking changes: - Field names changed from camelCase to snake_case - Many fields changed from Optional to required - String fields changed to enums where appropriate - Use populate_by_name=True for backward compatibility
0 parents  commit b5018f4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

82 files changed

+29932
-0
lines changed

‎.flake8‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
[flake8]
2+
max-line-length = 100
3+
exclude =
4+
.git,
5+
__pycache__,
6+
build,
7+
dist,
8+
.eggs,
9+
*.egg-info,
10+
.venv,
11+
venv,
12+
env
13+
14+
# Ignore specific rules
15+
ignore =
16+
E203, # whitespace before ':'
17+
E501, # line too long (handled by black)
18+
W503, # line break before binary operator
19+
20+
per-file-ignores =
21+
__init__.py:F401
22+

‎.github/workflows/README.md‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
# GitHub Actions Workflows
2+
3+
This directory contains automated CI/CD workflows for the elexon-bmrs project.
4+
5+
## Workflows
6+
7+
### 1. Tests (`test.yml`)
8+
9+
**Triggers:**
10+
- Push to `main` or `develop` branches
11+
- Pull requests to `main`
12+
13+
**What it does:**
14+
- ✅ Tests on Python 3.8, 3.9, 3.10, 3.11, 3.12
15+
- ✅ Code coverage reporting (Codecov)
16+
- ✅ Linting (Black, isort, flake8)
17+
- ✅ Type checking (mypy)
18+
- ✅ Build verification
19+
20+
**Status Badge:**
21+
```markdown
22+
![Tests](https://github.com/benjaminwatts/elexon-bmrs/workflows/Tests/badge.svg)
23+
```
24+
25+
### 2. Publish to PyPI (`publish.yml`)
26+
27+
**Triggers:**
28+
- **Automatic**: When a GitHub Release is published
29+
- **Manual**: Via workflow dispatch (Actions tab)
30+
31+
**What it does:**
32+
- ✅ Builds distribution packages (wheel + sdist)
33+
- ✅ Validates package integrity
34+
- ✅ Publishes to TestPyPI or PyPI
35+
- ✅ Uses trusted publishing (no API tokens needed!)
36+
37+
**Status Badge:**
38+
```markdown
39+
![Publish](https://github.com/benjaminwatts/elexon-bmrs/workflows/Publish%20to%20PyPI/badge.svg)
40+
```
41+
42+
## Setup Instructions
43+
44+
### 1. Enable Trusted Publishing (Recommended)
45+
46+
Trusted Publishing is the modern, secure way to publish to PyPI without API tokens.
47+
48+
#### For PyPI (Production):
49+
50+
1. Go to https://pypi.org/manage/account/publishing/
51+
2. Click "Add a new pending publisher"
52+
3. Fill in:
53+
- **PyPI Project Name**: `elexon-bmrs`
54+
- **Owner**: `benjaminwatts` (your GitHub username)
55+
- **Repository name**: `elexon-bmrs`
56+
- **Workflow name**: `publish.yml`
57+
- **Environment name**: `pypi`
58+
4. Click "Add"
59+
60+
#### For TestPyPI (Testing):
61+
62+
1. Go to https://test.pypi.org/manage/account/publishing/
63+
2. Repeat the same steps as above, but use:
64+
- **Environment name**: `testpypi`
65+
66+
### 2. Configure GitHub Environments (Optional but Recommended)
67+
68+
1. Go to your repository Settings → Environments
69+
2. Create two environments:
70+
- **`testpypi`**: For testing releases
71+
- **`pypi`**: For production releases
72+
3. For the `pypi` environment, add protection rules:
73+
- ✅ Required reviewers (yourself)
74+
- ✅ Wait timer (optional, e.g., 5 minutes to allow cancellation)
75+
76+
### 3. Alternative: Using API Tokens (Legacy Method)
77+
78+
If you prefer not to use trusted publishing:
79+
80+
1. Generate API tokens:
81+
- PyPI: https://pypi.org/manage/account/token/
82+
- TestPyPI: https://test.pypi.org/manage/account/token/
83+
84+
2. Add them as GitHub Secrets:
85+
- Go to repository Settings → Secrets → Actions
86+
- Add secrets:
87+
- `PYPI_API_TOKEN` (for PyPI)
88+
- TEST_PYPI_API_TOKEN` (for TestPyPI)
89+
90+
3. Modify `publish.yml` to use tokens instead of trusted publishing:
91+
```yaml
92+
- name: Publish to PyPI
93+
uses: pypa/gh-action-pypi-publish@release/v1
94+
with:
95+
password: ${{ secrets.PYPI_API_TOKEN }}
96+
```
97+
98+
## Usage
99+
100+
### Publishing a New Release
101+
102+
#### Method 1: GitHub Release (Automatic - Recommended)
103+
104+
1. **Update version** in `pyproject.toml`:
105+
```toml
106+
version = "0.1.0"
107+
```
108+
109+
2. **Update** `CHANGELOG.md`
110+
111+
3. **Commit and push**:
112+
```bash
113+
git add pyproject.toml CHANGELOG.md
114+
git commit -m "Release v0.1.0"
115+
git push origin main
116+
```
117+
118+
4. **Create a git tag**:
119+
```bash
120+
git tag v0.1.0
121+
git push origin v0.1.0
122+
```
123+
124+
5. **Create a GitHub Release**:
125+
- Go to: https://github.com/benjaminwatts/elexon-bmrs/releases/new
126+
- Tag: `v0.1.0`
127+
- Title: `v0.1.0`
128+
- Description: Copy from CHANGELOG.md
129+
- Click "Publish release"
130+
131+
6. **Watch the workflow**:
132+
- Go to Actions tab
133+
- The "Publish to PyPI" workflow will run automatically
134+
- It will publish to PyPI when complete!
135+
136+
#### Method 2: Manual Trigger
137+
138+
1. Go to: https://github.com/benjaminwatts/elexon-bmrs/actions/workflows/publish.yml
139+
2. Click "Run workflow"
140+
3. Choose:
141+
- Branch: `main`
142+
- Environment: `testpypi` or `pypi`
143+
4. Click "Run workflow"
144+
145+
### Testing Before Production Release
146+
147+
**Always test on TestPyPI first!**
148+
149+
1. Use manual trigger with `testpypi` environment
150+
2. Wait for workflow to complete
151+
3. Test installation:
152+
```bash
153+
pip install -i https://test.pypi.org/simple/ \
154+
--extra-index-url https://pypi.org/simple \
155+
elexon-bmrs
156+
```
157+
4. Verify it works:
158+
```bash
159+
python -c "from elexon_bmrs import BMRSClient; print('✓')"
160+
```
161+
5. If successful, create a GitHub Release for production
162+
163+
## Workflow Behavior
164+
165+
### Test Workflow
166+
167+
- **Runs on every push** to `main`/`develop`
168+
- **Runs on every PR** to `main`
169+
- Tests across Python 3.8-3.12
170+
- Fast feedback (~5-10 minutes)
171+
172+
### Publish Workflow
173+
174+
- **Automatic trigger**: GitHub Release published → PyPI
175+
- **Manual trigger**: You choose TestPyPI or PyPI
176+
- Always builds fresh packages
177+
- Always validates before publishing
178+
- Idempotent (safe to re-run)
179+
180+
## Security Features
181+
182+
### ✅ Trusted Publishing
183+
- No API tokens in GitHub Secrets
184+
- Uses OpenID Connect (OIDC)
185+
- PyPI verifies the workflow identity
186+
- Most secure method available
187+
188+
### ✅ Environment Protection
189+
- Production environment can require approval
190+
- Prevents accidental releases
191+
- Can add wait timers
192+
193+
### ✅ Read-only by Default
194+
- Workflows have minimal permissions
195+
- Only `id-token: write` for publishing
196+
- Follows principle of least privilege
197+
198+
## Monitoring
199+
200+
### Check Workflow Status
201+
202+
1. **Actions Tab**: https://github.com/benjaminwatts/elexon-bmrs/actions
203+
2. **Badges**: Add to README.md
204+
3. **Email Notifications**: GitHub sends on failure
205+
206+
### View Published Packages
207+
208+
- **PyPI**: https://pypi.org/project/elexon-bmrs/
209+
- **TestPyPI**: https://test.pypi.org/project/elexon-bmrs/
210+
211+
## Troubleshooting
212+
213+
### "Trusted publishing not configured"
214+
215+
**Solution**: Complete the Trusted Publishing setup (see above)
216+
217+
### "Package already exists"
218+
219+
**Solution**: Bump version in `pyproject.toml` - PyPI doesn't allow overwriting
220+
221+
### "Build failed"
222+
223+
**Solution**: Run locally first:
224+
```bash
225+
make pre-release # Run all checks
226+
make build # Test build
227+
```
228+
229+
### "Tests failing"
230+
231+
**Solution**:
232+
```bash
233+
make test # Run tests locally
234+
make lint # Check linting
235+
make type-check # Check types
236+
```
237+
238+
### "Manual trigger not working"
239+
240+
**Solution**: Ensure you have write access to the repository
241+
242+
## Best Practices
243+
244+
### Before Every Release
245+
246+
1. ✅ Run `make pre-release` locally
247+
2. ✅ Update `CHANGELOG.md`
248+
3. ✅ Bump version in `pyproject.toml`
249+
4. ✅ Test on TestPyPI first
250+
5. ✅ Create git tag
251+
6. ✅ Create GitHub Release
252+
253+
### Versioning
254+
255+
Follow Semantic Versioning:
256+
- `MAJOR.MINOR.PATCH` (e.g., `1.2.3`)
257+
- Increment MAJOR for breaking changes
258+
- Increment MINOR for new features
259+
- Increment PATCH for bug fixes
260+
261+
### Release Notes
262+
263+
Always include in GitHub Release description:
264+
- What's new
265+
- What changed
266+
- What was fixed
267+
- Breaking changes (if any)
268+
- Migration guide (if needed)
269+
270+
## Comparing Approaches
271+
272+
### GitHub Actions vs Manual Publishing
273+
274+
| Feature | GitHub Actions | Manual (`make upload`) |
275+
|---------|---------------|------------------------|
276+
| **Speed** | ⚡ Fast | 🐌 Slower |
277+
| **Consistency** | ✅ Always same | ⚠️ Varies |
278+
| **Security** | ✅ Trusted publishing | ⚠️ Needs tokens |
279+
| **Tracking** | ✅ Full history | ❌ No tracking |
280+
| **Rollback** | ✅ Easy | ❌ Manual |
281+
| **Collaboration** | ✅ Team friendly | ⚠️ Individual |
282+
| **Testing** | ✅ Automatic | ⚠️ Manual |
283+
284+
**Recommendation**: Use GitHub Actions for production, keep `make upload` for emergency fixes.
285+
286+
## Additional Resources
287+
288+
- [GitHub Actions Documentation](https://docs.github.com/en/actions)
289+
- [PyPI Trusted Publishing Guide](https://docs.pypi.org/trusted-publishers/)
290+
- [Python Packaging User Guide](https://packaging.python.org/)
291+
- [Semantic Versioning](https://semver.org/)
292+
293+
---
294+
295+
**Need Help?**
296+
- Check workflow logs in Actions tab
297+
- Review [PYPI_DISTRIBUTION.md](../../PYPI_DISTRIBUTION.md)
298+
- Open an issue: https://github.com/benjaminwatts/elexon-bmrs/issues
299+

‎.github/workflows/docs.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: Documentation
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
branches:
9+
- main
10+
workflow_dispatch:
11+
12+
# Sets permissions for GitHub Pages deployment
13+
permissions:
14+
contents: read
15+
pages: write
16+
id-token: write
17+
18+
# Allow one concurrent deployment
19+
concurrency:
20+
group: "pages"
21+
cancel-in-progress: true
22+
23+
jobs:
24+
build:
25+
runs-on: ubuntu-latest
26+
steps:
27+
- name: Checkout code
28+
uses: actions/checkout@v4
29+
with:
30+
fetch-depth: 0
31+
32+
- name: Set up Python
33+
uses: actions/setup-python@v4
34+
with:
35+
python-version: '3.11'
36+
cache: 'pip'
37+
38+
- name: Install dependencies
39+
run: |
40+
python -m pip install --upgrade pip
41+
pip install -r requirements.txt
42+
pip install -r requirements-dev.txt
43+
44+
- name: Build documentation
45+
run: |
46+
mkdocs build --clean --strict
47+
48+
- name: Upload artifact
49+
uses: actions/upload-pages-artifact@v2
50+
with:
51+
path: ./site
52+
53+
# Deploy job - only runs on push to main
54+
deploy:
55+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
56+
environment:
57+
name: github-pages
58+
url: ${{ steps.deployment.outputs.page_url }}
59+
runs-on: ubuntu-latest
60+
needs: build
61+
steps:
62+
- name: Deploy to GitHub Pages
63+
id: deployment
64+
uses: actions/deploy-pages@v3
65+

0 commit comments

Comments
 (0)