Skip to content

Commit 43fd07b

Browse files
xaviviroJustar96
xaviviro
and
Justar96
authored
chore: initial version (#6)
* first commit * first * uv fix * uv fix * uv fix * code formatted for lint * code formatted for lint mypi * fixes * templates * from python-toon to toon_format * Resolve .gitignore merge conflict Keep both reference repositories section and standard Python gitignore structure. Co-authored-by: Justar96 * refactor: add file headers and expand test coverage ## Code Organization - Add Google-style headers to all 18 source files - Copyright (c) 2025 TOON Format Organization - SPDX-License-Identifier: MIT - Comprehensive module docstrings - Format all source code with Ruff ## Test Suite Expansion - Increase test coverage from 78% to 91% (792 tests) - Add comprehensive test modules: - test_security.py: 24 tests for injection prevention and resource exhaustion - test_internationalization.py: 24 tests for Unicode/UTF-8 support - test_cli.py: 30 integration tests for command-line interface - test_scanner.py: 31 tests for scanner module (100% coverage) - test_string_utils.py: 42 tests for string utilities (100% coverage) - test_normalize_functions.py: 37 tests for normalization (95% coverage) - test_parsing_utils.py: Complete parsing utility coverage - Add 306 official spec compliance tests via test_spec_fixtures.py - Create test fixture infrastructure with JSON schema validation ## Files Changed - Modified: All 18 source files in src/toon_format/ - Added: 8 new test modules - Added: Test fixtures and schema - Added: New utility module _parsing_utils.py * Add token counting utilities and update documentation Features: - Add benchmark dependency group with tiktoken>=0.4.0 to pyproject.toml - Export count_tokens, estimate_savings, and compare_formats utilities - Implement token counting using tiktoken with o200k_base encoding (gpt5/gpt5-mini) Documentation Updates: - Add Token Counting & Comparison section to main README with examples - Update docs/README.md with new utility functions in API reference list - Add roadmap section announcing planned comprehensive benchmarks - Add complete Utility Functions section to docs/api.md covering: * count_tokens() - Token counting with tiktoken * estimate_savings() - JSON vs TOON comparison metrics * compare_formats() - Formatted comparison tables - Add Token Efficiency examples with cost estimation patterns - Update LLM integration guide with Measuring Token Savings section - Include cost calculation examples and integration patterns - Update model references from GPT-4 to gpt5 throughout docs - Add benchmark disclaimer noting comprehensive benchmarks coming soon Technical Details: - Update tokenizer documentation from GPT-4o/GPT-4 to gpt5/gpt5-mini - Fix TypedDict usage examples in docs/api.md (EncodeOptions uses dict syntax) - Clarify DecodeOptions is a class while EncodeOptions is a TypedDict - Add toon-spec/ submodule files (CHANGELOG.md and SPEC.md v1.3) * Fix coverage configuration for GitHub Actions * Fix linting errors in test suite * Fix format --------- Co-authored-by: Justar <nalongkon1996@gmail.com> Co-authored-by: Justar96 <justar96@users.noreply.github.com>
1 parent f360486 commit 43fd07b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+12115
-295
lines changed

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,11 +51,11 @@ body:
5151
description: |
5252
Please provide:
5353
- Python version
54-
- toon-format version
54+
- toon_format version (from `pip show toon_format`)
5555
- Operating system
5656
placeholder: |
5757
- Python 3.12.0
58-
- toon-format 0.1.0
58+
- toon_format 1.0.0
5959
- macOS 14.0
6060
validations:
6161
required: true

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 65 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -42,24 +42,83 @@ Closes #
4242

4343
- [ ] All existing tests pass
4444
- [ ] Added new tests for changes
45+
- [ ] Tested on Python 3.8
46+
- [ ] Tested on Python 3.9
47+
- [ ] Tested on Python 3.10
4548
- [ ] Tested on Python 3.11
4649
- [ ] Tested on Python 3.12
47-
- [ ] Tested on Python 3.13
48-
- [ ] Tested on Python 3.14
50+
51+
### Test Output
52+
53+
```bash
54+
# Paste test output here
55+
```
56+
57+
## Code Quality
58+
59+
<!-- Confirm code quality checks -->
60+
61+
- [ ] Ran `ruff check src/toon_format tests` - no issues
62+
- [ ] Ran `ruff format src/toon_format tests` - code formatted
63+
- [ ] Ran `mypy src/toon_format` - no critical errors
64+
- [ ] All tests pass: `pytest tests/ -v`
4965

5066
## Checklist
5167

5268
<!-- Mark completed items with an [x] -->
5369

54-
- [ ] My code follows the project's coding standards
70+
- [ ] My code follows the project's coding standards (PEP 8, line length 100)
5571
- [ ] I have added type hints to new code
56-
- [ ] I have run `ruff check` and `ruff format`
57-
- [ ] I have run `mypy` on my changes
5872
- [ ] I have added tests that prove my fix/feature works
5973
- [ ] New and existing tests pass locally
60-
- [ ] I have updated documentation (if needed)
74+
- [ ] I have updated documentation (README.md, CLAUDE.md if needed)
6175
- [ ] My changes do not introduce new dependencies
76+
- [ ] I have maintained Python 3.8+ compatibility
77+
- [ ] I have reviewed the [TOON specification](https://github.com/toon-format/spec) for relevant sections
78+
79+
## Performance Impact
80+
81+
<!-- If applicable, describe any performance implications -->
82+
83+
- [ ] No performance impact
84+
- [ ] Performance improvement (describe below)
85+
- [ ] Potential performance regression (describe and justify below)
86+
87+
<!-- Details: -->
88+
89+
## Breaking Changes
90+
91+
<!-- If this is a breaking change, describe the migration path for users -->
92+
93+
- [ ] No breaking changes
94+
- [ ] Breaking changes (describe migration path below)
95+
96+
<!-- Migration path: -->
97+
98+
## Screenshots / Examples
99+
100+
<!-- If applicable, add screenshots or example output -->
101+
102+
```python
103+
# Example usage
104+
```
105+
106+
Output:
107+
```
108+
# Example output
109+
```
62110

63111
## Additional Context
64112

65113
<!-- Add any other context about the PR here (optional) -->
114+
115+
## Checklist for Reviewers
116+
117+
<!-- For maintainers reviewing this PR -->
118+
119+
- [ ] Code changes are clear and well-documented
120+
- [ ] Tests adequately cover the changes
121+
- [ ] Documentation is updated
122+
- [ ] No security concerns
123+
- [ ] Follows TOON specification
124+
- [ ] Backward compatible (or breaking changes are justified and documented)

‎.github/dependabot.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Dependabot configuration for automated dependency updates
2+
# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
3+
4+
version: 2
5+
updates:
6+
# Monitor GitHub Actions for updates
7+
- package-ecosystem: "github-actions"
8+
directory: "/"
9+
schedule:
10+
interval: "weekly"
11+
day: "monday"
12+
labels:
13+
- "dependencies"
14+
- "github-actions"
15+
commit-message:
16+
prefix: "ci"
17+
include: "scope"
18+
19+
# Monitor pip dependencies (compatible with uv)
20+
- package-ecosystem: "pip"
21+
directory: "/"
22+
schedule:
23+
interval: "weekly"
24+
day: "monday"
25+
labels:
26+
- "dependencies"
27+
- "python"
28+
commit-message:
29+
prefix: "deps"
30+
include: "scope"
31+
# Group dev dependencies together
32+
groups:
33+
dev-dependencies:
34+
patterns:
35+
- "pytest*"
36+
- "mypy*"
37+
- "ruff*"
38+
update-types:
39+
- "minor"
40+
- "patch"

‎.github/workflows/publish.yml‎

Lines changed: 60 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,28 +3,78 @@ name: Publish to PyPI
33
on:
44
release:
55
types: [published]
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
610

711
jobs:
8-
publish:
9-
name: Publish to PyPI
12+
build:
13+
name: Build distribution
1014
runs-on: ubuntu-latest
11-
permissions:
12-
id-token: write
13-
contents: read
1415

1516
steps:
1617
- uses: actions/checkout@v4
1718

18-
- name: Install uv
19-
uses: astral-sh/setup-uv@v5
20-
2119
- name: Set up Python
2220
uses: actions/setup-python@v5
2321
with:
24-
python-version: "3.12"
22+
python-version: "3.x"
23+
24+
- name: Install build dependencies
25+
run: |
26+
python -m pip install --upgrade pip
27+
pip install build
2528
2629
- name: Build package
27-
run: uv build
30+
run: python -m build
31+
32+
- name: Store distribution packages
33+
uses: actions/upload-artifact@v4
34+
with:
35+
name: python-package-distributions
36+
path: dist/
37+
38+
publish-to-pypi:
39+
name: Publish to PyPI
40+
if: github.event_name == 'release' && github.event.action == 'published'
41+
needs: build
42+
runs-on: ubuntu-latest
43+
environment:
44+
name: pypi
45+
url: https://pypi.org/p/toon_format
46+
permissions:
47+
id-token: write
48+
49+
steps:
50+
- name: Download distributions
51+
uses: actions/download-artifact@v4
52+
with:
53+
name: python-package-distributions
54+
path: dist/
2855

2956
- name: Publish to PyPI
3057
uses: pypa/gh-action-pypi-publish@release/v1
58+
59+
publish-to-testpypi:
60+
name: Publish to TestPyPI
61+
if: github.event_name == 'workflow_dispatch'
62+
needs: build
63+
runs-on: ubuntu-latest
64+
environment:
65+
name: testpypi
66+
url: https://test.pypi.org/p/toon_format
67+
permissions:
68+
id-token: write
69+
70+
steps:
71+
- name: Download distributions
72+
uses: actions/download-artifact@v4
73+
with:
74+
name: python-package-distributions
75+
path: dist/
76+
77+
- name: Publish to TestPyPI
78+
uses: pypa/gh-action-pypi-publish@release/v1
79+
with:
80+
repository-url: https://test.pypi.org/legacy/

‎.github/workflows/test.yml‎

Lines changed: 22 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,17 @@ name: Tests
22

33
on:
44
push:
5-
branches: [main]
5+
branches: [main, develop]
66
pull_request:
7-
branches: [main]
7+
branches: [main, develop]
88

99
jobs:
1010
test:
11-
name: Python ${{ matrix.python-version }} on ${{ matrix.os }}
12-
runs-on: ${{ matrix.os }}
11+
name: Test Python ${{ matrix.python-version }}
12+
runs-on: ubuntu-latest
1313
strategy:
14-
fail-fast: false
1514
matrix:
16-
os: [ubuntu-latest, macos-latest, windows-latest]
17-
python-version: ["3.11", "3.12", "3.13", "3.14"]
15+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
1816

1917
steps:
2018
- uses: actions/checkout@v4
@@ -32,17 +30,23 @@ jobs:
3230
- name: Install dependencies
3331
run: uv sync
3432

35-
- name: Run tests
36-
run: uv run pytest tests/ -v
37-
3833
- name: Run tests with coverage
39-
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
40-
run: |
41-
uv run pytest tests/ --cov=src/toon_format --cov-report=xml --cov-report=term-missing
34+
run: uv run pytest --cov=toon_format --cov-report=xml --cov-report=term --cov-report=html --cov-fail-under=85
4235

43-
- name: Upload coverage to Codecov
44-
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
45-
uses: codecov/codecov-action@v4
36+
- name: Upload coverage reports as artifact
37+
uses: actions/upload-artifact@v4
38+
if: matrix.python-version == '3.12'
39+
with:
40+
name: coverage-reports
41+
path: |
42+
coverage.xml
43+
htmlcov/
44+
retention-days: 30
45+
46+
- name: Coverage comment on PR
47+
uses: py-cov-action/python-coverage-comment-action@v3
48+
if: matrix.python-version == '3.12' && github.event_name == 'pull_request'
4649
with:
47-
file: ./coverage.xml
48-
fail_ci_if_error: false
50+
GITHUB_TOKEN: ${{ github.token }}
51+
MINIMUM_GREEN: 90
52+
MINIMUM_ORANGE: 85

‎.gitignore‎

Lines changed: 60 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,12 @@
1-
# Python
1+
# Byte-compiled / optimized / DLL files
22
__pycache__/
33
*.py[cod]
44
*$py.class
5+
6+
# C extensions
57
*.so
8+
9+
# Distribution / packaging
610
.Python
711
build/
812
develop-eggs/
@@ -23,7 +27,36 @@ share/python-wheels/
2327
*.egg
2428
MANIFEST
2529

26-
# Virtual environments
30+
# Package-specific
31+
toon_format.egg-info/
32+
33+
# Ruff cache
34+
.ruff_cache/
35+
36+
# Mypy cache
37+
.mypy_cache/
38+
.dmypy.json
39+
dmypy.json
40+
41+
# PyInstaller
42+
*.manifest
43+
*.spec
44+
45+
# Unit test / coverage reports
46+
htmlcov/
47+
.tox/
48+
.nox/
49+
.coverage
50+
.coverage.*
51+
.cache
52+
nosetests.xml
53+
coverage.xml
54+
*.cover
55+
*.py,cover
56+
.hypothesis/
57+
.pytest_cache/
58+
59+
# Environments
2760
.env
2861
.venv
2962
env/
@@ -38,21 +71,35 @@ venv.bak/
3871
*.swp
3972
*.swo
4073
*~
74+
.claude/
75+
CLAUDE.md
76+
77+
# macOS
4178
.DS_Store
79+
.AppleDouble
80+
.LSOverride
81+
._*
4282

43-
# Testing
44-
.pytest_cache/
45-
.coverage
46-
htmlcov/
47-
.tox/
48-
.nox/
83+
# Files that might appear in the root of a volume
84+
.DocumentRevisions-V100
85+
.fseventsd
86+
.Spotlight-V100
87+
.TemporaryItems
88+
.Trashes
89+
.VolumeIcon.icns
90+
.com.apple.timemachine.donotpresent
4991

50-
# Type checking
51-
.mypy_cache/
52-
.pytype/
53-
.pyre/
54-
.pyright/
92+
# Directories potentially created on remote AFP share
93+
.AppleDB
94+
.AppleDesktop
95+
Network Trash Folder
96+
Temporary Items
97+
.apdisk
5598

5699
# uv
57100
.uv/
58101
uv.lock
102+
103+
PR_DESCRIPTION.md
104+
AGENTS.md
105+
.augment/

0 commit comments

Comments
 (0)