Merge pull request #93 from OoriData/feature/kb-rearch

OgbujiPT 0.10.0 Phase 1: Foundation & Rearchitecture

Author: uogbuji

.github/workflows/main.yml

@@ -1,13 +1,13 @@
-name: Python package
+name: OgbujiPT CI
 
-on: [push]
+on: [push, pull_request]
 
 jobs:
-  build:
+  test:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.12", "3.13"]
     services:
       mockdb:
         image: pgvector/pgvector:pg16
@@ -20,28 +20,27 @@ jobs:
           - 5432:5432
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install ruff pytest pytest-mock pytest-asyncio respx # pytest-httpx # Now using respx instead
-          pip install pgvector asyncpg pytest-asyncio # Added by Kai, Osi
-          
+          # pip install ruff pytest pytest-mock pytest-asyncio respx
           if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
-          # Install OgbujiPT itself, as checked out
-          pip install -U $GITHUB_WORKSPACE
+          # Install OgbujiPT itself, plus dependencies needed to run the full CI, with test suite
+          pip install -U ".[testall]"
+
       - name: Lint with ruff
         run: |
-          # stop the build if there are Python syntax errors or undefined names
-          # ruff --format=github --select=E9,F63,F7,F82 --target-version=py311 .  # No longer works
-          ruff check --select=E9,F63,F7,F82 --target-version=py311 --exclude **/*.ipynb .
-          # default set of ruff rules with GitHub Annotations
-          # ruff --format=github --target-version=py311 .  # No longer works
-          ruff check --target-version=py311 --exclude **/*.ipynb .
+          # Stop the build if there are Python syntax errors or undefined names
+          ruff check --select=E9,F63,F7,F82 --target-version=py312 --exclude **/*.ipynb .
+          # Run default ruff checks
+          ruff check --target-version=py312 --exclude **/*.ipynb .
+
       - name: Test with pytest
         env:
           PG_HOST: 0.0.0.0
@@ -50,4 +49,4 @@ jobs:
           PG_PASSWORD: mock_password
           PG_PORT: 5432
         run: |
-          pytest
+          pytest test/ -v

.github/workflows/publish.yml

@@ -0,0 +1,35 @@
+name: Publish to PyPI  # https://pypi.org/p/OgbujiPT
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi  # GitHub environment for additional protection
+    permissions:
+      id-token: write  # Required for trusted publishing
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install build tools
+        run: |
+          python -m pip install --upgrade pip
+          pip install build twine
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        # with:
+          # Uses OIDC trusted publishing - no token needed if configured
+          # If you need to use a token instead, uncomment below:
+          # password: ${{ secrets.PYPI_API_TOKEN }}

.github/workflows/python-publish.yml

@@ -0,0 +1,43 @@
+Additional context on this repository for AI tools & coding agents
+
+- Python 3.12+ code, unless otherwise specified
+- Python code uses single outer quotes, including triple single quotes for e.g. docstrings
+- prefer absolute imports to relative imports
+- Use a decent amount of comments
+  - not *too* many, just enough that anybody familiar with the code can use them as a reference point. Not meant to teach somebody new every intricacy of the code, just help keep the reader oriented.
+- if it saves a line, put a comment after a line rather than above it
+  - use the standard two spaces before the comment character, eg. `CODE  # COMMENT`
+- Try to stick to 120 characters per line
+  - if one of those comments would break this guideline, just put that comment above the line instead, as is standard convention
+- If there is a pyproject.toml in place, use it as a reference for builds, installs, etc. The basic packaging and dev preference, including if you have to supply your own pyproject.toml, is as follows:
+  - Use pyproject.toml with hatchling, not e.g. setup.py
+  - Reusable Python code modules are developed in the `pylib` folder, and installed using e.g. `uv pip install -U .`, which includes proper mapping to Python library package namespace via `tool.hatch.build.sources`. The `__init__.py` and other modules in the top-level package go directly in `pylib`, though submodules can use subdirectories, e.g. `pylib/a/b` becomes `installed_library_name.a.b`. Ultimately this will mean the installed package is importable as `from installed_library_name.etc import …`
+  - Yes this means editable and "dev mode" environments are NOT desirable, nor are shenanigans adding pylib to `sys.path`. Layer-efficient dockerization is an option if that's needed.
+  - The ethos is to always develop keeping things properly installable. No dev mode shortcuts
+  - Prefer hatchling build system over setuptools, poetry, etc. Avoid setuptools as much as possible. Use `[tool.hatch.build.sources]` to map source directories to package namespaces (e.g., `"pylib" = "installed_library_name"`).
+  - Use `[tool.hatch.build.targets.wheel]` with `only-include = ["pylib"]` to ensure the pylib directory structure gets included properly in the wheel, avoiding the duplication issue that can occur with sources mapping
+- **Debugging package issues**: When modules aren't importing correctly after installation, check:
+  - That you are in the correct virtualenv (you may have to ask the developer)
+  - Package structure in site-packages (e.g., `ls -la /path/to/site-packages/package_name/`)
+- Use uv, but pay attention to the above
+  - Again always use `uv pip install -U .` for full installation, never editable installs (`pip install -e`). This ensures proper testing of the actual distribution.
+- Use async (e.g. asyncio) wherever it makes sense. Avoid multithreading, though multiprocessing is OK. Multiprocess for CPU-bound concurrency, and asyncIO for I/O bound, cooperative etc.
+- Be pythonic. Avoid e.g. complex abstract class hierarchies for the sake of them, though classes are also fine in many usage patterns. We love dictionaries, dynamic dispatch, etc.
+  - I don't consider Pydantic very Pythonic, so we can tolerate it if need be (e.g. we're using a toolkit that strictly works with Pydantic), but otherwise, simple dataclasses are better.
+- Type hints are OK in moderation, but avoid absolutely littering the code with them.
+  - No excess imports & symbols, e.g. Use type | None rather than Optional[type]
+- use iterator patterns as much as practical. Also functional programming approaches, including partials (currying) and decorators
+- Prefereed tools:
+  - Logging: structlog
+  - Retries on failure: tenacity
+  - CLI argument processing: fire—avoid argparse except for truly trivial usage
+  - CLI formatting: rich
+  - HTTP client: httpx (async)
+  - HTML/XML parsing: selectolax (though for now we're using html5-modern as the base implementation for our html5 features)
+  - Browser-like Web crawling/scraping: Python playwright (with playwright_stealth if needed)
+  - pytest, as well as pytest-mock, pytest-httpx, pytest-asyncio
+  - rapidfuzz for fuzzy text matching
+- AVOID the following unless explicitly requested or otherwise unavoidable:
+  - langchain
+
+- Once again PREFER SINGLE QUOTES