switch hatch to dependency groups, split deps to data file

[sneakers-the-rat]

fix: #143

so this does two things, switched us over to dependency groups and splits our dependencies out into data so that the version constraints aren't all mushed up with the form of the template (which is about structure not data).

however for the life of me i can't tell what's going on - the pyproject.toml seems to be correctly generated, here is an example with the minimal option:

expand/collapse pyproject.toml

################################################################################
# Build Configuration
################################################################################

[build-system]
build-backend = "hatchling.build"
requires = ["hatchling"]

################################################################################
# Project Configuration
################################################################################

[project]
name = "aaa"
# You can chose to use dynamic versioning with hatch or static where you add it manually.
version = "0.1.0"

description = "a"
authors = [
    { name = "a", email = "a@a.com" },
]
license = "MIT"
requires-python = ">= 3.10" # Adjust based on the minimum version of Python that you support
readme = {"file" = "README.md", "content-type" = "text/markdown"}
# Please consult https://pypi.org/classifiers/ for a full list.
classifiers = [
    "Development Status :: 2 - Pre-Alpha",
    "Intended Audience :: Science/Research",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3 :: Only",
]
# TODO: add keywords
keywords = []
# TODO: add dependencies
dependencies = []

[project.urls]
Homepage = "https://github.com/aaa/aaa"
"Source Code" = "https://github.com/aaa/aaa"
"Bug Tracker" = "https://github.com/aaa/aaa/issues"
Documentation = "https://github.com/aaa/aaa/blob/main/README.md"
Download = "https://pypi.org/project/aaa/#files"

[dependency-groups]
dev = [
    "hatch",
    "pre-commit",
    {include-group = "docs"},
    {include-group = "tests"},
]
docs = [
    "sphinx~=8.0",
    "myst-parser>=4.0",
    "pydata-sphinx-theme~=0.16",
    "sphinx-autobuild>=2024.10.3",
    "sphinx-autoapi>=3.6.0",
    "sphinx_design>=0.6.1",
    "sphinx-copybutton>=0.5.2",
]
build = [
    "pip-audit",
    "twine",
]
tests = [
    "pytest",
    "pytest-cov",
    "pytest-raises",
    "pytest-randomly",
    "pytest-xdist",
]

################################################################################
# Tool Configuration
################################################################################

# Hatch is building your package's wheel and sdist
# This tells hatch to only include Python packages (i.e., folders with __init__.py) in the build.
# read more about package building, here:
# https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-distribution-files-sdist-wheel.html
[tool.hatch.build]
only-packages = true

# This tells Hatch to build the package from the src/ directory.
# Read more about src layouts here: https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-structure.html
[tool.hatch.build.targets.wheel]
packages = ["src/aaa"]



######## Configure pytest for your test suite ########
[tool.pytest.ini_options]
testpaths = ["tests"] # Tells pytest what directory tests are in
markers = ["raises"] # Tells pytest to not raise a warning if you use @pytest.mark.raises

[tool.coverage.paths]
source = [
    "src/aaa",
    "*/site-packages/aaa",
]

[tool.coverage.run]
# Ensures code coverage is measured for branches (conditional statements with different outcomes) in your code.
branch = true
parallel = true

[tool.coverage.report]
# This configures the output test coverage report
exclude_lines = ["pragma: no cover"]
precision = 2


# Use UV to create Hatch environments
[tool.hatch.envs.default]
installer = "uv"
builder = true

################################################################################
# Hatch Environments
################################################################################

#--------------- Build and check your package ---------------#

# This table installs the tools you need to test and build your package
[tool.hatch.envs.build]
description = """Test the installation the package."""
dependency-groups = [
    "build",
]
detached = true
builder = true

# This table installs created the command hatch run install:check which will build and check your package.
[tool.hatch.envs.build.scripts]
check = [
    "hatch build {args:--clean}",
    "twine check dist/*",
]

#--------------- Run tests ---------------#
[tool.hatch.envs.test]
description = """Run the test suite."""
dependency-groups = [
    "tests",
]

[[tool.hatch.envs.test.matrix]]
python = ["3.10", "3.11", "3.12", "3.13"]

[tool.hatch.envs.test.scripts]
run = "pytest {args:--cov=aaa --cov-report=term-missing --cov-report=xml}"

#--------------- Build and preview your documentation ---------------#

# This sets up a hatch environment with associated dependencies that need to be installed
[tool.hatch.envs.docs]
description = """Build or serve the documentation."""
# Install optional dependency test for docs
dependency-groups = [
    "docs",
]

# This table contains the scripts that you can use to build and serve your docs
# hatch run docs:build will build your documentation
# hatch run docs:serve will serve them 'live' on your computer locally
[tool.hatch.envs.docs.scripts]
build = ["sphinx-build {args:-W -b html docs docs/_build}"]
serve = ["sphinx-autobuild docs --watch src/aaa {args:-b html docs/_build/serve}"]



#--------------- Check security for your dependencies ---------------#

[tool.hatch.envs.audit]
description = """Check dependencies for security vulnerabilities."""
dependency-groups = [
    "build",
]

[tool.hatch.envs.audit.scripts]
check = ["pip-audit"]

but when we go to actually run the functional tests, hatch doesn't seem to actually be installing any of the dependency group deps, or the package itself... I am pretty positive this has nothing to do with the templating change, when i diff the generated pyproject.toml before and after this PR, the only thing that changes pretty much is renaming the table key to dependency-groups . I paused the pytest run on errors and went to the generated directory and it also looked fine, and same result - when i tried to run hatch manually it didn't install any of the deps and then errored out.

I have been up and down hatch's documentation and i can't figure out why it doesn't seem to want to install anything!

the only other thing i can think of is it being related to pypa/hatch#2113 which seems to require that the builder=true flag be present for pretty much all the envs.

i lay this PR on the mercy of the court, if anyone can figure this out, gr8

I am opening this up here to see if CI behaves differently and something is just wrong on my machine. idk.

---

edit: there does seem to be something going wrong where the outer hatch environment that’s running the tests for the template seems to interfere with the inner hatch environment that’s being tested - when i removed the test matrix from the template, another test failed saying that the python 3.10 environment didn’t exist, and it started working again when i removed the matrix from the outer (template) package…

i can also replicate the problems here by removing all the template changes except strictly changing the keys to dependency-group, i think something broken in hatch.

(call me old fashioned, but i am sorta liking my old fashioned ways of "simply using venvs" right now, at least with a venv i know exactly where it is and what's in it and there's no tool in between me and python, but all these environment virtualization tools seem to want to get too fancy and break lol)

---

edit2: ope wait no it's just this: pypa/hatch#2152

[lwasser]

My guess is the tests here are breaking not because of this pr but they have been broken. I am not sure why however as they started breaking when we got the latest dep update and I made changes in november but they were ok then. It is possible that I made something brittle.

[sneakers-the-rat]

I think its just this! pypa/hatch#2152

[Midnighter]

@sneakers-the-rat taking a brief look through this, I'm not convinced that defining dependencies in a separate YAML file provides real value. To me it seems to introduce more complexity to the template and separates the definition of the groups from where they are used. Basically, I don't see the disadvantage of having the constraints in the template for the pyproject.toml.

[sneakers-the-rat]

maintainable templates separate data from structure with some reasonable balance of which things might be changed or be reused.

so now rather than the dependencies section having the logical conditions that control rendering interspersed with data, we have a compact section of logical conditions without needing to repeat the formatting that is shared between all the things of the same kind.

the circumstance where it is a useful change is if we ever want to do any logic on the deps - e.g., maybe allow someone to use extras instead of dependency-groups for some subset of the groups (i would like that for shipping tests in the package), or declare additional dependencies in some tool-specific way like in hatch build env dependencies, allow someone to enable/disable using the minimum constraints, provide their own default constraints, update their generated project from .copier-answers.yml while keeping any changes they have made to the deps, write tests that ensure that a set of expected deps is toggled by some option without needing to maintain the same data that's in the yml file in the test data anyway, etc.

specifically, the reason why i had this in mind for awhile was thinking about how it would be a bunch of continuous annoying work keep any minimums updated e.g. for security problems from audit, so having them as data lets them use them as data - can write an action to check all the packages used in the template, across all options that might change, with the specific version dependency, and then autogenerate a PR that bumps the version. doing that by parsing all the deps as text in the template would be a pain to say the least.

the implementation as is should only ever need to be changed if toml, copier, or jinja changed core features, and at that point we'd need to probably refactor things anyway.

idrc, close this if you want, not a change i'd fight for, if it's controversial i'd rather just close it and forget about it

edit: the reason why i thought this made sense for the deps specifically and not other things is that they are the most likely thing to want to programmatically control or introspect. like i am not trying to put all strings in separate files, some things make sense to do in place, etc.

[lwasser]

This, in theory, should work, but I found it doesn't.

this pr - pypa/hatch#2155 is working towards. fix. so i think we should add builder=True here to the individual environments. and then open an issue to remove that when they have fixed the bug!

[lwasser]

Oh, OK. So we are modifying this function rather than using what the copier uses, to allow us to import the dependency files effectively in a way that we can trust the "API" part won't change on us? Is that the gist?

[sneakers-the-rat]

it's basically so that we do effectively what they do in the tests where we are loading the copier template directly (rather than calling copier, which we do in most of the tests) without reaching inside their private methods. also they use pyyaml and we are using ruamel.yaml, so had to adapt it for that too

[lwasser]

@sneakers-the-rat, see if my suggested changes fix CI? I haven't pulled this down to test.

Given they are actively working on a fix to this bug, I think it's easy enough to switch things over, use builder = True, and then remove it once hte patch is released. This way we are following the modern pep convention of using dependency groups but using a quirky work-around (a hatch feature that we don't need for pure python) that we can easily and simply remove once hatch supports it properly.

[Midnighter]

@sneakers-the-rat I have a slight concern about increase in complexity and decrease in cohesion, but your arguments make sense even if we don't make use of these new facilities immediately. And importantly, if this brings you joy, I have no wish to stand in the way or even ask to abandon the work.

I'll follow with some inline comments.

[Midnighter]

I would do away with most of these checks. Instead of extracting the group from the global deps variable, rather expect the group parameter to already be an appropriate mapping. So I'd call it with something like render_deps(deps["tests"]). And I'd favor calling it render_dependencies.

[sneakers-the-rat]

I tried to get copier to render it as an appropriate mapping pretty much exactly like this, but copier insists on loading the yaml as a length one array for some reason, and it seems to vary by version where older python double nests it. This was the one point where i almost gave up and would say would be reason not to make the change if we were to pick one, but i figured if this was the only way to provide data to it as one can do it most other templating systems, this would be the workaround needed.

[Midnighter]

Ouch 😬

[sneakers-the-rat]

Yeah its not great, i figured there would be some way of e.g. declaring a directory of data to load into the jinja environment, but the only way i found to do it really well was to do this: https://copier.readthedocs.io/en/stable/faq/#how-can-i-alter-the-context-before-rendering-the-project
But that makes the template "unsafe" which obviously is too large of a downgrade to warrant it.

[Midnighter]

We had the discussion on Slack regarding a lack of being able to require a minimum hatch version, despite that, we need hatch as a global dependency to even create the hatch environments, so this dependency does not make sense to me.

[sneakers-the-rat]

I was hoping that it would be able to detect a mismatch of version between the installed hatch and the requested hatch when it created the build env it used to install the package, but that might have been an empty hope. Im not sure where this dep should get declared but we need to declare it somewhere, because now the template wouldnt work with hatch versions older than this. So i guess this is a problem we would need to fix for whatever implementation of dependency-groups we do here

[lwasser]

Could we think about the hatch dependency in the same way we think about python. This template supports hatch >= version x.x.x in our documentation? I know it's not a perfect solution.

The better solution would be something akin to python-requires

[sneakers-the-rat]

yeah i think in any case it's sort of at a different level than the template itself, it seems like a feature that hatch needs to me. the thing i worry about with putting dependencies in the README is that it's much easier to not see them than programmatic specifications that force them to be true, but I added a note in the readme here: b1dd00d

[lwasser]

definitely not a concern for your pr here! agree 💯 !!

[Midnighter]

I always have to look up the whitespace behavior of jinja2, but isn't it possible to have each group start on a newline without inserting a literal \n?

[Midnighter]

You could also consider render_deps having only the inner=True behavior, that would likely solve the whitespace issues here. As you'd have to specify something like:

{%- if use_test %}
tests = [
{{ render_deps("tests") }}
]
{%- endif %}

[sneakers-the-rat]

I did it this way mostly because i wanted the top level pyproject.toml to be a logical template (what to display) and the macro to be the structural template (how to display it), and to link the keys in the dict with the output to avoid key discrepancies (tests/test). Its a little awkward but this way of writing them makes them self contained lines that dont have different start/end whitespace behavior that would need to be handled by, e.g., remembering to suppress whitespace at the end of the last line and start of first, and future groups could copy/paste the one line and just modify the key

[sneakers-the-rat]

looking at #145, and did this: 6c7bafb

and apparently now the thing that makes the dependency groups work is to not have builder in the environments? and then the ones that do have builder need to have explicit deps?

however in copying the change from hatch build --clean to hatch run build:check i feel like we might we masking an error, because running hatch build --clean within the tests still causes a "Environment test.py3.10 is not a builder environment" error, but i think that might be referring to the outer hatch environment, the ones that the tests are running in, because when i run hatch build --clean on a generated project outside of the tests it works.

[sneakers-the-rat]

this, btw, is why it's nice to have the dependencies separated out as data - if we need to do weird things with them like render them twice, we can do it without needing to remember to keep multiple places updated, all the deps stay specified in the yaml

[lwasser]

@sneakers-the-rat i am testing this too...

what do you think about separating concerns: move the dependencies to data files to a separate issue and pr so we can focus this one here only on supporting dependency groups?

I am getting some inconsistent results ? Are you seeing that too? And what version of hatch are you testing on? i am testing on the new dev version. The ci here is i think working on the latest december release.

I can run the docs environment without builder=true but i can't run the build:check without builder=true.

also hatch env remove doesn't actually remove the envt as far as i can see. I am happy to trouble shoot more next week when i'm back online. But i'm not convinced that the fix on their dev branch right now is actually working either.

[lwasser]

i posted here in hopes that will help. i also noticed that if you try hatch env remove it doesn't actually delete the environment. i have some theories around that (space in my path maybe) but there is something really odd happening with hatch.

[sneakers-the-rat]

move the dependencies to data files to a separate issue and pr so we can focus this one here only on supporting dependency groups?

sure, the change to dependency groups is so small it doesn't really matter either way to me. the macro makes it easier to do the change if we need to render duplicate dependency groups in the hatch envs like the current workaround, but idrc.

i'm using the latest pypi release (1.16.2) locally