Skip to content

multi-output: document full/lite splits with optional dependencies #2168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

multi-output: document full/lite splits with optional dependencies #2168

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
77b9535
multi-output: document full/lite splits for optional dependencies.
croth1 May 1, 2024
36eacbd
update README
croth1 May 1, 2024
939bfbd
Mark pre-commit instructions optional
croth1 May 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ If you have questions or need help, please check out our documentation for a [li
Run the following commands.
- `conda env create -f ./.ci_scripts/environment.yml`
- `conda activate conda-forge-docs`
- For live builds, `npm install && npm run start`
- enable pre-commit hooks with `pre-commit install`
- For live builds, `npm install && npm run start` - you can view the docs at `http://localhost:3000`
- For production builds, run `.ci_scripts/update_docs`
4. Make and commit your changes.
5. Submit a [pull request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests) to the main repository proposing your changes.
Expand Down
31 changes: 31 additions & 0 deletions docs/maintainer/knowledge_base.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1655,6 +1655,37 @@ add some information on r packages which make heavy use of `noarch: generic`
- [importlib_metadata and importlib-metadata](https://github.com/conda-forge/importlib_metadata-feedstock/blob/main/recipe/meta.yaml)
- [typing_extensions and typing-extensions](https://github.com/conda-forge/typing_extensions-feedstock/blob/main/recipe/meta.yaml)

### Common patterns

#### Splitting out heavy optional dependencies
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous (low-effort) approach for this was to add the example in the list above (under "Distributing the same project with different sets of dependencies"), and then the pitfall in the section below. This section has a bit both, hence the confusion. I'd rather:

  • Add the run_constrained pin_subpackage problem to the pitfalls (link to issue).
  • Add a shorter summary of the solution you are proposing to the aforementioned list to avoid duplicating information. Briefly warn about the pitfall here.

That said, the information you have written here is accurate and I don't want to bikeshed too much. Feel free to ignore my feedback or only take into account parts of it.

Copy link
Contributor Author

@croth1 croth1 May 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmh, fair point, doing it this way is a bit of mess. I found the list structure a bit annoying because it somewhat strongly enforces brevity and discourages explanations beyond a short summary, but it is just a more detailed explanation of a special case of Distributing the same project with different sets of dependencies. Hmm...


When a package has especially large optional dependencies it can be useful to split the package into a "lite" version that depends only on the core dependencies and a "full" version that depends on the "lite" version and all optional dependencies. The `pin_subpackage` function helps to ensure that the "full" version will pull the exact version of the "lite" version. As an example consider the [seaborn recipe](https://github.com/conda-forge/seaborn-feedstock/blob/eec88f0116a26c2e205daab10193e38e96407cab/recipe/meta.yaml) with optional statsmodels dependency:

```yaml
- name: seaborn
build:
noarch: python
requirements:
run:
- statsmodels >=0.12
- {{ pin_subpackage('seaborn-base', exact=True) }}
```

In order to prevent the "lite" version `seaborn-base` to pull in seaborn versions before the package split, `run_constrained` will ensure that the `seaborn-base` package can only be installed together with the matching version of the `seaborn` package.

```yaml
- name: seaborn-base
build:
noarch: python
requirements:
run_constrained:
- seaborn ={{ version }}=*_{{ build }}
```

:::important
Counter-intuitively `{{ pin_subpackage('seaborn-base', exact=True) }}` will only work in the requirements section but not in `run_constrained` (see [conda-build#4415](https://github.com/conda/conda-build/issues/4415)). Use `seaborn ={{ version }}=*_{{ build }}` to ensure that the `seaborn-base` package can only be installed with the matching version of the `seaborn` package.
:::

### Common pitfalls with `outputs`

This is a non-exhaustive list of common pitfalls when using `outputs`.
Expand Down