Skip to content

Commit d7ab551

Browse files
baszalmstrabaszalmstra
committed
wip: initial integration in pixi
1 parent 48ecd0b commit d7ab551

File tree

2 files changed

+294
-6
lines changed

2 files changed

+294
-6
lines changed

‎IMPLEMENTATION_PLAN.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# Implementation Plan: Develop Dependencies Feature
2+
3+
Issue: [#4721](https://github.com/prefix-dev/pixi/issues/4721)
4+
5+
## Progress Summary
6+
7+
| Stage | Status | Description |
8+
|-------|--------|-------------|
9+
| Stage 1 | ✅ Complete | Data Model + TOML Parsing (with 12 tests) |
10+
| Stage 2 | ✅ Complete | Feature Access Methods |
11+
| Stage 3 | 🚧 In Progress | Solving Integration |
12+
13+
**Last Updated**: Stage 1 & 2 completed with comprehensive test coverage
14+
15+
---
16+
17+
## Overview
18+
Add support for `[develop]` section in pixi.toml that allows specifying source dependencies whose build/host/run dependencies should be installed without building the packages themselves.
19+
20+
## Design Decisions
21+
- Store as `IndexMap<PackageName, SourceSpec>` to support Git/Path/Url sources
22+
- Expand develop dependencies BEFORE solving (not during)
23+
- Merge expanded dependencies into regular dependencies
24+
- Support platform-specific and feature-specific develop dependencies
25+
- Always extract all dependency types (build, host, run)
26+
27+
---
28+
29+
## Stage 1: Data Model + TOML Parsing
30+
**Goal**: Add develop dependencies to data model and parse from TOML
31+
**Status**: ✅ Complete
32+
33+
### Tasks
34+
- [x] Add `develop_dependencies: Option<IndexMap<PackageName, SourceSpec>>` to `WorkspaceTarget` in `crates/pixi_manifest/src/target.rs`
35+
- [x] Add `develop: Option<IndexMap<PackageName, TomlLocationSpec>>` to `TomlTarget` in `crates/pixi_manifest/src/toml/target.rs`
36+
- [x] Parse `develop` field in `TomlTarget::into_workspace_target()`
37+
- [x] Add `develop: Option<IndexMap<PackageName, TomlLocationSpec>>` to `TomlFeature` in `crates/pixi_manifest/src/toml/feature.rs`
38+
- [x] Add `develop` field to `TomlManifest` in `crates/pixi_manifest/src/toml/manifest.rs`
39+
- [x] Parse `develop` field in all TOML deserializers
40+
- [x] Convert `TomlLocationSpec` to `SourceSpec` during parsing
41+
- [x] Write comprehensive unit tests for parsing develop dependencies
42+
43+
### Success Criteria
44+
- [x] Code compiles
45+
- [x] Unit test passes for parsing `[develop]` section
46+
- [x] Unit test passes for parsing `[feature.X.develop]` section
47+
- [x] Unit test passes for parsing `[target.linux-64.develop]` section
48+
49+
### Files Modified
50+
- `crates/pixi_manifest/src/target.rs`
51+
- `crates/pixi_manifest/src/toml/target.rs`
52+
- `crates/pixi_manifest/src/toml/feature.rs`
53+
- `crates/pixi_manifest/src/toml/manifest.rs`
54+
- `crates/pixi_manifest/src/toml/mod.rs`
55+
- `crates/pixi_manifest/src/toml/test_develop.rs` (new)
56+
57+
### Tests Added
58+
Created comprehensive test suite with 12 tests:
59+
- `test_parse_develop_path` - Parse path-based develop dependencies
60+
- `test_parse_develop_git` - Parse git-based develop dependencies
61+
- `test_parse_develop_url` - Parse URL-based develop dependencies
62+
- `test_parse_develop_multiple` - Parse multiple develop dependencies
63+
- `test_parse_feature_develop` - Parse feature-specific develop dependencies
64+
- `test_parse_target_develop` - Parse platform-specific develop dependencies
65+
- `test_parse_develop_override` - Test platform override behavior
66+
- `test_parse_develop_git_with_rev` - Parse git with rev/tag
67+
- `test_parse_develop_empty` - Handle missing develop dependencies
68+
- `test_parse_feature_target_develop` - Parse feature + target develop dependencies
69+
- `test_parse_develop_invalid_no_source_type` - Error handling
70+
- `test_parse_develop_invalid_multiple_sources` - Error handling
71+
72+
### Commit Message
73+
```
74+
feat: add develop dependencies to manifest data model
75+
76+
Add support for parsing [develop] sections from pixi.toml:
77+
- Add develop_dependencies field to WorkspaceTarget
78+
- Parse develop section in TomlTarget, TomlFeature, and TomlManifest
79+
- Support platform-specific develop dependencies
80+
- Add comprehensive test suite with 12 tests
81+
82+
Part of #4721
83+
```
84+
85+
---
86+
87+
## Stage 2: Feature Access Methods
88+
**Goal**: Add methods to access develop dependencies with platform resolution
89+
**Status**: ✅ Complete
90+
91+
### Tasks
92+
- [x] Add `develop_dependencies(&self, platform: Option<Platform>)` method to `Feature` in `crates/pixi_manifest/src/feature.rs`
93+
- [x] Follow pattern of existing `run_dependencies()`, `build_dependencies()` methods
94+
- [x] Handle platform-specific resolution via `targets.resolve(platform)`
95+
- [x] Platform resolution tested via Stage 1 tests
96+
97+
### Success Criteria
98+
- [x] Code compiles
99+
- [x] All existing tests pass
100+
- [x] Platform-specific develop dependency resolution works correctly
101+
- [x] Method returns correct develop dependencies for given platform
102+
103+
### Files Modified
104+
- `crates/pixi_manifest/src/feature.rs`
105+
106+
### Commit Message
107+
```
108+
feat: add methods to access develop dependencies from features
109+
110+
Add develop_dependencies() method to Feature that:
111+
- Resolves platform-specific develop dependencies
112+
- Follows existing pattern for dependency access
113+
- Merges develop dependencies from all targets
114+
115+
Part of #4721
116+
```
117+
118+
---
119+
120+
## Stage 3: Solving Integration
121+
**Goal**: Integrate develop dependency expansion into environment solving
122+
**Status**: In Progress
123+
124+
### Tasks
125+
- [ ] Add `ExpandDevSourcesFailed` variant to `SolveCondaEnvironmentError` in `crates/pixi_core/src/lock_file/update.rs`
126+
- [ ] Create helper function `expand_develop_dependencies()` that:
127+
- Collects develop dependencies from all features in environment
128+
- Converts to `Vec<DependencyOnlySource>`
129+
- Calls `command_dispatcher.expand_dev_sources()`
130+
- Returns `ExpandedDevSources`
131+
- [ ] Integrate into `spawn_solve_conda_environment_task()` (around line 1907):
132+
- Call `expand_develop_dependencies()` before solving
133+
- Merge `expanded.dependencies` into regular dependencies
134+
- Merge `expanded.constraints` into constraints
135+
- [ ] Write integration test using `tests/data/workspaces/output-dependencies/` workspace
136+
137+
### Success Criteria
138+
- [ ] Code compiles
139+
- [ ] All existing tests pass
140+
- [ ] Integration test passes showing:
141+
- Develop source dependencies are expanded
142+
- Their dependencies are installed
143+
- The develop packages themselves are NOT built/installed
144+
- Dev sources that depend on other dev sources correctly filter them out
145+
146+
### Files Modified
147+
- `crates/pixi_core/src/lock_file/update.rs`
148+
- `crates/pixi_command_dispatcher/tests/integration/main.rs` (or new test file)
149+
150+
### Commit Message
151+
```
152+
feat: expand develop dependencies during environment solving
153+
154+
Integrate develop dependency expansion:
155+
- Expand develop sources before solving
156+
- Merge expanded dependencies into environment
157+
- Add comprehensive integration test
158+
- Handle errors from expansion gracefully
159+
160+
Closes #4721
161+
```
162+
163+
---
164+
165+
## Testing Strategy
166+
167+
### Unit Tests
168+
- Parse `[develop]` section with valid source specs
169+
- Parse `[feature.X.develop]` section
170+
- Parse `[target.platform.develop]` section
171+
- Platform-specific resolution of develop dependencies
172+
173+
### Integration Test
174+
Use `tests/data/workspaces/output-dependencies/` workspace:
175+
1. Add `[develop]` section with test-package as dev source
176+
2. Add `[feature.extra.develop]` with package-a as dev source
177+
3. Verify:
178+
- test-package's dependencies (cmake, make, openssl, zlib, numpy, python) are installed
179+
- test-package itself is NOT built/installed
180+
- package-b is included (not a dev source)
181+
- test-package is filtered from package-a's dependencies (both are dev sources)
182+
183+
---
184+
185+
## Notes
186+
- Command dispatcher already has `expand_dev_sources()` implemented
187+
- `TomlLocationSpec` already supports parsing source specs
188+
- Follow existing patterns for dependencies/host-dependencies/build-dependencies

‎crates/pixi_core/src/lock_file/update.rs‎

Lines changed: 106 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
use std::{
22
cmp::PartialEq,
3-
collections::{HashMap, HashSet, hash_map::Entry},
3+
collections::{BTreeMap, HashMap, HashSet, hash_map::Entry},
44
future::{Future, ready},
55
iter,
66
path::PathBuf,
@@ -19,16 +19,16 @@ use indicatif::ProgressBar;
1919
use itertools::{Either, Itertools};
2020
use miette::{Diagnostic, IntoDiagnostic, MietteDiagnostic, Report, WrapErr};
2121
use pixi_command_dispatcher::{
22-
BuildEnvironment, CommandDispatcher, CommandDispatcherError, PixiEnvironmentSpec,
23-
SolvePixiEnvironmentError,
22+
BuildEnvironment, CommandDispatcher, CommandDispatcherError, DependencyOnlySource,
23+
ExpandDevSourcesSpec, ExpandedDevSources, PixiEnvironmentSpec, SolvePixiEnvironmentError,
2424
};
2525
use pixi_consts::consts;
2626
use pixi_glob::GlobHashCache;
2727
use pixi_install_pypi::{
2828
LazyEnvironmentVariables, PyPIBuildConfig, PyPIContextConfig, PyPIEnvironmentUpdater,
2929
PyPIUpdateConfig,
3030
};
31-
use pixi_manifest::{ChannelPriority, EnvironmentName, FeaturesExt};
31+
use pixi_manifest::{ChannelPriority, EnvironmentName, FeaturesExt, HasFeaturesIter};
3232
use pixi_progress::global_multi_progress;
3333
use pixi_record::{ParseLockFileError, PixiRecord};
3434
use pixi_utils::prefix::Prefix;
@@ -40,7 +40,10 @@ use pixi_uv_conversions::{
4040
use pypi_mapping::{self, MappingClient};
4141
use pypi_modifiers::pypi_marker_env::determine_marker_environment;
4242
use rattler::package_cache::PackageCache;
43-
use rattler_conda_types::{Arch, GenericVirtualPackage, PackageName, ParseChannelError, Platform};
43+
use rattler_conda_types::{
44+
Arch, ChannelConfig, ChannelUrl, GenericVirtualPackage, PackageName, ParseChannelError,
45+
Platform,
46+
};
4447
use rattler_lock::{LockFile, LockedPackageRef, ParseCondaLockError};
4548
use serde::{Deserialize, Serialize};
4649
use thiserror::Error;
@@ -228,6 +231,15 @@ pub enum SolveCondaEnvironmentError {
228231
source: CommandDispatcherError<SolvePixiEnvironmentError>,
229232
},
230233

234+
#[error("failed to expand develop dependencies for environment '{}' for platform '{}'", .environment_name.fancy_display(), .platform)]
235+
ExpandDevSourcesFailed {
236+
environment_name: GroupedEnvironmentName,
237+
platform: Platform,
238+
#[source]
239+
#[diagnostic_source]
240+
source: CommandDispatcherError<pixi_command_dispatcher::ExpandDevSourcesError>,
241+
},
242+
231243
#[error(transparent)]
232244
#[diagnostic(transparent)]
233245
PypiMappingFailed(Box<dyn Diagnostic + Send + Sync + 'static>),
@@ -1893,6 +1905,69 @@ pub enum TaskResult {
18931905
),
18941906
}
18951907

1908+
/// Expands develop dependencies for a grouped environment.
1909+
///
1910+
/// Collects all develop dependencies from the features of the grouped environment,
1911+
/// converts them to DependencyOnlySource format, and uses the command dispatcher
1912+
/// to expand them into their build/host/run dependencies.
1913+
///
1914+
/// Returns `None` if there are no develop dependencies.
1915+
async fn expand_develop_dependencies(
1916+
group: &GroupedEnvironment<'_>,
1917+
platform: Platform,
1918+
channel_config: &ChannelConfig,
1919+
channels: &[ChannelUrl],
1920+
virtual_packages: &[GenericVirtualPackage],
1921+
variants: &BTreeMap<String, Vec<String>>,
1922+
command_dispatcher: &CommandDispatcher,
1923+
) -> Result<Option<ExpandedDevSources>, SolveCondaEnvironmentError> {
1924+
// Collect develop dependencies from all features in the group
1925+
let mut develop_deps = IndexMap::new();
1926+
for feature in group.features() {
1927+
if let Some(deps) = feature.develop_dependencies(Some(platform)) {
1928+
develop_deps.extend(deps.into_owned());
1929+
}
1930+
}
1931+
1932+
// If there are no develop dependencies, return early
1933+
if develop_deps.is_empty() {
1934+
return Ok(None);
1935+
}
1936+
1937+
// Convert to Vec<DependencyOnlySource>
1938+
let dev_sources: Vec<DependencyOnlySource> = develop_deps
1939+
.into_iter()
1940+
.map(|(name, spec)| DependencyOnlySource {
1941+
source: spec,
1942+
output_name: name,
1943+
})
1944+
.collect();
1945+
1946+
// Create the spec for expanding dev sources
1947+
let spec = ExpandDevSourcesSpec {
1948+
dev_sources,
1949+
channel_config: channel_config.clone(),
1950+
channels: channels.to_vec(),
1951+
build_environment: BuildEnvironment::simple(platform, virtual_packages.to_vec()),
1952+
variants: Some(variants.clone()),
1953+
enabled_protocols: Default::default(),
1954+
};
1955+
1956+
// Expand the dev sources
1957+
let expanded = command_dispatcher
1958+
.expand_dev_sources(spec)
1959+
.await
1960+
.map_err(
1961+
|source| SolveCondaEnvironmentError::ExpandDevSourcesFailed {
1962+
environment_name: group.name(),
1963+
platform,
1964+
source,
1965+
},
1966+
)?;
1967+
1968+
Ok(Some(expanded))
1969+
}
1970+
18961971
/// A task that solves the conda dependencies for a given environment.
18971972
#[allow(clippy::too_many_arguments)]
18981973
async fn spawn_solve_conda_environment_task(
@@ -1951,14 +2026,39 @@ async fn spawn_solve_conda_environment_task(
19512026
// Determine the build variants
19522027
let variants = group.workspace().variants(platform);
19532028

2029+
// Expand develop dependencies if any
2030+
let expanded_develop = expand_develop_dependencies(
2031+
&group,
2032+
platform,
2033+
&channel_config,
2034+
&channels,
2035+
&virtual_packages,
2036+
&variants,
2037+
&command_dispatcher,
2038+
)
2039+
.await?;
2040+
2041+
// Merge expanded develop dependencies into regular dependencies
2042+
let mut dependencies = dependencies;
2043+
let constraints = if let Some(expanded) = expanded_develop {
2044+
// Merge the expanded dependencies
2045+
for (name, spec) in expanded.dependencies.into_specs() {
2046+
dependencies.insert(name, spec);
2047+
}
2048+
// Use the expanded constraints
2049+
expanded.constraints
2050+
} else {
2051+
Default::default()
2052+
};
2053+
19542054
let start = Instant::now();
19552055

19562056
// Solve the environment using the command dispatcher.
19572057
let mut records = command_dispatcher
19582058
.solve_pixi_environment(PixiEnvironmentSpec {
19592059
name: Some(group_name.to_string()),
19602060
dependencies,
1961-
constraints: Default::default(),
2061+
constraints,
19622062
installed: existing_repodata_records.records.clone(),
19632063
build_environment: BuildEnvironment::simple(platform, virtual_packages),
19642064
channels,

0 commit comments

Comments
 (0)