Skip to content

Fix conda environment yaml parsing errors #4078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 15, 2025
Merged

Fix conda environment yaml parsing errors #4078

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c9c023e
Fix conda environment yaml parsing errors
AyanSinhaMahapatra Jan 7, 2025
2de7396
Merge branch 'develop' into fix-package-scan-bugs
AyanSinhaMahapatra Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/packagedcode/conda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ def parse(cls, location, package_only=False):
primary_language=cls.default_primary_language,
dependencies=dependencies,
extra_data=extra_data,
is_private=True,
Copy link
Member

@pombredanne pombredanne Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why private?

Copy link
Member Author

@AyanSinhaMahapatra AyanSinhaMahapatra Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conda environment YAMLs are ostly used to export a particular conda environment (this is like a lockfile used in projects) and to import the environment from them to replicate the conda environment. See https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file and https://docs.conda.io/projects/conda/en/stable/user-guide/concepts/environments.html
This is not a manifest that you can publish a conda package, for that you need build recipes in meta.yaml files, since this manifest is not used to publish packages at all, and only publish environments, I've marked the packages created as private.

)
yield models.PackageData.from_data(package_data, package_only)

Expand Down Expand Up @@ -205,6 +206,8 @@ def get_conda_yaml_dependencies(conda_data):

if "::" in dep:
namespace, dep = dep.split("::")
if "/" in namespace or ":" in namespace:
namespace = None

req = parse_requirement_line(dep)
if req:
Expand Down
2 changes: 1 addition & 1 deletion tests/packagedcode/data/conda/conda-yaml/phc-gnn/environment_gpu.yml-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
"notice_text": null,
"source_packages": [],
"file_references": [],
"is_private": false,
"is_private": true,
"is_virtual": false,
"extra_data": {
"channels": [
Expand Down
2 changes: 1 addition & 1 deletion tests/packagedcode/data/conda/conda-yaml/ringer/environment.yaml-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
"notice_text": null,
"source_packages": [],
"file_references": [],
"is_private": false,
"is_private": true,
"is_virtual": false,
"extra_data": {
"channels": [
Expand Down
3 changes: 3 additions & 0 deletions tests/packagedcode/data/conda/conda-yaml/test/environment_host_port.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
name: test
dependencies:
- https://104.17.92.24:443::flask=1.0.2
Copy link
Member

@pombredanne pombredanne Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the URL a namespace, or is is rather a repository_url qualifier?

Copy link
Member Author

@AyanSinhaMahapatra AyanSinhaMahapatra Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a repository URL qualifier allright, there are common ones like conda-forge, anaconda, bioconda and the URL is usually not specified fully, but denoted by just a reference word.

Copy link
Member Author

@AyanSinhaMahapatra AyanSinhaMahapatra Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I am not sure, this probably requires a bit more research, there are examples which seem more like a namespace, less like a repository URL:

Copy link
Member Author

@AyanSinhaMahapatra AyanSinhaMahapatra Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#4094

58 changes: 58 additions & 0 deletions tests/packagedcode/data/conda/conda-yaml/test/environment_host_port.yml-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
[
{
"type": "conda",
"namespace": null,
"name": "test",
"version": null,
"qualifiers": {},
"subpath": null,
"primary_language": "Python",
"description": null,
"release_date": null,
"parties": [],
"keywords": [],
"homepage_url": null,
"download_url": null,
"size": null,
"sha1": null,
"md5": null,
"sha256": null,
"sha512": null,
"bug_tracking_url": null,
"code_view_url": null,
"vcs_url": null,
"copyright": null,
"holder": null,
"declared_license_expression": null,
"declared_license_expression_spdx": null,
"license_detections": [],
"other_license_expression": null,
"other_license_expression_spdx": null,
"other_license_detections": [],
"extracted_license_statement": null,
"notice_text": null,
"source_packages": [],
"file_references": [],
"is_private": true,
"is_virtual": false,
"extra_data": {},
"dependencies": [
{
"purl": "pkg:conda/[email protected]",
"extracted_requirement": "=1.0.2",
"scope": "dependencies",
"is_runtime": true,
"is_optional": false,
"is_pinned": true,
"is_direct": true,
"resolved_package": {},
"extra_data": {}
}
],
"repository_homepage_url": null,
"repository_download_url": null,
"api_data_url": null,
"datasource_id": "conda_yaml",
"purl": "pkg:conda/test"
}
]
2 changes: 1 addition & 1 deletion tests/packagedcode/data/misc/conda/scenicplus.yaml-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
"notice_text": null,
"source_packages": [],
"file_references": [],
"is_private": false,
"is_private": true,
"is_virtual": false,
"extra_data": {
"channels": [
Expand Down
6 changes: 6 additions & 0 deletions tests/packagedcode/test_conda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,3 +113,9 @@ def test_parse_conda_yaml_simple_dependencies(self):
package = conda.CondaYamlHandler.parse(test_file)
expected_loc = self.get_test_loc('conda/conda-yaml/ringer/environment.yaml-expected.json')
self.check_packages_data(package, expected_loc, regen=REGEN_TEST_FIXTURES)

def test_parse_conda_yaml_does_not_fail_on_test_files_with_port(self):
test_file = self.get_test_loc('conda/conda-yaml/test/environment_host_port.yml')
package = conda.CondaYamlHandler.parse(test_file)
expected_loc = self.get_test_loc('conda/conda-yaml/test/environment_host_port.yml-expected.json')
self.check_packages_data(package, expected_loc, regen=REGEN_TEST_FIXTURES)