Skip to content

Commit ee5d36d

Browse files
madpahmadpah
authored
Merge pull request #38 from CycloneDX/feat/conda-support
feat: add support for Conda
2 parents f132c92 + 2d01116 commit ee5d36d

File tree

12 files changed

+850
-35
lines changed

12 files changed

+850
-35
lines changed

‎README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ You can use one of the parsers to obtain information about your project or envir
4949

5050
| Parser | Class / Import | Description |
5151
| ------- | ------ | ------ |
52+
| CondaListJsonParser | `from cyclonedx.parser.conda import CondaListJsonParser` | Parses input provided as a `str` that is output from `conda list --json` |
53+
| CondaListExplicitParser | `from cyclonedx.parser.conda import CondaListExplicitParser` | Parses input provided as a `str` that is output from `conda list --explicit` or `conda list --explicit --md5` |
5254
| Environment | `from cyclonedx.parser.environment import EnvironmentParser` | Looks at the packaged installed in your current Python environment. |
5355
| PipEnvParser | `from cyclonedx.parser.pipenv import PipEnvParser` | Parses `Pipfile.lock` content passed in as a string. |
5456
| PipEnvFileParser | `from cyclonedx.parser.pipenv import PipEnvFileParser` | Parses the `Pipfile.lock` file at the supplied path. |
@@ -194,6 +196,11 @@ _Note: We refer throughout using XPath, but the same is true for both XML and JS
194196
<td>Y</td><td>Y</td><td>Y</td><td>Y</td>
195197
<td>&nbsp;</td>
196198
</tr>
199+
<tr>
200+
<td><code>./externalReferences</code></td>
201+
<td>Y</td><td>Y</td><td>Y</td><td>N/A</td>
202+
<td>Not all Parsers have this information. It will be populated where there is information available.</td>
203+
</tr>
197204
<tr>
198205
<td><code>./hashes</code></td>
199206
<td>Y</td><td>Y</td><td>Y</td><td>Y</td>

‎cyclonedx/parser/__init__.py

Lines changed: 24 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -24,28 +24,28 @@
2424
information is obtained by each set of Parsers. It does NOT guarantee the information is output in the resulting
2525
CycloneDX BOM document.
2626
27-
| Data Path | Environment | Pipenv | Poetry | Requirements |
27+
| Data Path | Conda | Environment | Pipenv | Poetry | Requirements |
2828
| ----------- | ----------- | ----------- | ----------- | ----------- |
29-
| `component.supplier` | N (if in package METADATA) | N/A | | |
30-
| `component.author` | Y (if in package METADATA) | N/A | | |
31-
| `component.publisher` | N (if in package METADATA) | N/A | | |
32-
| `component.group` | - | - | - | - |
33-
| `component.name` | Y | Y | Y | Y |
34-
| `component.version` | Y | Y | Y | Y |
35-
| `component.description` | N | N/A | N | N/A |
36-
| `component.scope` | N | N/A | N | N/A |
37-
| `component.hashes` | N/A | Y - see below (1) | Y - see below (1) | N/A |
38-
| `component.licenses` | Y (if in package METADATA) | N/A | N/A | N/A |
39-
| `component.copyright` | N (if in package METADATA) | N/A | N/A | N/A |
40-
| `component.cpe` | _Deprecated_ | _Deprecated_ | _Deprecated_ | _Deprecated_ |
41-
| `component.purl` | Y | Y | Y | Y |
42-
| `component.swid` | N/A | N/A | N/A | N/A |
43-
| `component.modified` | _Deprecated_ | _Deprecated_ | _Deprecated_ | _Deprecated_ |
44-
| `component.pedigree` | N/A | N/A | N/A | N/A |
45-
| `component.externalReferences` | N/A | Y - see below (1) | Y - see below (1) | N/A |
46-
| `component.properties` | N/A | N/A | N/A | N/A |
47-
| `component.components` | N/A | N/A | N/A | N/A |
48-
| `component.evidence` | N/A | N/A | N/A | N/A |
29+
| `component.supplier` | N | N (if in package METADATA) | N/A | | |
30+
| `component.author` | N | Y (if in package METADATA) | N/A | | |
31+
| `component.publisher` | N | N (if in package METADATA) | N/A | | |
32+
| `component.group` | - | - | - | - | - |
33+
| `component.name` | Y |Y | Y | Y | Y |
34+
| `component.version` | Y |Y | Y | Y | Y |
35+
| `component.description` | N |N | N/A | N | N/A |
36+
| `component.scope` | N |N | N/A | N | N/A |
37+
| `component.hashes` | Y - see below (2) | N/A | Y - see below (1) | Y - see below (1) | N/A |
38+
| `component.licenses` | N | Y (if in package METADATA) | N/A | N/A | N/A |
39+
| `component.copyright` | N |N (if in package METADATA) | N/A | N/A | N/A |
40+
| `component.cpe` | _Deprecated_ |_Deprecated_ | _Deprecated_ | _Deprecated_ | _Deprecated_ |
41+
| `component.purl` | Y |Y | Y | Y | Y |
42+
| `component.swid` | N/A |N/A | N/A | N/A | N/A |
43+
| `component.modified` | _Deprecated_ |_Deprecated_ | _Deprecated_ | _Deprecated_ | _Deprecated_ |
44+
| `component.pedigree` | N/A |N/A | N/A | N/A | N/A |
45+
| `component.externalReferences` | Y - see below (3) | N/A | Y - see below (1) | Y - see below (1) | N/A |
46+
| `component.properties` | N/A | N/A | N/A | N/A | N/A |
47+
| `component.components` | N/A | N/A | N/A | N/A | N/A |
48+
| `component.evidence` | N/A | N/A | N/A | N/A | N/A |
4949
5050
**Legend:**
5151
@@ -61,6 +61,9 @@
6161
supports only a single set of hashes identifying a single artefact at `component.hashes`. To cater for this
6262
situation in Python, we add the hashes to `component.externalReferences`, as we cannot determine which package was
6363
actually obtained and installed to meet a given dependency.
64+
2. MD5 hashses are available when using the `CondaListExplicitParser` with output from the conda command
65+
`conda list --explicit --md5` only.
66+
3. For Conda, we provide a link to the registry as provided in the Conda output.
6467
6568
"""
6669

‎cyclonedx/parser/conda.py

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# encoding: utf-8
2+
3+
# This file is part of CycloneDX Python Lib
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# SPDX-License-Identifier: Apache-2.0
18+
# Copyright (c) OWASP Foundation. All Rights Reserved.
19+
import json
20+
from abc import abstractmethod
21+
from typing import List
22+
23+
from . import BaseParser
24+
from ..model import ExternalReference, ExternalReferenceType
25+
from ..model.component import Component
26+
from ..utils.conda import parse_conda_json_to_conda_package, parse_conda_list_str_to_conda_package, CondaPackage
27+
28+
29+
class _BaseCondaParser(BaseParser):
30+
"""
31+
Internal abstract parser - not for programatic use.
32+
33+
"""
34+
def __init__(self, conda_data: str):
35+
super().__init__()
36+
self._conda_packages: List[CondaPackage] = []
37+
self._parse_to_conda_packages(data_str=conda_data)
38+
self._conda_packages_to_components()
39+
40+
@abstractmethod
41+
def _parse_to_conda_packages(self, data_str: str):
42+
"""
43+
Abstract method for implementation by concrete Conda Parsers
44+
45+
Params:
46+
data_str:
47+
`str` data passed into the Parser
48+
49+
Returns:
50+
A `list` of `CondaPackage` instances parsed.
51+
"""
52+
pass
53+
54+
def _conda_packages_to_components(self):
55+
"""
56+
Converts the parsed `CondaPackage` instances into `Component` instances.
57+
58+
"""
59+
for conda_package in self._conda_packages:
60+
c = Component(
61+
name=conda_package['name'], version=str(conda_package['version'])
62+
)
63+
c.add_external_reference(ExternalReference(
64+
reference_type=ExternalReferenceType.DISTRIBUTION,
65+
url=conda_package['base_url'],
66+
comment=f"Distribution name {conda_package['dist_name']}"
67+
))
68+
69+
self._components.append(c)
70+
71+
72+
class CondaListJsonParser(_BaseCondaParser):
73+
"""
74+
This parser is intended to receive the output from the command `conda list --json`.
75+
"""
76+
77+
def _parse_to_conda_packages(self, data_str: str):
78+
conda_list_content = json.loads(data_str)
79+
80+
for package in conda_list_content:
81+
conda_package = parse_conda_json_to_conda_package(conda_json_str=json.dumps(package))
82+
if conda_package:
83+
self._conda_packages.append(conda_package)
84+
85+
86+
class CondaListExplicitParser(_BaseCondaParser):
87+
"""
88+
This parser is intended to receive the output from the command `conda list --explicit` or
89+
`conda list --explicit --md5`.
90+
"""
91+
92+
def _parse_to_conda_packages(self, data_str: str):
93+
for line in data_str.replace('\r\n', '\n').split('\n'):
94+
line = line.strip()
95+
conda_package = parse_conda_list_str_to_conda_package(conda_list_str=line)
96+
if conda_package:
97+
self._conda_packages.append(conda_package)

‎cyclonedx/utils/__init__.py

Whitespace-only changes.

‎cyclonedx/utils/conda.py

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# encoding: utf-8
2+
3+
# This file is part of CycloneDX Python Lib
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# SPDX-License-Identifier: Apache-2.0
18+
# Copyright (c) OWASP Foundation. All Rights Reserved.
19+
import json
20+
import sys
21+
from json import JSONDecodeError
22+
from typing import Union
23+
24+
if sys.version_info >= (3, 8, 0):
25+
from typing import TypedDict
26+
else:
27+
from typing_extensions import TypedDict
28+
29+
from urllib.parse import urlparse
30+
31+
32+
class CondaPackage(TypedDict):
33+
"""
34+
Internal package for unifying Conda package definitions to.
35+
"""
36+
base_url: str
37+
build_number: int
38+
build_string: str
39+
channel: str
40+
dist_name: str
41+
name: str
42+
platform: str
43+
version: str
44+
md5_hash: str
45+
46+
47+
def parse_conda_json_to_conda_package(conda_json_str: str) -> Union[CondaPackage, None]:
48+
try:
49+
package_data = json.loads(conda_json_str)
50+
except JSONDecodeError:
51+
print(f'Failed to decode JSON: {conda_json_str}')
52+
raise ValueError(f'Invalid JSON supplied - cannot be parsed: {conda_json_str}')
53+
54+
if 'md5_hash' not in package_data.keys():
55+
package_data['md5_hash'] = None
56+
57+
if isinstance(package_data, dict):
58+
return CondaPackage(**package_data)
59+
60+
return None
61+
62+
63+
def parse_conda_list_str_to_conda_package(conda_list_str: str) -> Union[CondaPackage, None]:
64+
"""
65+
Helper method for parsing a line of output from `conda list --explicit` into our internal `CondaPackage` object.
66+
67+
Params:
68+
conda_list_str:
69+
Line of output from `conda list --explicit`
70+
71+
Returns:
72+
Instance of `CondaPackage` else `None`.
73+
"""
74+
75+
line = conda_list_str.strip()
76+
77+
if line[0:1] == '#' or line[0:1] == '@' or len(line) == 0:
78+
# Skip comments, @EXPLICT or empty lines
79+
return None
80+
81+
# Remove any hash
82+
package_hash: str = None
83+
if '#' in line:
84+
hash_parts = line.split('#')
85+
if len(hash_parts) > 1:
86+
package_hash = hash_parts.pop()
87+
line = ''.join(hash_parts)
88+
89+
package_parts = line.split('/')
90+
package_name_version_build_string = package_parts.pop()
91+
package_arch = package_parts.pop()
92+
package_url = urlparse('/'.join(package_parts))
93+
94+
try:
95+
package_nvbs_parts = package_name_version_build_string.split('-')
96+
build_number_with_opt_string = package_nvbs_parts.pop()
97+
if '.' in build_number_with_opt_string:
98+
# Remove any .conda at the end if present or other package type eg .tar.gz
99+
pos = build_number_with_opt_string.find('.')
100+
build_number_with_opt_string = build_number_with_opt_string[0:pos]
101+
102+
if '_' in build_number_with_opt_string:
103+
bnbs_parts = build_number_with_opt_string.split('_')
104+
if len(bnbs_parts) == 2:
105+
build_number = int(bnbs_parts.pop())
106+
build_string = build_number_with_opt_string
107+
else:
108+
raise ValueError(f'Unexpected build version string for Conda Package: {conda_list_str}')
109+
else:
110+
build_string = None
111+
build_number = int(build_number_with_opt_string)
112+
113+
build_version = package_nvbs_parts.pop()
114+
package_name = '-'.join(package_nvbs_parts)
115+
except IndexError as e:
116+
raise ValueError(f'Error parsing {package_nvbs_parts} from {conda_list_str} IndexError: {str(e)}')
117+
118+
return CondaPackage(
119+
base_url=package_url.geturl(), build_number=build_number, build_string=build_string,
120+
channel=package_url.path[1:], dist_name=f'{package_name}-{build_version}-{build_string}',
121+
name=package_name, platform=package_arch, version=build_version, md5_hash=package_hash
122+
)

‎poetry.lock

Lines changed: 13 additions & 13 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,9 @@ python = "^3.6"
4141
packageurl-python = "^0.9.4"
4242
requirements_parser = "^0.2.0"
4343
setuptools = "^50.3.2"
44-
importlib-metadata = "^4.8.1"
44+
importlib-metadata = { version = "^4.8.1", python = "~3.6 | ~3.7" }
4545
toml = "^0.10.2"
46+
typing-extensions = { version = "^3.10.0", python = "~3.6 | ~3.7" }
4647

4748
[tool.poetry.dev-dependencies]
4849
tox = "^3.24.3"

0 commit comments

Comments
 (0)