---

Black Duck SCA Scan Yocto Script - bd_scan_yocto_via_sbom.py v1.4.1

---

Provision of This Script

This script is provided under the MIT license (see LICENSE file).

It does not extend the licensed functionality of Black Duck Software. It is provided as-is, without warranty or liability.

This script is for licensed users of Black Duck Software only. You will need a Black Duck SCA server and an API key to use it.

For comments or issues, please raise a GitHub issue in this repository. Black Duck Support cannot respond to support tickets for this open-source utility. Users are encouraged to engage with the authors to address any identified issues.

Refer to How to Report Issues below for the minimum required information to report issues with this script.

---

Quick Start

See the quick start guide.

---

Introduction

Overview of bd_scan_yocto_via_sbom

While Black Duck Detect is the default scan utility for Black Duck SCA and supports Yocto projects, its Yocto scans have limitations. A BD Detect scan will only identify standard recipes from OpenEmbedded.org and does not cover:

• Modified or custom recipes.
• Recipes moved to new layers or with changed package versions/revisions.
• Custom kernel identification or kernel vulnerability mapping.
• Copyright and deep license data.
• Snippet or other code scanning within custom recipes.

This utility addresses these gaps by creating a comprehensive Black Duck SCA project from your Yocto build. It achieves this by:

• Scanning Yocto project artifacts to generate an SPDX SBOM (Software Bill of Materials), which is then uploaded to your Black Duck server to create a project version (--modes OE_RECIPES)
• Filtering and "fixing-up" recipes using data from the OpenEmbedded (OE) APIs to correctly identify local recipes moved to new layers or with different local versions/revisions.
• Signature scanning packages and downloaded archives for recipes not matched by OE data (with the option to scan all packages - --modes SIG_SCAN or --modes SIG_SCAN_ALL)
• Optionally adding unmatched components via CPE lookup or by creating custom components through a second SBOM upload (--modes CPE_COMPS and/or --modes CUSTOM_COMPS)
• Applying patches for locally patched CVEs identified from cve_check data (if available - --modes CVE_PATCHES)
• Optionally processing the Linux Kernel to filter out vulnerabilities not within the sources compiled into your specific kernel build (--modes IMAGE_MANIFEST,KERNEL_VULNS)

For guidance on optimizing Yocto project scans with this utility, refer to the Best Practice Recommendations section below.

Note the addition of new --modes parameter in v1.2.0+ to control scans to be performed (legacy scan parameters still supported) - see Scan Mode section below.

Why is This Script Needed?

Yocto is a powerful, local build system for creating custom Linux images. It's not a package manager but a highly customizable environment. Key characteristics that make comprehensive scanning challenging include:

• No Centralized Repository: Unlike other Linux distributions, Yocto downloads software packages from external locations at build time.
• Recipe Misconception: Yocto recipe names do not fully identify software packages. Standard recipes are provided as templates but are only local text files in a project and can be modified to download different software from anywhere - with the potential for supply chain hacking.
• Extensive Customization: Yocto projects can be heavily modified, with unlimited changes to the build environment, layers, recipes, and the addition of custom recipes and patches.
• Hidden Dependencies: Packages within recipes can embed other open-source software not explicitly listed in the Yocto manifest.
• Post-Build Analysis: Due to Yocto's dynamic nature, it's only feasible to accurately determine what was built and packaged into an image after the build is complete.

This script employs multiple techniques to reverse-engineer a built Yocto project to identify its included packages, report license information, and map vulnerabilities from the package origins.

The script can also identify and process the Linux kernel, determining its custom configuration and modules to ignore vulnerabilities reported against excluded modules.

Understanding Yocto Recipes

Openembedded.org (OE) provides a list of standard Yocto builds, layers and recipes for use as templates (see layers.openembedded.org). Most Yocto projects start using a template but can be extensively modified thereafter.

Yocto recipes are essentially text config files used by Bitbake (the Yocto build tool) to download software packages or sources from external locations.

The Black Duck SCA Knowledgebase has onboarded the majority of standard Yocto recipes by creating mappings to the original component referenced in the recipe (download location). The advantage of mapping recipes to the original component means that vulnerabilities are reported accurately, and the original license definition will be used (as opposed to the potentially inaccurate license definition reported in the recipe file).

This utility downloads the list of standard OE recipes from openembedded.org (OE data) during a scan, checking recipes from the project manifest to ensure they are correctly specified (this is required because Yocto layers can be renamed and recipes moved to new layers locally). In most cases a standard OE recipe in the manifest should result in a direct component match in the BD project (Bill of Materials or BOM).

However, Yocto recipes can be modified in a project to upgrade packages and add patches, or other (non-standard) recipes can be imported or created to generate Custom Recipes.

Custom (non-standard) Recipes have either been created by:

1. Using a recipe from a non-OE source (comprising either original OSS, modified OSS or custom/commercial code from 3rd parties)
2. Modifying an existing standard OE recipe to use a new package version
3. Creating a new recipe to install different OSS
4. Creating a recipe for your own code

This utility provides alternative scan techniques to support identification of non-OE (Custom) recipes including SIG_SCAN, CPE_COMPS and CUSTOM_COMPS modes which should be specified to ensure a complete bill of materials (BOM) - see FAQ 1.

A small number of standard OE recipes may not exist in the Black Duck KB, and will not map to a component using the OE_RECIPES mode as expected, so the SIG_SCAN, CPE_COMPS and CUSTOM_COMPS modes may be required to map these into the BD project.

Note: For Custom Recipes of type 2 above, you can also consider using the ‘--max_oe_version_distance’ option to map the most recent known version from the OE data.

---

Prerequisites

Before running the script, ensure you meet the following requirements:

1. Yocto v2.1 or newer
2. Python 3.10 or newer
3. Black Duck SCA server 2024.7 or newer
4. Black Duck SCA API Token with either:
   • Global Code Scanner and Global Project Manager roles, OR
   • Project Code Scanner and BOM Manager roles for an existing project (also Project Manager role to patch project vulnerabilities if CVE_PATCH mode used).
   • (Also Global Component Manager needed to create custom components if --mode CUSTOM_COMPS specified)
5. Single-target Bitbake configurations only are supported. Run this utility on one target at a time.
6. A built Yocto project with access to the build platform. Alternatively, specific outputs from the build can be used, though many script features may be limited so not recommended.
7. Ensure license text entries in manifest files are SPDX compliant to support SBOM upload. Custom licenses are not supported for the creation of custom components (mode=CUSTOM_COMPS) - use --ignore_licenses to bypass license checks.

---

Installation

You have a few options for installing this utility:

Option 1: Install from PyPI (Recommended)

1. Create a virtual environment:

   python3 -m venv venv
   source venv/bin/activate # On Windows: .\venv\Scripts\activate

2. Install the package:

   pip3 install bd_scan_yocto_via_sbom --upgrade

Option 2: Build and Install Locally

1. Clone the repository:

   git clone <repository-url>
   cd bd_scan_yocto_via_sbom

2. Create a virtual environment (if you haven't already):

   python3 -m venv venv
   source venv/bin/activate # On Windows: .\venv\Scripts\activate

3. Build the utility:

   python3 -m build

4. Install the package:

   pip3 install dist/bd_scan_yocto_via_sbom-1.0.X-py3-none-any.whl --upgrade

   (Replace 1.0.X with the actual version number of the built wheel file.)

Option 3: Clone Repository and Install Prerequisites

1. Clone the repository:

   git clone <repository-url>
   cd bd_scan_yocto_via_sbom

2. Ensure prerequisite packages are installed (see the pyproject.toml file for a list).

   pip install .

---

How to Run

If you installed the utility as a package:

1. Set the Bitbake environment (e.g., source oe-init-build-env).
2. Activate your virtual environment (e.g., source venv/bin/activate).
3. Run the script:

   bd-scan-yocto-via-sbom PARAMETERS

If you cloned the repository locally:

1. Set the Bitbake environment (e.g., source oe-init-build-env).
2. Activate your virtual environment where dependency packages were installed (e.g., source venv/bin/activate).
3. Run the script:

   python3 PATH_TO_REPOSITORY/run.py PARAMETERS

---

Best Practice Recommendations

For optimal Yocto scan results, review the following:

1. Check required scan modes using --modes: - the default scans (if --modes not specified) are OE_RECIPES,SIG_SCAN,CVE_PATCHES (same as --modes DEFAULT) - see Scan Modes below.
   
2. Optionally override Bitbake Environment Values: By default, the utility calls Bitbake -e to extract environment and layer information, and will refer to the latest Yocto build. You can override values including license.manifest, machine, target, download_dir, package_dir, and image_package_type using command-line parameters.
3. Generate a Recipe Report: Use the --recipe_report REPFILE parameter to create a report of matched and unmatched recipes in the Bill of Materials (BOM), required for analysis and debugging. IMPORTANT Unmatched IDs shown the BD project version are only from the 1st scan stage and should not be used to determine missing recipes if you use scan modes in addition to OE_RECIPES - see FAQs.
4. Cache OE Data: The --oe_data_folder FOLDER parameter allows you to cache downloaded OE data (approx. 300MB) and reuse it in subsequent runs, saving download time. OE data doesn't change very frequently.
5. Apply Patched CVEs to BD Project: Add the cve_check class to your Bitbake local.conf to identify patched CVEs. Ensure PHASE 7 picks up the cve-check file. Optionally, specify the output CVE check file using --cve_check_file FILE if an alternative location is needed. See CVE Patching for more information.
6. Fuzzy Match Modified Recipes: For recipes modified from standard OE versions, optionally use --max_oe_version_distance X.X.X (e.g., 0.0.1 to 0.0.10) for fuzzy matching against OE recipes. Be cautious, as this can sometimes disable correct matches. It's recommended to create two projects and compare results with and without this parameter. See OE Difference Calculations for more information.
7. Process Image Manifest: To include the Linux kernel and other packages specified in the image manifest, consider adding --modes IMAGE_MANIFEST. Optionally, specify the image_license.manifest file path (--image_license_manifest FILEPATH) if the latest build is not desired. You may also need to add mode CPE_COMPS to add the kernel.
8. Add Components by CPE: Add --modes CPE_COMPS to create packages not matched by other methods through CPE lookup. Note that only packages with reported vulnerabilities have legitimate published CPEs - see OWASP report on component identification (PDF).
9. Process Kernel Vulnerabilities: To ignore kernel vulnerabilities not within compiled kernel sources, add --modes KERNEL_VULNS (assumes IMAGE_MANIFEST mode).
10. Add Custom Components for Unmatched recipes: Where OSS recipes are not matched by any other method, and for custom or commercial recipes, you can consider adding --modes CUSTOM_COMPS to create Custom Components for unmatched recipes and add them to the BD project as placeholders. However, note that Custom Components do not have any vulnerabilities mapped, will use the license specified in the license.manifest and cannot be easily deleted from the BD server once created without deleting all projects where referenced. Also note that all future scans for the same recipe will map to the created Custom Components even when KB updates add new recipes (unless the associated PURL is deleted under 'Management-->Unmatched Components').

---

Scan Modes

The --modes option is a comma-delimited list of modes (no spaces), used to control the multiple types of scan supported in the script. It can be used to replace existing scan control parameters and simplify the command line.

Explanation of modes:

• DEFAULT - Includes OE_RECIPES,SIG_SCAN,CVE_PATCHES
• OE_RECIPES - Map components by OE recipe lookup (set by DEFAULT)
• IMAGE_MANIFEST - Process image manifest in addition to standard manifest (usually to add linux kernel). You may need to use mode CPE_COMPS to add a custom kernel version
• SIG_SCAN - Scan unmatched recipes/packages using Signature scan (set in DEFAULT mode) - see FAQs
• SIG_SCAN_ALL - Scan all recipes/packages using Signature scan
• CPE_COMPS - Add unmatched recipes as packages by CPE lookup (where CPEs available - only works for components with reported vulnerabilities)- See OWASP report on component identification (PDF).
• CUSTOM_COMPS - Create Custom Components for unmatched recipes (note Custom Components are only placeholders for SBOM export - no vulnerability or other data is provided) - see FAQs
• CVE_PATCHES - Process locally patched CVEs from cve_check class (set in DEFAULT mode) - See CVE Patching
• KERNEL_VULNS - Process kernel modules and mark vulns as unaffected where associated modules do not exist in the kernel
• ALL - Includes OE_RECIPES,IMAGE_MANIFEST,SIG_SCAN,CVE_PATCHES,CPE_COMPS,CUSTOM_COMPS,KERNEL_VULNS (but not SIG_SCAN_ALL)

Notes:

• DEFAULT is assumed if --modes not specified.
• To add scan modes to the default set use, for example --modes DEFAULT,IMAGE_MANIFEST,KERNEL_VULNS.
• To add SIG_SCAN_ALL mode (scanning all packages) to ALL scan mode use --modes ALL,SIG_SCAN_ALL.

Mapping of legacy (pre-v1.2.0) scan control parameters to scan modes:

• --process_image_manifest = mode IMAGE_MANIFEST
• --scan_all_packages = mode SIG_SCAN_ALL
• --add_comps_by_cpe = mode CPE_COMPS
• --process_kernel_vulns = mode KERNEL_VULNS
• --sbom_create_custom_components = mode CUSTOM_COMPS

Specifying legacy parameters in addition to --modes will override scan modes defined in the modes list (including modes in DEFAULT).

---

Other Scan Options

• Improve Recipe Release Identification: Optionally run bitbake -g to create a task-depends.dot file which is specified in --task_depends_dot_file FILE along with -l license.manifest. If -l license.manifest is not also specified, all development dependencies will be processed (not just those within the license manifest).
• Signature scan ALL packages (as opposed to only unmatched packages): Add --modes SIG_SCAN_ALL to scan all packages.
• Turn on Snippet scanning within Signature scanned packages: Requires --modes SIG_SCAN or --modes SIG_SCAN_ALL - add the parameter --detect_opts detect.blackduck.signature.scanner.snippet.matching=SNIPPET_MATCHING to enable snippet scanning of sources within scanned packages.

---

Command Line Parameters

usage: bd-scan-yocto-via-sbom [-h] [--blackduck_url BLACKDUCK_URL] [--blackduck_api_token BLACKDUCK_API_TOKEN] [--blackduck_trust_cert] [-p PROJECT] [-v VERSION] <OTHER PARAMETERS>

Create BD-SCA project version from Yocto project

• -h, --help: Show help message and exit.

Black Duck Project Parameters - REQUIRED:

• --blackduck_url BLACKDUCK_URL: Black Duck server URL (also uses BLACKDUCK_URL environment variable).
• --blackduck_api_token BLACKDUCK_API_TOKEN: Black Duck API token (also uses BLACKDUCK_API_TOKEN environment variable).
• -p PROJECT, --project PROJECT: Black Duck project to create or update.
• -v VERSION, --version VERSION: Black Duck project version to create or update.
• -t TARGET, --target TARGET: Yocto target (e.g., 'core-image-sato' - single target configuration only supported).
• --blackduck_trust_cert: Trust Black Duck server certificate (also uses BLACKDUCK_TRUST_CERT environment variable) - OPTIONAL.

Scan Mode Configuration Parameter - OPTIONAL:

• --modes MODES: A comma-delimited list of scan modes (no spaces) selected from [ALL,DEFAULT,OE_RECIPES,IMAGE_MANIFEST,SIG_SCAN,SIG_SCAN_ALL,CVE_PATCHES,CPE_COMPS,CUSTOM_COMPS,KERNEL_VULNS] - see Scan Modes.

Yocto Project Configuration Parameters - OPTIONAL:

• -l LICENSE_MANIFEST, --license_manifest LICENSE_MANIFEST: The most recent Yocto license.manifest file is located by default, usually located in tmp/deploy/licenses/<yocto-image>-<yocto-machine>/license.manifest. Use this option to specify a different build or if the file is in a non-standard location (or if --skip_bitbake is used).
• -i IMAGE_LICENSE_MANIFEST, --image_license_manifest IMAGE_LICENSE_MANIFEST: If --modes IMAGE_MANIFEST specified, the latest image manifest will be identified by default from the environment. Specify the path to an alternate image_license.manifest file if the latest build is not desired or the file exists in another location.
• --task_depends_dot_file FILE: Specify the path to the task-depends.dot file created by bitbake -g for processing. If -l license.manifest is not also specified, all recipes, including dev dependencies, will be processed (--target is also required).
• -b BITBAKE_LAYERS, --bitbake_layers_file BITBAKE_LAYERS: Optionally specify the output of the command bitbake-layers show-recipes stored in a file. Should be used with --skip_bitbake (although it will prevent many other script modes from operating, so it's not recommended).
• -c CVE_CHECK_FILE, --cve_check_file CVE_CHECK_FILE: CVE check output file (in .cve or .json format) to mark locally patched CVEs. The most recent file will be located by default; use this parameter to specify an alternate file. Usually located in build/tmp/deploy/images/XXX. Should be determined from the Bitbake environment by default (unless --skip_bitbake is used).
• --build_dir BUILD_DIR: Alternate Yocto build folder (defaults to poky/build).
• --download_dir DOWNLOAD_DIR: Alternate directory where original OSS source is downloaded (defaults to poky/build/downloads).
• --package_dir PACKAGE_DIR: Alternate directory where package files are downloaded (e.g., poky/build/tmp/deploy/rpm/<ARCH>).
• --image_package_type IMAGE_PACKAGE_TYPE: Package type used for installing packages (specify one of rpm, deb, or ipx - default rpm).
• --kernel_recipe RECIPE_NAME: Define a non-standard kernel recipe name (defaults to 'linux-yocto').
• --exclude_recipes RECIPE_LIST: Exclude specified recipes from BOM (comma-delimited).
• --exclude_layers LAYER_LIST: Exclude specified layers from BOM (comma-delimited).
• --ignore_licenses: Ignore licenses when creating custom components (Use this option if Phase 5 SBOM upload fails when trying to add custom components - custom components will have Unknown license).

Script Behavior Parameters - OPTIONAL:

• --skip_oe_data: Do not use layers/recipes/layers from layers.openembedded.org to review origin layers and revisions within recipes to ensure more components are matched against the Black Duck KnowledgeBase (KB). If mode OE_RECIPES is still specified then recipes will be uploaded to create the project but without any checking against OE data.
• --oe_data_folder OE_DATA_FOLDER: Folder to contain OE data files. If files don't exist, they will be downloaded; if they exist, they will be used without re-downloading. Creates OE data files in the specified folder if they don't already exist. If files exist, they are used without re-downloading. This allows offline usage of OE data or reduces large data transfers if the script is run frequently. RECOMMENDED.
• --max_oe_version_distance MAX_OE_VERSION_DISTANCE: When no exact match, use the closest previous recipe version up to the specified distance against OE data. Setting this value allows close (previous) recipe version matching. The value must be in MAJOR.MINOR.PATCH format (e.g., 0.10.0). CAUTION: Setting this value too high may cause components to be matched against older recipes in the OE data, potentially leading to different vulnerability reports. It's generally better to maintain a close relationship between matched versions and project versions. Consider values in the range 0.0.1 to 0.0.10. See OE Difference Calculations.
• --skip_sig_scan: Do not signature scan downloads and packages. By default, only recipes not matched from OE data are scanned (equivalent to removing SIG_SCAN from --modes)

Connection & BD Detect Configuration Parameters - OPTIONAL:

• --detect_jar_path DETECT_JAR_PATH: Path to the BD Detect JAR file.
• --detect_opts DETECT_OPTS: Additional BD Detect options, comma separated list (remove leading -- from BD Detect options).
• --api_timeout: Specify API timeout in seconds (default 60). Used in BD Detect as --detect.timeout.
• --unmap: Unmap previous code locations (scans) when running the initial scan (default is not to unmap).

Legacy Scan Mode Parameters (replaced by --modes):

• --process_image_manifest: Process the image_license.manifest file (default location) to include recipes from the core image - equivalent to --modes IMAGE_MANIFEST.
• --add_comps_by_cpe: Look up CPEs for packages not mapped in the BOM or discovered by signature scan and add missing packages. Note: not all packages have published CPEs - equivalent to --modes CPE_COMPS.
• --process_kernel_vulns: Process compiled kernel sources to ignore vulnerabilities affecting modules not compiled into the custom kernel. Requires --process_image_manifest - equivalent to --modes KERNEL_VULNS.
• --scan_all_packages: Signature scan all packages (default: only recipes not matched from OE data are scanned) - equivalent to --modes SIG_SCAN_ALL.
• --sbom_create_custom_components: Create Custom Components when all other scan techniques do not match recipes. The license reported in license.manifest will be used. Custom Components are designed for unknown components but lack associated vulnerabilities, deep license data, or copyrights. Caution: This parameter creates new, permanent custom components mapped to PURLs. These PURLs will be matched in all future scans, even if new data is added to the KB for the recipe. Use the Manage --> Unmatched Origins view to disassociate PURLs from custom components if you don't want them matched in future scans. Deleting custom components once created and referenced in projects is not possible unless the projects themselves are removed. Do not use this parameter until you fully understand its implications - equivalent to --modes CUSTOM_COMPS.

General Parameters:

• --recipe_report REPFILE: Create a report file with a list of recipes, including those not matched in the BOM.
• -o OUTPUT, --output OUTPUT: Specify output SPDX SBOM file. If specified, only the initial SBOM will be created and all other script features will be skipped (use for debug purposes only).
• --debug: Enable debug logging mode.
• --logfile LOGFILE: Output logging messages to a specified file.

---

Minimum Required Parameters

To run the script, you must provide:

--blackduck_url BD_URL --blackduck_api_token BD_API -p BD_PROJECT -v BD_VERSION -t YOCTO_TARGET

Optionally, add --blackduck_trust_cert to trust the Black Duck server certificate.

Consider using --oe_data_folder FOLDER with an existing folder to cache OE recipe data on subsequent runs, saving large downloads.

Server credentials can also be specified using the standard environment variables: BLACKDUCK_URL, BLACKDUCK_API_TOKEN, and BLACKDUCK_TRUST_CERT.

---

Example Distance Calculations for --max_oe_version_difference

When no exact match, use this option to find the most recent previous recipe version up to the specified distance against OE data. Setting this value allows close (previous) recipe version matching. The value must be in MAJOR.MINOR.PATCH format (e.g., 0.10.0). CAUTION: Setting this value too high may cause components to be matched against older recipes in the OE data, potentially leading to different vulnerability reports. It's generally better to maintain a close relationship between matched versions and project versions. Consider values in the range 0.0.1 to 0.0.10.

• Recipe version is 3.2.4 - closest previous OE recipe version is 3.2.1: Distance value needed: minimum 0.0.3
• Recipe version is 3.2.4 - closest previous OE recipe version is 3.0.1: Distance value needed: minimum 0.2.0
• Recipe version is 3.2.4 - closest previous OE recipe version is 2.0.1: Distance value needed: minimum 1.0.0

Note that lower-order values are overridden by higher-order values (e.g., a distance of 1.0.0 is equivalent to 1.999.999).

---

CVE Patching

The Yocto cve_check class processes Bitbake dependencies within the development environment, generating a list of CVEs identified from the NVD for all packages in that environment.

For patched CVE remediation in your Black Duck project, you need to add the cve_check bbclass to your Yocto build configuration to generate the CVE check log output. Add the following line to your build/conf/local.conf (or similar) file:

INHERIT += "cve-check"

Then, rebuild your project (e.g., using bitbake core-image-sato) to run the CVE check action and generate the required CVE log files without a full rebuild.

The script should automatically locate the cve-check output file (usually under poky/tmp/deploy/images/licenses), but you can also specify the file using the --cve_check_file CVE_CHECK_FILE parameter.

The CVE_PATCHES mode enables this feature which is included in the DEFAULT mode. Remove CVE_PATCHES from the list of modes to disable.

---

Troubleshooting

• Ensure required parameters are specified: Double-check that you've provided all minimum required parameters, including the Yocto target (-t).
• Verify prerequisites and build status: Confirm all prerequisites are met and that you have a built Yocto project with a generated license.manifest file - see Prerequisites
• Enable debug logging: Use the --debug parameter to turn on detailed logging.
• Check code locations in Black Duck: After the scan, verify that your Black Duck project's Source tab shows 2 or 3 separate code locations: one for the initial SBOM import, one for Signature Scan, and optionally one for the second SBOM used to create CPE matching/custom components.
• Review Recipe matching information: Specify --recipe_report FILE to create a full list of recipes and examine missing recipes; consider using --max_oe_version_difference X.X.X to enable fuzzy matching of OE recipes, or adding additional scan modes to add missing recipes.
• Missing packages: If a specific package appears missing from the project, confirm you are looking for the recipe name, not the package name (See FAQs for an explanation of Yocto recipes vs. packages). The scan modes SIG_SCAN, CPE_COMPS and CUSTOM_COMPS can be specified in --modes to process missing recipes using various techniques (alternatively use --modes ALL).
• Linux kernel missing: Consider adding the --modes IMAGE_MANIFEST mode to include packages in the image manifest, which usually includes the Linux kernel. Use --image_license_manifest PATH to specify a different path to the image_license.manifest file if in a non-standard location or the latest build is not desired. Check the Linux Kernel recipe name ('linux-yocto' is expected - use parameter --kernel_recipe mylinux to specify a non-standard recipe name for the kernel). If the kernel still isn't identified due to a custom name format, consider adding the required kernel version manually to the project.
• Unable to upload SPDX file during script run: All licenses in the license.manifest must be valid SPDX licenses for the SBOM to be importable. Check licenses at https://spdx.org/licenses/. Note that license text in the license.manifest is only used for components added as Custom Components (CUSTOM_COMPS), and is not referenced for components added by the OE_RECIPES,IMAGE_MANIFEST,SIG_SCAN,CPE_COMPS options as the licenses from the KB will be used instead, but the license entries must be SPDX compliant to be uploadable.

---

How to Report Issues with This Script

If you encounter issues, please create an issue in GitHub and include the following information:

• Your Organization name (to validate your Black Duck license).
• A clear description of the issue.
• Yocto version in use (obtain with bitbake -e | grep DISTRO_CODENAME).
• The license.manifest file for your build (and optionally image_license.manifest) if mode IMAGE_MANIFEST is used.
• The output of the command bitbake-layers show-recipes in a file.
• The log output from this script with --debug enabled (use --logfile LOGFILE to save to a file).
• The recipe report file (use --recipe_report REPFILE).

---

Additional Scan Possibilities (Outside This Script)

For custom C/C++ recipes or recipes built with other languages and package managers, you can combine this script with other scan types:

• C/C++ Recipes: The advanced blackduck_c_cpp utility can be used during the build process to identify compiled sources, system includes, and operating system dependencies. You would need to modify the recipe's build command to call blackduck-c-cpp as part of a scanning cycle after configuring it to connect to your Black Duck server.
• Package Manager-Based Recipes: For recipes utilizing a package manager, a standard BD Detect scan in DETECTOR mode can be used to analyze project dependencies separately.

Combining Multiple Scans: Multiple scans can be combined into the same Black Duck project. Ensure you use the BD Detect parameter --detect.project.codelocation.unmap=false to prevent previous scans from being unmapped.

---

Release Notes

• v1.4.1
  • Minor fix to layer detection for OE recipes, backup & increment recipe_report file
• v1.4.0
  • Multiple improvements to reporting and output, other recipe reporting and 2nd origin lookup, stopped running Detect if project exists. Also implemented fall-back when show-layers data missing to use OE data lookup.
• v1.3.2
  • Upgraded to BD Detect 11 and replaced --no_unmap with --unmap (default is not to unmap scans). Added --ignore_licenses option for custom components.
• v1.3.1
  • Added --exclude_recipes and --exclude_layers options
• v1.3.0
  • Fix for max_oe_version_distance and exact matches
• v1.2.5
  • Added extra comments in remediated vulns
• v1.2.4
  • Updated CVE patching using linked vulns from BDSAs using async calls - should mark vulns in BD project as patched/ignored
• v1.2.3
  • Removed license entries from SBOM in phase 3 to support non-compliant license types.
• v1.2.2
  • Merged PR to support IGNORED vulnerability types (in addition to PATCHED) and
• v1.2.1
  • Minor fix for CPE lookup version strings
• v1.2.0
  • Addition of --modes option to simplify multiple scan mode options:
  • Modes string can contain comma separated list (no spaces) from [ALL, OE_RECIPES*, IMAGE_MANIFEST, SIG_SCAN*, SIG_SCAN_ALL, CVE_PATCHES*, CPE_COMPS, CUSTOM_COMPS, KERNEL_VULNS] (modes marked with * run by default)
  • Note that legacy options to control scan modes are still supported
• v1.1.4
  • Minor bug fix for sbom_create_custom_components and more
• v1.1.3
  • Minor fix to enforce --process_image_manifest if --process_kernel_vulns specified (as image manifest required to locate kernel image package and extract kernel modules).
• v1.1.2
  • Improved custom component creation by SBOM import: added to the second SBOM import (which runs when --add_comps_by_cpe is specified).
  • Moved SBOM custom component creation step to after Signature scan alongside optional addition of packages using CPE - to ensure custom packages only added if truly unmatched.
  • Removed date/time from SBOM document name and signature scan temporary folder names, allowing rescanning without unmapping (required for BD Detect 11). Default unmapping will be removed when BD Detect 11 is implemented.
  • Removed asyncio as dependency (bundled in Python 3.4+)
• v1.1.1
  • Minor fix for cve_check output file identification.
• v1.1.0
  • Added --add_comps_by_cpe parameter to look for unmatched packages via CPE.
  • Changed vulnerability patching to use asynchronous updates for improved speed.
  • Added custom kernel source scanning to ignore vulnerabilities affecting uncompiled kernel modules (requires --process_kernel_vulns and process_image_manifest).

---

FAQs

1. Why is Recipe XXX missing from the Black Duck project?
   Recipes can be added to the BOM using multiple scan modes.
   Yocto recipes which are listed at layers.openembedded.org and known in the Black Duck KB should be added by the OE_RECIPES mode.
   However, if the recipe has been modified to use a new version or does not exist in the OE list, then the --max_oe_version_distance option can be used to try to find the most recent known OE recipe version.
   If still no match (or if a standard OE recipe is not known to the BD KB), then the SIG_SCAN mode can be used to scan the package to look for exact or modified Signature matches for known OSS - see FAQ 12 for more about Signature scanning. If still no match then CPE_COMPS mode can be used to lookup OSS components by CPE (only works for packages with reported vulnerabilities).
   Finally, the CUSTOM_COMPS mode will create placeholder custom components in the BOM for the recipe. If you use ANY non-OE recipes, then only if all options have been specified can all recipes be matched (see also FAQ 12 below for the requirement to have valid license identifiers for custom recipes).
   

2. I cannot see a specific package in the Black Duck project.
   Black Duck reports recipes in the Yocto project, not individual packages. Multiple packages can be combined into a single recipe and are generally considered part of that recipe's main component. Use the --recipe_report FILE parameter to list matched and missing recipes, and conider using other scan modes (--modes ALL or --modes CPE_COMPS,CUSTOM_COMPS) to add missing recipes. Note also that Signature scan (--modes SIG_SCAN or --modes SIG_SCAN_ALL) can identify missing recipes as well as embedded OSS within recipes.
   

3. Why can I see unmatched IDs in the project view?
   Unmatched IDs are shown from the first SBOM upload performed by the script in mode OE_RECIPES. The Unmatched IDs is a list of recipes which could not be matched because they are not listed at layers.openembedded.org (because they have been modified, they come from another non-OE source, are your own custom recipes containing your code or are unknown in the KB).
   If you have used other scan modes (SIG_SCAN, CPE_COMPS or CUSTOM_COMPS) then these unmatched recipes may have been matched by the Signature scan or CPE_COMPS and CUSTOM_COMPS modes. If you use all scan modes (and your custom recipes use valid SPDX license definitions; see FAQ 11 AND you have a recent BD server version) then you should have a full SBOM containing all recipes - custom/commercial/unknown recipes will be added as custom components.
   The only valid way to see the list matched and matched recipes is to view the console output or use the --recipe_report option - do not refer to the Unmatched IDs in the BOM view.

4. Why can't I rely on the license data provided by Yocto in the license.manifest file?
   The licenses reported by Bitbake come directly from the recipe files. However, the true applicable license for each package is the one declared in its origin repository, which may differ.
   Many OSS packages also embed other OSS with different licenses, sometimes with re-licensing restrictions. Black Duck uses licenses from the origin packages, can support full license text/copyrights and optional deep license analysis to identify embedded licenses within packages (for components created by SIG_SCAN or CPE_COMPS modes). Licenses from recipes are used when Custom Components are created using --modes CUSTOM_COMPS.
   

5. Can this utility be used on a Yocto image without access to the build environment?
   Yes, potentially. Use the parameters --skip_bitbake -l LIC_MANIFEST_FILE --bitbake_layers_file LAYERS_FILE, where LIC_MANIFEST_FILE is the path to your license.manifest file and LAYERS_FILE contains the output of bitbake-layers show-recipes.
   However, several scan modes will be disabled due to the lack of build artefacts (including Signature scanning of packages --modes SIG_SCAN, CVE patching --modes CVE_PATCHES and kernel vulnerability applicability --modes KERNEL_VULNS).
   

6. Why can't I simply use the cve-check class provided by Yocto to determine unpatched vulnerabilities?
   The cve-check class processes all recipes in the build and attempts to associate CVEs from the NVD using CPEs. This often leads to a large number of false positive CVEs, as it reports all packages (including build dependencies) rather than just those in the distributed image and the CPE is a wildcard often associating many vulnerabilities which are false positive. Furthermore, the CPE association data from the NVD is frequently inaccurate with no earliest affected version meaning that newer vulnerabilities are shown for all previous verisons of packages. Black Duck Security Advistories are expert-curated to reduce false positives including marking CVEs as ignored where they should not apply.
   

7. Why couldn't I just use the create-spdx class provided by Yocto to export a full SBOM?
   The Yocto create-spdx class generates SPDX JSON files for packages, including data such as file lists and hashes. However, many SPDX fields (e.g., license text, copyrights) are blank (NO-ASSERTION), and packages are not identified by PURL, stopping import into other tools (including Black Duck). The SPDX export also covers all recipes in the build environment as opposed to only those in the manifest.
   

8. I cannot see the Linux kernel in the Black Duck project.
   Consider using the --modes IMAGE_MANIFEST (optionally with --image_license_manifest PATH) to process packages in the image manifest, which usually includes the Linux kernel. You may also need to add the CPE_COMPS mode to add the kernel. If the kernel still cannot be identified due to a custom name format, consider adding the required kernel version manually to the project.
   

9. I am using another Yocto wrapper like KAS (https://kas.readthedocs.io/) and cannot run Bitbake, or the script fails for some other reason.
   Use the parameters --skip_bitbake -l LIC_MANIFEST_FILE --bitbake_layers_file LAYERS_FILE, where LIC_MANIFEST_FILE is the path to your license.manifest file and LAYERS_FILE contains the output of bitbake-layers show-recipes. Note that this will disable several features, including CVE patching, kernel vulnerability identification, and package scanning.
   

10. I want to scan all recipes, including development dependencies, as opposed to only those in the delivered image.
    Run the command bitbake -g to create a task-depends.dot file, then use the parameter --task_depends_dot_file FILE, where FILE is the path to the generated file.
    

11. Unable to upload SPDX file during script run in phase 5
    Where mode=CUSTOM_COMPS, licenses for custom components to be added from the license.manifest must be valid SPDX licenses for the SBOM to be importable. Use the --ignore_licenses option (custom components will have 'Unknown license') or check licenses for custom components at https://spdx.org/licenses/ (suggest checking compliance via an LLM) and modify any non-compliant license entries in the manifest files. Note that license text in the manifest files is only used to define the licenses for components added as Custom Components (CUSTOM_COMPS), but is not used for components added by the OE_RECIPES,IMAGE_MANIFEST,SIG_SCAN,CPE_COMPS options which will reference the licenses from the BD KB instead. Resolved licenses can be modified within the project version or globally once components have been added to the BOM.
    

12. Why do I see multiple extra components in the BOM when using SIG_SCAN or SIG_SCAN_ALL modes?
    Signature scanning examines package archives to look for hierarchical folder matches.
    If a package is an exact, unmodified copy of an original package known to the BD KB, then a component match will be shown for this component in the BOM. However, Signature scanning will continue to look for sub-matches within the package which can report embedded OSS components, or partial matches deeper in the hierarchy.
    The additional utility https://github.com/blackducksoftware/bd_sig_filter can be used to filter Signature matches to remove obvious false positives. Consider also using the --ignore_archive_submatches option in bd_sig_filter to remove all sub-matches (leaving only those which match top-level packages).