Performance Measurement of cbuild setup Command

Performance Measurement and Monitoring of cbuild setup Command

Overview

This document provides a structured approach to measuring and monitoring the performance of the cbuild setup command within the CMSIS-Toolbox ecosystem. Any delay in its execution can negatively impact the responsiveness of the other development environment. The goal is to ensure that the execution time of cbuild setup is tracked, analyzed, and monitored automatically to maintain optimal performance.

Requirement Breakdown

• Performance Measurement:
  • cbuild acts as an orchestration utility that triggers various underlying commands and tools. Any delay could originate from these commands, making it necessary to measure execution times at a granular level.
• Machine-Readable Output:
  • The generated performance data should be in a structured, machine-readable format (e.g., JSON) to enable automated threshold checking.
• Non-Disruptive Implementation:
  • The performance monitoring mechanism should not impact the existing functionality of cbuild setup.

Implementation

Go's Build Tags provide a robust mechanism for including or excluding specific code segments during the build process. By leveraging build tags, developers can selectively enable performance tracking without affecting the standard binary.

• A new build tag named performance has been introduced.
• Performance tracking code is included in the binary only when it is compiled with the performance tag.
• Example build command:

  go build -tags performance -o ./cbuild.exe ./cmd/cbuild

• In normal build mode (without the performance tag), the performance tracking code is omitted, and a stub implementation is used to prevent build errors.

Output Format

The output is generated in JSON format, providing detailed insights into execution performance. The recorded information includes:

• Example project used for performance measurement.
• Operating System (OS) and system architecture details.
• List of triggered commands.
• Execution time for each command (in milliseconds).

Example Output

[
  {
    "example": "vscode-get-started",
    "os": "windows",
    "arch": "amd64",
    "performance": [
      {
        "tool": "csolution.exe",
        "args": "list packs --solution=./get_started.csolution.yml --context-set --toolchain=AC6 -m -q -n",
        "time_ms": 280
      },
      {
        "tool": "csolution.exe",
        "args": "convert --solution=./get_started.csolution.yml --context-set --toolchain=AC6 --quiet",
        "time_ms": 679
      },
      {
        "tool": "cbuild2cmake.exe",
        "args": "C:/Temp/vscode-get-started/get_started.cbuild-idx.yml --context-set",
        "time_ms": 339
      },
      {
        "tool": "cmake.exe",
        "args": "-G Ninja -S C:\\Temp\\vscode-get-started\\tmp -B C:\\Temp\\sourabh\\vscode-get-started\\tmp -Wno-dev",
        "time_ms": 1549
      },
      {
        "tool": "ninja",
        "args": "--version",
        "time_ms": 348
      },
      {
        "tool": "cmake.exe",
        "args": "--build C:\\Temp\\vscode-get-started\\tmp -j 8 --target hello.debug+avh-database -- --quiet",
        "time_ms": 1359
      },
      {
        "tool": "cbuild",
        "args": "setup ./get_started.csolution.yml --context-set=true --packs=true --toolchain=AC6 --update-rte=true",
        "time_ms": 4656
      }
    ]
  }
]

Customizing the Performance Report Output

By default, the performance output file is generated in the current working directory. However, users can customize the location and name of the performance report using the hidden --perf-report option.

Example command:

cbuild setup ./test.csolution.yml -S -p --update-rte --perf-report ./reports/performance_report.json

In this example, the performance report will be generated at ./reports/performance_report.json instead of the default location.

Summary

This document outlines the structured approach to performance measurement for the cbuild setup command, ensuring that delays are detected and monitored efficiently. The use of Go's build tags allows seamless integration without affecting regular builds, while the JSON output format enables automation in performance threshold checks. This implementation will contribute to improved responsiveness and stability within IDEs leveraging CMSIS-Toolbox.