diff --git a/llvm/docs/CommandGuide/index.rst b/llvm/docs/CommandGuide/index.rst index f85f32a1fdd51..86de450d459f1 100644 --- a/llvm/docs/CommandGuide/index.rst +++ b/llvm/docs/CommandGuide/index.rst @@ -85,6 +85,7 @@ Developer Tools llvm-tblgen mlir-tblgen lit + llvm-advisor llvm-exegesis llvm-ifs llvm-locstats diff --git a/llvm/docs/CommandGuide/llvm-advisor.rst b/llvm/docs/CommandGuide/llvm-advisor.rst new file mode 100644 index 0000000000000..5262d342b79ee --- /dev/null +++ b/llvm/docs/CommandGuide/llvm-advisor.rst @@ -0,0 +1,312 @@ +llvm-advisor - LLVM compilation analysis tool +============================================= + +.. program:: llvm-advisor + +SYNOPSIS +-------- + +:program:`llvm-advisor` [*options*] *compiler* [*compiler-args...*] + +:program:`llvm-advisor` **view** [*options*] *compiler* [*compiler-args...*] + +DESCRIPTION +----------- + +The :program:`llvm-advisor` tool is a compilation analysis utility that acts as +a wrapper around compiler commands to collect detailed information about the +compilation process. It captures compilation data, optimization information, +and diagnostic details that can be used to analyze and improve build performance +and code optimization. The tool requires no external dependencies beyond a +standard LLVM/Clang installation and Python 3 for the web viewer. + +:program:`llvm-advisor` intercepts compiler invocations and instruments them +to extract and collect data including: + +* LLVM IR, assembly, and preprocessed source code output +* AST dumps in both text and JSON formats +* Include dependency trees and macro expansion information +* Compiler diagnostics, warnings, and static analysis results +* Debug information and DWARF data analysis +* Compilation timing reports (via ``-ftime-report``) +* Runtime profiling and coverage data (when profiler is enabled) +* Binary size analysis and symbol information +* Optimization remarks from compiler passes + +The tool supports two primary modes of operation: + +**Data Collection Mode**: The default mode where :program:`llvm-advisor` wraps +the compiler command, collects analysis data, and stores it in a +hierarchically organized output directory for later analysis. + +**View Mode**: When invoked with the **view** subcommand, :program:`llvm-advisor` +performs data collection and then automatically launches a web-based interface +with interactive visualization and analysis capabilities. The web viewer provides +a REST API for programmatic access to the collected data. + +All collected data is stored in a timestamped, structured format within the output +directory (default: ``.llvm-advisor``) and can be analyzed using the built-in web +viewer or external tools. + +OPTIONS +------- + +.. option:: --config + + Specify a configuration file to customize :program:`llvm-advisor` behavior. + The configuration file uses JSON format and can override default settings + for output directory, verbosity, and other options. + +.. option:: --output-dir + + Specify the directory where compilation analysis data will be stored. + If the directory doesn't exist, it will be created. The default output + directory is ``.llvm-advisor`` in the current working directory. + +.. option:: --verbose + + Enable verbose output to display detailed information about the analysis + process, including the compiler command being executed and the location + of collected data. + +.. option:: --keep-temps + + Preserve temporary files created during the analysis process. By default, + temporary files are cleaned up automatically. This option is useful for + debugging or when you need to examine intermediate analysis results. + +.. option:: --no-profiler + + Disable the automatic addition of compiler profiling flags during compilation. + By default, :program:`llvm-advisor` adds flags like ``-fprofile-instr-generate`` + and ``-fcoverage-mapping`` to collect runtime profiling data. This option + disables that behavior, reducing compilation overhead but limiting the + coverage and profiling data available for analysis. + +.. option:: --port + + Specify the port number for the web server when using the **view** command. + The default port is 8000. The web viewer will be accessible at + ``http://localhost:``. + +.. option:: --help, -h + + Display usage information and available options. + +COMMANDS +-------- + +:program:`llvm-advisor` supports the following commands: + +Data Collection (Default) +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When no subcommand is specified, :program:`llvm-advisor` operates in data +collection mode: + +.. code-block:: console + + llvm-advisor [options] [compiler-args...] + +This mode wraps the specified compiler command, collects analysis data during +compilation, and stores the results in the output directory. + +View Mode +~~~~~~~~~ + +The **view** subcommand combines data collection with automatic web viewer +launch: + +.. code-block:: console + + llvm-advisor view [options] [compiler-args...] + +In this mode, :program:`llvm-advisor` first performs compilation with data +collection, then launches a web server providing an interactive interface +to analyze the collected data. The web viewer remains active until manually +terminated. + +EXAMPLES +-------- + +Basic Usage +~~~~~~~~~~~ + +Analyze a simple C compilation: + +.. code-block:: console + + llvm-advisor clang -O2 -g main.c -o main + +This command will compile ``main.c`` using clang with ``-O2`` optimization +and debug information, while collecting analysis data in the +``.llvm-advisor`` directory. The output will be organized as: + +.. code-block:: text + + .llvm-advisor/ + └── main/ + └── main_20250825_143022/ # Timestamped compilation session + ├── ir/main.ll # LLVM IR output + ├── assembly/main.s # Assembly output + ├── ast/main.ast # AST dump + ├── diagnostics/ # Compiler warnings/errors + └── ... # Additional analysis data + +Complex C++ Project +~~~~~~~~~~~~~~~~~~~ + +Analyze a C++ compilation with custom output directory: + +.. code-block:: console + + llvm-advisor --output-dir analysis-results clang++ -O3 -std=c++17 app.cpp lib.cpp -o app + +Compile with maximum optimization and store analysis results in the +``analysis-results`` directory. + +Interactive Analysis +~~~~~~~~~~~~~~~~~~~~ + +Compile and immediately launch the web viewer: + +.. code-block:: console + + llvm-advisor view --port 8080 clang -O2 main.c + +This will compile ``main.c``, collect analysis data, and launch a web interface +accessible at ``http://localhost:8080`` for interactive analysis. + +Configuration File Usage +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use a custom configuration file: + +.. code-block:: console + + llvm-advisor --config custom-config.json --verbose clang++ -O1 project.cpp + +Example configuration file (``custom-config.json``): + +.. code-block:: json + + { + "outputDir": "compilation-analysis", + "verbose": true, + "keepTemps": false, + "runProfiler": true, + "timeout": 120 + } + +Integration with Build Systems +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:program:`llvm-advisor` can be integrated into existing build systems by +substituting the compiler command: + +.. code-block:: console + + # Instead of: make CC=clang CXX=clang++ + make CC="llvm-advisor clang" CXX="llvm-advisor clang++" + + # For CMake projects: + cmake -DCMAKE_C_COMPILER="llvm-advisor clang" \ + -DCMAKE_CXX_COMPILER="llvm-advisor clang++" \ + .. + +Accessing Historical Data +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The timestamped directory structure allows you to analyze compilation trends +over time: + +.. code-block:: console + + # View most recent compilation results + llvm-advisor view --output-dir .llvm-advisor + + # Each unit directory contains multiple timestamped runs + ls .llvm-advisor/myproject/ + # Output: myproject_20250825_140512 myproject_20250825_143022 + +The web viewer automatically uses the most recent compilation run for analysis, +but all historical data remains accessible in the timestamped directories. + +CONFIGURATION +------------- + +:program:`llvm-advisor` can be configured using a JSON configuration file +specified with the :option:`--config` option. The configuration file supports +the following options: + +**outputDir** (string) + Default output directory for analysis data. + +**verbose** (boolean) + Enable verbose output by default. + +**keepTemps** (boolean) + Preserve temporary files by default. + +**runProfiler** (boolean) + Enable performance profiling during compilation. + +**timeout** (integer) + Timeout in seconds for compilation analysis (default: 60). + +OUTPUT FORMAT +------------- + +:program:`llvm-advisor` generates analysis data in a structured format within +the output directory. The tool organizes data hierarchically by compilation unit +and timestamp, allowing multiple compilation sessions to be tracked over time. + +The typical output structure includes: + +.. code-block:: text + + .llvm-advisor/ + └── {compilation-unit}/ # One directory per compilation unit + └── {unit-name}_{timestamp}/ # Timestamped compilation runs + ├── ir/ # LLVM IR files (.ll) + ├── assembly/ # Assembly output (.s) + ├── ast/ # AST dumps (.ast) and JSON (.ast.json) + ├── preprocessed/ # Preprocessed source (.i/.ii) + ├── include-tree/ # Include hierarchy information + ├── dependencies/ # Dependency analysis (.deps.txt) + ├── debug/ # Debug information and DWARF data + ├── static-analyzer/ # Static analysis results + ├── diagnostics/ # Compiler diagnostics and warnings + ├── coverage/ # Code coverage data + ├── time-trace/ # Compilation time traces + ├── runtime-trace/ # Runtime tracing information + ├── binary-analysis/ # Binary size and symbol analysis + ├── pgo/ # Profile-guided optimization data + ├── ftime-report/ # Compilation timing reports + ├── version-info/ # Compiler version information + └── sources/ # Source file copies and metadata + +Each compilation run creates a new timestamped directory, preserving the history +of compilation sessions. The most recent run is automatically used by the web +viewer for analysis. + +EXIT STATUS +----------- + +:program:`llvm-advisor` returns the same exit status as the wrapped compiler +command. If the compilation succeeds, it returns 0. If the compilation fails +or :program:`llvm-advisor` encounters an internal error, it returns a non-zero +exit status. + +:program:`llvm-advisor` returns exit code 1 for various error conditions including: + +* Invalid command line arguments or missing compiler command +* Configuration file parsing errors +* Output directory creation failures +* Web viewer launch failures (view mode only) +* Data collection or extraction errors + +SEE ALSO +-------- + +:manpage:`clang(1)`, :manpage:`opt(1)`, :manpage:`llc(1)` diff --git a/llvm/tools/llvm-advisor/CMakeLists.txt b/llvm/tools/llvm-advisor/CMakeLists.txt new file mode 100644 index 0000000000000..d2389bdd1e0fa --- /dev/null +++ b/llvm/tools/llvm-advisor/CMakeLists.txt @@ -0,0 +1,15 @@ +cmake_minimum_required(VERSION 3.18) + +set(LLVM_TOOL_LLVM_ADVISOR_BUILD_DEFAULT ON) +set(LLVM_REQUIRE_EXE_NAMES llvm-advisor) + +add_subdirectory(src) + +# Set the executable name +set_target_properties(llvm-advisor PROPERTIES + OUTPUT_NAME llvm-advisor) + +# Install the binary +install(TARGETS llvm-advisor + RUNTIME DESTINATION bin + COMPONENT llvm-advisor) diff --git a/llvm/tools/llvm-advisor/config/config.json b/llvm/tools/llvm-advisor/config/config.json new file mode 100644 index 0000000000000..9e94a41ff46c4 --- /dev/null +++ b/llvm/tools/llvm-advisor/config/config.json @@ -0,0 +1,7 @@ +{ + "outputDir": ".llvm-advisor", + "verbose": false, + "keepTemps": false, + "runProfiler": true, + "timeout": 60 +} diff --git a/llvm/tools/llvm-advisor/src/CMakeLists.txt b/llvm/tools/llvm-advisor/src/CMakeLists.txt new file mode 100644 index 0000000000000..28e7b8f1425ab --- /dev/null +++ b/llvm/tools/llvm-advisor/src/CMakeLists.txt @@ -0,0 +1,35 @@ +# Gather all .cpp sources in this directory tree +file(GLOB_RECURSE LLVM_ADVISOR_SOURCES CONFIGURE_DEPENDS + ${CMAKE_CURRENT_SOURCE_DIR}/*.cpp +) + +# Define the executable target +add_llvm_tool(llvm-advisor + ${LLVM_ADVISOR_SOURCES} +) + +# Link required LLVM libraries +target_link_libraries(llvm-advisor PRIVATE + LLVMSupport + LLVMCore + LLVMIRReader + LLVMBitWriter + LLVMRemarks + LLVMProfileData +) + +# Set include directories +target_include_directories(llvm-advisor PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR} +) + +# Install the Python webview module alongside the binary +install(DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/../tools/ + DESTINATION share/llvm-advisor/tools + FILES_MATCHING + PATTERN "*.py" + PATTERN "*.html" + PATTERN "*.css" + PATTERN "*.js" + PATTERN "*.md" +) diff --git a/llvm/tools/llvm-advisor/src/Config/AdvisorConfig.cpp b/llvm/tools/llvm-advisor/src/Config/AdvisorConfig.cpp new file mode 100644 index 0000000000000..7eff4d1af3dc4 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Config/AdvisorConfig.cpp @@ -0,0 +1,73 @@ +//===------------------ AdvisorConfig.cpp - LLVM Advisor ------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This file implements the AdvisorConfig class. +// +//===----------------------------------------------------------------------===// + +#include "AdvisorConfig.h" +#include "llvm/Support/Error.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/JSON.h" +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/Support/Path.h" + +namespace llvm { +namespace advisor { + +AdvisorConfig::AdvisorConfig() { OutputDir_ = ".llvm-advisor"; } + +Expected AdvisorConfig::loadFromFile(llvm::StringRef path) { + auto BufferOrError = MemoryBuffer::getFile(path); + if (!BufferOrError) { + return createStringError(BufferOrError.getError(), + "Cannot read config file"); + } + + auto Buffer = std::move(*BufferOrError); + Expected JsonOrError = json::parse(Buffer->getBuffer()); + if (!JsonOrError) { + return JsonOrError.takeError(); + } + + auto &Json = *JsonOrError; + auto *Obj = Json.getAsObject(); + if (!Obj) { + return createStringError(std::make_error_code(std::errc::invalid_argument), + "Config file must contain JSON object"); + } + + if (auto outputDirOpt = Obj->getString("outputDir"); outputDirOpt) { + OutputDir_ = outputDirOpt->str(); + } + + if (auto verboseOpt = Obj->getBoolean("verbose"); verboseOpt) { + Verbose_ = *verboseOpt; + } + + if (auto keepTempsOpt = Obj->getBoolean("keepTemps"); keepTempsOpt) { + KeepTemps_ = *keepTempsOpt; + } + + if (auto runProfileOpt = Obj->getBoolean("runProfiler"); runProfileOpt) { + RunProfiler_ = *runProfileOpt; + } + + if (auto timeoutOpt = Obj->getInteger("timeout"); timeoutOpt) { + TimeoutSeconds_ = static_cast(*timeoutOpt); + } + + return true; +} + +std::string AdvisorConfig::getToolPath(llvm::StringRef tool) const { + return tool.str(); +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Config/AdvisorConfig.h b/llvm/tools/llvm-advisor/src/Config/AdvisorConfig.h new file mode 100644 index 0000000000000..25808a3e52db2 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Config/AdvisorConfig.h @@ -0,0 +1,56 @@ +//===------------------- AdvisorConfig.h - LLVM Advisor -------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the AdvisorConfig code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_ADVISOR_CONFIG_H +#define LLVM_ADVISOR_CONFIG_H + +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/Error.h" +#include + +namespace llvm { +namespace advisor { + +class AdvisorConfig { +public: + AdvisorConfig(); + + Expected loadFromFile(llvm::StringRef path); + + void setOutputDir(const std::string &dir) { OutputDir_ = dir; } + void setVerbose(bool verbose) { Verbose_ = verbose; } + void setKeepTemps(bool keep) { KeepTemps_ = keep; } + void setRunProfiler(bool run) { RunProfiler_ = run; } + void setTimeout(int seconds) { TimeoutSeconds_ = seconds; } + + const std::string &getOutputDir() const { return OutputDir_; } + bool getVerbose() const { return Verbose_; } + bool getKeepTemps() const { return KeepTemps_; } + bool getRunProfiler() const { return RunProfiler_; } + int getTimeout() const { return TimeoutSeconds_; } + + std::string getToolPath(llvm::StringRef tool) const; + +private: + std::string OutputDir_; + bool Verbose_ = false; + bool KeepTemps_ = false; + bool RunProfiler_ = true; + int TimeoutSeconds_ = 60; +}; + +} // namespace advisor +} // namespace llvm + +#endif diff --git a/llvm/tools/llvm-advisor/src/Core/BuildContext.h b/llvm/tools/llvm-advisor/src/Core/BuildContext.h new file mode 100644 index 0000000000000..18156f3634bea --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/BuildContext.h @@ -0,0 +1,68 @@ +//===------------------- BuildContext.h - LLVM Advisor --------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the BuildContext code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_ADVISOR_CORE_BUILDCONTEXT_H +#define LLVM_ADVISOR_CORE_BUILDCONTEXT_H + +#include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include +#include + +namespace llvm { +namespace advisor { + +enum class BuildPhase { + Unknown, + Preprocessing, + Compilation, + Assembly, + Linking, + Archiving, + CMakeConfigure, + CMakeBuild, + MakefileBuild +}; + +enum class BuildTool { + Unknown, + Clang, + GCC, + LLVM_Tools, + CMake, + Make, + Ninja, + Linker, + Archiver +}; + +struct BuildContext { + BuildPhase phase; + BuildTool tool; + std::string workingDirectory; + std::string outputDirectory; + llvm::SmallVector inputFiles; + llvm::SmallVector outputFiles; + llvm::SmallVector expectedGeneratedFiles; + std::unordered_map metadata; + bool hasOffloading = false; + bool hasDebugInfo = false; + bool hasOptimization = false; +}; + +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_CORE_BUILDCONTEXT_H diff --git a/llvm/tools/llvm-advisor/src/Core/BuildExecutor.cpp b/llvm/tools/llvm-advisor/src/Core/BuildExecutor.cpp new file mode 100644 index 0000000000000..1fa19bd543203 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/BuildExecutor.cpp @@ -0,0 +1,153 @@ +//===---------------- BuildExecutor.cpp - LLVM Advisor --------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the BuildExecutor code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "BuildExecutor.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/Path.h" +#include "llvm/Support/Program.h" +#include "llvm/Support/raw_ostream.h" + +namespace llvm { +namespace advisor { + +BuildExecutor::BuildExecutor(const AdvisorConfig &config) : config_(config) {} + +llvm::Expected +BuildExecutor::execute(llvm::StringRef compiler, + const llvm::SmallVectorImpl &args, + BuildContext &buildContext, llvm::StringRef tempDir) { + auto instrumentedArgs = instrumentCompilerArgs(args, buildContext, tempDir); + + auto compilerPath = llvm::sys::findProgramByName(compiler); + if (!compilerPath) { + return llvm::createStringError( + std::make_error_code(std::errc::no_such_file_or_directory), + "Compiler not found: " + compiler.str()); + } + + llvm::SmallVector execArgs; + execArgs.push_back(compiler); + for (const auto &arg : instrumentedArgs) { + execArgs.push_back(arg); + } + + if (config_.getVerbose()) { + llvm::outs() << "Executing: " << compiler; + for (const auto &arg : instrumentedArgs) { + llvm::outs() << " " << arg; + } + llvm::outs() << "\n"; + } + + return llvm::sys::ExecuteAndWait(*compilerPath, execArgs); +} + +llvm::SmallVector BuildExecutor::instrumentCompilerArgs( + const llvm::SmallVectorImpl &args, BuildContext &buildContext, + llvm::StringRef tempDir) { + + llvm::SmallVector result(args.begin(), args.end()); + llvm::DenseSet existingFlags; + + // Scan existing flags to avoid duplication + for (const auto &arg : args) { + if (llvm::StringRef(arg).starts_with("-g")) + existingFlags.insert("debug"); + if (llvm::StringRef(arg).contains("-fsave-optimization-record")) + existingFlags.insert("remarks"); + if (llvm::StringRef(arg).contains("-fprofile-instr-generate")) + existingFlags.insert("profile"); + } + + // Add debug info if not present + if (!existingFlags.contains("debug")) { + result.push_back("-g"); + } + + // Add optimization remarks with proper redirection + if (!existingFlags.contains("remarks")) { + result.push_back("-fsave-optimization-record"); + result.push_back("-foptimization-record-file=" + tempDir.str() + + "/remarks.opt.yaml"); + buildContext.expectedGeneratedFiles.push_back(tempDir.str() + + "/remarks.opt.yaml"); + } else { + // If user already specified remarks, find and redirect the file + bool foundFileFlag = false; + for (auto &arg : result) { + if (llvm::StringRef(arg).contains("-foptimization-record-file=")) { + // Extract filename and redirect to temp + llvm::StringRef existingPath = llvm::StringRef(arg).substr(26); + llvm::StringRef filename = llvm::sys::path::filename(existingPath); + arg = "-foptimization-record-file=" + tempDir.str() + "/" + + filename.str(); + buildContext.expectedGeneratedFiles.push_back(tempDir.str() + "/" + + filename.str()); + foundFileFlag = true; + break; + } + } + // If no explicit file specified, add our own + if (!foundFileFlag) { + result.push_back("-foptimization-record-file=" + tempDir.str() + + "/remarks.opt.yaml"); + buildContext.expectedGeneratedFiles.push_back(tempDir.str() + + "/remarks.opt.yaml"); + } + } + + // Add profiling if enabled and not present, redirect to temp directory + if (config_.getRunProfiler() && !existingFlags.contains("profile")) { + result.push_back("-fprofile-instr-generate=" + tempDir.str() + + "/profile.profraw"); + result.push_back("-fcoverage-mapping"); + buildContext.expectedGeneratedFiles.push_back(tempDir.str() + + "/profile.profraw"); + } + + // Add remark extraction flags if none present + bool hasRpass = false; + for (const auto &arg : result) { + if (llvm::StringRef(arg).starts_with("-Rpass=")) { + hasRpass = true; + break; + } + } + if (!hasRpass) { + // For now we add offloading and general analysis passes + result.push_back("-Rpass=kernel-info"); + result.push_back("-Rpass=analysis"); + } + + // Add diagnostic output format for better parsing + bool hasDiagFormat = false; + for (const auto &arg : result) { + if (llvm::StringRef(arg).contains("-fdiagnostics-format")) { + hasDiagFormat = true; + break; + } + } + if (!hasDiagFormat) { + result.push_back("-fdiagnostics-parseable-fixits"); + result.push_back("-fdiagnostics-absolute-paths"); + } + + return result; +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Core/BuildExecutor.h b/llvm/tools/llvm-advisor/src/Core/BuildExecutor.h new file mode 100644 index 0000000000000..965ad86158944 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/BuildExecutor.h @@ -0,0 +1,48 @@ +//===------------------- BuildExecutor.h - LLVM Advisor -------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the BuildExecutor code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_ADVISOR_CORE_BUILDEXECUTOR_H +#define LLVM_ADVISOR_CORE_BUILDEXECUTOR_H + +#include "../Config/AdvisorConfig.h" +#include "BuildContext.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/Error.h" +#include + +namespace llvm { +namespace advisor { + +class BuildExecutor { +public: + BuildExecutor(const AdvisorConfig &config); + + llvm::Expected execute(llvm::StringRef compiler, + const llvm::SmallVectorImpl &args, + BuildContext &buildContext, + llvm::StringRef tempDir); + +private: + llvm::SmallVector + instrumentCompilerArgs(const llvm::SmallVectorImpl &args, + BuildContext &buildContext, llvm::StringRef tempDir); + + const AdvisorConfig &config_; +}; + +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_CORE_BUILDEXECUTOR_H diff --git a/llvm/tools/llvm-advisor/src/Core/CommandAnalyzer.cpp b/llvm/tools/llvm-advisor/src/Core/CommandAnalyzer.cpp new file mode 100644 index 0000000000000..aeda9d3dfe2b6 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/CommandAnalyzer.cpp @@ -0,0 +1,182 @@ +//===----------------- CommandAnalyzer.cpp - LLVM Advisor -----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the CommandAnalyzer code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "CommandAnalyzer.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/ADT/StringSwitch.h" +#include "llvm/Support/Error.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/Path.h" + +namespace llvm { +namespace advisor { + +CommandAnalyzer::CommandAnalyzer(llvm::StringRef command, + const llvm::SmallVectorImpl &args) + : command_(command.str()), args_(args.data(), args.data() + args.size()) {} + +BuildContext CommandAnalyzer::analyze() const { + BuildContext context; + llvm::SmallString<256> cwd; + llvm::sys::fs::current_path(cwd); + context.workingDirectory = cwd.str().str(); + + context.tool = detectBuildTool(); + context.phase = detectBuildPhase(context.tool); + context.inputFiles = extractInputFiles(); + context.outputFiles = extractOutputFiles(); + detectBuildFeatures(context); + + return context; +} + +BuildTool CommandAnalyzer::detectBuildTool() const { + return llvm::StringSwitch(llvm::sys::path::filename(command_)) + .StartsWith("clang", BuildTool::Clang) + .StartsWith("gcc", BuildTool::GCC) + .StartsWith("g++", BuildTool::GCC) + .Case("cmake", BuildTool::CMake) + .Case("make", BuildTool::Make) + .Case("ninja", BuildTool::Ninja) + .EndsWith("-ld", BuildTool::Linker) + .Case("ld", BuildTool::Linker) + .Case("ar", BuildTool::Archiver) + .Case("llvm-ar", BuildTool::Archiver) + .StartsWith("llvm-", BuildTool::LLVM_Tools) + .Default(BuildTool::Unknown); +} + +BuildPhase CommandAnalyzer::detectBuildPhase(BuildTool tool) const { + if (tool == BuildTool::CMake) { + for (const auto &arg : args_) { + if (arg == "--build") + return BuildPhase::CMakeBuild; + } + return BuildPhase::CMakeConfigure; + } + + if (tool == BuildTool::Make || tool == BuildTool::Ninja) { + return BuildPhase::MakefileBuild; + } + + if (tool == BuildTool::Linker) { + return BuildPhase::Linking; + } + + if (tool == BuildTool::Archiver) { + return BuildPhase::Archiving; + } + + if (tool == BuildTool::Clang || tool == BuildTool::GCC) { + for (const auto &arg : args_) { + if (arg == "-E") + return BuildPhase::Preprocessing; + if (arg == "-S") + return BuildPhase::Assembly; + if (arg == "-c") + return BuildPhase::Compilation; + } + + bool hasObjectFile = false; + for (const auto &Arg : args_) { + llvm::StringRef argRef(Arg); + if (argRef.ends_with(".o") || argRef.ends_with(".O") || + argRef.ends_with(".obj") || argRef.ends_with(".OBJ")) { + hasObjectFile = true; + break; + } + } + if (hasObjectFile) { + return BuildPhase::Linking; + } + + bool hasSourceFile = false; + for (const auto &Arg : args_) { + llvm::StringRef argRef(Arg); + if (argRef.ends_with(".c") || argRef.ends_with(".C") || + argRef.ends_with(".cpp") || argRef.ends_with(".CPP") || + argRef.ends_with(".cc") || argRef.ends_with(".CC") || + argRef.ends_with(".cxx") || argRef.ends_with(".CXX")) { + hasSourceFile = true; + break; + } + } + if (hasSourceFile) { + return BuildPhase::Compilation; // Default for source files + } + } + + return BuildPhase::Unknown; +} + +void CommandAnalyzer::detectBuildFeatures(BuildContext &context) const { + for (const auto &arg : args_) { + if (arg == "-g" || llvm::StringRef(arg).starts_with("-g")) { + context.hasDebugInfo = true; + } + + if (llvm::StringRef(arg).starts_with("-O") && arg.length() > 2) { + context.hasOptimization = true; + } + + llvm::StringRef argRef(arg); + if (argRef.contains("openmp") || argRef.contains("openacc") || + argRef.contains("cuda") || argRef.contains("offload")) { + context.hasOffloading = true; + } + + if (llvm::StringRef(arg).starts_with("-march=")) { + context.metadata["target_arch"] = arg.substr(7); + } + if (llvm::StringRef(arg).starts_with("-mtune=")) { + context.metadata["tune"] = arg.substr(7); + } + if (llvm::StringRef(arg).starts_with("--offload-arch=")) { + context.metadata["offload_arch"] = arg.substr(15); + } + } +} + +llvm::SmallVector CommandAnalyzer::extractInputFiles() const { + llvm::SmallVector inputs; + for (size_t i = 0; i < args_.size(); ++i) { + const auto &arg = args_[i]; + if (llvm::StringRef(arg).starts_with("-")) { + if (arg == "-o" || arg == "-I" || arg == "-L" || arg == "-D") { + i++; + } + continue; + } + if (llvm::sys::fs::exists(arg)) { + inputs.push_back(arg); + } + } + return inputs; +} + +llvm::SmallVector CommandAnalyzer::extractOutputFiles() const { + llvm::SmallVector outputs; + for (size_t i = 0; i < args_.size(); ++i) { + const auto &arg = args_[i]; + if (arg == "-o" && i + 1 < args_.size()) { + outputs.push_back(args_[i + 1]); + i++; + } + } + return outputs; +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Core/CommandAnalyzer.h b/llvm/tools/llvm-advisor/src/Core/CommandAnalyzer.h new file mode 100644 index 0000000000000..cb31279121d66 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/CommandAnalyzer.h @@ -0,0 +1,46 @@ +//===------------------- CommandAnalyzer.h - LLVM Advisor -----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the CommandAnalyzer code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_CORE_COMMANDANALYZER_H +#define LLVM_ADVISOR_CORE_COMMANDANALYZER_H + +#include "BuildContext.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include + +namespace llvm { +namespace advisor { + +class CommandAnalyzer { +public: + CommandAnalyzer(llvm::StringRef command, + const llvm::SmallVectorImpl &args); + + BuildContext analyze() const; + +private: + BuildTool detectBuildTool() const; + BuildPhase detectBuildPhase(BuildTool tool) const; + void detectBuildFeatures(BuildContext &context) const; + llvm::SmallVector extractInputFiles() const; + llvm::SmallVector extractOutputFiles() const; + + std::string command_; + llvm::SmallVector args_; +}; + +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_CORE_COMMANDANALYZER_H diff --git a/llvm/tools/llvm-advisor/src/Core/CompilationManager.cpp b/llvm/tools/llvm-advisor/src/Core/CompilationManager.cpp new file mode 100644 index 0000000000000..ab2cf58365acc --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/CompilationManager.cpp @@ -0,0 +1,338 @@ +//===---------------- CompilationManager.cpp - LLVM Advisor ---------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the CompilationManager code generator driver. It provides a +// convenient command-line interface for generating an assembly file or a +// relocatable file, given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "CompilationManager.h" +#include "../Detection/UnitDetector.h" +#include "../Utils/FileManager.h" +#include "../Utils/UnitMetadata.h" +#include "CommandAnalyzer.h" +#include "DataExtractor.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/Path.h" +#include +#include +#include +#include + +namespace llvm { +namespace advisor { + +CompilationManager::CompilationManager(const AdvisorConfig &config) + : config_(config), buildExecutor_(config) { + + // Get current working directory first + llvm::SmallString<256> currentDir; + llvm::sys::fs::current_path(currentDir); + initialWorkingDir_ = currentDir.str().str(); + + // Create temp directory with proper error handling + llvm::SmallString<128> tempDirPath; + if (auto EC = + llvm::sys::fs::createUniqueDirectory("llvm-advisor", tempDirPath)) { + // Use timestamp for temp folder naming + auto now = std::chrono::system_clock::now(); + auto timestamp = + std::chrono::duration_cast(now.time_since_epoch()) + .count(); + tempDir_ = "/tmp/llvm-advisor-" + std::to_string(timestamp); + llvm::sys::fs::create_directories(tempDir_); + } else { + tempDir_ = tempDirPath.str().str(); + } + + // Ensure the directory actually exists + if (!llvm::sys::fs::exists(tempDir_)) { + llvm::sys::fs::create_directories(tempDir_); + } + + if (config_.getVerbose()) { + llvm::outs() << "Using temporary directory: " << tempDir_ << "\n"; + } + + // Initialize unit metadata tracking + llvm::SmallString<256> outputDirPath; + if (llvm::sys::path::is_absolute(config_.getOutputDir())) { + outputDirPath = config_.getOutputDir(); + } else { + outputDirPath = initialWorkingDir_; + llvm::sys::path::append(outputDirPath, config_.getOutputDir()); + } + + unitMetadata_ = std::make_unique(outputDirPath.str()); + if (auto Err = unitMetadata_->loadMetadata()) { + if (config_.getVerbose()) { + llvm::errs() << "Failed to load metadata: " + << llvm::toString(std::move(Err)) << "\n"; + } + } +} + +CompilationManager::~CompilationManager() { + if (!config_.getKeepTemps() && llvm::sys::fs::exists(tempDir_)) { + llvm::sys::fs::remove_directories(tempDir_); + } +} + +llvm::Expected CompilationManager::executeWithDataCollection( + const std::string &compiler, + const llvm::SmallVectorImpl &args) { + + // Analyze the build command + BuildContext buildContext = CommandAnalyzer(compiler, args).analyze(); + + if (config_.getVerbose()) { + llvm::outs() << "Build phase: " << static_cast(buildContext.phase) + << "\n"; + } + + // Skip data collection for linking/archiving phases + if (buildContext.phase == BuildPhase::Linking || + buildContext.phase == BuildPhase::Archiving) { + return buildExecutor_.execute(compiler, args, buildContext, tempDir_); + } + + // Detect compilation units + UnitDetector detector(config_); + auto detectedUnits = detector.detectUnits(compiler, args); + if (!detectedUnits) { + return detectedUnits.takeError(); + } + + llvm::SmallVector, 4> units; + for (auto &unitInfo : *detectedUnits) { + units.push_back(std::make_unique(unitInfo, tempDir_)); + + // Register unit in metadata tracker + unitMetadata_->registerUnit(unitInfo.name); + } + + // Scan existing files before compilation + auto existingFiles = scanDirectory(initialWorkingDir_); + + // Execute compilation with instrumentation + auto execResult = + buildExecutor_.execute(compiler, args, buildContext, tempDir_); + if (!execResult) { + return execResult; + } + int exitCode = *execResult; + + // Collect generated files (even if compilation failed for analysis) + collectGeneratedFiles(existingFiles, units); + + // Extract additional data + DataExtractor extractor(config_); + for (auto &unit : units) { + if (auto Err = extractor.extractAllData(*unit, tempDir_)) { + if (config_.getVerbose()) { + llvm::errs() << "Data extraction failed: " + << llvm::toString(std::move(Err)) << "\n"; + } + // Mark unit as failed if data extraction fails + unitMetadata_->updateUnitStatus(unit->getName(), "failed"); + } else { + // Update unit metadata with file counts and artifact types + const auto &generatedFiles = unit->getAllGeneratedFiles(); + size_t totalFiles = 0; + for (const auto &category : generatedFiles) { + totalFiles += category.second.size(); + unitMetadata_->addArtifactType(unit->getName(), category.first); + } + unitMetadata_->updateUnitFileCount(unit->getName(), totalFiles); + } + } + + // Organize output + if (auto Err = organizeOutput(units)) { + if (config_.getVerbose()) { + llvm::errs() << "Output organization failed: " + << llvm::toString(std::move(Err)) << "\n"; + } + // Mark units as failed if output organization fails + for (auto &unit : units) { + unitMetadata_->updateUnitStatus(unit->getName(), "failed"); + } + } else { + // Mark units as completed on successful organization + for (auto &unit : units) { + unitMetadata_->updateUnitStatus(unit->getName(), "completed"); + } + } + + // Save metadata to disk + if (auto Err = unitMetadata_->saveMetadata()) { + if (config_.getVerbose()) { + llvm::errs() << "Failed to save metadata: " + << llvm::toString(std::move(Err)) << "\n"; + } + } + + // Clean up leaked files from source directory + cleanupLeakedFiles(); + + return exitCode; +} + +std::unordered_set +CompilationManager::scanDirectory(llvm::StringRef dir) const { + std::unordered_set files; + std::error_code EC; + for (llvm::sys::fs::directory_iterator DI(dir, EC), DE; DI != DE && !EC; + DI.increment(EC)) { + if (DI->type() != llvm::sys::fs::file_type::directory_file) { + files.insert(DI->path()); + } + } + return files; +} + +void CompilationManager::collectGeneratedFiles( + const std::unordered_set &existingFiles, + llvm::SmallVectorImpl> &units) { + FileClassifier classifier; + + // Collect files from temp directory + std::error_code EC; + for (llvm::sys::fs::recursive_directory_iterator DI(tempDir_, EC), DE; + DI != DE && !EC; DI.increment(EC)) { + if (DI->type() != llvm::sys::fs::file_type::directory_file) { + std::string filePath = DI->path(); + if (classifier.shouldCollect(filePath)) { + auto classification = classifier.classifyFile(filePath); + + // Add to appropriate unit + if (!units.empty()) { + units[0]->addGeneratedFile(classification.category, filePath); + } + } + } + } + + // Also check for files that leaked into source directory + auto currentFiles = scanDirectory(initialWorkingDir_); + for (const auto &file : currentFiles) { + if (existingFiles.find(file) == existingFiles.end()) { + if (classifier.shouldCollect(file)) { + auto classification = classifier.classifyFile(file); + + // Move leaked file to temp directory + std::string destPath = + tempDir_ + "/" + llvm::sys::path::filename(file).str(); + if (!FileManager::moveFile(file, destPath)) { + if (!units.empty()) { + units[0]->addGeneratedFile(classification.category, destPath); + } + } + } + } + } +} + +llvm::Error CompilationManager::organizeOutput( + const llvm::SmallVectorImpl> &units) { + // Resolve output directory as absolute path from initial working directory + llvm::SmallString<256> outputDirPath; + if (llvm::sys::path::is_absolute(config_.getOutputDir())) { + outputDirPath = config_.getOutputDir(); + } else { + outputDirPath = initialWorkingDir_; + llvm::sys::path::append(outputDirPath, config_.getOutputDir()); + } + + std::string outputDir = outputDirPath.str().str(); + + if (config_.getVerbose()) { + llvm::outs() << "Output directory: " << outputDir << "\n"; + } + + // Generate timestamp for this compilation run + auto now = std::chrono::system_clock::now(); + auto time_t = std::chrono::system_clock::to_time_t(now); + auto tm = *std::localtime(&time_t); + + char timestampStr[20]; + std::strftime(timestampStr, sizeof(timestampStr), "%Y%m%d_%H%M%S", &tm); + + // Move collected files to organized structure + for (const auto &unit : units) { + // Create base unit directory if it doesn't exist + std::string baseUnitDir = outputDir + "/" + unit->getName(); + llvm::sys::fs::create_directories(baseUnitDir); + + // Create timestamped run directory + std::string runDirName = unit->getName() + "_" + std::string(timestampStr); + std::string unitDir = baseUnitDir + "/" + runDirName; + + if (config_.getVerbose()) { + llvm::outs() << "Creating run directory: " << unitDir << "\n"; + } + + // Create timestamped run directory + if (auto EC = llvm::sys::fs::create_directories(unitDir)) { + if (config_.getVerbose()) { + llvm::errs() << "Warning: Could not create run directory: " << unitDir + << "\n"; + } + continue; // Skip if we can't create the directory + } + + const auto &generatedFiles = unit->getAllGeneratedFiles(); + for (const auto &category : generatedFiles) { + std::string categoryDir = unitDir + "/" + category.first; + llvm::sys::fs::create_directories(categoryDir); + + for (const auto &file : category.second) { + std::string destFile = + categoryDir + "/" + llvm::sys::path::filename(file).str(); + if (auto Err = FileManager::copyFile(file, destFile)) { + if (config_.getVerbose()) { + llvm::errs() << "Failed to copy " << file << " to " << destFile + << "\n"; + } + } + } + } + } + + return llvm::Error::success(); +} + +void CompilationManager::cleanupLeakedFiles() { + // Clean up any remaining leaked files in source directory + auto currentFiles = scanDirectory(initialWorkingDir_); + for (const auto &file : currentFiles) { + llvm::StringRef filename = llvm::sys::path::filename(file); + + // Remove optimization remarks files that leaked + if (filename.ends_with(".opt.yaml") || filename.ends_with(".opt.yml")) { + llvm::sys::fs::remove(file); + if (config_.getVerbose()) { + llvm::outs() << "Cleaned up leaked file: " << file << "\n"; + } + } + + // Remove profile files that leaked + if (filename.ends_with(".profraw") || filename.ends_with(".profdata")) { + llvm::sys::fs::remove(file); + if (config_.getVerbose()) { + llvm::outs() << "Cleaned up leaked file: " << file << "\n"; + } + } + } +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Core/CompilationManager.h b/llvm/tools/llvm-advisor/src/Core/CompilationManager.h new file mode 100644 index 0000000000000..6c42095824faf --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/CompilationManager.h @@ -0,0 +1,64 @@ +//===---------------- CompilationManager.h - LLVM Advisor -----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the CompilationManager code generator driver. It provides a +// convenient command-line interface for generating an assembly file or a +// relocatable file, given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_CORE_COMPILATIONMANAGER_H +#define LLVM_ADVISOR_CORE_COMPILATIONMANAGER_H + +#include "../Config/AdvisorConfig.h" +#include "../Utils/FileClassifier.h" +#include "../Utils/UnitMetadata.h" +#include "BuildExecutor.h" +#include "CompilationUnit.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/Error.h" +#include +#include +#include + +namespace llvm { +namespace advisor { + +class CompilationManager { +public: + explicit CompilationManager(const AdvisorConfig &config); + ~CompilationManager(); + + llvm::Expected + executeWithDataCollection(const std::string &compiler, + const llvm::SmallVectorImpl &args); + +private: + std::unordered_set scanDirectory(llvm::StringRef dir) const; + + void collectGeneratedFiles( + const std::unordered_set &existingFiles, + llvm::SmallVectorImpl> &units); + + llvm::Error organizeOutput( + const llvm::SmallVectorImpl> &units); + + void cleanupLeakedFiles(); + + const AdvisorConfig &config_; + BuildExecutor buildExecutor_; + std::string tempDir_; + std::string initialWorkingDir_; + std::unique_ptr unitMetadata_; +}; + +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_CORE_COMPILATIONMANAGER_H diff --git a/llvm/tools/llvm-advisor/src/Core/CompilationUnit.cpp b/llvm/tools/llvm-advisor/src/Core/CompilationUnit.cpp new file mode 100644 index 0000000000000..414f30a3bc16e --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/CompilationUnit.cpp @@ -0,0 +1,85 @@ +//===---------------- CompilationUnit.cpp - LLVM Advisor ------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the CompilationUnit code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "CompilationUnit.h" +#include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/Error.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/Path.h" +#include + +namespace llvm { +namespace advisor { + +CompilationUnit::CompilationUnit(const CompilationUnitInfo &info, + const std::string &workDir) + : info_(info), workDir_(workDir) { + // Create unit-specific data directory + llvm::SmallString<128> dataDir; + llvm::sys::path::append(dataDir, workDir, "units", info.name); + llvm::sys::fs::create_directories(dataDir); +} + +std::string CompilationUnit::getPrimarySource() const { + if (info_.sources.empty()) { + return ""; + } + return info_.sources[0].path; +} + +std::string CompilationUnit::getDataDir() const { + llvm::SmallString<128> dataDir; + llvm::sys::path::append(dataDir, workDir_, "units", info_.name); + return dataDir.str().str(); +} + +std::string CompilationUnit::getExecutablePath() const { + return info_.outputExecutable; +} + +void CompilationUnit::addGeneratedFile(llvm::StringRef type, + llvm::StringRef path) { + generatedFiles_[type.str()].push_back(path.str()); +} + +bool CompilationUnit::hasGeneratedFiles(llvm::StringRef type) const { + if (type.empty()) { + return !generatedFiles_.empty(); + } + auto it = generatedFiles_.find(type.str()); + return it != generatedFiles_.end() && !it->second.empty(); +} + +llvm::SmallVector +CompilationUnit::getGeneratedFiles(llvm::StringRef type) const { + if (type.empty()) { + llvm::SmallVector allFiles; + for (const auto &pair : generatedFiles_) { + allFiles.append(pair.second.begin(), pair.second.end()); + } + return allFiles; + } + auto it = generatedFiles_.find(type.str()); + return it != generatedFiles_.end() ? it->second + : llvm::SmallVector(); +} + +const std::unordered_map> & +CompilationUnit::getAllGeneratedFiles() const { + return generatedFiles_; +} + +} // namespace advisor +} // namespace llvm \ No newline at end of file diff --git a/llvm/tools/llvm-advisor/src/Core/CompilationUnit.h b/llvm/tools/llvm-advisor/src/Core/CompilationUnit.h new file mode 100644 index 0000000000000..08461e68e3307 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/CompilationUnit.h @@ -0,0 +1,74 @@ +//===------------------- CompilationUnit.h - LLVM Advisor -----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the CompilationUnit code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_CORE_COMPILATIONUNIT_H +#define LLVM_ADVISOR_CORE_COMPILATIONUNIT_H + +#include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/Error.h" +#include +#include + +namespace llvm { +namespace advisor { + +struct SourceFile { + std::string path; + std::string language; + bool isHeader = false; + llvm::SmallVector dependencies; +}; + +struct CompilationUnitInfo { + std::string name; + llvm::SmallVector sources; + llvm::SmallVector compileFlags; + std::string targetArch; + bool hasOffloading = false; + std::string outputObject; + std::string outputExecutable; +}; + +class CompilationUnit { +public: + CompilationUnit(const CompilationUnitInfo &info, const std::string &workDir); + + const std::string &getName() const { return info_.name; } + const CompilationUnitInfo &getInfo() const { return info_; } + const std::string &getWorkDir() const { return workDir_; } + std::string getPrimarySource() const; + + std::string getDataDir() const; + std::string getExecutablePath() const; + + void addGeneratedFile(llvm::StringRef type, llvm::StringRef path); + + bool hasGeneratedFiles(llvm::StringRef type) const; + llvm::SmallVector + getGeneratedFiles(llvm::StringRef type = "") const; + const std::unordered_map> & + getAllGeneratedFiles() const; + +private: + CompilationUnitInfo info_; + std::string workDir_; + std::unordered_map> + generatedFiles_; +}; + +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_CORE_COMPILATIONUNIT_H \ No newline at end of file diff --git a/llvm/tools/llvm-advisor/src/Core/DataExtractor.cpp b/llvm/tools/llvm-advisor/src/Core/DataExtractor.cpp new file mode 100644 index 0000000000000..260fbb56281ab --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/DataExtractor.cpp @@ -0,0 +1,1485 @@ +//===------------------ DataExtractor.cpp - LLVM Advisor ------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the DataExtractor code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "DataExtractor.h" +#include "../Utils/FileManager.h" +#include "../Utils/ProcessRunner.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/Support/Path.h" +#include "llvm/Support/Process.h" +#include "llvm/Support/raw_ostream.h" +#include + +namespace llvm { +namespace advisor { + +DataExtractor::DataExtractor(const AdvisorConfig &config) : config_(config) {} + +Error DataExtractor::extractAllData(CompilationUnit &unit, + llvm::StringRef tempDir) { + if (config_.getVerbose()) { + outs() << "Extracting data for unit: " << unit.getName() << "\n"; + } + + // Create extraction subdirectories + sys::fs::create_directories(tempDir + "/ir"); + sys::fs::create_directories(tempDir + "/assembly"); + sys::fs::create_directories(tempDir + "/ast"); + sys::fs::create_directories(tempDir + "/preprocessed"); + sys::fs::create_directories(tempDir + "/include-tree"); + sys::fs::create_directories(tempDir + "/dependencies"); + sys::fs::create_directories(tempDir + "/debug"); + sys::fs::create_directories(tempDir + "/static-analyzer"); + sys::fs::create_directories(tempDir + "/diagnostics"); + sys::fs::create_directories(tempDir + "/coverage"); + sys::fs::create_directories(tempDir + "/time-trace"); + sys::fs::create_directories(tempDir + "/runtime-trace"); + sys::fs::create_directories(tempDir + "/binary-analysis"); + sys::fs::create_directories(tempDir + "/pgo"); + sys::fs::create_directories(tempDir + "/ftime-report"); + sys::fs::create_directories(tempDir + "/version-info"); + sys::fs::create_directories(tempDir + "/sources"); + + if (auto Err = extractIR(unit, tempDir)) + return Err; + if (auto Err = extractAssembly(unit, tempDir)) + return Err; + if (auto Err = extractAST(unit, tempDir)) + return Err; + if (auto Err = extractPreprocessed(unit, tempDir)) + return Err; + if (auto Err = extractIncludeTree(unit, tempDir)) + return Err; + if (auto Err = extractDependencies(unit, tempDir)) + return Err; + if (auto Err = extractDebugInfo(unit, tempDir)) + return Err; + if (auto Err = extractStaticAnalysis(unit, tempDir)) + return Err; + if (auto Err = extractMacroExpansion(unit, tempDir)) + return Err; + if (auto Err = extractCompilationPhases(unit, tempDir)) + return Err; + if (auto Err = extractFTimeReport(unit, tempDir)) + return Err; + if (auto Err = extractVersionInfo(unit, tempDir)) + return Err; + if (auto Err = extractSources(unit, tempDir)) + return Err; + + // Run additional extractors + for (size_t i = 0; i < numExtractors_; ++i) { + const auto &extractor = extractors_[i]; + if (auto Err = (this->*extractor.method)(unit, tempDir)) { + if (config_.getVerbose()) { + errs() << extractor.name + << " extraction failed: " << toString(std::move(Err)) << "\n"; + } + } + } + + return Error::success(); +} + +llvm::SmallVector +DataExtractor::getBaseCompilerArgs(const CompilationUnitInfo &unitInfo) const { + llvm::SmallVector baseArgs; + + // Preserve relevant compile flags and handle paired flags that forward + // arguments to specific toolchains (e.g. OpenMP target flags). + for (size_t i = 0; i < unitInfo.compileFlags.size(); ++i) { + const std::string &flag = unitInfo.compileFlags[i]; + + // Handle paired forwarding flags that must precede their next argument. + // Example: -Xopenmp-target -march=sm_70 + if (StringRef(flag) == "-Xopenmp-target" || + StringRef(flag).starts_with("-Xopenmp-target=")) { + baseArgs.push_back(flag); + // If the flag is the two-argument form, also copy the next arg if + // present. + if (StringRef(flag) == "-Xopenmp-target" && + i + 1 < unitInfo.compileFlags.size()) { + baseArgs.push_back(unitInfo.compileFlags[i + 1]); + ++i; // consume the next argument + } + continue; + } + + // Commonly needed flags for reproducing preprocessing/IR/ASM + if (StringRef(flag).starts_with("-I") || + StringRef(flag).starts_with("-D") || + StringRef(flag).starts_with("-U") || + StringRef(flag).starts_with("-std=") || + StringRef(flag).starts_with("-m") || + StringRef(flag).starts_with("-f") || + StringRef(flag).starts_with("-W") || + StringRef(flag).starts_with("-O")) { + // Skip instrumentation/file-emission flags added by the executor + if (StringRef(flag).starts_with("-fsave-optimization-record") || + StringRef(flag).starts_with("-fprofile-instr-generate") || + StringRef(flag).starts_with("-fcoverage-mapping") || + StringRef(flag).starts_with("-foptimization-record-file")) { + continue; + } + baseArgs.push_back(flag); + continue; + } + + // Preserve explicit target specification when present + if (StringRef(flag).starts_with("--target=") || + StringRef(flag) == "-target") { + baseArgs.push_back(flag); + if (StringRef(flag) == "-target" && + i + 1 < unitInfo.compileFlags.size()) { + baseArgs.push_back(unitInfo.compileFlags[i + 1]); + ++i; + } + continue; + } + } + + return baseArgs; +} + +Error DataExtractor::extractIR(CompilationUnit &unit, llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/ir/" + sys::path::stem(source.path).str() + ".ll").str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-emit-llvm"); + baseArgs.push_back("-S"); + baseArgs.push_back("-o"); + baseArgs.push_back(outputFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + if (config_.getVerbose()) { + errs() << "Failed to extract IR for " << source.path << "\n"; + } + continue; + } + + if (sys::fs::exists(outputFile)) { + unit.addGeneratedFile("ir", outputFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractAssembly(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/assembly/" + sys::path::stem(source.path).str() + ".s") + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-S"); + baseArgs.push_back("-o"); + baseArgs.push_back(outputFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + if (config_.getVerbose()) { + errs() << "Failed to extract assembly for " << source.path << "\n"; + } + continue; + } + + if (sys::fs::exists(outputFile)) { + unit.addGeneratedFile("assembly", outputFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractAST(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/ast/" + sys::path::stem(source.path).str() + ".ast").str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-ast-dump"); + baseArgs.push_back("-fsyntax-only"); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + if (result && result->exitCode == 0) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << result->stdout; + unit.addGeneratedFile("ast", outputFile); + } + } + } + return Error::success(); +} + +Error DataExtractor::extractPreprocessed(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string ext = (source.language == "C++") ? ".ii" : ".i"; + std::string outputFile = + (tempDir + "/preprocessed/" + sys::path::stem(source.path).str() + ext) + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-E"); + baseArgs.push_back("-o"); + baseArgs.push_back(outputFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + if (config_.getVerbose()) { + errs() << "Failed to extract preprocessed for " << source.path << "\n"; + } + continue; + } + + if (sys::fs::exists(outputFile)) { + unit.addGeneratedFile("preprocessed", outputFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractIncludeTree(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/include-tree/" + sys::path::stem(source.path).str() + + ".include.txt") + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-H"); + baseArgs.push_back("-fsyntax-only"); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + if (result && !result->stderr.empty()) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << result->stderr; // Include tree goes to stderr + unit.addGeneratedFile("include-tree", outputFile); + } + } + } + return Error::success(); +} + +Error DataExtractor::extractDependencies(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = (tempDir + "/dependencies/" + + sys::path::stem(source.path).str() + ".deps.txt") + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-MM"); // Generate dependencies in Makefile format + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + if (result && result->exitCode == 0 && !result->stdout.empty()) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << result->stdout; // Dependencies go to stdout + unit.addGeneratedFile("dependencies", outputFile); + } + } + } + return Error::success(); +} + +Error DataExtractor::extractDebugInfo(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = (tempDir + "/debug/" + + sys::path::stem(source.path).str() + ".debug.txt") + .str(); + std::string objectFile = + (tempDir + "/debug/" + sys::path::stem(source.path).str() + ".o").str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-g"); + baseArgs.push_back("-c"); + baseArgs.push_back("-o"); + baseArgs.push_back(objectFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + if (config_.getVerbose()) { + errs() << "Failed to extract debug info for " << source.path << "\n"; + } + continue; + } + + // Extract DWARF info using llvm-dwarfdump + if (sys::fs::exists(objectFile)) { + llvm::SmallVector dwarfArgs = {objectFile}; + auto result = + ProcessRunner::run("llvm-dwarfdump", dwarfArgs, config_.getTimeout()); + if (result && result->exitCode == 0) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << result->stdout; + unit.addGeneratedFile("debug", outputFile); + } + } + } + } + return Error::success(); +} + +Error DataExtractor::extractStaticAnalysis(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/static-analyzer/" + sys::path::stem(source.path).str() + + ".analysis.txt") + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("--analyze"); + baseArgs.push_back("-Xanalyzer"); + baseArgs.push_back("-analyzer-output=text"); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + if (result) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << "STDOUT:\n" << result->stdout << "\nSTDERR:\n" << result->stderr; + unit.addGeneratedFile("static-analyzer", outputFile); + } + } + } + return Error::success(); +} + +Error DataExtractor::extractMacroExpansion(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/preprocessed/" + sys::path::stem(source.path).str() + + ".macro-expanded" + ((source.language == "C++") ? ".ii" : ".i")) + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-E"); + baseArgs.push_back("-dM"); // Show macro definitions + baseArgs.push_back("-o"); + baseArgs.push_back(outputFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + if (config_.getVerbose()) { + errs() << "Failed to extract macro expansion for " << source.path + << "\n"; + } + continue; + } + + if (sys::fs::exists(outputFile)) { + unit.addGeneratedFile("macro-expansion", outputFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractCompilationPhases(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/debug/" + sys::path::stem(source.path).str() + + ".phases.txt") + .str(); + std::string bindingsFile = + (tempDir + "/debug/" + sys::path::stem(source.path).str() + + ".bindings.txt") + .str(); + + // First: Extract compilation bindings with -ccc-print-bindings + llvm::SmallVector bindingsArgs = + getBaseCompilerArgs(unit.getInfo()); + bindingsArgs.push_back( + "-ccc-print-bindings"); // Print compilation bindings/phases + bindingsArgs.push_back("-fsyntax-only"); + bindingsArgs.push_back(source.path); + + auto bindingsResult = ProcessRunner::run( + config_.getToolPath("clang"), bindingsArgs, config_.getTimeout()); + if (bindingsResult) { + std::error_code EC; + raw_fd_ostream bindingsOS(bindingsFile, EC); + if (!EC) { + bindingsOS << bindingsResult->stderr; // Bindings output goes to stderr + unit.addGeneratedFile("compilation-phases", bindingsFile); + } + } + + // Second: Extract verbose compiler info with -v + llvm::SmallVector verboseArgs = + getBaseCompilerArgs(unit.getInfo()); + verboseArgs.push_back("-v"); // Verbose compilation phases + verboseArgs.push_back("-fsyntax-only"); + verboseArgs.push_back(source.path); + + auto verboseResult = ProcessRunner::run(config_.getToolPath("clang"), + verboseArgs, config_.getTimeout()); + if (verboseResult) { + std::error_code EC; + raw_fd_ostream verboseOS(outputFile, EC); + if (!EC) { + verboseOS << "COMPILATION PHASES:\n" + << verboseResult->stderr; // Verbose output goes to stderr + unit.addGeneratedFile("compilation-phases", outputFile); + } + } + } + return Error::success(); +} + +Error DataExtractor::extractFTimeReport(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = (tempDir + "/ftime-report/" + + sys::path::stem(source.path).str() + ".ftime.txt") + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-ftime-report"); + baseArgs.push_back("-fsyntax-only"); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + if (result) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << "FTIME REPORT:\n" + << result->stderr; // ftime-report output goes to stderr + unit.addGeneratedFile("ftime-report", outputFile); + if (config_.getVerbose()) { + outs() << "FTime Report: " << outputFile << "\n"; + } + } + } + } + return Error::success(); +} + +Error DataExtractor::extractVersionInfo(CompilationUnit &unit, + llvm::StringRef tempDir) { + std::string outputFile = (tempDir + "/version-info/clang-version.txt").str(); + + llvm::SmallVector args; + args.push_back("--version"); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), args, + config_.getTimeout()); + if (result) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << result->stdout; + unit.addGeneratedFile("version-info", outputFile); + if (config_.getVerbose()) { + outs() << "Version Info: " << outputFile << "\n"; + } + } + } + return Error::success(); +} + +Error DataExtractor::runCompilerWithFlags( + const llvm::SmallVector &args) { + auto result = ProcessRunner::run(config_.getToolPath("clang"), args, + config_.getTimeout()); + if (result && result->exitCode == 0) { + return Error::success(); + } + + // Fallback: retry without offloading-specific flags to at least produce + // host-side artifacts when device toolchains are unavailable. + llvm::SmallVector sanitizedArgs; + for (size_t i = 0; i < args.size(); ++i) { + llvm::StringRef a(args[i]); + + if (a.starts_with("-fopenmp-targets")) { + continue; // drop device list + } + if (a == "-Xopenmp-target") { + // drop the flag and its immediate argument, if present + if (i + 1 < args.size()) { + ++i; + } + continue; + } + if (a.starts_with("-Xopenmp-target=")) { + continue; // drop single-arg form + } + sanitizedArgs.push_back(args[i]); + } + + auto retry = ProcessRunner::run(config_.getToolPath("clang"), sanitizedArgs, + config_.getTimeout()); + if (!retry || retry->exitCode != 0) { + return createStringError(std::make_error_code(std::errc::io_error), + "Compiler failed"); + } + return Error::success(); +} + +Error DataExtractor::extractASTJSON(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/ast/" + sys::path::stem(source.path).str() + ".ast.json") + .str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-Xclang"); + baseArgs.push_back("-ast-dump=json"); + baseArgs.push_back("-fsyntax-only"); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + if (result && result->exitCode == 0) { + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << result->stdout; + unit.addGeneratedFile("ast-json", outputFile); + if (config_.getVerbose()) { + outs() << "AST JSON: " << outputFile << "\n"; + } + } + } + } + return Error::success(); +} + +Error DataExtractor::extractDiagnostics(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string outputFile = + (tempDir + "/diagnostics/" + sys::path::stem(source.path).str() + + ".diagnostics.txt") + .str(); + + std::error_code EC; + raw_fd_ostream OS(outputFile, EC); + if (!EC) { + OS << "Diagnostics for: " << source.path << "\n"; + + // Run basic diagnostics + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-fdiagnostics-parseable-fixits"); + baseArgs.push_back("-fdiagnostics-absolute-paths"); + baseArgs.push_back("-Wall"); + baseArgs.push_back("-Wextra"); + baseArgs.push_back("-fsyntax-only"); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + if (result) { + OS << "Exit code: " << result->exitCode << "\n"; + if (!result->stderr.empty()) { + OS << result->stderr << "\n"; + } + if (!result->stdout.empty()) { + OS << result->stdout << "\n"; + } + } + + // Run additional diagnostics with more flags + llvm::SmallVector extraArgs = + getBaseCompilerArgs(unit.getInfo()); + extraArgs.push_back("-Weverything"); + extraArgs.push_back("-Wno-c++98-compat"); + extraArgs.push_back("-Wno-c++98-compat-pedantic"); + extraArgs.push_back("-fsyntax-only"); + extraArgs.push_back(source.path); + + auto extraResult = ProcessRunner::run(config_.getToolPath("clang"), + extraArgs, config_.getTimeout()); + if (extraResult) { + OS << "\nExtended diagnostics:\n"; + OS << "Exit code: " << extraResult->exitCode << "\n"; + if (!extraResult->stderr.empty()) { + OS << extraResult->stderr << "\n"; + } + if (!extraResult->stdout.empty()) { + OS << extraResult->stdout << "\n"; + } + } + + unit.addGeneratedFile("diagnostics", outputFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractCoverage(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string objectFile = (tempDir + "/" + sourceStem + "_cov.o").str(); + std::string executableFile = (tempDir + "/" + sourceStem + "_cov").str(); + std::string profrawFile = (tempDir + "/" + sourceStem + ".profraw").str(); + std::string profdataFile = (tempDir + "/" + sourceStem + ".profdata").str(); + std::string coverageFile = + (tempDir + "/coverage/" + sourceStem + ".coverage.json").str(); + + // Compile with coverage instrumentation to create executable + llvm::SmallVector compileArgs = + getBaseCompilerArgs(unit.getInfo()); + compileArgs.push_back("-fprofile-instr-generate=" + profrawFile); + compileArgs.push_back("-fcoverage-mapping"); + compileArgs.push_back("-o"); + compileArgs.push_back(executableFile); + compileArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(compileArgs)) { + continue; + } + + if (sys::fs::exists(executableFile)) { + // Run the executable to generate profile data (if it doesn't require + // input) + auto runResult = + ProcessRunner::run(executableFile, {}, config_.getTimeout()); + + // Convert raw profile to indexed format if profraw exists + if (sys::fs::exists(profrawFile)) { + llvm::SmallVector mergeArgs = { + "merge", "-sparse", "-o", profdataFile, profrawFile}; + auto mergeResult = ProcessRunner::run("llvm-profdata", mergeArgs, + config_.getTimeout()); + + if (mergeResult && mergeResult->exitCode == 0 && + sys::fs::exists(profdataFile)) { + // Generate coverage report in JSON format + llvm::SmallVector covArgs = { + "export", executableFile, "-instr-profile=" + profdataFile, + "-format=json"}; + auto covResult = + ProcessRunner::run("llvm-cov", covArgs, config_.getTimeout()); + + if (covResult && covResult->exitCode == 0) { + std::error_code EC; + raw_fd_ostream OS(coverageFile, EC); + if (!EC) { + OS << covResult->stdout; + unit.addGeneratedFile("coverage", coverageFile); + if (config_.getVerbose()) { + outs() << "Coverage: " << coverageFile << "\n"; + } + } + } + } + // Clean up temporary files + sys::fs::remove(profrawFile); + sys::fs::remove(profdataFile); + } + // Clean up temporary executable + sys::fs::remove(executableFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractTimeTrace(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string traceFile = + (tempDir + "/time-trace/" + sourceStem + ".trace.json").str(); + + // Create a temporary object file in temp directory + std::string tempObject = (tempDir + "/" + sourceStem + "_trace.o").str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-ftime-trace"); + baseArgs.push_back("-ftime-trace-granularity=0"); + baseArgs.push_back("-c"); + baseArgs.push_back("-o"); + baseArgs.push_back(tempObject); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + + // The trace file is generated next to the object file with .json extension + std::string expectedTraceFile = + (tempDir + "/" + sourceStem + "_trace.json").str(); + + // Also check the working directory in case trace went there + std::string workingDirTrace = sourceStem + ".json"; + + if (sys::fs::exists(expectedTraceFile)) { + if (auto EC = sys::fs::rename(expectedTraceFile, traceFile)) { + if (config_.getVerbose()) { + errs() << "Failed to move trace file: " << EC.message() << "\n"; + } + } else { + unit.addGeneratedFile("time-trace", traceFile); + if (config_.getVerbose()) { + outs() << "Time trace: " << traceFile << "\n"; + } + } + } else if (sys::fs::exists(workingDirTrace)) { + if (auto EC = sys::fs::rename(workingDirTrace, traceFile)) { + if (config_.getVerbose()) { + errs() << "Failed to move trace file from working dir: " + << EC.message() << "\n"; + } + } else { + unit.addGeneratedFile("time-trace", traceFile); + if (config_.getVerbose()) { + outs() << "Time trace: " << traceFile << "\n"; + } + } + } else if (config_.getVerbose()) { + errs() << "Time trace file not found for " << source.path << "\n"; + } + + // Clean up temporary object file + sys::fs::remove(tempObject); + } + return Error::success(); +} + +Error DataExtractor::extractRuntimeTrace(CompilationUnit &unit, + llvm::StringRef tempDir) { + + // Check for OpenMP offloading flags + bool hasOffloading = false; + for (const auto &flag : unit.getInfo().compileFlags) { + StringRef flagStr(flag); + // Check for various OpenMP offloading indicators + if (flagStr.contains("offload") || // Generic offload flags + flagStr.contains("-fopenmp-targets") || // OpenMP target specification + flagStr.contains("-Xopenmp-target") || // OpenMP target-specific flags + flagStr.contains("nvptx") || // NVIDIA GPU targets + flagStr.contains("amdgcn") || // AMD GPU targets + flagStr.contains("-fopenmp")) { // Basic OpenMP (may have runtime) + hasOffloading = true; + break; + } + } + + if (!hasOffloading) { + if (config_.getVerbose()) + outs() << "Runtime trace skipped - no OpenMP offloading flags detected\n"; + return Error::success(); + } + + if (config_.getVerbose()) + outs() + << "OpenMP offloading detected, attempting runtime trace extraction\n"; + + // Find executable name from compile flags + std::string executableStr = "a.out"; // Default executable name + for (size_t i = 0; i < unit.getInfo().compileFlags.size(); ++i) { + if (unit.getInfo().compileFlags[i] == "-o" && + i + 1 < unit.getInfo().compileFlags.size()) { + executableStr = unit.getInfo().compileFlags[i + 1]; + break; + } + } + + StringRef executable = executableStr; + + if (config_.getVerbose()) + outs() << "Looking for executable: " << executable << "\n"; + + if (!sys::fs::exists(executable)) { + if (config_.getVerbose()) { + outs() << "Runtime trace skipped - executable not found: " << executable + << "\n"; + outs() << "Note: Executable is needed to generate runtime profile data\n"; + outs() << "Checked current directory for: " << executable << "\n"; + + // List current directory contents for debugging + outs() << "Current directory contents:\n"; + std::error_code EC; + for (sys::fs::directory_iterator I(".", EC), E; I != E && !EC; + I.increment(EC)) { + outs() << " " << I->path() << "\n"; + } + } + + return Error::success(); + } + + // Create a trace directory + SmallString<256> traceDir; + sys::path::append(traceDir, tempDir, "runtime-trace"); + if (auto ec = sys::fs::create_directories(traceDir, true)) { + if (config_.getVerbose()) + outs() << "Warning: Failed to create runtime trace directory: " + << ec.message() << "\n"; + } + + // Prepare trace file + SmallString<256> traceFile; + sys::path::append(traceFile, traceDir, "profile.json"); + + if (config_.getVerbose()) { + outs() << "Running runtime trace: " << executable << "\n"; + outs() << "Trace file: " << traceFile << "\n"; + } + + // Set environment variable for OpenMP target profiling + SmallVector env; + env.push_back("LIBOMPTARGET_PROFILE=" + traceFile.str().str()); + + if (config_.getVerbose()) { + outs() << "Setting environment: LIBOMPTARGET_PROFILE=" << traceFile << "\n"; + outs() << "Executing: " << executable << "\n"; + } + + // Run executable with profiling environment + auto result = + ProcessRunner::runWithEnv(executable, {}, env, config_.getTimeout()); + + if (config_.getVerbose()) { + if (result) { + outs() << "Runtime trace completed with exit code: " << result->exitCode + << "\n"; + + if (!result->stdout.empty()) + outs() << "STDOUT: " << result->stdout << "\n"; + if (!result->stderr.empty()) + outs() << "STDERR: " << result->stderr << "\n"; + } else { + outs() << "Runtime trace failed: " << toString(result.takeError()) + << "\n"; + } + } + + // Register trace file if generated + if (sys::fs::exists(traceFile)) { + unit.addGeneratedFile("runtime-trace", traceFile.str()); + if (config_.getVerbose()) + outs() << "Runtime trace saved: " << traceFile << "\n"; + } else { + if (config_.getVerbose()) { + outs() << "Runtime trace failed - no trace file generated at: " + << traceFile << "\n"; + outs() << "This may happen if:\n"; + outs() << " 1. The program didn't use OpenMP target offloading\n"; + outs() << " 2. The runtime doesn't support LIBOMPTARGET_PROFILE\n"; + outs() << " 3. The program crashed before generating profile data\n"; + } + } + + return Error::success(); +} + +Error DataExtractor::extractSARIF(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string sarifFile = + (tempDir + "/static-analyzer/" + sourceStem + ".sarif").str(); + + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("--analyze"); + baseArgs.push_back("-Xanalyzer"); + baseArgs.push_back("-analyzer-output=sarif"); + baseArgs.push_back("-o"); + baseArgs.push_back(sarifFile); + baseArgs.push_back(source.path); + + auto result = ProcessRunner::run(config_.getToolPath("clang"), baseArgs, + config_.getTimeout()); + + if (result && sys::fs::exists(sarifFile)) { + // Check if the SARIF file has content + uint64_t fileSize; + auto fileSizeErr = sys::fs::file_size(sarifFile, fileSize); + if (!fileSizeErr && fileSize > 0) { + unit.addGeneratedFile("static-analysis-sarif", sarifFile); + if (config_.getVerbose()) { + outs() << "SARIF static analysis extracted: " << sarifFile << "\n"; + } + } else if (config_.getVerbose()) { + outs() << "SARIF file created but empty for " << source.path << "\n"; + } + } else if (config_.getVerbose()) { + errs() << "Failed to extract SARIF static analysis for " << source.path + << "\n"; + } + } + return Error::success(); +} + +Error DataExtractor::extractBinarySize(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string objectFile = (tempDir + "/" + sourceStem + "_size.o").str(); + std::string sizeFile = + (tempDir + "/binary-analysis/" + sourceStem + ".size.txt").str(); + + // Compile object file + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-c"); + baseArgs.push_back("-o"); + baseArgs.push_back(objectFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + continue; + } + + if (sys::fs::exists(objectFile)) { + std::error_code EC; + raw_fd_ostream OS(sizeFile, EC); + if (!EC) { + OS << "Binary size analysis for: " << source.path << "\n"; + OS << "Generated from object file: " << objectFile << "\n\n"; + + // Try different llvm-size formats + llvm::SmallVector berkeleyArgs = {objectFile}; + auto berkeleyResult = + ProcessRunner::run("llvm-size", berkeleyArgs, config_.getTimeout()); + if (berkeleyResult && berkeleyResult->exitCode == 0) { + OS << "Berkeley format (default):\n" + << berkeleyResult->stdout << "\n"; + } + + // Also try -A format for more details + llvm::SmallVector sysVArgs = {"-A", objectFile}; + auto sysVResult = + ProcessRunner::run("llvm-size", sysVArgs, config_.getTimeout()); + if (sysVResult && sysVResult->exitCode == 0) { + OS << "System V format:\n" << sysVResult->stdout << "\n"; + } + + unit.addGeneratedFile("binary-size", sizeFile); + if (config_.getVerbose()) { + outs() << "Binary size: " << sizeFile << "\n"; + } + } + + // Clean up temporary object file + sys::fs::remove(objectFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractPGO(CompilationUnit &unit, + llvm::StringRef tempDir) { + // Look for existing profile raw data file from compilation + std::string profrawFile = tempDir.str() + "/profile.profraw"; + + if (sys::fs::exists(profrawFile)) { + std::string profileFile = (tempDir + "/pgo/merged.profdata").str(); + std::string profileText = (tempDir + "/pgo/profile.txt").str(); + + // Convert raw profile to indexed format + llvm::SmallVector mergeArgs = {"merge", "-sparse", "-o", + profileFile, profrawFile}; + auto mergeResult = + ProcessRunner::run("llvm-profdata", mergeArgs, config_.getTimeout()); + + if (mergeResult && mergeResult->exitCode == 0 && + sys::fs::exists(profileFile)) { + // Generate human-readable profile summary + llvm::SmallVector showArgs = { + "show", "-all-functions", "-text", "-detailed-summary", profileFile}; + auto showResult = + ProcessRunner::run("llvm-profdata", showArgs, config_.getTimeout()); + + if (showResult && showResult->exitCode == 0) { + std::error_code EC; + raw_fd_ostream OS(profileText, EC); + if (!EC) { + OS << "PGO Profile Data Summary\n"; + OS << "========================\n"; + OS << "Source profile: " << profrawFile << "\n"; + OS << "Indexed profile: " << profileFile << "\n\n"; + OS << showResult->stdout; + unit.addGeneratedFile("pgo-profile", profileText); + if (config_.getVerbose()) { + outs() << "PGO profile data extracted: " << profileText << "\n"; + } + } + } + + // Also create a JSON-like summary if possible + llvm::SmallVector jsonArgs = {"show", "-json", + profileFile}; + auto jsonResult = + ProcessRunner::run("llvm-profdata", jsonArgs, config_.getTimeout()); + + if (jsonResult && jsonResult->exitCode == 0) { + std::string jsonFile = (tempDir + "/pgo/profile.json").str(); + std::error_code EC; + raw_fd_ostream jsonOS(jsonFile, EC); + if (!EC) { + jsonOS << jsonResult->stdout; + unit.addGeneratedFile("pgo-profile-json", jsonFile); + if (config_.getVerbose()) { + outs() << "PGO profile JSON extracted: " << jsonFile << "\n"; + } + } + } + sys::fs::remove(profileFile); + } else if (config_.getVerbose()) { + errs() << "Failed to merge PGO profile data\n"; + } + } else if (config_.getVerbose()) { + outs() << "No PGO profile data found to extract\n"; + } + + return Error::success(); +} + +Error DataExtractor::extractSymbols(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string objectFile = (tempDir + "/" + sourceStem + "_symbols.o").str(); + std::string symbolsFile = + (tempDir + "/binary-analysis/" + sourceStem + ".symbols.txt").str(); + + // Compile object file + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-c"); + baseArgs.push_back("-o"); + baseArgs.push_back(objectFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + continue; + } + + if (sys::fs::exists(objectFile)) { + std::error_code EC; + raw_fd_ostream OS(symbolsFile, EC); + if (!EC) { + OS << "Symbol table for: " << source.path << "\n"; + OS << "Generated from object file: " << objectFile << "\n\n"; + + // Extract symbols using llvm-nm + llvm::SmallVector nmArgs = {"-C", "-a", objectFile}; + auto nmResult = + ProcessRunner::run("llvm-nm", nmArgs, config_.getTimeout()); + if (nmResult && nmResult->exitCode == 0) { + OS << "Symbols:\n" << nmResult->stdout << "\n"; + } + + unit.addGeneratedFile("symbols", symbolsFile); + if (config_.getVerbose()) { + outs() << "Symbols: " << symbolsFile << "\n"; + } + } + + // Clean up temporary object file + sys::fs::remove(objectFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractObjdump(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string objectFile = (tempDir + "/" + sourceStem + "_objdump.o").str(); + std::string objdumpFile = + (tempDir + "/binary-analysis/" + sourceStem + ".objdump.txt").str(); + + // Compile object file + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-c"); + baseArgs.push_back("-o"); + baseArgs.push_back(objectFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + continue; + } + + if (sys::fs::exists(objectFile)) { + std::error_code EC; + raw_fd_ostream OS(objdumpFile, EC); + if (!EC) { + OS << "Object dump for: " << source.path << "\n"; + OS << "Generated from object file: " << objectFile << "\n\n"; + + // Disassemble using llvm-objdump + llvm::SmallVector objdumpArgs = {"-d", "-t", "-r", + objectFile}; + auto objdumpResult = ProcessRunner::run("llvm-objdump", objdumpArgs, + config_.getTimeout()); + if (objdumpResult && objdumpResult->exitCode == 0) { + OS << objdumpResult->stdout << "\n"; + } + + unit.addGeneratedFile("objdump", objdumpFile); + if (config_.getVerbose()) { + outs() << "Objdump: " << objdumpFile << "\n"; + } + } + + // Clean up temporary object file + sys::fs::remove(objectFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractXRay(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string executableFile = (tempDir + "/" + sourceStem + "_xray").str(); + std::string xrayFile = + (tempDir + "/binary-analysis/" + sourceStem + ".xray.txt").str(); + + // Compile with XRay instrumentation + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-fxray-instrument"); + baseArgs.push_back("-fxray-instruction-threshold=1"); + baseArgs.push_back("-o"); + baseArgs.push_back(executableFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + continue; + } + + if (sys::fs::exists(executableFile)) { + std::error_code EC; + raw_fd_ostream OS(xrayFile, EC); + if (!EC) { + OS << "XRay analysis for: " << source.path << "\n"; + OS << "Generated from executable: " << executableFile << "\n\n"; + + // Extract XRay instrumentation info + llvm::SmallVector xrayArgs = {"extract", + executableFile}; + auto xrayResult = + ProcessRunner::run("llvm-xray", xrayArgs, config_.getTimeout()); + if (xrayResult && xrayResult->exitCode == 0) { + OS << "XRay instrumentation:\n" << xrayResult->stdout << "\n"; + } + + unit.addGeneratedFile("xray", xrayFile); + if (config_.getVerbose()) { + outs() << "XRay: " << xrayFile << "\n"; + } + } + + // Clean up temporary executable + sys::fs::remove(executableFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractOptDot(CompilationUnit &unit, + llvm::StringRef tempDir) { + for (const auto &source : unit.getInfo().sources) { + if (source.isHeader) + continue; + + std::string sourceStem = sys::path::stem(source.path).str(); + std::string bitcodeFile = (tempDir + "/" + sourceStem + ".bc").str(); + std::string dotFile = (tempDir + "/ir/" + sourceStem + ".cfg.dot").str(); + + // Compile to bitcode + llvm::SmallVector baseArgs = + getBaseCompilerArgs(unit.getInfo()); + baseArgs.push_back("-emit-llvm"); + baseArgs.push_back("-c"); + baseArgs.push_back("-o"); + baseArgs.push_back(bitcodeFile); + baseArgs.push_back(source.path); + + if (auto Err = runCompilerWithFlags(baseArgs)) { + continue; + } + + if (sys::fs::exists(bitcodeFile)) { + // Generate CFG dot file using opt + llvm::SmallVector optArgs = { + "-dot-cfg", "-disable-output", bitcodeFile}; + auto optResult = ProcessRunner::run("opt", optArgs, config_.getTimeout()); + + // The dot file is typically generated in current directory + std::string expectedDotFile = "." + sourceStem + ".dot"; + if (sys::fs::exists(expectedDotFile)) { + if (auto EC = sys::fs::rename(expectedDotFile, dotFile)) { + if (config_.getVerbose()) { + errs() << "Failed to move dot file: " << EC.message() << "\n"; + } + } else { + unit.addGeneratedFile("cfg-dot", dotFile); + if (config_.getVerbose()) { + outs() << "CFG DOT: " << dotFile << "\n"; + } + } + } + + // Clean up temporary bitcode file + sys::fs::remove(bitcodeFile); + } + } + return Error::success(); +} + +Error DataExtractor::extractSources(CompilationUnit &unit, + llvm::StringRef tempDir) { + if (config_.getVerbose()) { + outs() << "Extracting source files based on dependencies...\n"; + } + + // Create sources directory + SmallString<256> sourcesDir; + sys::path::append(sourcesDir, tempDir, "sources"); + if (auto EC = sys::fs::create_directories(sourcesDir)) { + if (config_.getVerbose()) { + outs() << "Warning: Failed to create sources directory: " << EC.message() + << "\n"; + } + return Error::success(); // Continue even if we can't create directory + } + + // Find and parse dependencies files + SmallString<256> depsDir; + sys::path::append(depsDir, tempDir, "dependencies"); + + if (!sys::fs::exists(depsDir)) { + if (config_.getVerbose()) { + outs() << "No dependencies directory found, skipping source extraction\n"; + } + return Error::success(); + } + + std::error_code EC; + for (sys::fs::directory_iterator I(depsDir, EC), E; I != E && !EC; + I.increment(EC)) { + StringRef filePath = I->path(); + if (!filePath.ends_with(".deps.txt")) { + continue; + } + + if (config_.getVerbose()) { + outs() << "Processing dependencies file: " << filePath << "\n"; + } + + // Read and parse dependencies file + auto bufferOrErr = MemoryBuffer::getFile(filePath); + if (!bufferOrErr) { + if (config_.getVerbose()) { + outs() << "Warning: Failed to read dependencies file: " << filePath + << "\n"; + } + continue; + } + + StringRef content = bufferOrErr.get()->getBuffer(); + SmallVector lines; + content.split(lines, '\n'); + + for (StringRef line : lines) { + line = line.trim(); + if (line.empty() || !line.contains(":")) { + continue; + } + + // Parse line format: "target.o: source1.c source2.h ..." + auto colonPos = line.find(':'); + if (colonPos == StringRef::npos) { + continue; + } + + StringRef sourcesStr = line.substr(colonPos + 1).trim(); + SmallVector sourceFiles; + sourcesStr.split(sourceFiles, ' ', -1, false); + + for (StringRef sourceFile : sourceFiles) { + sourceFile = sourceFile.trim(); + if (sourceFile.empty()) { + continue; + } + + // Convert relative paths to absolute paths + SmallString<256> absoluteSourcePath; + if (sys::path::is_absolute(sourceFile)) { + absoluteSourcePath = sourceFile; + } else { + // Get current working directory + SmallString<256> currentDir; + sys::fs::current_path(currentDir); + absoluteSourcePath = currentDir; + sys::path::append(absoluteSourcePath, sourceFile); + } + + // Check if source file exists + if (!sys::fs::exists(absoluteSourcePath)) { + if (config_.getVerbose()) { + outs() << "Warning: Source file not found: " << absoluteSourcePath + << "\n"; + } + continue; + } + + // Create destination path preserving directory structure + SmallString<256> destPath; + sys::path::append(destPath, sourcesDir, + sys::path::filename(sourceFile)); + + // Copy source file to sources directory + if (auto copyErr = FileManager::copyFile(absoluteSourcePath.str().str(), + destPath.str().str())) { + if (config_.getVerbose()) { + outs() << "Warning: Failed to copy source file " + << absoluteSourcePath << " to " << destPath << "\n"; + } + } else { + unit.addGeneratedFile("sources", destPath.str().str()); + if (config_.getVerbose()) { + outs() << "Copied source: " << sourceFile << " -> " << destPath + << "\n"; + } + } + } + } + } + + return Error::success(); +} + +const DataExtractor::ExtractorInfo DataExtractor::extractors_[] = { + {&DataExtractor::extractASTJSON, "AST JSON"}, + {&DataExtractor::extractDiagnostics, "Diagnostics"}, + {&DataExtractor::extractCoverage, "Coverage"}, + {&DataExtractor::extractTimeTrace, "Time trace"}, + {&DataExtractor::extractRuntimeTrace, "Runtime trace"}, + {&DataExtractor::extractSARIF, "SARIF"}, + {&DataExtractor::extractBinarySize, "Binary size"}, + {&DataExtractor::extractPGO, "PGO"}, + {&DataExtractor::extractSymbols, "Symbols"}, + {&DataExtractor::extractObjdump, "Objdump"}, + {&DataExtractor::extractXRay, "XRay"}, + {&DataExtractor::extractOptDot, "Opt DOT"}}; + +const size_t DataExtractor::numExtractors_ = + sizeof(DataExtractor::extractors_) / sizeof(DataExtractor::ExtractorInfo); + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Core/DataExtractor.h b/llvm/tools/llvm-advisor/src/Core/DataExtractor.h new file mode 100644 index 0000000000000..0cd4ee71cdd1a --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/DataExtractor.h @@ -0,0 +1,82 @@ +//===------------------ DataExtractor.h - LLVM Advisor --------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the DataExtractor code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_DATA_EXTRACTOR_H +#define LLVM_ADVISOR_DATA_EXTRACTOR_H + +#include "../Config/AdvisorConfig.h" +#include "CompilationUnit.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/Error.h" +#include + +namespace llvm { +namespace advisor { + +class DataExtractor { +public: + DataExtractor(const AdvisorConfig &config); + + Error extractAllData(CompilationUnit &unit, llvm::StringRef tempDir); + +private: + llvm::SmallVector + getBaseCompilerArgs(const CompilationUnitInfo &unitInfo) const; + + Error extractIR(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractAssembly(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractAST(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractPreprocessed(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractIncludeTree(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractDependencies(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractDebugInfo(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractStaticAnalysis(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractMacroExpansion(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractCompilationPhases(CompilationUnit &unit, + llvm::StringRef tempDir); + Error extractFTimeReport(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractVersionInfo(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractSources(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractASTJSON(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractDiagnostics(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractCoverage(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractTimeTrace(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractRuntimeTrace(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractSARIF(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractBinarySize(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractPGO(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractSymbols(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractObjdump(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractXRay(CompilationUnit &unit, llvm::StringRef tempDir); + Error extractOptDot(CompilationUnit &unit, llvm::StringRef tempDir); + + Error runCompilerWithFlags(const llvm::SmallVector &args); + + using ExtractorMethod = Error (DataExtractor::*)(CompilationUnit &, + llvm::StringRef); + struct ExtractorInfo { + ExtractorMethod method; + const char *name; + }; + + static const ExtractorInfo extractors_[]; + static const size_t numExtractors_; + + const AdvisorConfig &config_; +}; + +} // namespace advisor +} // namespace llvm + +#endif diff --git a/llvm/tools/llvm-advisor/src/Core/ViewerLauncher.cpp b/llvm/tools/llvm-advisor/src/Core/ViewerLauncher.cpp new file mode 100644 index 0000000000000..94871d0185080 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/ViewerLauncher.cpp @@ -0,0 +1,117 @@ +//===---------------- ViewerLauncher.cpp - LLVM Advisor ------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "ViewerLauncher.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/Path.h" +#include "llvm/Support/Program.h" +#include "llvm/Support/raw_ostream.h" + +using namespace llvm; +using namespace llvm::advisor; + +Expected ViewerLauncher::findPythonExecutable() { + std::vector candidates = {"python3", "python"}; + + for (const auto &candidate : candidates) { + if (auto path = sys::findProgramByName(candidate)) { + return *path; + } + } + + return createStringError( + std::make_error_code(std::errc::no_such_file_or_directory), + "Python executable not found. Please install Python 3."); +} + +Expected ViewerLauncher::getViewerScript() { + SmallString<256> scriptPath; + + // Try to find the server script relative to the executable + auto mainExecutable = sys::fs::getMainExecutable(nullptr, nullptr); + if (mainExecutable.empty()) { + return createStringError( + std::make_error_code(std::errc::no_such_file_or_directory), + "Cannot determine executable path"); + } + + // Try: relative to binary (development/build tree) + sys::path::append(scriptPath, sys::path::parent_path(mainExecutable)); + sys::path::append(scriptPath, ".."); + sys::path::append(scriptPath, "tools"); + sys::path::append(scriptPath, "webserver"); + sys::path::append(scriptPath, "server.py"); + + if (sys::fs::exists(scriptPath)) { + return scriptPath.str().str(); + } + + // Try: relative to binary (same directory as executable) + scriptPath.clear(); + sys::path::append(scriptPath, sys::path::parent_path(mainExecutable)); + sys::path::append(scriptPath, "tools"); + sys::path::append(scriptPath, "webserver"); + sys::path::append(scriptPath, "server.py"); + + if (sys::fs::exists(scriptPath)) { + return scriptPath.str().str(); + } + + // Try: installed location + scriptPath.clear(); + sys::path::append(scriptPath, sys::path::parent_path(mainExecutable)); + sys::path::append(scriptPath, ".."); + sys::path::append(scriptPath, "share"); + sys::path::append(scriptPath, "llvm-advisor"); + sys::path::append(scriptPath, "tools"); + sys::path::append(scriptPath, "webserver"); + sys::path::append(scriptPath, "server.py"); + + if (sys::fs::exists(scriptPath)) { + return scriptPath.str().str(); + } + + return createStringError( + std::make_error_code(std::errc::no_such_file_or_directory), + "Web server script not found. Please ensure tools/webserver/server.py " + "exists."); +} + +Expected ViewerLauncher::launch(const std::string &outputDir, int port) { + auto pythonOrErr = findPythonExecutable(); + if (!pythonOrErr) { + return pythonOrErr.takeError(); + } + + auto scriptOrErr = getViewerScript(); + if (!scriptOrErr) { + return scriptOrErr.takeError(); + } + + // Verify output directory exists and has data + if (!sys::fs::exists(outputDir)) { + return createStringError( + std::make_error_code(std::errc::no_such_file_or_directory), + "Output directory does not exist: " + outputDir); + } + + std::vector args = {*pythonOrErr, *scriptOrErr, + "--data-dir", outputDir, + "--port", std::to_string(port)}; + + // Execute the Python web server + int result = sys::ExecuteAndWait(*pythonOrErr, args); + + if (result != 0) { + return createStringError(std::make_error_code(std::errc::io_error), + "Web server failed with exit code: " + + std::to_string(result)); + } + + return result; +} diff --git a/llvm/tools/llvm-advisor/src/Core/ViewerLauncher.h b/llvm/tools/llvm-advisor/src/Core/ViewerLauncher.h new file mode 100644 index 0000000000000..9ad2b07d215dd --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Core/ViewerLauncher.h @@ -0,0 +1,36 @@ +//===---------------- ViewerLauncher.h - LLVM Advisor --------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// ViewerLauncher handles launching the Python web server to visualize +// the collected compilation data. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_ADVISOR_CORE_VIEWERLAUNCHER_H +#define LLVM_ADVISOR_CORE_VIEWERLAUNCHER_H + +#include "llvm/Support/Error.h" +#include + +namespace llvm { +namespace advisor { + +class ViewerLauncher { +public: + static llvm::Expected launch(const std::string &outputDir, + int port = 8000); + +private: + static llvm::Expected findPythonExecutable(); + static llvm::Expected getViewerScript(); +}; + +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_CORE_VIEWERLAUNCHER_H diff --git a/llvm/tools/llvm-advisor/src/Detection/UnitDetector.cpp b/llvm/tools/llvm-advisor/src/Detection/UnitDetector.cpp new file mode 100644 index 0000000000000..35492a4a1108b --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Detection/UnitDetector.cpp @@ -0,0 +1,130 @@ +//===--------------------- UnitDetector.cpp - LLVM Advisor ----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the UnitDetector code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "UnitDetector.h" +#include "llvm/ADT/Hashing.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/Path.h" + +namespace llvm { +namespace advisor { + +UnitDetector::UnitDetector(const AdvisorConfig &config) : config_(config) {} + +llvm::Expected> +UnitDetector::detectUnits(llvm::StringRef compiler, + const llvm::SmallVectorImpl &args) { + + auto sources = findSourceFiles(args); + if (sources.empty()) { + return llvm::createStringError( + std::make_error_code(std::errc::invalid_argument), + "No source files found"); + } + + CompilationUnitInfo unit; + unit.name = generateUnitName(sources); + unit.sources = sources; + + // Store original args but filter out source files for the compile flags + for (const auto &arg : args) { + // Skip source files when adding to compile flags + llvm::StringRef extension = llvm::sys::path::extension(arg); + if (!arg.empty() && arg[0] != '-' && + (extension == ".c" || extension == ".cpp" || extension == ".cc" || + extension == ".cxx" || extension == ".C")) { + continue; + } + unit.compileFlags.push_back(arg); + } + + // Extract output files and features + extractBuildInfo(args, unit); + + return llvm::SmallVector{unit}; +} + +llvm::SmallVector UnitDetector::findSourceFiles( + const llvm::SmallVectorImpl &args) const { + llvm::SmallVector sources; + + for (const auto &arg : args) { + if (arg.empty() || arg[0] == '-') + continue; + + llvm::StringRef extension = llvm::sys::path::extension(arg); + if (extension == ".c" || extension == ".cpp" || extension == ".cc" || + extension == ".cxx" || extension == ".C") { + + SourceFile source; + source.path = arg; + source.language = classifier_.getLanguage(arg); + source.isHeader = false; + sources.push_back(source); + } + } + + return sources; +} + +void UnitDetector::extractBuildInfo( + const llvm::SmallVectorImpl &args, CompilationUnitInfo &unit) { + for (size_t i = 0; i < args.size(); ++i) { + const auto &arg = args[i]; + + if (arg == "-o" && i + 1 < args.size()) { + llvm::StringRef output = args[i + 1]; + llvm::StringRef ext = llvm::sys::path::extension(output); + if (ext == ".o") { + unit.outputObject = args[i + 1]; + } else { + unit.outputExecutable = args[i + 1]; + } + } + + llvm::StringRef argRef(arg); + if (argRef.contains("openmp") || argRef.contains("offload") || + argRef.contains("cuda")) { + unit.hasOffloading = true; + } + + if (llvm::StringRef(arg).starts_with("-march=")) { + unit.targetArch = arg.substr(7); + } + } +} + +std::string UnitDetector::generateUnitName( + const llvm::SmallVectorImpl &sources) const { + if (sources.empty()) + return "unknown"; + + // Use first source file name as base + std::string baseName = llvm::sys::path::stem(sources[0].path).str(); + + // Add hash for uniqueness when multiple sources + if (sources.size() > 1) { + std::string combined; + for (const auto &source : sources) { + combined += source.path; + } + auto hash = llvm::hash_value(combined); + baseName += "_" + std::to_string(static_cast(hash) % 10000); + } + + return baseName; +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Detection/UnitDetector.h b/llvm/tools/llvm-advisor/src/Detection/UnitDetector.h new file mode 100644 index 0000000000000..2c721726e14c5 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Detection/UnitDetector.h @@ -0,0 +1,50 @@ +//===------------------- UnitDetector.h - LLVM Advisor --------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the UnitDetector code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_DETECTION_UNITDETECTOR_H +#define LLVM_ADVISOR_DETECTION_UNITDETECTOR_H + +#include "../Config/AdvisorConfig.h" +#include "../Core/CompilationUnit.h" +#include "../Utils/FileClassifier.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/Error.h" +#include + +namespace llvm { +namespace advisor { + +class UnitDetector { +public: + explicit UnitDetector(const AdvisorConfig &config); + + llvm::Expected> + detectUnits(llvm::StringRef compiler, + const llvm::SmallVectorImpl &args); + +private: + llvm::SmallVector + findSourceFiles(const llvm::SmallVectorImpl &args) const; + void extractBuildInfo(const llvm::SmallVectorImpl &args, + CompilationUnitInfo &unit); + std::string + generateUnitName(const llvm::SmallVectorImpl &sources) const; + + const AdvisorConfig &config_; + FileClassifier classifier_; +}; + +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_DETECTION_UNITDETECTOR_H diff --git a/llvm/tools/llvm-advisor/src/Utils/FileClassifier.cpp b/llvm/tools/llvm-advisor/src/Utils/FileClassifier.cpp new file mode 100644 index 0000000000000..a5fbb6f059eb5 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/FileClassifier.cpp @@ -0,0 +1,151 @@ +//===------------------- FileClassifier.cpp - LLVM Advisor ----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the FileClassifier code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "FileClassifier.h" +#include "llvm/Support/Error.h" +#include "llvm/Support/Path.h" + +namespace llvm { +namespace advisor { + +FileClassification +FileClassifier::classifyFile(llvm::StringRef filePath) const { + StringRef filename = sys::path::filename(filePath); + StringRef extension = sys::path::extension(filePath); + + FileClassification classification; + classification.isGenerated = true; + classification.isTemporary = false; + + // LLVM IR files + if (extension == ".ll") { + classification.category = "ir"; + classification.description = "LLVM IR text"; + return classification; + } + + // Assembly files + if (extension == ".s" || extension == ".S") { + classification.category = "assembly"; + classification.description = "Assembly"; + return classification; + } + + // Optimization remarks + if (filename.ends_with(".opt.yaml") || filename.ends_with(".opt.yml")) { + classification.category = "remarks"; + classification.description = "Optimization remarks"; + return classification; + } + + // Preprocessed files + if (extension == ".i" || extension == ".ii") { + classification.category = "preprocessed"; + classification.description = "Preprocessed source"; + return classification; + } + + // AST dumps + if (extension == ".ast" || filename.contains("ast-dump")) { + classification.category = "ast"; + classification.description = "AST dump"; + return classification; + } + + // Profile data + if (extension == ".profraw" || extension == ".profdata") { + classification.category = "profile"; + classification.description = "Profile data"; + return classification; + } + + // Include trees + if (filename.contains(".include.") || filename.contains("include-tree")) { + classification.category = "include-tree"; + classification.description = "Include tree"; + return classification; + } + + // Debug info + if (filename.contains("debug") || filename.contains("dwarf")) { + classification.category = "debug"; + classification.description = "Debug information"; + return classification; + } + + // Static analyzer output + if (filename.contains("analysis") || filename.contains("analyzer")) { + classification.category = "static-analyzer"; + classification.description = "Static analyzer output"; + return classification; + } + + // Macro expansion + if (filename.contains("macro-expanded")) { + classification.category = "macro-expansion"; + classification.description = "Macro expansion"; + return classification; + } + + // Compilation phases + if (filename.contains("phases")) { + classification.category = "compilation-phases"; + classification.description = "Compilation phases"; + return classification; + } + + // Control flow graph + if (extension == ".dot" || filename.contains("cfg")) { + classification.category = "cfg"; + classification.description = "Control flow graph"; + return classification; + } + + // Template instantiation + if (filename.contains("template") || filename.contains("instantiation")) { + classification.category = "template-instantiation"; + classification.description = "Template instantiation"; + return classification; + } + + // Default for unknown files + classification.category = "unknown"; + classification.description = "Unknown file type"; + classification.isGenerated = false; + return classification; +} + +bool FileClassifier::shouldCollect(llvm::StringRef filePath) const { + auto classification = classifyFile(filePath); + return classification.category != "unknown" && classification.isGenerated && + !classification.isTemporary; +} + +std::string FileClassifier::getLanguage(llvm::StringRef filePath) const { + StringRef extension = sys::path::extension(filePath); + + if (extension == ".c") + return "C"; + if (extension == ".cpp" || extension == ".cc" || extension == ".cxx" || + extension == ".C") + return "C++"; + if (extension == ".h" || extension == ".hpp" || extension == ".hh" || + extension == ".hxx") + return "Header"; + + return "Unknown"; +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Utils/FileClassifier.h b/llvm/tools/llvm-advisor/src/Utils/FileClassifier.h new file mode 100644 index 0000000000000..d669a8c42324c --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/FileClassifier.h @@ -0,0 +1,40 @@ +//===----------------- FileClassifier.h - LLVM Advisor --------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the FileClassifier code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_FILE_CLASSIFIER_H +#define LLVM_ADVISOR_FILE_CLASSIFIER_H + +#include "llvm/ADT/StringRef.h" +#include + +namespace llvm { +namespace advisor { + +struct FileClassification { + std::string category; + std::string description; + bool isTemporary = false; + bool isGenerated = true; +}; + +class FileClassifier { +public: + FileClassification classifyFile(llvm::StringRef filePath) const; + bool shouldCollect(llvm::StringRef filePath) const; + std::string getLanguage(llvm::StringRef filePath) const; +}; + +} // namespace advisor +} // namespace llvm + +#endif diff --git a/llvm/tools/llvm-advisor/src/Utils/FileManager.cpp b/llvm/tools/llvm-advisor/src/Utils/FileManager.cpp new file mode 100644 index 0000000000000..5252c5adc0943 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/FileManager.cpp @@ -0,0 +1,215 @@ +//===------------------- FileManager.cpp - LLVM Advisor -------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the FileManager code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "FileManager.h" +#include "llvm/ADT/STLExtras.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/Error.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/Path.h" + +namespace llvm { +namespace advisor { + +Expected FileManager::createTempDir(llvm::StringRef prefix) { + SmallString<128> tempDirPath; + if (std::error_code ec = + sys::fs::createUniqueDirectory(prefix, tempDirPath)) { + return createStringError(ec, "Failed to create unique temporary directory"); + } + return tempDirPath.str().str(); +} + +Error FileManager::copyDirectory(llvm::StringRef source, llvm::StringRef dest) { + std::error_code EC; + + SmallString<128> sourcePathNorm(source); + // Remove trailing slash manually if present + if (sourcePathNorm.ends_with("/") && sourcePathNorm.size() > 1) { + sourcePathNorm.pop_back(); + } + + for (sys::fs::recursive_directory_iterator I(source, EC), E; I != E && !EC; + I.increment(EC)) { + StringRef currentPath = I->path(); + SmallString<128> destPath(dest); + + StringRef relativePath = currentPath; + if (!relativePath.consume_front(sourcePathNorm)) { + return createStringError( + std::make_error_code(std::errc::invalid_argument), + "Path '" + currentPath.str() + "' not in source dir '" + source + + "'"); + } + // Remove leading slash manually if present + if (relativePath.starts_with("/")) { + relativePath = relativePath.drop_front(1); + } + + sys::path::append(destPath, relativePath); + + if (sys::fs::is_directory(currentPath)) { + if (sys::fs::create_directories(destPath)) { + return createStringError(std::make_error_code(std::errc::io_error), + "Failed to create directory: " + + destPath.str().str()); + } + } else { + if (sys::fs::create_directories(sys::path::parent_path(destPath))) { + return createStringError(std::make_error_code(std::errc::io_error), + "Failed to create parent directory for: " + + destPath.str().str()); + } + if (sys::fs::copy_file(currentPath, destPath)) { + return createStringError(std::make_error_code(std::errc::io_error), + "Failed to copy file: " + currentPath.str()); + } + } + } + + if (EC) { + return createStringError(EC, "Failed to iterate directory: " + source); + } + + return Error::success(); +} + +Error FileManager::removeDirectory(llvm::StringRef path) { + if (!sys::fs::exists(path)) { + return Error::success(); + } + + std::error_code EC; + SmallVector Dirs; + for (sys::fs::recursive_directory_iterator I(path, EC), E; I != E && !EC; + I.increment(EC)) { + if (I->type() == sys::fs::file_type::directory_file) { + Dirs.push_back(I->path()); + } else { + if (auto E = sys::fs::remove(I->path())) { + return createStringError(E, "Failed to remove file: " + I->path()); + } + } + } + + if (EC) { + return createStringError(EC, "Error iterating directory " + path); + } + + for (const auto &Dir : llvm::reverse(Dirs)) { + if (auto E = sys::fs::remove(Dir)) { + return createStringError(E, "Failed to remove directory: " + Dir); + } + } + + if (auto E = sys::fs::remove(path)) { + return createStringError(E, + "Failed to remove top-level directory: " + path); + } + + return Error::success(); +} + +SmallVector FileManager::findFiles(llvm::StringRef directory, + llvm::StringRef pattern) { + SmallVector files; + std::error_code EC; + for (sys::fs::recursive_directory_iterator I(directory, EC), E; I != E && !EC; + I.increment(EC)) { + if (I->type() != sys::fs::file_type::directory_file) { + StringRef filename = sys::path::filename(I->path()); + if (filename.find(pattern) != StringRef::npos) { + files.push_back(I->path()); + } + } + } + return files; +} + +SmallVector FileManager::findFilesByExtension( + llvm::StringRef directory, const SmallVector &extensions) { + SmallVector files; + std::error_code EC; + for (sys::fs::recursive_directory_iterator I(directory, EC), E; I != E && !EC; + I.increment(EC)) { + if (I->type() != sys::fs::file_type::directory_file) { + StringRef filepath = I->path(); + for (const auto &ext : extensions) { + if (filepath.ends_with(ext)) { + files.push_back(filepath.str()); + break; + } + } + } + } + return files; +} + +Error FileManager::moveFile(llvm::StringRef source, llvm::StringRef dest) { + if (source == dest) { + return Error::success(); + } + + if (sys::fs::create_directories(sys::path::parent_path(dest))) { + return createStringError( + std::make_error_code(std::errc::io_error), + "Failed to create parent directory for destination: " + dest); + } + + if (sys::fs::rename(source, dest)) { + // If rename fails, try copy and remove + if (sys::fs::copy_file(source, dest)) { + return createStringError(std::make_error_code(std::errc::io_error), + "Failed to move file (copy failed): " + source); + } + if (sys::fs::remove(source)) { + return createStringError(std::make_error_code(std::errc::io_error), + "Failed to move file (source removal failed): " + + source); + } + } + + return Error::success(); +} + +Error FileManager::copyFile(llvm::StringRef source, llvm::StringRef dest) { + if (source == dest) { + return Error::success(); + } + + if (sys::fs::create_directories(sys::path::parent_path(dest))) { + return createStringError( + std::make_error_code(std::errc::io_error), + "Failed to create parent directory for destination: " + dest); + } + + if (sys::fs::copy_file(source, dest)) { + return createStringError(std::make_error_code(std::errc::io_error), + "Failed to copy file: " + source); + } + + return Error::success(); +} + +Expected FileManager::getFileSize(llvm::StringRef path) { + sys::fs::file_status status; + if (auto EC = sys::fs::status(path, status)) { + return createStringError(EC, "File not found: " + path); + } + + return status.getSize(); +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Utils/FileManager.h b/llvm/tools/llvm-advisor/src/Utils/FileManager.h new file mode 100644 index 0000000000000..9d52dc8485b1b --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/FileManager.h @@ -0,0 +1,59 @@ +//===-------------------- FileManager.h - LLVM Advisor --------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the FileManager code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_FILE_MANAGER_H +#define LLVM_ADVISOR_FILE_MANAGER_H + +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/Error.h" +#include + +namespace llvm { +namespace advisor { + +class FileManager { +public: + /// Create unique temporary directory with pattern llvm-advisor-xxxxx + static Expected + createTempDir(llvm::StringRef prefix = "llvm-advisor"); + + /// Recursively copy directory + static Error copyDirectory(llvm::StringRef source, llvm::StringRef dest); + + /// Remove directory and contents + static Error removeDirectory(llvm::StringRef path); + + /// Find files matching pattern + static llvm::SmallVector findFiles(llvm::StringRef directory, + llvm::StringRef pattern); + + /// Find files by extension + static llvm::SmallVector + findFilesByExtension(llvm::StringRef directory, + const llvm::SmallVector &extensions); + + /// Move file from source to destination + static Error moveFile(llvm::StringRef source, llvm::StringRef dest); + + /// Copy file from source to destination + static Error copyFile(llvm::StringRef source, llvm::StringRef dest); + + /// Get file size + static Expected getFileSize(llvm::StringRef path); +}; + +} // namespace advisor +} // namespace llvm + +#endif diff --git a/llvm/tools/llvm-advisor/src/Utils/ProcessRunner.cpp b/llvm/tools/llvm-advisor/src/Utils/ProcessRunner.cpp new file mode 100644 index 0000000000000..68ca1783ca6cf --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/ProcessRunner.cpp @@ -0,0 +1,158 @@ +//===-------------------- ProcessRunner.cpp - LLVM Advisor ----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the ProcessRunner code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "ProcessRunner.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/Support/Path.h" +#include "llvm/Support/Process.h" +#include "llvm/Support/Program.h" +#include + +extern char **environ; + +namespace llvm { +namespace advisor { + +Expected +ProcessRunner::run(llvm::StringRef program, + const llvm::SmallVector &args, + int timeoutSeconds) { + + auto programPath = sys::findProgramByName(program); + if (!programPath) { + return createStringError(programPath.getError(), + "Tool not found: " + program); + } + + llvm::SmallVector execArgs; + execArgs.push_back(program); + for (const auto &arg : args) { + execArgs.push_back(arg); + } + + SmallString<128> stdoutPath, stderrPath; + sys::fs::createTemporaryFile("stdout", "tmp", stdoutPath); + sys::fs::createTemporaryFile("stderr", "tmp", stderrPath); + + std::optional redirects[] = { + std::nullopt, // stdin + StringRef(stdoutPath), // stdout + StringRef(stderrPath) // stderr + }; + + int exitCode = sys::ExecuteAndWait(*programPath, execArgs, std::nullopt, + redirects, timeoutSeconds); + + ProcessResult result; + result.exitCode = exitCode; + result.executionTime = 0; // not tracking time + + auto stdoutBuffer = MemoryBuffer::getFile(stdoutPath); + if (stdoutBuffer) { + result.stdout = (*stdoutBuffer)->getBuffer().str(); + } + + auto stderrBuffer = MemoryBuffer::getFile(stderrPath); + if (stderrBuffer) { + result.stderr = (*stderrBuffer)->getBuffer().str(); + } + + sys::fs::remove(stdoutPath); + sys::fs::remove(stderrPath); + + return result; +} + +Expected ProcessRunner::runWithEnv( + llvm::StringRef program, const llvm::SmallVector &args, + const llvm::SmallVector &env, int timeoutSeconds) { + + // Prepare environment variables (current environment + our additions) + SmallVector environment; + + // Get current environment variables + char **envp = environ; + while (*envp) { + environment.emplace_back(*envp); + ++envp; + } + + // Add our additional environment variables + for (const auto &var : env) { + environment.emplace_back(var); + } + + // Prepare arguments + llvm::SmallVector argv; + argv.push_back(program); + for (const auto &arg : args) + argv.push_back(arg); + + // Create temporary files for stdout and stderr + SmallString<64> stdoutFile, stderrFile; + sys::fs::createTemporaryFile("stdout", "tmp", stdoutFile); + sys::fs::createTemporaryFile("stderr", "tmp", stderrFile); + + // Set up redirects + SmallVector, 3> redirects; + redirects.push_back(std::nullopt); // stdin + redirects.push_back(stdoutFile.str()); // stdout + redirects.push_back(stderrFile.str()); // stderr + + // Run the process with custom environment + bool executionFailed = false; + int status = sys::ExecuteAndWait(program, argv, + environment, // Custom environment + redirects, // Redirects + timeoutSeconds, // Timeout + 0, // Memory limit (no limit) + nullptr, // Standard output (using file) + &executionFailed, // Execution failed flag + nullptr // Process statistics + ); + + if (executionFailed) { + return createStringError(std::errc::no_such_file_or_directory, + "Failed to execute process"); + } + + // Read stdout and stderr from temporary files + std::string stdoutStr, stderrStr; + + auto stdoutBuffer = MemoryBuffer::getFile(stdoutFile); + if (stdoutBuffer) { + stdoutStr = stdoutBuffer.get()->getBuffer().str(); + } + + auto stderrBuffer = MemoryBuffer::getFile(stderrFile); + if (stderrBuffer) { + stderrStr = stderrBuffer.get()->getBuffer().str(); + } + + // Clean up temporary files + sys::fs::remove(stdoutFile); + sys::fs::remove(stderrFile); + + // Process result + ProcessResult result; + result.exitCode = status; + result.stdout = stdoutStr; + result.stderr = stderrStr; + result.executionTime = 0; // Not tracking time + return result; +} + +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Utils/ProcessRunner.h b/llvm/tools/llvm-advisor/src/Utils/ProcessRunner.h new file mode 100644 index 0000000000000..0182257c25534 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/ProcessRunner.h @@ -0,0 +1,45 @@ +//===------------------- ProcessRunner.h - LLVM Advisor -------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the ProcessRunner code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// +#ifndef LLVM_ADVISOR_PROCESS_RUNNER_H +#define LLVM_ADVISOR_PROCESS_RUNNER_H + +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/Error.h" +#include + +namespace llvm { +namespace advisor { + +class ProcessRunner { +public: + struct ProcessResult { + int exitCode; + std::string stdout; + std::string stderr; + double executionTime; + }; + + static Expected + run(llvm::StringRef program, const llvm::SmallVector &args, + int timeoutSeconds = 60); + + static Expected runWithEnv( + llvm::StringRef program, const llvm::SmallVector &args, + const llvm::SmallVector &env, int timeoutSeconds = 60); +}; + +} // namespace advisor +} // namespace llvm + +#endif diff --git a/llvm/tools/llvm-advisor/src/Utils/UnitMetadata.cpp b/llvm/tools/llvm-advisor/src/Utils/UnitMetadata.cpp new file mode 100644 index 0000000000000..97c1f2b1ce02f --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/UnitMetadata.cpp @@ -0,0 +1,367 @@ +//===-- UnitMetadata.cpp - Compilation Unit Metadata Management -*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This file implements the UnitMetadata class for tracking compilation unit +// metadata including timestamps, file counts, and processing status. +// +//===----------------------------------------------------------------------===// + +#include "UnitMetadata.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/Support/Path.h" +#include "llvm/Support/raw_ostream.h" +#include +#include +#include +#include + +using namespace llvm; + +namespace llvm { +namespace advisor { +namespace utils { + +UnitMetadata::UnitMetadata(StringRef OutputDirectory) + : OutputDirectory(OutputDirectory.str()) { + MetadataFilePath = getMetadataPath(); +} + +Error UnitMetadata::loadMetadata() { + if (!fileExists(MetadataFilePath)) { + return Error::success(); // No existing metadata, start fresh + } + + auto BufferOrError = MemoryBuffer::getFile(MetadataFilePath); + if (!BufferOrError) { + return createStringError(BufferOrError.getError(), + "Failed to open metadata file: " + + MetadataFilePath); + } + + return fromJson(BufferOrError->get()->getBuffer()); +} + +Error UnitMetadata::saveMetadata() { + // Ensure output directory exists + if (auto Err = createDirectoryIfNeeded(OutputDirectory)) { + return Err; + } + + auto JsonStr = toJson(); + if (!JsonStr) { + return JsonStr.takeError(); + } + + std::error_code EC; + raw_fd_ostream File(MetadataFilePath, EC); + if (EC) { + return createStringError(EC, "Failed to create metadata file: " + + MetadataFilePath); + } + + File << *JsonStr; + return Error::success(); +} + +void UnitMetadata::clear() { Units.clear(); } + +void UnitMetadata::registerUnit(StringRef UnitName) { + CompilationUnitInfo Info; + Info.Name = UnitName.str(); + Info.Timestamp = std::chrono::system_clock::now(); + Info.TotalFiles = 0; + Info.Status = "in_progress"; + Info.OutputPath = OutputDirectory + "/" + UnitName.str(); + + Units[UnitName.str()] = Info; +} + +void UnitMetadata::updateUnitStatus(StringRef UnitName, StringRef Status) { + auto it = Units.find(UnitName.str()); + if (it != Units.end()) { + it->second.Status = Status.str(); + // Update timestamp when status changes + it->second.Timestamp = std::chrono::system_clock::now(); + } +} + +void UnitMetadata::updateUnitFileCount(StringRef UnitName, size_t FileCount) { + auto it = Units.find(UnitName.str()); + if (it != Units.end()) { + it->second.TotalFiles = FileCount; + } +} + +void UnitMetadata::addArtifactType(StringRef UnitName, StringRef Type) { + auto it = Units.find(UnitName.str()); + if (it != Units.end()) { + auto &Types = it->second.ArtifactTypes; + if (std::find(Types.begin(), Types.end(), Type.str()) == Types.end()) { + Types.push_back(Type.str()); + } + } +} + +void UnitMetadata::setUnitProperty(StringRef UnitName, StringRef Key, + StringRef Value) { + auto it = Units.find(UnitName.str()); + if (it != Units.end()) { + it->second.Properties[Key.str()] = Value.str(); + } +} + +bool UnitMetadata::hasUnit(StringRef UnitName) const { + return Units.find(UnitName.str()) != Units.end(); +} + +Expected +UnitMetadata::getUnitInfo(StringRef UnitName) const { + auto it = Units.find(UnitName.str()); + if (it != Units.end()) { + return it->second; + } + return createStringError( + std::make_error_code(std::errc::no_such_file_or_directory), + "Unit not found: " + UnitName); +} + +std::vector UnitMetadata::getAllUnits() const { + std::vector Result; + for (const auto &Pair : Units) { + Result.push_back(Pair.second); + } + + // Sort by timestamp (most recent first) + std::sort(Result.begin(), Result.end(), + [](const CompilationUnitInfo &A, const CompilationUnitInfo &B) { + return A.Timestamp > B.Timestamp; + }); + + return Result; +} + +std::vector UnitMetadata::getRecentUnits(size_t Count) const { + auto AllUnits = getAllUnits(); + std::vector Result; + + size_t MaxCount = std::min(Count, AllUnits.size()); + for (size_t i = 0; i < MaxCount; ++i) { + Result.push_back(AllUnits[i].Name); + } + + return Result; +} + +Expected UnitMetadata::getMostRecentUnit() const { + auto RecentUnits = getRecentUnits(1); + if (RecentUnits.empty()) { + return createStringError( + std::make_error_code(std::errc::no_such_file_or_directory), + "No units found"); + } + return RecentUnits[0]; +} + +void UnitMetadata::removeUnit(StringRef UnitName) { + Units.erase(UnitName.str()); +} + +void UnitMetadata::cleanupOldUnits(int MaxAgeInDays) { + auto Now = std::chrono::system_clock::now(); + auto MaxAge = std::chrono::hours(24 * MaxAgeInDays); + + std::vector UnitsToRemove; + for (const auto &Pair : Units) { + if (Now - Pair.second.Timestamp > MaxAge) { + UnitsToRemove.push_back(Pair.first); + } + } + + for (const auto &UnitName : UnitsToRemove) { + removeUnit(UnitName); + } +} + +size_t UnitMetadata::getUnitCount() const { return Units.size(); } + +Expected UnitMetadata::toJson() const { + json::Object Root; + json::Array UnitsArray; + + for (const auto &Pair : Units) { + const auto &Info = Pair.second; + json::Object UnitObj; + + UnitObj["name"] = Info.Name; + UnitObj["timestamp"] = formatTimestamp(Info.Timestamp); + UnitObj["total_files"] = static_cast(Info.TotalFiles); + UnitObj["status"] = Info.Status; + UnitObj["output_path"] = Info.OutputPath; + + // Artifact types + json::Array ArtifactTypesArray; + for (const auto &Type : Info.ArtifactTypes) { + ArtifactTypesArray.push_back(Type); + } + UnitObj["artifact_types"] = std::move(ArtifactTypesArray); + + // Properties + json::Object PropertiesObj; + for (const auto &Prop : Info.Properties) { + PropertiesObj[Prop.first] = Prop.second; + } + UnitObj["properties"] = std::move(PropertiesObj); + + UnitsArray.push_back(std::move(UnitObj)); + } + + Root["units"] = std::move(UnitsArray); + + std::string Result; + raw_string_ostream OS(Result); + OS << json::Value(std::move(Root)); + return Result; +} + +Error UnitMetadata::fromJson(StringRef JsonStr) { + Units.clear(); + + Expected JsonOrError = json::parse(JsonStr); + if (!JsonOrError) { + return JsonOrError.takeError(); + } + + auto *Root = JsonOrError->getAsObject(); + if (!Root) { + return createStringError(std::make_error_code(std::errc::invalid_argument), + "Invalid JSON: root must be an object"); + } + + auto *UnitsArray = Root->getArray("units"); + if (!UnitsArray) { + return Error::success(); // No units section found, return success with + // empty units + } + + for (const auto &UnitValue : *UnitsArray) { + auto *UnitObj = UnitValue.getAsObject(); + if (!UnitObj) { + continue; // Skip invalid unit objects + } + + CompilationUnitInfo Info; + + // Parse name + if (auto NameOpt = UnitObj->getString("name")) { + Info.Name = NameOpt->str(); + } else { + continue; // Skip units without name + } + + // Parse timestamp + if (auto TimestampOpt = UnitObj->getString("timestamp")) { + auto TimestampOrError = parseTimestamp(*TimestampOpt); + if (TimestampOrError) { + Info.Timestamp = *TimestampOrError; + } else { + // Use current time if parsing fails + Info.Timestamp = std::chrono::system_clock::now(); + } + } else { + Info.Timestamp = std::chrono::system_clock::now(); + } + + // Parse total_files + if (auto FilesOpt = UnitObj->getInteger("total_files")) { + Info.TotalFiles = static_cast(*FilesOpt); + } else { + Info.TotalFiles = 0; + } + + // Parse status + if (auto StatusOpt = UnitObj->getString("status")) { + Info.Status = StatusOpt->str(); + } else { + Info.Status = "unknown"; + } + + // Parse output_path + if (auto PathOpt = UnitObj->getString("output_path")) { + Info.OutputPath = PathOpt->str(); + } else { + Info.OutputPath = OutputDirectory + "/" + Info.Name; + } + + // Parse artifact_types + if (auto TypesArray = UnitObj->getArray("artifact_types")) { + for (const auto &TypeValue : *TypesArray) { + if (auto TypeStr = TypeValue.getAsString()) { + Info.ArtifactTypes.push_back(TypeStr->str()); + } + } + } + + // Parse properties + if (auto PropertiesObj = UnitObj->getObject("properties")) { + for (const auto &Prop : *PropertiesObj) { + if (auto ValueStr = Prop.second.getAsString()) { + Info.Properties[Prop.first.str()] = ValueStr->str(); + } + } + } + + Units[Info.Name] = Info; + } + + return Error::success(); +} + +std::string UnitMetadata::getMetadataPath() const { + return OutputDirectory + "/.llvm-advisor-metadata.json"; +} + +std::string UnitMetadata::formatTimestamp( + const std::chrono::system_clock::time_point &TimePoint) const { + auto TimeT = std::chrono::system_clock::to_time_t(TimePoint); + std::stringstream ss; + ss << std::put_time(std::gmtime(&TimeT), "%Y-%m-%dT%H:%M:%SZ"); + return ss.str(); +} + +Expected +UnitMetadata::parseTimestamp(StringRef TimestampStr) const { + std::tm tm = {}; + std::istringstream ss(TimestampStr.str()); + ss >> std::get_time(&tm, "%Y-%m-%dT%H:%M:%SZ"); + + if (ss.fail()) { + return createStringError(std::make_error_code(std::errc::invalid_argument), + "Invalid timestamp format: " + TimestampStr); + } + + return std::chrono::system_clock::from_time_t(std::mktime(&tm)); +} + +bool UnitMetadata::fileExists(StringRef Path) const { + return sys::fs::exists(Path); +} + +Error UnitMetadata::createDirectoryIfNeeded(StringRef Path) const { + if (!sys::fs::exists(Path)) { + std::error_code EC = sys::fs::create_directories(Path); + if (EC) { + return createStringError(EC, "Error creating directory: " + Path); + } + } + return Error::success(); +} + +} // namespace utils +} // namespace advisor +} // namespace llvm diff --git a/llvm/tools/llvm-advisor/src/Utils/UnitMetadata.h b/llvm/tools/llvm-advisor/src/Utils/UnitMetadata.h new file mode 100644 index 0000000000000..f1e4ec6ea3585 --- /dev/null +++ b/llvm/tools/llvm-advisor/src/Utils/UnitMetadata.h @@ -0,0 +1,91 @@ +//===-- UnitMetadata.h - Compilation Unit Metadata Management --*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This file declares the UnitMetadata class for tracking compilation unit +// metadata including timestamps, file counts, and processing status. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_ADVISOR_UTILS_UNITMETADATA_H +#define LLVM_ADVISOR_UTILS_UNITMETADATA_H + +#include "llvm/Support/Error.h" +#include "llvm/Support/JSON.h" +#include "llvm/Support/raw_ostream.h" +#include +#include +#include +#include + +namespace llvm { +namespace advisor { +namespace utils { + +struct CompilationUnitInfo { + std::string Name; + std::chrono::system_clock::time_point Timestamp; + size_t TotalFiles; + std::vector ArtifactTypes; + std::string Status; // "in_progress", "completed", "failed" + std::string OutputPath; + std::map Properties; +}; + +class UnitMetadata { +public: + UnitMetadata(StringRef OutputDirectory); + ~UnitMetadata() = default; + + // Main operations + Error loadMetadata(); + Error saveMetadata(); + void clear(); + + // Unit management + void registerUnit(StringRef UnitName); + void updateUnitStatus(StringRef UnitName, StringRef Status); + void updateUnitFileCount(StringRef UnitName, size_t FileCount); + void addArtifactType(StringRef UnitName, StringRef Type); + void setUnitProperty(StringRef UnitName, StringRef Key, StringRef Value); + + // Query operations + bool hasUnit(StringRef UnitName) const; + Expected getUnitInfo(StringRef UnitName) const; + std::vector getAllUnits() const; + std::vector getRecentUnits(size_t Count = 5) const; + Expected getMostRecentUnit() const; + + // Utility operations + void removeUnit(StringRef UnitName); + void cleanupOldUnits(int MaxAgeInDays = 30); + size_t getUnitCount() const; + + // JSON operations + Expected toJson() const; + Error fromJson(StringRef JsonStr); + +private: + std::string OutputDirectory; + std::string MetadataFilePath; + std::map Units; + + // Helper methods + std::string getMetadataPath() const; + std::string + formatTimestamp(const std::chrono::system_clock::time_point &TimePoint) const; + Expected + parseTimestamp(StringRef TimestampStr) const; + bool fileExists(StringRef Path) const; + Error createDirectoryIfNeeded(StringRef Path) const; +}; + +} // namespace utils +} // namespace advisor +} // namespace llvm + +#endif // LLVM_ADVISOR_UTILS_UNITMETADATA_H diff --git a/llvm/tools/llvm-advisor/src/llvm-advisor.cpp b/llvm/tools/llvm-advisor/src/llvm-advisor.cpp new file mode 100644 index 0000000000000..4adc2cb059dfb --- /dev/null +++ b/llvm/tools/llvm-advisor/src/llvm-advisor.cpp @@ -0,0 +1,206 @@ +//===-------------- llvm-advisor.cpp - LLVM Advisor -----------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This is the llvm-advisor code generator driver. It provides a convenient +// command-line interface for generating an assembly file or a relocatable file, +// given LLVM bitcode. +// +//===----------------------------------------------------------------------===// + +#include "Config/AdvisorConfig.h" +#include "Core/CompilationManager.h" +#include "Core/ViewerLauncher.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/CommandLine.h" +#include "llvm/Support/FileSystem.h" +#include "llvm/Support/InitLLVM.h" +#include "llvm/Support/Path.h" +#include "llvm/Support/raw_ostream.h" + +static llvm::cl::opt + ConfigFile("config", llvm::cl::desc("Configuration file"), + llvm::cl::value_desc("filename")); +static llvm::cl::opt OutputDir("output-dir", + llvm::cl::desc("Output directory"), + llvm::cl::value_desc("directory")); +static llvm::cl::opt Verbose("verbose", llvm::cl::desc("Verbose output")); +static llvm::cl::opt KeepTemps("keep-temps", + llvm::cl::desc("Keep temporary files")); +static llvm::cl::opt NoProfiler("no-profiler", + llvm::cl::desc("Disable profiler")); +static llvm::cl::opt + Port("port", llvm::cl::desc("Web server port (for view command)"), + llvm::cl::value_desc("port"), llvm::cl::init(8000)); + +int main(int argc, char **argv) { + llvm::InitLLVM X(argc, argv); + + // Handle help and subcommands before argument parsing + if (argc > 1) { + llvm::StringRef firstArg(argv[1]); + if (firstArg == "--help" || firstArg == "-h") { + llvm::outs() << "LLVM Advisor - Compilation analysis tool\n\n"; + llvm::outs() << "Usage:\n"; + llvm::outs() << " llvm-advisor [options] [compiler-args...] " + " - Compile with data collection\n"; + llvm::outs() << " llvm-advisor view [options] " + "[compiler-args...] - Compile and launch web viewer\n\n"; + llvm::outs() << "Examples:\n"; + llvm::outs() << " llvm-advisor clang -O2 -g main.c\n"; + llvm::outs() << " llvm-advisor view --port 8080 clang++ -O3 app.cpp\n\n"; + llvm::outs() << "Options:\n"; + llvm::outs() << " --config Configuration file\n"; + llvm::outs() << " --output-dir Output directory (default: " + ".llvm-advisor)\n"; + llvm::outs() << " --verbose Verbose output\n"; + llvm::outs() << " --keep-temps Keep temporary files\n"; + llvm::outs() << " --no-profiler Disable profiler\n"; + llvm::outs() << " --port Web server port for view command " + "(default: 8000)\n"; + return 0; + } + } + + // Check for 'view' subcommand + bool isViewCommand = false; + int argOffset = 0; + + if (argc > 1 && llvm::StringRef(argv[1]) == "view") { + isViewCommand = true; + argOffset = 1; + } + + // Parse llvm-advisor options until we find the compiler + llvm::SmallVector advisorArgs; + advisorArgs.push_back(argv[0]); + + int compilerArgStart = 1 + argOffset; + bool foundCompiler = false; + + for (int i = 1 + argOffset; i < argc; ++i) { + llvm::StringRef arg(argv[i]); + if (arg.starts_with("--") || + (arg.starts_with("-") && arg.size() > 1 && arg != "-")) { + advisorArgs.push_back(argv[i]); + if (arg == "--config" || arg == "--output-dir" || arg == "--port") { + if (i + 1 < argc && !llvm::StringRef(argv[i + 1]).starts_with("-")) { + advisorArgs.push_back(argv[++i]); + } + } + } else { + compilerArgStart = i; + foundCompiler = true; + break; + } + } + + if (!foundCompiler) { + llvm::errs() << "Error: No compiler command provided.\n"; + if (isViewCommand) { + llvm::errs() << "Usage: llvm-advisor view [options] " + "[compiler-args...]\n"; + } else { + llvm::errs() + << "Usage: llvm-advisor [options] [compiler-args...]\n"; + } + return 1; + } + + // Parse llvm-advisor options + int advisorArgc = static_cast(advisorArgs.size()); + llvm::cl::ParseCommandLineOptions(advisorArgc, + const_cast(advisorArgs.data()), + "LLVM Compilation Advisor"); + + // Extract compiler and arguments + std::string compiler = argv[compilerArgStart]; + llvm::SmallVector compilerArgs; + for (int i = compilerArgStart + 1; i < argc; ++i) { + compilerArgs.push_back(argv[i]); + } + + // Configure advisor + llvm::advisor::AdvisorConfig config; + if (!ConfigFile.empty()) { + if (auto Err = config.loadFromFile(ConfigFile).takeError()) { + llvm::errs() << "Error loading config: " << llvm::toString(std::move(Err)) + << "\n"; + return 1; + } + } + + if (!OutputDir.empty()) { + config.setOutputDir(OutputDir); + } else { + config.setOutputDir(".llvm-advisor"); // Default hidden directory + } + + config.setVerbose(Verbose); + config.setKeepTemps(KeepTemps || + isViewCommand); // Keep temps for view command + config.setRunProfiler(!NoProfiler); + + // Create output directory + if (auto EC = llvm::sys::fs::create_directories(config.getOutputDir())) { + llvm::errs() << "Error creating output directory: " << EC.message() << "\n"; + return 1; + } + + if (config.getVerbose()) { + llvm::outs() << "LLVM Compilation Advisor\n"; + llvm::outs() << "Compiler: " << compiler << "\n"; + llvm::outs() << "Output: " << config.getOutputDir() << "\n"; + if (isViewCommand) { + llvm::outs() << "Mode: Compile and launch web viewer\n"; + } + } + + // Execute with data collection + llvm::advisor::CompilationManager manager(config); + auto result = manager.executeWithDataCollection(compiler, compilerArgs); + + if (!result) { + llvm::errs() << "Error: " << llvm::toString(result.takeError()) << "\n"; + return 1; + } + + if (config.getVerbose()) { + llvm::outs() << "Compilation completed (exit code: " << *result << ")\n"; + } + + // If this is a view command and compilation succeeded, launch the web viewer + if (isViewCommand && *result == 0) { + if (config.getVerbose()) { + llvm::outs() << "Launching web viewer...\n"; + } + + // Convert output directory to absolute path for web viewer + llvm::SmallString<256> absoluteOutputDir; + if (llvm::sys::path::is_absolute(config.getOutputDir())) { + absoluteOutputDir = config.getOutputDir(); + } else { + llvm::sys::fs::current_path(absoluteOutputDir); + llvm::sys::path::append(absoluteOutputDir, config.getOutputDir()); + } + + auto viewerResult = llvm::advisor::ViewerLauncher::launch( + absoluteOutputDir.str().str(), Port); + if (!viewerResult) { + llvm::errs() << "Error launching web viewer: " + << llvm::toString(viewerResult.takeError()) << "\n"; + llvm::errs() << "Compilation data is still available in: " + << config.getOutputDir() << "\n"; + return 1; + } + + return *viewerResult; + } + + return *result; +} diff --git a/llvm/tools/llvm-advisor/tools/__init__.py b/llvm/tools/llvm-advisor/tools/__init__.py new file mode 100644 index 0000000000000..0e977a146fa04 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/__init__.py @@ -0,0 +1,7 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// diff --git a/llvm/tools/llvm-advisor/tools/common/__init__.py b/llvm/tools/llvm-advisor/tools/common/__init__.py new file mode 100644 index 0000000000000..0e977a146fa04 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/__init__.py @@ -0,0 +1,7 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// diff --git a/llvm/tools/llvm-advisor/tools/common/collector.py b/llvm/tools/llvm-advisor/tools/common/collector.py new file mode 100644 index 0000000000000..420dbb89bdf43 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/collector.py @@ -0,0 +1,268 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// +# +# This is the artifact collector module. It provides logic for discovering and +# parsing build artifacts for LLVM Advisor analysis. +# +# ===----------------------------------------------------------------------===# + +import os +from typing import Dict, List, Any, Optional +from pathlib import Path + +from .models import FileType, CompilationUnit, ParsedFile +from .parsers import ( + RemarksParser, + TimeTraceParser, + DiagnosticsParser, + ASTParser, + PGOProfileParser, + XRayParser, + StaticAnalyzerParser, + IRParser, + ObjdumpParser, + IncludeTreeParser, + AssemblyParser, + PreprocessedParser, + SARIFParser, + MacroExpansionParser, + DependenciesParser, + BinarySizeParser, + DebugParser, + SymbolsParser, + RuntimeTraceParser, + CompilationPhasesParser, + FTimeReportParser, + VersionInfoParser, + PreprocessedParser as SourcesParser, # Reuse for simple text files +) + + +class ArtifactCollector: + def __init__(self): + self.parsers = { + FileType.REMARKS: RemarksParser(), + FileType.TIME_TRACE: TimeTraceParser(), + FileType.DIAGNOSTICS: DiagnosticsParser(), + FileType.AST_JSON: ASTParser(), + FileType.PGO_PROFILE: PGOProfileParser(), + FileType.XRAY: XRayParser(), + FileType.STATIC_ANALYZER: StaticAnalyzerParser(), + FileType.IR: IRParser(), + FileType.OBJDUMP: ObjdumpParser(), + FileType.INCLUDE_TREE: IncludeTreeParser(), + FileType.ASSEMBLY: AssemblyParser(), + FileType.PREPROCESSED: PreprocessedParser(), + FileType.STATIC_ANALYSIS_SARIF: SARIFParser(), + FileType.MACRO_EXPANSION: MacroExpansionParser(), + FileType.DEPENDENCIES: DependenciesParser(), + FileType.BINARY_SIZE: BinarySizeParser(), + FileType.DEBUG: DebugParser(), + FileType.SYMBOLS: SymbolsParser(), + FileType.RUNTIME_TRACE: RuntimeTraceParser(), + FileType.COMPILATION_PHASES: CompilationPhasesParser(), + FileType.FTIME_REPORT: FTimeReportParser(), + FileType.VERSION_INFO: VersionInfoParser(), + FileType.SOURCES: SourcesParser(), + } + + # Map directory names to file types + self.dir_to_type = { + "remarks": FileType.REMARKS, + "time-trace": FileType.TIME_TRACE, + "diagnostics": FileType.DIAGNOSTICS, + "ast-json": FileType.AST_JSON, + "pgo-profile": FileType.PGO_PROFILE, + "xray": FileType.XRAY, + "static-analyzer": FileType.STATIC_ANALYZER, + "ir": FileType.IR, + "objdump": FileType.OBJDUMP, + "include-tree": FileType.INCLUDE_TREE, + "assembly": FileType.ASSEMBLY, + "preprocessed": FileType.PREPROCESSED, + "static-analysis-sarif": FileType.STATIC_ANALYSIS_SARIF, + "macro-expansion": FileType.MACRO_EXPANSION, + "dependencies": FileType.DEPENDENCIES, + "binary-size": FileType.BINARY_SIZE, + "debug": FileType.DEBUG, + "symbols": FileType.SYMBOLS, + "runtime-trace": FileType.RUNTIME_TRACE, + "compilation-phases": FileType.COMPILATION_PHASES, + "ftime-report": FileType.FTIME_REPORT, + "version-info": FileType.VERSION_INFO, + "sources": FileType.SOURCES, + } + + def discover_compilation_units(self, advisor_dir: str) -> List[CompilationUnit]: + """Discover all compilation units in the .llvm-advisor directory.""" + compilation_units = [] + advisor_path = Path(advisor_dir) + + if not advisor_path.exists(): + return compilation_units + + # Each subdirectory represents a compilation unit + for unit_dir in advisor_path.iterdir(): + if not unit_dir.is_dir(): + continue + + # Check if this is the new nested structure or old flat structure + units = self._scan_compilation_unit_with_runs(unit_dir) + compilation_units.extend(units) + + return compilation_units + + def _scan_compilation_unit_with_runs(self, unit_dir: Path) -> List[CompilationUnit]: + """Scan a compilation unit directory that contains timestamped runs.""" + units = [] + + # unit_dir contains timestamped run directories + run_dirs = [] + for item in unit_dir.iterdir(): + if item.is_dir() and item.name.startswith(unit_dir.name + "_"): + run_dirs.append(item) + + if not run_dirs: + # No timestamped runs found, skip this unit + return units + + # Sort by timestamp (newest first) + run_dirs.sort(key=lambda x: x.name, reverse=True) + + # Use the most recent run + latest_run = run_dirs[0] + unit = self._scan_single_run(latest_run, unit_dir.name) + if unit: + # Store run timestamp info in metadata + unit.metadata = getattr(unit, "metadata", {}) + unit.metadata["run_timestamp"] = latest_run.name.split("_", 1)[-1] + unit.metadata["run_path"] = str(latest_run) + unit.metadata["available_runs"] = [r.name for r in run_dirs] + units.append(unit) + + return units + + def _scan_single_run( + self, run_dir: Path, unit_name: str + ) -> Optional[CompilationUnit]: + """Scan a single run directory for artifacts.""" + artifacts = {} + + # Scan each artifact type directory + for artifact_dir in run_dir.iterdir(): + if not artifact_dir.is_dir(): + continue + + dir_name = artifact_dir.name + if dir_name not in self.dir_to_type: + continue + + file_type = self.dir_to_type[dir_name] + artifact_files = [] + + # Collect all files in this artifact directory + for file_path in artifact_dir.rglob("*"): + if file_path.is_file(): + artifact_files.append(str(file_path)) + + if artifact_files: + artifacts[file_type] = artifact_files + + if artifacts: + return CompilationUnit( + name=unit_name, path=str(run_dir), artifacts=artifacts + ) + + return None + + def parse_compilation_unit( + self, unit: CompilationUnit + ) -> Dict[FileType, List[ParsedFile]]: + """Parse all artifacts for a compilation unit.""" + parsed_artifacts = {} + + for file_type, file_paths in unit.artifacts.items(): + if file_type not in self.parsers: + continue + + parser = self.parsers[file_type] + parsed_files = [] + + for file_path in file_paths: + try: + if parser.can_parse(file_path): + parsed_file = parser.parse(file_path) + parsed_files.append(parsed_file) + except Exception as e: + # Create error entry for failed parsing + error_file = ParsedFile( + file_type=file_type, + file_path=file_path, + data={}, + metadata={"error": f"Failed to parse: {str(e)}"}, + ) + parsed_files.append(error_file) + + if parsed_files: + parsed_artifacts[file_type] = parsed_files + + return parsed_artifacts + + def parse_all_units( + self, advisor_dir: str + ) -> Dict[str, Dict[FileType, List[ParsedFile]]]: + """Parse all compilation units in the advisor directory.""" + units = self.discover_compilation_units(advisor_dir) + parsed_units = {} + + for unit in units: + parsed_artifacts = self.parse_compilation_unit(unit) + if parsed_artifacts: + parsed_units[unit.name] = parsed_artifacts + + return parsed_units + + def get_summary_statistics( + self, parsed_units: Dict[str, Dict[FileType, List[ParsedFile]]] + ) -> Dict[str, Any]: + """Generate summary statistics for all parsed data.""" + stats = { + "total_units": len(parsed_units), + "total_files": 0, + "file_types": {}, + "errors": 0, + "units": {}, + } + + for unit_name, artifacts in parsed_units.items(): + unit_stats = {"file_types": {}, "total_files": 0, "errors": 0} + + for file_type, parsed_files in artifacts.items(): + type_name = file_type.value + file_count = len(parsed_files) + error_count = sum(1 for f in parsed_files if "error" in f.metadata) + + unit_stats["file_types"][type_name] = { + "count": file_count, + "errors": error_count, + } + unit_stats["total_files"] += file_count + unit_stats["errors"] += error_count + + # Update global stats + if type_name not in stats["file_types"]: + stats["file_types"][type_name] = {"count": 0, "errors": 0} + + stats["file_types"][type_name]["count"] += file_count + stats["file_types"][type_name]["errors"] += error_count + + stats["units"][unit_name] = unit_stats + stats["total_files"] += unit_stats["total_files"] + stats["errors"] += unit_stats["errors"] + + return stats diff --git a/llvm/tools/llvm-advisor/tools/common/models.py b/llvm/tools/llvm-advisor/tools/common/models.py new file mode 100644 index 0000000000000..97cdbe6fdefa6 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/models.py @@ -0,0 +1,116 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from dataclasses import dataclass +from typing import Dict, List, Any, Optional +from enum import Enum + + +class FileType(Enum): + REMARKS = "remarks" + TIME_TRACE = "time-trace" + DIAGNOSTICS = "diagnostics" + AST_JSON = "ast-json" + PGO_PROFILE = "pgo-profile" + XRAY = "xray" + STATIC_ANALYZER = "static-analyzer" + IR = "ir" + OBJDUMP = "objdump" + INCLUDE_TREE = "include-tree" + ASSEMBLY = "assembly" + PREPROCESSED = "preprocessed" + STATIC_ANALYSIS_SARIF = "static-analysis-sarif" + MACRO_EXPANSION = "macro-expansion" + DEPENDENCIES = "dependencies" + BINARY_SIZE = "binary-size" + DEBUG = "debug" + SYMBOLS = "symbols" + RUNTIME_TRACE = "runtime-trace" + COMPILATION_PHASES = "compilation-phases" + FTIME_REPORT = "ftime-report" + VERSION_INFO = "version-info" + SOURCES = "sources" + + +@dataclass +class SourceLocation: + file: Optional[str] = None + line: Optional[int] = None + column: Optional[int] = None + + +@dataclass +class CompilationUnit: + name: str + path: str + artifacts: Dict[FileType, List[str]] + metadata: Dict[str, Any] = None + + def __post_init__(self): + if self.metadata is None: + self.metadata = {} + + +@dataclass +class ParsedFile: + file_type: FileType + file_path: str + data: Any + metadata: Dict[str, Any] + + +@dataclass +class Diagnostic: + level: str + message: str + location: Optional[SourceLocation] = None + code: Optional[str] = None + + +@dataclass +class Remark: + pass_name: str + function: str + message: str + location: Optional[SourceLocation] = None + args: Dict[str, Any] = None + + +@dataclass +class TraceEvent: + name: str + category: str + phase: str + timestamp: int + duration: Optional[int] = None + pid: Optional[int] = None + tid: Optional[int] = None + args: Dict[str, Any] = None + + +@dataclass +class Symbol: + name: str + address: Optional[str] = None + size: Optional[int] = None + type: Optional[str] = None + section: Optional[str] = None + + +@dataclass +class BinarySize: + section: str + size: int + percentage: Optional[float] = None + + +@dataclass +class Dependency: + source: str + target: str + type: Optional[str] = None diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/__init__.py b/llvm/tools/llvm-advisor/tools/common/parsers/__init__.py new file mode 100644 index 0000000000000..589e854df39f0 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/__init__.py @@ -0,0 +1,57 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from .base_parser import BaseParser +from .remarks_parser import RemarksParser +from .time_trace_parser import TimeTraceParser +from .diagnostics_parser import DiagnosticsParser +from .ast_parser import ASTParser +from .pgo_profile_parser import PGOProfileParser +from .xray_parser import XRayParser +from .static_analyzer_parser import StaticAnalyzerParser +from .ir_parser import IRParser +from .objdump_parser import ObjdumpParser +from .include_tree_parser import IncludeTreeParser +from .assembly_parser import AssemblyParser +from .preprocessed_parser import PreprocessedParser +from .sarif_parser import SARIFParser +from .macro_expansion_parser import MacroExpansionParser +from .dependencies_parser import DependenciesParser +from .binary_size_parser import BinarySizeParser +from .debug_parser import DebugParser +from .symbols_parser import SymbolsParser +from .runtime_trace_parser import RuntimeTraceParser +from .compilation_phases_parser import CompilationPhasesParser +from .ftime_report_parser import FTimeReportParser +from .version_info_parser import VersionInfoParser + +__all__ = [ + "BaseParser", + "RemarksParser", + "TimeTraceParser", + "DiagnosticsParser", + "ASTParser", + "PGOProfileParser", + "XRayParser", + "StaticAnalyzerParser", + "IRParser", + "ObjdumpParser", + "IncludeTreeParser", + "AssemblyParser", + "PreprocessedParser", + "SARIFParser", + "MacroExpansionParser", + "DependenciesParser", + "BinarySizeParser", + "DebugParser", + "SymbolsParser", + "RuntimeTraceParser", + "CompilationPhasesParser", + "FTimeReportParser", + "VersionInfoParser", +] diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/assembly_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/assembly_parser.py new file mode 100644 index 0000000000000..ab14b1a2d1b3c --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/assembly_parser.py @@ -0,0 +1,138 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class AssemblyParser(BaseParser): + def __init__(self): + super().__init__(FileType.ASSEMBLY) + self.label_pattern = re.compile(r"^(\w+):") + self.instruction_pattern = re.compile(r"^\s+(\w+)") + self.section_pattern = re.compile(r"^\s*\.(text|data|bss|rodata)") + + def parse(self, file_path: str) -> ParsedFile: + if self.is_large_file(file_path): + return self._parse_large_assembly(file_path) + + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + asm_data = self._analyze_assembly_content(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_lines": len(lines), + **asm_data["summary"], + } + + return self.create_parsed_file(file_path, asm_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _parse_large_assembly(self, file_path: str) -> ParsedFile: + try: + asm_data = {"labels": [], "instructions": {}, "sections": [], "summary": {}} + line_count = 0 + + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + for line in f: + line_count += 1 + + # Only parse first 5000 lines for large files + if line_count > 5000: + break + + line = line.strip() + if not line or line.startswith("#") or line.startswith(";"): + continue + + # Parse labels + label_match = self.label_pattern.match(line) + if label_match: + asm_data["labels"].append(label_match.group(1)) + + # Parse instructions + inst_match = self.instruction_pattern.match(line) + if inst_match: + inst = inst_match.group(1) + asm_data["instructions"][inst] = ( + asm_data["instructions"].get(inst, 0) + 1 + ) + + # Parse sections + section_match = self.section_pattern.match(line) + if section_match: + asm_data["sections"].append(section_match.group(1)) + + asm_data["summary"] = { + "label_count": len(asm_data["labels"]), + "instruction_types": len(asm_data["instructions"]), + "total_instructions": sum(asm_data["instructions"].values()), + "section_count": len(set(asm_data["sections"])), + "analyzed_lines": line_count, + "is_partial": True, + } + + metadata = { + "file_size": self.get_file_size(file_path), + **asm_data["summary"], + } + + return self.create_parsed_file(file_path, asm_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _analyze_assembly_content(self, lines: List[str]) -> Dict[str, Any]: + asm_data = {"labels": [], "instructions": {}, "sections": [], "summary": {}} + + for line in lines: + original_line = line + line = line.strip() + + if not line or line.startswith("#") or line.startswith(";"): + continue + + # Parse labels + label_match = self.label_pattern.match(line) + if label_match: + asm_data["labels"].append(label_match.group(1)) + continue + + # Parse instructions + inst_match = self.instruction_pattern.match(original_line) + if inst_match: + inst = inst_match.group(1) + asm_data["instructions"][inst] = ( + asm_data["instructions"].get(inst, 0) + 1 + ) + continue + + # Parse sections + section_match = self.section_pattern.match(line) + if section_match: + asm_data["sections"].append(section_match.group(1)) + + asm_data["summary"] = { + "label_count": len(asm_data["labels"]), + "instruction_types": len(asm_data["instructions"]), + "total_instructions": sum(asm_data["instructions"].values()), + "section_count": len(set(asm_data["sections"])), + } + + return asm_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/ast_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/ast_parser.py new file mode 100644 index 0000000000000..b7f2d816c9ce4 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/ast_parser.py @@ -0,0 +1,111 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import json +from typing import Dict, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class ASTParser(BaseParser): + def __init__(self): + super().__init__(FileType.AST_JSON) + + def parse(self, file_path: str) -> ParsedFile: + if self.is_large_file(file_path): + return self._parse_large_ast(file_path) + + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + ast_data = json.loads(content) + + # Extract summary information + summary = self._extract_ast_summary(ast_data) + + metadata = { + "file_size": self.get_file_size(file_path), + "ast_summary": summary, + } + + return self.create_parsed_file(file_path, ast_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _parse_large_ast(self, file_path: str) -> ParsedFile: + try: + # For large AST files, just extract basic info + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + # Read first chunk to get basic structure + chunk = f.read(10000) # 10KB + + # Try to parse at least the root node + if chunk.startswith("{"): + bracket_count = 0 + for i, char in enumerate(chunk): + if char == "{": + bracket_count += 1 + elif char == "}": + bracket_count -= 1 + if bracket_count == 0: + try: + partial_data = json.loads(chunk[: i + 1]) + summary = self._extract_ast_summary( + partial_data, partial=True + ) + + metadata = { + "file_size": self.get_file_size(file_path), + "ast_summary": summary, + "is_partial": True, + } + + return self.create_parsed_file( + file_path, partial_data, metadata + ) + except: + break + + metadata = { + "file_size": self.get_file_size(file_path), + "error": "File too large to parse completely", + } + + return self.create_parsed_file(file_path, {}, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _extract_ast_summary( + self, ast_data: Dict[str, Any], partial: bool = False + ) -> Dict[str, Any]: + summary = { + "root_kind": ast_data.get("kind", "unknown"), + "root_id": ast_data.get("id", "unknown"), + "has_inner": "inner" in ast_data, + "is_partial": partial, + } + + if "inner" in ast_data and isinstance(ast_data["inner"], list): + summary["inner_count"] = len(ast_data["inner"]) + + # Count node types + node_types = {} + for node in ast_data["inner"]: + if isinstance(node, dict) and "kind" in node: + kind = node["kind"] + node_types[kind] = node_types.get(kind, 0) + 1 + + summary["node_types"] = node_types + + return summary diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/base_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/base_parser.py new file mode 100644 index 0000000000000..ac1b29756665e --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/base_parser.py @@ -0,0 +1,62 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from abc import ABC, abstractmethod +from typing import Any, Dict, List, Optional +import os +from ..models import ParsedFile, FileType + + +class BaseParser(ABC): + def __init__(self, file_type: FileType): + self.file_type = file_type + + @abstractmethod + def parse(self, file_path: str) -> ParsedFile: + pass + + def can_parse(self, file_path: str) -> bool: + return os.path.exists(file_path) and os.path.isfile(file_path) + + def get_file_size(self, file_path: str) -> int: + return os.path.getsize(file_path) if os.path.exists(file_path) else 0 + + def is_large_file(self, file_path: str, threshold: int = 100 * 1024 * 1024) -> bool: + return self.get_file_size(file_path) > threshold + + def read_file_safe( + self, file_path: str, max_size: int = 100 * 1024 * 1024 + ) -> Optional[str]: + try: + if self.is_large_file(file_path, max_size): + return None + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + return f.read() + except Exception: + return None + + def read_file_chunked(self, file_path: str, chunk_size: int = 1024 * 1024): + try: + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + while True: + chunk = f.read(chunk_size) + if not chunk: + break + yield chunk + except Exception: + return + + def create_parsed_file( + self, file_path: str, data: Any, metadata: Dict[str, Any] = None + ) -> ParsedFile: + return ParsedFile( + file_type=self.file_type, + file_path=file_path, + data=data, + metadata=metadata or {}, + ) diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/binary_size_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/binary_size_parser.py new file mode 100644 index 0000000000000..7fb2cc6f33aa7 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/binary_size_parser.py @@ -0,0 +1,109 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, BinarySize + + +class BinarySizeParser(BaseParser): + def __init__(self): + super().__init__(FileType.BINARY_SIZE) + # Pattern for size output like: "1234 5678 90 12345 section_name" + self.size_pattern = re.compile(r"^\s*(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(.+)$") + # Pattern for nm-style output with size + self.nm_pattern = re.compile( + r"^([0-9a-fA-F]+)\s+([0-9a-fA-F]+)\s+([A-Za-z])\s+(.+)$" + ) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + size_data = self._parse_size_output(lines) + + total_size = sum(item.size for item in size_data) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_sections": len(size_data), + "total_binary_size": total_size, + } + + return self.create_parsed_file(file_path, size_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_size_output(self, lines: List[str]) -> List[BinarySize]: + size_data = [] + total_size = 0 + + for line in lines: + line = line.strip() + if not line or line.startswith("#"): + continue + + # Try standard size format first + size_match = self.size_pattern.match(line) + if size_match: + text_size = int(size_match.group(1)) + data_size = int(size_match.group(2)) + bss_size = int(size_match.group(3)) + total = int(size_match.group(4)) + name = size_match.group(5) + + # Add individual sections if they have non-zero sizes + if text_size > 0: + size_data.append(BinarySize(section=f"{name}.text", size=text_size)) + if data_size > 0: + size_data.append(BinarySize(section=f"{name}.data", size=data_size)) + if bss_size > 0: + size_data.append(BinarySize(section=f"{name}.bss", size=bss_size)) + + total_size += total + continue + + # Try nm-style format + nm_match = self.nm_pattern.match(line) + if nm_match: + address = nm_match.group(1) + size = int(nm_match.group(2), 16) + symbol_type = nm_match.group(3) + name = nm_match.group(4) + + size_data.append(BinarySize(section=name, size=size)) + total_size += size + continue + + # Try to parse generic "section: size" format + if ":" in line: + parts = line.split(":", 1) + if len(parts) == 2: + section_name = parts[0].strip() + size_part = parts[1].strip() + + # Extract numeric size + size_numbers = re.findall(r"\d+", size_part) + if size_numbers: + size = int(size_numbers[0]) + size_data.append(BinarySize(section=section_name, size=size)) + total_size += size + + # Calculate percentages + if total_size > 0: + for item in size_data: + item.percentage = (item.size / total_size) * 100 + + return size_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/compilation_phases_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/compilation_phases_parser.py new file mode 100644 index 0000000000000..b56f95add194c --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/compilation_phases_parser.py @@ -0,0 +1,225 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class CompilationPhasesParser(BaseParser): + def __init__(self): + super().__init__(FileType.COMPILATION_PHASES) + # Pattern for -ccc-print-bindings output: # "target" - "tool", inputs: [...], output: "..." + self.binding_pattern = re.compile( + r'^#\s+"([^"]+)"\s+-\s+"([^"]+)",\s+inputs:\s+\[([^\]]*)\],\s+output:\s+"([^"]*)"' + ) + # Fallback patterns for other compilation phase formats + self.phase_pattern = re.compile(r"^(\w+):\s*(.+)") + self.timing_pattern = re.compile(r"(\d+(?:\.\d+)?)\s*(ms|s|us)") + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + phases_data = self._parse_compilation_phases(lines) + + total_time = sum( + phase.get("duration", 0) + for phase in phases_data["phases"] + if phase.get("duration") is not None + ) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_phases": len(phases_data["phases"]), + "total_bindings": len(phases_data["bindings"]), + "unique_tools": len(phases_data["tool_counts"]), + "total_time": total_time, + "time_unit": phases_data["time_unit"], + "tool_counts": phases_data["tool_counts"], + } + + return self.create_parsed_file(file_path, phases_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_compilation_phases(self, lines: List[str]) -> Dict[str, Any]: + phases_data = { + "phases": [], + "bindings": [], + "tool_counts": {}, + "time_unit": "ms", + "summary": {}, + "clang_version": None, + "target": None, + "thread_model": None, + "installed_dir": None, + "file_type": None, # Track what type of file this is + } + + # First pass: determine file type based on content + has_bindings = any( + self.binding_pattern.match(line.strip()) for line in lines if line.strip() + ) + has_compilation_phases_header = any( + line.strip() == "COMPILATION PHASES:" for line in lines + ) + + if has_bindings: + phases_data["file_type"] = "bindings" + elif has_compilation_phases_header: + phases_data["file_type"] = "phases" + else: + phases_data["file_type"] = "unknown" + + for line in lines: + line = line.strip() + if not line: + continue + + # Parse -ccc-print-bindings output (only for bindings files) + if phases_data["file_type"] == "bindings": + binding_match = self.binding_pattern.match(line) + if binding_match: + target = binding_match.group(1) + tool = binding_match.group(2) + inputs_str = binding_match.group(3) + output = binding_match.group(4) + + # Parse inputs array + inputs = [] + if inputs_str.strip(): + # Simple parsing of quoted inputs: "file1", "file2", ... + import re + + input_matches = re.findall(r'"([^"]*)"', inputs_str) + inputs = input_matches + + binding_entry = { + "target": target, + "tool": tool, + "inputs": inputs, + "output": output, + } + + phases_data["bindings"].append(binding_entry) + + # Count tools for summary + if tool in phases_data["tool_counts"]: + phases_data["tool_counts"][tool] += 1 + else: + phases_data["tool_counts"][tool] = 1 + + continue + + # Extract compiler information (only for phases files) + if phases_data["file_type"] == "phases": + # Extract clang version + if line.startswith("clang version"): + phases_data["clang_version"] = line + continue + + # Extract target + if line.startswith("Target:"): + phases_data["target"] = line.replace("Target:", "").strip() + continue + + # Extract thread model + if line.startswith("Thread model:"): + phases_data["thread_model"] = line.replace( + "Thread model:", "" + ).strip() + continue + + # Extract installed directory + if line.startswith("InstalledDir:"): + phases_data["installed_dir"] = line.replace( + "InstalledDir:", "" + ).strip() + continue + + # Parse phase information (fallback for timing data) + phase_match = self.phase_pattern.match(line) + if phase_match: + phase_name = phase_match.group(1) + phase_info = phase_match.group(2) + + # Extract timing information if present + timing_match = self.timing_pattern.search(phase_info) + duration = None + time_unit = "ms" + + if timing_match: + duration = float(timing_match.group(1)) + time_unit = timing_match.group(2) + + # Convert to consistent unit (milliseconds) + if time_unit == "s": + duration *= 1000 + elif time_unit == "us": + duration /= 1000 + + phase_entry = { + "name": phase_name, + "info": phase_info, + "duration": duration, + "time_unit": time_unit, + } + + phases_data["phases"].append(phase_entry) + continue + + # Handle simple timing lines like "Frontend: 123.45ms" + if ":" in line: + parts = line.split(":", 1) + if len(parts) == 2: + phase_name = parts[0].strip() + timing_info = parts[1].strip() + + timing_match = self.timing_pattern.search(timing_info) + if timing_match: + duration = float(timing_match.group(1)) + time_unit = timing_match.group(2) + + # Convert to milliseconds + if time_unit == "s": + duration *= 1000 + elif time_unit == "us": + duration /= 1000 + + phase_entry = { + "name": phase_name, + "info": timing_info, + "duration": duration, + "time_unit": "ms", + } + + phases_data["phases"].append(phase_entry) + + # Calculate summary statistics + durations = [ + p["duration"] for p in phases_data["phases"] if p["duration"] is not None + ] + phases_data["summary"] = { + "total_time": sum(durations) if durations else 0, + "avg_time": sum(durations) / len(durations) if durations else 0, + "max_time": max(durations) if durations else 0, + "min_time": min(durations) if durations else 0, + "total_bindings": len(phases_data["bindings"]), + "unique_tools": len(phases_data["tool_counts"]), + "tool_counts": phases_data["tool_counts"], + } + + return phases_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/debug_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/debug_parser.py new file mode 100644 index 0000000000000..410ba35e28923 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/debug_parser.py @@ -0,0 +1,106 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class DebugParser(BaseParser): + def __init__(self): + super().__init__(FileType.DEBUG) + self.dwarf_pattern = re.compile(r"^\s*<(\d+)><([0-9a-fA-F]+)>:\s*(.+)") + self.debug_line_pattern = re.compile( + r"^\s*Line\s+(\d+),\s*column\s+(\d+),\s*(.+)" + ) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + debug_data = self._parse_debug_info(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_lines": len(lines), + **debug_data["summary"], + } + + return self.create_parsed_file(file_path, debug_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _parse_debug_info(self, lines: List[str]) -> Dict[str, Any]: + debug_data = { + "dwarf_entries": [], + "line_info": [], + "sections": {}, + "summary": {}, + } + + current_section = None + + for line in lines: + original_line = line + line = line.strip() + + if not line: + continue + + # Detect debug sections + if line.startswith(".debug_"): + current_section = line + debug_data["sections"][current_section] = [] + continue + + # Parse DWARF entries + dwarf_match = self.dwarf_pattern.match(original_line) + if dwarf_match: + entry = { + "depth": int(dwarf_match.group(1)), + "offset": dwarf_match.group(2), + "content": dwarf_match.group(3), + } + debug_data["dwarf_entries"].append(entry) + + if current_section: + debug_data["sections"][current_section].append(entry) + continue + + # Parse debug line information + line_match = self.debug_line_pattern.match(original_line) + if line_match: + line_info = { + "line": int(line_match.group(1)), + "column": int(line_match.group(2)), + "info": line_match.group(3), + } + debug_data["line_info"].append(line_info) + + if current_section: + debug_data["sections"][current_section].append(line_info) + continue + + # Store other debug information by section + if current_section: + debug_data["sections"][current_section].append({"raw": line}) + + debug_data["summary"] = { + "dwarf_entry_count": len(debug_data["dwarf_entries"]), + "line_info_count": len(debug_data["line_info"]), + "section_count": len(debug_data["sections"]), + } + + return debug_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/dependencies_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/dependencies_parser.py new file mode 100644 index 0000000000000..2b41fd57bb1ca --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/dependencies_parser.py @@ -0,0 +1,106 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Dependency + + +class DependenciesParser(BaseParser): + def __init__(self): + super().__init__(FileType.DEPENDENCIES) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + dependencies = self._parse_dependencies(lines) + + # Calculate statistics + sources = set() + targets = set() + for dep in dependencies: + sources.add(dep.source) + targets.add(dep.target) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_dependencies": len(dependencies), + "unique_sources": len(sources), + "unique_targets": len(targets), + "unique_files": len(sources.union(targets)), + } + + return self.create_parsed_file(file_path, dependencies, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_dependencies(self, lines: List[str]) -> List[Dependency]: + dependencies = [] + current_target = None + + for line in lines: + line = line.strip() + if not line: + continue + + # Handle make-style dependencies (target: source1 source2 ...) + if ":" in line and not line.startswith(" ") and not line.startswith("\t"): + parts = line.split(":", 1) + if len(parts) == 2: + target = parts[0].strip() + sources = parts[1].strip() + current_target = target + + if sources: + for source in sources.split(): + source = source.strip() + if source and source != "\\": + dependencies.append( + Dependency( + source=source, target=target, type="dependency" + ) + ) + + # Handle continuation lines + elif (line.startswith(" ") or line.startswith("\t")) and current_target: + sources = line.strip() + for source in sources.split(): + source = source.strip() + if source and source != "\\": + dependencies.append( + Dependency( + source=source, target=current_target, type="dependency" + ) + ) + + # Handle simple dependency lists (one per line) + elif "->" in line or "=>" in line: + if "->" in line: + parts = line.split("->", 1) + else: + parts = line.split("=>", 1) + + if len(parts) == 2: + source = parts[0].strip() + target = parts[1].strip() + dependencies.append( + Dependency(source=source, target=target, type="dependency") + ) + + # Reset current target for new sections + elif not line.startswith(" ") and not line.startswith("\t"): + current_target = None + + return dependencies diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/diagnostics_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/diagnostics_parser.py new file mode 100644 index 0000000000000..5d21e06fe6a68 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/diagnostics_parser.py @@ -0,0 +1,83 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import List +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Diagnostic, SourceLocation + + +class DiagnosticsParser(BaseParser): + def __init__(self): + super().__init__(FileType.DIAGNOSTICS) + # Pattern to match diagnostic lines like: "file.c:5:9: warning: message" + self.diagnostic_pattern = re.compile( + r"(?P[^:]+):(?P\d+):(?P\d+):\s*(?P\w+):\s*(?P.+)" + ) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + diagnostics = [] + lines = content.split("\n") + + for line in lines: + line = line.strip() + if not line: + continue + + diagnostic = self._parse_diagnostic_line(line) + if diagnostic: + diagnostics.append(diagnostic) + + # Count by level + level_counts = {} + for diag in diagnostics: + level_counts[diag.level] = level_counts.get(diag.level, 0) + 1 + + metadata = { + "total_diagnostics": len(diagnostics), + "level_counts": level_counts, + "file_size": self.get_file_size(file_path), + } + + return self.create_parsed_file(file_path, diagnostics, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_diagnostic_line(self, line: str) -> Diagnostic: + match = self.diagnostic_pattern.match(line) + if match: + try: + location = SourceLocation( + file=match.group("file"), + line=int(match.group("line")), + column=int(match.group("column")), + ) + + return Diagnostic( + level=match.group("level"), + message=match.group("message"), + location=location, + ) + except ValueError: + pass + + # Fallback for lines that don't match the pattern + if any(level in line.lower() for level in ["error", "warning", "note", "info"]): + for level in ["error", "warning", "note", "info"]: + if level in line.lower(): + return Diagnostic(level=level, message=line) + + return None diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/ftime_report_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/ftime_report_parser.py new file mode 100644 index 0000000000000..5f7433055f5e8 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/ftime_report_parser.py @@ -0,0 +1,128 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class FTimeReportParser(BaseParser): + def __init__(self): + super().__init__(FileType.FTIME_REPORT) + # Patterns to match ftime-report output + # Pattern for: 0.0112 (100.0%) 0.0020 (100.0%) 0.0132 (100.0%) 0.0132 (100.0%) Front end + # This needs to be reviewed with more files and outputs + self.timing_line_pattern = re.compile( + r"^\s*(\d+\.\d+)\s+\((\d+\.\d+)%\)\s+(\d+\.\d+)\s+\((\d+\.\d+)%\)\s+(\d+\.\d+)\s+\((\d+\.\d+)%\)\s+(\d+\.\d+)\s+\((\d+\.\d+)%\)\s+(.+)$" + ) + self.total_pattern = re.compile(r"Total Execution Time:\s+(\d+\.\d+)\s+seconds") + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + timing_data = self._parse_ftime_report(lines) + + # Calculate statistics + total_time = timing_data.get("total_execution_time", 0) + timing_entries = timing_data.get("timings", []) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_execution_time": total_time, + "timing_entries_count": len(timing_entries), + "top_time_consumer": ( + timing_entries[0]["name"] if timing_entries else None + ), + "top_time_percentage": ( + timing_entries[0]["percentage"] if timing_entries else 0 + ), + } + + return self.create_parsed_file(file_path, timing_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_ftime_report(self, lines: List[str]) -> Dict[str, Any]: + timing_data = {"timings": [], "total_execution_time": 0, "summary": {}} + + parsing_timings = False + + for line in lines: + line = line.strip() + if not line: + continue + + # Check for total execution time + total_match = self.total_pattern.search(line) + if total_match: + timing_data["total_execution_time"] = float(total_match.group(1)) + continue + + # Check if we're in the timing section + if "---User Time---" in line and "--System Time--" in line: + parsing_timings = True + continue + + # Parse timing lines + if parsing_timings: + # Check if this line ends the timing section + if not line or "===" in line: + parsing_timings = False + continue + + timing_match = self.timing_line_pattern.match(line) + if timing_match: + user_time = float(timing_match.group(1)) + user_percent = float(timing_match.group(2)) + system_time = float(timing_match.group(3)) + system_percent = float(timing_match.group(4)) + total_time = float(timing_match.group(5)) + total_percent = float(timing_match.group(6)) + wall_time = float(timing_match.group(7)) + wall_percent = float(timing_match.group(8)) + name = timing_match.group(9).strip() + + timing_entry = { + "name": name, + "user_time": user_time, + "user_percentage": user_percent, + "system_time": system_time, + "system_percentage": system_percent, + "total_time": total_time, + "total_percentage": total_percent, + "wall_time": wall_time, + "wall_percentage": wall_percent, + "time_seconds": wall_time, # Use wall time as primary metric + "percentage": wall_percent, # Use wall percentage as primary metric + "time_ms": wall_time * 1000, + } + + timing_data["timings"].append(timing_entry) + + # Sort timings by time (descending) + timing_data["timings"].sort(key=lambda x: x["time_seconds"], reverse=True) + + # Calculate summary + if timing_data["timings"]: + timing_data["summary"] = { + "total_phases": len(timing_data["timings"]), + "slowest_phase": timing_data["timings"][0]["name"], + "slowest_time": timing_data["timings"][0]["time_seconds"], + "fastest_phase": timing_data["timings"][-1]["name"], + "fastest_time": timing_data["timings"][-1]["time_seconds"], + } + + return timing_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/include_tree_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/include_tree_parser.py new file mode 100644 index 0000000000000..59a0164eb918b --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/include_tree_parser.py @@ -0,0 +1,91 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Dependency + + +class IncludeTreeParser(BaseParser): + def __init__(self): + super().__init__(FileType.INCLUDE_TREE) + self.include_pattern = re.compile(r"^(\s*)(\S+)\s*(?:\(([^)]+)\))?") + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + include_data = self._parse_include_tree(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_includes": len(include_data["dependencies"]), + "unique_files": len(include_data["files"]), + "max_depth": include_data["max_depth"], + } + + return self.create_parsed_file(file_path, include_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_include_tree(self, lines: List[str]) -> Dict[str, Any]: + include_data = {"dependencies": [], "files": set(), "tree": [], "max_depth": 0} + + stack = [] # Stack to track parent files + + for line in lines: + if not line.strip(): + continue + + match = self.include_pattern.match(line) + if match: + indent = len(match.group(1)) + file_path = match.group(2) + extra_info = match.group(3) + + depth = indent // 2 # Assuming 2 spaces per indent level + include_data["max_depth"] = max(include_data["max_depth"], depth) + + # Adjust stack based on current depth + while len(stack) > depth: + stack.pop() + + # Add to files set + include_data["files"].add(file_path) + + # Create dependency relationship + if stack: + parent = stack[-1] + dependency = Dependency( + source=parent, target=file_path, type="include" + ) + include_data["dependencies"].append(dependency) + + # Add tree entry + tree_entry = { + "file": file_path, + "depth": depth, + "parent": stack[-1] if stack else None, + "extra_info": extra_info, + } + include_data["tree"].append(tree_entry) + + # Push current file onto stack + stack.append(file_path) + + # Convert set to list for JSON serialization + include_data["files"] = list(include_data["files"]) + + return include_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/ir_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/ir_parser.py new file mode 100644 index 0000000000000..098f213524f21 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/ir_parser.py @@ -0,0 +1,140 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class IRParser(BaseParser): + def __init__(self): + super().__init__(FileType.IR) + self.function_pattern = re.compile(r"^define\s+.*@(\w+)\s*\(") + self.global_pattern = re.compile(r"^@(\w+)\s*=") + self.type_pattern = re.compile(r"^%(\w+)\s*=\s*type") + + def parse(self, file_path: str) -> ParsedFile: + if self.is_large_file(file_path): + return self._parse_large_ir(file_path) + + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + ir_data = self._analyze_ir_content(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_lines": len(lines), + **ir_data["summary"], + } + + return self.create_parsed_file(file_path, ir_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _parse_large_ir(self, file_path: str) -> ParsedFile: + try: + ir_data = {"functions": [], "globals": [], "types": [], "summary": {}} + line_count = 0 + + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + for line in f: + line_count += 1 + line = line.strip() + + # Only parse first 10000 lines for large files + if line_count > 10000: + break + + if line.startswith("define"): + match = self.function_pattern.match(line) + if match: + ir_data["functions"].append(match.group(1)) + elif line.startswith("@") and "=" in line: + match = self.global_pattern.match(line) + if match: + ir_data["globals"].append(match.group(1)) + elif line.startswith("%") and "type" in line: + match = self.type_pattern.match(line) + if match: + ir_data["types"].append(match.group(1)) + + ir_data["summary"] = { + "function_count": len(ir_data["functions"]), + "global_count": len(ir_data["globals"]), + "type_count": len(ir_data["types"]), + "analyzed_lines": line_count, + "is_partial": True, + } + + metadata = { + "file_size": self.get_file_size(file_path), + **ir_data["summary"], + } + + return self.create_parsed_file(file_path, ir_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _analyze_ir_content(self, lines: List[str]) -> Dict[str, Any]: + ir_data = { + "functions": [], + "globals": [], + "types": [], + "instructions": {}, + "summary": {}, + } + + instruction_count = 0 + + for line in lines: + line = line.strip() + if not line or line.startswith(";"): + continue + + # Count instructions + if any( + line.strip().startswith(inst) + for inst in ["%", "call", "ret", "br", "load", "store"] + ): + instruction_count += 1 + + # Extract functions + if line.startswith("define"): + match = self.function_pattern.match(line) + if match: + ir_data["functions"].append(match.group(1)) + + # Extract globals + elif line.startswith("@") and "=" in line: + match = self.global_pattern.match(line) + if match: + ir_data["globals"].append(match.group(1)) + + # Extract types + elif line.startswith("%") and "type" in line: + match = self.type_pattern.match(line) + if match: + ir_data["types"].append(match.group(1)) + + ir_data["summary"] = { + "function_count": len(ir_data["functions"]), + "global_count": len(ir_data["globals"]), + "type_count": len(ir_data["types"]), + "instruction_count": instruction_count, + } + + return ir_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/macro_expansion_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/macro_expansion_parser.py new file mode 100644 index 0000000000000..108790b977f54 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/macro_expansion_parser.py @@ -0,0 +1,134 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class MacroExpansionParser(BaseParser): + def __init__(self): + super().__init__(FileType.MACRO_EXPANSION) + self.macro_pattern = re.compile(r"^#define\s+(\w+)(?:\(([^)]*)\))?\s*(.*)") + self.expansion_pattern = re.compile( + r'^\s*//\s*expanded\s*from\s*[\'"]([^\'"]+)[\'"]' + ) + + def parse(self, file_path: str) -> ParsedFile: + if self.is_large_file(file_path): + return self._parse_large_macro_expansion(file_path) + + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + macro_data = self._analyze_macro_content(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_lines": len(lines), + **macro_data["summary"], + } + + return self.create_parsed_file(file_path, macro_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _parse_large_macro_expansion(self, file_path: str) -> ParsedFile: + try: + macro_data = {"macros": {}, "expansions": [], "summary": {}} + + line_count = 0 + + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + for line in f: + line_count += 1 + + # Only analyze first 5000 lines for large files + if line_count > 5000: + break + + line = line.strip() + + if not line: + continue + + # Parse macro definitions + macro_match = self.macro_pattern.match(line) + if macro_match: + macro_name = macro_match.group(1) + params = macro_match.group(2) + definition = macro_match.group(3) + + macro_data["macros"][macro_name] = { + "parameters": params.split(",") if params else [], + "definition": definition, + } + continue + + # Parse expansion comments + expansion_match = self.expansion_pattern.match(line) + if expansion_match: + macro_data["expansions"].append(expansion_match.group(1)) + + macro_data["summary"] = { + "macro_count": len(macro_data["macros"]), + "expansion_count": len(macro_data["expansions"]), + "analyzed_lines": line_count, + "is_partial": True, + } + + metadata = { + "file_size": self.get_file_size(file_path), + **macro_data["summary"], + } + + return self.create_parsed_file(file_path, macro_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _analyze_macro_content(self, lines: List[str]) -> Dict[str, Any]: + macro_data = {"macros": {}, "expansions": [], "summary": {}} + + for line in lines: + line = line.strip() + + if not line: + continue + + # Parse macro definitions + macro_match = self.macro_pattern.match(line) + if macro_match: + macro_name = macro_match.group(1) + params = macro_match.group(2) + definition = macro_match.group(3) + + macro_data["macros"][macro_name] = { + "parameters": params.split(",") if params else [], + "definition": definition, + } + continue + + # Parse expansion comments + expansion_match = self.expansion_pattern.match(line) + if expansion_match: + macro_data["expansions"].append(expansion_match.group(1)) + + macro_data["summary"] = { + "macro_count": len(macro_data["macros"]), + "expansion_count": len(macro_data["expansions"]), + } + + return macro_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/objdump_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/objdump_parser.py new file mode 100644 index 0000000000000..89091f7db6ee7 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/objdump_parser.py @@ -0,0 +1,118 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Symbol + + +class ObjdumpParser(BaseParser): + def __init__(self): + super().__init__(FileType.OBJDUMP) + self.symbol_pattern = re.compile( + r"^([0-9a-fA-F]+)\s+([lgw!])\s+([dDfFoO])\s+(\S+)\s+([0-9a-fA-F]+)\s+(.+)" + ) + self.section_pattern = re.compile( + r"^Idx\s+Name\s+Size\s+VMA\s+LMA\s+File Offset\s+Algn" + ) + self.disasm_pattern = re.compile( + r"^\s*([0-9a-fA-F]+):\s+([0-9a-fA-F\s]+)\s+(.+)" + ) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + objdump_data = self._parse_objdump_content(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_lines": len(lines), + **objdump_data["summary"], + } + + return self.create_parsed_file(file_path, objdump_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _parse_objdump_content(self, lines: List[str]) -> Dict[str, Any]: + objdump_data = { + "symbols": [], + "sections": [], + "disassembly": [], + "headers": [], + "summary": {}, + } + + current_section = None + in_symbol_table = False + in_disassembly = False + + for line in lines: + line = line.rstrip() + + if not line: + continue + + # Detect sections + if "SYMBOL TABLE:" in line: + in_symbol_table = True + continue + elif "Disassembly of section" in line: + in_disassembly = True + current_section = line + continue + elif line.startswith("Contents of section"): + current_section = line + continue + + # Parse symbol table + if in_symbol_table and self.symbol_pattern.match(line): + match = self.symbol_pattern.match(line) + if match: + symbol = Symbol( + name=match.group(6), + address=match.group(1), + type=match.group(3), + section=match.group(4), + ) + objdump_data["symbols"].append(symbol) + + # Parse disassembly + elif in_disassembly and self.disasm_pattern.match(line): + match = self.disasm_pattern.match(line) + if match: + objdump_data["disassembly"].append( + { + "address": match.group(1), + "bytes": match.group(2).strip(), + "instruction": match.group(3), + } + ) + + # Collect headers and other info + elif line.startswith("Program Header:") or line.startswith( + "Section Headers:" + ): + objdump_data["headers"].append(line) + + objdump_data["summary"] = { + "symbol_count": len(objdump_data["symbols"]), + "disasm_count": len(objdump_data["disassembly"]), + "section_count": len(objdump_data["sections"]), + "header_count": len(objdump_data["headers"]), + } + + return objdump_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/pgo_profile_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/pgo_profile_parser.py new file mode 100644 index 0000000000000..6419c4d335374 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/pgo_profile_parser.py @@ -0,0 +1,55 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class PGOProfileParser(BaseParser): + def __init__(self): + super().__init__(FileType.PGO_PROFILE) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + profile_data = {"functions": [], "counters": [], "raw_lines": []} + + current_function = None + + for line in lines: + line = line.strip() + if not line: + continue + + profile_data["raw_lines"].append(line) + + # Simple pattern matching for PGO profile data + if line.startswith("# Func Hash:") or line.startswith("Function:"): + current_function = line + profile_data["functions"].append(line) + elif line.startswith("# Num Counters:") or line.isdigit(): + profile_data["counters"].append(line) + + metadata = { + "total_functions": len(profile_data["functions"]), + "total_counters": len(profile_data["counters"]), + "total_lines": len(profile_data["raw_lines"]), + "file_size": self.get_file_size(file_path), + } + + return self.create_parsed_file(file_path, profile_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/preprocessed_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/preprocessed_parser.py new file mode 100644 index 0000000000000..2bdf9a7929370 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/preprocessed_parser.py @@ -0,0 +1,159 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class PreprocessedParser(BaseParser): + def __init__(self): + super().__init__(FileType.PREPROCESSED) + self.line_directive_pattern = re.compile(r'^#\s*(\d+)\s+"([^"]+)"') + self.pragma_pattern = re.compile(r"^#\s*pragma\s+(.+)") + + def parse(self, file_path: str) -> ParsedFile: + if self.is_large_file(file_path): + return self._parse_large_preprocessed(file_path) + + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, {}, {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + preprocessed_data = self._analyze_preprocessed_content(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_lines": len(lines), + **preprocessed_data["summary"], + } + + return self.create_parsed_file(file_path, preprocessed_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _parse_large_preprocessed(self, file_path: str) -> ParsedFile: + try: + preprocessed_data = { + "source_files": set(), + "pragmas": [], + "directives": [], + "summary": {}, + } + + line_count = 0 + code_lines = 0 + + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + for line in f: + line_count += 1 + + # Only analyze first 10000 lines for large files + if line_count > 10000: + break + + line = line.strip() + + if not line: + continue + + # Parse line directives + line_match = self.line_directive_pattern.match(line) + if line_match: + source_file = line_match.group(2) + preprocessed_data["source_files"].add(source_file) + preprocessed_data["directives"].append( + {"line": int(line_match.group(1)), "file": source_file} + ) + continue + + # Parse pragma directives + pragma_match = self.pragma_pattern.match(line) + if pragma_match: + preprocessed_data["pragmas"].append(pragma_match.group(1)) + continue + + # Count actual code lines + if not line.startswith("#"): + code_lines += 1 + + preprocessed_data["source_files"] = list(preprocessed_data["source_files"]) + + preprocessed_data["summary"] = { + "source_file_count": len(preprocessed_data["source_files"]), + "pragma_count": len(preprocessed_data["pragmas"]), + "directive_count": len(preprocessed_data["directives"]), + "code_lines": code_lines, + "analyzed_lines": line_count, + "is_partial": True, + } + + metadata = { + "file_size": self.get_file_size(file_path), + **preprocessed_data["summary"], + } + + return self.create_parsed_file(file_path, preprocessed_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, {}, {"error": str(e)}) + + def _analyze_preprocessed_content(self, lines: List[str]) -> Dict[str, Any]: + preprocessed_data = { + "source_files": set(), + "pragmas": [], + "directives": [], + "summary": {}, + } + + code_lines = 0 + + for line in lines: + original_line = line + line = line.strip() + + if not line: + continue + + # Parse line directives + line_match = self.line_directive_pattern.match(line) + if line_match: + source_file = line_match.group(2) + preprocessed_data["source_files"].add(source_file) + preprocessed_data["directives"].append( + {"line": int(line_match.group(1)), "file": source_file} + ) + continue + + # Parse pragma directives + pragma_match = self.pragma_pattern.match(line) + if pragma_match: + preprocessed_data["pragmas"].append(pragma_match.group(1)) + continue + + # Count actual code lines (not preprocessor directives) + if not line.startswith("#"): + code_lines += 1 + + # Convert set to list for JSON serialization + preprocessed_data["source_files"] = list(preprocessed_data["source_files"]) + + preprocessed_data["summary"] = { + "source_file_count": len(preprocessed_data["source_files"]), + "pragma_count": len(preprocessed_data["pragmas"]), + "directive_count": len(preprocessed_data["directives"]), + "code_lines": code_lines, + } + + return preprocessed_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/remarks_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/remarks_parser.py new file mode 100644 index 0000000000000..23dc1e2de9809 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/remarks_parser.py @@ -0,0 +1,96 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import yaml +from typing import List, Dict, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Remark, SourceLocation + + +class RemarksParser(BaseParser): + def __init__(self): + super().__init__(FileType.REMARKS) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + remarks = [] + # Handle custom YAML tags by creating a loader + loader = yaml.SafeLoader + loader.add_constructor( + "!Passed", lambda loader, node: loader.construct_mapping(node) + ) + loader.add_constructor( + "!Missed", lambda loader, node: loader.construct_mapping(node) + ) + loader.add_constructor( + "!Analysis", lambda loader, node: loader.construct_mapping(node) + ) + + yaml_docs = yaml.load_all(content, Loader=loader) + + for doc in yaml_docs: + if not doc: + continue + + remark = self._parse_remark(doc) + if remark: + remarks.append(remark) + + metadata = { + "total_remarks": len(remarks), + "file_size": self.get_file_size(file_path), + } + + return self.create_parsed_file(file_path, remarks, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_remark(self, doc: Dict[str, Any]) -> Remark: + try: + pass_name = doc.get("Pass", "") + function = doc.get("Function", "") + + # Extract location information + location = None + debug_loc = doc.get("DebugLoc") + if debug_loc: + location = SourceLocation( + file=debug_loc.get("File"), + line=debug_loc.get("Line"), + column=debug_loc.get("Column"), + ) + + # Build message from args or use Name + message = doc.get("Name", "") + args = doc.get("Args", []) + if args: + arg_strings = [] + for arg in args: + if isinstance(arg, dict) and "String" in arg: + arg_strings.append(arg["String"]) + elif isinstance(arg, str): + arg_strings.append(arg) + if arg_strings: + message = "".join(arg_strings) + + return Remark( + pass_name=pass_name, + function=function, + message=message, + location=location, + args=doc.get("Args", {}), + ) + except Exception: + return None diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/runtime_trace_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/runtime_trace_parser.py new file mode 100644 index 0000000000000..eba58fbd1c0b4 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/runtime_trace_parser.py @@ -0,0 +1,17 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from .time_trace_parser import TimeTraceParser +from ..models import FileType + + +class RuntimeTraceParser(TimeTraceParser): + def __init__(self): + # Runtime trace uses the same Chrome trace format as time-trace + super().__init__() + self.file_type = FileType.RUNTIME_TRACE diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/sarif_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/sarif_parser.py new file mode 100644 index 0000000000000..dcd41101a1ae0 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/sarif_parser.py @@ -0,0 +1,81 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import json +from typing import List, Dict, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Diagnostic, SourceLocation + + +class SARIFParser(BaseParser): + def __init__(self): + super().__init__(FileType.STATIC_ANALYSIS_SARIF) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + sarif_data = json.loads(content) + diagnostics = [] + + # Parse SARIF format + runs = sarif_data.get("runs", []) + for run in runs: + results = run.get("results", []) + for result in results: + diagnostic = self._parse_sarif_result(result, run) + if diagnostic: + diagnostics.append(diagnostic) + + metadata = { + "total_results": len(diagnostics), + "file_size": self.get_file_size(file_path), + "sarif_version": sarif_data.get("$schema", ""), + "runs_count": len(runs), + } + + return self.create_parsed_file(file_path, diagnostics, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_sarif_result( + self, result: Dict[str, Any], run: Dict[str, Any] + ) -> Diagnostic: + try: + message = result.get("message", {}).get("text", "") + rule_id = result.get("ruleId", "") + + # Extract level from result + level = result.get("level", "info") + + # Extract location + location = None + locations = result.get("locations", []) + if locations: + physical_location = locations[0].get("physicalLocation", {}) + artifact_location = physical_location.get("artifactLocation", {}) + region = physical_location.get("region", {}) + + if artifact_location.get("uri"): + location = SourceLocation( + file=artifact_location.get("uri"), + line=region.get("startLine"), + column=region.get("startColumn"), + ) + + return Diagnostic( + level=level, message=message, location=location, code=rule_id + ) + + except Exception: + return None diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/static_analyzer_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/static_analyzer_parser.py new file mode 100644 index 0000000000000..cb642cefb5df1 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/static_analyzer_parser.py @@ -0,0 +1,81 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import List, Dict, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Diagnostic, SourceLocation + + +class StaticAnalyzerParser(BaseParser): + def __init__(self): + super().__init__(FileType.STATIC_ANALYZER) + # Pattern for static analyzer output + self.analyzer_pattern = re.compile( + r"(?P[^:]+):(?P\d+):(?P\d+):\s*(?P\w+):\s*(?P.+)" + ) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + results = [] + + for line in lines: + line = line.strip() + if not line: + continue + + # Try to parse as diagnostic + diagnostic = self._parse_analyzer_line(line) + if diagnostic: + results.append(diagnostic) + else: + # Store as raw line for other analysis results + results.append({"type": "raw", "content": line}) + + # Count diagnostic types + diagnostic_count = sum(1 for r in results if isinstance(r, Diagnostic)) + raw_count = len(results) - diagnostic_count + + metadata = { + "total_results": len(results), + "diagnostic_count": diagnostic_count, + "raw_count": raw_count, + "file_size": self.get_file_size(file_path), + } + + return self.create_parsed_file(file_path, results, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_analyzer_line(self, line: str) -> Diagnostic: + match = self.analyzer_pattern.match(line) + if match: + try: + location = SourceLocation( + file=match.group("file"), + line=int(match.group("line")), + column=int(match.group("column")), + ) + + return Diagnostic( + level=match.group("level"), + message=match.group("message"), + location=location, + ) + except ValueError: + pass + + return None diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/symbols_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/symbols_parser.py new file mode 100644 index 0000000000000..832c99e5b4086 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/symbols_parser.py @@ -0,0 +1,111 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, Symbol + + +class SymbolsParser(BaseParser): + def __init__(self): + super().__init__(FileType.SYMBOLS) + # Pattern for nm output: "address type name" + self.nm_pattern = re.compile(r"^([0-9a-fA-F]+)\s+([A-Za-z?])\s+(.+)$") + # Pattern for objdump symbol table + self.objdump_pattern = re.compile( + r"^([0-9a-fA-F]+)\s+([lgw!])\s+([dDfFoO])\s+(\S+)\s+([0-9a-fA-F]+)\s+(.+)$" + ) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + symbols = self._parse_symbols(lines) + + # Calculate statistics + symbol_types = {} + sections = set() + + for symbol in symbols: + if symbol.type: + symbol_types[symbol.type] = symbol_types.get(symbol.type, 0) + 1 + if symbol.section: + sections.add(symbol.section) + + metadata = { + "file_size": self.get_file_size(file_path), + "total_symbols": len(symbols), + "symbol_types": symbol_types, + "unique_sections": len(sections), + } + + return self.create_parsed_file(file_path, symbols, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_symbols(self, lines: List[str]) -> List[Symbol]: + symbols = [] + + for line in lines: + line = line.strip() + if not line or line.startswith("#"): + continue + + # Try nm format first + nm_match = self.nm_pattern.match(line) + if nm_match: + symbol = Symbol( + name=nm_match.group(3), + address=nm_match.group(1), + type=nm_match.group(2), + ) + symbols.append(symbol) + continue + + # Try objdump format + objdump_match = self.objdump_pattern.match(line) + if objdump_match: + symbol = Symbol( + name=objdump_match.group(6), + address=objdump_match.group(1), + type=objdump_match.group(3), + section=objdump_match.group(4), + size=( + int(objdump_match.group(5), 16) + if objdump_match.group(5) != "0" + else None + ), + ) + symbols.append(symbol) + continue + + # Try simple format: "name address size" + parts = line.split() + if len(parts) >= 2: + name = parts[0] + address = ( + parts[1] + if parts[1].replace("0x", "").replace("0X", "").isalnum() + else None + ) + size = None + + if len(parts) >= 3 and parts[2].isdigit(): + size = int(parts[2]) + + symbol = Symbol(name=name, address=address, size=size) + symbols.append(symbol) + + return symbols diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/time_trace_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/time_trace_parser.py new file mode 100644 index 0000000000000..fbe80fab27065 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/time_trace_parser.py @@ -0,0 +1,244 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import json +from typing import List, Dict, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile, TraceEvent + + +class TimeTraceParser(BaseParser): + def __init__(self): + super().__init__(FileType.TIME_TRACE) + + def parse(self, file_path: str) -> ParsedFile: + if self.is_large_file(file_path): + return self._parse_large_file(file_path) + + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + data = json.loads(content) + events = self._parse_trace_events(data.get("traceEvents", [])) + + metadata = { + "total_events": len(events), + "file_size": self.get_file_size(file_path), + "beginning_of_time": data.get("beginningOfTime"), + } + + return self.create_parsed_file(file_path, events, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_large_file(self, file_path: str) -> ParsedFile: + events = [] + try: + # For large files, parse in streaming mode + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + content = f.read(1024 * 1024) # Read first 1MB + if content.startswith('{"traceEvents":['): + # Try to extract just a sample of events + bracket_count = 0 + event_start = content.find("[") + 1 + event_end = event_start + + for i, char in enumerate(content[event_start:], event_start): + if char == "{": + bracket_count += 1 + elif char == "}": + bracket_count -= 1 + if bracket_count == 0 and len(events) < 1000: + try: + event_json = content[event_start : i + 1] + event_data = json.loads(event_json) + event = self._parse_trace_event(event_data) + if event: + events.append(event) + except: + pass + # Find next event + comma_pos = content.find(",", i) + if comma_pos == -1: + break + event_start = comma_pos + 1 + + metadata = { + "total_events": len(events), + "file_size": self.get_file_size(file_path), + "is_partial": True, + } + + return self.create_parsed_file(file_path, events, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_trace_events(self, events: List[Dict[str, Any]]) -> List[TraceEvent]: + parsed_events = [] + for event_data in events: + event = self._parse_trace_event(event_data) + if event: + parsed_events.append(event) + return parsed_events + + def _parse_trace_event(self, event_data: Dict[str, Any]) -> TraceEvent: + try: + return TraceEvent( + name=event_data.get("name", ""), + category=event_data.get("cat", ""), + phase=event_data.get("ph", ""), + timestamp=event_data.get("ts", 0), + duration=event_data.get("dur"), + pid=event_data.get("pid"), + tid=event_data.get("tid"), + args=event_data.get("args", {}), + ) + except Exception: + return None + + def get_flamegraph_data(self, events: List[TraceEvent]) -> Dict[str, Any]: + """Convert trace events to chrome tracing format with proper stacking""" + stacks = [] + + # Filter for duration events and sort by timestamp + duration_events = [e for e in events if e.duration and e.duration > 0] + duration_events.sort(key=lambda x: x.timestamp) + + # Build proper call stacks using begin/end events + call_stack = [] + stack_frames = [] + + # Group events by thread for proper stacking + thread_events = {} + for event in duration_events: + tid = event.tid or 0 + if tid not in thread_events: + thread_events[tid] = [] + thread_events[tid].append(event) + + # Process each thread separately + all_samples = [] + for tid, events_list in thread_events.items(): + samples = self._build_call_stacks(events_list, tid) + all_samples.extend(samples) + + # Sort all samples by timestamp for time order view + all_samples.sort(key=lambda x: x["timestamp"]) + + # Calculate total duration for normalization + if all_samples: + min_time = min(s["timestamp"] for s in all_samples) + max_time = max(s["timestamp"] + s["duration"] for s in all_samples) + total_duration = max_time - min_time + else: + total_duration = 0 + + return { + "samples": all_samples, + "total_duration": total_duration, + "thread_count": len(thread_events), + "total_events": len(all_samples), + } + + def _build_call_stacks( + self, events: List[TraceEvent], thread_id: int + ) -> List[Dict]: + """Build proper call stacks from trace events""" + samples = [] + active_frames = [] # Stack of currently active frames + + for event in events: + # Create frame info + frame = { + "name": event.name, + "category": event.category, + "timestamp": event.timestamp, + "duration": event.duration, + "thread_id": thread_id, + "depth": 0, + "args": event.args or {}, + } + + # Find proper depth by checking overlaps with active frames + frame["depth"] = self._calculate_frame_depth(frame, active_frames) + + # Add frame to samples + samples.append(frame) + + # Update active frames list + self._update_active_frames(frame, active_frames) + + return samples + + def _calculate_frame_depth(self, frame: Dict, active_frames: List[Dict]) -> int: + """Calculate the proper depth for a frame based on overlapping frames""" + frame_start = frame["timestamp"] + frame_end = frame["timestamp"] + frame["duration"] + + # Find the maximum depth of overlapping frames + max_depth = 0 + for active in active_frames: + active_start = active["timestamp"] + active_end = active["timestamp"] + active["duration"] + + # Check if frames overlap + if frame_start < active_end and frame_end > active_start: + max_depth = max(max_depth, active["depth"] + 1) + + return max_depth + + def _update_active_frames(self, new_frame: Dict, active_frames: List[Dict]): + """Update the list of active frames, removing expired ones""" + current_time = new_frame["timestamp"] + + # Remove frames that have ended + active_frames[:] = [ + frame + for frame in active_frames + if frame["timestamp"] + frame["duration"] > current_time + ] + + # Add new frame + active_frames.append(new_frame) + + def get_sandwich_data(self, events: List[TraceEvent]) -> Dict[str, Any]: + """Aggregate events by function name for sandwich view""" + function_stats = {} + + duration_events = [e for e in events if e.duration and e.duration > 0] + + for event in duration_events: + name = event.name + if name not in function_stats: + function_stats[name] = { + "name": name, + "total_time": 0, + "call_count": 0, + "category": event.category, + "avg_time": 0, + } + + function_stats[name]["total_time"] += event.duration + function_stats[name]["call_count"] += 1 + + # Calculate averages and sort by total time + for stats in function_stats.values(): + if stats["call_count"] > 0: + stats["avg_time"] = stats["total_time"] / stats["call_count"] + + sorted_functions = sorted( + function_stats.values(), key=lambda x: x["total_time"], reverse=True + ) + + return {"functions": sorted_functions} diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/version_info_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/version_info_parser.py new file mode 100644 index 0000000000000..37fb882599def --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/version_info_parser.py @@ -0,0 +1,88 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import re +from typing import Dict, List, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class VersionInfoParser(BaseParser): + def __init__(self): + super().__init__(FileType.VERSION_INFO) + self.version_pattern = re.compile(r"clang version\s+([\d\w\.\-\+]+)") + self.target_pattern = re.compile(r"Target:\s+(.+)") + self.thread_pattern = re.compile(r"Thread model:\s+(.+)") + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + version_data = self._parse_version_info(lines) + + metadata = { + "file_size": self.get_file_size(file_path), + "clang_version": version_data.get("clang_version"), + "target": version_data.get("target"), + "thread_model": version_data.get("thread_model"), + "full_version_string": version_data.get("full_version_string"), + } + + return self.create_parsed_file(file_path, version_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) + + def _parse_version_info(self, lines: List[str]) -> Dict[str, Any]: + version_data = { + "clang_version": None, + "target": None, + "thread_model": None, + "full_version_string": "", + "install_dir": None, + "libraries": [], + } + + for line in lines: + line = line.strip() + if not line: + continue + + # Store the full version string (usually the first line) + if not version_data["full_version_string"] and "clang version" in line: + version_data["full_version_string"] = line + + # Extract version number + version_match = self.version_pattern.search(line) + if version_match: + version_data["clang_version"] = version_match.group(1) + + # Extract target + target_match = self.target_pattern.search(line) + if target_match: + version_data["target"] = target_match.group(1) + + # Extract thread model + thread_match = self.thread_pattern.search(line) + if thread_match: + version_data["thread_model"] = thread_match.group(1) + + # Extract install directory + if line.startswith("InstalledDir:"): + version_data["install_dir"] = line.replace("InstalledDir:", "").strip() + + # Extract library paths + if "library" in line.lower() or "lib" in line: + version_data["libraries"].append(line) + + return version_data diff --git a/llvm/tools/llvm-advisor/tools/common/parsers/xray_parser.py b/llvm/tools/llvm-advisor/tools/common/parsers/xray_parser.py new file mode 100644 index 0000000000000..9e57c903fad43 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/common/parsers/xray_parser.py @@ -0,0 +1,67 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from typing import Dict, Any +from .base_parser import BaseParser +from ..models import FileType, ParsedFile + + +class XRayParser(BaseParser): + def __init__(self): + super().__init__(FileType.XRAY) + + def parse(self, file_path: str) -> ParsedFile: + content = self.read_file_safe(file_path) + if content is None: + return self.create_parsed_file( + file_path, [], {"error": "File too large or unreadable"} + ) + + try: + lines = content.split("\n") + xray_data = {"entries": [], "functions": set(), "events": []} + + for line in lines: + line = line.strip() + if not line or line.startswith("#"): + continue + + # Parse XRay log entries + parts = line.split() + if len(parts) >= 4: + entry = { + "timestamp": ( + parts[0] if parts[0].replace(".", "").isdigit() else None + ), + "function": parts[1] if len(parts) > 1 else None, + "event_type": parts[2] if len(parts) > 2 else None, + "data": " ".join(parts[3:]) if len(parts) > 3 else None, + } + + xray_data["entries"].append(entry) + + if entry["function"]: + xray_data["functions"].add(entry["function"]) + + if entry["event_type"]: + xray_data["events"].append(entry["event_type"]) + + # Convert sets to lists for JSON serialization + xray_data["functions"] = list(xray_data["functions"]) + + metadata = { + "total_entries": len(xray_data["entries"]), + "unique_functions": len(xray_data["functions"]), + "event_types": list(set(xray_data["events"])), + "file_size": self.get_file_size(file_path), + } + + return self.create_parsed_file(file_path, xray_data, metadata) + + except Exception as e: + return self.create_parsed_file(file_path, [], {"error": str(e)}) diff --git a/llvm/tools/llvm-advisor/tools/webserver/__init__.py b/llvm/tools/llvm-advisor/tools/webserver/__init__.py new file mode 100644 index 0000000000000..0e977a146fa04 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/__init__.py @@ -0,0 +1,7 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/__init__.py b/llvm/tools/llvm-advisor/tools/webserver/api/__init__.py new file mode 100644 index 0000000000000..0e977a146fa04 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/__init__.py @@ -0,0 +1,7 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/artifacts.py b/llvm/tools/llvm-advisor/tools/webserver/api/artifacts.py new file mode 100644 index 0000000000000..708fb81fecffd --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/artifacts.py @@ -0,0 +1,153 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import os +import sys +from typing import Dict, Any +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent +sys.path.insert(0, str(tools_dir)) + +from .base import BaseEndpoint, APIResponse +from common.models import FileType + + +class ArtifactsEndpoint(BaseEndpoint): + """GET /api/artifacts/{file_type} - Get aggregated data for a file type across all units""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + if len(path_parts) < 3: + return APIResponse.invalid_request("File type required") + + file_type_str = path_parts[2] + + # Validate file type + try: + file_type = FileType(file_type_str) + except ValueError: + return APIResponse.invalid_request(f"Invalid file type: {file_type_str}") + + parsed_data = self.get_parsed_data() + + # Aggregate data from all units for this file type + aggregated_data = { + "file_type": file_type.value, + "units": {}, + "global_summary": { + "total_files": 0, + "total_errors": 0, + "units_with_type": 0, + }, + } + + for unit_name, unit_data in parsed_data.items(): + if file_type in unit_data: + unit_files = [] + error_count = 0 + unit_summary_stats = {} + + for parsed_file in unit_data[file_type]: + has_error = "error" in parsed_file.metadata + if has_error: + error_count += 1 + + file_summary = { + "file_name": os.path.basename(parsed_file.file_path), + "file_path": parsed_file.file_path, + "file_size_bytes": parsed_file.metadata.get("file_size", 0), + "has_error": has_error, + "metadata": parsed_file.metadata, + } + + # Include relevant summary data based on file type + if ( + isinstance(parsed_file.data, dict) + and "summary" in parsed_file.data + ): + file_summary["summary"] = parsed_file.data["summary"] + + # Aggregate numeric summary stats + for key, value in parsed_file.data["summary"].items(): + if isinstance(value, (int, float)): + unit_summary_stats[key] = ( + unit_summary_stats.get(key, 0) + value + ) + + elif isinstance(parsed_file.data, list): + file_summary["item_count"] = len(parsed_file.data) + unit_summary_stats["total_items"] = unit_summary_stats.get( + "total_items", 0 + ) + len(parsed_file.data) + + unit_files.append(file_summary) + + aggregated_data["units"][unit_name] = { + "files": unit_files, + "count": len(unit_files), + "errors": error_count, + "summary_stats": unit_summary_stats, + } + + aggregated_data["global_summary"]["total_files"] += len(unit_files) + aggregated_data["global_summary"]["total_errors"] += error_count + aggregated_data["global_summary"]["units_with_type"] += 1 + + # Add file type specific aggregations + if file_type_str == "diagnostics": + aggregated_data["global_summary"]["total_diagnostics"] = sum( + unit["summary_stats"].get("total_diagnostics", 0) + for unit in aggregated_data["units"].values() + ) + elif file_type_str == "remarks": + aggregated_data["global_summary"]["total_remarks"] = sum( + unit["summary_stats"].get("total_remarks", 0) + for unit in aggregated_data["units"].values() + ) + elif file_type_str in ["time-trace", "runtime-trace"]: + aggregated_data["global_summary"]["total_events"] = sum( + unit["summary_stats"].get("total_events", 0) + for unit in aggregated_data["units"].values() + ) + + return APIResponse.success(aggregated_data) + + +class ArtifactTypesEndpoint(BaseEndpoint): + """GET /api/artifacts - List all available artifact types with counts""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + parsed_data = self.get_parsed_data() + + # Count files by type across all units + type_counts = {} + + for unit_name, unit_data in parsed_data.items(): + for file_type, parsed_files in unit_data.items(): + type_name = file_type.value + if type_name not in type_counts: + type_counts[type_name] = { + "total_files": 0, + "total_errors": 0, + "units": [], + } + + error_count = sum(1 for f in parsed_files if "error" in f.metadata) + type_counts[type_name]["total_files"] += len(parsed_files) + type_counts[type_name]["total_errors"] += error_count + type_counts[type_name]["units"].append(unit_name) + + response_data = { + "supported_types": [ft.value for ft in FileType], + "available_types": type_counts, + "total_types_found": len(type_counts), + } + + return APIResponse.success(response_data) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/base.py b/llvm/tools/llvm-advisor/tools/webserver/api/base.py new file mode 100644 index 0000000000000..135a2233d952b --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/base.py @@ -0,0 +1,55 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import json +from abc import ABC, abstractmethod +from typing import Dict, Any, Optional + + +class BaseEndpoint(ABC): + def __init__(self, data_dir: str, collector): + self.data_dir = data_dir + self.collector = collector + self._cache = {} + + @abstractmethod + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + pass + + def get_compilation_units(self): + if "units" not in self._cache: + self._cache["units"] = self.collector.discover_compilation_units( + self.data_dir + ) + return self._cache["units"] + + def get_parsed_data(self): + if "parsed_data" not in self._cache: + self._cache["parsed_data"] = self.collector.parse_all_units(self.data_dir) + return self._cache["parsed_data"] + + def clear_cache(self): + self._cache.clear() + + +class APIResponse: + @staticmethod + def success(data: Any, status: int = 200) -> Dict[str, Any]: + return {"success": True, "data": data, "status": status} + + @staticmethod + def error(message: str, status: int = 400) -> Dict[str, Any]: + return {"success": False, "error": message, "status": status} + + @staticmethod + def not_found(resource: str) -> Dict[str, Any]: + return APIResponse.error(f"{resource} not found", 404) + + @staticmethod + def invalid_request(message: str) -> Dict[str, Any]: + return APIResponse.error(f"Invalid request: {message}", 400) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/explorer.py b/llvm/tools/llvm-advisor/tools/webserver/api/explorer.py new file mode 100644 index 0000000000000..0490d7751a90e --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/explorer.py @@ -0,0 +1,714 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import os +import sys +from typing import Dict, Any, List +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent +sys.path.insert(0, str(tools_dir)) + +from .base import BaseEndpoint, APIResponse +from common.models import FileType + + +class ExplorerEndpoint(BaseEndpoint): + """Explorer endpoints for source code and artifact viewing""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + """Route explorer requests to specific handlers""" + if len(path_parts) < 3: + return APIResponse.invalid_request("Explorer endpoint required") + + endpoint = path_parts[2] + + if endpoint == "files": + return self.handle_files(path_parts, query_params) + elif endpoint == "source": + return self.handle_source(path_parts, query_params) + elif endpoint in [ + "assembly", + "ir", + "optimized-ir", + "object", + "ast-json", + "preprocessed", + "macro-expansion", + ]: + return self.handle_artifact(path_parts, query_params, endpoint) + else: + return APIResponse.not_found(f"Explorer endpoint '{endpoint}'") + + def handle_files( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/explorer/files - Get available source files and their artifacts""" + try: + parsed_data = self.get_parsed_data() + + # Check if unit parameter is provided + unit_filter = query_params.get("unit", [None])[0] + + if not unit_filter: + return APIResponse.invalid_request("Unit parameter is required") + + # Only process the specified unit + if unit_filter not in parsed_data: + return APIResponse.not_found(f"Unit '{unit_filter}' not found") + + unit_artifacts = parsed_data[unit_filter] + files_info = [] + + print(f"Processing files for unit: {unit_filter}") + + # Get available artifact types for this unit + available_types = set(unit_artifacts.keys()) + + # PRIORITY: Get source files from collected sources (timestamped) + source_files = set() + + # Use directly collected source files + if FileType.SOURCES in unit_artifacts: + for parsed_file in unit_artifacts[FileType.SOURCES]: + # Add collected source files + source_file_path = os.path.basename(parsed_file.file_path) + source_files.add(source_file_path) + + # Fallback to dependencies parsing + if not source_files: + if FileType.DEPENDENCIES in unit_artifacts: + for parsed_file in unit_artifacts[FileType.DEPENDENCIES]: + if isinstance(parsed_file.data, list): + for dependency in parsed_file.data: + # Handle both object-style and dict-style dependencies + source_path = None + target_path = None + + if hasattr(dependency, "source"): + source_path = dependency.source + elif ( + isinstance(dependency, dict) + and "source" in dependency + ): + source_path = dependency["source"] + + if hasattr(dependency, "target"): + target_path = dependency.target + elif ( + isinstance(dependency, dict) + and "target" in dependency + ): + target_path = dependency["target"] + + # Filter for actual source files (skip object files, etc.) + if source_path and self._is_source_file(source_path): + source_files.add(source_path) + + if target_path and self._is_source_file(target_path): + source_files.add(target_path) + + # Extract source file references from diagnostics and remarks as fallback + for file_type in [FileType.DIAGNOSTICS, FileType.REMARKS]: + if file_type in unit_artifacts: + for parsed_file in unit_artifacts[file_type]: + if isinstance(parsed_file.data, list): + for item in parsed_file.data: + if isinstance(item, dict) and "file" in item: + source_path = item["file"] + if self._is_source_file(source_path): + source_files.add(source_path) + + # For each identified source file, check what artifacts are available + for source_file in source_files: + available_artifacts = self._get_available_artifacts_for_source( + source_file, unit_artifacts + ) + + if available_artifacts: # Only include files that have artifacts + # Make the path relative to make it cleaner for display + display_path = source_file + if source_file.startswith("./"): + display_path = source_file[2:] + + files_info.append( + { + "path": source_file, + "name": os.path.basename(source_file), + "display_name": display_path, + "unit": unit_filter, # Use unit_filter instead of unit_name + "available_artifacts": available_artifacts, + } + ) + + # Remove duplicates and sort + unique_files = {} + for file_info in files_info: + key = file_info["path"] + if key not in unique_files: + unique_files[key] = file_info + else: + # Merge available artifacts + existing_artifacts = set(unique_files[key]["available_artifacts"]) + new_artifacts = set(file_info["available_artifacts"]) + unique_files[key]["available_artifacts"] = list( + existing_artifacts | new_artifacts + ) + + final_files = sorted(unique_files.values(), key=lambda x: x["display_name"]) + + return APIResponse.success( + {"files": final_files, "count": len(final_files)} + ) + + except Exception as e: + return APIResponse.error(f"Failed to load source files: {str(e)}") + + def handle_source( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/explorer/source/{file_path} - Get source code content""" + if len(path_parts) < 4: + return APIResponse.invalid_request("File path required") + + file_path = "/".join(path_parts[3:]) + + try: + # Look for source files in the timestamped runs first + source_content = None + parsed_data = self.get_parsed_data() + + # Check parsed sources from timestamped runs + for unit_name, unit_artifacts in parsed_data.items(): + if FileType.SOURCES in unit_artifacts: + for parsed_file in unit_artifacts[FileType.SOURCES]: + # Extract filename from both the requested path and stored path + requested_filename = os.path.basename(file_path) + stored_filename = os.path.basename(parsed_file.file_path) + + if requested_filename == stored_filename: + # Found the source file in our collected sources + if os.path.exists(parsed_file.file_path): + with open( + parsed_file.file_path, + "r", + encoding="utf-8", + errors="ignore", + ) as f: + source_content = f.read() + break + + if source_content: + break + + # Fallback to original filesystem search + if not source_content: + # Try absolute path first + if os.path.exists(file_path): + with open(file_path, "r", encoding="utf-8", errors="ignore") as f: + source_content = f.read() + + # Try relative to data directory (where .llvm-advisor is) + if not source_content: + # Remove leading ./ if present + clean_path = file_path + if clean_path.startswith("./"): + clean_path = clean_path[2:] + + relative_path = os.path.join(self.data_dir, clean_path) + if os.path.exists(relative_path): + with open( + relative_path, "r", encoding="utf-8", errors="ignore" + ) as f: + source_content = f.read() + + # Try one level up from data directory (since source files might be outside .llvm-advisor) + if not source_content: + parent_dir = os.path.dirname(self.data_dir) + clean_path = file_path + if clean_path.startswith("./"): + clean_path = clean_path[2:] + + parent_relative_path = os.path.join(parent_dir, clean_path) + if os.path.exists(parent_relative_path): + with open( + parent_relative_path, "r", encoding="utf-8", errors="ignore" + ) as f: + source_content = f.read() + + if not source_content: + return APIResponse.not_found(f"Source file '{file_path}'") + + # Get inline data (diagnostics, remarks) from parsed data + inline_data = self._get_inline_data_for_file(parsed_data, file_path) + + return APIResponse.success( + { + "source": source_content, + "file_path": file_path, + "language": self._detect_language(file_path), + "inline_data": inline_data, + } + ) + + except Exception as e: + return APIResponse.error(f"Failed to load source file: {str(e)}") + + def handle_artifact( + self, path_parts: list, query_params: Dict[str, list], artifact_type: str + ) -> Dict[str, Any]: + """GET /api/explorer/{artifact_type}/{file_path} - Get artifact content""" + if len(path_parts) < 4: + return APIResponse.invalid_request("File path required") + + file_path = "/".join(path_parts[3:]) + + try: + parsed_data = self.get_parsed_data() + + # Map endpoint names to FileType enums + type_mapping = { + "assembly": FileType.ASSEMBLY, + "ir": FileType.IR, + "optimized-ir": FileType.IR, + "object": FileType.OBJDUMP, + "ast-json": FileType.AST_JSON, + "preprocessed": FileType.PREPROCESSED, + "macro-expansion": FileType.MACRO_EXPANSION, + } + + if artifact_type not in type_mapping: + return APIResponse.invalid_request( + f"Unknown artifact type: {artifact_type}" + ) + + file_type = type_mapping[artifact_type] + base_name = os.path.splitext(os.path.basename(file_path))[0] + + # Try to read the raw file directly from .llvm-advisor + raw_content = self._try_read_raw_artifact( + file_path, artifact_type, base_name + ) + if raw_content: + return APIResponse.success( + { + "content": raw_content, + "file_path": file_path, + "artifact_type": artifact_type, + } + ) + + # Fall back to parsed data if raw file not found + content = None + for unit_name, unit_artifacts in parsed_data.items(): + if file_type in unit_artifacts: + for parsed_file in unit_artifacts[file_type]: + artifact_base = os.path.splitext( + os.path.basename(parsed_file.file_path) + )[0] + # More flexible matching to ensure we find the right content + if ( + base_name in artifact_base + or artifact_base in base_name + or self._matches_artifact_to_source( + parsed_file.file_path, file_path + ) + ): + # Try to read the raw file directly based on parsed_file.file_path + if os.path.exists(parsed_file.file_path): + try: + with open( + parsed_file.file_path, + "r", + encoding="utf-8", + errors="ignore", + ) as f: + raw_file_content = f.read() + content = raw_file_content + break + except Exception as e: + pass + + # Use the already parsed content as fallback + if not content: + content = self._format_parsed_content(parsed_file.data) + if ( + content + and content.strip() + and content.strip() != "# No data available" + ): + break + if ( + content + and content.strip() + and content.strip() != "# No data available" + ): + break + + if not content: + return APIResponse.not_found(f"{artifact_type} for file '{file_path}'") + + return APIResponse.success( + { + "content": content, + "file_path": file_path, + "artifact_type": artifact_type, + } + ) + + except Exception as e: + return APIResponse.error(f"Failed to load {artifact_type}: {str(e)}") + + def _format_parsed_content(self, data: Any) -> str: + """Format already parsed content for display""" + if isinstance(data, str): + return data + elif isinstance(data, dict): + if "content" in data: + return data["content"] + elif "text" in data: + return data["text"] + elif "data" in data: + # Sometimes the actual content is nested in a 'data' field + return self._format_parsed_content(data["data"]) + elif "instructions" in data: + # Handle assembly with instructions field + if isinstance(data["instructions"], list): + return "\n".join(str(inst) for inst in data["instructions"]) + else: + return str(data["instructions"]) + elif "assembly" in data: + # Handle assembly content + return str(data["assembly"]) + elif "ir" in data: + # Handle LLVM IR content + return str(data["ir"]) + elif "source" in data: + # Handle source content + return str(data["source"]) + else: + # For structured data, format as JSON with proper indentation + import json + + return json.dumps(data, indent=2) + elif isinstance(data, list): + # Handle lists - could be lines of code, assembly instructions, etc. + if len(data) > 0: + first_item = data[0] + if isinstance(first_item, str): + # Simple list of strings - join with newlines + return "\n".join(str(item) for item in data) + elif isinstance(first_item, dict): + # List of structured data - try to extract meaningful content + lines = [] + for item in data: + if isinstance(item, dict): + # Try to extract text/content/instruction from each item + if "instruction" in item: + lines.append(str(item["instruction"])) + elif "text" in item: + lines.append(str(item["text"])) + elif "content" in item: + lines.append(str(item["content"])) + elif "line" in item: + lines.append(str(item["line"])) + elif "assembly" in item: + lines.append(str(item["assembly"])) + elif "ir" in item: + lines.append(str(item["ir"])) + else: + # For complex objects, try to stringify them meaningfully + if hasattr(item, "__str__") and not isinstance( + item, dict + ): + lines.append(str(item)) + else: + # Extract all values and join them + values = [ + str(v) for v in item.values() if v is not None + ] + if values: + lines.append(" ".join(values)) + else: + lines.append(json.dumps(item, indent=2)) + else: + lines.append(str(item)) + return "\n".join(lines) + else: + return "\n".join(str(item) for item in data) + else: + return "# No data available" + else: + # Handle any other data types including custom objects + if hasattr(data, "__dict__"): + # For custom objects, try to extract meaningful content + return str(data) + else: + return str(data) + + def _get_inline_data_for_file( + self, parsed_data: Dict, file_path: str + ) -> Dict[str, List[Dict]]: + """Get inline data (diagnostics, remarks) for a specific file using parsed data""" + inline_data = {"diagnostics": [], "remarks": []} + + for unit_name, unit_artifacts in parsed_data.items(): + # Get diagnostics + if FileType.DIAGNOSTICS in unit_artifacts: + for parsed_file in unit_artifacts[FileType.DIAGNOSTICS]: + if isinstance(parsed_file.data, list): + for diagnostic in parsed_file.data: + # Handle both dataclass objects and dictionaries + if hasattr(diagnostic, "location") and diagnostic.location: + if self._matches_file( + diagnostic.location.file or "", file_path + ): + inline_data["diagnostics"].append( + { + "line": diagnostic.location.line or 0, + "column": diagnostic.location.column or 0, + "level": diagnostic.level, + "message": diagnostic.message, + "type": "diagnostic", + } + ) + elif isinstance(diagnostic, dict) and self._matches_file( + diagnostic.get("file", ""), file_path + ): + inline_data["diagnostics"].append( + { + "line": diagnostic.get("line", 0), + "column": diagnostic.get("column", 0), + "level": diagnostic.get("level", "info"), + "message": diagnostic.get("message", ""), + "type": "diagnostic", + } + ) + + # Get remarks + if FileType.REMARKS in unit_artifacts: + for parsed_file in unit_artifacts[FileType.REMARKS]: + if isinstance(parsed_file.data, list): + for remark in parsed_file.data: + # Handle both dataclass objects and dictionaries + if hasattr(remark, "location") and remark.location: + if self._matches_file( + remark.location.file or "", file_path + ): + inline_data["remarks"].append( + { + "line": remark.location.line or 0, + "column": remark.location.column or 0, + "level": "remark", + "message": remark.message, + "pass": remark.pass_name, + "type": "remark", + } + ) + elif isinstance(remark, dict) and self._matches_file( + remark.get("file", ""), file_path + ): + inline_data["remarks"].append( + { + "line": remark.get("line", 0), + "column": remark.get("column", 0), + "level": remark.get("level", "info"), + "message": remark.get("message", ""), + "pass": remark.get("pass", ""), + "type": "remark", + } + ) + + return inline_data + + def _matches_file(self, path1: str, path2: str) -> bool: + """Check if two paths refer to the same source file""" + if not path1 or not path2: + return False + + # Normalize paths + norm1 = os.path.normpath(path1).replace("\\", "/") + norm2 = os.path.normpath(path2).replace("\\", "/") + + # Direct match or basename match + return ( + norm1 == norm2 + or os.path.basename(norm1) == os.path.basename(norm2) + or norm1.endswith(norm2) + or norm2.endswith(norm1) + ) + + def _detect_language(self, file_path: str) -> str: + """Detect programming language from file extension""" + ext = os.path.splitext(file_path)[1].lower() + + lang_map = { + ".cpp": "cpp", + ".cc": "cpp", + ".cxx": "cpp", + ".c++": "cpp", + ".c": "c", + ".h": "c", + ".hpp": "cpp", + ".hxx": "cpp", + ".h++": "cpp", + ".py": "python", + ".js": "javascript", + ".rs": "rust", + ".go": "go", + ".java": "java", + } + + return lang_map.get(ext, "text") + + def _is_source_file(self, file_path: str) -> bool: + """Check if a file path represents a source code file""" + if not file_path: + return False + + # Get file extension + ext = os.path.splitext(file_path)[1].lower() + + # List of source file extensions + source_extensions = { + ".c", + ".cpp", + ".cc", + ".cxx", + ".c++", + ".h", + ".hpp", + ".hxx", + ".h++", + ".py", + ".js", + ".rs", + ".go", + ".java", + ".swift", + ".kt", + ".scala", + ".rb", + ".php", + ".pl", + ".sh", + ".bash", + } + + return ext in source_extensions + + def _get_available_artifacts_for_source( + self, source_file_path: str, unit_artifacts: Dict + ) -> List[str]: + """Check which artifacts are available for a given source file""" + available_artifacts = [] + base_name = os.path.splitext(os.path.basename(source_file_path))[0] + + # Map artifact types to their display names + type_map = { + FileType.ASSEMBLY: "assembly", + FileType.IR: "ir", + FileType.AST_JSON: "ast-json", + FileType.OBJDUMP: "object", + FileType.PREPROCESSED: "preprocessed", + } + + for file_type, display_name in type_map.items(): + if file_type in unit_artifacts: + # Check if this source file has this type of artifact + for parsed_file in unit_artifacts[file_type]: + artifact_base = os.path.splitext( + os.path.basename(parsed_file.file_path) + )[0] + # Try various matching strategies + if ( + artifact_base == base_name + or base_name in artifact_base + or artifact_base in base_name + or self._matches_artifact_to_source( + parsed_file.file_path, source_file_path + ) + ): + available_artifacts.append(display_name) + break + + return available_artifacts + + def _try_read_raw_artifact( + self, file_path: str, artifact_type: str, base_name: str + ) -> str: + """Try to read the raw artifact file directly from .llvm-advisor directory""" + try: + # Map artifact types to their directory names and file extensions + artifact_mapping = { + "assembly": ("assembly", [".s", ".asm"]), + "ir": ("ir", [".ll"]), + "optimized-ir": ("ir", [".ll"]), + "object": ("objdump", [".objdump", ".obj", ".txt"]), + "ast-json": ("ast-json", [".json"]), + "preprocessed": ("preprocessed", [".i", ".ii"]), + "macro-expansion": ("macro-expansion", [".i", ".ii"]), + } + + if artifact_type not in artifact_mapping: + return None + + dir_name, extensions = artifact_mapping[artifact_type] + + # Look for the artifact file in all compilation units + for root, dirs, files in os.walk(self.data_dir): + if dir_name in os.path.basename(root): + # Try to find files that match our base name + for file in files: + file_base = os.path.splitext(file)[0] + file_ext = os.path.splitext(file)[1] + if ( + file_base == base_name + or base_name in file_base + or file_base in base_name + ): + # Check if file has the right extension + if file_ext in extensions or not extensions: + artifact_path = os.path.join(root, file) + with open( + artifact_path, + "r", + encoding="utf-8", + errors="ignore", + ) as f: + content = f.read() + return content + return None + + except Exception as e: + return None + + def _matches_artifact_to_source(self, artifact_path: str, source_path: str) -> bool: + """Check if an artifact matches a source file using various heuristics""" + artifact_name = os.path.basename(artifact_path) + source_name = os.path.basename(source_path) + + # Remove extensions for comparison + artifact_base = os.path.splitext(artifact_name)[0] + source_base = os.path.splitext(source_name)[0] + + # Direct match + if artifact_base == source_base: + return True + + # Handle cases like test.cpp -> test.s, test.ll, etc. + if artifact_base.startswith(source_base) or source_base.startswith( + artifact_base + ): + return True + + # Handle mangled names or similar patterns + # This could be expanded based on actual file naming patterns found + return False diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/files.py b/llvm/tools/llvm-advisor/tools/webserver/api/files.py new file mode 100644 index 0000000000000..531210ea9c232 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/files.py @@ -0,0 +1,124 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import os +import sys +from typing import Dict, Any +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent +sys.path.insert(0, str(tools_dir)) + +from .base import BaseEndpoint, APIResponse +from common.models import FileType + + +class FileContentEndpoint(BaseEndpoint): + """GET /api/file/{unit_name}/{file_type}/{file_name} - Get parsed content of a specific file""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + if len(path_parts) < 5: + return APIResponse.invalid_request( + "File path must include unit_name, file_type, and file_name" + ) + + unit_name = path_parts[2] + file_type_str = path_parts[3] + file_name = path_parts[4] + + # Validate file type + try: + file_type = FileType(file_type_str) + except ValueError: + return APIResponse.invalid_request(f"Invalid file type: {file_type_str}") + + parsed_data = self.get_parsed_data() + + # Validate unit exists + if unit_name not in parsed_data: + return APIResponse.not_found(f"Compilation unit '{unit_name}'") + + # Validate file type exists in unit + if file_type not in parsed_data[unit_name]: + return APIResponse.not_found( + f"File type '{file_type_str}' in unit '{unit_name}'" + ) + + # Find the specific file + target_file = None + for parsed_file in parsed_data[unit_name][file_type]: + if os.path.basename(parsed_file.file_path) == file_name: + target_file = parsed_file + break + + if not target_file: + return APIResponse.not_found( + f"File '{file_name}' of type '{file_type_str}' in unit '{unit_name}'" + ) + + # Check for streaming/partial parsing + include_full_data = query_params.get("full", ["false"])[0].lower() == "true" + + # For code artifacts (assembly, ir, preprocessed, etc.), return raw file content + code_artifact_types = { + FileType.ASSEMBLY, + FileType.IR, + FileType.PREPROCESSED, + FileType.MACRO_EXPANSION, + FileType.AST_JSON, + } + + if file_type in code_artifact_types: + # Return raw file content for code viewing + try: + with open(target_file.file_path, "r", encoding="utf-8") as f: + raw_content = f.read() + + response_data = { + "file_path": target_file.file_path, + "file_name": file_name, + "unit_name": unit_name, + "file_type": file_type.value, + "content": raw_content, + "data_type": "raw", + "metadata": target_file.metadata, + "has_error": "error" in target_file.metadata, + } + + return APIResponse.success(response_data) + + except Exception as e: + return APIResponse.server_error( + f"Failed to read file content: {str(e)}" + ) + + response_data = { + "file_path": target_file.file_path, + "file_name": file_name, + "unit_name": unit_name, + "file_type": file_type.value, + "metadata": target_file.metadata, + "has_error": "error" in target_file.metadata, + } + + # Include data based on query parameters and file size + if target_file.metadata.get("is_partial", False) and not include_full_data: + # For large files that were partially parsed, provide summary only + if isinstance(target_file.data, dict) and "summary" in target_file.data: + response_data["summary"] = target_file.data["summary"] + response_data["data_type"] = "summary" + else: + response_data["data"] = target_file.data + response_data["data_type"] = "partial" + else: + response_data["data"] = target_file.data + response_data["data_type"] = "full" + + return APIResponse.success(response_data) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/health.py b/llvm/tools/llvm-advisor/tools/webserver/api/health.py new file mode 100644 index 0000000000000..5674a57c6e75b --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/health.py @@ -0,0 +1,43 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import os +from typing import Dict, Any +from .base import BaseEndpoint, APIResponse + + +class HealthEndpoint(BaseEndpoint): + """GET /api/health - System health and data directory status""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + data_dir_exists = os.path.exists(self.data_dir) + + # Count units and files if data directory exists + unit_count = 0 + file_count = 0 + + if data_dir_exists: + try: + units = self.get_compilation_units() + unit_count = len(units) + file_count = sum( + sum(len(files) for files in unit.artifacts.values()) + for unit in units + ) + except Exception: + pass + + health_data = { + "status": "healthy" if data_dir_exists else "no_data", + "data_dir": self.data_dir, + "data_dir_exists": data_dir_exists, + "compilation_units": unit_count, + "total_files": file_count, + } + + return APIResponse.success(health_data) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/__init__.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/__init__.py new file mode 100644 index 0000000000000..0e977a146fa04 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/__init__.py @@ -0,0 +1,7 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/base_specialized.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/base_specialized.py new file mode 100644 index 0000000000000..848d1b4e4eb6f --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/base_specialized.py @@ -0,0 +1,36 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from typing import Dict, Any +import sys +from pathlib import Path +from ..base import APIResponse + + +class BaseSpecializedEndpoint: + """Base class for specialized file-type endpoints""" + + def __init__(self, data_dir: str, collector): + self.data_dir = data_dir + self.collector = collector + self._cache = {} + + def get_compilation_units(self): + if "units" not in self._cache: + self._cache["units"] = self.collector.discover_compilation_units( + self.data_dir + ) + return self._cache["units"] + + def get_parsed_data(self): + if "parsed_data" not in self._cache: + self._cache["parsed_data"] = self.collector.parse_all_units(self.data_dir) + return self._cache["parsed_data"] + + def clear_cache(self): + self._cache.clear() diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/binary_size_api.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/binary_size_api.py new file mode 100644 index 0000000000000..8ceb21755e811 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/binary_size_api.py @@ -0,0 +1,448 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import sys +from collections import defaultdict, Counter +from typing import Dict, Any +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent.parent +sys.path.insert(0, str(tools_dir)) + +from common.models import FileType, BinarySize +from ..base import BaseEndpoint, APIResponse + + +class BinarySizeEndpoint(BaseEndpoint): + """Specialized endpoints for binary size analysis""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + """Route requests to specific handlers based on path""" + if len(path_parts) >= 3: + sub_endpoint = path_parts[2] + else: + sub_endpoint = "overview" + + method_name = f"handle_{sub_endpoint.replace('-', '_')}" + + if hasattr(self, method_name): + handler_method = getattr(self, method_name) + return handler_method(path_parts, query_params) + else: + available_methods = [ + method[7:].replace("_", "-") + for method in dir(self) + if method.startswith("handle_") and method != "handle" + ] + + return APIResponse.error( + f"Sub-endpoint '{sub_endpoint}' not available. " + f"Available: {', '.join(available_methods)}", + 404, + ) + + def handle_overview( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/binary-size/overview - Overall binary size statistics""" + parsed_data = self.get_parsed_data() + + total_size = 0 + section_sizes = defaultdict(int) + section_counts = Counter() + size_distribution = [] + + for unit_name, unit_data in parsed_data.items(): + if FileType.BINARY_SIZE in unit_data: + for parsed_file in unit_data[FileType.BINARY_SIZE]: + if isinstance(parsed_file.data, list): + for size_entry in parsed_file.data: + if isinstance(size_entry, BinarySize): + total_size += size_entry.size + section_sizes[size_entry.section] += size_entry.size + section_counts[size_entry.section] += 1 + size_distribution.append(size_entry.size) + + # Calculate size statistics + size_stats = {} + if size_distribution: + size_distribution.sort() + size_stats = { + "total_size": total_size, + "average_section_size": total_size / len(size_distribution), + "median_section_size": size_distribution[len(size_distribution) // 2], + "largest_section_size": max(size_distribution), + "smallest_section_size": min(size_distribution), + "total_sections": len(size_distribution), + } + + overview_data = { + "size_statistics": size_stats, + "section_breakdown": dict( + sorted(section_sizes.items(), key=lambda x: x[1], reverse=True)[:15] + ), + "section_counts": dict(section_counts.most_common(10)), + "size_insights": self._generate_size_insights(section_sizes, total_size), + } + + return APIResponse.success(overview_data) + + def handle_sections( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/binary-size/sections - Detailed analysis by sections""" + parsed_data = self.get_parsed_data() + + sections_data = defaultdict( + lambda: { + "total_size": 0, + "occurrences": 0, + "units": set(), + "size_distribution": [], + } + ) + + for unit_name, unit_data in parsed_data.items(): + if FileType.BINARY_SIZE in unit_data: + for parsed_file in unit_data[FileType.BINARY_SIZE]: + if isinstance(parsed_file.data, list): + for size_entry in parsed_file.data: + if isinstance(size_entry, BinarySize): + section = size_entry.section + sections_data[section]["total_size"] += size_entry.size + sections_data[section]["occurrences"] += 1 + sections_data[section]["units"].add(unit_name) + sections_data[section]["size_distribution"].append( + size_entry.size + ) + + # Convert to detailed analysis + result = {} + for section, data in sections_data.items(): + sizes = data["size_distribution"] + sizes.sort() + + result[section] = { + "total_size": data["total_size"], + "occurrences": data["occurrences"], + "units_involved": len(data["units"]), + "average_size": ( + data["total_size"] / data["occurrences"] + if data["occurrences"] > 0 + else 0 + ), + "size_range": { + "min": min(sizes) if sizes else 0, + "max": max(sizes) if sizes else 0, + "median": sizes[len(sizes) // 2] if sizes else 0, + }, + "section_type": self._classify_section_type(section), + "optimization_potential": self._assess_optimization_potential( + section, data["total_size"] + ), + } + + # Sort by total size + sorted_result = dict( + sorted(result.items(), key=lambda x: x[1]["total_size"], reverse=True) + ) + + return APIResponse.success( + {"sections": sorted_result, "total_unique_sections": len(sorted_result)} + ) + + def handle_optimization( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/binary-size/optimization - Size optimization opportunities""" + parsed_data = self.get_parsed_data() + + large_sections = [] + redundant_sections = defaultdict(list) + optimization_opportunities = [] + + # Collect all size data + all_sections = defaultdict(list) + + for unit_name, unit_data in parsed_data.items(): + if FileType.BINARY_SIZE in unit_data: + for parsed_file in unit_data[FileType.BINARY_SIZE]: + if isinstance(parsed_file.data, list): + for size_entry in parsed_file.data: + if isinstance(size_entry, BinarySize): + all_sections[size_entry.section].append( + { + "size": size_entry.size, + "unit": unit_name, + "percentage": size_entry.percentage or 0, + } + ) + + # Find optimization opportunities + total_binary_size = sum( + sum(entries[0]["size"] for entries in all_sections.values()) + ) + + for section, entries in all_sections.items(): + total_section_size = sum(entry["size"] for entry in entries) + section_percentage = (total_section_size / max(total_binary_size, 1)) * 100 + + # Large sections (>5% of total) + if section_percentage > 5: + large_sections.append( + { + "section": section, + "total_size": total_section_size, + "percentage": section_percentage, + "occurrences": len(entries), + "optimization_type": "large_section", + } + ) + + # Redundant sections (same name, multiple occurrences) + if len(entries) > 1: + redundant_sections[section] = { + "total_size": total_section_size, + "occurrences": len(entries), + "average_size": total_section_size / len(entries), + "units": [entry["unit"] for entry in entries], + } + + # Generate specific optimization recommendations + optimization_opportunities = self._generate_optimization_recommendations( + large_sections, redundant_sections, total_binary_size + ) + + optimization_data = { + "large_sections": sorted( + large_sections, key=lambda x: x["total_size"], reverse=True + ), + "redundant_sections": dict(redundant_sections), + "optimization_opportunities": optimization_opportunities, + "potential_savings": self._calculate_potential_savings( + large_sections, redundant_sections + ), + "binary_size_breakdown": { + "total_size": total_binary_size, + "largest_contributors": [ + s["section"] + for s in sorted( + large_sections, key=lambda x: x["total_size"], reverse=True + )[:5] + ], + }, + } + + return APIResponse.success(optimization_data) + + def handle_comparison( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/binary-size/comparison - Compare sizes across compilation units""" + parsed_data = self.get_parsed_data() + + unit_sizes = {} + section_comparison = defaultdict(dict) + + for unit_name, unit_data in parsed_data.items(): + if FileType.BINARY_SIZE in unit_data: + unit_total = 0 + unit_sections = {} + + for parsed_file in unit_data[FileType.BINARY_SIZE]: + if isinstance(parsed_file.data, list): + for size_entry in parsed_file.data: + if isinstance(size_entry, BinarySize): + unit_total += size_entry.size + unit_sections[size_entry.section] = size_entry.size + section_comparison[size_entry.section][ + unit_name + ] = size_entry.size + + unit_sizes[unit_name] = { + "total_size": unit_total, + "section_count": len(unit_sections), + "sections": unit_sections, + "largest_section": ( + max(unit_sections.items(), key=lambda x: x[1]) + if unit_sections + else ("", 0) + ), + } + + # Calculate comparison metrics + if unit_sizes: + sizes = [data["total_size"] for data in unit_sizes.values()] + avg_size = sum(sizes) / len(sizes) + + comparison_metrics = { + "average_unit_size": avg_size, + "largest_unit": max( + unit_sizes.items(), key=lambda x: x[1]["total_size"] + ), + "smallest_unit": min( + unit_sizes.items(), key=lambda x: x[1]["total_size"] + ), + "size_variance": self._calculate_variance(sizes), + "units_above_average": [ + name + for name, data in unit_sizes.items() + if data["total_size"] > avg_size + ], + } + else: + comparison_metrics = {} + + comparison_data = { + "unit_comparison": dict( + sorted( + unit_sizes.items(), key=lambda x: x[1]["total_size"], reverse=True + ) + ), + "section_comparison": dict(section_comparison), + "comparison_metrics": comparison_metrics, + "insights": self._generate_comparison_insights( + unit_sizes, comparison_metrics + ), + } + + return APIResponse.success(comparison_data) + + def _generate_size_insights( + self, section_sizes: Dict[str, int], total_size: int + ) -> list: + """Generate insights about binary size""" + insights = [] + + if section_sizes: + largest_section = max(section_sizes.items(), key=lambda x: x[1]) + largest_percentage = (largest_section[1] / max(total_size, 1)) * 100 + + if largest_percentage > 50: + insights.append( + f"'{largest_section[0]}' dominates binary size ({largest_percentage:.1f}%)" + ) + + text_sections = [k for k in section_sizes.keys() if "text" in k.lower()] + if text_sections: + text_size = sum(section_sizes[k] for k in text_sections) + text_percentage = (text_size / max(total_size, 1)) * 100 + insights.append( + f"Code sections account for {text_percentage:.1f}% of binary size" + ) + + return insights + + def _classify_section_type(self, section_name: str) -> str: + """Classify section by type""" + section_lower = section_name.lower() + + if "text" in section_lower or "code" in section_lower: + return "code" + elif "data" in section_lower: + return "data" + elif "bss" in section_lower: + return "uninitialized_data" + elif "rodata" in section_lower or "const" in section_lower: + return "read_only_data" + elif "debug" in section_lower: + return "debug_info" + else: + return "other" + + def _assess_optimization_potential(self, section: str, size: int) -> str: + """Assess optimization potential for a section""" + if size > 1024 * 1024: # >1MB + return "high" + elif size > 256 * 1024: # >256KB + return "medium" + else: + return "low" + + def _generate_optimization_recommendations( + self, large_sections: list, redundant_sections: dict, total_size: int + ) -> list: + """Generate specific optimization recommendations""" + recommendations = [] + + if large_sections: + recommendations.append( + { + "type": "large_sections", + "priority": "high", + "description": f"Consider optimizing {len(large_sections)} large sections", + "sections": [s["section"] for s in large_sections[:3]], + } + ) + + if redundant_sections: + recommendations.append( + { + "type": "redundant_sections", + "priority": "medium", + "description": f"Found {len(redundant_sections)} potentially redundant sections", + "sections": list(redundant_sections.keys())[:3], + } + ) + + if total_size > 10 * 1024 * 1024: # >10MB + recommendations.append( + { + "type": "overall_size", + "priority": "medium", + "description": "Binary size is large - consider link-time optimization", + } + ) + + return recommendations + + def _calculate_potential_savings( + self, large_sections: list, redundant_sections: dict + ) -> Dict[str, Any]: + """Calculate potential size savings""" + large_section_savings = sum( + s["total_size"] * 0.1 for s in large_sections + ) # Assume 10% reduction + redundant_savings = sum( + data["total_size"] * 0.2 for data in redundant_sections.values() + ) # Assume 20% reduction + + return { + "from_large_sections": int(large_section_savings), + "from_redundant_sections": int(redundant_savings), + "total_potential": int(large_section_savings + redundant_savings), + } + + def _calculate_variance(self, values: list) -> float: + """Calculate variance of values""" + if len(values) <= 1: + return 0.0 + + mean = sum(values) / len(values) + return sum((x - mean) ** 2 for x in values) / len(values) + + def _generate_comparison_insights(self, unit_sizes: dict, metrics: dict) -> list: + """Generate insights from unit comparison""" + insights = [] + + if ( + metrics.get("size_variance", 0) + > (metrics.get("average_unit_size", 0) * 0.25) ** 2 + ): + insights.append( + "High size variance between units - investigate inconsistencies" + ) + + if metrics.get("units_above_average"): + insights.append( + f"{len(metrics['units_above_average'])} units are above average size" + ) + + return insights diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/compilation_phases_api.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/compilation_phases_api.py new file mode 100644 index 0000000000000..0d062f5f370ce --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/compilation_phases_api.py @@ -0,0 +1,526 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import sys +from collections import defaultdict +from typing import Dict, Any, List +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent.parent +sys.path.insert(0, str(tools_dir)) + +from common.models import FileType +from ..base import BaseEndpoint, APIResponse + + +class CompilationPhasesEndpoint(BaseEndpoint): + """Specialized endpoints for compilation phases timing analysis""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + """Route requests to specific handlers based on path""" + if len(path_parts) >= 3: + sub_endpoint = path_parts[2] + else: + sub_endpoint = "overview" + + method_name = f"handle_{sub_endpoint.replace('-', '_')}" + + if hasattr(self, method_name): + handler_method = getattr(self, method_name) + return handler_method(path_parts, query_params) + else: + available_methods = [ + method[7:].replace("_", "-") + for method in dir(self) + if method.startswith("handle_") and method != "handle" + ] + + return APIResponse.error( + f"Sub-endpoint '{sub_endpoint}' not available. " + f"Available: {', '.join(available_methods)}", + 404, + ) + + def handle_overview( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/compilation-phases/overview - Overall compilation timing statistics""" + parsed_data = self.get_parsed_data() + + total_time = 0 + phase_times = defaultdict(list) + unit_times = {} + + for unit_name, unit_data in parsed_data.items(): + if FileType.COMPILATION_PHASES in unit_data: + unit_total = 0 + for parsed_file in unit_data[FileType.COMPILATION_PHASES]: + if ( + isinstance(parsed_file.data, dict) + and "phases" in parsed_file.data + ): + for phase in parsed_file.data["phases"]: + if phase.get("duration") is not None: + duration = phase["duration"] + phase_name = phase["name"] + + phase_times[phase_name].append(duration) + unit_total += duration + total_time += duration + + if unit_total > 0: + unit_times[unit_name] = unit_total + + # Calculate phase statistics + phase_stats = {} + for phase_name, times in phase_times.items(): + if times: + phase_stats[phase_name] = { + "total_time": sum(times), + "avg_time": sum(times) / len(times), + "max_time": max(times), + "min_time": min(times), + "occurrences": len(times), + "percentage": (sum(times) / max(total_time, 1)) * 100, + } + + # Sort by total time + sorted_phases = dict( + sorted(phase_stats.items(), key=lambda x: x[1]["total_time"], reverse=True) + ) + + overview_data = { + "totals": { + "compilation_time": total_time, + "unique_phases": len(phase_stats), + "compilation_units": len(unit_times), + }, + "phase_breakdown": sorted_phases, + "unit_times": dict( + sorted(unit_times.items(), key=lambda x: x[1], reverse=True) + ), + "top_time_consumers": dict(list(sorted_phases.items())[:5]), + } + + return APIResponse.success(overview_data) + + def handle_phases( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/compilation-phases/phases - Detailed analysis by individual phases""" + parsed_data = self.get_parsed_data() + + phases_data = defaultdict(lambda: {"times": [], "units": set(), "files": []}) + + for unit_name, unit_data in parsed_data.items(): + if FileType.COMPILATION_PHASES in unit_data: + for parsed_file in unit_data[FileType.COMPILATION_PHASES]: + if ( + isinstance(parsed_file.data, dict) + and "phases" in parsed_file.data + ): + for phase in parsed_file.data["phases"]: + if phase.get("duration") is not None: + phase_name = phase["name"] + duration = phase["duration"] + + phases_data[phase_name]["times"].append(duration) + phases_data[phase_name]["units"].add(unit_name) + phases_data[phase_name]["files"].append( + { + "unit": unit_name, + "file": parsed_file.file_path, + "duration": duration, + "info": phase.get("info", ""), + } + ) + + # Convert to detailed statistics + result = {} + for phase_name, data in phases_data.items(): + times = data["times"] + if times: + result[phase_name] = { + "statistics": { + "count": len(times), + "total_time": sum(times), + "average_time": sum(times) / len(times), + "median_time": sorted(times)[len(times) // 2], + "max_time": max(times), + "min_time": min(times), + "std_deviation": self._calculate_std_dev(times), + }, + "distribution": { + "units_involved": len(data["units"]), + "files_processed": len(data["files"]), + }, + "performance_insights": self._analyze_phase_performance(times), + "slowest_instances": sorted( + data["files"], key=lambda x: x["duration"], reverse=True + )[:3], + } + + # Sort by total time + sorted_result = dict( + sorted( + result.items(), + key=lambda x: x[1]["statistics"]["total_time"], + reverse=True, + ) + ) + + return APIResponse.success( + {"phases": sorted_result, "total_phases_analyzed": len(sorted_result)} + ) + + def handle_bottlenecks( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/compilation-phases/bottlenecks - Identify compilation bottlenecks""" + parsed_data = self.get_parsed_data() + + all_phase_times = [] + phase_outliers = defaultdict(list) + unit_bottlenecks = {} + + for unit_name, unit_data in parsed_data.items(): + if FileType.COMPILATION_PHASES in unit_data: + unit_phases = [] + + for parsed_file in unit_data[FileType.COMPILATION_PHASES]: + if ( + isinstance(parsed_file.data, dict) + and "phases" in parsed_file.data + ): + for phase in parsed_file.data["phases"]: + if phase.get("duration") is not None: + duration = phase["duration"] + phase_name = phase["name"] + + all_phase_times.append(duration) + unit_phases.append( + { + "name": phase_name, + "duration": duration, + "file": parsed_file.file_path, + } + ) + + if unit_phases: + # Find bottlenecks in this unit + unit_phases.sort(key=lambda x: x["duration"], reverse=True) + total_time = sum(p["duration"] for p in unit_phases) + + unit_bottlenecks[unit_name] = { + "total_time": total_time, + "slowest_phase": unit_phases[0] if unit_phases else None, + "top_3_phases": unit_phases[:3], + "phase_distribution": self._calculate_phase_distribution( + unit_phases + ), + } + + # Calculate global thresholds for outlier detection + if all_phase_times: + all_phase_times.sort() + p95_threshold = all_phase_times[int(len(all_phase_times) * 0.95)] + p99_threshold = all_phase_times[int(len(all_phase_times) * 0.99)] + + # Find outliers + for unit_name, unit_data in parsed_data.items(): + if FileType.COMPILATION_PHASES in unit_data: + for parsed_file in unit_data[FileType.COMPILATION_PHASES]: + if ( + isinstance(parsed_file.data, dict) + and "phases" in parsed_file.data + ): + for phase in parsed_file.data["phases"]: + if phase.get("duration") is not None: + duration = phase["duration"] + + if duration >= p99_threshold: + phase_outliers["p99"].append( + { + "unit": unit_name, + "phase": phase["name"], + "duration": duration, + "file": parsed_file.file_path, + } + ) + elif duration >= p95_threshold: + phase_outliers["p95"].append( + { + "unit": unit_name, + "phase": phase["name"], + "duration": duration, + "file": parsed_file.file_path, + } + ) + + bottlenecks_data = { + "global_thresholds": { + "p95_threshold": p95_threshold if all_phase_times else 0, + "p99_threshold": p99_threshold if all_phase_times else 0, + }, + "outliers": dict(phase_outliers), + "unit_bottlenecks": dict( + sorted( + unit_bottlenecks.items(), + key=lambda x: x[1]["total_time"], + reverse=True, + ) + ), + "recommendations": self._generate_bottleneck_recommendations( + unit_bottlenecks, phase_outliers + ), + } + + return APIResponse.success(bottlenecks_data) + + def handle_trends( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/compilation-phases/trends - Compilation time trends and patterns""" + parsed_data = self.get_parsed_data() + + # For this endpoint, we'll analyze patterns across units + # This needs more work to identify trends over time + phase_consistency = defaultdict(list) + unit_patterns = {} + + for unit_name, unit_data in parsed_data.items(): + if FileType.COMPILATION_PHASES in unit_data: + unit_phase_times = defaultdict(list) + + for parsed_file in unit_data[FileType.COMPILATION_PHASES]: + if ( + isinstance(parsed_file.data, dict) + and "phases" in parsed_file.data + ): + for phase in parsed_file.data["phases"]: + if phase.get("duration") is not None: + phase_name = phase["name"] + duration = phase["duration"] + + unit_phase_times[phase_name].append(duration) + phase_consistency[phase_name].append(duration) + + # Analyze patterns for this unit + if unit_phase_times: + unit_patterns[unit_name] = self._analyze_unit_patterns( + unit_phase_times + ) + + # Calculate consistency metrics + consistency_metrics = {} + for phase_name, times in phase_consistency.items(): + if len(times) > 1: + avg_time = sum(times) / len(times) + variance = sum((t - avg_time) ** 2 for t in times) / len(times) + coefficient_of_variation = ( + (variance**0.5) / avg_time if avg_time > 0 else 0 + ) + + consistency_metrics[phase_name] = { + "coefficient_of_variation": coefficient_of_variation, + "consistency_rating": ( + "high" + if coefficient_of_variation < 0.2 + else "medium" + if coefficient_of_variation < 0.5 + else "low" + ), + "sample_size": len(times), + } + + trends_data = { + "phase_consistency": consistency_metrics, + "unit_patterns": unit_patterns, + "insights": self._generate_trend_insights( + consistency_metrics, unit_patterns + ), + } + + return APIResponse.success(trends_data) + + def handle_bindings( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/compilation-phases/bindings - Compilation tool bindings from -ccc-print-bindings""" + parsed_data = self.get_parsed_data() + + all_bindings = [] + tool_counts = defaultdict(int) + target_counts = defaultdict(int) + unit_summaries = {} + + for unit_name, unit_data in parsed_data.items(): + if FileType.COMPILATION_PHASES in unit_data: + unit_bindings = [] + unit_tool_counts = defaultdict(int) + + for parsed_file in unit_data[FileType.COMPILATION_PHASES]: + if ( + isinstance(parsed_file.data, dict) + and "bindings" in parsed_file.data + ): + bindings = parsed_file.data["bindings"] + unit_bindings.extend(bindings) + all_bindings.extend(bindings) + + for binding in bindings: + tool = binding.get("tool", "unknown") + target = binding.get("target", "unknown") + + tool_counts[tool] += 1 + target_counts[target] += 1 + unit_tool_counts[tool] += 1 + + if unit_bindings: + unit_summaries[unit_name] = { + "total_bindings": len(unit_bindings), + "unique_tools": len(unit_tool_counts), + "tool_counts": dict(unit_tool_counts), + "bindings": unit_bindings, + } + + # Sort tools by usage count + sorted_tool_counts = dict( + sorted(tool_counts.items(), key=lambda x: x[1], reverse=True) + ) + + sorted_target_counts = dict( + sorted(target_counts.items(), key=lambda x: x[1], reverse=True) + ) + + bindings_data = { + "summary": { + "total_bindings": len(all_bindings), + "unique_tools": len(tool_counts), + "unique_targets": len(target_counts), + "compilation_units": len(unit_summaries), + }, + "tool_counts": sorted_tool_counts, + "target_counts": sorted_target_counts, + "unit_summaries": unit_summaries, + "all_bindings": all_bindings, + } + + return APIResponse.success(bindings_data) + + def _calculate_std_dev(self, values: List[float]) -> float: + """Calculate standard deviation""" + if len(values) <= 1: + return 0.0 + + mean = sum(values) / len(values) + variance = sum((x - mean) ** 2 for x in values) / len(values) + return variance**0.5 + + def _analyze_phase_performance(self, times: List[float]) -> Dict[str, Any]: + """Analyze performance characteristics of a phase""" + if not times: + return {} + + avg_time = sum(times) / len(times) + max_time = max(times) + + return { + "variability": "high" if max_time > avg_time * 2 else "low", + "performance_rating": "fast" if avg_time < 100 else "slow", + "consistency": "consistent" if max_time < avg_time * 1.5 else "variable", + } + + def _calculate_phase_distribution( + self, phases: List[Dict[str, Any]] + ) -> Dict[str, float]: + """Calculate time distribution across phases""" + total_time = sum(p["duration"] for p in phases) + if total_time == 0: + return {} + + distribution = {} + for phase in phases: + phase_name = phase["name"] + percentage = (phase["duration"] / total_time) * 100 + if phase_name not in distribution: + distribution[phase_name] = 0 + distribution[phase_name] += percentage + + return distribution + + def _generate_bottleneck_recommendations( + self, unit_bottlenecks: Dict, outliers: Dict + ) -> List[str]: + """Generate recommendations for addressing bottlenecks""" + recommendations = [] + + if outliers.get("p99"): + recommendations.append( + "Consider investigating phases with >99th percentile timing" + ) + + slow_units = [ + name + for name, data in unit_bottlenecks.items() + if data.get("total_time", 0) > 1000 + ] # > 1 second + + if slow_units: + recommendations.append( + f"Units with high compile times: {', '.join(slow_units[:3])}" + ) + + return recommendations + + def _analyze_unit_patterns(self, unit_phase_times: Dict) -> Dict[str, Any]: + """Analyze compilation patterns for a unit""" + total_phases = len(unit_phase_times) + dominant_phase = ( + max(unit_phase_times.items(), key=lambda x: sum(x[1])) + if unit_phase_times + else None + ) + + return { + "total_unique_phases": total_phases, + "dominant_phase": dominant_phase[0] if dominant_phase else None, + "phase_count": sum(len(times) for times in unit_phase_times.values()), + } + + def _generate_trend_insights( + self, consistency_metrics: Dict, unit_patterns: Dict + ) -> List[str]: + """Generate insights about compilation trends""" + insights = [] + + consistent_phases = [ + name + for name, metrics in consistency_metrics.items() + if metrics["consistency_rating"] == "high" + ] + + if consistent_phases: + insights.append( + f"Highly consistent phases: {', '.join(consistent_phases[:3])}" + ) + + variable_phases = [ + name + for name, metrics in consistency_metrics.items() + if metrics["consistency_rating"] == "low" + ] + + if variable_phases: + insights.append( + f"Variable phases needing attention: {', '.join(variable_phases[:3])}" + ) + + return insights diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/diagnostics_api.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/diagnostics_api.py new file mode 100644 index 0000000000000..aa5e5a4930886 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/diagnostics_api.py @@ -0,0 +1,310 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import os +import sys +from collections import defaultdict, Counter +from typing import Dict, Any +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent.parent +sys.path.insert(0, str(tools_dir)) + +from common.models import FileType, Diagnostic +from ..base import BaseEndpoint, APIResponse + + +class DiagnosticsEndpoint(BaseEndpoint): + """Specialized endpoints for compiler diagnostics analysis""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + """Route requests to specific handlers based on path""" + if len(path_parts) >= 3: + sub_endpoint = path_parts[2] + else: + sub_endpoint = "overview" + + method_name = f"handle_{sub_endpoint.replace('-', '_')}" + + if hasattr(self, method_name): + handler_method = getattr(self, method_name) + return handler_method(path_parts, query_params) + else: + available_methods = [ + method[7:].replace("_", "-") + for method in dir(self) + if method.startswith("handle_") and method != "handle" + ] + + return APIResponse.error( + f"Sub-endpoint '{sub_endpoint}' not available. " + f"Available: {', '.join(available_methods)}", + 404, + ) + + def handle_overview( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/diagnostics/overview - Overall diagnostics statistics""" + parsed_data = self.get_parsed_data() + + level_counts = Counter() + file_counts = Counter() + total_diagnostics = 0 + + for unit_name, unit_data in parsed_data.items(): + if FileType.DIAGNOSTICS in unit_data: + for parsed_file in unit_data[FileType.DIAGNOSTICS]: + if isinstance(parsed_file.data, list): + total_diagnostics += len(parsed_file.data) + + for diagnostic in parsed_file.data: + if isinstance(diagnostic, Diagnostic): + level_counts[diagnostic.level] += 1 + + if diagnostic.location and diagnostic.location.file: + file_counts[diagnostic.location.file] += 1 + + overview_data = { + "totals": { + "diagnostics": total_diagnostics, + "files_with_issues": len(file_counts), + "error_rate": level_counts.get("error", 0) + / max(total_diagnostics, 1) + * 100, + "warning_rate": level_counts.get("warning", 0) + / max(total_diagnostics, 1) + * 100, + }, + "by_level": dict(level_counts), + "top_problematic_files": dict(file_counts.most_common(10)), + } + + return APIResponse.success(overview_data) + + def handle_by_level( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/diagnostics/levels - Analysis by diagnostic levels (error, warning, note)""" + parsed_data = self.get_parsed_data() + + levels_data = defaultdict( + lambda: {"count": 0, "files": set(), "messages": Counter(), "examples": []} + ) + + for unit_name, unit_data in parsed_data.items(): + if FileType.DIAGNOSTICS in unit_data: + for parsed_file in unit_data[FileType.DIAGNOSTICS]: + if isinstance(parsed_file.data, list): + for diagnostic in parsed_file.data: + if isinstance(diagnostic, Diagnostic): + level = diagnostic.level + levels_data[level]["count"] += 1 + + if diagnostic.location and diagnostic.location.file: + levels_data[level]["files"].add( + diagnostic.location.file + ) + + # Count similar messages + message_key = diagnostic.message[ + :50 + ] # First 50 chars as key + levels_data[level]["messages"][message_key] += 1 + + # Keep examples + if len(levels_data[level]["examples"]) < 5: + example = { + "message": diagnostic.message, + "location": { + "file": ( + diagnostic.location.file + if diagnostic.location + else None + ), + "line": ( + diagnostic.location.line + if diagnostic.location + else None + ), + "column": ( + diagnostic.location.column + if diagnostic.location + else None + ), + }, + "code": diagnostic.code, + } + levels_data[level]["examples"].append(example) + + # Convert to serializable format + result = {} + for level, data in levels_data.items(): + result[level] = { + "count": data["count"], + "unique_files": len(data["files"]), + "common_messages": dict(data["messages"].most_common(5)), + "examples": data["examples"], + } + + return APIResponse.success({"levels": result, "total_levels": len(result)}) + + def handle_files( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/diagnostics/files - Analysis by files with issues""" + parsed_data = self.get_parsed_data() + + files_data = defaultdict( + lambda: { + "diagnostics": [], + "level_counts": Counter(), + "lines_with_issues": set(), + } + ) + + for unit_name, unit_data in parsed_data.items(): + if FileType.DIAGNOSTICS in unit_data: + for parsed_file in unit_data[FileType.DIAGNOSTICS]: + if isinstance(parsed_file.data, list): + for diagnostic in parsed_file.data: + if ( + isinstance(diagnostic, Diagnostic) + and diagnostic.location + and diagnostic.location.file + ): + file_path = diagnostic.location.file + + files_data[file_path]["level_counts"][ + diagnostic.level + ] += 1 + + if diagnostic.location.line: + files_data[file_path]["lines_with_issues"].add( + diagnostic.location.line + ) + + diagnostic_info = { + "level": diagnostic.level, + "message": diagnostic.message, + "line": diagnostic.location.line, + "column": diagnostic.location.column, + "code": diagnostic.code, + } + files_data[file_path]["diagnostics"].append( + diagnostic_info + ) + + # Convert to serializable format + result = {} + for file_path, data in files_data.items(): + total_issues = sum(data["level_counts"].values()) + result[file_path] = { + "file_name": os.path.basename(file_path), + "total_diagnostics": total_issues, + "level_breakdown": dict(data["level_counts"]), + "lines_affected": len(data["lines_with_issues"]), + "diagnostics": sorted( + data["diagnostics"], + key=lambda x: (x.get("line", 0), x.get("column", 0)), + ), + } + + # Sort by total diagnostics count + sorted_files = dict( + sorted( + result.items(), key=lambda x: x[1]["total_diagnostics"], reverse=True + ) + ) + + return APIResponse.success( + {"files": sorted_files, "total_files_with_issues": len(sorted_files)} + ) + + def handle_patterns( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/diagnostics/patterns - Common diagnostic patterns and trends""" + parsed_data = self.get_parsed_data() + + message_patterns = Counter() + code_patterns = Counter() + line_distribution = defaultdict(list) + + for unit_name, unit_data in parsed_data.items(): + if FileType.DIAGNOSTICS in unit_data: + for parsed_file in unit_data[FileType.DIAGNOSTICS]: + if isinstance(parsed_file.data, list): + for diagnostic in parsed_file.data: + if isinstance(diagnostic, Diagnostic): + # Pattern analysis on messages + words = diagnostic.message.lower().split() + if len(words) >= 2: + pattern = " ".join( + words[:3] + ) # First 3 words as pattern + message_patterns[pattern] += 1 + + # Code patterns + if diagnostic.code: + code_patterns[diagnostic.code] += 1 + + # Line distribution for hotspot analysis + if ( + diagnostic.location + and diagnostic.location.file + and diagnostic.location.line + ): + line_distribution[diagnostic.location.file].append( + diagnostic.location.line + ) + + # Find line clusters (areas with many diagnostics) + line_clusters = {} + for file_path, lines in line_distribution.items(): + if len(lines) > 1: + lines.sort() + clusters = [] + current_cluster = [lines[0]] + + for line in lines[1:]: + if line - current_cluster[-1] <= 5: # Within 5 lines + current_cluster.append(line) + else: + if len(current_cluster) >= 2: + clusters.append( + { + "start_line": min(current_cluster), + "end_line": max(current_cluster), + "diagnostic_count": len(current_cluster), + } + ) + current_cluster = [line] + + if len(current_cluster) >= 2: + clusters.append( + { + "start_line": min(current_cluster), + "end_line": max(current_cluster), + "diagnostic_count": len(current_cluster), + } + ) + + if clusters: + line_clusters[os.path.basename(file_path)] = clusters + + patterns_data = { + "common_message_patterns": dict(message_patterns.most_common(10)), + "common_diagnostic_codes": dict(code_patterns.most_common(10)), + "line_clusters": line_clusters, + "total_patterns_found": len(message_patterns) + len(code_patterns), + } + + return APIResponse.success(patterns_data) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/remarks_api.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/remarks_api.py new file mode 100644 index 0000000000000..2935584a10b84 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/remarks_api.py @@ -0,0 +1,263 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import os +import sys +from collections import defaultdict, Counter +from typing import Dict, Any, List +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent.parent +sys.path.insert(0, str(tools_dir)) + +from common.models import FileType, Remark +from ..base import APIResponse +from .base_specialized import BaseSpecializedEndpoint + + +class RemarksEndpoint(BaseSpecializedEndpoint): + """Specialized endpoints for optimization remarks analysis""" + + def handle_overview( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/remarks/overview - Overall remarks statistics""" + parsed_data = self.get_parsed_data() + + total_remarks = 0 + pass_distribution = Counter() + function_distribution = Counter() + location_distribution = defaultdict(int) + + for unit_name, unit_data in parsed_data.items(): + if FileType.REMARKS in unit_data: + for parsed_file in unit_data[FileType.REMARKS]: + if isinstance(parsed_file.data, list): + total_remarks += len(parsed_file.data) + + for remark in parsed_file.data: + if isinstance(remark, Remark): + pass_distribution[remark.pass_name] += 1 + function_distribution[remark.function] += 1 + + if remark.location and remark.location.file: + location_distribution[remark.location.file] += 1 + + overview_data = { + "totals": { + "remarks": total_remarks, + "unique_passes": len(pass_distribution), + "unique_functions": len(function_distribution), + "source_files": len(location_distribution), + }, + "top_passes": dict(pass_distribution.most_common(10)), + "top_functions": dict(function_distribution.most_common(10)), + "top_files": dict( + sorted(location_distribution.items(), key=lambda x: x[1], reverse=True)[ + :10 + ] + ), + } + + return APIResponse.success(overview_data) + + def handle_passes( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/remarks/passes - Analysis by optimization passes""" + parsed_data = self.get_parsed_data() + + passes_data = defaultdict( + lambda: {"count": 0, "functions": set(), "files": set(), "examples": []} + ) + + for unit_name, unit_data in parsed_data.items(): + if FileType.REMARKS in unit_data: + for parsed_file in unit_data[FileType.REMARKS]: + if isinstance(parsed_file.data, list): + for remark in parsed_file.data: + if isinstance(remark, Remark): + pass_name = remark.pass_name + passes_data[pass_name]["count"] += 1 + passes_data[pass_name]["functions"].add(remark.function) + + if remark.location and remark.location.file: + passes_data[pass_name]["files"].add( + remark.location.file + ) + + # Keep a few examples + if len(passes_data[pass_name]["examples"]) < 3: + example = { + "function": remark.function, + "message": ( + remark.message[:100] + "..." + if len(remark.message) > 100 + else remark.message + ), + "location": { + "file": ( + remark.location.file + if remark.location + else None + ), + "line": ( + remark.location.line + if remark.location + else None + ), + }, + } + passes_data[pass_name]["examples"].append(example) + + # Convert sets to counts for JSON serialization + result = {} + for pass_name, data in passes_data.items(): + result[pass_name] = { + "count": data["count"], + "unique_functions": len(data["functions"]), + "unique_files": len(data["files"]), + "examples": data["examples"], + } + + return APIResponse.success({"passes": result, "total_passes": len(result)}) + + def handle_functions( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/remarks/functions - Analysis by functions""" + parsed_data = self.get_parsed_data() + + functions_data = defaultdict( + lambda: { + "remarks_count": 0, + "passes": set(), + "locations": set(), + "messages": [], + } + ) + + for unit_name, unit_data in parsed_data.items(): + if FileType.REMARKS in unit_data: + for parsed_file in unit_data[FileType.REMARKS]: + if isinstance(parsed_file.data, list): + for remark in parsed_file.data: + if isinstance(remark, Remark): + func_name = remark.function + functions_data[func_name]["remarks_count"] += 1 + functions_data[func_name]["passes"].add( + remark.pass_name + ) + + if remark.location: + loc_str = ( + f"{remark.location.file}:{remark.location.line}" + ) + functions_data[func_name]["locations"].add(loc_str) + + # Keep sample messages + if len(functions_data[func_name]["messages"]) < 5: + functions_data[func_name]["messages"].append( + { + "pass": remark.pass_name, + "message": ( + remark.message[:150] + "..." + if len(remark.message) > 150 + else remark.message + ), + } + ) + + # Convert to serializable format + result = {} + for func_name, data in functions_data.items(): + result[func_name] = { + "remarks_count": data["remarks_count"], + "unique_passes": len(data["passes"]), + "unique_locations": len(data["locations"]), + "passes": list(data["passes"]), + "sample_messages": data["messages"], + } + + # Sort by remarks count + sorted_functions = dict( + sorted(result.items(), key=lambda x: x[1]["remarks_count"], reverse=True) + ) + + return APIResponse.success( + {"functions": sorted_functions, "total_functions": len(sorted_functions)} + ) + + def handle_hotspots( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/remarks/hotspots - Find optimization hotspots""" + parsed_data = self.get_parsed_data() + + file_hotspots = defaultdict( + lambda: { + "remarks_count": 0, + "line_distribution": defaultdict(int), + "passes": set(), + "functions": set(), + } + ) + + for unit_name, unit_data in parsed_data.items(): + if FileType.REMARKS in unit_data: + for parsed_file in unit_data[FileType.REMARKS]: + if isinstance(parsed_file.data, list): + for remark in parsed_file.data: + if ( + isinstance(remark, Remark) + and remark.location + and remark.location.file + ): + file_path = remark.location.file + file_hotspots[file_path]["remarks_count"] += 1 + + if remark.location.line: + file_hotspots[file_path]["line_distribution"][ + remark.location.line + ] += 1 + + file_hotspots[file_path]["passes"].add(remark.pass_name) + file_hotspots[file_path]["functions"].add( + remark.function + ) + + # Convert to serializable format and find top hotspots + hotspots = [] + for file_path, data in file_hotspots.items(): + hotspot = { + "file": file_path, + "file_name": os.path.basename(file_path), + "remarks_count": data["remarks_count"], + "unique_passes": len(data["passes"]), + "unique_functions": len(data["functions"]), + "hot_lines": dict( + sorted( + data["line_distribution"].items(), + key=lambda x: x[1], + reverse=True, + )[:10] + ), + } + hotspots.append(hotspot) + + # Sort by remarks count + hotspots.sort(key=lambda x: x["remarks_count"], reverse=True) + + return APIResponse.success( + { + "hotspots": hotspots[:20], # Top 20 hotspots + "total_files_with_remarks": len(hotspots), + } + ) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/runtime_trace_api.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/runtime_trace_api.py new file mode 100644 index 0000000000000..964b834730c5d --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/runtime_trace_api.py @@ -0,0 +1,269 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import sys +from collections import defaultdict, Counter +from typing import Dict, Any, List +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent.parent +sys.path.insert(0, str(tools_dir)) + +from common.models import FileType, TraceEvent +from ..base import BaseEndpoint, APIResponse + + +class RuntimeTraceEndpoint(BaseEndpoint): + """Specialized endpoints for runtime trace analysis (profile.json format)""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + """Route requests to specific handlers based on path""" + if len(path_parts) >= 3: + sub_endpoint = path_parts[2] + else: + sub_endpoint = "overview" + + method_name = f"handle_{sub_endpoint.replace('-', '_')}" + + if hasattr(self, method_name): + handler_method = getattr(self, method_name) + return handler_method(path_parts, query_params) + else: + available_methods = [ + method[7:].replace("_", "-") + for method in dir(self) + if method.startswith("handle_") and method != "handle" + ] + + return APIResponse.error( + f"Sub-endpoint '{sub_endpoint}' not available. " + f"Available: {', '.join(available_methods)}", + 404, + ) + + def handle_overview( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/runtime-trace/overview - Overall runtime timing statistics""" + parsed_data = self.get_parsed_data() + + total_events = 0 + event_categories = Counter() + event_phases = Counter() + duration_events = [] + + for unit_name, unit_data in parsed_data.items(): + if FileType.RUNTIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.RUNTIME_TRACE]: + if isinstance(parsed_file.data, list): + total_events += len(parsed_file.data) + + for event in parsed_file.data: + if isinstance(event, TraceEvent): + event_categories[event.category] += 1 + event_phases[event.phase] += 1 + + if event.duration is not None: + duration_events.append(event.duration) + + # Calculate timing statistics + timing_stats = {} + if duration_events: + duration_events.sort() + timing_stats = { + "total_duration": sum(duration_events), + "average_duration": sum(duration_events) / len(duration_events), + "median_duration": duration_events[len(duration_events) // 2], + "p95_duration": duration_events[int(len(duration_events) * 0.95)], + "max_duration": max(duration_events), + "events_with_duration": len(duration_events), + } + + overview_data = { + "totals": { + "events": total_events, + "categories": len(event_categories), + "unique_phases": len(event_phases), + }, + "timing_statistics": timing_stats, + "category_distribution": dict(event_categories.most_common(10)), + "phase_distribution": dict(event_phases.most_common(10)), + } + + return APIResponse.success(overview_data) + + def handle_flamegraph( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/runtime-trace/flamegraph - Get flamegraph data for runtime time order view""" + parsed_data = self.get_parsed_data() + unit_name = query_params.get("unit", [None])[0] + + if unit_name and unit_name in parsed_data: + unit_data = {unit_name: parsed_data[unit_name]} + else: + unit_data = parsed_data + + all_stacks = [] + + for unit, data in unit_data.items(): + if FileType.RUNTIME_TRACE in data: + for parsed_file in data[FileType.RUNTIME_TRACE]: + if isinstance(parsed_file.data, list): + from common.parsers.runtime_trace_parser import ( + RuntimeTraceParser, + ) + + parser = RuntimeTraceParser() + flamegraph_data = parser.get_flamegraph_data(parsed_file.data) + + for sample in flamegraph_data.get("samples", []): + sample["unit"] = unit + sample["source"] = "runtime" # Mark as runtime data + all_stacks.append(sample) + + # Sort by timestamp for time order view + all_stacks.sort(key=lambda x: x.get("timestamp", 0)) + + return APIResponse.success( + {"samples": all_stacks, "total_samples": len(all_stacks)} + ) + + def handle_sandwich( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/runtime-trace/sandwich - Get sandwich view data (aggregated by function)""" + parsed_data = self.get_parsed_data() + unit_name = query_params.get("unit", [None])[0] + + if unit_name and unit_name in parsed_data: + unit_data = {unit_name: parsed_data[unit_name]} + else: + unit_data = parsed_data + + all_functions = [] + + for unit, data in unit_data.items(): + if FileType.RUNTIME_TRACE in data: + for parsed_file in data[FileType.RUNTIME_TRACE]: + if isinstance(parsed_file.data, list): + from common.parsers.runtime_trace_parser import ( + RuntimeTraceParser, + ) + + parser = RuntimeTraceParser() + sandwich_data = parser.get_sandwich_data(parsed_file.data) + + for func in sandwich_data.get("functions", []): + func["unit"] = unit + func["source"] = "runtime" # Mark as runtime data + all_functions.append(func) + + # Merge functions with same name across units + function_map = {} + for func in all_functions: + name = func["name"] + if name not in function_map: + function_map[name] = { + "name": name, + "total_time": 0, + "call_count": 0, + "category": func.get("category", "Runtime"), + "units": [], + "source": "runtime", + } + + function_map[name]["total_time"] += func["total_time"] + function_map[name]["call_count"] += func["call_count"] + function_map[name]["units"].append(func["unit"]) + + # Calculate averages and sort + merged_functions = [] + for func_data in function_map.values(): + if func_data["call_count"] > 0: + func_data["avg_time"] = ( + func_data["total_time"] / func_data["call_count"] + ) + else: + func_data["avg_time"] = 0 + merged_functions.append(func_data) + + merged_functions.sort(key=lambda x: x["total_time"], reverse=True) + + return APIResponse.success( + {"functions": merged_functions, "total_functions": len(merged_functions)} + ) + + def handle_hotspots( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/runtime-trace/hotspots - Find runtime performance hotspots""" + parsed_data = self.get_parsed_data() + + event_durations = defaultdict(list) + category_times = defaultdict(float) + thread_times = defaultdict(float) + + for unit_name, unit_data in parsed_data.items(): + if FileType.RUNTIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.RUNTIME_TRACE]: + if isinstance(parsed_file.data, list): + for event in parsed_file.data: + if ( + isinstance(event, TraceEvent) + and event.duration is not None + ): + event_durations[event.name].append(event.duration) + category_times[ + event.category or "Runtime" + ] += event.duration + + if event.tid is not None: + thread_times[event.tid] += event.duration + + # Find hotspots by event name + event_hotspots = [] + for event_name, durations in event_durations.items(): + if durations: + total_time = sum(durations) + event_hotspots.append( + { + "event_name": event_name, + "total_time": total_time, + "average_time": total_time / len(durations), + "max_time": max(durations), + "occurrences": len(durations), + "percentage_of_total": 0, # Will be calculated below + "source": "runtime", + } + ) + + # Sort by total time + event_hotspots.sort(key=lambda x: x["total_time"], reverse=True) + + # Calculate percentages + total_trace_time = sum(h["total_time"] for h in event_hotspots) + for hotspot in event_hotspots: + hotspot["percentage_of_total"] = ( + hotspot["total_time"] / max(total_trace_time, 1) + ) * 100 + + hotspots_data = { + "event_hotspots": event_hotspots[:20], # Top 20 + "category_hotspots": dict( + sorted(category_times.items(), key=lambda x: x[1], reverse=True)[:10] + ), + "thread_hotspots": dict( + sorted(thread_times.items(), key=lambda x: x[1], reverse=True)[:10] + ), + "total_trace_time": total_trace_time, + } + + return APIResponse.success(hotspots_data) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized/time_trace_api.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/time_trace_api.py new file mode 100644 index 0000000000000..6b029656750e2 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized/time_trace_api.py @@ -0,0 +1,556 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import sys +from collections import defaultdict, Counter +from typing import Dict, Any, List +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent.parent +sys.path.insert(0, str(tools_dir)) + +from common.models import FileType, TraceEvent +from ..base import BaseEndpoint, APIResponse + + +class TimeTraceEndpoint(BaseEndpoint): + """Specialized endpoints for time trace analysis (Chrome trace format)""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + """Route requests to specific handlers based on path""" + if len(path_parts) >= 3: + sub_endpoint = path_parts[2] + else: + sub_endpoint = "overview" + + method_name = f"handle_{sub_endpoint.replace('-', '_')}" + + if hasattr(self, method_name): + handler_method = getattr(self, method_name) + return handler_method(path_parts, query_params) + else: + available_methods = [ + method[7:].replace("_", "-") + for method in dir(self) + if method.startswith("handle_") and method != "handle" + ] + + return APIResponse.error( + f"Sub-endpoint '{sub_endpoint}' not available. " + f"Available: {', '.join(available_methods)}", + 404, + ) + + def handle_overview( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/overview - Overall timing statistics""" + parsed_data = self.get_parsed_data() + + total_events = 0 + event_categories = Counter() + event_phases = Counter() + duration_events = [] + + for unit_name, unit_data in parsed_data.items(): + if FileType.TIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + total_events += len(parsed_file.data) + + for event in parsed_file.data: + if isinstance(event, TraceEvent): + event_categories[event.category] += 1 + event_phases[event.phase] += 1 + + if event.duration is not None: + duration_events.append(event.duration) + + # Calculate timing statistics + timing_stats = {} + if duration_events: + duration_events.sort() + timing_stats = { + "total_duration": sum(duration_events), + "average_duration": sum(duration_events) / len(duration_events), + "median_duration": duration_events[len(duration_events) // 2], + "p95_duration": duration_events[int(len(duration_events) * 0.95)], + "max_duration": max(duration_events), + "events_with_duration": len(duration_events), + } + + overview_data = { + "totals": { + "events": total_events, + "categories": len(event_categories), + "unique_phases": len(event_phases), + }, + "timing_statistics": timing_stats, + "category_distribution": dict(event_categories.most_common(10)), + "phase_distribution": dict(event_phases.most_common(10)), + } + + return APIResponse.success(overview_data) + + def handle_timeline( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/timeline - Timeline analysis of events""" + parsed_data = self.get_parsed_data() + + timeline_data = [] + processes = set() + threads = set() + + for unit_name, unit_data in parsed_data.items(): + if FileType.TIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + for event in parsed_file.data: + if isinstance(event, TraceEvent): + if event.pid is not None: + processes.add(event.pid) + if event.tid is not None: + threads.add(event.tid) + + timeline_entry = { + "unit": unit_name, + "timestamp": event.timestamp, + "name": event.name, + "category": event.category, + "phase": event.phase, + "duration": event.duration, + "pid": event.pid, + "tid": event.tid, + "args": event.args if event.args else {}, + } + timeline_data.append(timeline_entry) + + # Sort by timestamp + timeline_data.sort(key=lambda x: x["timestamp"]) + + # Limit to reasonable size for API response + max_events = int(query_params.get("limit", ["1000"])[0]) + timeline_data = timeline_data[:max_events] + + timeline_response = { + "timeline": timeline_data, + "metadata": { + "total_events_shown": len(timeline_data), + "unique_processes": len(processes), + "unique_threads": len(threads), + "time_range": { + "start": ( + min(e["timestamp"] for e in timeline_data) + if timeline_data + else 0 + ), + "end": ( + max(e["timestamp"] for e in timeline_data) + if timeline_data + else 0 + ), + }, + }, + } + + return APIResponse.success(timeline_response) + + def handle_hotspots( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/hotspots - Find performance hotspots""" + parsed_data = self.get_parsed_data() + + event_durations = defaultdict(list) + category_times = defaultdict(float) + thread_times = defaultdict(float) + + for unit_name, unit_data in parsed_data.items(): + if FileType.TIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + for event in parsed_file.data: + if ( + isinstance(event, TraceEvent) + and event.duration is not None + ): + event_durations[event.name].append(event.duration) + category_times[event.category] += event.duration + + if event.tid is not None: + thread_times[event.tid] += event.duration + + # Find hotspots by event name + event_hotspots = [] + for event_name, durations in event_durations.items(): + if durations: + total_time = sum(durations) + event_hotspots.append( + { + "event_name": event_name, + "total_time": total_time, + "average_time": total_time / len(durations), + "max_time": max(durations), + "occurrences": len(durations), + "percentage_of_total": 0, # Will be calculated below + } + ) + + # Sort by total time + event_hotspots.sort(key=lambda x: x["total_time"], reverse=True) + + # Calculate percentages + total_trace_time = sum(h["total_time"] for h in event_hotspots) + for hotspot in event_hotspots: + hotspot["percentage_of_total"] = ( + hotspot["total_time"] / max(total_trace_time, 1) + ) * 100 + + hotspots_data = { + "event_hotspots": event_hotspots[:20], # Top 20 + "category_hotspots": dict( + sorted(category_times.items(), key=lambda x: x[1], reverse=True)[:10] + ), + "thread_hotspots": dict( + sorted(thread_times.items(), key=lambda x: x[1], reverse=True)[:10] + ), + "total_trace_time": total_trace_time, + } + + return APIResponse.success(hotspots_data) + + def handle_categories( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/categories - Analysis by event categories""" + parsed_data = self.get_parsed_data() + + categories_data = defaultdict( + lambda: { + "events": [], + "total_time": 0, + "event_names": Counter(), + "phases": Counter(), + } + ) + + for unit_name, unit_data in parsed_data.items(): + if FileType.TIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + for event in parsed_file.data: + if isinstance(event, TraceEvent): + category = event.category or "uncategorized" + + categories_data[category]["event_names"][ + event.name + ] += 1 + categories_data[category]["phases"][event.phase] += 1 + + if event.duration is not None: + categories_data[category][ + "total_time" + ] += event.duration + + # Keep sample events (limited) + if len(categories_data[category]["events"]) < 5: + categories_data[category]["events"].append( + { + "name": event.name, + "duration": event.duration, + "timestamp": event.timestamp, + "args": event.args if event.args else {}, + } + ) + + # Convert to response format + result = {} + for category, data in categories_data.items(): + result[category] = { + "total_time": data["total_time"], + "unique_event_names": len(data["event_names"]), + "total_events": sum(data["event_names"].values()), + "top_events": dict(data["event_names"].most_common(5)), + "phase_distribution": dict(data["phases"]), + "sample_events": data["events"], + } + + # Sort by total time + sorted_result = dict( + sorted(result.items(), key=lambda x: x[1]["total_time"], reverse=True) + ) + + return APIResponse.success( + {"categories": sorted_result, "total_categories": len(sorted_result)} + ) + + def handle_parallelism( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/parallelism - Analyze parallelism and concurrency""" + parsed_data = self.get_parsed_data() + + thread_activity = defaultdict(list) + process_threads = defaultdict(set) + concurrent_events = [] + + for unit_name, unit_data in parsed_data.items(): + if FileType.TIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + for event in parsed_file.data: + if isinstance(event, TraceEvent): + if event.pid is not None and event.tid is not None: + process_threads[event.pid].add(event.tid) + + if event.duration is not None: + start_time = event.timestamp + end_time = event.timestamp + event.duration + + thread_activity[event.tid].append( + { + "start": start_time, + "end": end_time, + "duration": event.duration, + "event_name": event.name, + "category": event.category, + } + ) + + # Analyze thread utilization + thread_utilization = {} + for tid, activities in thread_activity.items(): + if activities: + total_active_time = sum(a["duration"] for a in activities) + activities.sort(key=lambda x: x["start"]) + + time_span = activities[-1]["end"] - activities[0]["start"] + utilization = ( + (total_active_time / max(time_span, 1)) if time_span > 0 else 0 + ) + + thread_utilization[tid] = { + "total_active_time": total_active_time, + "time_span": time_span, + "utilization_percentage": utilization * 100, + "activity_count": len(activities), + "average_activity_duration": total_active_time / len(activities), + } + + # Find overlapping events (basic concurrency analysis) + overlaps = 0 + sorted_activities = [] + for activities in thread_activity.values(): + sorted_activities.extend(activities) + + sorted_activities.sort(key=lambda x: x["start"]) + + for i in range(len(sorted_activities) - 1): + current = sorted_activities[i] + next_activity = sorted_activities[i + 1] + + if current["end"] > next_activity["start"]: + overlaps += 1 + + parallelism_data = { + "process_thread_mapping": { + pid: len(threads) for pid, threads in process_threads.items() + }, + "thread_utilization": dict( + sorted( + thread_utilization.items(), + key=lambda x: x[1]["utilization_percentage"], + reverse=True, + ) + ), + "concurrency_metrics": { + "total_threads": len(thread_activity), + "overlapping_activities": overlaps, + "max_threads_per_process": ( + max(len(threads) for threads in process_threads.values()) + if process_threads + else 0 + ), + }, + "insights": self._generate_parallelism_insights( + thread_utilization, process_threads + ), + } + + return APIResponse.success(parallelism_data) + + def _generate_parallelism_insights( + self, thread_utilization: Dict, process_threads: Dict + ) -> List[str]: + """Generate insights about parallelism""" + insights = [] + + if thread_utilization: + avg_utilization = sum( + t["utilization_percentage"] for t in thread_utilization.values() + ) / len(thread_utilization) + + if avg_utilization < 50: + insights.append( + "Low thread utilization detected - potential for better parallelization" + ) + elif avg_utilization > 90: + insights.append("High thread utilization - good parallelization") + + total_threads = sum(len(threads) for threads in process_threads.values()) + if total_threads > 8: + insights.append( + f"High thread count ({total_threads}) - monitor for contention" + ) + + return insights + + def handle_flamegraph( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/flamegraph - Get flamegraph data for time order view""" + parsed_data = self.get_parsed_data() + unit_name = query_params.get("unit", [None])[0] + + if unit_name and unit_name in parsed_data: + unit_data = {unit_name: parsed_data[unit_name]} + else: + unit_data = parsed_data + + all_stacks = [] + + for unit, data in unit_data.items(): + if FileType.TIME_TRACE in data: + for parsed_file in data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + from common.parsers.time_trace_parser import TimeTraceParser + + parser = TimeTraceParser() + flamegraph_data = parser.get_flamegraph_data(parsed_file.data) + + for sample in flamegraph_data.get("samples", []): + sample["unit"] = unit + all_stacks.append(sample) + + # Sort by timestamp for time order view + all_stacks.sort(key=lambda x: x.get("timestamp", 0)) + + return APIResponse.success( + {"samples": all_stacks, "total_samples": len(all_stacks)} + ) + + def handle_sandwich( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/sandwich - Get sandwich view data (aggregated by function)""" + parsed_data = self.get_parsed_data() + unit_name = query_params.get("unit", [None])[0] + + if unit_name and unit_name in parsed_data: + unit_data = {unit_name: parsed_data[unit_name]} + else: + unit_data = parsed_data + + all_functions = [] + + for unit, data in unit_data.items(): + if FileType.TIME_TRACE in data: + for parsed_file in data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + from common.parsers.time_trace_parser import TimeTraceParser + + parser = TimeTraceParser() + sandwich_data = parser.get_sandwich_data(parsed_file.data) + + for func in sandwich_data.get("functions", []): + func["unit"] = unit + all_functions.append(func) + + # Merge functions with same name across units + function_map = {} + for func in all_functions: + name = func["name"] + if name not in function_map: + function_map[name] = { + "name": name, + "total_time": 0, + "call_count": 0, + "category": func.get("category", ""), + "units": [], + } + + function_map[name]["total_time"] += func["total_time"] + function_map[name]["call_count"] += func["call_count"] + function_map[name]["units"].append(func["unit"]) + + # Calculate averages and sort + merged_functions = [] + for func_data in function_map.values(): + if func_data["call_count"] > 0: + func_data["avg_time"] = ( + func_data["total_time"] / func_data["call_count"] + ) + else: + func_data["avg_time"] = 0 + merged_functions.append(func_data) + + merged_functions.sort(key=lambda x: x["total_time"], reverse=True) + + return APIResponse.success( + {"functions": merged_functions, "total_functions": len(merged_functions)} + ) + + def handle_runtime_comparison( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """GET /api/time-trace/runtime-comparison - Compare time-trace vs runtime-trace""" + parsed_data = self.get_parsed_data() + + time_trace_data = {} + runtime_trace_data = {} + + # Collect both types of traces + for unit_name, unit_data in parsed_data.items(): + if FileType.TIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.TIME_TRACE]: + if isinstance(parsed_file.data, list): + time_trace_data[unit_name] = parsed_file.data + + if FileType.RUNTIME_TRACE in unit_data: + for parsed_file in unit_data[FileType.RUNTIME_TRACE]: + if isinstance(parsed_file.data, list): + runtime_trace_data[unit_name] = parsed_file.data + + comparison = {} + + for unit_name in set(time_trace_data.keys()) | set(runtime_trace_data.keys()): + time_events = time_trace_data.get(unit_name, []) + runtime_events = runtime_trace_data.get(unit_name, []) + + time_duration = sum(e.duration for e in time_events if e.duration) + runtime_duration = sum(e.duration for e in runtime_events if e.duration) + + comparison[unit_name] = { + "time_trace": { + "events": len(time_events), + "total_duration": time_duration, + "available": len(time_events) > 0, + }, + "runtime_trace": { + "events": len(runtime_events), + "total_duration": runtime_duration, + "available": len(runtime_events) > 0, + }, + } + + return APIResponse.success( + {"comparison": comparison, "units": list(comparison.keys())} + ) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/specialized_router.py b/llvm/tools/llvm-advisor/tools/webserver/api/specialized_router.py new file mode 100644 index 0000000000000..c6aa5aa0a3503 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/specialized_router.py @@ -0,0 +1,150 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import sys +from typing import Dict, Any, Optional +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent.parent +sys.path.insert(0, str(tools_dir)) + +from common.collector import ArtifactCollector +from .base import APIResponse + +# Import specialized endpoints +from .specialized.remarks_api import RemarksEndpoint +from .specialized.diagnostics_api import DiagnosticsEndpoint +from .specialized.compilation_phases_api import CompilationPhasesEndpoint +from .specialized.time_trace_api import TimeTraceEndpoint +from .specialized.runtime_trace_api import RuntimeTraceEndpoint +from .specialized.binary_size_api import BinarySizeEndpoint + + +class SpecializedRouter: + """Router for specialized file-type specific endpoints""" + + def __init__(self, data_dir: str, collector: ArtifactCollector): + self.data_dir = data_dir + self.collector = collector + + # Initialize specialized endpoints + self.endpoints = { + "remarks": RemarksEndpoint(data_dir, collector), + "diagnostics": DiagnosticsEndpoint(data_dir, collector), + "compilation-phases": CompilationPhasesEndpoint(data_dir, collector), + "time-trace": TimeTraceEndpoint(data_dir, collector), + "runtime-trace": RuntimeTraceEndpoint(data_dir, collector), + "binary-size": BinarySizeEndpoint(data_dir, collector), + } + + def route_request( + self, path_parts: list, query_params: Dict[str, list] + ) -> Dict[str, Any]: + """Route specialized requests to appropriate handlers""" + + if len(path_parts) < 2: + return APIResponse.invalid_request("File type required") + + file_type = path_parts[1] + + if file_type not in self.endpoints: + return APIResponse.error( + f"Specialized endpoint not available for '{file_type}'", 404 + ) + + endpoint = self.endpoints[file_type] + + # Determine sub-endpoint + if len(path_parts) >= 3: + sub_endpoint = path_parts[2] + else: + sub_endpoint = "overview" # Default to overview + + # Route to specific handler method + method_name = f"handle_{sub_endpoint.replace('-', '_')}" + + if hasattr(endpoint, method_name): + handler_method = getattr(endpoint, method_name) + return handler_method(path_parts, query_params) + else: + available_methods = [ + method[7:].replace("_", "-") + for method in dir(endpoint) + if method.startswith("handle_") and not method.startswith("handle__") + ] + + return APIResponse.error( + f"Sub-endpoint '{sub_endpoint}' not available for '{file_type}'. " + f"Available: {', '.join(available_methods)}", + 404, + ) + + def get_available_endpoints(self) -> Dict[str, Dict[str, str]]: + """Get all available specialized endpoints""" + endpoints_info = {} + + for file_type, endpoint in self.endpoints.items(): + # Get all handler methods + handlers = [ + method[7:].replace("_", "-") + for method in dir(endpoint) + if method.startswith("handle_") and not method.startswith("handle__") + ] + + endpoints_info[file_type] = { + "base_path": f"/api/{file_type}", + "available_endpoints": handlers, + "examples": [f"/api/{file_type}/{handler}" for handler in handlers[:3]], + } + + return endpoints_info + + +# Endpoint documentation for each file type +SPECIALIZED_ENDPOINTS_DOCS = { + "remarks": { + "overview": "Overall optimization remarks statistics and distribution", + "passes": "Analysis grouped by optimization passes", + "functions": "Analysis grouped by functions with remarks", + "hotspots": "Find files and locations with most optimization activity", + }, + "diagnostics": { + "overview": "Overall compiler diagnostics statistics", + "by-level": "Analysis by diagnostic levels (error, warning, note)", + "files": "Analysis by files with issues", + "patterns": "Common diagnostic patterns and trends", + }, + "compilation-phases": { + "overview": "Overall compilation timing statistics", + "phases": "Detailed analysis by individual compilation phases", + "bottlenecks": "Identify compilation bottlenecks and slow phases", + "trends": "Compilation time trends and consistency analysis", + }, + "time-trace": { + "overview": "Overall timing statistics from Chrome trace format", + "timeline": "Timeline analysis of compilation events", + "hotspots": "Find performance hotspots and slow operations", + "categories": "Analysis by event categories", + "parallelism": "Analyze parallelism and thread utilization", + }, + "runtime-trace": { + "overview": "Runtime profiling statistics", + "timeline": "Runtime event timeline analysis", + "hotspots": "Runtime performance hotspots", + "categories": "Runtime event categories", + "parallelism": "Runtime parallelism analysis", + }, + "binary-size": { + "overview": "Overall binary size statistics and breakdown", + "sections": "Detailed analysis by binary sections", + "optimization": "Size optimization opportunities and recommendations", + "comparison": "Compare sizes across compilation units", + }, +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/summary.py b/llvm/tools/llvm-advisor/tools/webserver/api/summary.py new file mode 100644 index 0000000000000..bdec50a88b30d --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/summary.py @@ -0,0 +1,31 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +from typing import Dict, Any +from .base import BaseEndpoint, APIResponse + + +class SummaryEndpoint(BaseEndpoint): + """GET /api/summary - Overall statistics summary across all compilation units""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + parsed_data = self.get_parsed_data() + stats = self.collector.get_summary_statistics(parsed_data) + + # Enhance with additional summary metrics + enhanced_stats = { + **stats, + "status": "success" if stats["errors"] == 0 else "partial_errors", + "success_rate": ( + (stats["total_files"] - stats["errors"]) / stats["total_files"] * 100 + if stats["total_files"] > 0 + else 0 + ), + } + + return APIResponse.success(enhanced_stats) diff --git a/llvm/tools/llvm-advisor/tools/webserver/api/units.py b/llvm/tools/llvm-advisor/tools/webserver/api/units.py new file mode 100644 index 0000000000000..e7f9020bdd5af --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/api/units.py @@ -0,0 +1,148 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +import os +import json +from typing import Dict, Any, List, Optional +from .base import BaseEndpoint, APIResponse + + +class UnitsEndpoint(BaseEndpoint): + """GET /api/units - List all compilation units with basic info""" + + def _load_metadata(self) -> Optional[Dict[str, Any]]: + """Load compilation unit metadata from the C++ tracking system""" + try: + metadata_path = os.path.join(self.data_dir, ".llvm-advisor-metadata.json") + if os.path.exists(metadata_path): + with open(metadata_path, "r") as f: + return json.load(f) + except Exception as e: + print(f"Warning: Failed to load metadata: {e}") + return None + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + units = self.get_compilation_units() + metadata = self._load_metadata() + metadata_units = {} + + if metadata and "units" in metadata: + for unit_meta in metadata["units"]: + metadata_units[unit_meta["name"]] = unit_meta + + unit_list = [] + + for unit in units: + unit_info = { + "name": unit.name, + "path": unit.path, + "artifact_types": [ft.value for ft in unit.artifacts.keys()], + "artifact_counts": { + ft.value: len(files) for ft, files in unit.artifacts.items() + }, + "total_files": sum(len(files) for files in unit.artifacts.values()), + } + + if hasattr(unit, "metadata") and unit.metadata: + if "run_timestamp" in unit.metadata: + unit_info["run_timestamp"] = unit.metadata["run_timestamp"] + if "available_runs" in unit.metadata: + unit_info["available_runs"] = unit.metadata["available_runs"] + if "run_path" in unit.metadata: + unit_info["run_path"] = unit.metadata["run_path"] + + if unit.name in metadata_units: + unit_meta = metadata_units[unit.name] + unit_info.update( + { + "metadata_timestamp": unit_meta.get("timestamp"), + "status": unit_meta.get("status", "unknown"), + "metadata": { + "output_path": unit_meta.get("output_path"), + "properties": unit_meta.get("properties", {}), + }, + } + ) + else: + unit_info.update( + {"timestamp": None, "status": "unknown", "metadata": {}} + ) + + unit_list.append(unit_info) + + if metadata_units: + unit_list.sort(key=lambda x: x.get("timestamp", ""), reverse=True) + + response_data = { + "units": unit_list, + "total_units": len(unit_list), + "total_files": sum(unit["total_files"] for unit in unit_list), + } + + return APIResponse.success(response_data) + + +class UnitDetailEndpoint(BaseEndpoint): + """GET /api/units/{unit_name} - Detailed information for a specific compilation unit""" + + def handle(self, path_parts: list, query_params: Dict[str, list]) -> Dict[str, Any]: + if len(path_parts) < 3: + return APIResponse.invalid_request("Unit name required") + + unit_name = path_parts[2] + parsed_data = self.get_parsed_data() + + if unit_name not in parsed_data: + return APIResponse.not_found(f"Compilation unit '{unit_name}'") + + unit_data = parsed_data[unit_name] + + response = { + "unit_name": unit_name, + "artifact_types": {}, + "summary": { + "total_artifact_types": len(unit_data), + "total_files": 0, + "total_errors": 0, + }, + } + + for file_type, parsed_files in unit_data.items(): + file_list = [] + error_count = 0 + + for parsed_file in parsed_files: + has_error = "error" in parsed_file.metadata + if has_error: + error_count += 1 + + file_info = { + "file_path": parsed_file.file_path, + "file_name": os.path.basename(parsed_file.file_path), + "file_size_bytes": parsed_file.metadata.get("file_size", 0), + "has_error": has_error, + "metadata": parsed_file.metadata, + } + + if isinstance(parsed_file.data, dict) and "summary" in parsed_file.data: + file_info["summary"] = parsed_file.data["summary"] + elif isinstance(parsed_file.data, list): + file_info["item_count"] = len(parsed_file.data) + + file_list.append(file_info) + + response["artifact_types"][file_type.value] = { + "files": file_list, + "count": len(file_list), + "errors": error_count, + } + + response["summary"]["total_files"] += len(file_list) + response["summary"]["total_errors"] += error_count + + return APIResponse.success(response) diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/css/custom.css b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/css/custom.css new file mode 100644 index 0000000000000..916dd0c3ce0b5 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/css/custom.css @@ -0,0 +1,805 @@ +/** + * LLVM Advisor Dashboard Custom Styles + * Complements Tailwind CSS with project-specific styling + */ + +/* =========================== + Custom CSS Properties + =========================== */ +:root { + --llvm-blue: #3b82f6; + --llvm-blue-dark: #2563eb; + --llvm-blue-light: #93c5fd; + --success: #10b981; + --warning: #f59e0b; + --error: #ef4444; + --gray-50: #f9fafb; + --gray-100: #f3f4f6; + --gray-200: #e5e7eb; + --gray-300: #d1d5db; + --gray-400: #9ca3af; + --gray-500: #6b7280; + --gray-600: #4b5563; + --gray-700: #374151; + --gray-800: #1f2937; + --gray-900: #111827; +} + +/* =========================== + Global Styles + =========================== */ +body { + font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; + -webkit-font-smoothing: antialiased; + -moz-osx-font-smoothing: grayscale; +} + +/* =========================== + Custom Utility Classes + =========================== */ +.bg-llvm-blue { + background-color: var(--llvm-blue); +} + +.bg-llvm-blue-dark { + background-color: var(--llvm-blue-dark); +} + +.bg-llvm-blue-light { + background-color: var(--llvm-blue-light); +} + +.text-llvm-blue { + color: var(--llvm-blue); +} + +.text-llvm-blue-dark { + color: var(--llvm-blue-dark); +} + +.border-llvm-blue { + border-color: var(--llvm-blue); +} + +.bg-success { + background-color: var(--success); +} + +.bg-warning { + background-color: var(--warning); +} + +.bg-error { + background-color: var(--error); +} + +.text-success { + color: var(--success); +} + +.text-warning { + color: var(--warning); +} + +.text-error { + color: var(--error); +} + +/* =========================== + Loading Animations + =========================== */ +.animate-pulse-slow { + animation: pulse 3s cubic-bezier(0.4, 0, 0.6, 1) infinite; +} + +.animate-fade-in { + animation: fadeIn 0.5s ease-in-out; +} + +.animate-slide-up { + animation: slideUp 0.3s ease-out; +} + +.animate-bounce-gentle { + animation: bounceGentle 2s infinite; +} + +@keyframes fadeIn { + from { + opacity: 0; + transform: translateY(10px); + } + to { + opacity: 1; + transform: translateY(0); + } +} + +@keyframes slideUp { + from { + opacity: 0; + transform: translateY(20px); + } + to { + opacity: 1; + transform: translateY(0); + } +} + +@keyframes bounceGentle { + 0%, 20%, 50%, 80%, 100% { + transform: translateY(0); + } + 40% { + transform: translateY(-5px); + } + 60% { + transform: translateY(-3px); + } +} + +/* =========================== + Component Styles + =========================== */ + +/* Loading Screen */ +#loading-screen { + background: linear-gradient(135deg, var(--gray-50) 0%, var(--gray-100) 100%); +} + +.loading-spinner { + border-top-color: var(--llvm-blue); + animation: spin 1s linear infinite; +} + +@keyframes spin { + to { + transform: rotate(360deg); + } +} + +/* Tab Navigation */ +.tab-button { + transition: all 0.2s ease-in-out; + position: relative; +} + +.tab-button:hover { + transform: translateY(-1px); +} + +.tab-button.active::after { + content: ''; + position: absolute; + bottom: -2px; + left: 0; + right: 0; + height: 2px; + background-color: var(--llvm-blue); + border-radius: 1px; +} + +.tab-content { + min-height: 500px; +} + +.tab-transition { + animation: fadeIn 0.3s ease-in-out; +} + +/* Metric Cards */ +.metric-card { + transition: all 0.2s ease-in-out; + border: 1px solid var(--gray-200); +} + +.metric-card:hover { + transform: translateY(-2px); + box-shadow: 0 10px 25px -5px rgba(0, 0, 0, 0.1), 0 10px 10px -5px rgba(0, 0, 0, 0.04); + border-color: var(--llvm-blue-light); +} + +.metric-value { + font-feature-settings: 'tnum'; + font-variant-numeric: tabular-nums; +} + +/* Charts */ +.chart-container { + position: relative; + height: 300px; + background: white; + border-radius: 8px; + padding: 16px; + box-shadow: 0 1px 3px rgba(0, 0, 0, 0.1); + border: 1px solid var(--gray-200); +} + +.chart-container canvas { + max-height: 100%; +} + +/* Alert Banner */ +.alert-banner { + transition: all 0.3s ease-in-out; + border-left-width: 4px; +} + +/* Status Indicator */ +.status-indicator { + display: flex; + align-items: center; + gap: 8px; +} + +.status-dot { + width: 8px; + height: 8px; + border-radius: 50%; + display: inline-block; +} + +.status-dot.online { + background-color: var(--success); + box-shadow: 0 0 0 2px rgba(16, 185, 129, 0.2); +} + +.status-dot.offline { + background-color: var(--error); + box-shadow: 0 0 0 2px rgba(239, 68, 68, 0.2); +} + +/* Unit Selector */ +.unit-selector { + transition: all 0.2s ease-in-out; +} + +.unit-selector:focus { + border-color: var(--llvm-blue); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.1); +} + +/* Insights Cards */ +.insight-card { + transition: all 0.2s ease-in-out; + border-left-width: 4px; +} + +.insight-card:hover { + transform: translateX(4px); +} + +.insight-card.error { + border-left-color: var(--error); + background-color: rgba(239, 68, 68, 0.05); +} + +.insight-card.warning { + border-left-color: var(--warning); + background-color: rgba(245, 158, 11, 0.05); +} + +.insight-card.info { + border-left-color: var(--llvm-blue); + background-color: rgba(59, 130, 246, 0.05); +} + +.insight-card.success { + border-left-color: var(--success); + background-color: rgba(16, 185, 129, 0.05); +} + +/* =========================== + Responsive Design + =========================== */ + +/* Mobile Adjustments */ +@media (max-width: 640px) { + .metric-card { + padding: 16px; + } + + .metric-value { + font-size: 1.5rem; + } + + .chart-container { + height: 250px; + padding: 12px; + } + + .tab-button { + padding: 8px 12px; + font-size: 14px; + } +} + +/* Tablet Adjustments */ +@media (min-width: 641px) and (max-width: 1024px) { + .chart-container { + height: 280px; + } +} + +/* Desktop Enhancements */ +@media (min-width: 1025px) { + .metric-card:hover .metric-value { + color: var(--llvm-blue); + } + + .chart-container:hover { + border-color: var(--llvm-blue-light); + } +} + +/* =========================== + Accessibility + =========================== */ + +/* Focus Styles */ +.focus\:ring-llvm:focus { + ring-color: var(--llvm-blue); + ring-opacity: 0.5; +} + +/* High Contrast Mode */ +@media (prefers-contrast: high) { + .metric-card { + border-width: 2px; + } + + .tab-button.active { + background-color: var(--llvm-blue); + color: white; + } +} + +/* Reduced Motion */ +@media (prefers-reduced-motion: reduce) { + * { + animation-duration: 0.01ms !important; + animation-iteration-count: 1 !important; + transition-duration: 0.01ms !important; + } +} + +/* Dark Mode Support (for future) */ +@media (prefers-color-scheme: dark) { + .dark-mode { + --gray-50: #1f2937; + --gray-100: #374151; + --gray-200: #4b5563; + --gray-300: #6b7280; + --gray-800: #f9fafb; + --gray-900: #f3f4f6; + } +} + +/* =========================== + Print Styles + =========================== */ +@media print { + .no-print { + display: none !important; + } + + .chart-container { + break-inside: avoid; + border: 1px solid #000; + } + + .metric-card { + border: 1px solid #000; + box-shadow: none; + } +} + +/* =========================== + Utilities + =========================== */ + +/* Text Selection */ +::selection { + background-color: var(--llvm-blue-light); + color: white; +} + +/* Scrollbar Styling */ +::-webkit-scrollbar { + width: 8px; + height: 8px; +} + +::-webkit-scrollbar-track { + background: var(--gray-100); + border-radius: 4px; +} + +::-webkit-scrollbar-thumb { + background: var(--gray-300); + border-radius: 4px; +} + +::-webkit-scrollbar-thumb:hover { + background: var(--gray-400); +} + +/* Custom Grid Classes */ +.grid-responsive { + display: grid; + gap: 1.5rem; + grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); +} + +.grid-metrics { + display: grid; + gap: 1rem; + grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); +} + +/* Custom Shadow */ +.shadow-subtle { + box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.05); +} + +.shadow-card { + box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06); +} + +/* =========================== + Code Explorer Styles + =========================== */ + +/* Code Container Styles */ +.code-container { + font-family: 'Monaco', 'Menlo', 'Ubuntu Mono', monospace; + font-size: 13px; + line-height: 1.5; + background-color: #1e1e1e; + color: #d4d4d4; + overflow: auto; + border-radius: 4px; +} + +.code-container pre { + margin: 0; + padding: 1rem; + overflow-x: auto; + white-space: pre; + background: transparent; +} + +.code-container code { + font-family: inherit; + font-size: inherit; + background: transparent; + color: inherit; +} + +/* Line Numbers */ +.code-line-numbers { + display: inline-block; + width: 40px; + text-align: right; + color: #858585; + user-select: none; + margin-right: 1rem; + padding-right: 0.5rem; + border-right: 1px solid #3e3e3e; +} + +.code-line { + display: block; + padding-left: 0.5rem; +} + +.code-line:hover { + background-color: #2a2a2a; +} + +/* Syntax Highlighting */ +.code-keyword { + color: #569cd6; + font-weight: bold; +} + +.code-string { + color: #ce9178; +} + +.code-comment { + color: #6a9955; + font-style: italic; +} + +.code-number { + color: #b5cea8; +} + +.code-function { + color: #dcdcaa; +} + +.code-type { + color: #4ec9b0; +} + +.code-instruction { + color: #c586c0; +} + +.code-register { + color: #9cdcfe; +} + +.code-address { + color: #d7ba7d; +} + +.code-label { + color: #ffc66d; + font-weight: bold; +} + +/* Assembly-specific highlighting */ +.asm-mnemonic { + color: #569cd6; + font-weight: bold; +} + +.asm-operand { + color: #9cdcfe; +} + +.asm-immediate { + color: #b5cea8; +} + +.asm-register { + color: #c586c0; +} + +.asm-memory { + color: #ce9178; +} + +.asm-comment { + color: #6a9955; + font-style: italic; +} + +/* LLVM IR-specific highlighting */ +.ir-instruction { + color: #569cd6; + font-weight: bold; +} + +.ir-type { + color: #4ec9b0; +} + +.ir-value { + color: #9cdcfe; +} + +.ir-constant { + color: #b5cea8; +} + +.ir-label { + color: #ffc66d; + font-weight: bold; +} + +.ir-attribute { + color: #c586c0; +} + +/* Source code highlighting */ +.src-keyword { + color: #569cd6; + font-weight: bold; +} + +.src-string { + color: #ce9178; +} + +.src-comment { + color: #6a9955; + font-style: italic; +} + +.src-preprocessor { + color: #c586c0; +} + +.src-type { + color: #4ec9b0; +} + +.src-function { + color: #dcdcaa; +} + +.src-variable { + color: #9cdcfe; +} + +.src-operator { + color: #d4d4d4; +} + +/* Split View Resize Handle */ +.split-resize-handle { + width: 4px; + background-color: var(--gray-300); + cursor: col-resize; + position: relative; + transition: background-color 0.2s; +} + +.split-resize-handle:hover { + background-color: var(--llvm-blue); +} + +.split-resize-handle::after { + content: ''; + position: absolute; + top: 50%; + left: 50%; + transform: translate(-50%, -50%); + width: 8px; + height: 20px; + background-image: repeating-linear-gradient( + 0deg, + transparent, + transparent 1px, + var(--gray-400) 1px, + var(--gray-400) 2px + ); +} + +/* Loading State for Code Viewers */ +.code-loading { + display: flex; + align-items: center; + justify-content: center; + height: 200px; + background-color: #1e1e1e; + color: #858585; + font-family: monospace; +} + +.code-loading::before { + content: ''; + width: 20px; + height: 20px; + border: 2px solid #3e3e3e; + border-top: 2px solid var(--llvm-blue); + border-radius: 50%; + animation: spin 1s linear infinite; + margin-right: 0.5rem; +} + +/* Error State for Code Viewers */ +.code-error { + display: flex; + align-items: center; + justify-content: center; + height: 200px; + background-color: #1e1e1e; + color: #f48771; + font-family: monospace; + flex-direction: column; +} + +.code-error svg { + width: 24px; + height: 24px; + margin-bottom: 0.5rem; +} + +/* Copy/Download Button Styles */ +.action-button { + transition: all 0.2s ease; + padding: 0.25rem 0.5rem; + border-radius: 0.25rem; + border: 1px solid transparent; +} + +.action-button:hover:not(:disabled) { + background-color: var(--llvm-blue); + color: white; + border-color: var(--llvm-blue); +} + +.action-button:disabled { + cursor: not-allowed; + opacity: 0.5; +} + +/* File Selector Enhancement */ +.file-item { + display: flex; + align-items: center; + padding: 0.5rem; + border-bottom: 1px solid var(--gray-200); + transition: background-color 0.2s; +} + +.file-item:hover { + background-color: var(--gray-50); +} + +.file-item.selected { + background-color: var(--llvm-blue-light); +} + +.file-icon { + width: 16px; + height: 16px; + margin-right: 0.5rem; + opacity: 0.7; +} + +/* Inline Data Styles */ +.inline-data-item { + display: block; + padding: 2px 4px; + margin: 1px 0; + border-radius: 2px; + font-size: 11px; + line-height: 1.2; + background-color: rgba(0, 0, 0, 0.1); + color: #fff; +} + +.inline-data-item.diagnostic.diagnostic-error { + background-color: rgba(239, 68, 68, 0.8); + border-left: 2px solid #ef4444; +} + +.inline-data-item.diagnostic.diagnostic-warning { + background-color: rgba(245, 158, 11, 0.8); + border-left: 2px solid #f59e0b; +} + +.inline-data-item.diagnostic.diagnostic-note { + background-color: rgba(59, 130, 246, 0.8); + border-left: 2px solid #3b82f6; +} + +.inline-data-item.diagnostic.diagnostic-info { + background-color: rgba(107, 114, 128, 0.8); + border-left: 2px solid #6b7280; +} + +.inline-data-item.remark { + background-color: rgba(16, 185, 129, 0.8); + border-left: 2px solid #10b981; +} + +.inline-data-icon { + margin-right: 4px; + font-size: 10px; +} + +.inline-data-message { + font-family: monospace; +} + +.inline-data-pass { + margin-left: 4px; + opacity: 0.8; + font-style: italic; +} + +/* Button states for inline data toggles */ +button.bg-llvm-blue { + background-color: var(--llvm-blue) !important; +} + +button.text-white { + color: white !important; +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/api-client.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/api-client.js new file mode 100644 index 0000000000000..32654be083824 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/api-client.js @@ -0,0 +1,436 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * API Client + * Handles all communication with the LLVM Advisor API backend + */ + +import {Utils} from './utils.js'; + +export class ApiClient { + constructor(baseUrl = '') { + this.baseUrl = baseUrl; + this.cache = new Map(); + this.cacheTimeout = 5 * 60 * 1000; // 5 minutes + } + + /** + * Generic HTTP request method with error handling and caching + */ + async request(endpoint, options = {}) { + const url = `${this.baseUrl}/api/${endpoint}`; + const cacheKey = `${url}${JSON.stringify(options)}`; + + // Check cache first for GET requests + if (!options.method || options.method === 'GET') { + const cached = this.cache.get(cacheKey); + if (cached && Date.now() - cached.timestamp < this.cacheTimeout) { + return cached.data; + } + } + + try { + const response = await fetch(url, { + method : 'GET', + headers : {'Content-Type' : 'application/json', ...options.headers}, + ...options + }); + + if (!response.ok) { + throw new Error(`HTTP ${response.status}: ${response.statusText}`); + } + + const data = await response.json(); + + // Cache successful GET responses + if (!options.method || options.method === 'GET') { + this.cache.set(cacheKey, {data, timestamp : Date.now()}); + } + + return data; + + } catch (error) { + console.error(`API request failed for ${endpoint}:`, error); + return { + success : false, + error : error.message, + status : error.status || 500 + }; + } + } + + /** + * Clear all cached responses + */ + clearCache() { this.cache.clear(); } + + // ============================================ + // Core API Endpoints + // ============================================ + + /** + * Get system health status + */ + async getHealth() { return await this.request('health'); } + + /** + * Get all compilation units + */ + async getUnits() { return await this.request('units'); } + + /** + * Get detailed information for a specific unit + */ + async getUnitDetail(unitName) { + return await this.request(`units/${encodeURIComponent(unitName)}`); + } + + /** + * Get overall summary statistics + */ + async getSummary() { return await this.request('summary'); } + + /** + * Get available artifact types + */ + async getArtifactTypes() { return await this.request('artifacts'); } + + /** + * Get aggregated data for a specific file type + */ + async getArtifactData(fileType) { + return await this.request(`artifacts/${encodeURIComponent(fileType)}`); + } + + /** + * Get build dependencies data + */ + async getBuildDependencies() { + return await this.request('artifacts/dependencies'); + } + + /** + * Get specific file content + */ + async getFileContent(unitName, fileType, fileName, full = false) { + const params = full ? '?full=true' : ''; + return await this.request( + `file/${encodeURIComponent(unitName)}/${encodeURIComponent(fileType)}/${ + encodeURIComponent(fileName)}${params}`); + } + + // ============================================ + // Specialized Endpoints - Remarks + // ============================================ + + /** + * Get optimization remarks overview + */ + async getRemarksOverview() { return await this.request('remarks/overview'); } + + /** + * Get remarks analysis by optimization passes + */ + async getRemarksPasses() { return await this.request('remarks/passes'); } + + /** + * Get remarks analysis by functions + */ + async getRemarksFunctions() { + return await this.request('remarks/functions'); + } + + /** + * Get optimization hotspots + */ + async getRemarksHotspots() { return await this.request('remarks/hotspots'); } + + // ============================================ + // Specialized Endpoints - Diagnostics + // ============================================ + + /** + * Get diagnostics overview + */ + async getDiagnosticsOverview() { + return await this.request('diagnostics/overview'); + } + + /** + * Get diagnostics by level (error, warning, note) + */ + async getDiagnosticsByLevel() { + return await this.request('diagnostics/by-level'); + } + + /** + * Get diagnostics by files + */ + async getDiagnosticsFiles() { + return await this.request('diagnostics/files'); + } + + /** + * Get diagnostic patterns + */ + async getDiagnosticsPatterns() { + return await this.request('diagnostics/patterns'); + } + + // ============================================ + // Specialized Endpoints - Compilation Analysis + // ============================================ + + /** + * Get ftime report data for compilation timing + */ + async getFTimeReport() { + return await this.request('artifacts/ftime-report'); + } + + /** + * Get version info data (clang version, target, etc.) + */ + async getVersionInfo() { + return await this.request('artifacts/version-info'); + } + + /** + * Get compilation phases bindings (from -ccc-print-bindings) + */ + async getCompilationPhasesBindings() { + return await this.request('compilation-phases/bindings'); + } + + // ============================================ + // Specialized Endpoints - Time Trace + // ============================================ + + /** + * Get time trace overview + */ + async getTimeTraceOverview() { + return await this.request('time-trace/overview'); + } + + /** + * Get time trace timeline (with optional limit) + */ + async getTimeTraceTimeline(limit = 1000) { + return await this.request(`time-trace/timeline?limit=${limit}`); + } + + /** + * Get time trace hotspots + */ + async getTimeTraceHotspots() { + return await this.request('time-trace/hotspots'); + } + + /** + * Get time trace categories analysis + */ + async getTimeTraceCategories() { + return await this.request('time-trace/categories'); + } + + /** + * Get parallelism analysis + */ + async getTimeTraceParallelism() { + return await this.request('time-trace/parallelism'); + } + + // ============================================ + // Specialized Endpoints - Binary Size + // ============================================ + + /** + * Get binary size overview + */ + async getBinarySizeOverview() { + return await this.request('binary-size/overview'); + } + + /** + * Get binary sections analysis + */ + async getBinarySizeSections() { + return await this.request('binary-size/sections'); + } + + /** + * Get binary size optimization opportunities + */ + async getBinarySizeOptimization() { + return await this.request('binary-size/optimization'); + } + + /** + * Get binary size comparison across units + */ + async getBinarySizeComparison() { + return await this.request('binary-size/comparison'); + } + + // ============================================ + // Specialized Endpoints - Runtime Trace + // ============================================ + + /** + * Get runtime trace overview + */ + async getRuntimeTraceOverview() { + return await this.request('runtime-trace/overview'); + } + + /** + * Get runtime trace timeline + */ + async getRuntimeTraceTimeline(limit = 1000) { + return await this.request(`runtime-trace/timeline?limit=${limit}`); + } + + /** + * Get runtime trace hotspots + */ + async getRuntimeTraceHotspots() { + return await this.request('runtime-trace/hotspots'); + } + + /** + * Get runtime trace categories + */ + async getRuntimeTraceCategories() { + return await this.request('runtime-trace/categories'); + } + + /** + * Get runtime parallelism analysis + */ + async getRuntimeTraceParallelism() { + return await this.request('runtime-trace/parallelism'); + } + + // ============================================ + // Specialized Endpoints - Code Explorer + // ============================================ + + /** + * Get list of available source files for a specific unit + */ + async getSourceFiles(unitName = null) { + const params = unitName ? `?unit=${encodeURIComponent(unitName)}` : ''; + return await this.request(`explorer/files${params}`); + } + + /** + * Get source code for a specific file + */ + async getSourceCode(filePath) { + return await this.request( + `explorer/source/${encodeURIComponent(filePath)}`); + } + + /** + * Get assembly output for a specific file + */ + async getAssembly(filePath) { + return await this.request( + `explorer/assembly/${encodeURIComponent(filePath)}`); + } + + /** + * Get LLVM IR for a specific file + */ + async getLLVMIR(filePath) { + return await this.request(`explorer/ir/${encodeURIComponent(filePath)}`); + } + + /** + * Get optimized LLVM IR for a specific file + */ + async getOptimizedIR(filePath) { + return await this.request( + `explorer/optimized-ir/${encodeURIComponent(filePath)}`); + } + + /** + * Get object code for a specific file + */ + async getObjectCode(filePath) { + return await this.request( + `explorer/object/${encodeURIComponent(filePath)}`); + } + + /** + * Get AST JSON for a specific file + */ + async getASTJSON(filePath) { + return await this.request( + `explorer/ast-json/${encodeURIComponent(filePath)}`); + } + + /** + * Get preprocessed source for a specific file + */ + async getPreprocessed(filePath) { + return await this.request( + `explorer/preprocessed/${encodeURIComponent(filePath)}`); + } + + /** + * Get macro expansion for a specific file + */ + async getMacroExpansion(filePath) { + return await this.request( + `explorer/macro-expansion/${encodeURIComponent(filePath)}`); + } + + // ============================================ + // Utility Methods + // ============================================ + + /** + * Check if the API is available + */ + async isApiAvailable() { + try { + const health = await this.getHealth(); + return health.success && health.data.status === 'healthy'; + } catch (error) { + return false; + } + } + + /** + * Get cache statistics + */ + getCacheStats() { + return {size : this.cache.size, keys : Array.from(this.cache.keys())}; + } + + /** + * Batch multiple API requests + */ + async batchRequests(requests) { + const promises = + requests.map(req => typeof req === 'string' + ? this.request(req) + : this.request(req.endpoint, req.options)); + + const results = await Promise.allSettled(promises); + + return results.map( + (result, index) => ({ + request : requests[index], + success : result.status === 'fulfilled' && result.value.success, + data : result.status === 'fulfilled' ? result.value.data : null, + error : result.status === 'rejected' + ? result.reason.message + : (result.value.success ? null : result.value.error) + })); + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/app.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/app.js new file mode 100644 index 0000000000000..c653ae35fc644 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/app.js @@ -0,0 +1,412 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Main Application Controller + * Orchestrates the entire LLVM Advisor dashboard application + */ + +import {ApiClient} from './api-client.js'; +import {CompilationUnitManager} from './compilation-unit-manager.js'; +import {Dashboard} from './dashboard.js'; +import {Explorer} from './explorer.js'; +import {TabManager} from './tab-manager.js'; +import {Utils} from './utils.js'; + +class LLVMAdvisorApp { + constructor() { + this.apiClient = new ApiClient(); + this.tabManager = new TabManager(); + this.dashboard = new Dashboard(this.apiClient); + this.explorer = new Explorer(this.apiClient, this); + this.unitManager = new CompilationUnitManager(this.apiClient); + this.performanceManager = null; + + this.currentUnit = null; + this.appData = null; + + this.init(); + } + + async init() { + try { + console.log('Initializing LLVM Advisor Dashboard...'); + + // Show loading screen + this.showLoadingScreen(); + + // Initialize components + await this.initializeComponents(); + + // Load initial data + await this.loadInitialData(); + + // Setup event listeners + this.setupEventListeners(); + + // Hide loading screen and show app + this.hideLoadingScreen(); + + console.log('Dashboard initialized successfully'); + + } catch (error) { + console.error('Failed to initialize dashboard:', error); + this.showError( + 'Failed to initialize dashboard. Please check your connection and try again.'); + } + } + + async initializeComponents() { + // Initialize tab manager + this.tabManager.init( + {onTabChange : (tabId) => this.handleTabChange(tabId)}); + + // Initialize compilation unit manager + await this.unitManager.init( + {onUnitChange : (unitName) => this.handleUnitChange(unitName)}); + + // Initialize dashboard + this.dashboard.init(); + + // Initialize explorer + this.explorer.init(); + + // Initialize performance manager (lazy loaded) + if (window.PerformanceManager) { + this.performanceManager = new window.PerformanceManager(); + window.performanceManager = this.performanceManager; + } + } + + async loadInitialData() { + try { + // Check API health + const healthStatus = await this.apiClient.getHealth(); + this.updateConnectionStatus(healthStatus.success); + + if (!healthStatus.success) { + throw new Error('API server is not responding'); + } + + // Get available compilation units + const unitsResponse = await this.apiClient.getUnits(); + if (unitsResponse.success) { + this.unitManager.updateUnits(unitsResponse.data.units); + + // Select the most recent unit (first in list, as they're sorted by + // recency) + if (unitsResponse.data.units.length > 0) { + this.currentUnit = unitsResponse.data.units[0].name; + this.unitManager.selectUnit(this.currentUnit); + + // Load dashboard data for the selected unit + await this.loadDashboardData(); + } + } + + } catch (error) { + console.error('Error loading initial data:', error); + this.updateConnectionStatus(false); + throw error; + } + } + + async loadDashboardData() { + if (!this.currentUnit) + return; + + try { + console.log(`Loading dashboard data for unit: ${this.currentUnit}`); + + // Load summary data with error handling + let summaryResponse; + try { + summaryResponse = await this.apiClient.getSummary(); + if (!summaryResponse.success) { + console.warn('Summary API failed:', summaryResponse.error); + summaryResponse = {success : false, data : null}; + } + } catch (error) { + console.warn('Summary API error:', error); + summaryResponse = {success : false, data : null}; + } + + // Load unit detail to get compilation unit file count + let unitDetail; + try { + unitDetail = await this.apiClient.getUnitDetail(this.currentUnit); + if (!unitDetail.success) { + console.warn('Unit detail API failed:', unitDetail.error); + unitDetail = {success : false, data : null}; + } + } catch (error) { + console.warn('Unit detail API error:', error); + unitDetail = {success : false, data : null}; + } + + // Load specialized data for dashboard with error handling + const [remarksOverview, remarksPasses, diagnosticsOverview, + compilationPhasesOverview, compilationPhasesBindings, + binarySizeOverview, buildDependencies, versionInfo] = + await Promise.allSettled([ + this.apiClient.getRemarksOverview().catch( + e => ({success : false, error : e.message})), + this.apiClient.getRemarksPasses().catch( + e => ({success : false, error : e.message})), + this.apiClient.getDiagnosticsOverview().catch( + e => ({success : false, error : e.message})), + this.apiClient.getFTimeReport().catch( + e => ({success : false, error : e.message})), + this.apiClient.getCompilationPhasesBindings().catch( + e => ({success : false, error : e.message})), + this.apiClient.getBinarySizeOverview().catch( + e => ({success : false, error : e.message})), + this.apiClient.getBuildDependencies().catch( + e => ({success : false, error : e.message})), + this.apiClient.getVersionInfo().catch( + e => ({success : false, error : e.message})) + ]); + + // Prepare dashboard data extraction + const dashboardData = { + summary : summaryResponse.success ? summaryResponse.data : null, + unitDetail : unitDetail.success ? unitDetail.data : null, + remarks : remarksOverview.status === 'fulfilled' && + remarksOverview.value.success + ? remarksOverview.value.data + : null, + remarksPasses : + remarksPasses.status === 'fulfilled' && remarksPasses.value.success + ? remarksPasses.value.data + : null, + diagnostics : diagnosticsOverview.status === 'fulfilled' && + diagnosticsOverview.value.success + ? diagnosticsOverview.value.data + : null, + compilationPhases : compilationPhasesOverview.status === 'fulfilled' && + compilationPhasesOverview.value.success + ? compilationPhasesOverview.value.data + : null, + compilationPhasesBindings : + compilationPhasesBindings.status === 'fulfilled' && + compilationPhasesBindings.value.success + ? compilationPhasesBindings.value.data + : null, + binarySize : binarySizeOverview.status === 'fulfilled' && + binarySizeOverview.value.success + ? binarySizeOverview.value.data + : null, + buildDependencies : buildDependencies.status === 'fulfilled' && + buildDependencies.value.success + ? buildDependencies.value.data + : null, + versionInfo : + versionInfo.status === 'fulfilled' && versionInfo.value.success + ? versionInfo.value.data + : null + }; + + // Update dashboard + this.dashboard.updateData(dashboardData); + + this.appData = dashboardData; + + } catch (error) { + console.error('Error loading dashboard data:', error); + this.showError( + 'Failed to load dashboard data. Some sections may not be available.'); + } + } + + setupEventListeners() { + // Global error handler + window.addEventListener('error', (event) => { + console.error('Global error:', event.error); + this.showError('An unexpected error occurred. Please refresh the page.'); + }); + + // Handle connection issues + window.addEventListener('online', () => { + this.updateConnectionStatus(true); + this.hideError(); + }); + + window.addEventListener('offline', () => { + this.updateConnectionStatus(false); + this.showError( + 'You are currently offline. Some features may not work properly.'); + }); + + // Auto-refresh data every 30 seconds if on dashboard tab + setInterval(() => { + if (this.tabManager.getCurrentTab() === 'dashboard' && this.currentUnit) { + this.refreshCurrentData(); + } + }, 30000); + } + + async handleTabChange(tabId) { + console.log(`📱 Switching to tab: ${tabId}`); + + try { + switch (tabId) { + case 'dashboard': + if (this.currentUnit && !this.appData) { + await this.loadDashboardData(); + } + break; + case 'explorer': + await this.explorer.onActivate(); + break; + case 'diagnostics': + // Future: Load diagnostics-specific data + break; + case 'performance': + if (this.performanceManager) { + await this.performanceManager.initialize(); + } + break; + } + } catch (error) { + console.error(`Error switching to tab ${tabId}:`, error); + this.showError(`Failed to load ${tabId} data.`); + } + } + + async handleUnitChange(unitName) { + if (unitName === this.currentUnit) + return; + + console.log(`Switching to compilation unit: ${unitName}`); + this.currentUnit = unitName; + + // Clear existing data + this.appData = null; + + // Reload data for new unit based on current tab + const currentTab = this.tabManager.getCurrentTab(); + if (currentTab === 'dashboard') { + await this.loadDashboardData(); + } else if (currentTab === 'explorer') { + // Reload explorer files for new unit + await this.explorer.loadAvailableFiles(); + } + + // If performance tab is active, notify performance manager + if (currentTab === 'performance' && this.performanceManager) { + this.performanceManager.onUnitChanged(unitName); + } + } + + async refreshCurrentData() { + try { + const currentTab = this.tabManager.getCurrentTab(); + + if (currentTab === 'dashboard') { + await this.loadDashboardData(); + } + + // Update last refresh time + this.updateLastRefreshTime(); + + } catch (error) { + console.error('Error refreshing data:', error); + // Don't show error for background refresh failures + } + } + + showLoadingScreen() { + const loadingScreen = document.getElementById('loading-screen'); + const app = document.getElementById('app'); + + if (loadingScreen) + loadingScreen.classList.remove('hidden'); + if (app) + app.classList.add('hidden'); + } + + hideLoadingScreen() { + const loadingScreen = document.getElementById('loading-screen'); + const app = document.getElementById('app'); + + if (loadingScreen) + loadingScreen.classList.add('hidden'); + if (app) + app.classList.remove('hidden'); + } + + showError(message) { + const alertBanner = document.getElementById('alert-banner'); + const alertMessage = document.getElementById('alert-message'); + + if (alertBanner && alertMessage) { + alertMessage.textContent = message; + alertBanner.classList.remove('hidden'); + alertBanner.classList.remove('bg-blue-50', 'border-blue-400'); + alertBanner.classList.add('bg-red-50', 'border-red-400'); + + const messageElement = alertBanner.querySelector('p'); + if (messageElement) { + messageElement.classList.remove('text-blue-700'); + messageElement.classList.add('text-red-700'); + } + } + } + + hideError() { + const alertBanner = document.getElementById('alert-banner'); + if (alertBanner) { + alertBanner.classList.add('hidden'); + } + } + + showInfo(message) { + const alertBanner = document.getElementById('alert-banner'); + const alertMessage = document.getElementById('alert-message'); + + if (alertBanner && alertMessage) { + alertMessage.textContent = message; + alertBanner.classList.remove('hidden'); + alertBanner.classList.remove('bg-red-50', 'border-red-400'); + alertBanner.classList.add('bg-blue-50', 'border-blue-400'); + + const messageElement = alertBanner.querySelector('p'); + if (messageElement) { + messageElement.classList.remove('text-red-700'); + messageElement.classList.add('text-blue-700'); + } + } + } + + updateConnectionStatus(isConnected) { + const statusIndicator = document.getElementById('status-indicator'); + if (!statusIndicator) + return; + + const dot = statusIndicator.querySelector('div'); + const text = statusIndicator.querySelector('span'); + + if (isConnected) { + dot.className = 'h-2 w-2 bg-success rounded-full'; + text.textContent = 'Connected'; + text.className = 'text-sm text-gray-600'; + } else { + dot.className = 'h-2 w-2 bg-error rounded-full'; + text.textContent = 'Disconnected'; + text.className = 'text-sm text-red-600'; + } + } + + updateLastRefreshTime() { + const now = new Date().toLocaleTimeString(); + console.log(`Data refreshed at ${now}`); + } +} + +// Initialize the application when DOM is loaded +document.addEventListener( + 'DOMContentLoaded', + () => { window.llvmAdvisorApp = new LLVMAdvisorApp(); }); + +export {LLVMAdvisorApp}; diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/chart-components.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/chart-components.js new file mode 100644 index 0000000000000..0f04d92540ba6 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/chart-components.js @@ -0,0 +1,372 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Chart Components + * Wrapper components for Chart.js with LLVM Advisor styling and functionality + */ + +export class ChartComponents { + constructor() { + this.defaultColors = { + primary : '#3b82f6', + secondary : '#64748b', + success : '#10b981', + warning : '#f59e0b', + error : '#ef4444', + info : '#06b6d4' + }; + + this.colorPalettes = { + blue : [ + '#dbeafe', '#bfdbfe', '#93c5fd', '#60a5fa', '#3b82f6', '#2563eb', + '#1d4ed8', '#1e40af' + ], + green : [ + '#d1fae5', '#a7f3d0', '#6ee7b7', '#34d399', '#10b981', '#059669', + '#047857', '#065f46' + ], + purple : [ + '#e9d5ff', '#d8b4fe', '#c084fc', '#a855f7', '#9333ea', '#7c3aed', + '#6d28d9', '#5b21b6' + ], + orange : [ + '#fed7aa', '#fdba74', '#fb923c', '#f97316', '#ea580c', '#dc2626', + '#b91c1c', '#991b1b' + ], + mixed : [ + '#3b82f6', '#10b981', '#f59e0b', '#ef4444', '#8b5cf6', '#06b6d4', + '#84cc16', '#f97316' + ] + }; + } + + /** + * Initialize Chart.js with global configurations + */ + init() { + if (typeof Chart === 'undefined') { + console.warn('Chart.js not loaded. Charts will not be available.'); + return; + } + + // Set global Chart.js defaults + Chart.defaults.font.family = 'Inter, system-ui, -apple-system, sans-serif'; + Chart.defaults.font.size = 12; + Chart.defaults.color = '#6b7280'; + Chart.defaults.borderColor = '#e5e7eb'; + Chart.defaults.backgroundColor = '#f9fafb'; + + // Configure default responsive options + Chart.defaults.responsive = true; + Chart.defaults.maintainAspectRatio = false; + + // Configure default animation + Chart.defaults.animation.duration = 400; + Chart.defaults.animation.easing = 'easeInOutQuart'; + + console.log('Chart components initialized'); + } + + /** + * Generate color palette for charts + */ + generateColors(count, palette = 'mixed', opacity = 1) { + const colors = this.colorPalettes[palette] || this.colorPalettes.mixed; + const result = []; + + for (let i = 0; i < count; i++) { + const colorIndex = i % colors.length; + const color = colors[colorIndex]; + + if (opacity < 1) { + // Convert hex to rgba + const r = parseInt(color.slice(1, 3), 16); + const g = parseInt(color.slice(3, 5), 16); + const b = parseInt(color.slice(5, 7), 16); + result.push(`rgba(${r}, ${g}, ${b}, ${opacity})`); + } else { + result.push(color); + } + } + + return result; + } + + /** + * Generate gradient colors for charts + */ + generateGradient(ctx, color1, color2) { + const gradient = ctx.createLinearGradient(0, 0, 0, 400); + gradient.addColorStop(0, color1); + gradient.addColorStop(1, color2); + return gradient; + } + + /** + * Get default chart options with LLVM Advisor styling + */ + getDefaultOptions(type = 'default') { + const baseOptions = { + responsive : true, + maintainAspectRatio : false, + plugins : { + legend : { + display : true, + position : 'top', + align : 'start', + labels : { + padding : 20, + usePointStyle : true, + font : {size : 11, weight : '500'} + } + }, + tooltip : { + backgroundColor : 'rgba(17, 24, 39, 0.95)', + titleColor : '#f9fafb', + bodyColor : '#f3f4f6', + borderColor : '#374151', + borderWidth : 1, + cornerRadius : 8, + displayColors : true, + padding : 12, + titleFont : {size : 12, weight : '600'}, + bodyFont : {size : 11} + } + }, + interaction : {intersect : false, mode : 'index'} + }; + + // Type-specific options + switch (type) { + case 'bar': + return { + ...baseOptions, + scales : { + x : {grid : {display : false}, border : {display : false}}, + y : { + beginAtZero : true, + grid : {color : '#f3f4f6', drawBorder : false}, + border : {display : false} + } + } + }; + + case 'line': + return { + ...baseOptions, + elements : { + point : {radius : 4, hoverRadius : 6, borderWidth : 2}, + line : {tension : 0.4, borderWidth : 2} + }, + scales : { + x : {grid : {display : false}, border : {display : false}}, + y : { + beginAtZero : true, + grid : {color : '#f3f4f6', drawBorder : false}, + border : {display : false} + } + } + }; + + case 'doughnut': + case 'pie': + return { + ...baseOptions, + cutout : type === 'doughnut' ? '60%' : 0, + plugins : { + ...baseOptions.plugins, + legend : {...baseOptions.plugins.legend, position : 'bottom'} + } + }; + + default: + return baseOptions; + } + } + + /** + * Create a loading placeholder for charts + */ + createLoadingPlaceholder(canvasId) { + const canvas = document.getElementById(canvasId); + if (!canvas) + return; + + const ctx = canvas.getContext('2d'); + const width = canvas.width; + const height = canvas.height; + + // Clear canvas + ctx.clearRect(0, 0, width, height); + + // Draw loading spinner + const centerX = width / 2; + const centerY = height / 2; + const radius = 20; + + ctx.strokeStyle = '#3b82f6'; + ctx.lineWidth = 3; + ctx.lineCap = 'round'; + + // Create animated spinner effect + const drawSpinner = (rotation) => { + ctx.clearRect(0, 0, width, height); + + ctx.beginPath(); + ctx.arc(centerX, centerY, radius, rotation, rotation + Math.PI * 1.5); + ctx.stroke(); + + // Add loading text + ctx.fillStyle = '#6b7280'; + ctx.font = '12px Inter, sans-serif'; + ctx.textAlign = 'center'; + ctx.fillText('Loading chart...', centerX, centerY + radius + 20); + }; + + let rotation = 0; + const interval = setInterval(() => { + rotation += 0.1; + drawSpinner(rotation); + }, 50); + + // Store interval reference for cleanup + canvas.dataset.loadingInterval = interval; + } + + /** + * Clear loading placeholder + */ + clearLoadingPlaceholder(canvasId) { + const canvas = document.getElementById(canvasId); + if (!canvas) + return; + + const interval = canvas.dataset.loadingInterval; + if (interval) { + clearInterval(interval); + delete canvas.dataset.loadingInterval; + } + + const ctx = canvas.getContext('2d'); + ctx.clearRect(0, 0, canvas.width, canvas.height); + } + + /** + * Create a chart with error state + */ + showChartError(canvasId, message = 'Failed to load chart data') { + const canvas = document.getElementById(canvasId); + if (!canvas) + return; + + const ctx = canvas.getContext('2d'); + const width = canvas.width; + const height = canvas.height; + + // Clear canvas + ctx.clearRect(0, 0, width, height); + + // Draw error icon and message + const centerX = width / 2; + const centerY = height / 2; + + // Error icon (triangle with exclamation) + ctx.fillStyle = '#ef4444'; + ctx.beginPath(); + ctx.moveTo(centerX, centerY - 15); + ctx.lineTo(centerX - 12, centerY + 10); + ctx.lineTo(centerX + 12, centerY + 10); + ctx.closePath(); + ctx.fill(); + + // Exclamation mark + ctx.fillStyle = '#ffffff'; + ctx.font = 'bold 16px Inter, sans-serif'; + ctx.textAlign = 'center'; + ctx.fillText('!', centerX, centerY + 5); + + // Error message + ctx.fillStyle = '#6b7280'; + ctx.font = '12px Inter, sans-serif'; + ctx.fillText(message, centerX, centerY + 35); + } + + /** + * Animate chart on data update + */ + animateChart(chart, newData) { + if (!chart || !newData) + return; + + // Update data with animation + chart.data = newData; + chart.update('active'); + } + + /** + * Export chart as image + */ + exportChart(canvasId, filename = 'chart.png') { + const canvas = document.getElementById(canvasId); + if (!canvas) + return; + + const link = document.createElement('a'); + link.download = filename; + link.href = canvas.toDataURL(); + link.click(); + } + + /** + * Get responsive font size based on container + */ + getResponsiveFontSize(container) { + const width = container.offsetWidth; + if (width < 400) + return 10; + if (width < 600) + return 11; + return 12; + } + + /** + * Format numbers for chart labels + */ + formatChartNumber(value, type = 'default') { + switch (type) { + case 'bytes': + return this.formatBytes(value); + case 'time': + return this.formatTime(value); + case 'percentage': + return `${value.toFixed(1)}%`; + case 'compact': + if (value >= 1000000) + return `${(value / 1000000).toFixed(1)}M`; + if (value >= 1000) + return `${(value / 1000).toFixed(1)}K`; + return value.toString(); + default: + return value.toLocaleString(); + } + } + + formatBytes(bytes) { + if (bytes === 0) + return '0 B'; + const k = 1024; + const sizes = [ 'B', 'KB', 'MB', 'GB' ]; + const i = Math.floor(Math.log(bytes) / Math.log(k)); + return `${parseFloat((bytes / Math.pow(k, i)).toFixed(1))} ${sizes[i]}`; + } + + formatTime(ms) { + if (ms < 1000) + return `${ms}ms`; + if (ms < 60000) + return `${(ms / 1000).toFixed(1)}s`; + return `${(ms / 60000).toFixed(1)}m`; + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/code-viewer.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/code-viewer.js new file mode 100644 index 0000000000000..24917e793dcfd --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/code-viewer.js @@ -0,0 +1,417 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Code Viewer Component + * Read-only code editor with Prism.js syntax highlighting and selection + */ + +import {Utils} from './utils.js'; + +export class CodeViewer { + constructor(container, options = {}) { + this.container = container; + this.options = { + language : 'text', + showLineNumbers : true, + readOnly : true, + ...options + }; + + this.content = ''; + this.inlineData = null; + this.inlineDataVisible = {diagnostics : false, remarks : false}; + + // References to DOM elements for scroll synchronization + this.lineNumbersDiv = null; + this.codeContentDiv = null; + } + + render(content, language = null, inlineData = null) { + if (!this.container) + return; + + this.content = content || ''; + this.language = language || this.options.language; + this.inlineData = inlineData; + + console.log('CodeViewer render:', { + hasContent : !!this.content, + language : this.language, + hasInlineData : !!this.inlineData, + contentLength : this.content.length + }); + + if (!this.content) { + this.renderEmpty(); + return; + } + + this.renderEditor(); + } + + renderEditor() { + const lines = this.content.split('\n'); + const maxLineDigits = lines.length.toString().length; + + // Check if file is too large and truncate if necessary + const MAX_LINES = 5000; // Reduced limit to prevent browser freeze + const isLargeFile = lines.length > MAX_LINES; + const displayLines = isLargeFile ? lines.slice(0, MAX_LINES) : lines; + + // Create the editor with proper structure for text selection + const editorDiv = document.createElement('div'); + editorDiv.className = + 'code-editor h-full bg-white border border-gray-200 rounded-lg overflow-hidden'; + + // Add warning for large files + if (isLargeFile) { + const warningDiv = document.createElement('div'); + warningDiv.className = + 'bg-yellow-50 border-b border-yellow-200 px-4 py-2 text-sm text-yellow-800'; + warningDiv.innerHTML = `⚠️ Large file detected. Showing first ${ + MAX_LINES} lines of ${lines.length} total lines.`; + editorDiv.appendChild(warningDiv); + } + + const flexContainer = document.createElement('div'); + flexContainer.className = 'flex h-full'; + + // Line numbers column + if (this.options.showLineNumbers) { + const lineNumbersDiv = document.createElement('div'); + lineNumbersDiv.className = + 'line-numbers bg-gray-50 border-r border-gray-200 px-3 py-2 select-none flex-shrink-0 text-right font-mono text-sm text-gray-500'; + lineNumbersDiv.style.minWidth = `${maxLineDigits * 10 + 24}px`; + lineNumbersDiv.style.cssText += + 'font-size: 14px; line-height: 1.5; padding-top: 8px;'; + + // Create line number entries that will match code lines + displayLines.forEach((_, index) => { + const lineNumber = index + 1; + const lineInlineData = this.getInlineDataForLine(lineNumber); + + const lineNumWrapper = document.createElement('div'); + lineNumWrapper.className = 'line-number-wrapper'; + + const lineNumDiv = document.createElement('div'); + lineNumDiv.className = 'line-number-content'; + lineNumDiv.style.cssText = + 'min-height: 21px; line-height: 1.5; padding-top: 0; padding-bottom: 0;'; + lineNumDiv.textContent = lineNumber.toString(); + + lineNumWrapper.appendChild(lineNumDiv); + + // Add spacer for inline data if present and visible + if (this.shouldShowInlineData(lineInlineData)) { + const spacerDiv = document.createElement('div'); + spacerDiv.className = 'line-number-spacer'; + // This will be dynamically sized to match the inline data height + lineNumWrapper.appendChild(spacerDiv); + } + + lineNumbersDiv.appendChild(lineNumWrapper); + }); + + flexContainer.appendChild(lineNumbersDiv); + + // Store reference for later scroll synchronization + this.lineNumbersDiv = lineNumbersDiv; + } + + // Create code content using Prism.js + const codeContentDiv = document.createElement('div'); + codeContentDiv.className = 'code-content flex-1 overflow-auto'; + + // Create container for line-by-line rendering + const linesContainer = document.createElement('div'); + linesContainer.className = 'font-mono text-sm'; + linesContainer.style.cssText = + 'padding: 8px; font-size: 14px; line-height: 1.5;'; + + // Render each line individually with potential inline data + displayLines.forEach((line, index) => { + const lineNumber = index + 1; + const lineInlineData = this.getInlineDataForLine(lineNumber); + + // Create line wrapper + const lineWrapper = document.createElement('div'); + lineWrapper.className = 'code-line-wrapper'; + + // Create the actual code line + const codeLine = document.createElement('div'); + codeLine.className = 'code-line'; + codeLine.style.cssText = 'min-height: 21px; line-height: 1.5;'; + + // Apply syntax highlighting to individual line + const tempPre = document.createElement('pre'); + tempPre.className = 'language-' + this.getPrismLanguage(this.language); + tempPre.style.cssText = + 'margin: 0; padding: 0; background: transparent; display: inline;'; + + const tempCode = document.createElement('code'); + tempCode.className = 'language-' + this.getPrismLanguage(this.language); + tempCode.textContent = line; + + tempPre.appendChild(tempCode); + + // Apply Prism highlighting to this line + if (window.Prism) { + try { + window.Prism.highlightElement(tempCode); + } catch (e) { + // Fallback: just display the text + tempCode.textContent = line; + } + } + + codeLine.appendChild(tempPre); + lineWrapper.appendChild(codeLine); + + // Add inline diagnostics/remarks if visible and present + if (this.shouldShowInlineData(lineInlineData)) { + const inlineDataContainer = document.createElement('div'); + inlineDataContainer.className = 'inline-data-container'; + inlineDataContainer.innerHTML = + this.renderInlineDataForLine(lineInlineData); + lineWrapper.appendChild(inlineDataContainer); + } + + linesContainer.appendChild(lineWrapper); + }); + + codeContentDiv.appendChild(linesContainer); + + // Synchronize line number spacer heights with inline data heights + if (this.options.showLineNumbers) { + this.synchronizeLineHeights(); + } + + // Set up scroll synchronization between line numbers and code content + if (this.options.showLineNumbers && this.lineNumbersDiv) { + codeContentDiv.addEventListener('scroll', () => { + if (this.lineNumbersDiv) { + this.lineNumbersDiv.scrollTop = codeContentDiv.scrollTop; + } + }); + } + + flexContainer.appendChild(codeContentDiv); + editorDiv.appendChild(flexContainer); + + // Store reference to code content for external access + this.codeContentDiv = codeContentDiv; + + // Replace container content + this.container.innerHTML = ''; + this.container.appendChild(editorDiv); + } + + renderEmpty() { + this.container.innerHTML = ` +
+
+ + + +

No code to display

+
+
+ `; + } + + toggleInlineData(type, visible) { + console.log(`toggleInlineData: ${type} = ${visible}`); + this.inlineDataVisible[type] = visible; + + // Re-render to show/hide inline data + if (this.content) { + console.log('Re-rendering editor after toggle'); + this.renderEditor(); + } + } + + updateContent(content, language = null, inlineData = null) { + this.render(content, language, inlineData); + } + + setLanguage(language) { + this.language = language; + if (this.content) { + this.renderEditor(); + } + } + + clear() { + this.content = ''; + this.inlineData = null; + this.renderEmpty(); + } + + getPrismLanguage(language) { + /** Map our language identifiers to Prism.js language classes */ + const languageMap = { + 'c' : 'c', + 'cpp' : 'cpp', + 'assembly' : 'nasm', // Use NASM for assembly + 'llvm-ir' : 'llvm', // Prism has LLVM support + 'json' : 'json', + 'python' : 'python', + 'rust' : 'rust', + // Probably this filetypes would be never used + // But we never know hehe + 'javascript' : 'javascript', + 'go' : 'go', + 'java' : 'java' + }; + return languageMap[language] || 'plaintext'; + } + + shouldShowInlineData(lineInlineData) { + const hasDiagnostics = this.inlineDataVisible.diagnostics && + lineInlineData.diagnostics.length > 0; + const hasRemarks = + this.inlineDataVisible.remarks && lineInlineData.remarks.length > 0; + return hasDiagnostics || hasRemarks; + } + + getInlineDataForLine(lineNumber) { + if (!this.inlineData) { + return {diagnostics : [], remarks : []}; + } + + // Handle the structured inline data format from the API + let diagnostics = []; + let remarks = []; + + // Check if diagnostics is an array of objects with line property + if (Array.isArray(this.inlineData.diagnostics)) { + diagnostics = + this.inlineData.diagnostics.filter(d => d.line === lineNumber); + } + + // Check if remarks is an array of objects with line property + if (Array.isArray(this.inlineData.remarks)) { + remarks = this.inlineData.remarks.filter(r => r.line === lineNumber); + } + + return {diagnostics, remarks}; + } + + renderInlineDataForLine(lineInlineData) { + let html = ''; + + // Show diagnostics if enabled + if (this.inlineDataVisible.diagnostics && + lineInlineData.diagnostics.length > 0) { + lineInlineData.diagnostics.forEach(diagnostic => { + const levelClass = this.getDiagnosticLevelClass(diagnostic.level); + const icon = this.getDiagnosticIcon(diagnostic.level); + + html += ` +
+
+ ${icon} +
+
${ + this.escapeHtml(diagnostic.message)}
+ ${ + diagnostic.column ? `
Column ${ + diagnostic.column}
` + : ''} +
+
+
+ `; + }); + } + + // Show remarks if enabled + if (this.inlineDataVisible.remarks && lineInlineData.remarks.length > 0) { + lineInlineData.remarks.forEach(remark => { + html += ` +
+
+ 💡 +
+
${ + this.escapeHtml(remark.message)}
+ ${ + remark.pass ? `
[${ + this.escapeHtml(remark.pass)}]
` + : ''} + ${ + remark.column ? `
Column ${ + remark.column}
` + : ''} +
+
+
+ `; + }); + } + + return html; + } + + getDiagnosticLevelClass(level) { + const levelMap = { + 'error' : 'bg-red-50 border-red-400 text-red-800', + 'warning' : 'bg-yellow-50 border-yellow-400 text-yellow-800', + 'note' : 'bg-blue-50 border-blue-400 text-blue-800', + 'info' : 'bg-gray-50 border-gray-400 text-gray-700' + }; + return levelMap[level] || levelMap['info']; + } + + getDiagnosticIcon(level) { + const iconMap = + {'error' : '❌', 'warning' : '⚠️', 'note' : 'ℹ️', 'info' : '💬'}; + return iconMap[level] || iconMap['info']; + } + + escapeHtml(text) { + const div = document.createElement('div'); + div.textContent = text; + return div.innerHTML; + } + + synchronizeLineHeights() { + /** + * Synchronize line number spacer heights with inline data container + * heights + */ + if (!this.lineNumbersDiv) + return; + + // Use setTimeout to ensure DOM has been rendered + setTimeout(() => { + const lineWrappers = + this.container.querySelectorAll('.code-line-wrapper'); + const lineNumberWrappers = + this.lineNumbersDiv.querySelectorAll('.line-number-wrapper'); + + lineWrappers.forEach((codeLineWrapper, index) => { + const lineNumberWrapper = lineNumberWrappers[index]; + if (!lineNumberWrapper) + return; + + const inlineDataContainer = + codeLineWrapper.querySelector('.inline-data-container'); + const lineNumberSpacer = + lineNumberWrapper.querySelector('.line-number-spacer'); + + if (inlineDataContainer && lineNumberSpacer) { + // Match the height of the inline data container + const inlineDataHeight = inlineDataContainer.offsetHeight; + lineNumberSpacer.style.height = `${inlineDataHeight}px`; + } else if (lineNumberSpacer) { + // If no inline data, remove the spacer height + lineNumberSpacer.style.height = '0px'; + } + }); + }, 0); + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/compilation-unit-manager.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/compilation-unit-manager.js new file mode 100644 index 0000000000000..9c64446abbd25 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/compilation-unit-manager.js @@ -0,0 +1,424 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Compilation Unit Manager + * Handles compilation unit selection and tracking + */ + +export class CompilationUnitManager { + constructor(apiClient) { + this.apiClient = apiClient; + this.units = []; + this.currentUnit = null; + this.onUnitChangeCallback = null; + this.selector = null; + this.metadata = new Map(); // Store metadata for each unit + } + + /** + * Initialize the compilation unit manager + */ + async init(options = {}) { + this.onUnitChangeCallback = options.onUnitChange; + + // Get the selector element + this.selector = document.getElementById('unit-selector'); + if (!this.selector) { + throw new Error('Unit selector element not found'); + } + + // Setup event listeners + this.setupEventListeners(); + + // Load compilation unit metadata + await this.loadUnitMetadata(); + + console.log('Compilation unit manager initialized'); + } + + /** + * Setup event listeners + */ + setupEventListeners() { + if (this.selector) { + this.selector.addEventListener('change', (event) => { + const selectedUnit = event.target.value; + if (selectedUnit && selectedUnit !== this.currentUnit) { + this.selectUnit(selectedUnit); + } + }); + } + } + + /** + * Load compilation unit metadata from API + */ + async loadUnitMetadata() { + try { + // Get units from API + const response = await this.apiClient.getUnits(); + + if (response.success && response.data.units) { + this.units = response.data.units; + + // Sort units by most recent first + this.units.sort((a, b) => { + // If units have timestamps, sort by those + if (a.timestamp && b.timestamp) { + return new Date(b.timestamp) - new Date(a.timestamp); + } + // Otherwise, sort alphabetically + return a.name.localeCompare(b.name); + }); + + // Store metadata for each unit + this.units.forEach(unit => { + this.metadata.set(unit.name, { + ...unit, + lastSelected : null, + isRecent : this.isRecentUnit(unit) + }); + }); + + console.log(`📋 Loaded ${this.units.length} compilation units`); + return true; + } else { + console.warn('No compilation units found'); + return false; + } + } catch (error) { + console.error('Failed to load unit metadata:', error); + return false; + } + } + + /** + * Update the units list + */ + updateUnits(units) { + this.units = units || []; + this.updateSelector(); + + // Update metadata + this.units.forEach(unit => { + if (!this.metadata.has(unit.name)) { + this.metadata.set( + unit.name, + {...unit, lastSelected : null, isRecent : this.isRecentUnit(unit)}); + } + }); + } + + /** + * Update the selector dropdown + */ + updateSelector() { + if (!this.selector) + return; + + // Clear existing options + this.selector.innerHTML = ''; + + if (this.units.length === 0) { + const option = document.createElement('option'); + option.value = ''; + option.textContent = 'No compilation units found'; + option.disabled = true; + this.selector.appendChild(option); + return; + } + + // Add default option + const defaultOption = document.createElement('option'); + defaultOption.value = ''; + defaultOption.textContent = 'Select a compilation unit...'; + defaultOption.disabled = true; + this.selector.appendChild(defaultOption); + + // Add unit options + this.units.forEach(unit => { + const option = document.createElement('option'); + option.value = unit.name; + + // Create descriptive text + let displayText = unit.name; + + // Add artifact count if available + if (unit.total_files) { + displayText += ` (${unit.total_files} artifacts)`; + } + + // Add run timestamp if available + if (unit.run_timestamp) { + const formattedTime = this.formatTimestamp(unit.run_timestamp); + displayText += ` - ${formattedTime}`; + } + + // Add recent indicator + if (this.isRecentUnit(unit)) { + displayText += ' 🔥'; + } + + option.textContent = displayText; + + // Set tooltip with more information + option.title = this.buildUnitTooltip(unit); + + this.selector.appendChild(option); + }); + } + + /** + * Select a specific compilation unit + */ + async selectUnit(unitName) { + if (!unitName || unitName === this.currentUnit) { + return; + } + + const unit = this.units.find(u => u.name === unitName); + if (!unit) { + console.error(`Unit not found: ${unitName}`); + return; + } + + try { + // Update current unit + const previousUnit = this.currentUnit; + this.currentUnit = unitName; + + // Update selector + if (this.selector) { + this.selector.value = unitName; + } + + // Update metadata + const metadata = this.metadata.get(unitName); + if (metadata) { + metadata.lastSelected = new Date().toISOString(); + } + + // Call the change callback + if (this.onUnitChangeCallback) { + await this.onUnitChangeCallback(unitName, previousUnit); + } + + // Update recent units tracking + this.updateRecentUnits(unitName); + + console.log(`Selected compilation unit: ${unitName}`); + + } catch (error) { + console.error(`Failed to select unit ${unitName}:`, error); + + // Revert selection on error + this.currentUnit = this.currentUnit; // Keep previous selection + if (this.selector) { + this.selector.value = this.currentUnit || ''; + } + + throw error; + } + } + + /** + * Get the currently selected unit + */ + getCurrentUnit() { return this.currentUnit; } + + /** + * Get information about the current unit + */ + getCurrentUnitInfo() { + if (!this.currentUnit) + return null; + + const unit = this.units.find(u => u.name === this.currentUnit); + const metadata = this.metadata.get(this.currentUnit); + + return {...unit, metadata}; + } + + /** + * Get all available units + */ + getAllUnits() { + return this.units.map( + unit => ({...unit, metadata : this.metadata.get(unit.name)})); + } + + /** + * Check if a unit is considered "recent" + */ + isRecentUnit(unit) { + // We are just considering the first 3 units as recent + const index = this.units.findIndex(u => u.name === unit.name); + return index < 3; + } + + /** + * Format timestamp from YYYYMMDD_HHMMSS format + */ + formatTimestamp(timestamp) { + if (!timestamp || timestamp.length !== 15) { + return timestamp; // Return as-is if not in expected format + } + + try { + const year = timestamp.substring(0, 4); + const month = timestamp.substring(4, 6); + const day = timestamp.substring(6, 8); + const hour = timestamp.substring(9, 11); + const minute = timestamp.substring(11, 13); + const second = timestamp.substring(13, 15); + + const date = new Date(year, month - 1, day, hour, minute, second); + return date.toLocaleString(); + } catch (e) { + return timestamp; // Fallback to original string + } + } + + /** + * Build tooltip text for a unit + */ + buildUnitTooltip(unit) { + const parts = []; + + parts.push(`Unit: ${unit.name}`); + + if (unit.total_files) { + parts.push(`Artifacts: ${unit.total_files}`); + } + + if (unit.run_timestamp) { + parts.push(`Run: ${this.formatTimestamp(unit.run_timestamp)}`); + } + + if (unit.available_runs && unit.available_runs.length > 1) { + parts.push(`Available runs: ${unit.available_runs.length}`); + } + + if (unit.artifact_types && unit.artifact_types.length > 0) { + parts.push(`Types: ${unit.artifact_types.join(', ')}`); + } + + const metadata = this.metadata.get(unit.name); + if (metadata && metadata.lastSelected) { + const lastSelected = new Date(metadata.lastSelected); + parts.push(`Last selected: ${lastSelected.toLocaleString()}`); + } + + return parts.join('\n'); + } + + /** + * Update recent units tracking + */ + updateRecentUnits(unitName) { + // Move selected unit to the front of recent units + const unitIndex = this.units.findIndex(u => u.name === unitName); + if (unitIndex > 0) { + const unit = this.units.splice(unitIndex, 1)[0]; + this.units.unshift(unit); + + // Update the selector to reflect new order + this.updateSelector(); + } + + // Store in localStorage for persistence + try { + const recentUnits = + JSON.parse(localStorage.getItem('llvm_advisor_recent_units') || '[]'); + + // Remove if already exists + const filteredRecent = recentUnits.filter(name => name !== unitName); + + // Add to front + filteredRecent.unshift(unitName); + + // Keep only last 5 recent units + const updatedRecent = filteredRecent.slice(0, 5); + + localStorage.setItem('llvm_advisor_recent_units', + JSON.stringify(updatedRecent)); + } catch (error) { + console.warn('Failed to update recent units in localStorage:', error); + } + } + + /** + * Get recent units from localStorage + */ + getRecentUnits() { + try { + return JSON.parse(localStorage.getItem('llvm_advisor_recent_units') || + '[]'); + } catch (error) { + console.warn('Failed to get recent units from localStorage:', error); + return []; + } + } + + /** + * Auto-select the most appropriate unit + */ + autoSelectUnit() { + if (this.units.length === 0) + return null; + + // Try to select from recent units first + const recentUnits = this.getRecentUnits(); + for (const recentUnit of recentUnits) { + if (this.units.some(u => u.name === recentUnit)) { + this.selectUnit(recentUnit); + return recentUnit; + } + } + + // Otherwise, select the first unit (most recent by default) + const firstUnit = this.units[0]; + if (firstUnit) { + this.selectUnit(firstUnit.name); + return firstUnit.name; + } + + return null; + } + + /** + * Refresh units list from API + */ + async refreshUnits() { + const success = await this.loadUnitMetadata(); + if (success) { + this.updateSelector(); + + // If current unit no longer exists, auto-select a new one + if (this.currentUnit && + !this.units.some(u => u.name === this.currentUnit)) { + this.currentUnit = null; + this.autoSelectUnit(); + } + } + return success; + } + + /** + * Get unit statistics + */ + getUnitStats() { + return { + totalUnits : this.units.length, + currentUnit : this.currentUnit, + recentUnits : this.getRecentUnits(), + unitsByFileCount : + this.units.filter(u => u.total_files) + .sort((a, b) => b.total_files - a.total_files) + .slice(0, 5) + .map(u => ({name : u.name, artifacts : u.total_files})) + }; + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/dashboard.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/dashboard.js new file mode 100644 index 0000000000000..21a4cbc6722fb --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/dashboard.js @@ -0,0 +1,1010 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Dashboard + * Main dashboard logic and data visualization + */ + +import {ChartComponents} from './chart-components.js'; +import {Utils} from './utils.js'; + +export class Dashboard { + constructor(apiClient) { + this.apiClient = apiClient; + this.chartComponents = new ChartComponents(); + this.charts = {}; + this.currentData = null; + this.isInitialized = false; + } + + /** + * Initialize the dashboard + */ + init() { + console.log('📊 Initializing dashboard...'); + + // Initialize chart components + this.chartComponents.init(); + + this.isInitialized = true; + console.log('Dashboard initialized'); + } + + /** + * Update dashboard with new data + */ + async updateData(data) { + if (!this.isInitialized) { + console.warn('Dashboard not initialized, cannot update data'); + return; + } + + this.currentData = data; + + try { + // Update metrics cards + this.updateMetricsCards(data); + + // Update charts + await this.updateCharts(data); + + // Update insights and recommendations + this.updateInsights(data); + + console.log('Dashboard updated successfully'); + + } catch (error) { + console.error('Failed to update dashboard:', error); + this.showDashboardError('Failed to update dashboard data'); + } + } + + /** + * Update the key metrics cards + */ + updateMetricsCards(data) { + const summary = data.summary; + const diagnostics = data.diagnostics; + const remarks = data.remarks; + const buildDependencies = data.buildDependencies; + + // Total Source Files (from dependencies - count unique source files for + // current unit only) + let totalSourceFiles = 0; + + // Get current unit name to filter data + const currentUnitName = this.currentData?.unitDetail?.name || + document.getElementById('unit-selector')?.value; + + console.log('Calculating total files for unit:', currentUnitName); + + if (buildDependencies && currentUnitName) { + // Look for the specific unit's data + if (buildDependencies.units && buildDependencies.units[currentUnitName]) { + const unit = buildDependencies.units[currentUnitName]; + console.log(`Processing unit ${currentUnitName}:`, unit); + + // Look for summary_stats in the current unit + if (unit.summary_stats && unit.summary_stats.unique_sources) { + console.log(`Found unique_sources in summary_stats: ${ + unit.summary_stats.unique_sources}`); + totalSourceFiles = unit.summary_stats.unique_sources; + } + // Fallback: look in metadata + else if (unit.metadata && unit.metadata.unique_sources) { + console.log(`Found unique_sources in metadata: ${ + unit.metadata.unique_sources}`); + totalSourceFiles = unit.metadata.unique_sources; + } + // Check individual files and their metadata for unique_sources count + else if (unit.files && Array.isArray(unit.files)) { + console.log(`Checking files array for unit ${ + currentUnitName}, length: ${unit.files.length}`); + const uniqueSourceFiles = new Set(); + + unit.files.forEach(file => { + // Check if this is a dependencies file and has metadata with + // unique_sources + if (file.metadata && file.metadata.unique_sources) { + // For dependencies files, use the unique_sources count + console.log(`Found unique_sources in file metadata: ${ + file.metadata.unique_sources}`); + totalSourceFiles = + Math.max(totalSourceFiles, file.metadata.unique_sources); + } + // Also count sources directory files if available + else if (file.file_path && file.file_path.includes('/sources/')) { + const fileName = file.file_path.split('/').pop(); + if (fileName && !fileName.startsWith('.')) { + uniqueSourceFiles.add(fileName); + } + } + }); + + // If no unique_sources found in metadata, use the count from sources + // directory + if (totalSourceFiles === 0 && uniqueSourceFiles.size > 0) { + totalSourceFiles = uniqueSourceFiles.size; + console.log( + `Counted ${uniqueSourceFiles.size} unique sources from files`); + } + } + } + } + + // Fallback: if no current unit or no data found, show 0 + if (!currentUnitName || totalSourceFiles === 0) { + console.log( + 'No current unit selected or no source files found, showing 0'); + totalSourceFiles = 0; + } + + console.log('Calculated source files:', totalSourceFiles); + + this.updateMetricCard('metric-total-files', + Utils.formatNumber(totalSourceFiles)); + + // Success Rate + const successRate = summary?.success_rate || 0; + this.updateMetricCard('metric-success-rate', `${successRate.toFixed(1)}%`); + + // Total Errors + const totalErrors = summary?.errors || 0; + this.updateMetricCard('metric-total-errors', + Utils.formatNumber(totalErrors)); + + // Compilation Phases (from compilation phases bindings) + const compilationPhases = + data.compilationPhasesBindings?.summary?.total_bindings || 0; + this.updateMetricCard('metric-timing-phases', + Utils.formatNumber(compilationPhases)); + } + + /** + * Update a single metric card + */ + updateMetricCard(elementId, value) { + const element = document.getElementById(elementId); + if (element) { + // Add animation + element.style.opacity = '0.6'; + setTimeout(() => { + element.textContent = value; + element.style.opacity = '1'; + }, 150); + } + } + + /** + * Update all charts + */ + async updateCharts(data) { + const summary = data.summary; + const diagnostics = data.diagnostics; + const compilationPhases = data.compilationPhases; + const binarySize = data.binarySize; + const remarks = data.remarks; + const remarksPasses = data.remarksPasses; + const versionInfo = data.versionInfo; + + // Remarks Distribution Chart + if (remarks || remarksPasses) { + await this.updateRemarksDistributionChart(remarks, remarksPasses); + } + + // Diagnostic Levels Chart + if (diagnostics?.by_level) { + await this.updateDiagnosticLevelsChart(diagnostics.by_level); + } + + // Compilation Info Table + if (compilationPhases || versionInfo) { + this.updateCompilationInfoTable(compilationPhases, versionInfo); + } + + // Binary Size Chart + if (binarySize?.section_breakdown) { + await this.updateBinarySizeChart(binarySize.section_breakdown); + } + } + + /** + * Update remarks distribution chart + */ + async updateRemarksDistributionChart(remarksData, remarksPassesData) { + const canvas = document.getElementById('remarks-distribution-chart'); + if (!canvas) + return; + + // Try to get remarks distribution by type from passes data + let chartData = []; + + if (remarksPassesData && remarksPassesData.passes) { + // Get top optimization passes by count + const passesArray = + Object.entries(remarksPassesData.passes) + .sort(([, a ], [, b ]) => (b.count || 0) - (a.count || 0)) + .slice(0, 8); // Top 8 passes + + if (passesArray.length > 0) { + const labels = passesArray.map( + ([ name ]) => + name.length > 20 ? name.substring(0, 20) + '...' : name); + const counts = passesArray.map(([, data ]) => data.count || 0); + + const colors = [ + '#3b82f6', '#1e40af', '#1d4ed8', '#2563eb', '#60a5fa', '#93c5fd', + '#dbeafe', '#eff6ff' + ]; + + const config = { + type : 'doughnut', + data : { + labels : labels, + datasets : [ { + data : counts, + backgroundColor : colors.slice(0, labels.length), + borderWidth : 2, + borderColor : '#ffffff' + } ] + }, + options : { + responsive : true, + maintainAspectRatio : false, + plugins : { + legend : { + position : 'bottom', + labels : + {padding : 20, usePointStyle : true, font : {size : 11}} + }, + tooltip : { + callbacks : { + label : (context) => { + const label = context.label; + const value = context.parsed; + const total = + context.dataset.data.reduce((a, b) => a + b, 0); + const percentage = ((value / total) * 100).toFixed(1); + return `${label}: ${value} remarks (${percentage}%)`; + } + } + } + } + } + }; + + // Destroy existing chart if it exists + if (this.charts.remarksDistribution) { + this.charts.remarksDistribution.destroy(); + } + + this.charts.remarksDistribution = new Chart(canvas, config); + return; + } + } + + // Fallback: Show placeholder when no remarks data is available + this.showPlaceholderChart('remarks-distribution-chart', + 'No Remarks Data Available'); + } + + /** + * Update diagnostic levels chart + */ + async updateDiagnosticLevelsChart(diagnosticData) { + const canvas = document.getElementById('diagnostic-levels-chart'); + if (!canvas) + return; + + // Extract diagnostic level counts + const levels = [ 'error', 'warning', 'note', 'info' ]; + const colors = { + 'error' : '#ef4444', + 'warning' : '#f59e0b', + 'note' : '#3b82f6', + 'info' : '#10b981' + }; + + const data = levels.map(level => { + if (diagnosticData.by_level && diagnosticData.by_level[level]) { + return diagnosticData.by_level[level]; + } + // Fallback: look for the data in different format + return diagnosticData[level] || 0; + }); + + const config = { + type : 'bar', + data : { + labels : levels.map(level => Utils.capitalize(level)), + datasets : [ { + label : 'Diagnostics', + data : data, + backgroundColor : levels.map(level => colors[level]), + borderRadius : 4, + borderWidth : 0 + } ] + }, + options : { + responsive : true, + maintainAspectRatio : false, + plugins : { + legend : {display : false}, + tooltip : { + callbacks : { + label : (context) => { + return `${context.label}: ${context.parsed.y} issues`; + } + } + } + }, + scales : { + y : {beginAtZero : true, ticks : {precision : 0}}, + x : {grid : {display : false}} + } + } + }; + + if (this.charts.diagnosticLevels) { + this.charts.diagnosticLevels.destroy(); + } + + this.charts.diagnosticLevels = new Chart(canvas, config); + } + + /** + * Update compilation info table + */ + updateCompilationInfoTable(compilationData, versionInfo) { + const container = document.getElementById('compilation-info-table'); + if (!container) + return; + + // Check if we have valid compilation data + if (!compilationData && !versionInfo) { + container.innerHTML = ` +
+ + + + No compilation information available +
+ `; + return; + } + + // Extract compilation info from the actual parsed data + let compilationInfo = []; + + // Get current unit name to filter data + const currentUnitName = this.currentData?.unitDetail?.name || + document.getElementById('unit-selector')?.value; + + console.log('Processing compilation info for unit:', currentUnitName); + + // Use Map to store unique info for current unit only + const infoMap = new Map(); + + const addOrAggregateInfo = + (label, value, icon, shouldAggregate = false) => { + if (!value) + return; + + if (shouldAggregate && infoMap.has(label)) { + // Aggregate numeric values (for timing info within this unit) + const existing = infoMap.get(label); + if (typeof value === 'string' && value.endsWith('s')) { + // Parse timing values like "0.0171s" + const currentTime = parseFloat(value.replace('s', '')); + const existingTime = parseFloat(existing.value.replace('s', '')); + if (!isNaN(currentTime) && !isNaN(existingTime)) { + existing.value = `${(existingTime + currentTime).toFixed(4)}s`; + return; + } + } + // For non-timing aggregatable values, prefer the first one + return; + } else if (!shouldAggregate && infoMap.has(label)) { + // For non-aggregatable values, just ignore duplicates + return; + } + + infoMap.set(label, {label, value, icon}); + }; + + // Debug: log the structure to understand what we have + console.log('FTime report data structure:', compilationData); + console.log('Version info data structure:', versionInfo); + + // Process ftime-report data (compilation timing) - only for current unit + if (compilationData && compilationData.units) { + Object.entries(compilationData.units).forEach(([ unitName, unit ]) => { + // Skip if not the current unit + if (currentUnitName && unitName !== currentUnitName) { + console.log( + `Skipping unit ${unitName}, current unit is ${currentUnitName}`); + return; + } + if (unit.files && Array.isArray(unit.files)) { + unit.files.forEach(file => { + console.log('Processing ftime file:', file); + + // Process ftime report file + if (file.file_name?.includes('ftime') || + file.file_path?.includes('ftime-report')) { + console.log('Found ftime file metadata:', file.metadata); + + // Extract compilation timing information + if (file.metadata) { + // Add timing information (aggregate timing values, keep first + // for others) + if (file.metadata.total_execution_time !== undefined) { + addOrAggregateInfo( + 'Total Execution Time', + `${file.metadata.total_execution_time.toFixed(4)}s`, + 'M12 8v4l3 3m6-3a9 9 0 11-18 0 9 9 0 0118 0z', true); + } + + if (file.metadata.timing_entries_count !== undefined) { + addOrAggregateInfo( + 'Timing Phases', file.metadata.timing_entries_count, + 'M9 5H7a2 2 0 00-2 2v6a2 2 0 002 2h6a2 2 0 002-2V7a2 2 0 00-2-2h-2m-2 0V3a2 2 0 012-2h2a2 2 0 012 2v2m-6 0h6', + false); + } + + if (file.metadata.top_time_consumer !== undefined && + file.metadata.top_time_consumer) { + addOrAggregateInfo('Top Time Consumer', + file.metadata.top_time_consumer, + 'M13 10V3L4 14h7v7l9-11h-7z', false); + } + } + } + }); + } + }); + } + + // Process version-info data (clang version, target, etc.) - only for + // current unit + if (versionInfo && versionInfo.units) { + Object.entries(versionInfo.units).forEach(([ unitName, unit ]) => { + // Skip if not the current unit + if (currentUnitName && unitName !== currentUnitName) { + console.log(`Skipping version-info unit ${ + unitName}, current unit is ${currentUnitName}`); + return; + } + if (unit.files && Array.isArray(unit.files)) { + unit.files.forEach(file => { + console.log('Processing version info file:', file); + + // Process version info file + if (file.file_name?.includes('version') || + file.file_path?.includes('version-info')) { + console.log('Found version info file metadata:', file.metadata); + + // Extract version information + if (file.metadata || file.data) { + const metadata = file.metadata || {}; + const data = file.data || {}; + + // Extract unique compilation info (no aggregation needed for + // compiler info) + const clangVersion = + metadata.clang_version || data.clang_version; + addOrAggregateInfo( + 'Clang Version', clangVersion, + 'M9 12l2 2 4-4M7.835 4.697a3.42 3.42 0 001.946-.806 3.42 3.42 0 014.438 0 3.42 3.42 0 001.946.806 3.42 3.42 0 713.138 3.138 3.42 3.42 0 00.806 1.946 3.42 3.42 0 010 4.438 3.42 3.42 0 00-.806 1.946 3.42 3.42 0 01-3.138 3.138 3.42 3.42 0 00-1.946.806 3.42 3.42 0 01-4.438 0 3.42 3.42 0 00-1.946-.806 3.42 3.42 0 01-3.138-3.138 3.42 3.42 0 00-.806-1.946 3.42 3.42 0 010-4.438 3.42 3.42 0 00.806-1.946 3.42 3.42 0 713.138-3.138z', + false); + + const target = metadata.target || data.target; + addOrAggregateInfo( + 'Target Architecture', target, + 'M9 3v2m6-2v2M9 19v2m6-2v2M5 9H3m2 6H3m18-6h-2m2 6h-2M7 19h10a2 2 0 002-2V7a2 2 0 00-2-2H7a2 2 0 00-2 2v10a2 2 0 002 2zM9 9h6v6H9V9z', + false); + + const threadModel = metadata.thread_model || data.thread_model; + addOrAggregateInfo('Thread Model', threadModel, + 'M19 11H5m14-7l2 7-2 7M5 18l-2-7 2-7', + false); + } + } + }); + } + }); + } + + // Add timestamp from unit detail if available + if (this.currentData && this.currentData.unitDetail) { + const unitDetail = this.currentData.unitDetail; + console.log('Unit detail for timestamp:', unitDetail); + + if (unitDetail.timestamp) { + const timestamp = new Date(unitDetail.timestamp); + addOrAggregateInfo('Compilation Time', timestamp.toLocaleString(), + 'M12 8v4l3 3m6-3a9 9 0 11-18 0 9 9 0 0118 0z', + false); + } + } + + // Convert map to array for display + compilationInfo = Array.from(infoMap.values()); + + console.log('Final compilation info for unit', currentUnitName, ':', + compilationInfo); + + // Only show table if we have actual data + if (compilationInfo.length === 0) { + container.innerHTML = ` +
+ + + + No compilation information available from parsed data +
+ `; + } else { + // Create table HTML with actual parsed data + container.innerHTML = ` +
+
+ ${ + compilationInfo + .map(info => ` +
+
+ + + + ${ + info.label} +
+ ${ + info.value} +
+ `) + .join('')} +
+
+ `; + } + } + + /** + * Update binary size chart + */ + async updateBinarySizeChart(sizeData) { + const canvas = document.getElementById('binary-size-chart'); + if (!canvas) + return; + + // Check if we have valid size data + if (!sizeData || Object.keys(sizeData).length === 0) { + this.showPlaceholderChart('binary-size-chart', 'No Binary Size Data'); + return; + } + + // Get top sections by size + const sections = Object.entries(sizeData) + .filter(([, size ]) => size > 0) + .sort(([, a ], [, b ]) => b - a) + .slice(0, 8); + + if (sections.length === 0) { + this.showPlaceholderChart('binary-size-chart', 'No Binary Size Data'); + return; + } + + const labels = sections.map( + ([ name ]) => + Utils.formatSectionName ? Utils.formatSectionName(name) : name); + const sizes = sections.map(([, size ]) => size); + + const colors = + this.chartComponents.generateColors + ? this.chartComponents.generateColors(labels.length, 'blue') + : [ + '#3b82f6', '#1e40af', '#1d4ed8', '#2563eb', '#60a5fa', + '#93c5fd', '#dbeafe', '#eff6ff' + ]; + + const config = { + type : 'pie', + data : { + labels : labels, + datasets : [ { + data : sizes, + backgroundColor : colors.slice(0, labels.length), + borderWidth : 2, + borderColor : '#ffffff' + } ] + }, + options : { + responsive : true, + maintainAspectRatio : false, + plugins : { + legend : { + position : 'bottom', + labels : {padding : 20, usePointStyle : true} + }, + tooltip : { + callbacks : { + label : (context) => { + const label = context.label; + const value = context.parsed; + const total = context.dataset.data.reduce((a, b) => a + b, 0); + const percentage = ((value / total) * 100).toFixed(1); + const formattedSize = this.formatBytes ? this.formatBytes(value) + : `${value} bytes`; + return `${label}: ${formattedSize} (${percentage}%)`; + } + } + } + } + } + }; + + if (this.charts.binarySize) { + this.charts.binarySize.destroy(); + } + + this.charts.binarySize = new Chart(canvas, config); + } + + /** + * Show placeholder chart for missing data + */ + showPlaceholderChart(canvasId, message) { + const canvas = document.getElementById(canvasId); + if (!canvas) + return; + + const chartKey = canvasId.replace('-chart', '') + .replace(/-([a-z])/g, (g) => g[1].toUpperCase()); + + const config = { + type : 'doughnut', + data : { + labels : [ message ], + datasets : + [ {data : [ 1 ], backgroundColor : [ '#f3f4f6' ], borderWidth : 0} ] + }, + options : { + responsive : true, + maintainAspectRatio : false, + plugins : {legend : {display : false}, tooltip : {enabled : false}} + } + }; + + if (this.charts[chartKey]) { + this.charts[chartKey].destroy(); + } + + this.charts[chartKey] = new Chart(canvas, config); + } + + /** + * Format bytes helper function + */ + formatBytes(bytes) { + if (bytes === 0) + return '0 B'; + const k = 1024; + const sizes = [ 'B', 'KB', 'MB', 'GB' ]; + const i = Math.floor(Math.log(bytes) / Math.log(k)); + return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i]; + } + + /** + * Update insights and recommendations + */ + updateInsights(data) { + this.updateRemarksSummary(data); + this.updateOptimizationPasses(data); + } + + /** + * Update remarks summary + */ + updateRemarksSummary(data) { + const container = document.getElementById('remarks-summary-list'); + if (!container) + return; + + const remarks = data.remarks; + + container.innerHTML = ''; + + if (!remarks || !remarks.totals) { + container.innerHTML = ` +
+ + + + No optimization remarks available +
+ `; + return; + } + + const summaryItems = [ + { + title : 'Total Optimization Remarks', + value : remarks.totals.remarks || 0, + description : 'Total number of optimization opportunities found', + icon : 'M13 10V3L4 14h7v7l9-11h-7z', + color : 'bg-blue-50 border-blue-200' + }, + { + title : 'Unique Optimization Passes', + value : remarks.totals.unique_passes || 0, + description : + 'Number of different optimization passes that generated remarks', + icon : + 'M9 3v2m6-2v2M9 19v2m6-2v2M5 9H3m2 6H3m18-6h-2m2 6h-2M7 19h10a2 2 0 002-2V7a2 2 0 00-2-2H7a2 2 0 00-2 2v10a2 2 0 002 2zM9 9h6v6H9V9z', + color : 'bg-green-50 border-green-200' + }, + { + title : 'Functions with Remarks', + value : remarks.totals.unique_functions || 0, + description : 'Functions that have optimization remarks', + icon : + 'M7 21a4 4 0 01-4-4V5a2 2 0 012-2h4a2 2 0 012 2v12a4 4 0 01-4 4zM21 5a2 2 0 012 2v12a4 4 0 01-4 4h-4a2 2 0 01-2-2V5a2 2 0 012-2h4z', + color : 'bg-purple-50 border-purple-200' + } + ]; + + summaryItems.forEach(item => { + const itemElement = document.createElement('div'); + itemElement.className = + `flex items-start space-x-3 p-3 rounded-lg ${item.color}`; + + itemElement.innerHTML = ` +
+ + + +
+
+

${ + item.title}

+

${ + item.description}

+ ${ + item.value} +
+ `; + + container.appendChild(itemElement); + }); + } + + /** + * Update optimization passes + */ + updateOptimizationPasses(data) { + const container = document.getElementById('optimization-passes-list'); + if (!container) + return; + + const passesData = data.remarksPasses; + + container.innerHTML = ''; + + if (!passesData || !passesData.passes) { + container.innerHTML = ` +
+ + + + No optimization passes data available +
+ `; + return; + } + + // Get top optimization passes by remark count + const topPasses = + Object.entries(passesData.passes) + .sort(([, a ], [, b ]) => (b.count || 0) - (a.count || 0)) + .slice(0, 5); // Top 5 passes + + topPasses.forEach(([ passName, passInfo ]) => { + const passElement = document.createElement('div'); + passElement.className = + 'flex items-start space-x-3 p-3 rounded-lg bg-gradient-to-r from-blue-50 to-indigo-50 border border-blue-200'; + + const description = + passInfo.examples && passInfo.examples.length > 0 + ? passInfo.examples[0].message || 'Optimization pass execution' + : 'Optimization pass with multiple improvements'; + + passElement.innerHTML = ` +
+ + + +
+
+

${ + passName.replace(/-/g, ' ').replace(/\b\w/g, + l => l.toUpperCase())}

+

${ + description.length > 80 ? description.substring(0, 80) + '...' + : description}

+
+ ${ + passInfo.count || 0} remarks + ${ + passInfo.unique_functions || 0} functions +
+
+ `; + + container.appendChild(passElement); + }); + } + + /** + * Generate top issues from data + */ + generateTopIssues(data) { + const issues = []; + + // Check for compilation errors + if (data.summary?.errors > 0) { + issues.push({ + severity : 'error', + title : 'Compilation Errors', + description : `${data.summary.errors} files failed to parse correctly`, + count : data.summary.errors + }); + } + + // Check for high error rate in diagnostics + if (data.diagnostics?.by_level?.error > 10) { + issues.push({ + severity : 'error', + title : 'High Error Count', + description : 'Large number of compilation errors detected', + count : data.diagnostics.by_level.error + }); + } + + // Check for many warnings + if (data.diagnostics?.by_level?.warning > 50) { + issues.push({ + severity : 'warning', + title : 'Many Warnings', + description : + 'High number of compiler warnings that should be addressed', + count : data.diagnostics.by_level.warning + }); + } + + return issues.slice(0, 5); // Return top 5 issues + } + + /** + * Generate optimization recommendations + */ + generateOptimizationRecommendations(data) { + const recommendations = []; + + // Optimization remarks suggestions + if (data.remarks?.totals?.remarks > 0) { + recommendations.push({ + title : 'Review Optimization Remarks', + description : `${ + data.remarks.totals + .remarks} optimization opportunities found in your code`, + impact : 'Potential performance improvement' + }); + } + + // Binary size optimization + if (data.binarySize?.size_statistics?.total_size > + 10 * 1024 * 1024) { // > 10MB + recommendations.push({ + title : 'Consider Binary Size Optimization', + description : + 'Binary size is large, consider link-time optimization or unused code removal', + impact : 'Reduce binary size' + }); + } + + // Warning cleanup + if (data.diagnostics?.by_level?.warning > 20) { + recommendations.push({ + title : 'Clean Up Warnings', + description : + 'Addressing compiler warnings can improve code quality and catch potential bugs', + impact : 'Better code quality' + }); + } + + return recommendations.slice(0, 4); // Return top 4 recommendations + } + + /** + * Get CSS class for issue severity + */ + getIssueColorClass(severity) { + switch (severity) { + case 'error': + return 'bg-red-50 border border-red-200'; + case 'warning': + return 'bg-yellow-50 border border-yellow-200'; + case 'info': + return 'bg-blue-50 border border-blue-200'; + default: + return 'bg-gray-50 border border-gray-200'; + } + } + + /** + * Get icon for issue severity + */ + getIssueIcon(severity) { + const iconClass = severity === 'error' ? 'text-red-500' + : severity === 'warning' ? 'text-yellow-500' + : 'text-blue-500'; + + const iconPath = + severity === 'error' + ? 'M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-2.5L13.732 4c-.77-.833-1.964-.833-2.732 0L3.732 16.5c-.77.833.192 2.5 1.732 2.5z' + : 'M13 16h-1v-4h-1m1-4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z'; + + return ` + + + + `; + } + + /** + * Show dashboard error + */ + showDashboardError(message) { + console.error('Dashboard error:', message); + } + + /** + * Clear all charts + */ + clearCharts() { + Object.values(this.charts).forEach(chart => { + if (chart && typeof chart.destroy === 'function') { + chart.destroy(); + } + }); + this.charts = {}; + } + + /** + * Refresh dashboard data + */ + async refresh() { + if (this.currentData) { + await this.updateData(this.currentData); + } + } + + /** + * Get dashboard statistics + */ + getStats() { + return { + chartsActive : Object.keys(this.charts).length, + lastUpdate : this.currentData ? new Date().toISOString() : null, + isInitialized : this.isInitialized + }; + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/explorer.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/explorer.js new file mode 100644 index 0000000000000..efd889d81f22a --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/explorer.js @@ -0,0 +1,588 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Explorer Module + */ + +import {CodeViewer} from './code-viewer.js'; +import {Utils} from './utils.js'; + +export class Explorer { + constructor(apiClient, app) { + this.apiClient = apiClient; + this.app = app; + this.currentFile = null; + this.currentViewType = 'assembly'; + this.sourceContent = null; + this.outputContent = null; + this.availableFiles = []; + this.isLoading = false; + + this.initializeElements(); + } + + initializeElements() { + this.fileSelector = document.getElementById('file-selector'); + this.viewTypeSelector = document.getElementById('view-type-selector'); + this.sourceContainer = document.getElementById('source-code-container'); + this.outputContainer = document.getElementById('output-container'); + this.rightPanelTitle = document.getElementById('right-panel-title'); + this.copyBtn = document.getElementById('copy-output-btn'); + this.downloadBtn = document.getElementById('download-output-btn'); + + // Inline data toggle buttons + this.toggleDiagnosticsBtn = + document.getElementById('toggle-diagnostics-btn'); + this.toggleRemarksBtn = document.getElementById('toggle-remarks-btn'); + + // Initialize inline data state + this.inlineDataVisible = {diagnostics : false, remarks : false}; + this.inlineData = null; + + // Initialize code viewers + this.sourceViewer = null; + this.outputViewer = null; + } + + init() { + this.setupEventListeners(); + console.log('🔍 Explorer initialized'); + } + + setupEventListeners() { + // File selection change + this.fileSelector?.addEventListener('change', (event) => { + const selectedFile = event.target.value; + if (selectedFile && selectedFile !== this.currentFile) { + const fileData = + this.availableFiles.find(f => (f.path || f.name) === selectedFile); + this.updateViewTypeSelector(fileData); + this.loadFile(selectedFile); + } + }); + + // View type selection change + this.viewTypeSelector?.addEventListener('change', async (event) => { + const newViewType = event.target.value; + if (newViewType !== this.currentViewType) { + this.currentViewType = newViewType; + this.updateRightPanelTitle(); + if (this.currentFile) { + console.log(`Loading new view type: ${newViewType} for file: ${ + this.currentFile}`); + const outputResponse = + await this.loadOutput(this.currentFile, newViewType); + if (outputResponse) { + this.displayOutput(outputResponse); + } + this.updateActionButtons(); + } + } + }); + + // Copy and download buttons + this.copyBtn?.addEventListener('click', () => this.copyToClipboard()); + this.downloadBtn?.addEventListener('click', () => this.downloadOutput()); + + // Inline data toggle buttons + this.toggleDiagnosticsBtn?.addEventListener('click', (e) => { + console.log('Diagnostics button clicked'); + e.preventDefault(); + this.toggleInlineData('diagnostics'); + }); + + this.toggleRemarksBtn?.addEventListener('click', (e) => { + console.log('Remarks button clicked'); + e.preventDefault(); + this.toggleInlineData('remarks'); + }); + } + + async loadAvailableFiles() { + try { + // Get current unit name + const currentUnitName = this.app.currentUnit || + document.getElementById('unit-selector')?.value; + + if (!currentUnitName) { + console.log('No unit selected, cannot load files'); + this.availableFiles = []; + this.updateFileSelector(); + return; + } + + console.log('Loading files for unit:', currentUnitName); + const response = await this.apiClient.getSourceFiles(currentUnitName); + if (response.success && response.data) { + this.availableFiles = response.data.files || []; + console.log('Loaded files:', this.availableFiles); + this.updateFileSelector(); + } else { + console.warn('Failed to load source files:', response.error); + this.showError('Failed to load available files'); + } + } catch (error) { + console.error('Error loading available files:', error); + this.showError('Error loading available files'); + } + } + + updateFileSelector() { + if (!this.fileSelector) + return; + + this.fileSelector.innerHTML = ''; + + this.availableFiles.forEach(file => { + const option = document.createElement('option'); + option.value = file.path || file.name; + option.textContent = file.display_name || file.name; + this.fileSelector.appendChild(option); + }); + + // Auto-select first file if available + if (this.availableFiles.length > 0) { + const firstFile = this.availableFiles[0]; + this.fileSelector.value = firstFile.path || firstFile.name; + this.updateViewTypeSelector(firstFile); + this.loadFile(firstFile.path || firstFile.name); + } + } + + updateViewTypeSelector(selectedFile = null) { + if (!this.viewTypeSelector) + return; + + if (!selectedFile) { + const selectedPath = this.fileSelector?.value; + selectedFile = + this.availableFiles.find(f => (f.path || f.name) === selectedPath); + } + + this.viewTypeSelector.innerHTML = ''; + + const availableArtifacts = selectedFile?.available_artifacts || []; + + const artifactDisplayNames = { + 'assembly' : 'Assembly', + 'ir' : 'LLVM IR', + 'ast-json' : 'AST JSON', + 'object' : 'Object Code', + 'preprocessed' : 'Preprocessed', + 'macro-expansion' : 'Macro Expansion' + }; + + availableArtifacts.forEach(artifact => { + const option = document.createElement('option'); + option.value = artifact; + option.textContent = + artifactDisplayNames[artifact] || Utils.capitalize(artifact); + this.viewTypeSelector.appendChild(option); + }); + + // Set default selection (prefer assembly, then ir, then other code + // artifacts) + if (availableArtifacts.length > 0) { + const preferredOrder = [ + 'assembly', 'ir', 'ast-json', 'object', 'preprocessed', + 'macro-expansion' + ]; + let defaultType = availableArtifacts[0]; + + for (const preferred of preferredOrder) { + if (availableArtifacts.includes(preferred)) { + defaultType = preferred; + break; + } + } + + this.viewTypeSelector.value = defaultType; + this.currentViewType = defaultType; + this.updateRightPanelTitle(); + } + } + + async loadFile(filePath) { + if (this.isLoading) + return; + + this.isLoading = true; + this.currentFile = filePath; + this.showLoadingStates(); + + try { + const [sourceResponse, outputResponse] = await Promise.all([ + this.loadSourceCode(filePath), + this.loadOutput(filePath, this.currentViewType) + ]); + + if (sourceResponse) { + this.displaySourceCode(sourceResponse); + } + + if (outputResponse) { + this.displayOutput(outputResponse); + } + + this.updateActionButtons(); + + } catch (error) { + console.error('Error loading file:', error); + this.showError('Failed to load file content'); + } finally { + this.isLoading = false; + } + } + + async loadSourceCode(filePath) { + try { + const response = await this.apiClient.getSourceCode(filePath); + if (response.success) { + this.sourceContent = response.data; + console.log('Loaded source code with inline data:', + response.data.inline_data); + return response.data; + } else { + console.warn('Failed to load source code:', response.error); + return null; + } + } catch (error) { + console.error('Error loading source code:', error); + return null; + } + } + + async loadOutput(filePath, viewType) { + try { + const endpoint = `explorer/${encodeURIComponent(viewType)}/${ + encodeURIComponent(filePath)}`; + const response = await this.apiClient.request(endpoint); + console.log(`Response for ${viewType}:`, response); + + if (response?.success && response.data) { + const content = + response.data.content || response.data.data || response.data || ''; + + // Debug: Check if content contains HTML tags + if (typeof content === 'string' && content.includes('Loading source code...'; + } + + if (this.outputContainer) { + this.outputContainer.innerHTML = + '
Loading output...
'; + } + } + + showSourceError(message) { + if (this.sourceContainer) { + this.sourceContainer.innerHTML = ` +
+
+ + + +

${message}

+
+
+ `; + } + } + + showOutputError(message) { + if (this.outputContainer) { + this.outputContainer.innerHTML = ` +
+
+ + + +

${message}

+
+
+ `; + } + } + + showError(message) { + console.error('Explorer error:', message); + this.showSourceError('Failed to load content'); + this.showOutputError('Failed to load content'); + } + + updateRightPanelTitle() { + if (!this.rightPanelTitle) + return; + + const titles = { + 'assembly' : 'Assembly', + 'ir' : 'LLVM IR', + 'optimized-ir' : 'Optimized IR', + 'ast-json' : 'AST JSON', + 'object' : 'Object Code' + }; + + this.rightPanelTitle.textContent = titles[this.currentViewType] || 'Output'; + } + + updateActionButtons() { + const hasContent = this.outputContent && (this.outputContent.content || + this.outputContent.output); + + if (this.copyBtn) { + this.copyBtn.disabled = !hasContent; + } + + if (this.downloadBtn) { + this.downloadBtn.disabled = !hasContent; + } + } + + async copyToClipboard() { + if (!this.outputContent) + return; + + const text = this.outputContent.content || this.outputContent.output || ''; + + try { + await navigator.clipboard.writeText(text); + + const originalText = this.copyBtn?.textContent; + if (this.copyBtn) { + this.copyBtn.textContent = 'Copied!'; + setTimeout(() => { this.copyBtn.textContent = originalText; }, 2000); + } + } catch (error) { + console.error('Failed to copy to clipboard:', error); + } + } + + downloadOutput() { + if (!this.outputContent || !this.currentFile) + return; + + const content = + this.outputContent.content || this.outputContent.output || ''; + const filename = + `${this.currentFile.split('/').pop()}.${this.currentViewType}`; + + const blob = new Blob([ content ], {type : 'text/plain'}); + const url = window.URL.createObjectURL(blob); + + const a = document.createElement('a'); + a.style.display = 'none'; + a.href = url; + a.download = filename; + document.body.appendChild(a); + a.click(); + + window.URL.revokeObjectURL(url); + document.body.removeChild(a); + } + + async refresh() { + if (this.currentFile) { + await this.loadFile(this.currentFile); + } else { + await this.loadAvailableFiles(); + } + } + + async onActivate() { + if (this.availableFiles.length === 0) { + await this.loadAvailableFiles(); + } + } + + onDeactivate() { + // Clean up if needed + } + + // Inline data methods + updateInlineDataButtons() { + let diagnosticsCount = 0; + let remarksCount = 0; + + // Handle both array and string formats + if (this.inlineData?.diagnostics) { + if (Array.isArray(this.inlineData.diagnostics)) { + diagnosticsCount = this.inlineData.diagnostics.length; + } else if (typeof this.inlineData.diagnostics === 'string') { + // Count diagnostic lines in string format + diagnosticsCount = (this.inlineData.diagnostics.match( + /:\d+:\d+:\s+(warning|error|note|info):/g) || + []).length; + } + } + + if (this.inlineData?.remarks) { + if (Array.isArray(this.inlineData.remarks)) { + remarksCount = this.inlineData.remarks.length; + } else if (typeof this.inlineData.remarks === 'string') { + // Count YAML blocks in string format + remarksCount = + (this.inlineData.remarks.match(/^---\s+!/gm) || []).length; + } + } + + if (this.toggleDiagnosticsBtn) { + this.toggleDiagnosticsBtn.disabled = diagnosticsCount === 0; + this.toggleDiagnosticsBtn.textContent = + diagnosticsCount > 0 ? `Diagnostics (${diagnosticsCount})` + : 'Diagnostics'; + } + + if (this.toggleRemarksBtn) { + this.toggleRemarksBtn.disabled = remarksCount === 0; + this.toggleRemarksBtn.textContent = + remarksCount > 0 ? `Remarks (${remarksCount})` : 'Remarks'; + } + } + + toggleInlineData(type) { + console.log(`Explorer.toggleInlineData called: ${type}`); + console.log('Current state:', this.inlineDataVisible[type]); + console.log('Available inline data:', this.inlineData); + + this.inlineDataVisible[type] = !this.inlineDataVisible[type]; + + // Update button appearance + const button = type === 'diagnostics' ? this.toggleDiagnosticsBtn + : this.toggleRemarksBtn; + console.log('Button found:', !!button); + + if (button) { + if (this.inlineDataVisible[type]) { + button.classList.add('bg-blue-600', 'text-white', 'border-blue-600'); + button.classList.remove('text-gray-600', 'border-gray-300', + 'hover:bg-gray-200'); + } else { + button.classList.remove('bg-blue-600', 'text-white', 'border-blue-600'); + button.classList.add('text-gray-600', 'border-gray-300', + 'hover:bg-gray-200'); + } + } + + // Update the source viewer inline data visibility + if (this.sourceViewer) { + console.log('Calling sourceViewer.toggleInlineData'); + this.sourceViewer.toggleInlineData(type, this.inlineDataVisible[type]); + } else { + console.log('No sourceViewer available'); + } + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/performance.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/performance.js new file mode 100644 index 0000000000000..5ee40685f6a4a --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/performance.js @@ -0,0 +1,1473 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +class PerformanceManager { + constructor() { + this.currentViewType = 'time-order'; + this.currentUnit = null; + this.timeTraceData = null; + this.runtimeTraceData = null; + + // Interactive state + this.viewports = { + timeTrace : {offsetX : 0, scaleX : 1, offsetY : 0, scaleY : 1}, + runtimeTrace : {offsetX : 0, scaleX : 1, offsetY : 0, scaleY : 1} + }; + + // Mouse/touch interaction state + this.isDragging = false; + this.lastMousePos = {x : 0, y : 0}; + this.isZooming = false; + + // Search + this.searchQuery = ''; + this.searchResults = {timeTrace : [], runtimeTrace : []}; + + // Constants + this.FRAME_HEIGHT = 18; + this.MINIMAP_HEIGHT = 50; + this.MIN_FRAME_WIDTH_FOR_TEXT = 25; + this.PADDING = 4; + this.MIN_ZOOM = 0.1; + this.MAX_ZOOM = 100; + + this.initializeEventListeners(); + } + + initializeEventListeners() { + const viewTypeSelector = document.getElementById('view-type-selector-perf'); + if (viewTypeSelector) { + viewTypeSelector.addEventListener('change', (e) => { + this.currentViewType = e.target.value; + this.renderBothViews(); + }); + } + + const refreshBtn = document.getElementById('refresh-performance-btn'); + if (refreshBtn) { + refreshBtn.addEventListener('click', + () => { this.loadAllPerformanceData(); }); + } + + const unitSelector = document.getElementById('unit-selector'); + if (unitSelector) { + unitSelector.addEventListener('change', (e) => { + this.currentUnit = e.target.value; + this.loadAllPerformanceData(); + }); + } + + this.initializeSearch(); + } + + initializeSearch() { + const controlsDiv = document.querySelector( + '#performance-content .flex.items-center.space-x-4'); + if (controlsDiv && !document.getElementById('perf-search')) { + const searchDiv = document.createElement('div'); + searchDiv.className = 'flex items-center space-x-2'; + searchDiv.innerHTML = ` + + + + `; + controlsDiv.appendChild(searchDiv); + + const searchInput = document.getElementById('perf-search'); + searchInput.addEventListener('input', (e) => { + this.searchQuery = e.target.value.toLowerCase(); + this.performSearch(); + }); + } + } + + async loadAllPerformanceData() { + try { + const params = new URLSearchParams(); + if (this.currentUnit) { + params.append('unit', this.currentUnit); + } + + // Load both time-trace and runtime-trace data + const [timeTraceResponse, runtimeTraceResponse] = await Promise.all([ + this.loadTraceData('time-trace', params), + this.loadTraceData('runtime-trace', params) + ]); + + this.timeTraceData = timeTraceResponse; + this.runtimeTraceData = runtimeTraceResponse; + + // Reset viewports when new data is loaded + this.resetViewports(); + + this.renderBothViews(); + this.updateIndividualStats(); + } catch (error) { + console.error('Failed to load performance data:', error); + this.showError('Failed to load performance data: ' + error.message); + } + } + + async loadTraceData(traceType, params) { + const endpoints = { + 'time-order' : `${traceType}/flamegraph`, + 'sandwich' : `${traceType}/sandwich` + }; + + const endpoint = `/api/${endpoints[this.currentViewType]}`; + + console.log(`Loading ${traceType} data from:`, endpoint, + 'params:', params.toString()); + + try { + const response = await fetch(`${endpoint}?${params}`); + const result = await response.json(); + + console.log(`${traceType} response:`, result); + + if (result.success) { + // Add source marking to help distinguish data + if (result.data && result.data.samples) { + result.data.samples.forEach(sample => { sample.source = traceType; }); + } + if (result.data && result.data.functions) { + result.data.functions.forEach(func => { func.source = traceType; }); + } + return result.data; + } else { + console.warn(`No ${traceType} data available:`, + result.message || result.error); + return null; + } + } catch (error) { + console.error(`Failed to load ${traceType} data:`, error); + return null; + } + } + + renderBothViews() { + switch (this.currentViewType) { + case 'time-order': + this.renderTimeOrderViews(); + break; + case 'sandwich': + this.renderSandwichViews(); + break; + } + } + + renderTimeOrderViews() { + // Create dual time-order layout + const container = document.getElementById('performance-visualization'); + container.innerHTML = ` +
+ +
+
+

+ + + + Compilation Time Trace +
+ Loading... + ? +
+

+
+
+ ${this.createInteractiveCanvasContainer('time-trace')} +
+
+ + +
+
+

+ + + + Runtime Offloading Trace +
+ Loading... + ? +
+

+
+
+ ${ + this.createInteractiveCanvasContainer('runtime-trace')} +
+
+
+ `; + + this.setupInteractiveCanvases(); + } + + createInteractiveCanvasContainer(traceType) { + return ` +
+ +
+ + 100% +
+ + + + + + + + + + + +
+ Loading... +
+ + +
+ ? Keys +
+
+ `; + } + + setupInteractiveCanvases() { + ['time-trace', 'runtime-trace'].forEach( + traceType => { this.setupSingleInteractiveCanvas(traceType); }); + + // Add keyboard listeners + document.addEventListener('keydown', this.handleKeyDown.bind(this)); + document.addEventListener('keyup', this.handleKeyUp.bind(this)); + } + + setupSingleInteractiveCanvas(traceType) { + const mainCanvas = document.getElementById(`${traceType}-main`); + const minimapCanvas = document.getElementById(`${traceType}-minimap`); + const resetBtn = document.getElementById(`${traceType}-reset-btn`); + + if (!mainCanvas || !minimapCanvas) + return; + + // Setup main canvas + const container = mainCanvas.parentElement; + const containerWidth = container.clientWidth; + + mainCanvas.width = containerWidth * window.devicePixelRatio; + mainCanvas.height = 350 * window.devicePixelRatio; + mainCanvas.style.width = containerWidth + 'px'; + mainCanvas.style.height = '350px'; + + const mainCtx = mainCanvas.getContext('2d'); + mainCtx.scale(window.devicePixelRatio, window.devicePixelRatio); + + // Setup minimap + minimapCanvas.width = containerWidth * window.devicePixelRatio; + minimapCanvas.height = this.MINIMAP_HEIGHT * window.devicePixelRatio; + minimapCanvas.style.width = containerWidth + 'px'; + minimapCanvas.style.height = this.MINIMAP_HEIGHT + 'px'; + + const minimapCtx = minimapCanvas.getContext('2d'); + minimapCtx.scale(window.devicePixelRatio, window.devicePixelRatio); + + // Add interactive event listeners + this.addCanvasInteractions(mainCanvas, minimapCanvas, traceType); + + // Reset button + if (resetBtn) { + resetBtn.addEventListener('click', + () => { this.resetViewport(traceType); }); + } + + // Render trace data + const data = + traceType === 'time-trace' ? this.timeTraceData : this.runtimeTraceData; + this.renderSingleTimeOrder(traceType, data, mainCtx, minimapCtx, mainCanvas, + minimapCanvas); + } + + renderSingleTimeOrder(traceType, data, mainCtx, minimapCtx, mainCanvas, + minimapCanvas) { + const statusEl = document.getElementById(`${traceType}-status`); + + if (!data || !data.samples || data.samples.length === 0) { + statusEl.textContent = 'No data available'; + statusEl.className = + statusEl.className.replace('text-gray-500', 'text-red-500'); + + const canvasWidth = mainCanvas.width / window.devicePixelRatio; + const canvasHeight = mainCanvas.height / window.devicePixelRatio; + + mainCtx.clearRect(0, 0, canvasWidth, canvasHeight); + mainCtx.fillStyle = '#f3f4f6'; + mainCtx.fillRect(0, 0, canvasWidth, canvasHeight); + + mainCtx.fillStyle = '#6b7280'; + mainCtx.font = '14px system-ui'; + mainCtx.textAlign = 'center'; + mainCtx.fillText('No trace data available', canvasWidth / 2, + canvasHeight / 2); + + minimapCtx.clearRect(0, 0, minimapCanvas.width / window.devicePixelRatio, + this.MINIMAP_HEIGHT); + return; + } + + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + + statusEl.textContent = `${data.samples.length} events • Zoom: ${ + Math.round(viewport.scaleX * 100)}%`; + statusEl.className = + statusEl.className.replace('text-red-500', 'text-gray-500'); + + // Render flamechart with viewport + this.renderFlamechart(data.samples, mainCtx, mainCanvas, traceType); + this.renderMinimap(data.samples, minimapCtx, minimapCanvas, traceType); + + // Update zoom display + this.updateZoomDisplay(traceType); + } + + renderFlamechart(samples, ctx, canvas, traceType) { + const canvasWidth = canvas.width / window.devicePixelRatio; + const canvasHeight = canvas.height / window.devicePixelRatio; + + ctx.clearRect(0, 0, canvasWidth, canvasHeight); + + if (!samples || samples.length === 0) + return; + + // Get viewport for this trace + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + + // Calculate time bounds + const times = + samples.map(s => [s.timestamp, s.timestamp + s.duration]).flat(); + const minTime = Math.min(...times); + const maxTime = Math.max(...times); + const totalDuration = maxTime - minTime; + + if (totalDuration === 0) + return; + + // Build layers to avoid overlaps + const layers = this.buildNonOverlappingLayers(samples); + + // Apply viewport transformations + const viewportLeft = viewport.offsetX * totalDuration; + const viewportWidth = totalDuration / viewport.scaleX; + const visibleTimeStart = minTime + viewportLeft; + const visibleTimeEnd = visibleTimeStart + viewportWidth; + + // Calculate scaling with zoom + const timeScale = canvasWidth / viewportWidth; + const maxVisibleLayers = Math.floor(canvasHeight / this.FRAME_HEIGHT); + + // Apply vertical offset + const layerOffset = Math.floor(viewport.offsetY * layers.length); + const visibleLayers = + layers.slice(layerOffset, layerOffset + maxVisibleLayers); + + // Render visible layers + visibleLayers.forEach((layer, layerIndex) => { + const y = layerIndex * this.FRAME_HEIGHT; + + layer.forEach(sample => { + const sampleStart = sample.timestamp; + const sampleEnd = sample.timestamp + sample.duration; + + // Only render if sample is visible in viewport + if (sampleEnd >= visibleTimeStart && sampleStart <= visibleTimeEnd) { + const x = (sampleStart - visibleTimeStart) * timeScale; + const width = Math.max(1, sample.duration * timeScale); + + // Only draw if within canvas bounds + if (x < canvasWidth && x + width > 0) { + this.drawFrame(ctx, sample, x, y, width, this.FRAME_HEIGHT); + } + } + }); + }); + + // Render time axis for visible range + this.renderTimeAxis(ctx, visibleTimeStart, viewportWidth, canvasWidth, + canvasHeight); + + // Store current viewport bounds for interaction + canvas.dataset.visibleTimeStart = visibleTimeStart; + canvas.dataset.visibleTimeEnd = visibleTimeEnd; + canvas.dataset.timeScale = timeScale; + } + + buildNonOverlappingLayers(samples) { + const layers = []; + const sortedSamples = + [...samples ].sort((a, b) => a.timestamp - b.timestamp); + + for (const sample of sortedSamples) { + let placed = false; + + // Try to place in existing layer + for (let i = 0; i < layers.length; i++) { + const layer = layers[i]; + const lastInLayer = layer[layer.length - 1]; + + if (!lastInLayer || + lastInLayer.timestamp + lastInLayer.duration <= sample.timestamp) { + layer.push(sample); + placed = true; + break; + } + } + + // Create new layer if needed + if (!placed) { + layers.push([ sample ]); + } + } + + return layers; + } + + drawFrame(ctx, sample, x, y, width, height) { + const isSearchMatch = this.searchQuery && + sample.name.toLowerCase().includes(this.searchQuery); + + // Frame background + let color = this.getCategoryColor(sample.category); + if (this.searchQuery && !isSearchMatch) { + color = this.fadeColor(color); + } + + ctx.fillStyle = color; + ctx.fillRect(x, y, width, height); + + // Frame border + ctx.strokeStyle = 'rgba(255, 255, 255, 0.3)'; + ctx.lineWidth = 0.5; + ctx.strokeRect(x, y, width, height); + + // Search highlight + if (isSearchMatch) { + ctx.strokeStyle = '#fbbf24'; + ctx.lineWidth = 2; + ctx.strokeRect(x, y, width, height); + } + + // Frame text + if (width > this.MIN_FRAME_WIDTH_FOR_TEXT) { + ctx.fillStyle = this.getTextColor(color); + ctx.font = '11px system-ui'; + ctx.textAlign = 'left'; + ctx.textBaseline = 'middle'; + + const text = this.truncateText(sample.name, width - 2 * this.PADDING); + ctx.fillText(text, x + this.PADDING, y + height / 2); + } + } + + renderTimeAxis(ctx, minTime, totalDuration, canvasWidth, canvasHeight) { + if (totalDuration === 0) + return; + + // Calculate intervals + const targetInterval = totalDuration / 8; + const magnitude = Math.pow(10, Math.floor(Math.log10(targetInterval))); + let interval = magnitude; + + if (targetInterval / interval > 5) + interval *= 5; + else if (targetInterval / interval > 2) + interval *= 2; + + // Draw axis + ctx.strokeStyle = '#e5e7eb'; + ctx.lineWidth = 1; + ctx.fillStyle = '#6b7280'; + ctx.font = '10px system-ui'; + ctx.textAlign = 'center'; + + const timeScale = canvasWidth / totalDuration; + + for (let time = Math.ceil(minTime / interval) * interval; + time <= minTime + totalDuration; time += interval) { + + const x = (time - minTime) * timeScale; + + if (x >= 0 && x <= canvasWidth) { + // Grid line + ctx.beginPath(); + ctx.moveTo(x, 0); + ctx.lineTo(x, canvasHeight - 20); + ctx.stroke(); + + // Time label + const timeMs = (time / 1000).toFixed(1); + ctx.fillText(`${timeMs}ms`, x, canvasHeight - 5); + } + } + } + + renderMinimap(samples, ctx, canvas, traceType) { + const canvasWidth = canvas.width / window.devicePixelRatio; + const canvasHeight = canvas.height / window.devicePixelRatio; + + ctx.clearRect(0, 0, canvasWidth, canvasHeight); + + if (!samples || samples.length === 0) + return; + + const times = + samples.map(s => [s.timestamp, s.timestamp + s.duration]).flat(); + const minTime = Math.min(...times); + const maxTime = Math.max(...times); + const totalDuration = maxTime - minTime; + + if (totalDuration === 0) + return; + + const timeScale = canvasWidth / totalDuration; + const barHeight = canvasHeight - 10; + + // Render simplified bars + samples.forEach(sample => { + const x = (sample.timestamp - minTime) * timeScale; + const width = Math.max(1, sample.duration * timeScale); + const y = 5; + + ctx.fillStyle = this.getCategoryColor(sample.category); + ctx.fillRect(x, y, width, barHeight); + }); + } + + renderSandwichViews() { + const container = document.getElementById('performance-visualization'); + container.innerHTML = ` +
+ +
+
+
+

Functions

+
+ + 0 functions +
+
+
+
+
+ + + + + + + + + + + +
+ Total + + + + + Self + + + + + Function + + + +
+
+
+
+ + +
+
+

Select a function to view details

+
+
+ +
+
+ Callers (functions that call this) +
+
+ Select a function to see its callers +
+
+ + +
+
+ Callees (functions called by this) +
+
+ Select a function to see its callees +
+
+
+
+
+ `; + + this.renderSpeedscopeSandwich(); + } + + renderSpeedscopeSandwich() { + // Combine data from both traces + const allFunctions = []; + + // Add time-trace functions + if (this.timeTraceData && this.timeTraceData.functions) { + this.timeTraceData.functions.forEach(func => { + allFunctions.push({ + ...func, + source : 'compilation', + color : this.getCategoryColor(func.category) + }); + }); + } + + // Add runtime-trace functions + if (this.runtimeTraceData && this.runtimeTraceData.functions) { + this.runtimeTraceData.functions.forEach(func => { + allFunctions.push({ + ...func, + source : 'runtime', + color : this.getCategoryColor(func.category) + }); + }); + } + + // Merge functions with same name + const functionMap = new Map(); + allFunctions.forEach(func => { + const key = func.name; + if (functionMap.has(key)) { + const existing = functionMap.get(key); + existing.total_time += func.total_time; + existing.call_count += func.call_count; + existing.sources = existing.sources || [ existing.source ]; + if (!existing.sources.includes(func.source)) { + existing.sources.push(func.source); + } + } else { + functionMap.set(key, { + ...func, + self_time : func.total_time, + sources : [ func.source ] + }); + } + }); + + const mergedFunctions = Array.from(functionMap.values()); + const totalTime = mergedFunctions.reduce((sum, f) => sum + f.total_time, 0); + + // Sort by total time by default + mergedFunctions.sort((a, b) => b.total_time - a.total_time); + + this.renderFunctionTable(mergedFunctions, totalTime); + this.setupSandwichInteractions(mergedFunctions); + + // Update count + const countEl = document.getElementById('sandwich-count'); + if (countEl) { + countEl.textContent = `${mergedFunctions.length} functions`; + } + } + + renderFunctionTable(functions, totalTime) { + const tbody = document.getElementById('sandwich-table-body'); + if (!tbody) + return; + + tbody.innerHTML = + functions + .map((func, index) => { + const totalPerc = + totalTime > 0 ? (func.total_time / totalTime * 100) : 0; + const selfPerc = + totalTime > 0 ? (func.self_time / totalTime * 100) : 0; + const isMatch = + this.searchQuery && + func.name.toLowerCase().includes(this.searchQuery); + + return ` + + +
+
+
+
+ ${ + (func.total_time / 1000).toFixed(2)}ms + (${ + totalPerc.toFixed(1)}%) +
+ + +
+
+
+
+ ${ + (func.self_time / 1000).toFixed(2)}ms + (${ + selfPerc.toFixed(1)}%) +
+ + +
+
+
+
+ ${this.highlightSearchInText(func.name)} +
+
+ ${func.call_count.toLocaleString()} calls + ${ + func.sources ? ' • ' + func.sources.join(', ') : ''} +
+
+
+ + + `; + }) + .join(''); + } + + setupSandwichInteractions(functions) { + // Set up table row clicks + const rows = document.querySelectorAll('.sandwich-row'); + rows.forEach(row => { + row.addEventListener('click', () => { + // Remove previous selection + document.querySelectorAll('.sandwich-row') + .forEach(r => r.classList.remove('bg-blue-50')); + + // Add selection to clicked row + row.classList.add('bg-blue-50'); + + const index = parseInt(row.dataset.index); + const selectedFunction = functions[index]; + + this.showFunctionDetails(selectedFunction); + }); + }); + + // Set up sorting + const sortSelect = document.getElementById('sandwich-sort'); + if (sortSelect) { + sortSelect.addEventListener( + 'change', () => { this.sortFunctions(functions, sortSelect.value); }); + } + + // Set up column header sorting + const headers = document.querySelectorAll('th[data-sort]'); + headers.forEach(header => { + header.addEventListener('click', () => { + const sortBy = header.dataset.sort; + this.sortFunctions(functions, sortBy); + }); + }); + } + + sortFunctions(functions, sortBy) { + switch (sortBy) { + case 'total': + functions.sort((a, b) => b.total_time - a.total_time); + break; + case 'self': + functions.sort((a, b) => b.self_time - a.self_time); + break; + case 'name': + functions.sort((a, b) => a.name.localeCompare(b.name)); + break; + } + + const totalTime = functions.reduce((sum, f) => sum + f.total_time, 0); + this.renderFunctionTable(functions, totalTime); + this.setupSandwichInteractions(functions); + } + + showFunctionDetails(func) { + const titleEl = document.getElementById('sandwich-detail-title'); + const callersEl = document.getElementById('callers-chart'); + const calleesEl = document.getElementById('callees-chart'); + + if (titleEl) { + titleEl.innerHTML = ` +
+
+ ${func.name} + ${ + (func.total_time / 1000).toFixed(2)}ms total +
+ `; + } + + // For now, show placeholder content in callers/callees + if (callersEl) { + callersEl.innerHTML = ` +
+
Callers for ${ + func.name}
+
Implementation would show flamegraph of functions that call this
+
+
+ Total calls: ${func.call_count.toLocaleString()}
+ Sources: ${ + func.sources ? func.sources.join(', ') : 'unknown'} +
+
+
+ `; + } + + if (calleesEl) { + calleesEl.innerHTML = ` +
+
Callees for ${ + func.name}
+
Implementation would show flamegraph of functions called by this
+
+
+ Self time: ${ + (func.self_time / 1000).toFixed(2)}ms
+ Category: ${func.category || 'Unknown'} +
+
+
+ `; + } + } + + performSearch() { + if (this.currentViewType === 'sandwich') { + this.renderSandwichTables(); + } else { + this.renderTimeOrderViews(); + } + this.updateSearchResultsDisplay(); + } + + updateSearchResultsDisplay() { + const countEl = document.getElementById('search-results-count'); + if (!countEl) + return; + + let totalResults = 0; + + if (this.timeTraceData && this.timeTraceData.functions) { + totalResults += + this.timeTraceData.functions + .filter(f => !this.searchQuery || + f.name.toLowerCase().includes(this.searchQuery)) + .length; + } + + if (this.runtimeTraceData && this.runtimeTraceData.functions) { + totalResults += + this.runtimeTraceData.functions + .filter(f => !this.searchQuery || + f.name.toLowerCase().includes(this.searchQuery)) + .length; + } + + countEl.textContent = this.searchQuery ? `${totalResults} results` : ''; + } + + updateIndividualStats() { + // Update individual trace stats in headers + this.updateTraceStats('time-trace', this.timeTraceData); + this.updateTraceStats('runtime-trace', this.runtimeTraceData); + + // Update global stats if elements exist + const totalEventsEl = document.getElementById('perf-total-events'); + const totalDurationEl = document.getElementById('perf-total-duration'); + const avgDurationEl = document.getElementById('perf-avg-duration'); + const viewModeEl = document.getElementById('perf-view-mode'); + + if (totalEventsEl || totalDurationEl || avgDurationEl || viewModeEl) { + let totalEvents = 0; + let totalDuration = 0; + + // Combine stats from both traces + [this.timeTraceData, this.runtimeTraceData].forEach(data => { + if (data) { + if (data.samples) { + totalEvents += data.samples.length; + totalDuration += + data.samples.reduce((sum, s) => sum + s.duration, 0); + } else if (data.functions) { + totalEvents += + data.functions.reduce((sum, f) => sum + f.call_count, 0); + totalDuration += + data.functions.reduce((sum, f) => sum + f.total_time, 0); + } + } + }); + + if (totalEventsEl) + totalEventsEl.textContent = totalEvents.toLocaleString(); + if (totalDurationEl) + totalDurationEl.textContent = `${(totalDuration / 1000).toFixed(2)} ms`; + if (avgDurationEl) { + const avg = totalEvents > 0 ? totalDuration / totalEvents : 0; + avgDurationEl.textContent = `${(avg / 1000).toFixed(2)} ms`; + } + if (viewModeEl) { + viewModeEl.textContent = this.currentViewType.charAt(0).toUpperCase() + + this.currentViewType.slice(1); + } + } + } + + updateTraceStats(traceType, data) { + const statsEl = document.getElementById(`${traceType}-stats`); + if (!statsEl) + return; + + if (!data) { + statsEl.textContent = 'No data'; + return; + } + + let events = 0; + let duration = 0; + let sources = new Set(); + + if (data.samples) { + events = data.samples.length; + duration = data.samples.reduce((sum, s) => sum + (s.duration || 0), 0); + data.samples.forEach(s => { + if (s.source) + sources.add(s.source); + }); + } else if (data.functions) { + events = data.functions.reduce((sum, f) => sum + f.call_count, 0); + duration = data.functions.reduce((sum, f) => sum + f.total_time, 0); + data.functions.forEach(f => { + if (f.source) + sources.add(f.source); + }); + } + + const sourceInfo = + sources.size > 0 ? ` • ${Array.from(sources).join(', ')}` : ''; + statsEl.textContent = `${events.toLocaleString()} events • ${ + (duration / 1000).toFixed(2)}ms${sourceInfo}`; + + // Update debug info in UI + const debugEl = document.getElementById(`${traceType}-debug`); + if (debugEl) { + const sourceList = Array.from(sources); + const debugInfo = + sourceList.length > 0 ? sourceList.join(',') : 'unknown'; + debugEl.textContent = debugInfo; + debugEl.title = `Data source: ${debugInfo}\nSamples: ${ + data.samples + ? data.samples.length + : 0}\nFunctions: ${data.functions ? data.functions.length : 0}`; + } + + // Add debug info in console + console.log(`${traceType} data:`, { + events, + duration, + sources : Array.from(sources), + sampleData : data.samples ? data.samples.slice(0, 3) : null, + functionData : data.functions ? data.functions.slice(0, 3) : null + }); + + // Add visual indicator if data appears to be identical between traces + if (traceType === 'runtime-trace' && window.timeTraceDataHash) { + const currentHash = this.hashData(data); + if (currentHash === window.timeTraceDataHash) { + debugEl.style.backgroundColor = '#ef4444'; + debugEl.style.color = 'white'; + debugEl.textContent = '⚠ SAME'; + debugEl.title = + 'WARNING: This data appears identical to time-trace data'; + } + } else if (traceType === 'time-trace') { + window.timeTraceDataHash = this.hashData(data); + } + } + + // Utility methods + getCategoryColor(category) { + const colors = { + 'Source' : '#10b981', // emerald-500 + 'Frontend' : '#3b82f6', // blue-500 + 'Backend' : '#f59e0b', // amber-500 + 'CodeGen' : '#ef4444', // red-500 + 'Optimizer' : '#8b5cf6', // violet-500 + 'Parse' : '#6b7280', // gray-500 + 'Runtime' : '#ec4899', // pink-500 + '' : '#9ca3af' // gray-400 + }; + return colors[category] || colors['']; + } + + getTextColor(backgroundColor) { + // Simple contrast calculation + const hex = backgroundColor.replace('#', ''); + const r = parseInt(hex.substr(0, 2), 16); + const g = parseInt(hex.substr(2, 2), 16); + const b = parseInt(hex.substr(4, 2), 16); + const brightness = (r * 299 + g * 587 + b * 114) / 1000; + return brightness > 128 ? '#1f2937' : '#f9fafb'; + } + + fadeColor(color) { + const hex = color.replace('#', ''); + const r = parseInt(hex.substr(0, 2), 16); + const g = parseInt(hex.substr(2, 2), 16); + const b = parseInt(hex.substr(4, 2), 16); + return `rgba(${r}, ${g}, ${b}, 0.3)`; + } + + truncateText(text, maxWidth) { + if (text.length * 6 <= maxWidth) + return text; + const maxChars = Math.floor(maxWidth / 6) - 3; + return text.substring(0, Math.max(0, maxChars)) + '...'; + } + + highlightSearchInText(text) { + if (!this.searchQuery) + return text; + const regex = new RegExp(`(${this.searchQuery})`, 'gi'); + return text.replace(regex, + '$1'); + } + + showError(message) { + const container = document.getElementById('performance-visualization'); + if (container) { + container.innerHTML = ` +
+
+ + + +

Error Loading Performance Data

+

${message}

+ +
+
+ `; + } + } + + async initialize() { + this.currentUnit = document.getElementById('unit-selector')?.value || null; + // Auto-load data without requiring refresh button + await this.loadAllPerformanceData(); + + // Set up auto-refresh if unit changes + const unitSelector = document.getElementById('unit-selector'); + if (unitSelector) { + unitSelector.addEventListener('change', () => { + this.currentUnit = unitSelector.value; + this.loadAllPerformanceData(); + }); + } + } + + // Interactive methods + resetViewports() { + this.viewports + .timeTrace = {offsetX : 0, scaleX : 1, offsetY : 0, scaleY : 1}; + this.viewports + .runtimeTrace = {offsetX : 0, scaleX : 1, offsetY : 0, scaleY : 1}; + } + + resetViewport(traceType) { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + this.viewports[key] = {offsetX : 0, scaleX : 1, offsetY : 0, scaleY : 1}; + this.redrawTrace(traceType); + this.updateZoomDisplay(traceType); + } + + updateZoomDisplay(traceType) { + const zoomEl = document.getElementById(`${traceType}-zoom-level`); + if (zoomEl) { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const zoom = Math.round(this.viewports[key].scaleX * 100); + zoomEl.textContent = `${zoom}%`; + } + } + + addCanvasInteractions(mainCanvas, minimapCanvas, traceType) { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + + // Main canvas interactions + mainCanvas.addEventListener('wheel', (e) => { + e.preventDefault(); + this.handleWheel(e, traceType); + }); + + mainCanvas.addEventListener('mousedown', + (e) => { this.handleMouseDown(e, traceType); }); + + mainCanvas.addEventListener('mousemove', + (e) => { this.handleMouseMove(e, traceType); }); + + mainCanvas.addEventListener('mouseup', + (e) => { this.handleMouseUp(e, traceType); }); + + mainCanvas.addEventListener( + 'dblclick', (e) => { this.handleDoubleClick(e, traceType); }); + + // Minimap interactions + minimapCanvas.addEventListener( + 'click', (e) => { this.handleMinimapClick(e, traceType); }); + + minimapCanvas.addEventListener( + 'mousedown', (e) => { this.handleMinimapDrag(e, traceType); }); + } + + handleMinimapDrag(e, traceType) { + let isDragging = false; + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + + const startDrag = (startE) => { + isDragging = true; + + const onDrag = (moveE) => { + if (!isDragging) + return; + + const rect = e.target.getBoundingClientRect(); + const currentX = (moveE.clientX - rect.left) / rect.width; + + const viewportWidthRatio = 1 / viewport.scaleX; + viewport.offsetX = currentX - viewportWidthRatio / 2; + + // Clamp to valid range + viewport.offsetX = + Math.max(0, Math.min(1 - viewportWidthRatio, viewport.offsetX)); + + this.redrawTrace(traceType); + }; + + const endDrag = () => { + isDragging = false; + document.removeEventListener('mousemove', onDrag); + document.removeEventListener('mouseup', endDrag); + }; + + document.addEventListener('mousemove', onDrag); + document.addEventListener('mouseup', endDrag); + }; + + startDrag(e); + } + + handleWheel(e, traceType) { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + + const zoomSpeed = 0.1; + const zoomFactor = e.deltaY < 0 ? (1 + zoomSpeed) : (1 - zoomSpeed); + + const newScale = Math.max( + this.MIN_ZOOM, Math.min(this.MAX_ZOOM, viewport.scaleX * zoomFactor)); + + if (newScale !== viewport.scaleX) { + const rect = e.target.getBoundingClientRect(); + const mouseX = e.clientX - rect.left; + + // Zoom towards mouse position + const canvasWidth = rect.width; + const normalizedMouseX = mouseX / canvasWidth; + + viewport.offsetX = + normalizedMouseX - + (normalizedMouseX - viewport.offsetX) * (newScale / viewport.scaleX); + viewport.scaleX = newScale; + + this.redrawTrace(traceType); + this.updateZoomDisplay(traceType); + } + } + + handleMouseDown(e, traceType) { + this.isDragging = true; + this.lastMousePos = {x : e.clientX, y : e.clientY}; + e.target.style.cursor = 'grabbing'; + } + + handleMouseMove(e, traceType) { + if (this.isDragging) { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + + const rect = e.target.getBoundingClientRect(); + const deltaX = (e.clientX - this.lastMousePos.x) / rect.width; + const deltaY = (e.clientY - this.lastMousePos.y) / rect.height; + + viewport.offsetX -= deltaX / viewport.scaleX; + viewport.offsetY -= deltaY / viewport.scaleY; + + this.lastMousePos = {x : e.clientX, y : e.clientY}; + this.redrawTrace(traceType); + } else { + // Show tooltip + this.showTooltip(e, traceType); + } + } + + handleMouseUp(e, traceType) { + this.isDragging = false; + e.target.style.cursor = 'grab'; + } + + handleDoubleClick(e, traceType) { + // Fit frame functionality - find frame under cursor and zoom to it + const frame = this.getFrameAtPosition(e, traceType); + if (frame) { + this.fitToFrame(frame, traceType); + } else { + this.resetViewport(traceType); + } + } + + handleMinimapClick(e, traceType) { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + + const rect = e.target.getBoundingClientRect(); + const clickX = (e.clientX - rect.left) / rect.width; + + viewport.offsetX = clickX - 0.5 / viewport.scaleX; + this.redrawTrace(traceType); + } + + handleKeyDown(e) { + let handled = false; + + switch (e.code) { + case 'Equal': + case 'NumpadAdd': + if (e.ctrlKey || e.metaKey) { + this.zoomIn(); + handled = true; + } + break; + case 'Minus': + case 'NumpadSubtract': + if (e.ctrlKey || e.metaKey) { + this.zoomOut(); + handled = true; + } + break; + case 'Escape': + this.resetViewports(); + this.renderTimeOrderViews(); + handled = true; + break; + case 'ArrowLeft': + this.panLeft(); + handled = true; + break; + case 'ArrowRight': + this.panRight(); + handled = true; + break; + } + + if (handled) { + e.preventDefault(); + } + } + + handleKeyUp(e) { + // Handle key release if needed + } + + zoomIn() { + ['time-trace', 'runtime-trace'].forEach(traceType => { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + viewport.scaleX = Math.min(this.MAX_ZOOM, viewport.scaleX * 1.2); + this.redrawTrace(traceType); + this.updateZoomDisplay(traceType); + }); + } + + zoomOut() { + ['time-trace', 'runtime-trace'].forEach(traceType => { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + viewport.scaleX = Math.max(this.MIN_ZOOM, viewport.scaleX / 1.2); + this.redrawTrace(traceType); + this.updateZoomDisplay(traceType); + }); + } + + panLeft() { + ['time-trace', 'runtime-trace'].forEach(traceType => { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + this.viewports[key].offsetX -= 0.1; + this.redrawTrace(traceType); + }); + } + + panRight() { + ['time-trace', 'runtime-trace'].forEach(traceType => { + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + this.viewports[key].offsetX += 0.1; + this.redrawTrace(traceType); + }); + } + + redrawTrace(traceType) { + const mainCanvas = document.getElementById(`${traceType}-main`); + const minimapCanvas = document.getElementById(`${traceType}-minimap`); + + if (!mainCanvas || !minimapCanvas) + return; + + const mainCtx = mainCanvas.getContext('2d'); + const minimapCtx = minimapCanvas.getContext('2d'); + + const data = + traceType === 'time-trace' ? this.timeTraceData : this.runtimeTraceData; + this.renderSingleTimeOrder(traceType, data, mainCtx, minimapCtx, mainCanvas, + minimapCanvas); + } + + getFrameAtPosition(e, traceType) { + // Find frame under cursor for double-click to fit + const rect = e.target.getBoundingClientRect(); + const x = e.clientX - rect.left; + const y = e.clientY - rect.top; + + return null; + } + + fitToFrame(frame, traceType) { + // Zoom and pan to fit specific frame + const key = traceType === 'time-trace' ? 'timeTrace' : 'runtimeTrace'; + const viewport = this.viewports[key]; + + // Calculate optimal zoom and offset for frame + viewport.scaleX = 2.0; + viewport.offsetX = 0; + + this.redrawTrace(traceType); + this.updateZoomDisplay(traceType); + } + + showTooltip(e, traceType) { + const tooltip = document.getElementById(`${traceType}-tooltip`); + if (!tooltip) + return; + + // Find frame at mouse position and show tooltip + const frame = this.getFrameAtPosition(e, traceType); + + if (frame) { + tooltip.innerHTML = ` +
${frame.name}
+
Duration: ${ + (frame.duration / 1000).toFixed(2)}ms
+
Category: ${ + frame.category || 'Unknown'}
+ `; + + tooltip.style.left = `${e.offsetX + 10}px`; + tooltip.style.top = `${e.offsetY - 50}px`; + tooltip.classList.remove('hidden'); + } else { + tooltip.classList.add('hidden'); + } + } + + hashData(data) { + // Simple hash function to detect if data is identical + if (!data) + return null; + + let hashString = ''; + if (data.samples) { + hashString = + data.samples.map(s => `${s.name}:${s.duration}:${s.timestamp}`) + .join('|'); + } else if (data.functions) { + hashString = + data.functions.map(f => `${f.name}:${f.total_time}:${f.call_count}`) + .join('|'); + } + + // Simple hash + let hash = 0; + for (let i = 0; i < hashString.length; i++) { + const char = hashString.charCodeAt(i); + hash = ((hash << 5) - hash) + char; + hash = hash & hash; // Convert to 32bit integer + } + return hash; + } + + cleanup() { + if (this.animationFrame) { + cancelAnimationFrame(this.animationFrame); + this.animationFrame = null; + } + + // Remove event listeners + document.removeEventListener('keydown', this.handleKeyDown); + document.removeEventListener('keyup', this.handleKeyUp); + } +} + +window.PerformanceManager = PerformanceManager; +window.performanceManager = null; diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/tab-manager.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/tab-manager.js new file mode 100644 index 0000000000000..4eac538f42a57 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/tab-manager.js @@ -0,0 +1,337 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Tab Manager + * Handles tab switching and navigation in the LLVM Advisor dashboard + */ + +export class TabManager { + constructor() { + this.currentTab = 'dashboard'; + this.tabs = new Map(); + this.onTabChangeCallback = null; + } + + /** + * Initialize the tab manager + */ + init(options = {}) { + this.onTabChangeCallback = options.onTabChange; + + // Register all tabs + this.registerTabs(); + + // Setup event listeners + this.setupEventListeners(); + + // Set initial tab state + this.setActiveTab(this.currentTab); + + console.log('Tab manager initialized'); + } + + /** + * Register all available tabs + */ + registerTabs() { + const tabButtons = document.querySelectorAll('.tab-button'); + const tabContents = document.querySelectorAll('.tab-content'); + + tabButtons.forEach(button => { + const tabId = button.dataset.tab; + const content = document.getElementById(`${tabId}-content`); + + if (content) { + this.tabs.set(tabId, { + button, + content, + isLoaded : tabId === 'dashboard', // Dashboard is loaded by default + title : button.textContent.trim() + }); + } + }); + + console.log(`📋 Registered ${this.tabs.size} tabs:`, + Array.from(this.tabs.keys())); + } + + /** + * Setup event listeners for tab interactions + */ + setupEventListeners() { + // Handle tab button clicks + document.addEventListener('click', (event) => { + if (event.target.classList.contains('tab-button')) { + event.preventDefault(); + const tabId = event.target.dataset.tab; + if (tabId && this.tabs.has(tabId)) { + this.switchTab(tabId); + } + } + }); + + // Handle keyboard navigation (Tab key to cycle through tabs) + document.addEventListener('keydown', (event) => { + if (event.key === 'Tab' && event.ctrlKey) { + event.preventDefault(); + this.switchToNextTab(); + } + }); + + // Handle URL hash changes for deep linking + window.addEventListener('hashchange', () => { this.handleHashChange(); }); + + // Set initial hash if none exists + if (!window.location.hash && this.currentTab) { + window.location.hash = `#${this.currentTab}`; + } + } + + /** + * Switch to a specific tab + */ + async switchTab(tabId) { + if (!this.tabs.has(tabId) || tabId === this.currentTab) { + return; + } + + const previousTab = this.currentTab; + + try { + // Update current tab + this.currentTab = tabId; + + // Update UI + this.setActiveTab(tabId); + + // Update URL hash + window.location.hash = `#${tabId}`; + + // Call the tab change callback + if (this.onTabChangeCallback) { + await this.onTabChangeCallback(tabId, previousTab); + } + + // Mark tab as loaded + const tab = this.tabs.get(tabId); + if (tab) { + tab.isLoaded = true; + } + + // Track tab switch for analytics + this.trackTabSwitch(tabId, previousTab); + + console.log(`📱 Switched from ${previousTab} to ${tabId}`); + + } catch (error) { + console.error(`Failed to switch to tab ${tabId}:`, error); + + // Revert to previous tab on error + this.currentTab = previousTab; + this.setActiveTab(previousTab); + + // Show error notification + this.showTabSwitchError(tabId, error.message); + } + } + + /** + * Set the visual active state for a tab + */ + setActiveTab(tabId) { + // Update all tab buttons + this.tabs.forEach((tab, id) => { + if (id === tabId) { + // Activate current tab + tab.button.classList.add('active'); + tab.button.classList.remove('text-gray-500', 'hover:text-gray-700', + 'hover:border-gray-300', + 'border-transparent'); + tab.button.classList.add('border-llvm-blue', 'text-llvm-blue'); + + // Show current tab content + tab.content.classList.remove('hidden'); + tab.content.classList.add('tab-transition'); + + } else { + // Deactivate other tabs + tab.button.classList.remove('active', 'border-llvm-blue', + 'text-llvm-blue'); + tab.button.classList.add('border-transparent', 'text-gray-500', + 'hover:text-gray-700', + 'hover:border-gray-300'); + + // Hide other tab contents + tab.content.classList.add('hidden'); + tab.content.classList.remove('tab-transition'); + } + }); + } + + /** + * Switch to the next tab in sequence + */ + switchToNextTab() { + const tabIds = Array.from(this.tabs.keys()); + const currentIndex = tabIds.indexOf(this.currentTab); + const nextIndex = (currentIndex + 1) % tabIds.length; + const nextTabId = tabIds[nextIndex]; + + this.switchTab(nextTabId); + } + + /** + * Switch to the previous tab in sequence + */ + switchToPreviousTab() { + const tabIds = Array.from(this.tabs.keys()); + const currentIndex = tabIds.indexOf(this.currentTab); + const prevIndex = currentIndex === 0 ? tabIds.length - 1 : currentIndex - 1; + const prevTabId = tabIds[prevIndex]; + + this.switchTab(prevTabId); + } + + /** + * Handle URL hash changes for deep linking + */ + handleHashChange() { + const hash = window.location.hash.slice(1); // Remove the '#' + + if (hash && this.tabs.has(hash) && hash !== this.currentTab) { + this.switchTab(hash); + } + } + + /** + * Get the currently active tab + */ + getCurrentTab() { return this.currentTab; } + + /** + * Get information about a specific tab + */ + getTabInfo(tabId) { return this.tabs.get(tabId); } + + /** + * Get all registered tabs + */ + getAllTabs() { + const result = {}; + this.tabs.forEach((tab, id) => { + result[id] = { + title : tab.title, + isLoaded : tab.isLoaded, + isActive : id === this.currentTab + }; + }); + return result; + } + + /** + * Check if a tab has been loaded + */ + isTabLoaded(tabId) { + const tab = this.tabs.get(tabId); + return tab ? tab.isLoaded : false; + } + + /** + * Mark a tab as loaded + */ + markTabAsLoaded(tabId) { + const tab = this.tabs.get(tabId); + if (tab) { + tab.isLoaded = true; + } + } + + /** + * Show loading state for a specific tab + */ + showTabLoading(tabId) { + const tab = this.tabs.get(tabId); + if (tab && tab.content) { + const loadingHtml = ` +
+
+
+

Loading ${ + tab.title}...

+
+
+ `; + + // Store original content if not already stored + if (!tab.originalContent) { + tab.originalContent = tab.content.innerHTML; + } + + tab.content.innerHTML = loadingHtml; + } + } + + /** + * Hide loading state for a specific tab + */ + hideTabLoading(tabId) { + const tab = this.tabs.get(tabId); + if (tab && tab.originalContent) { + tab.content.innerHTML = tab.originalContent; + delete tab.originalContent; + } + } + + /** + * Show error state for tab switching + */ + showTabSwitchError(tabId, errorMessage) { + console.error(`Tab switch error for ${tabId}:`, errorMessage); + + const tab = this.tabs.get(tabId); + if (tab) { + alert(`Failed to switch to ${tab.title}: ${errorMessage}`); + } + } + + /** + * Track tab switches for analytics/debugging + */ + trackTabSwitch(newTab, previousTab) { + const timestamp = new Date().toISOString(); + + console.log( + `Tab Analytics: ${previousTab} -> ${newTab} at ${timestamp}`); + } + + /** + * Enable/disable a specific tab + */ + setTabEnabled(tabId, enabled) { + const tab = this.tabs.get(tabId); + if (tab) { + if (enabled) { + tab.button.removeAttribute('disabled'); + tab.button.classList.remove('opacity-50', 'cursor-not-allowed'); + } else { + tab.button.setAttribute('disabled', 'true'); + tab.button.classList.add('opacity-50', 'cursor-not-allowed'); + + // If this was the current tab, switch to another one + if (tabId === this.currentTab) { + const enabledTabs = + Array.from(this.tabs.keys()) + .filter( + id => id !== tabId && + !this.tabs.get(id).button.hasAttribute('disabled')); + + if (enabledTabs.length > 0) { + this.switchTab(enabledTabs[0]); + } + } + } + } + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/utils.js b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/utils.js new file mode 100644 index 0000000000000..8f1ac1c98ac38 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/static/js/utils.js @@ -0,0 +1,478 @@ +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +/** + * Utility Functions + * Common helper functions used throughout the LLVM Advisor dashboard + */ + +export class Utils { + /** + * Format numbers with proper separators and abbreviations + */ + static formatNumber(num) { + if (num === null || num === undefined) + return '0'; + + const number = parseInt(num); + if (isNaN(number)) + return '0'; + + // Use compact notation for large numbers + if (number >= 1000000) { + return `${(number / 1000000).toFixed(1)}M`; + } else if (number >= 1000) { + return `${(number / 1000).toFixed(1)}K`; + } + + return number.toLocaleString(); + } + + /** + * Format file type names for display + */ + static formatFileType(type) { + if (!type) + return 'Unknown'; + + const typeMap = { + 'opt_record' : 'Optimization Records', + 'opt_remarks' : 'Optimization Remarks', + 'time_trace' : 'Time Trace', + 'runtime_trace' : 'Runtime Trace', + 'binary_size' : 'Binary Size', + 'compilation_units' : 'Compilation Units', + 'diagnostics' : 'Diagnostics', + 'clang_diagnostics' : 'Clang Diagnostics', + 'coverage_report' : 'Coverage Report', + 'profile_data' : 'Profile Data', + 'ast_dump' : 'AST Dump', + 'ir_code' : 'IR Code', + 'assembly_code' : 'Assembly Code', + 'debug_info' : 'Debug Info', + 'static_analysis' : 'Static Analysis', + 'memory_usage' : 'Memory Usage', + 'compilation_commands' : 'Compilation Commands', + 'build_log' : 'Build Log', + 'link_map' : 'Link Map', + 'symbol_table' : 'Symbol Table' + }; + + return typeMap[type] || this.capitalize(type.replace(/_/g, ' ')); + } + + /** + * Capitalize the first letter of a string + */ + static capitalize(str) { + if (!str) + return ''; + return str.charAt(0).toUpperCase() + str.slice(1).toLowerCase(); + } + + /** + * Format compilation phase names for display + */ + static formatPhaseName(name) { + if (!name) + return 'Unknown Phase'; + + // Common LLVM phase name mappings + const phaseMap = { + 'frontend' : 'Frontend', + 'backend' : 'Backend', + 'codegen' : 'Code Generation', + 'optimization' : 'Optimization', + 'linking' : 'Linking', + 'parsing' : 'Parsing', + 'semantic' : 'Semantic Analysis', + 'irgen' : 'IR Generation', + 'opt' : 'Optimization', + 'asm' : 'Assembly Generation', + 'obj' : 'Object Generation' + }; + + // Try direct mapping first + if (phaseMap[name.toLowerCase()]) { + return phaseMap[name.toLowerCase()]; + } + + // Format by replacing underscores and capitalizing + return name.replace(/_/g, ' ') + .replace(/\b\w/g, l => l.toUpperCase()) + .replace(/\bIr\b/g, 'IR') + .replace(/\bLlvm\b/g, 'LLVM') + .replace(/\bCpp\b/g, 'C++') + .replace(/\bAst\b/g, 'AST'); + } + + /** + * Format time values (milliseconds) for display + */ + static formatTime(timeMs) { + if (timeMs === null || timeMs === undefined) + return '0ms'; + + const time = parseFloat(timeMs); + if (isNaN(time)) + return '0ms'; + + if (time < 1000) { + return `${time.toFixed(0)}ms`; + } else if (time < 60000) { + return `${(time / 1000).toFixed(2)}s`; + } else if (time < 3600000) { + const minutes = Math.floor(time / 60000); + const seconds = ((time % 60000) / 1000).toFixed(0); + return `${minutes}m ${seconds}s`; + } else { + const hours = Math.floor(time / 3600000); + const minutes = Math.floor((time % 3600000) / 60000); + return `${hours}h ${minutes}m`; + } + } + + /** + * Format byte sizes for display + */ + static formatBytes(bytes) { + if (bytes === null || bytes === undefined) + return '0 B'; + + const size = parseInt(bytes); + if (isNaN(size) || size === 0) + return '0 B'; + + const units = [ 'B', 'KB', 'MB', 'GB', 'TB' ]; + const threshold = 1024; + + if (size < threshold) + return `${size} B`; + + let unitIndex = 0; + let value = size; + + while (value >= threshold && unitIndex < units.length - 1) { + value /= threshold; + unitIndex++; + } + + return `${value.toFixed(1)} ${units[unitIndex]}`; + } + + /** + * Format binary section names for display + */ + static formatSectionName(name) { + if (!name) + return 'Unknown Section'; + + // Common binary section mappings + const sectionMap = { + '.text' : 'Code (.text)', + '.data' : 'Data (.data)', + '.bss' : 'BSS (.bss)', + '.rodata' : 'Read-Only Data (.rodata)', + '.debug' : 'Debug Info (.debug)', + '.symtab' : 'Symbol Table (.symtab)', + '.strtab' : 'String Table (.strtab)', + '.rela' : 'Relocations (.rela)', + '.dynamic' : 'Dynamic (.dynamic)', + '.interp' : 'Interpreter (.interp)', + '.note' : 'Notes (.note)', + '.comment' : 'Comments (.comment)', + '.plt' : 'PLT (.plt)', + '.got' : 'GOT (.got)' + }; + + // Try direct mapping first + if (sectionMap[name]) { + return sectionMap[name]; + } + + // If it starts with a dot, assume it's a section name + if (name.startsWith('.')) { + return `${this.capitalize(name.slice(1))} (${name})`; + } + + return this.capitalize(name); + } + + /** + * Format percentage values + */ + static formatPercentage(value, decimals = 1) { + if (value === null || value === undefined) + return '0%'; + + const num = parseFloat(value); + if (isNaN(num)) + return '0%'; + + return `${num.toFixed(decimals)}%`; + } + + /** + * Format diagnostic level names + */ + static formatDiagnosticLevel(level) { + const levelMap = { + 'error' : 'Error', + 'warning' : 'Warning', + 'note' : 'Note', + 'info' : 'Info', + 'fatal' : 'Fatal Error', + 'remark' : 'Remark' + }; + + return levelMap[level?.toLowerCase()] || + this.capitalize(level || 'unknown'); + } + + /** + * Truncate text to specified length with ellipsis + */ + static truncateText(text, maxLength = 50) { + if (!text) + return ''; + if (text.length <= maxLength) + return text; + + return text.substring(0, maxLength - 3) + '...'; + } + + /** + * Debounce function calls + */ + static debounce(func, wait) { + let timeout; + return function executedFunction(...args) { + const later = () => { + clearTimeout(timeout); + func(...args); + }; + clearTimeout(timeout); + timeout = setTimeout(later, wait); + }; + } + + /** + * Throttle function calls + */ + static throttle(func, limit) { + let inThrottle; + return function executedFunction(...args) { + if (!inThrottle) { + func.apply(this, args); + inThrottle = true; + setTimeout(() => inThrottle = false, limit); + } + }; + } + + /** + * Deep clone an object + */ + static deepClone(obj) { + if (obj === null || typeof obj !== 'object') + return obj; + if (obj instanceof Date) + return new Date(obj.getTime()); + if (obj instanceof Array) + return obj.map(item => this.deepClone(item)); + if (typeof obj === 'object') { + const clonedObj = {}; + Object.keys(obj).forEach( + key => { clonedObj[key] = this.deepClone(obj[key]); }); + return clonedObj; + } + } + + /** + * Check if two objects are equal (deep comparison) + */ + static isEqual(obj1, obj2) { + if (obj1 === obj2) + return true; + if (obj1 == null || obj2 == null) + return false; + if (typeof obj1 !== typeof obj2) + return false; + + if (typeof obj1 === 'object') { + const keys1 = Object.keys(obj1); + const keys2 = Object.keys(obj2); + + if (keys1.length !== keys2.length) + return false; + + for (let key of keys1) { + if (!keys2.includes(key)) + return false; + if (!this.isEqual(obj1[key], obj2[key])) + return false; + } + + return true; + } + + return obj1 === obj2; + } + + /** + * Generate a random color + */ + static getRandomColor() { + const colors = [ + '#3b82f6', '#10b981', '#f59e0b', '#ef4444', '#8b5cf6', '#06b6d4', + '#84cc16', '#f97316', '#ec4899', '#6366f1', '#14b8a6', '#f59e0b' + ]; + return colors[Math.floor(Math.random() * colors.length)]; + } + + /** + * Get contrast color (black or white) for a given background color + */ + static getContrastColor(hexColor) { + // Remove # if present + hexColor = hexColor.replace('#', ''); + + // Convert to RGB + const r = parseInt(hexColor.substr(0, 2), 16); + const g = parseInt(hexColor.substr(2, 2), 16); + const b = parseInt(hexColor.substr(4, 2), 16); + + // Calculate luminance + const luminance = ((0.299 * r) + (0.587 * g) + (0.114 * b)) / 255; + + return luminance > 0.5 ? '#000000' : '#ffffff'; + } + + /** + * Format date/time for display + */ + static formatDateTime(date, options = {}) { + if (!date) + return 'Unknown'; + + const dateObj = date instanceof Date ? date : new Date(date); + + const defaultOptions = { + year : 'numeric', + month : 'short', + day : 'numeric', + hour : '2-digit', + minute : '2-digit' + }; + + return dateObj.toLocaleDateString('en-US', {...defaultOptions, ...options}); + } + + /** + * Format relative time (e.g., "2 minutes ago") + */ + static formatRelativeTime(date) { + if (!date) + return 'Unknown'; + + const now = new Date(); + const dateObj = date instanceof Date ? date : new Date(date); + const diffMs = now - dateObj; + + const diffSecs = Math.floor(diffMs / 1000); + const diffMins = Math.floor(diffSecs / 60); + const diffHours = Math.floor(diffMins / 60); + const diffDays = Math.floor(diffHours / 24); + + if (diffSecs < 60) + return 'Just now'; + if (diffMins < 60) + return `${diffMins} minute${diffMins > 1 ? 's' : ''} ago`; + if (diffHours < 24) + return `${diffHours} hour${diffHours > 1 ? 's' : ''} ago`; + if (diffDays < 7) + return `${diffDays} day${diffDays > 1 ? 's' : ''} ago`; + + return this.formatDateTime(dateObj); + } + + /** + * Validate email address + */ + static isValidEmail(email) { + const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/; + return emailRegex.test(email); + } + + /** + * Escape HTML to prevent XSS + */ + static escapeHtml(text) { + const div = document.createElement('div'); + div.textContent = text; + return div.innerHTML; + } + + /** + * Generate a simple hash from a string + */ + static hashString(str) { + let hash = 0; + if (str.length === 0) + return hash; + + for (let i = 0; i < str.length; i++) { + const char = str.charCodeAt(i); + hash = ((hash << 5) - hash) + char; + hash = hash & hash; // Convert to 32bit integer + } + + return Math.abs(hash); + } + + /** + * Check if value is empty (null, undefined, empty string, empty array, etc.) + */ + static isEmpty(value) { + if (value == null) + return true; + if (typeof value === 'string') + return value.trim().length === 0; + if (Array.isArray(value)) + return value.length === 0; + if (typeof value === 'object') + return Object.keys(value).length === 0; + return false; + } + + /** + * Sort array of objects by a property + */ + static sortBy(array, property, ascending = true) { + return array.sort((a, b) => { + const aVal = a[property]; + const bVal = b[property]; + + if (aVal < bVal) + return ascending ? -1 : 1; + if (aVal > bVal) + return ascending ? 1 : -1; + return 0; + }); + } + + /** + * Group array of objects by a property + */ + static groupBy(array, property) { + return array.reduce((groups, item) => { + const key = item[property]; + if (!groups[key]) { + groups[key] = []; + } + groups[key].push(item); + return groups; + }, {}); + } +} diff --git a/llvm/tools/llvm-advisor/tools/webserver/frontend/templates/index.html b/llvm/tools/llvm-advisor/tools/webserver/frontend/templates/index.html new file mode 100644 index 0000000000000..fb06d7165e5f4 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/frontend/templates/index.html @@ -0,0 +1,890 @@ + + + + + + + + LLVM Advisor - Compilation Analysis Dashboard + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+

+ Loading LLVM Advisor Dashboard... +

+

Analyzing compilation data

+
+
+ + + + + + + + + diff --git a/llvm/tools/llvm-advisor/tools/webserver/server.py b/llvm/tools/llvm-advisor/tools/webserver/server.py new file mode 100644 index 0000000000000..2fd01dff44f13 --- /dev/null +++ b/llvm/tools/llvm-advisor/tools/webserver/server.py @@ -0,0 +1,342 @@ +# ===----------------------------------------------------------------------===// +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===----------------------------------------------------------------------===// + +#!/usr/bin/env python3 + +import os +import sys +import json +import argparse +import mimetypes +from http.server import HTTPServer, BaseHTTPRequestHandler +from urllib.parse import urlparse, parse_qs +from pathlib import Path + +# Add parent directories to path for imports +current_dir = Path(__file__).parent +tools_dir = current_dir.parent +sys.path.insert(0, str(tools_dir)) + +from common.collector import ArtifactCollector +from common.models import FileType + +# Import API endpoints +from api.health import HealthEndpoint +from api.units import UnitsEndpoint, UnitDetailEndpoint +from api.summary import SummaryEndpoint +from api.files import FileContentEndpoint +from api.artifacts import ArtifactsEndpoint, ArtifactTypesEndpoint +from api.explorer import ExplorerEndpoint +from api.specialized_router import SpecializedRouter, SPECIALIZED_ENDPOINTS_DOCS + + +class APIHandler(BaseHTTPRequestHandler): + """API handler""" + + def __init__(self, data_dir, *args, **kwargs): + self.data_dir = data_dir + self.collector = ArtifactCollector() + + # Set up frontend paths + current_dir = Path(__file__).parent + self.frontend_dir = current_dir / "frontend" + self.static_dir = self.frontend_dir / "static" + self.templates_dir = self.frontend_dir / "templates" + + # Initialize endpoints + self.endpoints = { + "health": HealthEndpoint(data_dir, self.collector), + "units": UnitsEndpoint(data_dir, self.collector), + "unit_detail": UnitDetailEndpoint(data_dir, self.collector), + "summary": SummaryEndpoint(data_dir, self.collector), + "files": FileContentEndpoint(data_dir, self.collector), + "artifacts": ArtifactsEndpoint(data_dir, self.collector), + "artifact_types": ArtifactTypesEndpoint(data_dir, self.collector), + "explorer": ExplorerEndpoint(data_dir, self.collector), + } + + # Initialize specialized router + self.specialized_router = SpecializedRouter(data_dir, self.collector) + + super().__init__(*args, **kwargs) + + def _send_json_response(self, response_data): + """Send JSON response with proper headers""" + if "status" in response_data: + status = response_data["status"] + else: + status = 200 + + response_json = json.dumps(response_data, indent=2, default=str) + + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Access-Control-Allow-Origin", "*") + self.send_header("Access-Control-Allow-Methods", "GET, OPTIONS") + self.send_header("Access-Control-Allow-Headers", "Content-Type") + self.end_headers() + self.wfile.write(response_json.encode("utf-8")) + + def _send_error(self, message, status=500): + """Send error response""" + self._send_json_response({"success": False, "error": message, "status": status}) + + def _send_static_file(self, file_path): + """Send a static file with appropriate headers""" + try: + if not file_path.exists(): + self.send_error(404, "File not found") + return + + # Determine content type + content_type, _ = mimetypes.guess_type(str(file_path)) + if content_type is None: + content_type = "application/octet-stream" + + # Read file content + with open(file_path, "rb") as f: + content = f.read() + + # Send response + self.send_response(200) + self.send_header("Content-Type", content_type) + self.send_header("Content-Length", str(len(content))) + self.send_header("Access-Control-Allow-Origin", "*") + self.end_headers() + self.wfile.write(content) + + except Exception as e: + self.send_error(500, f"Error serving file: {str(e)}") + + def _send_html_file(self, file_path): + """Send an HTML file""" + try: + if not file_path.exists(): + self.send_error(404, "File not found") + return + + with open(file_path, "r", encoding="utf-8") as f: + content = f.read() + + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(content.encode("utf-8")))) + self.send_header("Access-Control-Allow-Origin", "*") + self.end_headers() + self.wfile.write(content.encode("utf-8")) + + except Exception as e: + self.send_error(500, f"Error serving HTML file: {str(e)}") + + def do_OPTIONS(self): + """Handle CORS preflight requests""" + self.send_response(200) + self.send_header("Access-Control-Allow-Origin", "*") + self.send_header("Access-Control-Allow-Methods", "GET, OPTIONS") + self.send_header("Access-Control-Allow-Headers", "Content-Type") + self.end_headers() + + def do_GET(self): + """Route GET requests to endpoints or serve static files""" + try: + parsed_url = urlparse(self.path) + path = parsed_url.path.rstrip("/") + path_parts = path.strip("/").split("/") + query_params = parse_qs(parsed_url.query) + + # Serve main UI at root + if path == "" or path == "/": + index_file = self.templates_dir / "index.html" + self._send_html_file(index_file) + return + + # Serve static files + if path.startswith("/static/"): + # Remove '/static' from path and get relative path + static_path = path[8:] # Remove '/static/' + file_path = self.static_dir / static_path + self._send_static_file(file_path) + return + + # API Documentation endpoint + if path == "/api" or path == "/api/": + response = self._get_api_documentation() + self._send_json_response(response) + return + + # Route to appropriate API endpoints + if path == "/api/health": + response = self.endpoints["health"].handle(path_parts, query_params) + elif path == "/api/units": + response = self.endpoints["units"].handle(path_parts, query_params) + elif path == "/api/summary": + response = self.endpoints["summary"].handle(path_parts, query_params) + elif path == "/api/artifacts": + response = self.endpoints["artifact_types"].handle( + path_parts, query_params + ) + elif path.startswith("/api/units/") and len(path_parts) >= 3: + response = self.endpoints["unit_detail"].handle( + path_parts, query_params + ) + elif path.startswith("/api/file/") and len(path_parts) >= 5: + response = self.endpoints["files"].handle(path_parts, query_params) + elif path.startswith("/api/artifacts/") and len(path_parts) >= 3: + response = self.endpoints["artifacts"].handle(path_parts, query_params) + elif path.startswith("/api/explorer/") and len(path_parts) >= 3: + response = self.endpoints["explorer"].handle(path_parts, query_params) + elif self._is_specialized_endpoint(path_parts): + # Route to specialized file-type endpoints + response = self.specialized_router.route_request( + path_parts, query_params + ) + else: + response = { + "success": False, + "error": "Endpoint not found", + "status": 404, + "available_endpoints": self._get_available_endpoints(), + } + + self._send_json_response(response) + + except Exception as e: + self._send_error(f"Internal server error: {str(e)}") + + def _is_specialized_endpoint(self, path_parts: list) -> bool: + """Check if this is a specialized file-type endpoint""" + if len(path_parts) >= 2 and path_parts[0] == "api": + file_type = path_parts[1] + specialized_types = [ + "remarks", + "diagnostics", + "compilation-phases", + "time-trace", + "runtime-trace", + "binary-size", + "ast-json", + "sarif", + "symbols", + "ir", + "assembly", + "preprocessed", + "macro-expansion", + ] + return file_type in specialized_types + return False + + def _get_api_documentation(self): + """Generate API documentation""" + return { + "success": True, + "data": { + "llvm_advisor_api": "1.0", + "description": "API for LLVM Advisor compilation data", + "data_directory": self.data_dir, + "endpoints": self._get_available_endpoints(), + "specialized_endpoints": self.specialized_router.get_available_endpoints(), + "supported_file_types": [ft.value for ft in FileType], + }, + "status": 200, + } + + def _get_available_endpoints(self): + """Get list of available endpoints with descriptions""" + return { + "GET /api/health": { + "description": "System health check and data directory status", + "returns": "Health status, data directory info, basic statistics", + }, + "GET /api/units": { + "description": "List all compilation units with basic metadata", + "returns": "Array of compilation units with file counts by type", + }, + "GET /api/units/{unit_name}": { + "description": "Detailed information for a specific compilation unit", + "returns": "Complete metadata for all files in the unit", + }, + "GET /api/summary": { + "description": "Overall statistics summary across all units", + "returns": "Aggregated statistics, error counts, success rates", + }, + "GET /api/artifacts": { + "description": "List all available artifact types with global counts", + "returns": "Summary of all file types found across units", + }, + "GET /api/artifacts/{file_type}": { + "description": "Aggregated data for a specific file type across all units", + "returns": "All files of the specified type with summary statistics", + }, + "GET /api/file/{unit_name}/{file_type}/{file_name}": { + "description": "Get parsed content of a specific file", + "returns": "Complete parsed data for the requested file", + "query_params": { + "full": "Set to 'true' to get complete data for large files (default: summary only)" + }, + }, + "Specialized Endpoints": { + "description": "File-type specific analysis endpoints", + "pattern": "GET /api/{file_type}/{analysis_type}", + "examples": { + "GET /api/remarks/overview": "Optimization remarks statistics", + "GET /api/diagnostics/patterns": "Common diagnostic patterns", + "GET /api/compilation-phases/bottlenecks": "Compilation bottlenecks", + "GET /api/time-trace/hotspots": "Performance hotspots", + "GET /api/binary-size/optimization": "Size optimization opportunities", + }, + }, + } + + +def create_handler(data_dir): + """Factory function to create handler with data directory""" + + def handler(*args, **kwargs): + return APIHandler(data_dir, *args, **kwargs) + + return handler + + +def main(): + parser = argparse.ArgumentParser(description="LLVM Advisor API Server") + parser.add_argument( + "--data-dir", required=True, help="Directory containing .llvm-advisor data" + ) + parser.add_argument("--port", type=int, default=8000, help="Port to listen on") + parser.add_argument("--host", default="localhost", help="Host to bind to") + + args = parser.parse_args() + + # Verify data directory exists + if not os.path.exists(args.data_dir): + print(f"Error: Data directory does not exist: {args.data_dir}") + sys.exit(1) + + # Create handler with data directory + handler_class = create_handler(args.data_dir) + + # Start server + server = HTTPServer((args.host, args.port), handler_class) + + print(f"LLVM Advisor Web Interface") + print(f"==========================") + print(f"Starting web server on http://{args.host}:{args.port}") + print(f"Loading data from: {args.data_dir}") + print(f"Press Ctrl+C to stop the server") + print() + + try: + server.serve_forever() + except KeyboardInterrupt: + print("\nShutting down server...") + server.shutdown() + server.server_close() + + +if __name__ == "__main__": + main()