[mlir][tblgen] Adds support for embedded LIT tests in TableGen records #158017
+529
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces a new Testable base class that allows TableGen records (starting
with Pass records) to embed LIT test definitions directly within their
definitions. This enables co-locating tests with pass definitions for better
maintainability.
Key components:
- Testable.td: Base class for records that can have embedded tests
- LitTestGen.cpp: TableGen backend to extract and generate LIT test files
- AddMLIR.cmake: CMake function to process embedded tests with usage examples
- PassBase.td: Updated Pass class to extend Testable
Usage example in CMake:
add_embedded_lit_tests(MyPassesEmbeddedTests
${CMAKE_CURRENT_SOURCE_DIR}/include/MyPasses.td
${CMAKE_CURRENT_SOURCE_DIR}/test/Passes/)
# Add LIT test generation target as a dependency to some
# other target
add_library(someLib DEPENDS MyPassesEmbeddedTests)
Member
|
@llvm/pr-subscribers-mlir @llvm/pr-subscribers-mlir-core Author: Kshitij Jain (jkshtj) ChangesIntroduces a new Testable base class that allows TableGen records (starting with Pass records) to embed LIT test definitions directly within their definitions. This enables co-locating tests with pass definitions for better maintainability. Key components:
Usage example in CMake: Full diff: https://github.com/llvm/llvm-project/pull/158017.diff 6 Files Affected:
diff --git a/mlir/cmake/modules/AddMLIR.cmake b/mlir/cmake/modules/AddMLIR.cmake
index 6589458ab7894..9b05b70231dba 100644
--- a/mlir/cmake/modules/AddMLIR.cmake
+++ b/mlir/cmake/modules/AddMLIR.cmake
@@ -762,3 +762,103 @@ function(mlir_target_link_libraries target type)
target_link_libraries(${target} ${type} ${ARGN})
endif()
endfunction()
+
+# Extracts LIT tests embedded in `Testable` records in `tblgen_file`
+# and generates a file per test in `output_dir`
+#
+# Example usage:
+# # Extract tests from MyPasses.td and generate them in test/Passes/
+# add_embedded_lit_tests(MyPassesEmbeddedTests
+# ${CMAKE_CURRENT_SOURCE_DIR}/include/MyPasses.td
+# ${CMAKE_CURRENT_SOURCE_DIR}/test/Passes/)
+#
+# # This will:
+# # 1. Process MyPasses.td with mlir-tblgen --gen-lit-tests
+# # 2. Extract individual test files to test/Passes/
+# # 3. Generate files like: test/Passes/generated_MyPass_test1.mlir
+#
+function(add_embedded_lit_tests target tblgen_file output_dir)
+ set(LLVM_TARGET_DEFINITIONS ${tblgen_file})
+
+ # Extraction script content
+ set(EXTRACT_SCRIPT_CONTENT [[
+ # Generated extraction script
+ if(NOT CONSOLIDATED_FILE)
+ message(FATAL_ERROR "CONSOLIDATED_FILE variable is required")
+ endif()
+
+ if(NOT OUTPUT_DIR)
+ message(FATAL_ERROR "OUTPUT_DIR variable is required")
+ endif()
+
+ if(NOT EXISTS ${CONSOLIDATED_FILE})
+ message(FATAL_ERROR "Consolidated file does not exist: ${CONSOLIDATED_FILE}")
+ endif()
+
+ # Read the consolidated file
+ file(READ ${CONSOLIDATED_FILE} file_content)
+
+ # Split into lines for processing
+ string(REPLACE "\n" ";" lines "${file_content}")
+
+ set(current_filename "")
+ set(current_content "")
+ set(in_test_block FALSE)
+ set(extracted_test_files)
+
+ foreach(line IN LISTS lines)
+ # Check for filename line
+ if(line MATCHES "^// File: (.+)$")
+ set(current_filename "${CMAKE_MATCH_1}")
+ endif()
+
+ # Check for BEGIN marker
+ if(line MATCHES "^// --- BEGIN .+ ---$")
+ set(in_test_block TRUE)
+ set(current_content "")
+ # Check for END marker
+ elseif(line MATCHES "^// --- END .+ ---$")
+ set(in_test_block FALSE)
+
+ # Write the extracted content to file
+ if(current_filename AND current_content)
+ file(MAKE_DIRECTORY ${OUTPUT_DIR})
+ file(WRITE ${OUTPUT_DIR}/${current_filename} "${current_content}")
+ message(STATUS "Extracted test file: ${current_filename}")
+ list(APPEND extracted_test_files ${current_filename})
+ endif()
+
+ set(current_filename "")
+ set(current_content "")
+ # Collect content within BEGIN/END block
+ elseif(in_test_block)
+ string(APPEND current_content "${line}\n")
+ endif()
+ endforeach()
+
+ list(LENGTH extracted_test_files num_extracted_files)
+ message(STATUS "Extracted ${num_extracted_files} test files to ${OUTPUT_DIR}")
+ ]])
+
+ # Write extraction script to a file in the build directory
+ file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/extract_lit_tests.cmake "${EXTRACT_SCRIPT_CONTENT}")
+
+ # Process tblgen_file and generate a file with all embedded LIT
+ # tests in tblgen_file
+ get_filename_component(tblgen_name ${tblgen_file} NAME_WE)
+ set(consolidated_output_file ${tblgen_name}_extracted_lit_tests.txt)
+ mlir_tablegen(${consolidated_output_file} --gen-lit-tests)
+
+ # Add public tablegen target to trigger builds on changes in tblgen_file
+ add_public_tablegen_target(${target})
+
+ # Call the extraction script to extract all LIT tests into individual
+ # `.mlir` test files
+ add_custom_command(TARGET ${target} POST_BUILD
+ COMMAND ${CMAKE_COMMAND}
+ -DCONSOLIDATED_FILE=${CMAKE_CURRENT_BINARY_DIR}/${consolidated_output_file}
+ -DOUTPUT_DIR=${output_dir}
+ -P ${CMAKE_CURRENT_BINARY_DIR}/extract_lit_tests.cmake
+ COMMENT "Extracting LIT tests to individual files"
+ )
+endfunction()
\ No newline at end of file
diff --git a/mlir/include/mlir/IR/Testable.td b/mlir/include/mlir/IR/Testable.td
new file mode 100644
index 0000000000000..15814ed1bd939
--- /dev/null
+++ b/mlir/include/mlir/IR/Testable.td
@@ -0,0 +1,40 @@
+//===-- Testable.td - Testable type definition file --------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains the definition of the `Testable` type.
+//
+// Any type whose records can have corresponding LIT tests (eg - Pass) can extend
+// `Testable` in order to be able to embed LIT tests within record definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef TESTABLE
+#define TESTABLE
+
+// Represents a LIT test record in TableGen
+class LitTest<string name, code snippet, list<string> run = [], list<string> check = []> {
+ // The name of the generated test file
+ string testFileName = name;
+
+ // The IR snippet/code to be tested
+ code irSnippet = snippet;
+
+ // The RUN commands for the test (e.g., "mlir-opt %s")
+ list<string> runLines = run;
+
+ // Expected output patterns (CHECK lines)
+ list<string> checkLines = check;
+}
+
+// Base class for elements that can have auto-generated LIT tests
+class Testable {
+ // List of LIT tests associated with this element
+ list<LitTest> tests = [];
+}
+
+#endif // TESTABLE
\ No newline at end of file
diff --git a/mlir/include/mlir/Pass/PassBase.td b/mlir/include/mlir/Pass/PassBase.td
index e37f9735e2241..50ea44419ca24 100644
--- a/mlir/include/mlir/Pass/PassBase.td
+++ b/mlir/include/mlir/Pass/PassBase.td
@@ -14,6 +14,8 @@
#ifndef MLIR_PASS_PASSBASE
#define MLIR_PASS_PASSBASE
+include "mlir/IR/Testable.td"
+
//===----------------------------------------------------------------------===//
// Options
//===----------------------------------------------------------------------===//
@@ -62,7 +64,7 @@ class Statistic<string varName, string statName, string desc> {
// Pass
//===----------------------------------------------------------------------===//
-class PassBase<string passArg, string base> {
+class PassBase<string passArg, string base> : Testable {
// The command line argument of the pass.
string argument = passArg;
diff --git a/mlir/test/mlir-tblgen/gen-lit-tests.td b/mlir/test/mlir-tblgen/gen-lit-tests.td
new file mode 100644
index 0000000000000..40a03fb2b2d60
--- /dev/null
+++ b/mlir/test/mlir-tblgen/gen-lit-tests.td
@@ -0,0 +1,65 @@
+// RUN: mlir-tblgen -gen-lit-tests -I %S/../../include -dialect=test %s | FileCheck %s
+
+include "mlir/Pass/PassBase.td"
+include "mlir/IR/Testable.td"
+
+def TestPassWithEmbeddedLitTests : Pass<"test-pass-with-embedded-lit-tests"> {
+ let summary = "pass summary";
+ let description = [{
+ Pass description
+ }];
+
+ let tests = [
+ LitTest<
+ "lit_test_file_1.mlir",
+ [{
+ func.func @test1() {
+ return 42;
+ }
+ }],
+ [
+ "// RUN: mlir-opt %s --verify-roundtrip | FileCheck %s",
+ ],
+ [
+ "// RANDOM-CHECK-LABEL: func.func @test1",
+ ]
+ >,
+ LitTest<
+ "lit_test_file_2.mlir",
+ [{
+ func.func @test2() {
+ return 42;
+ }
+ }],
+ [
+ "// RUN: mlir-opt %s --verify-roundtrip | FileCheck %s",
+ ],
+ [
+ "// RANDOM-CHECK-LABEL: func.func @test2",
+ ]
+ >,
+ ];
+}
+
+// CHECK-LABEL: // Generated 2 LIT test files
+// CHECK: // Use the following files for LIT testing:
+
+// CHECK: // File: generated_TestPassWithEmbeddedLitTests_lit_test_file_1.mlir
+// CHECK: // --- BEGIN generated_TestPassWithEmbeddedLitTests_lit_test_file_1.mlir ---
+// CHECK: // RUN: mlir-opt %s --verify-roundtrip | FileCheck %s
+// CHECK: // Generated from TableGen definition: TestPassWithEmbeddedLitTests
+// CHECK: func.func @test1() {
+// CHECK: return 42;
+// CHECK: }
+// CHECK: // RANDOM-CHECK-LABEL: func.func @test1
+// CHECK: --- END generated_TestPassWithEmbeddedLitTests_lit_test_file_1.mlir ---
+
+// CHECK: // File: generated_TestPassWithEmbeddedLitTests_lit_test_file_2.mlir
+// CHECK: // --- BEGIN generated_TestPassWithEmbeddedLitTests_lit_test_file_2.mlir ---
+// CHECK: // RUN: mlir-opt %s --verify-roundtrip | FileCheck %s
+// CHECK: // Generated from TableGen definition: TestPassWithEmbeddedLitTests
+// CHECK: func.func @test2() {
+// CHECK: return 42;
+// CHECK: }
+// CHECK: // RANDOM-CHECK-LABEL: func.func @test2
+// CHECK: // --- END generated_TestPassWithEmbeddedLitTests_lit_test_file_2.mlir ---
\ No newline at end of file
diff --git a/mlir/tools/mlir-tblgen/CMakeLists.txt b/mlir/tools/mlir-tblgen/CMakeLists.txt
index 2a7ef7e0576c8..e721f1e26a2bd 100644
--- a/mlir/tools/mlir-tblgen/CMakeLists.txt
+++ b/mlir/tools/mlir-tblgen/CMakeLists.txt
@@ -16,6 +16,7 @@ add_tablegen(mlir-tblgen MLIR
EnumsGen.cpp
EnumPythonBindingGen.cpp
FormatGen.cpp
+ LitTestGen.cpp
LLVMIRConversionGen.cpp
LLVMIRIntrinsicGen.cpp
mlir-tblgen.cpp
diff --git a/mlir/tools/mlir-tblgen/LitTestGen.cpp b/mlir/tools/mlir-tblgen/LitTestGen.cpp
new file mode 100644
index 0000000000000..49a092fa9879f
--- /dev/null
+++ b/mlir/tools/mlir-tblgen/LitTestGen.cpp
@@ -0,0 +1,170 @@
+//===- LitTestGen.cpp - LIT test generator ----------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// LitTestGen extracts `LitTest` records from `Testable` TableGen records and
+// generates corresponding LIT test files.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/TableGen/GenInfo.h"
+#include "mlir/TableGen/Operator.h"
+#include "mlir/TableGen/Pass.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/Path.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/Record.h"
+
+#include <set>
+
+using namespace mlir;
+using namespace mlir::tblgen;
+using llvm::formatv;
+using llvm::RecordKeeper;
+
+static llvm::cl::OptionCategory litTestGenCategory("Options for -gen-lit-tests");
+static llvm::cl::opt<std::string>
+ outputDir("output-dir",
+ llvm::cl::desc("Output directory for generated test files"),
+ llvm::cl::cat(litTestGenCategory),
+ llvm::cl::value_desc("directory"));
+
+
+/// Cpp type corresponding to the `LitTest` record type in TableGen
+struct LitTest {
+ std::string sourceDefName;
+ std::string testFileName;
+ std::string irSnippet;
+ llvm::SmallVector<std::string> runLines;
+ llvm::SmallVector<std::string> checkLines;
+};
+
+static llvm::SmallVector<LitTest> extractTestsFromRecord(const llvm::Record *record,
+ llvm::StringRef dialectName = "") {
+ llvm::SmallVector<LitTest> tests;
+
+ // Check if the record has a tests field
+ const llvm::RecordVal *testsVal = record->getValue("tests");
+ if (!testsVal)
+ return tests;
+
+ const llvm::ListInit *testsList =
+ llvm::dyn_cast_or_null<llvm::ListInit>(testsVal->getValue());
+ if (!testsList)
+ return tests;
+
+ for (const llvm::Init *init : testsList->getElements()) {
+ const llvm::DefInit *defInit = llvm::dyn_cast<llvm::DefInit>(init);
+ if (!defInit)
+ continue;
+
+ const llvm::Record *testRec = defInit->getDef();
+
+ // Extract fields from LitTest record
+ std::string name = testRec->getValueAsString("testFileName").str();
+ std::string irSnippet = testRec->getValueAsString("irSnippet").str();
+
+ llvm::SmallVector<std::string> runLines;
+ llvm::for_each(*testRec->getValueAsListInit("runLines"), [&](const llvm::Init *init) {
+ runLines.emplace_back(llvm::cast<llvm::StringInit>(init)->getValue());
+ });
+
+ llvm::SmallVector<std::string> checkLines;
+ llvm::for_each(*testRec->getValueAsListInit("checkLines"), [&](const llvm::Init *init) {
+ checkLines.emplace_back(llvm::cast<llvm::StringInit>(init)->getValue());
+ });
+
+ tests.push_back(LitTest {
+ record->getName().str(),
+ name,
+ irSnippet,
+ runLines,
+ checkLines,
+ });
+ }
+
+ return tests;
+}
+
+/// Extract tests from passes
+static llvm::SmallVector<LitTest> extractPassTests(const RecordKeeper &records) {
+ llvm::SmallVector<LitTest> tests;
+
+ // Check if PassBase class exists before trying to get derived definitions
+ if (records.getClass("PassBase")) {
+ for (const llvm::Record *def : records.getAllDerivedDefinitions("PassBase")) {
+ if (def->isAnonymous())
+ continue;
+
+ auto passTests = extractTestsFromRecord(def, "passes");
+ tests.insert(tests.end(), passTests.begin(), passTests.end());
+ }
+ }
+
+ return tests;
+}
+
+/// Generate a LIT test file for an IR test
+static void generateTestFile(const LitTest &test, llvm::raw_ostream &os) {
+ // Add RUN lines
+ for (const auto& runLine : test.runLines) {
+ os << "\n" << runLine << "\n";
+ }
+
+ os << "// Generated from TableGen definition: " << test.sourceDefName << "\n\n";
+
+ // Add the test body
+ os << test.irSnippet << "\n";
+
+ // Add CHECK lines
+ for (const auto& checkLine : test.checkLines) {
+ os << "\n" << checkLine << "\n";
+ }
+}
+
+/// Main function to generate all IR test test files
+static void generateLitTests(const RecordKeeper &records, raw_ostream &os) {
+ llvm::SmallVector<LitTest> allTests;
+
+ // Extract tests from different definition types (only passes for now)
+ auto passTests = extractPassTests(records);
+
+ allTests.insert(allTests.end(), passTests.begin(), passTests.end());
+
+ if (allTests.empty()) {
+ os << "// No LitTest record found in any TableGen definition\n";
+ return;
+ }
+
+ // Generate summary
+ os << "// Generated " << allTests.size() << " LIT test files\n";
+ os << "// Use the following files for LIT testing:\n\n";
+
+ // Generate file list and content for each test
+ for (const auto& test : allTests) {
+ std::string testFileName = formatv("generated_{0}_{1}", test.sourceDefName, test.testFileName);
+ os << "// File: " << testFileName << "\n";
+
+ os << "// --- BEGIN " << testFileName << " ---\n";
+ generateTestFile(test, os);
+ os << "// --- END " << testFileName << " ---\n\n";
+ }
+}
+
+//===----------------------------------------------------------------------===//
+// Generator Registration
+//===----------------------------------------------------------------------===//
+
+static mlir::GenRegistration
+ genLitTests("gen-lit-tests", "Generate LIT test files for `Testable` TableGen records",
+ [](const RecordKeeper &records, raw_ostream &os) {
+ generateLitTests(records, os);
+ return false;
+ });
\ No newline at end of file
|
Member
|
@llvm/pr-subscribers-mlir-ods Author: Kshitij Jain (jkshtj) ChangesIntroduces a new Testable base class that allows TableGen records (starting with Pass records) to embed LIT test definitions directly within their definitions. This enables co-locating tests with pass definitions for better maintainability. Key components:
Usage example in CMake: Full diff: https://github.com/llvm/llvm-project/pull/158017.diff 6 Files Affected:
diff --git a/mlir/cmake/modules/AddMLIR.cmake b/mlir/cmake/modules/AddMLIR.cmake
index 6589458ab7894..9b05b70231dba 100644
--- a/mlir/cmake/modules/AddMLIR.cmake
+++ b/mlir/cmake/modules/AddMLIR.cmake
@@ -762,3 +762,103 @@ function(mlir_target_link_libraries target type)
target_link_libraries(${target} ${type} ${ARGN})
endif()
endfunction()
+
+# Extracts LIT tests embedded in `Testable` records in `tblgen_file`
+# and generates a file per test in `output_dir`
+#
+# Example usage:
+# # Extract tests from MyPasses.td and generate them in test/Passes/
+# add_embedded_lit_tests(MyPassesEmbeddedTests
+# ${CMAKE_CURRENT_SOURCE_DIR}/include/MyPasses.td
+# ${CMAKE_CURRENT_SOURCE_DIR}/test/Passes/)
+#
+# # This will:
+# # 1. Process MyPasses.td with mlir-tblgen --gen-lit-tests
+# # 2. Extract individual test files to test/Passes/
+# # 3. Generate files like: test/Passes/generated_MyPass_test1.mlir
+#
+function(add_embedded_lit_tests target tblgen_file output_dir)
+ set(LLVM_TARGET_DEFINITIONS ${tblgen_file})
+
+ # Extraction script content
+ set(EXTRACT_SCRIPT_CONTENT [[
+ # Generated extraction script
+ if(NOT CONSOLIDATED_FILE)
+ message(FATAL_ERROR "CONSOLIDATED_FILE variable is required")
+ endif()
+
+ if(NOT OUTPUT_DIR)
+ message(FATAL_ERROR "OUTPUT_DIR variable is required")
+ endif()
+
+ if(NOT EXISTS ${CONSOLIDATED_FILE})
+ message(FATAL_ERROR "Consolidated file does not exist: ${CONSOLIDATED_FILE}")
+ endif()
+
+ # Read the consolidated file
+ file(READ ${CONSOLIDATED_FILE} file_content)
+
+ # Split into lines for processing
+ string(REPLACE "\n" ";" lines "${file_content}")
+
+ set(current_filename "")
+ set(current_content "")
+ set(in_test_block FALSE)
+ set(extracted_test_files)
+
+ foreach(line IN LISTS lines)
+ # Check for filename line
+ if(line MATCHES "^// File: (.+)$")
+ set(current_filename "${CMAKE_MATCH_1}")
+ endif()
+
+ # Check for BEGIN marker
+ if(line MATCHES "^// --- BEGIN .+ ---$")
+ set(in_test_block TRUE)
+ set(current_content "")
+ # Check for END marker
+ elseif(line MATCHES "^// --- END .+ ---$")
+ set(in_test_block FALSE)
+
+ # Write the extracted content to file
+ if(current_filename AND current_content)
+ file(MAKE_DIRECTORY ${OUTPUT_DIR})
+ file(WRITE ${OUTPUT_DIR}/${current_filename} "${current_content}")
+ message(STATUS "Extracted test file: ${current_filename}")
+ list(APPEND extracted_test_files ${current_filename})
+ endif()
+
+ set(current_filename "")
+ set(current_content "")
+ # Collect content within BEGIN/END block
+ elseif(in_test_block)
+ string(APPEND current_content "${line}\n")
+ endif()
+ endforeach()
+
+ list(LENGTH extracted_test_files num_extracted_files)
+ message(STATUS "Extracted ${num_extracted_files} test files to ${OUTPUT_DIR}")
+ ]])
+
+ # Write extraction script to a file in the build directory
+ file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/extract_lit_tests.cmake "${EXTRACT_SCRIPT_CONTENT}")
+
+ # Process tblgen_file and generate a file with all embedded LIT
+ # tests in tblgen_file
+ get_filename_component(tblgen_name ${tblgen_file} NAME_WE)
+ set(consolidated_output_file ${tblgen_name}_extracted_lit_tests.txt)
+ mlir_tablegen(${consolidated_output_file} --gen-lit-tests)
+
+ # Add public tablegen target to trigger builds on changes in tblgen_file
+ add_public_tablegen_target(${target})
+
+ # Call the extraction script to extract all LIT tests into individual
+ # `.mlir` test files
+ add_custom_command(TARGET ${target} POST_BUILD
+ COMMAND ${CMAKE_COMMAND}
+ -DCONSOLIDATED_FILE=${CMAKE_CURRENT_BINARY_DIR}/${consolidated_output_file}
+ -DOUTPUT_DIR=${output_dir}
+ -P ${CMAKE_CURRENT_BINARY_DIR}/extract_lit_tests.cmake
+ COMMENT "Extracting LIT tests to individual files"
+ )
+endfunction()
\ No newline at end of file
diff --git a/mlir/include/mlir/IR/Testable.td b/mlir/include/mlir/IR/Testable.td
new file mode 100644
index 0000000000000..15814ed1bd939
--- /dev/null
+++ b/mlir/include/mlir/IR/Testable.td
@@ -0,0 +1,40 @@
+//===-- Testable.td - Testable type definition file --------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains the definition of the `Testable` type.
+//
+// Any type whose records can have corresponding LIT tests (eg - Pass) can extend
+// `Testable` in order to be able to embed LIT tests within record definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef TESTABLE
+#define TESTABLE
+
+// Represents a LIT test record in TableGen
+class LitTest<string name, code snippet, list<string> run = [], list<string> check = []> {
+ // The name of the generated test file
+ string testFileName = name;
+
+ // The IR snippet/code to be tested
+ code irSnippet = snippet;
+
+ // The RUN commands for the test (e.g., "mlir-opt %s")
+ list<string> runLines = run;
+
+ // Expected output patterns (CHECK lines)
+ list<string> checkLines = check;
+}
+
+// Base class for elements that can have auto-generated LIT tests
+class Testable {
+ // List of LIT tests associated with this element
+ list<LitTest> tests = [];
+}
+
+#endif // TESTABLE
\ No newline at end of file
diff --git a/mlir/include/mlir/Pass/PassBase.td b/mlir/include/mlir/Pass/PassBase.td
index e37f9735e2241..50ea44419ca24 100644
--- a/mlir/include/mlir/Pass/PassBase.td
+++ b/mlir/include/mlir/Pass/PassBase.td
@@ -14,6 +14,8 @@
#ifndef MLIR_PASS_PASSBASE
#define MLIR_PASS_PASSBASE
+include "mlir/IR/Testable.td"
+
//===----------------------------------------------------------------------===//
// Options
//===----------------------------------------------------------------------===//
@@ -62,7 +64,7 @@ class Statistic<string varName, string statName, string desc> {
// Pass
//===----------------------------------------------------------------------===//
-class PassBase<string passArg, string base> {
+class PassBase<string passArg, string base> : Testable {
// The command line argument of the pass.
string argument = passArg;
diff --git a/mlir/test/mlir-tblgen/gen-lit-tests.td b/mlir/test/mlir-tblgen/gen-lit-tests.td
new file mode 100644
index 0000000000000..40a03fb2b2d60
--- /dev/null
+++ b/mlir/test/mlir-tblgen/gen-lit-tests.td
@@ -0,0 +1,65 @@
+// RUN: mlir-tblgen -gen-lit-tests -I %S/../../include -dialect=test %s | FileCheck %s
+
+include "mlir/Pass/PassBase.td"
+include "mlir/IR/Testable.td"
+
+def TestPassWithEmbeddedLitTests : Pass<"test-pass-with-embedded-lit-tests"> {
+ let summary = "pass summary";
+ let description = [{
+ Pass description
+ }];
+
+ let tests = [
+ LitTest<
+ "lit_test_file_1.mlir",
+ [{
+ func.func @test1() {
+ return 42;
+ }
+ }],
+ [
+ "// RUN: mlir-opt %s --verify-roundtrip | FileCheck %s",
+ ],
+ [
+ "// RANDOM-CHECK-LABEL: func.func @test1",
+ ]
+ >,
+ LitTest<
+ "lit_test_file_2.mlir",
+ [{
+ func.func @test2() {
+ return 42;
+ }
+ }],
+ [
+ "// RUN: mlir-opt %s --verify-roundtrip | FileCheck %s",
+ ],
+ [
+ "// RANDOM-CHECK-LABEL: func.func @test2",
+ ]
+ >,
+ ];
+}
+
+// CHECK-LABEL: // Generated 2 LIT test files
+// CHECK: // Use the following files for LIT testing:
+
+// CHECK: // File: generated_TestPassWithEmbeddedLitTests_lit_test_file_1.mlir
+// CHECK: // --- BEGIN generated_TestPassWithEmbeddedLitTests_lit_test_file_1.mlir ---
+// CHECK: // RUN: mlir-opt %s --verify-roundtrip | FileCheck %s
+// CHECK: // Generated from TableGen definition: TestPassWithEmbeddedLitTests
+// CHECK: func.func @test1() {
+// CHECK: return 42;
+// CHECK: }
+// CHECK: // RANDOM-CHECK-LABEL: func.func @test1
+// CHECK: --- END generated_TestPassWithEmbeddedLitTests_lit_test_file_1.mlir ---
+
+// CHECK: // File: generated_TestPassWithEmbeddedLitTests_lit_test_file_2.mlir
+// CHECK: // --- BEGIN generated_TestPassWithEmbeddedLitTests_lit_test_file_2.mlir ---
+// CHECK: // RUN: mlir-opt %s --verify-roundtrip | FileCheck %s
+// CHECK: // Generated from TableGen definition: TestPassWithEmbeddedLitTests
+// CHECK: func.func @test2() {
+// CHECK: return 42;
+// CHECK: }
+// CHECK: // RANDOM-CHECK-LABEL: func.func @test2
+// CHECK: // --- END generated_TestPassWithEmbeddedLitTests_lit_test_file_2.mlir ---
\ No newline at end of file
diff --git a/mlir/tools/mlir-tblgen/CMakeLists.txt b/mlir/tools/mlir-tblgen/CMakeLists.txt
index 2a7ef7e0576c8..e721f1e26a2bd 100644
--- a/mlir/tools/mlir-tblgen/CMakeLists.txt
+++ b/mlir/tools/mlir-tblgen/CMakeLists.txt
@@ -16,6 +16,7 @@ add_tablegen(mlir-tblgen MLIR
EnumsGen.cpp
EnumPythonBindingGen.cpp
FormatGen.cpp
+ LitTestGen.cpp
LLVMIRConversionGen.cpp
LLVMIRIntrinsicGen.cpp
mlir-tblgen.cpp
diff --git a/mlir/tools/mlir-tblgen/LitTestGen.cpp b/mlir/tools/mlir-tblgen/LitTestGen.cpp
new file mode 100644
index 0000000000000..49a092fa9879f
--- /dev/null
+++ b/mlir/tools/mlir-tblgen/LitTestGen.cpp
@@ -0,0 +1,170 @@
+//===- LitTestGen.cpp - LIT test generator ----------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// LitTestGen extracts `LitTest` records from `Testable` TableGen records and
+// generates corresponding LIT test files.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/TableGen/GenInfo.h"
+#include "mlir/TableGen/Operator.h"
+#include "mlir/TableGen/Pass.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/Path.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/Record.h"
+
+#include <set>
+
+using namespace mlir;
+using namespace mlir::tblgen;
+using llvm::formatv;
+using llvm::RecordKeeper;
+
+static llvm::cl::OptionCategory litTestGenCategory("Options for -gen-lit-tests");
+static llvm::cl::opt<std::string>
+ outputDir("output-dir",
+ llvm::cl::desc("Output directory for generated test files"),
+ llvm::cl::cat(litTestGenCategory),
+ llvm::cl::value_desc("directory"));
+
+
+/// Cpp type corresponding to the `LitTest` record type in TableGen
+struct LitTest {
+ std::string sourceDefName;
+ std::string testFileName;
+ std::string irSnippet;
+ llvm::SmallVector<std::string> runLines;
+ llvm::SmallVector<std::string> checkLines;
+};
+
+static llvm::SmallVector<LitTest> extractTestsFromRecord(const llvm::Record *record,
+ llvm::StringRef dialectName = "") {
+ llvm::SmallVector<LitTest> tests;
+
+ // Check if the record has a tests field
+ const llvm::RecordVal *testsVal = record->getValue("tests");
+ if (!testsVal)
+ return tests;
+
+ const llvm::ListInit *testsList =
+ llvm::dyn_cast_or_null<llvm::ListInit>(testsVal->getValue());
+ if (!testsList)
+ return tests;
+
+ for (const llvm::Init *init : testsList->getElements()) {
+ const llvm::DefInit *defInit = llvm::dyn_cast<llvm::DefInit>(init);
+ if (!defInit)
+ continue;
+
+ const llvm::Record *testRec = defInit->getDef();
+
+ // Extract fields from LitTest record
+ std::string name = testRec->getValueAsString("testFileName").str();
+ std::string irSnippet = testRec->getValueAsString("irSnippet").str();
+
+ llvm::SmallVector<std::string> runLines;
+ llvm::for_each(*testRec->getValueAsListInit("runLines"), [&](const llvm::Init *init) {
+ runLines.emplace_back(llvm::cast<llvm::StringInit>(init)->getValue());
+ });
+
+ llvm::SmallVector<std::string> checkLines;
+ llvm::for_each(*testRec->getValueAsListInit("checkLines"), [&](const llvm::Init *init) {
+ checkLines.emplace_back(llvm::cast<llvm::StringInit>(init)->getValue());
+ });
+
+ tests.push_back(LitTest {
+ record->getName().str(),
+ name,
+ irSnippet,
+ runLines,
+ checkLines,
+ });
+ }
+
+ return tests;
+}
+
+/// Extract tests from passes
+static llvm::SmallVector<LitTest> extractPassTests(const RecordKeeper &records) {
+ llvm::SmallVector<LitTest> tests;
+
+ // Check if PassBase class exists before trying to get derived definitions
+ if (records.getClass("PassBase")) {
+ for (const llvm::Record *def : records.getAllDerivedDefinitions("PassBase")) {
+ if (def->isAnonymous())
+ continue;
+
+ auto passTests = extractTestsFromRecord(def, "passes");
+ tests.insert(tests.end(), passTests.begin(), passTests.end());
+ }
+ }
+
+ return tests;
+}
+
+/// Generate a LIT test file for an IR test
+static void generateTestFile(const LitTest &test, llvm::raw_ostream &os) {
+ // Add RUN lines
+ for (const auto& runLine : test.runLines) {
+ os << "\n" << runLine << "\n";
+ }
+
+ os << "// Generated from TableGen definition: " << test.sourceDefName << "\n\n";
+
+ // Add the test body
+ os << test.irSnippet << "\n";
+
+ // Add CHECK lines
+ for (const auto& checkLine : test.checkLines) {
+ os << "\n" << checkLine << "\n";
+ }
+}
+
+/// Main function to generate all IR test test files
+static void generateLitTests(const RecordKeeper &records, raw_ostream &os) {
+ llvm::SmallVector<LitTest> allTests;
+
+ // Extract tests from different definition types (only passes for now)
+ auto passTests = extractPassTests(records);
+
+ allTests.insert(allTests.end(), passTests.begin(), passTests.end());
+
+ if (allTests.empty()) {
+ os << "// No LitTest record found in any TableGen definition\n";
+ return;
+ }
+
+ // Generate summary
+ os << "// Generated " << allTests.size() << " LIT test files\n";
+ os << "// Use the following files for LIT testing:\n\n";
+
+ // Generate file list and content for each test
+ for (const auto& test : allTests) {
+ std::string testFileName = formatv("generated_{0}_{1}", test.sourceDefName, test.testFileName);
+ os << "// File: " << testFileName << "\n";
+
+ os << "// --- BEGIN " << testFileName << " ---\n";
+ generateTestFile(test, os);
+ os << "// --- END " << testFileName << " ---\n\n";
+ }
+}
+
+//===----------------------------------------------------------------------===//
+// Generator Registration
+//===----------------------------------------------------------------------===//
+
+static mlir::GenRegistration
+ genLitTests("gen-lit-tests", "Generate LIT test files for `Testable` TableGen records",
+ [](const RecordKeeper &records, raw_ostream &os) {
+ generateLitTests(records, os);
+ return false;
+ });
\ No newline at end of file
|
You can test this locally with the following command:git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/tools/mlir-tblgen/LitTestGen.cpp mlir/tools/mlir-tblgen/OpDocGen.cpp --diff_from_common_commit
View the diff from clang-format here.diff --git a/mlir/tools/mlir-tblgen/LitTestGen.cpp b/mlir/tools/mlir-tblgen/LitTestGen.cpp
index 9cf6b987a..bdc3c8319 100644
--- a/mlir/tools/mlir-tblgen/LitTestGen.cpp
+++ b/mlir/tools/mlir-tblgen/LitTestGen.cpp
@@ -1,4 +1,5 @@
-//===- LitTestGen.cpp - LIT test generator ----------------------------------===//
+//===- LitTestGen.cpp - LIT test generator
+//----------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
@@ -6,7 +7,7 @@
//
//===----------------------------------------------------------------------===//
//
-// LitTestGen extracts `LitTest` records from `Testable` TableGen records and
+// LitTestGen extracts `LitTest` records from `Testable` TableGen records and
// generates corresponding LIT test files.
//
//===----------------------------------------------------------------------===//
@@ -28,13 +29,11 @@ using namespace mlir::tblgen;
using llvm::formatv;
using llvm::RecordKeeper;
-static llvm::cl::OptionCategory litTestGenCategory("Options for -gen-lit-tests");
-static llvm::cl::opt<std::string>
- outputDir("output-dir",
- llvm::cl::desc("Output directory for generated test files"),
- llvm::cl::cat(litTestGenCategory),
- llvm::cl::value_desc("directory"));
-
+static llvm::cl::OptionCategory
+ litTestGenCategory("Options for -gen-lit-tests");
+static llvm::cl::opt<std::string> outputDir(
+ "output-dir", llvm::cl::desc("Output directory for generated test files"),
+ llvm::cl::cat(litTestGenCategory), llvm::cl::value_desc("directory"));
/// Cpp type corresponding to the `LitTest` record type in TableGen
struct LitTest {
@@ -146,17 +145,18 @@ extractTestsFromRecord(const llvm::Record *record) {
/// Generate a LIT test file for an IR test
static void generateTestFile(const LitTest &test, llvm::raw_ostream &os) {
// Add RUN lines
- for (const auto& runLine : test.runLines) {
+ for (const auto &runLine : test.runLines) {
os << "\n" << runLine << "\n";
}
- os << "// Generated from TableGen definition: " << test.sourceDefName << "\n\n";
-
+ os << "// Generated from TableGen definition: " << test.sourceDefName
+ << "\n\n";
+
// Add the test body
os << test.irSnippet << "\n";
-
+
// Add CHECK lines
- for (const auto& checkLine : test.checkLines) {
+ for (const auto &checkLine : test.checkLines) {
os << "\n" << checkLine << "\n";
}
}
@@ -186,8 +186,9 @@ static void generateLitTests(const RecordKeeper &records, raw_ostream &os) {
os << "// Use the following files for LIT testing:\n\n";
// Generate file list and content for each test
- for (const auto& test : allTests) {
- std::string testFileName = formatv("generated_{0}_{1}", test.sourceDefName, test.testFileName);
+ for (const auto &test : allTests) {
+ std::string testFileName =
+ formatv("generated_{0}_{1}", test.sourceDefName, test.testFileName);
os << "// File: " << testFileName << "\n";
os << "// --- BEGIN " << testFileName << " ---\n";
@@ -201,8 +202,9 @@ static void generateLitTests(const RecordKeeper &records, raw_ostream &os) {
//===----------------------------------------------------------------------===//
static mlir::GenRegistration
- genLitTests("gen-lit-tests", "Generate LIT test files for `Testable` TableGen records",
- [](const RecordKeeper &records, raw_ostream &os) {
- generateLitTests(records, os);
- return false;
- });
\ No newline at end of file
+ genLitTests("gen-lit-tests",
+ "Generate LIT test files for `Testable` TableGen records",
+ [](const RecordKeeper &records, raw_ostream &os) {
+ generateLitTests(records, os);
+ return false;
+ });
\ No newline at end of file
|
Contributor
|
If I understand correctly, this PR does not change the documentation to reflect the pass behavior, correct? It seems like it only moves the definition of the lit test to the tablegen. Having the lit test definition closer to the documentation would certainly help people remember to keep the docs in sync, but the original idea of https://discourse.llvm.org/t/survey-interested-in-discussing-richer-tablegen-descriptions-in-mlir/87959/2 was to replace hand-written before/after examples in the description string with actual pass executions. Maybe that could be combined with a change like this, so that the lit test generated in the tablegen is also rendered on the website? |
j2kun
reviewed
Sep 11, 2025
| LitTest< | ||
| "lit_test_file_1.mlir", | ||
| [{ | ||
| func.func @test1() { |
Contributor
There was a problem hiding this comment.
Rather than try to reconstruct the list test from separate records, it seems like it would be simpler to have a single text field that contains the entire lit test, and this feature would limit the allowed complexity of the RUN line. (e.g., only one RUN line with one pipe)
Or you could have the entire lit test in one field, except for the RUN line, and autogenerate the run line as // RUN: mlir-opt %s <pass-flag> | FileCheck %s, since <pass-flag> is already part of the tablegen record and also it would be a good practice to ensure that the behavior tested/documented for a pass is restricted to just what that one pass does.
If you render the before/after example from the lit test, then, you could just strip out any lines starting with \s*// CHECK.
Collaborator
There was a problem hiding this comment.
I feel the same, we actually have a prototype downstream.
let mlirExamples = [[{
# func.func @std_for(%arg0 : index, %arg1 : index, %arg2 : index) {
// Two nested for loops
scf.for %i0 = %arg0 to %arg1 step %arg2 {
scf.for %i1 = %arg0 to %arg1 step %arg2 {
%min_cmp = arith.cmpi slt, %i0, %i1 : index
%min = arith.select %min_cmp, %i0, %i1 : index
%max_cmp = arith.cmpi sge, %i0, %i1 : index
%max = arith.select %max_cmp, %i0, %i1 : index
// A loop with an empty body
scf.for %i2 = %min to %max step %i1 {
}
}
}
# return
# }
}]];
We automatically generate the round-trip testing from this, and use it to generate an "example" section in the documentation, while stripping the lines starting with # from the documentation, which allows to build more complex example without clobbering the documentation with the surrounding noise.
Collaborator
There was a problem hiding this comment.
Instead of a separate tablegen field, I would consider some markdown in the description field, that we can do:
let description = [[{
This op is doing ... and ...
The basic usage is ....
```mlir_example
# func.func @foo(....) {
"my.op"() ......
# }
` ``
It optional takes an extra flag to indicate ...
```mlir_example
# func.func @foo(....) {
"my.op"() some_flag ......
# }
` ``
etc.
}]]];
This can then be similarly extracted for round-trip test generation by default, while also integrating nicely when generating the markdown documentation (eliding the commented out lines).
Contributor
Author
There was a problem hiding this comment.
Thank you for the suggestions. In general I agree with the ideas you guys mentioned. I have a few questions to get a better understanding of the desired end product.
Do we always want to include the IR examples in the documentation or do we want to give the user an option to only generate LIT tests from them?
-
If it's the former then we'd always need the user to also specify the full "after" version of the IR. This "after" version obviously can be used as the CHECK lines in the test. It does increase the onus on the user a bit, but should give high-quality before/after versions of the IR to go along with documentation, as well as precise LIT tests.
-
If it's the latter, we'd still need the user to specify CHECK lines, but It'd be quite complicated to generate the "after" versions of the IR for documentation.
-
It seemed to me, from the comments, that there may be a third option of just generating a
--verify-roundtriptest.This can then be similarly extracted for round-trip test generation by default...
If so, I'm not sure what utility such IR can provide either for documentation or for testing.
I personally prefer option 1, and have the user always always specify IR examples in before/after pairs.
Thoughts?
Collaborator
There was a problem hiding this comment.
Do we always want to include the IR examples in the documentation or do we want to give the user an option to only generate LIT tests from them?
If it's not the documentation, I'd question why is it in ODS?
If it's the former then we'd always need the user to also specify the full "after" version of the IR.
I don't quite get what is the "after" version in the context of the documentation?
It seemed to me, from the comments, that there may be a third option of just generating a --verify-roundtrip test.
Right.
If so, I'm not sure what utility such IR can provide either for documentation or for testing.
I don't follow: the IR here isn't meant to be useful for testing, we already have tests for it. The problem statement as I understand it is that we want examples in the doc that are maintained up-to-date in terms of being "correct". That is: we want to test the documentation to actually have examples reflecting the implementation.
Is there another problem to solve that I'm missing?
Contributor
Author
There was a problem hiding this comment.
If it's the former then we'd always need the user to also specify the full "after" version of the IR.
I don't quite get what is the "after" version in the context of the documentation?
It seemed to me, from the comments, that there may be a third option of just generating a --verify-roundtrip test.
Right.
Ok, I see the misalignment in our expectations now.
You're primarily thinking about tests for operations while I started out with working on tests for passes. For operations, I agree that all we need are --verify-roundtrip tests. For passes however, I think it makes sense to have the user specify before/after IR snippets that we generate the tests from.
And I do like the idea of having the snippets be a part of the description field itself, so as to not introduce additional complications for documentation rendering. Let me take another shot at things based on the feedback.
Thank you for your comments!
Collaborator
There was a problem hiding this comment.
Oh passes is a whole other world, I'm not sure what would make sense for passes, I would need to see any proposal with concrete example on a few actual passes in the code base to figure out.
jpienaar
reviewed
Nov 2, 2025
| This operation demonstrates the mlir_example feature for ops. | ||
|
|
||
| Basic usage: | ||
| ```mlir_example |
Member
There was a problem hiding this comment.
If done this way, you may have to change the identication of syntax highlighting to support this (and also add newline above to make it trigger). Could you test by using mlir-www doc gen flow?
Member
There was a problem hiding this comment.
( I may suggest also considering other variants of this, else one has to change that GitHub syntax highlighting side too, so ideally one where you can match without needing changes elsewhere would be preferred)
Contributor
Author
There was a problem hiding this comment.
Hi @jpienaar, thank you for taking a look at this and your comments. I do want to clarify that I did not intend the new PR revision to be review-ready yet. As such, I have marked it as a draft for now.
That said, UX for this feature is going to look similar to what you see here, so I have some questions to understand your comments better.
you may have to change the identication of syntax highlighting to support this (and also add newline above to make it trigger).
Could you please elaborate what you mean here?
Could you test by using mlir-www doc gen flow?
What is the mlir-www doc gen flow, how do I use it, and what am I testing for exactly?
( I may suggest also considering other variants of this, else one has to change that GitHub syntax highlighting side too, so ideally one where you can match without needing changes elsewhere would be preferred)
Also might be missing some prior knowledge/context here to understand what we're talking about. Could you please explain further?
P.S. - I'd be happy to continue this conversation on discord or via a call for faster iteration, if you think it's needed.
Member
There was a problem hiding this comment.
Syntax highlighting in markdown in the generated website via Hugo ( https://github.com/llvm/mlir-www/blob/main/.github/workflows/main.yml ) and GitHub have syntax highlighting support for code blocks in markdown. But for that, there is an identifier that's used to determine which highlighting to use (there is auto detection too but doubt MLIR detection is great). So
func.func @foo(%arg0: i32){
// Do something
}looks different from
func.func @foo(%arg0: i32){
// Do something
}
as the latter is not considered a block of MLIR and treated as unknown format. So ideally the method to delineate MLIR examples is such that we don't lose syntax highlighting across either - I believe that's doable and just needs testing to verify.
Contributor
Author
There was a problem hiding this comment.
Makes sense. Let me take a look at things.
jkshtj
force-pushed
the
jkshtj/tblgen-embedded-lit-tests
branch
3 times, most recently
from
0395bd6 to
d3eb351
Compare
jkshtj
force-pushed
the
jkshtj/tblgen-embedded-lit-tests
branch
from
d3eb351 to
890cf81
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a new Testable base class that allows TableGen records (starting with Pass records) to embed LIT test definitions directly within their definitions. This enables co-locating tests with pass definitions for better maintainability.
Key components:
Usage example in CMake: