Show test coverage data generated by Test Coverage Tool package

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 2.0.0 (23-Jul-2025)
+* Use [Test Coverage Tool](https://openexchange.intersystems.com/package/Test-Coverage-Tool) to present coverage information (#24)
+
 ## 0.2.3 (13-Nov-2024)
 * Add manual refresh to test trees, and automatic refresh when client setting changes (#20)
 * Fix overprompting during authentication.

README.md

@@ -1,6 +1,10 @@
 # InterSystems Testing Manager
 
-This preview extension uses VS Code's [Testing API](https://code.visualstudio.com/api/extension-guides/testing) to discover, run and debug unit test classes built with the [%UnitTest testing framework](https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=TUNT_WhatIsPercentUnitTest) of the InterSystems IRIS platforms, plus Caché-based predecessors supporting the `/api/atelier` REST service.
+> **New in Version 2.0 - Test Coverage**
+> 
+> The v2.0 release has been entered into the [InterSystems Developer Tools Contest 2025](https://openexchange.intersystems.com/contest/42). Please support it with your vote between 28th July and 3rd August.
+
+This extension uses VS Code's [Testing API](https://code.visualstudio.com/api/extension-guides/testing) to discover, run and debug unit test classes built with the [%UnitTest testing framework](https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=TUNT_WhatIsPercentUnitTest) of the InterSystems IRIS platforms, plus Caché-based predecessors supporting the `/api/atelier` REST service.
 
 It augments the ObjectScript, InterSystems Language Server and Server Manager extensions, which are elements of the [InterSystems ObjectScript Extension Pack](https://marketplace.visualstudio.com/items?itemName=intersystems-community.objectscript-pack).
 
@@ -16,6 +20,12 @@ _Client-side editing workspace_
 
 _Server-side editing workspace_
 
+When used alongside [Test Coverage Tool](https://openexchange.intersystems.com/package/Test-Coverage-Tool) this extension presents coverage inside VS Code:
+
+![Code coverage example](images/README/Coverage-example.png)
+
+_Code coverage example showing coverage of Test Coverage Tool's own unit tests_
+
 In order to support topologies in which client-side-managed test classes have to be run in the namespace of a remote server, this extension uses the `/_vscode` web application on the test-running server, no matter whether local or remote.
 
 ## Server Preparations
@@ -34,6 +44,8 @@ In order to support topologies in which client-side-managed test classes have to
     ```
 > If you previously used the `%UnitTest` framework in a namespace, be aware that you are probably replacing an existing value. Consider taking a note of that in case you need to revert.
 
+3. If you want to gather and display test coverage data, set up [Test Coverage Tool](https://openexchange.intersystems.com/package/Test-Coverage-Tool) in the namespace(s) where your tests will execute.
+
 ## Workspace Preparations
 
 For a workspace using client-side editing, test classes are by default sought in `.cls` files under the `internal/testing/unit_tests` subfolder, using the conventional layout of one additional subfolder per package-name element. If your test classes are located elsewhere, use the `intersystems.testingManager.client.relativeTestRoot` setting to point there.
@@ -50,7 +62,7 @@ A subfolder is shown for each root folder of your workspace, which may be a mult
 
 At the level of an individual test class the final expansion shows a leaf for each `TestXXX` method.
 
-Hovering over any level of a tests tree will reveal action buttons that run all the tests from this level down. The 'Run' button does so without stopping at any breakpoints, in contrast to the 'Debug' button. At class or method level a 'Go to Test' button opens the class code and positions the cursor appropriately. At higher levels this button navigates to Explorer View.
+Hovering over any level of a tests tree will reveal action buttons that run all the tests from this level down. The 'Run Test' button does so without stopping at any breakpoints, in contrast to the 'Debug Test' button. At class or method level a 'Go to Test' button opens the class code and positions the cursor appropriately. At higher levels this button navigates to Explorer View.
 
 When a test class is open in an editor tab it displays icons in the gutter at the top of the class and at the start of each test method. These show the outcome of the most recent run, if any, and can be clicked to perform testing operations.
 
@@ -59,6 +71,9 @@ The `...` menu of the Testing panel in Test Explorer includes several useful com
 ## Debugging Tests
 After opening a test class, click in the gutter to set a VS Code breakpoint in the normal manner. Then launch the test-run with the Debug option on the context menu of the testing icons in the gutter.
 
+## Obtaining Test Coverage Information
+Use the 'Run with Coverage' option to submit your tests to [Test Coverage Tool](https://openexchange.intersystems.com/package/Test-Coverage-Tool). When the run finishes the 'TEST COVERAGE' view will appear, usually below the 'TEST EXPLORER'. Use this to discover what proportion of executable code lines were covered by the most recent coverage run. Open sources to see color markers on line numbers showing covered (green) and not covered (red) lines. Learn more in the [VS Code documentation](https://code.visualstudio.com/docs/debugtest/testing#_test-coverage).
+
 ## Recent Testing History
 
 The %UnitTest framework persists results of runs in server-side tables. The 'Recent History' root folder lets you explore the most recent ten sets of results for each server and namespace the workspace uses.

package.json

@@ -1,7 +1,7 @@
 {
   "name": "testingmanager",
   "displayName": "InterSystems Testing Manager",
-  "version": "0.2.4-SNAPSHOT",
+  "version": "2.0.0-SNAPSHOT",
   "preview": true,
   "publisher": "intersystems-community",
   "description": "Manage testing on InterSystems servers.",

src/commonRunTestsHandler.ts

@@ -1,9 +1,10 @@
 import * as vscode from 'vscode';
 import { IServerSpec } from "@intersystems-community/intersystems-servermanager";
-import { allTestRuns, extensionId, osAPI } from './extension';
+import { allTestRuns, extensionId, osAPI, OurTestItem } from './extension';
 import { relativeTestRoot } from './localTests';
 import logger from './logger';
 import { makeRESTRequest } from './makeRESTRequest';
+import { OurFileCoverage } from './ourFileCoverage';
 
 export async function commonRunTestsHandler(controller: vscode.TestController, resolveItemChildren: (item: vscode.TestItem) => Promise<void>, request: vscode.TestRunRequest, cancellation: vscode.CancellationToken) {
   logger.debug(`commonRunTestsHandler invoked by controller id=${controller.id}`);
@@ -14,14 +15,29 @@ export async function commonRunTestsHandler(controller: vscode.TestController, r
   // We don't yet support running only some TestXXX methods in a testclass
   const mapAuthorities = new Map<string, Map<string, vscode.TestItem>>();
   const runIndices: number[] =[];
-  const queue: vscode.TestItem[] = [];
+  const queue: OurTestItem[] = [];
+  const coverageRequest = request.profile?.kind === vscode.TestRunProfileKind.Coverage;
 
   // Loop through all included tests, or all known tests, and add them to our queue
   if (request.include) {
-    request.include.forEach(test => queue.push(test));
+    request.include.forEach((test: OurTestItem) => {
+      if (!coverageRequest || test.supportsCoverage) {
+        queue.push(test);
+      }
+    });
   } else {
     // Run was launched from controller's root level
-    controller.items.forEach(test => queue.push(test));
+    controller.items.forEach((test: OurTestItem) => {
+      if (!coverageRequest || test.supportsCoverage) {
+        queue.push(test);
+      }
+    });
+  }
+
+  if (coverageRequest && !queue.length) {
+    // No tests to run, but coverage requested
+    vscode.window.showErrorMessage("[Test Coverage Tool](https://openexchange.intersystems.com/package/Test-Coverage-Tool) not found.", );
+    return;
   }
 
   // Process every test that was queued. Recurse down to leaves (testmethods) and build a map of their parents (classes)
@@ -69,7 +85,7 @@ export async function commonRunTestsHandler(controller: vscode.TestController, r
 
   if (mapAuthorities.size === 0) {
     // Nothing included
-    vscode.window.showWarningMessage(`Empty test run`);
+    vscode.window.showErrorMessage("Empty test run.", { modal: true });
     return;
   }
 
@@ -143,6 +159,49 @@ export async function commonRunTestsHandler(controller: vscode.TestController, r
         console.log(error);
       }
 
+      // Map of uri strings checked for presence of a coverage.list file, recording the relative path of those that were found
+      const mapCoverageLists = new Map<string, string>();
+      for await (const mapInstance of mapTestClasses) {
+        const key = mapInstance[0];
+        const pathParts = key.split('/');
+        pathParts.pop();
+        const sourceBaseUri = mapInstance[1].uri?.with({ path: mapInstance[1].uri.path.split('/').slice(0, -pathParts.length).join('/') });
+        if (!sourceBaseUri) {
+          console.log(`No sourceBaseUri for key=${key}`);
+          continue;
+        }
+        while (pathParts.length > 1) {
+          const currentPath = pathParts.join('/');
+          // Check for coverage.list file here
+          const coverageListUri = sourceBaseUri.with({ path: sourceBaseUri.path.concat(`${currentPath}/coverage.list`) });
+          if (mapCoverageLists.has(coverageListUri.toString())) {
+            // Already checked this uri path, and therefore all its ancestors
+            break;
+          }
+          try {
+            await vscode.workspace.fs.stat(coverageListUri);
+            mapCoverageLists.set(coverageListUri.toString(), currentPath);
+          } catch (error) {
+            if (error.code !== vscode.FileSystemError.FileNotFound().code) {
+              console.log(`Error checking for ${coverageListUri.toString()}:`, error);
+            }
+            mapCoverageLists.set(coverageListUri.toString(), '');
+          }
+          pathParts.pop();
+        }
+      }
+      // Copy all coverage.list files found into the corresponding place under testRoot
+      for await (const [uriString, path] of mapCoverageLists) {
+        if (path.length > 0) {
+          const coverageListUri = vscode.Uri.parse(uriString, true);
+          try {
+            await vscode.workspace.fs.copy(coverageListUri, testRoot.with({ path: testRoot.path.concat(`${path}/coverage.list`) }));
+          } catch (error) {
+            console.log(`Error copying ${coverageListUri.path}:`, error);
+          }
+        }
+      }
+
       // Next, copy the classes into the folder as a package hierarchy
       for await (const mapInstance of mapTestClasses) {
         const key = mapInstance[0];
@@ -186,11 +245,18 @@ export async function commonRunTestsHandler(controller: vscode.TestController, r
         }
       }
 
+      let managerClass = "%UnitTest.Manager";
+      if (coverageRequest) {
+        managerClass = "TestCoverage.Manager";
+        request.profile.loadDetailedCoverage = async (testRun, fileCoverage, token) => {
+          return fileCoverage instanceof OurFileCoverage ? fileCoverage.loadDetailedCoverage() : [];
+        };
+      }
       const configuration = {
         "type": "objectscript",
         "request": "launch",
         "name": `${controller.id.split("-").pop()}Tests:${serverSpec.name}:${namespace}:${username}`,
-        "program": `##class(%UnitTest.Manager).RunTest("${testSpec}","${runQualifiers}")`,
+        "program": `##class(${managerClass}).RunTest("${testSpec}","${runQualifiers}")`,
 
         // Extra properties needed by our DebugAdapterTracker
         "testingRunIndex": runIndex,

src/coverage.ts

@@ -0,0 +1,70 @@
+import * as vscode from 'vscode';
+import { makeRESTRequest } from './makeRESTRequest';
+import logger from './logger';
+import { OurTestRun } from './extension';
+import { serverSpecForUri } from './historyExplorer';
+import { OurFileCoverage } from './ourFileCoverage';
+
+export async function processCoverage(serverName: string, namespace: string, run: OurTestRun): Promise<void> {
+  const uri = run.debugSession?.workspaceFolder?.uri;
+  const coverageIndex = run.debugSession?.configuration.coverageIndex;
+  logger.debug(`processCoverage: serverName=${serverName}, namespace=${namespace}, uri=${uri?.toString()}, coverageIndex=${coverageIndex}`);
+  if (uri) {
+    const fileCoverageResults = await getFileCoverageResults(uri, namespace, coverageIndex || 0);
+    if (fileCoverageResults.length > 0) {
+      logger.debug(`Coverage results for run ${coverageIndex}: ${JSON.stringify(fileCoverageResults)}`);
+      fileCoverageResults.forEach(fileCoverage => {
+        run.addCoverage(fileCoverage);
+      })
+    } else {
+      logger.debug(`No coverage results found for run ${coverageIndex}`);
+    }
+  }
+}
+
+export async function getFileCoverageResults(folderUri: vscode.Uri, namespace: string, coverageIndex: number): Promise<vscode.FileCoverage[]> {
+  const serverSpec = serverSpecForUri(folderUri);
+  const fileCoverageResults: vscode.FileCoverage[] = [];
+  if (!serverSpec) {
+    logger.error(`No server spec found for URI: ${folderUri.toString()}`);
+    return fileCoverageResults;
+  }
+  const exportSettings = vscode.workspace.getConfiguration('objectscript.export', folderUri);
+  const response = await makeRESTRequest(
+    "POST",
+    serverSpec,
+    { apiVersion: 1, namespace, path: "/action/query" },
+    {
+      query: "SELECT cu.Hash, cu.Name Name, cu.Type, abcu.ExecutableLines, CoveredLines, ExecutableMethods, CoveredMethods, RtnLine FROM TestCoverage_Data_Aggregate.ByCodeUnit abcu, TestCoverage_Data.CodeUnit cu WHERE abcu.CodeUnit = cu.Hash AND Run = ? ORDER BY Name",
+      parameters: [coverageIndex],
+    },
+  );
+  if (response) {
+    response?.data?.result?.content?.forEach(element => {
+      const fileType = element.Type.toLowerCase();
+      let pathPrefix = ''
+      if (folderUri.scheme === 'file') {
+        pathPrefix = exportSettings.folder;
+        if (pathPrefix && !pathPrefix.startsWith('/')) {
+          pathPrefix = `/${pathPrefix}`;
+        }
+        if (exportSettings.atelier) {
+          pathPrefix += '/' + fileType;
+        }
+      }
+      const fileUri = folderUri.with({ path: folderUri.path.concat(pathPrefix, `/${element.Name.replace(/\./g, '/')}.${fileType}`) });
+      logger.debug(`getFileCoverageResults element: ${JSON.stringify(element)}`);
+      logger.debug(`getFileCoverageResults fileUri: ${fileUri.toString()}`);
+      const fileCoverage = new OurFileCoverage(
+        coverageIndex,
+        element.Hash,
+        fileUri,
+        new vscode.TestCoverageCount(element.CoveredLines, element.ExecutableLines),
+        undefined,
+        new vscode.TestCoverageCount(element.CoveredMethods, element.ExecutableMethods)
+      );
+      fileCoverageResults.push(fileCoverage);
+    });
+  }
+  return fileCoverageResults;
+}

src/debugTracker.ts

@@ -1,14 +1,15 @@
 import * as vscode from 'vscode';
-import { allTestRuns, loadedTestController, localTestController, TestRun } from './extension';
+import { allTestRuns, loadedTestController, localTestController, OurTestRun } from './extension';
 import { refreshHistoryRootItem } from './historyExplorer';
+import { processCoverage } from './coverage';
 
 export class DebugTracker implements vscode.DebugAdapterTracker {
 
   private session: vscode.DebugSession;
   private serverName: string;
   private namespace: string;
   private testController: vscode.TestController
-  private run?: TestRun;
+  private run?: OurTestRun;
   private testingIdBase: string;
   private className?: string;
   private testMethodName?: string;
@@ -56,6 +57,14 @@ export class DebugTracker implements vscode.DebugAdapterTracker {
       }
       const line: string = (message.body.output as string).replace(/\n/, '');
       this.run.appendOutput(line + '\r\n');
+
+      const coverageMatch = line.match(/^(?:http|https):\/\/.*\/TestCoverage\.UI\.AggregateResultViewer\.cls\?Index=(\d+)/);
+      if (coverageMatch && this.run.debugSession) {
+        const coverageIndex = Number(coverageMatch[1]);
+        this.run.debugSession.configuration.coverageIndex = coverageIndex;
+        console.log(`Coverage index set to ${coverageIndex}`);
+      }
+
       if (this.className === undefined) {
         const classBegin = line.match(/^    ([%\dA-Za-z][\dA-Za-z0-9\.]*) begins \.\.\./);
         if (classBegin) {
@@ -139,10 +148,13 @@ export class DebugTracker implements vscode.DebugAdapterTracker {
     //console.log(`**Starting session ${this.session.name}, run.name = ${this.run?.name}`);
   }
 
-  onWillStopSession(): void {
-    //console.log(`**Stopping session ${this.session.name}`);
+  async onWillStopSession(): Promise<void> {
+    console.log(`**Stopping session ${this.session.name}`);
     if (this.run) {
+      await processCoverage(this.serverName, this.namespace, this.run);
+      //console.log(`**processCoverage done`);
       this.run.end();
+      //console.log(`**run.end() done`);
       refreshHistoryRootItem(this.serverName, this.namespace);
     }
 

src/extension.ts

@@ -14,10 +14,15 @@ export let historyBrowserController: vscode.TestController;
 export let osAPI: any;
 export let smAPI: serverManager.ServerManagerAPI | undefined;
 
-export interface TestRun extends vscode.TestRun {
+export interface OurTestRun extends vscode.TestRun {
   debugSession?: vscode.DebugSession
 }
-export const allTestRuns: (TestRun | undefined)[] = [];
+
+export interface OurTestItem extends vscode.TestItem {
+  supportsCoverage?: boolean
+}
+
+export const allTestRuns: (OurTestRun | undefined)[] = [];
 
 async function getServerManagerAPI(): Promise<serverManager.ServerManagerAPI | undefined> {
     const targetExtension = vscode.extensions.getExtension("intersystems-community.servermanager");