Merge pull request #222 from mathworks/2.8.1-SNAPSHOT-Qualification

2.8.1 snapshot qualification

Author: nbhoski

CONFIGDOC.md

@@ -24,43 +24,37 @@ To follow the steps in this example:
 Create a new project and configure it by following these steps:
 1. In your Jenkins interface, select **New Item** on the left. A new page opens where you can choose the type of your project. Enter a project name, and then click **Freestyle project**. To confirm your choices, click **OK**.
 
-![create_project](https://user-images.githubusercontent.com/48831250/105080784-520e6480-5a5f-11eb-9218-4d43013e2850.png)
+![create_project](https://user-images.githubusercontent.com/48831250/193948023-65f038c2-a81e-416b-8dea-0cf91ae5f105.png)
 
 2. On the project configuration page, in the **Source Code Management** section, specify the repository that hosts your tests.
 
 ![source_control](https://user-images.githubusercontent.com/48831250/94478391-37a73700-01a1-11eb-9f89-a5a71413baf0.png)
 
 3. In the **Build Environment** section, select **Use MATLAB version** and specify the MATLAB version you want to use in the build. If your desired MATLAB version is not listed under **Use MATLAB version**, enter the full path to its root folder in the **MATLAB root** box. 
 
-![build_environment](https://user-images.githubusercontent.com/48831250/105091260-943ea280-5a6d-11eb-8d11-48747df7ec32.png)
+![build_environment](https://user-images.githubusercontent.com/48831250/193948480-564ac249-0ef2-407b-b18c-ee52b5129ac0.png)
 
 4. In the **Build** section, select **Add build step > Run MATLAB Tests**. Then, specify the artifacts to be generated in the project workspace. In this example, the plugin generates Cobertura code coverage and JUnit-style test results reports. Furthermore, to generate the coverage report, the plugin uses only the code in the `source` folder located in the root of the repository. For more information about the build steps provided by the plugin, see [Plugin Configuration Guide](../CONFIGDOC.md).
 
-![run_matlab_tests](https://user-images.githubusercontent.com/48831250/105909903-2149a480-5ff6-11eb-81f1-c3b44e9b17d1.png)
+![run_,matlab_tests](https://user-images.githubusercontent.com/48831250/193948891-aa09960a-04ba-4a13-9eea-e4bebab9371d.png)
 
 5. In the **Post-build Actions** section, add two post-build actions to publish the Cobertura code coverage and JUnit-style test results reports. For each artifact, provide the path to the report.
 
-![post_build](https://user-images.githubusercontent.com/48831250/105082096-14aad680-5a61-11eb-9868-68d018199f9d.png)
+![post_build](https://user-images.githubusercontent.com/48831250/193949176-aced5dfb-f7b9-4978-8726-7d01bae4bc97.png)
 
 6. Click **Save** to save the project configuration settings. You can access and modify your settings at a later stage by selecting **Configure** in the project interface, which displays the project name at the top-left of the page.
 
 ## Run Tests and Inspect Artifacts
 To build your freestyle project, click **Build Now** in the project interface. Jenkins triggers a build, assigns it a number under **Build History**, and runs the build. If the build is successful, a blue circle icon appears next to the build number. If the build fails, Jenkins adds a red circle icon. In this example, the build succeeds because all the tests in the Times Table App project pass.
 
-![build_1](https://user-images.githubusercontent.com/48831250/105084788-dca59280-5a64-11eb-858d-664a5727a947.png)
-
 Navigate to the project workspace by clicking the **Workspace** icon in the project interface. The generated artifacts are in the `matlabTestArtifacts` folder of the workspace.
 
-![workspace](https://user-images.githubusercontent.com/48831250/105085419-b46a6380-5a65-11eb-8d46-747dd23291bf.png)
-
-Access the published Cobertura code coverage report by opening the **Coverage Report** link in the project interface.
-
-![cobertura_report](https://user-images.githubusercontent.com/48831250/105085331-9ac91c00-5a65-11eb-9628-efbf70520489.png)
+![workspace](https://user-images.githubusercontent.com/48831250/193951807-a09ef1f8-cf7e-49c8-8615-af5fbd97acd0.png)
 
-To view the published JUnit-style test results, open the **Latest Test Result** link in the project interface. In the new page, open the link in the **All Tests** table. The table expands and lists information for each of the test classes within the Times Table App project.  
+Click the **Status** icon in the project interface. You can access the published artifacts by opening the **Latest Test Result** and **Coverage Report** links. For example, open the **Latest Test Result** link to view the published JUnit-style test results. In the new page, open the link in the **All Tests** table. The table expands and lists information for each of the test classes within the Times Table App project.  
 
-![junit_report](https://user-images.githubusercontent.com/48831250/105088211-956dd080-5a69-11eb-931c-aef201eb9dbe.png)
+![test_results](https://user-images.githubusercontent.com/48831250/194086287-35f7c677-4c18-4316-b676-f2491ffa20a7.png)
 
 ## See Also
 * [Plugin Configuration Guide](../CONFIGDOC.md)<br/>
-* [Explore an Example Project (MATLAB)](https://www.mathworks.com/help/matlab/matlab_prog/explore-an-example-project.html)
+* [Explore an Example Project (MATLAB)](https://www.mathworks.com/help/matlab/matlab_prog/explore-an-example-project.html)

examples/Run-MATLAB-Tests.md

@@ -24,6 +24,7 @@ public class MatlabBuilderConstants {
     static final String COBERTURA_CODE_COVERAGE = "'CoberturaCodeCoverage'";
     static final String COBERTURA_MODEL_COVERAGE = "'CoberturaModelCoverage'";
 
+
     // Matlab Runner files 
     static final String BAT_RUNNER_SCRIPT = "run_matlab_command.bat";
     static final String SHELL_RUNNER_SCRIPT = "run_matlab_command.sh";

src/main/java/com/mathworks/ci/MatlabBuilderConstants.java

@@ -35,6 +35,7 @@
 import hudson.Launcher.ProcStarter;
 import hudson.tasks.BuildStepDescriptor;
 import hudson.tasks.Builder;
+import hudson.util.ListBoxModel;
 import jenkins.tasks.SimpleBuildStep;
 import net.sf.json.JSONObject;
 
@@ -60,6 +61,10 @@ public class RunMatlabTestsBuilder extends Builder implements SimpleBuildStep, M
     private SourceFolder sourceFolder;
     private SelectByFolder selectByFolder;
     private SelectByTag selectByTag;
+    private String loggingLevel = "default";
+    private String outputDetail = "default";
+    private boolean useParallel = false;
+    private boolean strict = false;
 
 
     @DataBoundConstructor
@@ -113,6 +118,27 @@ public void setSourceFolder(SourceFolder sourceFolder) {
     public void setSelectByFolder(SelectByFolder selectByFolder) {
     	this.selectByFolder = selectByFolder;
     }
+    
+
+    @DataBoundSetter
+    public void setLoggingLevel(String loggingLevel) {
+        this.loggingLevel = loggingLevel;
+    }
+
+    @DataBoundSetter
+    public void setOutputDetail(String outputDetail) {
+        this.outputDetail = outputDetail;
+    }
+
+    @DataBoundSetter
+    public void setUseParallel(boolean useParallel) {
+        this.useParallel = useParallel;
+    }
+
+    @DataBoundSetter
+    public void setStrict(boolean strict) {
+        this.strict = strict;
+    }
 
 
     public String getTapReportFilePath() {
@@ -178,6 +204,24 @@ private Artifact getArtifactObject(boolean isChecked, Artifact returnVal)  {
         return (isChecked) ? returnVal : new NullArtifact();
     }
 
+    // Verbosity level
+
+    public String getLoggingLevel() {
+        return loggingLevel == null ? "default" : this.loggingLevel;
+    }
+
+    public String getOutputDetail() {
+        return outputDetail == null ? "default" : this.outputDetail;
+    }
+
+    public boolean getStrict() {
+        return this.strict;
+    }
+
+    public boolean getUseParallel() {
+        return this.useParallel;
+    }
+    
 
     // To retain Backward compatibility
     protected Object readResolve() {
@@ -229,6 +273,31 @@ public boolean configure(StaplerRequest req, JSONObject formData) throws FormExc
             save();
             return super.configure(req, formData);
         }
+        
+        // Verbosity lists
+        public ListBoxModel doFillLoggingLevelItems() {
+            ListBoxModel items = new ListBoxModel();
+
+            items.add("Default", "default");
+            items.add("None", "none");
+            items.add("Terse", "terse");
+            items.add("Concise", "concise");
+            items.add("Detailed", "detailed");
+            items.add("Verbose", "verbose");
+            return items;
+        }
+
+        public ListBoxModel doFillOutputDetailItems() {
+            ListBoxModel items = new ListBoxModel();
+
+            items.add("Default", "default");
+            items.add("None", "none");
+            items.add("Terse", "terse");
+            items.add("Concise", "concise");
+            items.add("Detailed", "detailed");
+            items.add("Verbose", "verbose");
+            return items;
+        }
 
         /*
          * This is to identify which project type in jenkins this should be applicable.(non-Javadoc)
@@ -347,6 +416,24 @@ private String getInputArguments() {
         if (getSelectByTag() != null && !getSelectByTag().getTestTag().isEmpty()) {
             getSelectByTag().addTagToInputArgs(inputArgsList);
         }
+        
+        // Add Logging level
+        if (!getLoggingLevel().equalsIgnoreCase("default")) {
+            inputArgsList.add("'LoggingLevel'" + "," + "'"
+                    + getLoggingLevel() + "'");
+        }
+
+        // Add Output Detail
+        if (!getOutputDetail().equalsIgnoreCase("default")) {
+            inputArgsList.add("'OutputDetail'" + "," + "'"
+                    + getOutputDetail() + "'");
+        }
+
+        // Add UseParallel
+        inputArgsList.add("'UseParallel'" + "," + getUseParallel());
+
+        // Add Strict
+        inputArgsList.add("'Strict'" + "," + getStrict());
 
         return String.join(",", inputArgsList);
     }

src/main/java/com/mathworks/ci/RunMatlabTestsBuilder.java

@@ -33,6 +33,10 @@ public class RunMatlabTestsStep extends Step {
     private String testResultsSimulinkTest;
     private String modelCoverageCobertura;
     private String selectByTag;
+    private String loggingLevel;
+    private String outputDetail;
+    private boolean useParallel;
+    private boolean strict;
     private List<String> sourceFolder = new ArrayList<>();
     private List<String> selectByFolder = new ArrayList<>();
 
@@ -122,6 +126,42 @@ public List<String> getSelectByFolder() {
     public void setSelectByFolder(List<String> selectByFolder) {
         this.selectByFolder = selectByFolder;
     }
+    
+    public String getLoggingLevel() {
+        return loggingLevel;
+    }
+    
+    @DataBoundSetter
+    public void setLoggingLevel(String loggingLevel) {
+        this.loggingLevel = loggingLevel;
+    }
+
+    public String getOutputDetail() {
+        return outputDetail;
+    }
+    
+    @DataBoundSetter
+    public void setOutputDetail(String outputDetail) {
+        this.outputDetail = outputDetail;
+    }
+
+    public boolean getUseParallel() {
+        return useParallel;
+    }
+    
+    @DataBoundSetter
+    public void setUseParallel(boolean useParallel) {
+        this.useParallel = useParallel;
+    }
+
+    public boolean getStrict() {
+        return strict;
+    }
+    
+    @DataBoundSetter
+    public void setStrict(boolean strict) {
+        this.strict = strict;
+    }
 
     @Override
     public StepExecution start(StepContext context) throws Exception {
@@ -155,10 +195,14 @@ private String getInputArgs() {
         inputArgs.add("'Test'");
 
         args.forEach((key, val) -> {
-            if(key.equals("SourceFolder") || key.equals("SelectByFolder") && val != null){
-                inputArgs.add("'" + key + "'" + "," + val);
-            }else if(val != null){
-                inputArgs.add("'" + key + "'" + "," + "'" + val.replaceAll("'", "''") + "'");
+            if (val != null) {
+                if (key.equals("SourceFolder") || key.equals("SelectByFolder")
+                        || key.equals("UseParallel")
+                        || key.equals("Strict")) {
+                    inputArgs.add("'" + key + "'" + "," + val);             
+                } else {
+                    inputArgs.add("'" + key + "'" + "," + "'" + val.replaceAll("'", "''") + "'");
+                }
             }
         });
 
@@ -174,6 +218,10 @@ private Map<String, String> getGenscriptArgs() {
         args.put("CoberturaCodeCoverage", getCodeCoverageCobertura());
         args.put("CoberturaModelCoverage", getModelCoverageCobertura());
         args.put("SelectByTag", getSelectByTag());
+        args.put("UseParallel", String.valueOf(getUseParallel()));
+        args.put("Strict", String.valueOf(getStrict()));
+        args.put("LoggingLevel", getLoggingLevel());
+        args.put("OutputDetail", getOutputDetail());
         addFolderArgs("SourceFolder",getSourceFolder(),args);
         addFolderArgs("SelectByFolder",getSelectByFolder(),args);
         return args;

src/main/java/com/mathworks/ci/RunMatlabTestsStep.java

@@ -1,22 +1,46 @@
 <?jelly escape-by-default='true'?>
 <j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define" xmlns:l="/lib/layout" xmlns:t="/lib/hudson" xmlns:f="/lib/form">
 
-<f:block>
+<f:block> 
     <br></br>
     <f:optionalProperty field="sourceFolder" title="Source folder" />
 </f:block>
 
+<br></br>
+
+<f:section title="Filter Tests">
 <f:block>
-    <br></br>
-   <b>Filter Tests</b>
-   <f:optionalProperty field="selectByFolder" title="By folder name" />  
+   <f:optionalProperty field="selectByFolder" title="By folder name" />
    <br></br>
-   <f:optionalProperty field="selectByTag" title="By tag" />  
+   <f:optionalProperty field="selectByTag" title="By tag" /> 
 </f:block>
+</f:section>
+<br></br>
 
+<f:section title="Customize Test Run">
+<f:block>
+    <f:entry field="strict" title="Strict " checked="${instance.strict}">
+	    <f:checkbox/> 
+	</f:entry> 
+	<f:entry field="useParallel" title="Use parallel " checked="${instance.useParallel}">
+	    <f:checkbox/>
+	</f:entry>
+
+	<!-- Verbosity fields --> 
+	
+	<f:entry name="outputDetail" title="Output detail " field="outputDetail" default="Default">
+	    <f:select/>
+	</f:entry>
+	
+	<f:entry name="loggingLevel " title="Logging level" field="loggingLevel" default="Default">
+	    <f:select/>
+	</f:entry>
+</f:block>
+</f:section>
+
+
+<f:section title="Generate Test Artifacts">
 <f:block>
-    <br></br>
-    <b>Generate Test Artifacts</b>
     <f:optionalBlock name="pdfReportArtifact" field="pdfReportArtifact" title="PDF test report" checked="${instance.pdfReportArtifact.selected}">
   	 <f:entry field="pdfReportFilePath" title="File path: ">	
     	<f:textbox default="matlabTestArtifacts/testreport.pdf"/>
@@ -40,12 +64,12 @@
     	<f:textbox default="matlabTestArtifacts/simulinktestresults.mldatx"/>
      </f:entry>   
    </f:optionalBlock>
-  
 </f:block>
+</f:section>
+<br></br>
 
+<f:section title="Generate Coverage Artifacts">
 <f:block>
-    <br></br>
-    <b>Generate Coverage Artifacts</b>
    <f:optionalBlock name="coberturaArtifact" field="coberturaArtifact" title="Cobertura code coverage" checked="${instance.coberturaArtifact.selected}">
 	 <f:entry field="coberturaReportFilePath" title="File path: ">
     	<f:textbox default="matlabTestArtifacts/cobertura.xml"/>
@@ -58,4 +82,5 @@
   	  </f:entry>
    </f:optionalBlock>
 </f:block>
+</f:section>
 </j:jelly>

src/main/resources/com/mathworks/ci/RunMatlabTestsBuilder/config.jelly

@@ -0,0 +1,3 @@
+<div>
+    <p>Maximum verbosity level for logged diagnostics included for the test run.</p>  
+</div>

src/main/resources/com/mathworks/ci/RunMatlabTestsBuilder/help-loggingLevel.html

@@ -0,0 +1,3 @@
+<div>
+    <p>Display level for event details output by the test run.</p>  
+</div>

src/main/resources/com/mathworks/ci/RunMatlabTestsBuilder/help-outputDetail.html

@@ -0,0 +1,3 @@
+<div>
+    <p>Whether to apply strict checks when running the tests. For example, the framework generates a qualification failure if a test issues a warning.</p>  
+</div>

src/main/resources/com/mathworks/ci/RunMatlabTestsBuilder/help-strict.html

@@ -0,0 +1,4 @@
+<div>
+    <p>Whether to run tests in parallel. This feature requires a Parallel Computing Toolbox license, and might not be 
+      compatible with other options. If incompatible, testing occurs in serial regardless of the value of this feature.</p>  
+</div>