-
Notifications
You must be signed in to change notification settings - Fork 395
Changing TestAll.ps1 script to run tests locally with filtering #5868
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 4 commits
88a7666
1c42580
7ab8869
be3f0e4
263b4ac
8071b27
8f45946
62fe6aa
1a8ab9a
fb8a250
e6268f4
c8249c1
f19cfd6
17c7b36
012c0dd
44fb939
40502cf
2b1bd8c
b6a213d
268fdf6
c0a83cc
4f3da2d
5ab53e3
21837df
fcfbfe7
c50904e
c9e4183
52d2185
ea45810
3bf63ac
8a5cb37
f34382d
4050fff
63deae4
dc75b5c
7c56b3f
f5c457d
37593d8
06101c6
5c578a1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| @echo off | ||
|
|
||
| powershell -ExecutionPolicy Unrestricted -NoLogo -NoProfile -File %~dp0\tools\RunTests.ps1 %* | ||
|
|
||
| exit /b %ERRORLEVEL% |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| <# | ||
| .SYNOPSIS | ||
| Run tests from a specific .testdef file | ||
|
|
||
| .DESCRIPTION | ||
| The RunTests script takes a testdef filename (without extension) and runs the tests defined in that | ||
| specific .testdef file. This is useful for development when you want to run tests for a specific | ||
| component rather than all tests. | ||
|
|
||
| The script searches for the specified .testdef file in the BuildOutput directory structure and | ||
| executes the tests according to the testdef configuration. | ||
|
|
||
| .PARAMETER TestDef | ||
| The name of the .testdef file to run (without the .testdef extension) | ||
| Example: "Deployment" to run "Deployment.testdef" | ||
|
|
||
| .PARAMETER OutputFolder | ||
| Set the base folder for the script to look for testdefs. Default: BuildOutput | ||
|
|
||
| .PARAMETER Platform | ||
| Only run tests for the selected platform (x86, x64, arm64). Default: x64 | ||
|
|
||
| .PARAMETER Configuration | ||
| Only run tests for the selected configuration (Debug, Release). Default: Release | ||
|
|
||
| .PARAMETER RunDisabled | ||
| If set, will run tests that are marked as "Disabled" in the testdef file. Default: false | ||
|
|
||
| .EXAMPLE | ||
| .\RunTests.ps1 -TestDef "DeploymentTests" | ||
|
|
||
| .EXAMPLE | ||
| .\RunTests.ps1 -TestDef "Deployment" -Platform "x86" -Configuration "Debug" | ||
| #> | ||
|
|
||
| param( | ||
| [Parameter(Mandatory=$true)] | ||
| [string]$TestDef, | ||
|
||
|
|
||
| [Parameter(Mandatory=$false)] | ||
| [string]$OutputFolder = "BuildOutput", | ||
guimafelipe marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| [Parameter(Mandatory=$false)] | ||
| [string]$Platform = "x64", | ||
guimafelipe marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
guimafelipe marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| [Parameter(Mandatory=$false)] | ||
guimafelipe marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| [string]$Configuration = "Release", | ||
guimafelipe marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| [Parameter(Mandatory=$false)] | ||
| [switch]$RunDisabled = $false | ||
| ) | ||
|
|
||
| function Find-TestDefFile | ||
| { | ||
| param($testDefName) | ||
|
|
||
| $configPlat = Join-Path $Configuration $Platform | ||
| $scriptParent = Split-Path -parent $PSScriptRoot | ||
| $OutputFolder = Join-Path $scriptParent $OutputFolder | ||
| $outputFolderPath = Join-Path $OutputFolder $configPlat | ||
| $testDefFileName = "$testDefName.testdef" | ||
| $testDefFile = Get-ChildItem -Recurse -Filter $testDefFileName $outputFolderPath -ErrorAction SilentlyContinue | Select-Object -First 1 | ||
|
|
||
| if ($null -eq $testDefFile) | ||
| { | ||
| Write-Error "Could not find testdef file '$testDefFileName' in '$outputFolderPath'" | ||
| Exit 1 | ||
| } | ||
|
|
||
| Write-Host "Found testdef file: $($testDefFile.FullName)" | ||
| return $testDefFile | ||
| } | ||
|
|
||
| function Get-Tests | ||
| { | ||
| param($testDefFile) | ||
|
|
||
| $tests = @() | ||
| $testJson = Get-Content -Raw $testDefFile.FullName | ConvertFrom-Json | ||
|
|
||
| $count = 0 | ||
| $baseId = $testDefFile.BaseName | ||
| foreach ($testConfig in $testJson.Tests) | ||
| { | ||
| $testConfig | Write-Host | ||
| if ($testConfig -contains 'Type') | ||
| { | ||
| $testType = $testConfig.Type | ||
| } | ||
| else | ||
| { | ||
| $testType = 'TAEF' | ||
| } | ||
|
|
||
| $id = $baseId + "-Test$count" | ||
| $t = [PSCustomObject]@{} | ||
| $t | Add-Member -MemberType NoteProperty -Name 'Test' -Value $id | ||
| $t | Add-Member -MemberType NoteProperty -Name 'Description' -Value $testConfig.Description | ||
| $t | Add-Member -MemberType NoteProperty -Name 'Filename' -Value $testConfig.Filename | ||
| $t | Add-Member -MemberType NoteProperty -Name 'Parameters' -Value $testConfig.Parameters | ||
| $t | Add-Member -MemberType NoteProperty -Name 'Architectures' -Value $testConfig.Architectures | ||
| $t | Add-Member -MemberType NoteProperty -Name 'Status' -Value $testConfig.Status | ||
| $t | Add-Member -MemberType NoteProperty -Name 'TestDef' -Value $testDefFile.FullName | ||
| $t | Add-Member -MemberType NoteProperty -Name 'Type' -Value $testType | ||
|
|
||
| $tests += $t | ||
| $count += 1 | ||
| } | ||
|
|
||
| $tests | ||
| } | ||
|
|
||
| function List-Tests | ||
| { | ||
| param($testDefFile) | ||
|
|
||
| $tests = Get-Tests $testDefFile | ||
| Write-Host "Tests in $($testDefFile.Name):" | ||
| $tests | Sort-Object -Property Test | Format-Table Test,Description,Type,Filename,Parameters,Architectures,Status -AutoSize | Out-String -Width 512 | ||
| } | ||
|
|
||
| function Run-TaefTest | ||
| { | ||
| param($test) | ||
|
|
||
| $testFolder = Split-Path -parent $test.TestDef | ||
| $tePath = Join-Path $testFolder "te.exe" | ||
| $dllFile = Join-Path $testFolder $test.Filename | ||
|
|
||
| & $tePath $dllFile $test.Parameters | ||
| } | ||
|
|
||
| function Run-Tests | ||
| { | ||
| param($testDefFile) | ||
|
|
||
| $tests = Get-Tests $testDefFile | ||
| Write-Host "Running tests from $($testDefFile.Name)..." | ||
|
|
||
| foreach ($test in $tests) | ||
| { | ||
| Write-Host "" | ||
| Write-Host "$($test.Filename) - $($test.Description)" | ||
| Write-Host "" | ||
|
|
||
| $validPlatform = $test.Architectures.Contains($Platform) | ||
| $testEnabled = $test.Status -eq "Enabled" | ||
|
|
||
| if ($validPlatform -and ($testEnabled -or $RunDisabled)) | ||
| { | ||
| Write-Host "Running test for platform $Platform..." | ||
| if ($test.Type -eq 'TAEF') | ||
| { | ||
| Run-TaefTest $test | ||
| } | ||
| elseif ($test.Type -eq 'Powershell') | ||
| { | ||
| Write-Host "Powershell tests not supported." | ||
| Exit 1 | ||
| } | ||
| else | ||
| { | ||
| Write-Host "Unknown test type '$($test.Type)'. Not running." | ||
| Exit 1 | ||
| } | ||
| } | ||
| elseif (-not($validPlatform)) | ||
| { | ||
| Write-Host "$Platform not listed in supported architectures: $($test.Architectures -join ', ')" | ||
| } | ||
| elseif (-not($testEnabled)) | ||
| { | ||
| Write-Host "Test is disabled. Not running." | ||
| } | ||
| } | ||
| } | ||
|
|
||
| Write-Host "RunTests.ps1 - Running tests for testdef: $TestDef" | ||
| Write-Host "Configuration: $Configuration, Platform: $Platform" | ||
| Write-Host "" | ||
|
|
||
| $StartTime = Get-Date | ||
|
|
||
| # Find the testdef file | ||
| $testDefFile = Find-TestDefFile $TestDef | ||
|
|
||
| List-Tests $testDefFile | ||
| Run-Tests $testDefFile | ||
|
|
||
| $TotalTime = (Get-Date)-$StartTime | ||
| $TotalMinutes = $TotalTime.Minutes | ||
| $TotalSeconds = $TotalTime.Seconds | ||
| Write-Host "" | ||
| Write-Host "Total Running Time: $TotalMinutes minutes and $TotalSeconds seconds" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's wrong with
TestAll.ps1(in root of the repository)?If that lacks functionality (e.g. run explicit subset of tests) why not improve
TestAll?Or perhaps fork it to
Test.cmdandtools\Test.ps1and change\TestAll.cmdto runTest.cmd ...options for existing TestAll.cmd behavior...There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used
TestAll.ps1as base to create this script. The idea for this one is to be a tool for developers to quickly iterate on testing based on what I was doing repetitively. I did not want this to merge with whatTestAll.ps1is doing as this later one is used on our build pipelines and requires way more stuff (log files and .wprp for example) than what is intended for a quick local test run to check if some changes broke anything.My usual loop was to call MSBuild locally to build the tests, and then go to
BuildOutputand run the tests. Now thinking back again, I realize it might be better to also include the test building in this script, as it makes sense we always would want to build and test together.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know about that. Depends if every test run requires build. I find that's more of a mixed bag. Sometimes I run a test multiple times between changing any code
Also, don't assume dev inner loop doesn't care about log files or .wprp.
Sounds like you're not reinventing TestAll.ps1 so much as BuildAll.ps1
It would be better to enhance those, or perhaps refactor the underlying plumbing from those with lots of options for dev inner-loop use (DoBuild.ps1, DoTest.ps1) and change those to call the new scripts with options to preserve their current behavior
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the script to receive more parameters and filtering based on the .testdef properties. Also, made the build step optional.
For making the
TestAll.ps1and theBuildAll.ps1scripts to share code with this one, it felt it should come in a separate pull request as this one would be way too much. I would like to hear your opinions on this and if it is better to proceed with this one being a separated change after some clean up.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I refactored the test code into a
Tests.psm1module and nowTestAll.ps1is using it