Run installer tests in pipeline #47313
There are no files selected for viewing
|
Member
There was a problem hiding this comment. I was expecting to see tests listed under the tests tab of the build you linked to - https://dev.azure.com/dnceng/internal/_build/results?buildId=2655424&view=ms.vss-test-web.build-test-results-tab Also do you happen to have build where there is a failure? I am curious what the UX is in this case on seeing what scenario test failed.
Member
Author
There was a problem hiding this comment. Hmm, interesting - will check what's going on. I do not have the pipeline run, with failing scenario tests. I could share the output of the local run.
Member
Author
There was a problem hiding this comment. Fixed with defc5d2 This will still, likely, get refactored further to utilize shared
Member
Author
There was a problem hiding this comment. There was an issue with search folder so tests weren't being published. This is now fixed with 08f3e6b Verification build with tests showing up correctly: https://dev.azure.com/dnceng/internal/_build/results?buildId=2659027&view=ms.vss-test-web.build-test-results-tab
Member
There was a problem hiding this comment. It doesn't look like the capture the inner scenario test log file. I'm curious how difficult it will be to diagnose failures when there is a scenario test failure. I am fine with adjusting as needed if this proves to be a problem. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| parameters: | ||
| - name: continueOnError | ||
| type: boolean | ||
| default: false | ||
|
|
||
| - name: targetArchitecture | ||
| type: string | ||
| default: '' | ||
|
|
||
| - name: OS | ||
| type: string | ||
| values: | ||
| - Windows_NT | ||
| - Linux | ||
| - Darwin | ||
|
|
||
| steps: | ||
| - task: DownloadBuildArtifacts@1 | ||
| inputs: | ||
| artifactName: BlobArtifacts | ||
| downloadPath: $(Build.ArtifactStagingDirectory) | ||
| checkDownloadedFiles: true | ||
| displayName: Download Blob Artifacts | ||
|
|
||
| - task: DownloadBuildArtifacts@1 | ||
| inputs: | ||
| artifactName: PackageArtifacts | ||
| downloadPath: $(Build.ArtifactStagingDirectory) | ||
| checkDownloadedFiles: true | ||
| displayName: Download Package Artifacts | ||
|
|
||
| # This is necessary whenever we want to publish/restore to an AzDO private feed | ||
| # Since sdk-task.ps1 tries to restore packages we need to do this authentication here | ||
| # otherwise it'll complain about accessing a private feed. | ||
| - task: NuGetAuthenticate@1 | ||
| displayName: 'Authenticate to AzDO Feeds' | ||
|
|
||
| - ${{ if eq(parameters.OS, 'Linux') }}: | ||
| - script: | | ||
| extraBuildProperties="/p:TestRpmPackages=true" | ||
| if [[ '${{ parameters.targetArchitecture }}' == 'x64' ]]; then | ||
| # At the moment Deb packages are only available for x64 | ||
| extraBuildProperties="$extraBuildProperties /p:TestDebPackages=true" | ||
| fi | ||
| ./build.sh \ | ||
|
Member
There was a problem hiding this comment. This additional entry point invocation worries me. The VMR is supposed to receive properties like TargetOS, TargetArchitecture, ContinuousIntegrationBuild, etc that help us distinguish different OSs, architectures, CI vs dev, etc. Could this yml import the steps/vmr-build.yml which handles all that?
Member
Author
There was a problem hiding this comment. That would be ideal. But, we don't currently use that for running tests - the only such instance also has its own custom invocation: sdk/eng/pipelines/templates/jobs/vmr-build.yml Lines 421 to 432 in d833ffb I'll think some more about this.
Member
Author
There was a problem hiding this comment. For this to work I would use This would also, likely, fix the issue with missing test status, as test logs are currently not uploaded from this job. The alternative is to add test-log publishing to the new job.
Member
There was a problem hiding this comment.
Right, we need the additional invocation as we can't build and run tests from a single invocation today. It also better separates the actions so I don't see that as a problem. That additional invocation in vmr-build.yml is kept in sync with the build invocation so that isn't a problem. It receives the properties that the build invocation receives. If it doesn't then that's a bug. My feedback here was around not introducing another YML entry point for the VMR. vmr-build.yml should be able to handle the different scenarios (building and testing today). The vmr-final-join has its own entry point and I think that should also get consolidated eventually.
Member
Author
There was a problem hiding this comment. Yeah, this makes sense. I will add the test-only feature to
Member
Author
There was a problem hiding this comment. OK. Three ways to refactor this. Option 1 is for steps template to run just one build steps - either regular build or tests. It seems clunky, as we'd need to supply some property that specifies the build type, regular or test. Based on such property (or properties) we'd only do regular or test build. This way we can share a single entry point across all VMR pipelines. Option 2 would run build and test (if requested). This implies that we'd also need to run any steps that are executed between main and test builds, i.e. sdk/eng/pipelines/templates/jobs/vmr-build.yml Lines 414 to 419 in d956fb2 sdk/eng/pipelines/templates/jobs/vmr-build.yml Lines 556 to 561 in d956fb2 Option 3 would move all of these steps to the new build steps yml: sdk/eng/pipelines/templates/jobs/vmr-build.yml Lines 383 to 621 in d956fb2
Member
There was a problem hiding this comment. Option 3 sounds best but I'm not sure what others think. cc @mmitche
Member
There was a problem hiding this comment. I think I am missing the difference between Option 1 and 3 here.
Member
Author
There was a problem hiding this comment. Option 1 would create a very small yml with a single step to invoke build.cmd or build.sh with various arguments. All conditioning, prep, and all other steps, like installing Node would stay in jobs/vmr-build.yml. This option would require a new parameter that would specify the build type (i.e. Option 3 would move the whole ~250 line block I linked above into the new steps yml. It keeps most of build and related steps together, and leaves the extra stuff like publishing in the current yml.
Member
Author
There was a problem hiding this comment. I will follow up with refactoring in a separate PR - still working out the issues.
Member
There was a problem hiding this comment. I think you are missing the
Member
Author
There was a problem hiding this comment. Oh, yeah, I don't use it locally, but other official legs/steps do use it - will fix, thanks. |
||
| -t \ | ||
| /p:TestInstallerPackages=true \ | ||
| /p:BlobArtifactsDir=$(Build.ArtifactStagingDirectory)/BlobArtifacts \ | ||
| /p:PackageArtifactsDir=$(Build.ArtifactStagingDirectory)/PackageArtifacts \ | ||
| $extraBuildProperties | ||
| displayName: Validate installer packages | ||
| workingDirectory: $(Build.SourcesDirectory) | ||
| continueOnError: ${{ parameters.continueOnError }} | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This timeout seems excessive. Looks like in your test run it took 20 minutes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'll adjust - thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed with d9895e4