Skip to content

Commit 2e3fb3d

Browse files
danieljurekdanieljurek
authored
[Ready for EngSys Review] ESRP Publishing (Azure#3187)
> [!WARNING] > SERVICE OWNERS: no need to review this PR. I'm making engsys changes to ci.yml files. Really fixes Azure#3035 [keeping as draft to avoid notifying everyone] A new release crate-packing strategy for Rust which uses `cargo publish --dry-run` to produce the crates: ``` cargo publish --dry-run --package <package1> --package <package2> ... ``` This uses existing tooling to do the build with no vendoring or separate packing. Release order sorting can be read from the tool output and doesn't need to be computed from metadata. `cargo publish` also verifies that dependencies defined in the package's `Cargo.toml` are published and fails if they are not published. This also supports the current Pre-GA approach of not incrementing core's versions after release as the workspace dependency includes For example: If a package depends on `[email protected]` and there is no `[email protected]` published on crates.io, the command will fail. It's possible to "atomically" generate packages for releases with unreleased dependencies if those dependencies are included in the package command (e.g. `storage_blobs` depending on `storage_common`) ## Formatting: Formatting attempts to match reasonable outputs. Redirecting stderr from `cargo publish` would normally result in red console text. The `cargo publish` command is also grouped: https://dev.azure.com/azure-sdk/public/_build/results?buildId=5461494&view=logs&j=b766ebde-1fdb-5f11-1350-46ddc53b23cf&t=7ef29fa0-f532-5653-3473-16dc04608431&l=104 <img width="678" height="188" alt="image" src="https://github.com/user-attachments/assets/33156db7-4567-4259-b69b-eb1720089ea5" /> ## Tests: | Scenario | Expected Outcome | Links | | -------- | ---------------- | ---- | | Release azure_canary w/o required azure_canary_core (intra-service dependency) | Fail | https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5460934&view=results | | Release azure_canary and azure_canary_core (intra-service dependency) | Success | https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5460932&view=results | | Release azure_canary_core | Success | https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5460854&view=results | | Release azure_canary_core which depends on **unreleased** azure_core | Fail | https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5460945&view=results | | Release azure_canary_core which depends on released azure_core | Success | https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5460949&view=results | | Manually queued live tests (no release) for Key Vault | Success | https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5460954&view=results | | PR pipeline works | Success | https://dev.azure.com/azure-sdk/public/_build/results?buildId=5460931&view=results |
1 parent de5bf3b commit 2e3fb3d

File tree

13 files changed

+376
-252
lines changed

13 files changed

+376
-252
lines changed

‎eng/pipelines/templates/jobs/pack.yml‎

Lines changed: 46 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -55,15 +55,52 @@ jobs:
5555
ServiceDirectory: ${{ parameters.ServiceDirectory }}
5656
PackageInfoDirectory: $(Build.ArtifactStagingDirectory)/PackageInfo
5757

58-
- task: Powershell@2
59-
displayName: "Pack Crates"
60-
condition: and(succeeded(), ne(variables['NoPackagesChanged'],'true'))
61-
inputs:
62-
pwsh: true
63-
filePath: $(Build.SourcesDirectory)/eng/scripts/Pack-Crates.ps1
64-
arguments: >
65-
-OutputPath '$(Build.ArtifactStagingDirectory)'
66-
-PackageInfoDirectory '$(Build.ArtifactStagingDirectory)/PackageInfo'
58+
- ${{ if eq('auto', parameters.ServiceDirectory) }}:
59+
- task: Powershell@2
60+
displayName: Pack Crates
61+
condition: and(succeeded(), ne(variables['NoPackagesChanged'],'true'))
62+
inputs:
63+
pwsh: true
64+
filePath: $(Build.SourcesDirectory)/eng/scripts/Pack-Crates.ps1
65+
arguments: >
66+
-OutputPath '$(Build.ArtifactStagingDirectory)'
67+
-PackageInfoDirectory '$(Build.ArtifactStagingDirectory)/PackageInfo'
68+
69+
- ${{ else }}:
70+
- pwsh: |
71+
$artifacts = '${{ convertToJson(parameters.Artifacts) }}' | ConvertFrom-Json
72+
$isReleaseBuild = $true
73+
$artifactsToBuild = $artifacts | Where-Object { $_.releaseInBatch -eq 'True' }
74+
75+
if (!$artifactsToBuild) {
76+
Write-Host "No packages to release. Building all packages in the service directory with no dependency validation."
77+
$artifactsToBuild = $artifacts
78+
$isReleaseBuild = $false
79+
}
80+
81+
$packageNames = $artifactsToBuild.name
82+
83+
Write-Host "##vso[task.setvariable variable=PackageNames]$($packageNames -join ',')"
84+
if ($isReleaseBuild) {
85+
Write-Host "##vso[task.setvariable variable=AdditionalPackageParams]-Release"
86+
} else {
87+
Write-Host "##vso[task.setvariable variable=AdditionalPackageParams]"
88+
}
89+
displayName: Configure crate packing
90+
condition: and(succeeded(), ne(variables['NoPackagesChanged'],'true'))
91+
92+
- task: Powershell@2
93+
displayName: Pack Crates
94+
condition: and(succeeded(), ne(variables['NoPackagesChanged'],'true'))
95+
inputs:
96+
pwsh: true
97+
filePath: $(Build.SourcesDirectory)/eng/scripts/Pack-Crates.ps1
98+
arguments: >
99+
-OutputPath '$(Build.ArtifactStagingDirectory)'
100+
-PackageNames $(PackageNames)
101+
-OutBuildOrderFile '$(Build.ArtifactStagingDirectory)/release-order.json'
102+
$(AdditionalPackageParams)
103+
67104
68105
- template: /eng/common/pipelines/templates/steps/publish-1es-artifact.yml
69106
parameters:

‎eng/pipelines/templates/stages/archetype-rust-release.yml‎

Lines changed: 103 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,13 @@ parameters:
1414
- name: DevFeedName
1515
type: string
1616
default: 'public/azure-sdk-for-rust'
17-
- name: Environment
18-
type: string
19-
default: 'cratesio'
2017

2118
stages:
2219
- ${{ if eq(variables['System.TeamProject'], 'internal') }}:
2320
- ${{ if in(variables['Build.Reason'], 'Manual', '') }}:
24-
- ${{ each artifact in parameters.Artifacts }}:
25-
- stage: Release_${{artifact.safeName}}
26-
displayName: "Release: ${{artifact.name}}"
21+
- ${{ if gt(length(parameters.Artifacts), 0) }}:
22+
- stage: Release_Batch
23+
displayName: "Releasing: ${{length(parameters.Artifacts)}} crates"
2724
dependsOn: ${{parameters.DependsOn}}
2825
condition: and(succeeded(), ne(variables['SetDevVersion'], 'true'), ne(variables['Skip.Release'], 'true'), ne(variables['Build.Repository.Name'], 'Azure/azure-sdk-for-rust-pr'))
2926
variables:
@@ -50,16 +47,17 @@ stages:
5047

5148
- template: /eng/common/pipelines/templates/steps/retain-run.yml
5249

53-
- script: |
54-
echo "##vso[build.addbuildtag]${{artifact.name}}"
55-
displayName: Add build tag '${{artifact.name}}'
50+
- ${{ each artifact in parameters.Artifacts }}:
51+
- script: |
52+
echo "##vso[build.addbuildtag]${{artifact.name}}"
53+
displayName: Add build tag '${{artifact.name}}'
5654
57-
- template: /eng/common/pipelines/templates/steps/create-tags-and-git-release.yml
58-
parameters:
59-
ArtifactLocation: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/${{artifact.name}}
60-
PackageRepository: Crates.io
61-
ReleaseSha: $(Build.SourceVersion)
62-
WorkingDirectory: $(Pipeline.Workspace)/_work
55+
- template: /eng/common/pipelines/templates/steps/create-tags-and-git-release.yml
56+
parameters:
57+
ArtifactLocation: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/${{artifact.name}}
58+
PackageRepository: Crates.io
59+
ReleaseSha: $(Build.SourceVersion)
60+
WorkingDirectory: $(Pipeline.Workspace)/_work
6361

6462
- deployment: PublishPackage
6563
displayName: "Publish to Crates.io"
@@ -71,7 +69,10 @@ stages:
7169
- input: pipelineArtifact # Required, type of the input artifact
7270
artifactName: ${{parameters.PipelineArtifactName}} # Required, name of the pipeline artifact
7371
targetPath: $(Pipeline.Workspace)/drop # Optional, specifies where the artifact is downloaded to
74-
environment: ${{parameters.Environment}}
72+
${{if parameters.TestPipeline}}:
73+
environment: none
74+
${{else}}:
75+
environment: package-publish
7576
# This timeout shouldn't be necessary once we're able to parallelize better. Right now,
7677
# this is here to ensure larger areas (30+) libraries don't time out.
7778
timeoutInMinutes: 120
@@ -84,33 +85,71 @@ stages:
8485
runOnce:
8586
deploy:
8687
steps:
87-
- template: /eng/pipelines/templates/steps/use-rust.yml@self
88-
parameters:
89-
Toolchain: stable
90-
91-
- pwsh: |
92-
$additionalOwners = @('heaths', 'hallipr')
93-
$token = $env:CARGO_REGISTRY_TOKEN
94-
$crateName = '${{artifact.name}}'
95-
96-
$manifestPath = "$(Pipeline.Workspace)/drop/$crateName/contents/Cargo.toml"
97-
Write-Host "> cargo publish --manifest-path `"$manifestPath`""
98-
cargo publish --manifest-path $manifestPath
99-
if (!$?) {
100-
Write-Error "Failed to publish package: '$crateName'"
101-
exit 1
102-
}
103-
104-
$existingOwners = (cargo owner --list $crateName) -replace " \(.*", ""
105-
$missingOwners = $additionalOwners | Where-Object { $existingOwners -notcontains $_ }
106-
107-
foreach ($owner in $missingOwners) {
108-
Write-Host "> cargo owner --add $owner $crateName"
109-
cargo owner --add $owner $crateName
110-
}
111-
displayName: Publish Crate
112-
env:
113-
CARGO_REGISTRY_TOKEN: $(azure-sdk-cratesio-token)
88+
# This loop over artifacts is used to produce the correct number
89+
# of ESRP release tasks. It has the side effect of also setting
90+
# the artifact name by looking up the index of the current
91+
# "artifact.name" in the parameters.Artifacts array, using that
92+
# as an "index" and then using that same index to look up the
93+
# actual artifact to release in the release-order.json file.
94+
- ${{ each artifact in parameters.Artifacts }}:
95+
- pwsh: |
96+
# From the DevOps template artifact loop calculate the current index
97+
$indexItem = '${{ artifact.name }}'
98+
[array] $indexList = ConvertFrom-Json '${{ convertToJson(parameters.Artifacts.*.name) }}'
99+
$index = $indexList.IndexOf($indexItem)
100+
Write-Host "Index of template artifact: $index"
101+
102+
[array] $artifacts = Get-Content '$(Pipeline.Workspace)/drop/release-order.json' | ConvertFrom-Json
103+
104+
$artifactName = $artifacts[$index]
105+
106+
Write-Host "Releasing artifact: $artifactName"
107+
108+
$artifactRootPath = '$(Pipeline.Workspace)/drop'
109+
$outDir = '$(Pipeline.Workspace)/esrp-release'
110+
111+
if (Test-Path $outDir) {
112+
Write-Host "Cleaning output directory: $outDir"
113+
Remove-Item -Path $outDir -Recurse -Force
114+
}
115+
New-Item -ItemType Directory -Path $outDir -Force | Out-Null
116+
117+
$packageMetadataPath = "$artifactRootPath/PackageInfo/$artifactName.json"
118+
if (!(Test-Path $packageMetadataPath)) {
119+
Write-Error "Package metadata file not found: $packageMetadataPath"
120+
exit 1
121+
}
122+
123+
$packageMetadata = Get-Content -Raw $packageMetadataPath | ConvertFrom-Json
124+
$packageVersion = $packageMetadata.version
125+
Write-Host "Package version: $packageVersion"
126+
127+
$cratePath = "$artifactRootPath/$artifactName/$artifactName-$packageVersion.crate"
128+
Copy-Item `
129+
-Path $cratePath `
130+
-Destination $outDir
131+
Write-Host "Contents of $outDir"
132+
Get-ChildItem -Path $outDir | ForEach-Object { Write-Host $_.FullName }
133+
displayName: 'Copy crate for ESRP'
134+
135+
- task: EsrpRelease@10
136+
displayName: 'ESRP Release'
137+
inputs:
138+
connectedservicename: 'Azure SDK PME Managed Identity'
139+
ClientId: '5f81938c-2544-4f1f-9251-dd9de5b8a81b'
140+
DomainTenantId: '975f013f-7f24-47e8-a7d3-abc4752bf346'
141+
Usemanagedidentity: true
142+
KeyVaultName: 'kv-azuresdk-codesign'
143+
SignCertName: 'azure-sdk-esrp-release-certificate'
144+
intent: 'packagedistribution'
145+
contenttype: 'Rust'
146+
contentsource: 'Folder'
147+
folderlocation: '$(Pipeline.Workspace)/esrp-release'
148+
waitforreleasecompletion: true
149+
owners: ${{ coalesce(variables['Build.RequestedForEmail'], '[email protected]') }}
150+
approvers: ${{ coalesce(variables['Build.RequestedForEmail'], '[email protected]') }}
151+
serviceendpointurl: 'https://api.esrp.microsoft.com/'
152+
mainpublisher: 'ESRPRELPACMANTEST'
114153

115154
- job: UpdatePackageVersion
116155
displayName: "API Review and Package Version Update"
@@ -130,69 +169,32 @@ stages:
130169
displayName: Download ${{parameters.PipelineArtifactName}} artifact
131170
artifact: ${{parameters.PipelineArtifactName}}
132171

133-
- template: /eng/common/pipelines/templates/steps/create-apireview.yml
134-
parameters:
135-
ArtifactPath: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}
136-
Artifacts: ${{parameters.Artifacts}}
137-
ConfigFileDir: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/PackageInfo
138-
MarkPackageAsShipped: true
139-
ArtifactName: ${{parameters.PipelineArtifactName}}
140-
SourceRootPath: $(System.DefaultWorkingDirectory)
141-
PackageName: ${{artifact.name}}
142-
143-
# Apply the version increment to each library, which updates the Cargo.toml and changelog files.
144-
- task: PowerShell@2
145-
displayName: Increment ${{artifact.name}} version
146-
inputs:
147-
targetType: filePath
148-
filePath: $(Build.SourcesDirectory)/eng/scripts/Update-PackageVersion.ps1
149-
arguments: >
150-
-ServiceDirectory '${{parameters.ServiceDirectory}}'
151-
-PackageName '${{artifact.name}}'
172+
- ${{ each artifact in parameters.Artifacts }}:
173+
- template: /eng/common/pipelines/templates/steps/create-apireview.yml
174+
parameters:
175+
ArtifactPath: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}
176+
Artifacts: ${{parameters.Artifacts}}
177+
ConfigFileDir: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/PackageInfo
178+
MarkPackageAsShipped: true
179+
ArtifactName: ${{parameters.PipelineArtifactName}}
180+
SourceRootPath: $(System.DefaultWorkingDirectory)
181+
PackageName: ${{artifact.name}}
182+
183+
# Apply the version increment to each library, which updates the Cargo.toml and changelog files.
184+
- task: PowerShell@2
185+
displayName: Increment ${{artifact.name}} version
186+
inputs:
187+
targetType: filePath
188+
filePath: $(Build.SourcesDirectory)/eng/scripts/Update-PackageVersion.ps1
189+
arguments: >
190+
-ServiceDirectory '${{parameters.ServiceDirectory}}'
191+
-PackageName '${{artifact.name}}'
152192
153193
- template: /eng/common/pipelines/templates/steps/create-pull-request.yml
154194
parameters:
155195
PRBranchName: increment-package-version-${{parameters.ServiceDirectory}}-$(Build.BuildId)
156-
CommitMsg: "Increment package version after release of ${{ artifact.name }}"
196+
CommitMsg: "Increment package version after release of ${{ join(', ', parameters.Artifacts.*.name) }}"
157197
PRTitle: "Increment versions for ${{parameters.ServiceDirectory}} releases"
158198
CloseAfterOpenForTesting: '${{parameters.TestPipeline}}'
159199
${{ if startsWith(variables['Build.SourceBranch'], 'refs/pull/') }}:
160200
BaseBranchName: main
161-
162-
- ${{ if eq(parameters.TestPipeline, true) }}:
163-
- job: ManualApproval
164-
displayName: "Manual approval"
165-
dependsOn: PublishPackage
166-
condition: ne(variables['Skip.PublishPackage'], 'true')
167-
pool: server
168-
timeoutInMinutes: 120 # 2 hours
169-
steps:
170-
- task: ManualValidation@1
171-
timeoutInMinutes: 60 # 1 hour
172-
inputs:
173-
notifyUsers: '' # Required, but empty string allowed
174-
allowApproversToApproveTheirOwnRuns: true
175-
instructions: "Approve yank of ${{ artifact.name }}"
176-
onTimeout: 'resume'
177-
178-
- job: YankCrates
179-
displayName: "Yank Crates"
180-
dependsOn: ManualApproval
181-
condition: and(succeeded(), ne(variables['Skip.PublishPackage'], 'true'))
182-
steps:
183-
- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
184-
185-
- download: current
186-
displayName: Download ${{parameters.PipelineArtifactName}} artifact
187-
artifact: ${{parameters.PipelineArtifactName}}
188-
189-
- task: PowerShell@2
190-
displayName: Yank Crates
191-
env:
192-
CARGO_REGISTRY_TOKEN: $(azure-sdk-cratesio-token)
193-
inputs:
194-
targetType: filePath
195-
filePath: $(Build.SourcesDirectory)/eng/scripts/Yank-Crates.ps1
196-
arguments:
197-
-CrateNames '${{artifact.name}}'
198-
-PackageInfoDirectory '$(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/PackageInfo'

‎eng/pipelines/templates/stages/archetype-sdk-client.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,9 @@ extends:
148148
parameters:
149149
DependsOn: "Build"
150150
ServiceDirectory: ${{ parameters.ServiceDirectory }}
151-
Artifacts: ${{ parameters.Artifacts }}
151+
Artifacts:
152+
- ${{ each artifact in parameters.Artifacts }}:
153+
- ${{ if ne(artifact.releaseInBatch, 'false')}}:
154+
- ${{ artifact }}
152155
TestPipeline: ${{ eq(parameters.ServiceDirectory, 'canary') }}
153156
PipelineArtifactName: packages

‎eng/scripts/Language-Settings.ps1‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
$Language = "rust"
22
$LanguageDisplayName = "Rust"
33
$PackageRepository = "crates.io"
4-
$packagePattern = "Cargo.toml"
4+
$packagePattern = "*.crate"
55
#$MetadataUri = "https://raw.githubusercontent.com/Azure/azure-sdk/main/_data/releases/latest/rust-packages.csv"
66
$GithubUri = "https://github.com/Azure/azure-sdk-for-rust"
77
$PackageRepositoryUri = "https://crates.io/crates"
@@ -139,15 +139,32 @@ function Get-rust-AdditionalValidationPackagesFromPackageSet ($packagesWithChang
139139
return $additionalPackages ?? @()
140140
}
141141

142+
# $GetPackageInfoFromPackageFileFn = "Get-${Language}-PackageInfoFromPackageFile"
142143
function Get-rust-PackageInfoFromPackageFile([IO.FileInfo]$pkg, [string]$workingDirectory) {
143-
#$pkg will be a FileInfo object for the Cargo.toml file in a package artifact directory
144-
$package = cargo read-manifest --manifest-path $pkg.FullName | ConvertFrom-Json
144+
# Create a temporary folder for extraction
145+
$extractionPath = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), [System.IO.Path]::GetRandomFileName())
146+
New-Item -ItemType Directory -Path $extractionPath | Out-Null
147+
148+
# Extract the .crate file (which is a tarball) to the temporary folder
149+
tar -xvf $pkg.FullName -C $extractionPath
150+
$cargoTomlPath = [System.IO.Path]::Combine($extractionPath, $pkg.BaseName, 'Cargo.toml')
151+
152+
Write-Host "Reading package info from $cargoTomlPath"
153+
if (!(Test-Path $cargoTomlPath)) {
154+
$message = "The Cargo.toml file was not found in the package artifact at $cargoTomlPath"
155+
LogError $message
156+
throw $message
157+
}
158+
159+
$package = cargo read-manifest --manifest-path $cargoTomlPath | ConvertFrom-Json
145160

146161
$packageName = $package.name
147162
$packageVersion = $package.version
148163

149-
$changeLogLoc = Get-ChildItem -Path $pkg.DirectoryName -Filter "CHANGELOG.md" | Select-Object -First 1
150-
$readmeContentLoc = Get-ChildItem -Path $pkg.DirectoryName -Filter "README.md" | Select-Object -First 1
164+
$packageAssetPath = [System.IO.Path]::Combine($extractionPath, "$packageName-$packageVersion")
165+
166+
$changeLogLoc = Get-ChildItem -Path $packageAssetPath -Filter "CHANGELOG.md" | Select-Object -First 1
167+
$readmeContentLoc = Get-ChildItem -Path $packageAssetPath -Filter "README.md" | Select-Object -First 1
151168

152169
if ($changeLogLoc) {
153170
$releaseNotes = Get-ChangeLogEntryAsString -ChangeLogLocation $changeLogLoc -VersionString $packageVersion

0 commit comments

Comments
 (0)