You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/container-registry/container-registry-transfer-images.md
+85-70Lines changed: 85 additions & 70 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
1
---
2
-
title: Transfer images
3
-
description: Transfer images in bulk from one container registry to another registry by creating a transfer pipeline using Azure storage accounts
2
+
title: Transfer artifacts
3
+
description: Transfer collections of images or other artifacts from one container registry to another registry by creating a transfer pipeline using Azure storage accounts
4
4
ms.topic: article
5
-
ms.date: 04/09/2020
5
+
ms.date: 04/10/2020
6
6
ms.custom:
7
7
---
8
8
@@ -24,11 +24,11 @@ This feature is available in the **Premium** container registry service tier. Fo
24
24
25
25
## Prerequisites
26
26
27
-
***Container registries** - You need an existing source registry with artifacts to transfer, and a target registry. ACR transfer is intended for movement across physically disconnected clouds. However, for testing, the source and target registries can be in the same or a different Azure subscription, Active Directory tenant, or cloud. If you need to create a registry, see [Quickstart: Create a private container registry using the Azure CLI](container-registry-get-started-azure-cli.md).
27
+
***Container registries** - You need an existing source registry with artifacts to transfer, and a target registry. ACR transfer is intended for movement across physically disconnected clouds. For testing, the source and target registries can be in the same or a different Azure subscription, Active Directory tenant, or cloud. If you need to create a registry, see [Quickstart: Create a private container registry using the Azure CLI](container-registry-get-started-azure-cli.md).
28
28
***Storage accounts** - Create source and target storage accounts in a subscription and location of your choice. For testing purposes, you can use the same subscription or subscriptions as your source and target registries. For cross-cloud scenarios, typically you create a separate storage account in each cloud. If needed, create the storage accounts with the [Azure CLI](../storage/common/storage-account-create.md?tabs=azure-cli) or other tools.
29
29
30
-
In each account, create a blob container for artifact transfer. For example, create a container named *transfer*. Two or more ACR Transfer pipelines can share the same storage account, but should use different storage container scopes.
31
-
***Key vaults** - Create key vaults to store secrets in the same Azure subscription or subscriptions as your source and target registries. If needed, create source and target key vaults with the [Azure CLI](../key-vault/quick-create-cli.md) or other tools.
30
+
Create a blob container for artifact transfer in each account. For example, create a container named *transfer*. Two or more transfer pipelines can share the same storage account, but should use different storage container scopes.
31
+
***Key vaults** - Key vaults are needed to store SAS token secrets used to access source and target storage accounts. Create the source and target key vaults in the same Azure subscription or subscriptions as your source and target registries. If needed, create key vaults with the [Azure CLI](../key-vault/quick-create-cli.md) or other tools.
32
32
***Environment variables** - For example commands in this article, set the following environment variables for the source and target environments. All examples are formatted for the Bash shell.
33
33
```console
34
34
SOURCE_RG="<source-resource-group>"
@@ -41,13 +41,16 @@ This feature is available in the **Premium** container registry service tier. Fo
41
41
42
42
## Scenario overview
43
43
44
-
You create the following three resources for image transfer between registries. All are created using PUT operations. These resources operate on your *source* and *target* registries and storage accounts.
44
+
You create the following three pipeline resources for image transfer between registries. All are created using PUT operations. These resources operate on your *source* and *target* registries and storage accounts.
45
45
46
-
***[ExportPipeline](#create-the-exportpipeline-resource-with-azure-resource-manager)** - Long-lasting resource that contains high-level information about the *source* registry and storage account. This information includes the source storage blob container URI and the key vault secret URI of the storage SAS token.
47
-
***[ImportPipeline](#create-the-importpipeline-resource)** - Long-lasting resource that contains high-level information about the *target* registry and storage account. This information includes the target storage blob container URI and the key vault secret URI of the storage SAS token. An import trigger is enabled by default, so the pipeline runs automatically when artifacts land in the target storage container.
48
-
***[PipelineRun](#create-a-pipelinerun-with-resource-manager)** - Resource used to invoke either an ExportPipeline or ImportPipeline resource.
46
+
Storage authentication uses SAS tokens, managed as secrets in key vaults. The pipelines use managed identities to read the secrets in the vaults.
47
+
48
+
***[ExportPipeline](#create-exportpipeline-with-resource-manager)** - Long-lasting resource that contains high-level information about the *source* registry and storage account. This information includes the source storage blob container URI and the key vault secret URI of the storage SAS token.
49
+
***[ImportPipeline](#create-importpipeline-with-resource-manager)** - Long-lasting resource that contains high-level information about the *target* registry and storage account. This information includes the target storage blob container URI and the key vault secret URI of the storage SAS token. An import trigger is enabled by default, so the pipeline runs automatically when artifacts land in the target storage container.
50
+
***[PipelineRun](#create-pipelinerun-with-resource-manager)** - Resource used to invoke either an ExportPipeline or ImportPipeline resource.
49
51
* You run the ExportPipeline manually by creating a PipelineRun resource and specify the artifacts to export.
50
52
* If an import trigger is enabled, the ImportPipeline runs automatically. It can also be run manually using a PipelineRun.
53
+
* Currently a maximum of **10 artifacts** can be transferred with each PipelineRun.
51
54
52
55
### Things to know
53
56
* The ExportPipeline and ImportPipeline will typically be in different Active Directory tenants associated with the source and destination clouds. This scenario requires separate managed identities and key vaults for the export and import resources. For testing purposes, these resources can be placed in the same cloud, sharing identities.
@@ -61,12 +64,12 @@ Transfer uses shared access signature (SAS) tokens to access the designated stor
61
64
62
65
Run the [az storage container generate-sas][az-storage-container-generate-sas] command to generate a SAS token for the container in the source storage account, used for artifact export.
63
66
64
-
Generate the token with the following permissions: Read, Write, List, Add, Create.
In the following example, command output is assigned to the EXPORT_SAS environment variable. Update the `--expiry` value for your environment:
69
+
In the following example, command output is assigned to the EXPORT_SAS environment variable, prefixed with the '?' character. Update the `--expiry` value for your environment:
67
70
68
71
```azurecli
69
-
EXPORT_SAS=$(az storage container generate-sas \
72
+
EXPORT_SAS=?$(az storage container generate-sas \
70
73
--name transfer \
71
74
--account-name $SA_SOURCE \
72
75
--expiry 2020-05-01 \
@@ -100,12 +103,12 @@ EXPORT_KV_URI=$(az keyvault secret show \
100
103
101
104
Run the [az storage container generate-sas][az-storage-container-generate-sas] command to generate a SAS token for the container in the target storage account, used for artifact import.
102
105
103
-
Generate the token with the following permissions: Read, Delete, List
106
+
*Recommended token permissions*: Read, Delete, List
104
107
105
-
In the following example, command output is assigned to the IMPORT_SAS environment variable. Update the `--expiry` value for your environment:
108
+
In the following example, command output is assigned to the IMPORT_SAS environment variable, prefixed with the '?' character. Update the `--expiry` value for your environment:
106
109
107
110
```azurecli
108
-
IMPORT_SAS=$(az storage container generate-sas \
111
+
IMPORT_SAS=?$(az storage container generate-sas \
109
112
--name transfer \
110
113
--account-name $SA_TARGET \
111
114
--expiry 2020-05-01 \
@@ -189,7 +192,7 @@ az keyvault set-policy --name $TARGET_KV \
189
192
--secret-permissions get
190
193
```
191
194
192
-
## Create ExportPipeline resource with Resource Manager
195
+
## Create ExportPipeline with Resource Manager
193
196
194
197
Create an ExportPipeline resource for your source container registry using Azure Resource Manager template deployment. The ExportPipeline resource is provisioned with the source user-assigned identity you created in the previous section.
195
198
@@ -231,51 +234,7 @@ Take note of the resource ID (`id`) of the pipeline, which is used in later step
## Create PipelineRun resource with Resource Manager
235
-
236
-
Create a PipelineRun resource for your source container registry using Azure Resource Manager template deployment. This resource runs the ExportPipeline resource you created in the previous step, and exports specified artifacts from your container registry to your source storage account.
237
-
238
-
Copy PipelineRun Resource Manager [template files](https://github.com/Azure/acr/tree/master/docs/image-transfer/PipelineRun) to a local folder.
239
-
240
-
Enter the following parameter values in the file `azuredeploy.parameters.json`:
241
-
242
-
|Parameter |Value |
243
-
|---------|---------|
244
-
|registryName | Name of your source container registry |
245
-
|pipelineRunName | Name you choose for the run |
246
-
|pipelineResourceId | Resource ID of the export pipeline.<br/>Example: `/subscriptions/<subscriptionID>/resourceGroups/<resourceGroupName>/providers/Microsoft.ContainerRegistry/registries/<sourceRegistryName>/exportPipelines/myExportPipeline`|
247
-
|targetName | Name you choose for the artifacts blob exported to your source storage account, such as *myblob*
248
-
|artifacts | Array of source artifacts to transfer, as tags or manifest digests<br/>Example: `[samples/hello-world:v1", "samples/nginx:v1"]`
249
-
250
-
Run [az deployment group create][az-deployment-group-create] to create the PipelineRun resource. The following example names the deployment *exportPipelineRun*.
251
-
252
-
```azurecli
253
-
az deployment group create \
254
-
--resource-group $SOURCE_RG \
255
-
--template-file azuredeploy.json \
256
-
--name exportPipelineRun \
257
-
--parameters azuredeploy.parameters.json
258
-
```
259
-
260
-
> [!IMPORTANT]
261
-
> For artifact export, if prompted, leave the `sourceName` blank. For testing, you can also leave `catalogDigest` and `forceUpdateTag` values blank.
262
-
263
-
When deployment completes successfully, verify artifact export by viewing the exported blob in the *transfer* container of the source storage account.
264
-
265
-
## Transfer blob (optional)
266
-
267
-
Use the AzCopy command to [transfer blob data](../storage/common/storage-use-azcopy-blobs.md#copy-blobs-between-storage-accounts) from the source storage account to the target storage account.
268
-
269
-
For example, the following [`azcopy sync`](/azure/storage/common/storage-ref-azcopy-sync) command replicates the *transfer* container from the source storage account to the *transfer* container in the target account. Authentication uses SAS tokens with appropriate permissions for the source and target containers. (Steps to create tokens are not shown).
## Create ImportPipeline resource with Resource Manager
237
+
## Create ImportPipeline with Resource Manager
279
238
280
239
Create an ImportPipeline resource in your target container registry using Azure Resource Manager template deployment. The ImportPipeline resource is provisioned with the target user-assigned identity you created previously. By default, the pipeline is enabled to import automatically when the storage account in the target environment has an artifact blob.
281
240
@@ -290,7 +249,7 @@ Parameter |Value |
290
249
|sourceUri | URI of the storage container in your target environment (the source for the import pipeline).<br/>Example: `https://targetstorage.blob.core.windows.net/transfer/`|
291
250
|keyVaultUri | URI of the SAS token secret in the target key vault, stored previously in the IMPORT_KV_URI variable.<br/>Example: `https://targetvault.vault-int.azure-int.net/secrets/acrimportsas/xxxxxxxxxx`|
292
251
293
-
### Export options
252
+
### Import options
294
253
295
254
The `Options` property for the import pipeline supports optional boolean values. The following values are recommended:
296
255
@@ -323,6 +282,61 @@ It can take several minutes for artifacts to import. When the import completes s
323
282
az acr repository list --name <target-registry-name>
324
283
```
325
284
285
+
## Create PipelineRun with Resource Manager
286
+
287
+
Create a PipelineRun resource for your source container registry using Azure Resource Manager template deployment. This resource runs the ExportPipeline resource you created in the previous step, and exports specified artifacts from your container registry as a blob to your source storage account.
288
+
289
+
Copy PipelineRun Resource Manager [template files](https://github.com/Azure/acr/tree/master/docs/image-transfer/PipelineRun) to a local folder.
290
+
291
+
Enter the following parameter values in the file `azuredeploy.parameters.json`:
292
+
293
+
|Parameter |Value |
294
+
|---------|---------|
295
+
|registryName | Name of your source container registry |
296
+
|pipelineRunName | Name you choose for the run |
297
+
|pipelineResourceId | Resource ID of the export pipeline.<br/>Example: `/subscriptions/<subscriptionID>/resourceGroups/<resourceGroupName>/providers/Microsoft.ContainerRegistry/registries/<sourceRegistryName>/exportPipelines/myExportPipeline`|
298
+
|targetName | Name you choose for the artifacts blob exported to your source storage account, such as *myblob*
299
+
|artifacts | Array of source artifacts to transfer, as tags or manifest digests<br/>Example: `[samples/hello-world:v1", "samples/nginx:v1"]`
300
+
301
+
Run [az deployment group create][az-deployment-group-create] to create the PipelineRun resource. The following example names the deployment *exportPipelineRun*.
302
+
303
+
```azurecli
304
+
az deployment group create \
305
+
--resource-group $SOURCE_RG \
306
+
--template-file azuredeploy.json \
307
+
--name exportPipelineRun \
308
+
--parameters azuredeploy.parameters.json
309
+
```
310
+
311
+
> [!IMPORTANT]
312
+
> For artifact export, if prompted, leave the `sourceName` blank. For testing, you can also leave `catalogDigest` and `forceUpdateTag` values blank.
313
+
314
+
It can take several minutes for artifacts to export. When deployment completes successfully, verify artifact export by listing the exported blob in the *transfer* container of the source storage account. For example, run the [az storage blob list][az-storage-blob-list] command:
315
+
316
+
```azurecli
317
+
az storage blob list \
318
+
--account-name $SA_SOURCE
319
+
--container transfer
320
+
--output table
321
+
```
322
+
323
+
## Transfer blob (optional)
324
+
325
+
Use the AzCopy tool or other methods to [transfer blob data](../storage/common/storage-use-azcopy-blobs.md#copy-blobs-between-storage-accounts) from the source storage account to the target storage account.
326
+
327
+
For example, the following [`azcopy sync`](/azure/storage/common/storage-ref-azcopy-sync) command replicates the *transfer* container from the source storage account to the *transfer* container in the target account. Authentication uses SAS tokens with appropriate permissions for the source and target containers. (Steps to create tokens are not shown.)
If you enabled the `sourceTriggerStatus` parameter of the import pipeline (the default value), the import pipeline will trigger. Import may take place after several minutes.
337
+
338
+
If you didn't enable the `sourceTriggerStatus` parameter of the import pipeline, run the ImportPipeline resource manually, as shown in the following section.
339
+
326
340
### Run the ImportPipeline resource manually (optional)
327
341
328
342
You can also use a PipelineRun resource to trigger an ImportPipeline for artifact import to your target container registry.
@@ -358,7 +372,7 @@ az acr repository list --name <target-registry-name>
358
372
359
373
## Delete pipeline resources
360
374
361
-
To delete a pipeline resource, delete its Resource Manager deployment by using the [az deployment group delete][azdeploymentgroupdelete] command. The following example deletes a an ExportPipeline deployment named *exportPipeline*:
375
+
To delete a pipeline resource, delete its Resource Manager deployment by using the [az deployment group delete][az-deployment-group-delete] command. The following example deletes an ExportPipeline deployment named *exportPipeline*:
362
376
363
377
```azurecli
364
378
az deployment group delete \
@@ -373,13 +387,13 @@ az deployment group delete \
373
387
* For common template deployment errors, see [Troubleshoot ARM template deployments](../azure-resource-manager/templates/template-tutorial-troubleshoot.md)
374
388
***Problems with export or import of storage blobs**
375
389
* SAS token may be expired, or may have insufficient permissions for the specified export or import run
376
-
* Existing storage blob in source storage account might not be overwritten during multiple export runs. Confirm that the OverwriteBlob option is set in the export run and the SAS token has
377
-
* Storage blob in target storage account might not be deleted after successful import run. Confirm that the DeleteBlobOnSuccess option is set in the import run.
378
-
*Container specified in export or import run does not exist, or specified storage blob doesn't exist for manual import run.
390
+
* Existing storage blob in source storage account might not be overwritten during multiple export runs. Confirm that the OverwriteBlob option is set in the export run and the SAS token has sufficient permissions.
391
+
* Storage blob in target storage account might not be deleted after successful import run. Confirm that the DeleteBlobOnSuccess option is set in the import run and the SAS token has sufficient permissions.
392
+
*Storage blob not created or deleted. Confirm that container specified in export or import run exists, or specified storage blob exists for manual import run.
379
393
***Artifacts transfer problems**
380
-
* Artifacts are transferred incompletely or not at all. Confirm spelling of artifacts in export run, and name of blob in export and import runs.
381
-
* Pipeline run might not have completed. An export or import run can take some time, especially for a large number of artifacts.
382
-
* For other pipeline issues, provide the deployment `correlationId` of the export run or import run to the Azure Container Registry team.
394
+
* Artifacts are transferred incompletely or not at all. Confirm spelling of artifacts in export run, and name of blob in export and import runs. Confirm you are transferring a maximum of 10 artifacts.
395
+
* Pipeline run might not have completed. An export or import run can take some time.
396
+
* For other pipeline issues, provide the deployment [correlation ID](../azure-resource-manager/templates/deployment-history.md) of the export run or import run to the Azure Container Registry team.
0 commit comments