Skip to content

Commit 7382651

Browse files
committed
add continuation token example
1 parent f055b5b commit 7382651

File tree

1 file changed

+50
-24
lines changed

1 file changed

+50
-24
lines changed

articles/storage/blobs/blob-cli.md

Lines changed: 50 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ author: StevenMatthew
77
ms.author: shaas
88
ms.service: azure-blob-storage
99
ms.topic: how-to
10-
ms.date: 05/08/2024
10+
ms.date: 06/11/2024
1111
ms.devlang: azurecli
1212
ms.custom: devx-track-azurecli
1313
---
@@ -104,24 +104,24 @@ az storage blob upload-batch \
104104

105105
By default, the `az storage blob list` command lists all blobs stored within a container. You can use various approaches to refine the scope of your search. There's no restriction on the number of containers or blobs a storage account can have. To potentially avoid retrieving thousands of blobs, it's a good idea to limit the amount of data returned.
106106

107-
Use the `--prefix` parameter to select either a single known file or a range of files whose names begin with a defined string.
107+
Use the `--prefix` parameter to select either a single known file or a range of files whose names begin with a defined string. You can specify a virtual directory as part of the `--prefix` parameter.
108108

109-
By default, only blobs are returned in a listing operation. In some scenarios, you might want to pass a value for the `--include` parameter to return additional types of objects such as soft-deleted blobs, snapshots, and versions. These values can be combined to return more than multiple object types.
109+
By default, only blobs are returned in a listing operation. In some scenarios, you might want to pass a value for the `--include` parameter to return additional types of objects such as soft-deleted blobs, snapshots, and versions. These values can be combined to return multiple object types.
110110

111-
The `--num-results` parameter can be used to limit the number of unfiltered blobs returned from a container. A service limit of 5,000 is imposed on all Azure resources. This limit ensures that manageable amounts of data are retrieved and that performance isn't impacted. If the number of blobs returned exceeds either the `--num-results` value or the service limit, a continuation token is returned. This token allows you to use multiple requests to retrieve any number of blobs. More information is available on [Enumerating blob resources](/rest/api/storageservices/enumerating-blob-resources).
111+
The `--num-results` parameter can be used to limit the number of blobs returned from a container. A service limit of 5,000 is imposed on all Azure resources. This limit ensures that manageable amounts of data are retrieved and that performance isn't impacted. If the number of blobs returned exceeds either the `--num-results` value or the service limit, a continuation token is returned. This token allows you to use multiple requests to retrieve any number of blobs. More information is available on [Enumerating blob resources](/rest/api/storageservices/enumerating-blob-resources).
112112

113-
The following example shows several approaches used to provide a list of blobs. The first approach lists a single blob within a specific container resource. The second approach uses the `--prefix` parameter to list all blobs in all containers with a prefix of *louis*. The search is restricted to five containers using the `--num-results` parameter. The third approach uses `--num-results` and `--marker` parameters to limit the retrieval of all blobs within a container.
113+
The following example shows several approaches used to provide a list of blobs. The first approach lists all blobs within a specified container. The second approach uses the `--prefix` parameter to list all blobs in the containers that begin with the specified prefix.The third approach uses the `--num-results` parameter to limit the results returned and the `--show-next-marker` parameter to include the continuation token in the results. When a continuation token is present in the results, it is passed to the subsequent call to `az storage blob list` to retrieve the next set of results.
114114

115115
For more information, see the [az storage blob list](/cli/azure/storage/blob#az-storage-blob-list) reference.
116116

117117
```azurecli-interactive
118118
#!/bin/bash
119119
storageAccount="<storage-account>"
120-
containerName="demo-container"
121-
blobPrefix="img-louis"
120+
containerName="<container-name>"
121+
blobPrefix="<prefix-string>"
122122
numResults=5
123123
124-
#Approach 1: List all blobs in a container
124+
#Approach 1: List all blobs in a container by name.
125125
126126
az storage blob list \
127127
--account-name $storageAccount \
@@ -130,7 +130,7 @@ az storage blob list \
130130
--auth-mode login \
131131
--output tsv
132132
133-
#Approach 2: Use the --prefix parameter to list blobs starting with specified prefix
133+
#Approach 2: Use the --prefix parameter to list blobs starting with specified prefix.
134134
135135
az storage blob list \
136136
--account-name $storageAccount \
@@ -140,22 +140,48 @@ az storage blob list \
140140
--auth-mode login \
141141
--output tsv
142142
143-
#Approach 3: Use the continuation token to return the next set of blobs.
144-
145-
$resultSet=$(az storage blob list `
146-
--account-name $storageAccount `
147-
--container $containerName `
148-
--prefix $blobPrefix `
149-
--num-results $numResults `
150-
--show-next-marker `
151-
--query "[].{Name:name, Marker:nextMarker}" `
152-
--auth-mode login `
153-
--output tsv)
154-
155-
$markerEntry=$resultSet[$numResults] | cut -f2
156-
$marker= echo $markerEntry | cut -f2
157-
143+
#Approach 3: Use the continuation token to return the complete set of results.
158144
145+
get_blobs() {
146+
if [ -z "$nextMarker" ]; then
147+
az storage blob list --container-name $containerName --num-results $numResults --account-name $storageAccount --show-next-marker --only-show-errors --auth-mode login
148+
else
149+
az storage blob list --container-name $containerName --num-results $numResults --marker $nextMarker --account-name $storageAccount --show-next-marker --only-show-errors --auth-mode login
150+
fi
151+
}
152+
153+
total=0
154+
nextMarker=""
155+
while true; do
156+
blobs=$(get_blobs $containerName $numResults $storageAccount $nextMarker)
157+
nextMarker=""
158+
blobsLength=$(echo $blobs | jq length)
159+
160+
if [ $blobsLength -le 0 ]; then
161+
break
162+
fi
163+
164+
blobIndex=0
165+
while read blob; do
166+
if [ $blobIndex -eq $(($blobsLength-1)) ];
167+
then
168+
nextMarker="$(echo $blob | jq -r .nextMarker)"
169+
else
170+
blobName=$(echo $blob | jq .name)
171+
echo "blobname: $blobName"
172+
fi
173+
((blobIndex++))
174+
done <<<$(echo $blobs | jq -c '.[]')
175+
176+
total=$(($total+$blobsLength-1))
177+
echo "Processed $total blobs so far"
178+
# echo "nextMarker: $nextMarker"
179+
if [ "${nextMarker}" = "null" ]; then
180+
echo -e "\nAccountName: $storageAccount, ContainerName: $containerName "
181+
echo "Processed $total blobs in total."
182+
break
183+
fi
184+
done
159185
```
160186

161187
## Download a blob

0 commit comments

Comments
 (0)