Skip to content

Commit 430d855

Browse files
committed
fix for spaces and tab & consistent indentation
1 parent a168239 commit 430d855

File tree

2 files changed

+100
-103
lines changed

2 files changed

+100
-103
lines changed

articles/cosmos-db/postgresql/howto-ingest-azure-blob-storage.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ author: mulander
66
ms.service: cosmos-db
77
ms.subservice: postgresql
88
ms.topic: how-to
9-
ms.date: 01/30/2023
9+
ms.date: 05/12/2023
1010
---
1111

1212
# How to ingest data using pg_azure_storage in Azure Cosmos DB for PostgreSQL
@@ -345,5 +345,5 @@ INSERT 0 264308
345345
346346
Congratulations, you just learned how to load data into Azure Cosmos DB for PostgreSQL directly from Azure Blob Storage.
347347
348-
Learn how to create a [real-time dashboard](tutorial-design-database-realtime.md) with Azure Cosmos DB for PostgreSQL.
349-
Learn more about [pg_azure_storage](reference-pg-azure-storage.md).
348+
- Learn how to create a [real-time dashboard](tutorial-design-database-realtime.md) with Azure Cosmos DB for PostgreSQL.
349+
- Learn more about [pg_azure_storage](reference-pg-azure-storage.md).

articles/cosmos-db/postgresql/reference-pg-azure-storage.md

Lines changed: 97 additions & 100 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ author: AvijitkGupta
66
ms.service: cosmos-db
77
ms.subservice: postgresql
88
ms.topic: reference
9-
ms.date: 04/28/2023
9+
ms.date: 05/12/2023
1010
---
1111

1212
# pg_azure_storage extension
@@ -40,12 +40,7 @@ FROM { 'filename' | PROGRAM 'command' | STDIN | Azure_blob_url}
4040
4141
### Arguments
4242
#### Azure_blob_url
43-
Allows unstructured data to be stored and accessed at a massive scale in block blobs. Objects in blob storage can be accessed from anywhere in the world via HTTP or HTTPS. The storage client libraries are available for multiple languages, including .NET, Java, Node.js, Python, PHP, and Ruby. User or client applications can access blobs via following methods
44-
1. URLs
45-
1. The Azure Storage REST API
46-
1. Azure PowerShell
47-
1. Azure CLI
48-
1. An Azure storage client library
43+
Allows unstructured data to be stored and accessed at a massive scale in block blobs. Objects in blob storage can be accessed from anywhere in the world via HTTPS. The storage client libraries are available for multiple languages, including .NET, Java, Node.js, Python, PHP, and Ruby.
4944

5045
### Option
5146
#### format
@@ -64,14 +59,14 @@ Function allows adding access to a storage account.
6459
```postgresql
6560
azure_storage.account_add
6661
(account_name_p text
67-
, account_key_p text);
62+
,account_key_p text);
6863
```
6964
### Arguments
7065
#### account_name_p
71-
An Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTP or HTTPS.
66+
An Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTPS.
7267

7368
#### account_key_p
74-
Your Azure blob storage (ABS) access keys are similar to a root password for your storage account. Always be careful to protect your access keys. Use Azure Key Vault to manage and rotate your keys securely. The account key is stored in a table that is only accessible by the postgres superuser. To see which storage accounts exist, use the account_list
69+
Your Azure blob storage (ABS) access keys are similar to a root password for your storage account. Always be careful to protect your access keys. Use Azure Key Vault to manage and rotate your keys securely. The account key is stored in a table that is accessible by the postgres superuser, azure_storage_admin and all roles granted those admin permissions. To see which storage accounts exist, use the function account_list.
7570

7671
## azure_storage.account_remove
7772
Function allows revoking account access to storage account.
@@ -83,7 +78,7 @@ azure_storage.account_remove
8378

8479
### Arguments
8580
#### account_name_p
86-
Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTP or HTTPS.
81+
Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTPS.
8782

8883
## azure_storage.account_user_add
8984
The function allows adding access for a role to a storage account.
@@ -96,7 +91,7 @@ azure_storage.account_add
9691

9792
### Arguments
9893
#### account_name_p
99-
An Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTP or HTTPS.
94+
An Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTPS.
10095

10196
#### user_p
10297
Role created by user visible on the cluster.
@@ -109,13 +104,13 @@ The function allows removing access for a role to a storage account.
109104

110105
```postgresql
111106
azure_storage.account_remove
112-
( account_name_p text
113-
, user_p regrole);
107+
(account_name_p text
108+
,user_p regrole);
114109
```
115110

116111
### Arguments
117112
#### account_name_p
118-
An Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTP or HTTPS.
113+
An Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTPS.
119114

120115
#### user_p
121116
Role created by user visible on the cluster.
@@ -125,15 +120,15 @@ The function lists the account & role having access to Azure blob storage.
125120

126121
```postgresql
127122
azure_storage.account_list
128-
( OUT account_name text
129-
, OUT allowed_users regrole[]
130-
)
123+
(OUT account_name text
124+
,OUT allowed_users regrole[]
125+
)
131126
Returns TABLE;
132127
```
133128

134129
### Arguments
135130
#### account_name
136-
Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTP or HTTPS.
131+
Azure blob storage (ABS) account contains all of your ABS objects: blobs, files, queues, and tables. The storage account provides a unique namespace for your ABS that is accessible from anywhere in the world over HTTPS.
137132

138133
#### allowed_users
139134
Lists the users having access to the Azure blob storage.
@@ -146,23 +141,24 @@ The function lists the available blob files within a user container with their p
146141

147142
```postgresql
148143
azure_storage.blob_list
149-
(account_name text
150-
, container_name text
151-
, prefix text DEFAULT ''::text
152-
, OUT path text
153-
, OUT bytes bigint
154-
, OUT last_modified timestamp with time zone
155-
, OUT etag text
156-
, OUT content_type text
157-
, OUT content_encoding text
158-
, OUT content_hash text
159-
)
144+
(account_name text
145+
,container_name text
146+
,prefix text DEFAULT ''::text
147+
,OUT path text
148+
,OUT bytes bigint
149+
,OUT last_modified timestamp with time zone
150+
,OUT etag text
151+
,OUT content_type text
152+
,OUT content_encoding text
153+
,OUT content_hash text
154+
)
160155
Returns SETOF record;
161156
```
162157

163158
### Arguments
164159
#### account_name
165-
The `storage account name` provides a unique namespace for your Azure storage data that's accessible from anywhere in the world over HTTP or HTTPS.
160+
The `storage account name` provides a unique namespace for your Azure storage data that's accessible from anywhere in the world over HTTPS.
161+
166162
#### container_name
167163
A container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs.
168164
A container name must be a valid DNS name, as it forms part of the unique URI used to address the container or its blobs. Follow these rules when naming a container:
@@ -172,8 +168,9 @@ A container name must be a valid DNS name, as it forms part of the unique URI us
172168
* Two or more consecutive dash characters aren't permitted in container names.
173169

174170
The URI for a container is similar to:
175-
https://<span></span>myaccount.blob.core.windows.net/mycontainer
176-
171+
```
172+
`https://<span></span>myaccount.blob.core.windows.net/mycontainer`
173+
```
177174
#### prefix
178175
returns file from blob container with matching string initials.
179176
#### path
@@ -204,31 +201,31 @@ The function allows loading the content of file \ files from within the containe
204201
```postgresql
205202
azure_storage.blob_get
206203
(account_name text
207-
, container_name text
208-
, path text
209-
, decoder text DEFAULT 'auto'::text
210-
, compression text DEFAULT 'auto'::text
211-
, options jsonb DEFAULT NULL::jsonb
204+
,container_name text
205+
,path text
206+
,decoder text DEFAULT 'auto'::text
207+
,compression text DEFAULT 'auto'::text
208+
,options jsonb DEFAULT NULL::jsonb
212209
)
213210
RETURNS SETOF record;
214211
```
215212
There's an overloaded version of function, containing rec parameter that allows you to conveniently define the output format record.
216213
```postgresql
217214
azure_storage.blob_get
218215
(account_name text
219-
, container_name text
220-
, path text
221-
, rec anyelement
222-
, decoder text DEFAULT 'auto'::text
223-
, compression text DEFAULT 'auto'::text
224-
, options jsonb DEFAULT NULL::jsonb
216+
,container_name text
217+
,path text
218+
,rec anyelement
219+
,decoder text DEFAULT 'auto'::text
220+
,compression text DEFAULT 'auto'::text
221+
,options jsonb DEFAULT NULL::jsonb
225222
)
226223
RETURNS SETOF anyelement;
227224
```
228225

229226
### Arguments
230227
#### account
231-
The storage account provides a unique namespace for your Azure Storage data that's accessible from anywhere in the world over HTTP or HTTPS.
228+
The storage account provides a unique namespace for your Azure Storage data that's accessible from anywhere in the world over HTTPS.
232229
#### container
233230
A container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs.
234231
A container name must be a valid DNS name, as it forms part of the unique URI used to address the container or its blobs.
@@ -263,15 +260,15 @@ The function acts as a utility function called as a parameter within blob_get, w
263260

264261
```postgresql
265262
azure_storage.options_csv_get
266-
(delimiter text DEFAULT NULL::text
267-
, null_string text DEFAULT NULL::text
268-
, header boolean DEFAULT NULL::boolean
269-
, quote text DEFAULT NULL::text
270-
, escape text DEFAULT NULL::text
271-
, force_not_null text[] DEFAULT NULL::text[]
272-
, force_null text[] DEFAULT NULL::text[]
273-
, content_encoding text DEFAULT NULL::text
274-
)
263+
(delimiter text DEFAULT NULL::text
264+
,null_string text DEFAULT NULL::text
265+
,header boolean DEFAULT NULL::boolean
266+
,quote text DEFAULT NULL::text
267+
,escape text DEFAULT NULL::text
268+
,force_not_null text[] DEFAULT NULL::text[]
269+
,force_null text[] DEFAULT NULL::text[]
270+
,content_encoding text DEFAULT NULL::text
271+
)
275272
Returns jsonb;
276273
```
277274

@@ -308,16 +305,16 @@ The function acts as a utility function called as a parameter within blob_get.
308305

309306
```postgresql
310307
azure_storage.options_copy
311-
(delimiter text DEFAULT NULL::text
312-
, null_string text DEFAULT NULL::text
313-
, header boolean DEFAULT NULL::boolean
314-
, quote text DEFAULT NULL::text
315-
, escape text DEFAULT NULL::text
316-
, force_quote text[] DEFAULT NULL::text[]
317-
, force_not_null text[] DEFAULT NULL::text[]
318-
, force_null text[] DEFAULT NULL::text[]
319-
, content_encoding text DEFAULT NULL::text
320-
)
308+
(delimiter text DEFAULT NULL::text
309+
,null_string text DEFAULT NULL::text
310+
,header boolean DEFAULT NULL::boolean
311+
,quote text DEFAULT NULL::text
312+
,escape text DEFAULT NULL::text
313+
,force_quote text[] DEFAULT NULL::text[]
314+
,force_not_null text[] DEFAULT NULL::text[]
315+
,force_null text[] DEFAULT NULL::text[]
316+
,content_encoding text DEFAULT NULL::text
317+
)
321318
Returns jsonb;
322319
```
323320

@@ -357,10 +354,10 @@ The function acts as a utility function called as a parameter within blob_get. I
357354

358355
```postgresql
359356
azure_storage.options_tsv
360-
(delimiter text DEFAULT NULL::text
361-
, null_string text DEFAULT NULL::text
362-
, content_encoding text DEFAULT NULL::text
363-
)
357+
(delimiter text DEFAULT NULL::text
358+
,null_string text DEFAULT NULL::text
359+
,content_encoding text DEFAULT NULL::text
360+
)
364361
Returns jsonb;
365362
```
366363

@@ -382,8 +379,7 @@ The function acts as a utility function called as a parameter within blob_get. I
382379

383380
```postgresql
384381
azure_storage.options_binary
385-
(content_encoding text DEFAULT NULL::text
386-
)
382+
(content_encoding text DEFAULT NULL::text)
387383
Returns jsonb;
388384
```
389385

@@ -403,17 +399,17 @@ Now you can list containers set to Private and Blob access levels for that stora
403399
The examples used make use of sample Azure storage account `(pgquickstart)` with custom files uploaded for adding to coverage of different use cases. We can start by creating table used across the set of example used.
404400
```sql
405401
CREATE TABLE IF NOT EXISTS public.events
406-
(
407-
event_id bigint,
408-
event_type text,
409-
event_public boolean,
410-
repo_id bigint,
411-
payload jsonb,
412-
repo jsonb,
413-
user_id bigint,
414-
org jsonb,
415-
created_at timestamp without time zone
416-
)
402+
(
403+
event_id bigint
404+
,event_type text
405+
,event_public boolean
406+
,repo_id bigint
407+
,payload jsonb
408+
,repo jsonb
409+
,user_id bigint
410+
,org jsonb
411+
,created_at timestamp without time zone
412+
);
417413
```
418414

419415
### Adding access key of storage account (mandatory for access level = private)
@@ -423,7 +419,7 @@ The example illustrates adding of access key for the storage account to get acce
423419
SELECT azure_storage.account_add('pgquickstart', 'SECRET_ACCESS_KEY');
424420
```
425421
> [!TIP]
426-
In your storage account, open **Access keys**. Copy the **Storage account name** and copy the **Key** from **key1** section (you have to select **Show** next to the key first).
422+
> In your storage account, open **Access keys**. Copy the **Storage account name** and copy the **Key** from **key1** section (you have to select **Show** next to the key first).
427423
428424
:::image type="content" source="media/howto-ingestion/azure-blob-storage-account-key.png" alt-text="Screenshot of Security + networking > Access keys section of an Azure Blob Storage page in the Azure portal." border="true":::
429425

@@ -455,7 +451,7 @@ SELECT * FROM azure_storage.blob_list('pgquickstart','publiccontainer','e');
455451
Alternatively
456452

457453
```sql
458-
SELECT * FROM azure_storage.blob_list('pgquickstart','publiccontainer','e') WHERE path LIKE 'e%';
454+
SELECT * FROM azure_storage.blob_list('pgquickstart','publiccontainer') WHERE path LIKE 'e%';
459455
```
460456

461457
### Read content from an object in a container
@@ -475,16 +471,16 @@ Alternatively, we can explicitly define the columns in the `FROM` clause.
475471
```sql
476472
SELECT * FROM azure_storage.blob_get('pgquickstart','publiccontainer','events.csv')
477473
AS res (
478-
event_id TEXT,
479-
event_type DATE,
480-
event_public INTEGER,
481-
repo_id INTEGER,
482-
payload JSONB,
483-
repo JSONB,
484-
user_id BIGINT,
485-
org JSONB,
486-
created_at TIMESTAMP WITHOUT TIME ZONE)
487-
LIMIT 5;
474+
event_id TEXT
475+
,event_type DATE
476+
,event_public INTEGER
477+
,repo_id INTEGER
478+
,payload JSONB
479+
,repo JSONB
480+
,user_id BIGINT
481+
,org JSONB
482+
,created_at TIMESTAMP WITHOUT TIME ZONE)
483+
LIMIT 5;
488484
```
489485

490486
### Use decoder option
@@ -510,7 +506,7 @@ SELECT * FROM azure_storage.blob_get
510506
,'events-compressed'
511507
, NULL::events
512508
, decoder := 'csv'
513-
, compression := 'gzip')
509+
, compression := 'gzip')
514510
LIMIT 5;
515511
```
516512

@@ -531,18 +527,19 @@ LIMIT 5;
531527
The example illustrates the use of `options` argument for processing files with headers, custom separators, escape characters, etc., you can either use the COPY command or pass COPY options to the blob_get function using the `azure_storage.options_copy` function.
532528
```sql
533529
SELECT * FROM azure_storage.blob_get
534-
( 'pgquickstart'
535-
,'publiccontainer'
536-
,'events_pipe.csv'
537-
, NULL::events
538-
, options := azure_storage.options_csv_get(delimiter := '|' , header := 'true'));
530+
('pgquickstart'
531+
,'publiccontainer'
532+
,'events_pipe.csv'
533+
,NULL::events
534+
,options := azure_storage.options_csv_get(delimiter := '|' , header := 'true')
535+
);
539536
```
540537

541538
### Aggregation query on content of an object in the container
542539
The example illustrates the ability to directly perform analysis on the data without importing the set into the database.
543540
```sql
544541
SELECT event_type,COUNT(1) FROM azure_storage.blob_get
545-
('pgquickstart'
542+
('pgquickstart'
546543
,'publiccontainer'
547544
,'events.csv'
548545
, NULL::events)

0 commit comments

Comments
 (0)