You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-add-storage.md
+13-6Lines changed: 13 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: How to define storage targets so that your Azure HPC Cache can use
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: conceptual
7
-
ms.date: 09/06/2019
7
+
ms.date: 09/24/2019
8
8
ms.author: v-erkell
9
9
---
10
10
@@ -16,11 +16,11 @@ You can define up to ten different storage targets for one cache. The cache pres
16
16
17
17
Remember that the storage exports must be accessible from your cache's virtual network. For on-premises hardware storage, you might need to set up a DNS server that can resolve hostnames for NFS storage access. Read more in [DNS access](hpc-cache-prereqs.md#dns-access).
18
18
19
-
You can add storage targets while creating your Azure HPC Cache, or afterward. The procedure is slightly different depending on whether you're adding Azure Blob storage or an NFS export. Details for each are below.
19
+
You can add storage targets while creating your cache, or afterward. The procedure is slightly different depending on whether you're adding Azure Blob storage or an NFS export. Details for each are below.
20
20
21
21
## Add storage targets while creating the cache
22
22
23
-
Use the **Storage targets** tab of the cache creation wizard to define storage at the same time you create the cache instance.
23
+
Use the **Storage targets** tab of the Azure HPC Cache creation wizard to define storage at the same time you create the cache instance.
24
24
25
25

26
26
@@ -40,6 +40,8 @@ To define an Azure Blob container, enter this information.
40
40
41
41

42
42
43
+
<!-- need to replace screenshot after note text is updated with both required RBAC roles -->
44
+
43
45
***Storage target name** - Set a name that identifies this storage target in the Azure HPC Cache.
44
46
***Target type** - Choose **Blob**.
45
47
***Storage account** - Select the account with the container to reference.
@@ -53,7 +55,7 @@ When finished, click **OK** to add the storage target.
53
55
54
56
### Add the access control roles to your account
55
57
56
-
The Azure HPC Cache uses [role-based access control (RBAC)](https://docs.microsoft.com/azure/role-based-access-control/index) to authorize the cache application to access your storage account for Azure Blob storage targets.
58
+
Azure HPC Cache uses [role-based access control (RBAC)](https://docs.microsoft.com/azure/role-based-access-control/index) to authorize the cache application to access your storage account for Azure Blob storage targets.
57
59
58
60
The storage account owner must explicitly add the roles [Storage Account Contributor](https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#storage-account-contributor) and [Storage Blob Data Contributor](https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) for the user "StorageCache Resource Provider".
59
61
@@ -93,9 +95,14 @@ Provide this information for an NFS-backed storage target:
93
95
94
96
***Usage model** - Choose one of the data caching profiles based on your workflow, described in [Choose a usage model, below](#choose-a-usage-model).
95
97
96
-
You can create multiple namespace paths to represent different exports on the same NFS storage system, but you must create them all from one storage target.
98
+
### NFS namespace paths
99
+
100
+
An NFS storage target can have multiple virtual paths, as long as each path represents a different export or subdirectory on the same storage system.
101
+
102
+
Create all of the paths from one storage target.
103
+
<!-- You can create multiple namespace paths to represent different exports on the same NFS storage system, but you must create them all from one storage target. -->
97
104
98
-
For each export, fill in these values:
105
+
Fill in these values for each namespace path:
99
106
100
107
***Virtual namespace path** - Set the client-facing filepath for this storage target. Read [Configure aggregated namespace](hpc-cache-namespace.md) to learn more about the virtual namespace feature.
Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-create.md
+10-8Lines changed: 10 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: How to create an Azure HPC Cache instance
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: tutorial
7
-
ms.date: 09/06/2019
7
+
ms.date: 09/24/2019
8
8
ms.author: v-erkell
9
9
---
10
10
@@ -18,18 +18,20 @@ Use the Azure portal to create your cache.
18
18
19
19

20
20
21
-
In **Project Details**, select the subscription and resource group that will host the Azure HPC Cache. Make sure the subscription is on the [preview access](hpc-cache-prereqs.md#azure-subscription) list.
21
+
In **Project Details**, select the subscription and resource group that will host the cache. Make sure the subscription is on the [preview access](hpc-cache-prereqs.md#azure-subscription) list.
22
22
23
23
In **Service Details**, set the cache name and these other attributes:
24
24
25
25
* Location - Select one of the [supported regions](hpc-cache-overview.md#region-availability).
26
26
* Virtual network - You can select an existing one or create a new virtual network.
27
-
* Subnet - Choose or create a subnet with at least 64 IP addresses (/24) that will be used only for the Azure HPC Cache.
27
+
* Subnet - Choose or create a subnet with at least 64 IP addresses (/24) that will be used only for this Azure HPC Cache instance.
28
28
29
29
## Set cache capacity
30
30
<!-- referenced from GUI - update aka.ms link if you change this header text -->
31
31
32
-
On the **Cache** page, you must set the capacity of your Azure HPC Cache. This value determines how much data your cache can hold and how quickly it can service client requests. After the public preview period, capacity will also affect the cache's cost.
32
+
On the **Cache** page, you must set the capacity of your cache. This value determines how much data your cache can hold and how quickly it can service client requests.
33
+
34
+
After the public preview period, capacity also will affect the cache's cost.
33
35
34
36
Cache capacity is measured in input/output operations per second (IOPS). Choose the capacity by setting these two values:
35
37
@@ -38,9 +40,9 @@ Cache capacity is measured in input/output operations per second (IOPS). Choose
38
40
39
41
Choose one of the available throughput values and cache storage sizes. The IOPS capacity is calculated and shown below the value selectors.
40
42
41
-
Keep in mind that the actual data transfer rate depends on workload, network speeds, and the type of storage targets. If a file isn't in the cache, or it is marked as stale, the service will use some throughput to fetch it from backend storage. The value you choose sets the maximum throughput for the entire cache, and not all of it is available for client requests.
43
+
Keep in mind that the actual data transfer rate depends on workload, network speeds, and the type of storage targets. The value you choose sets the maximum throughput for the entire cache, and not all of it is available for client requests. For example, if a client requests a file that isn't already stored in the cache, or if the file is marked as stale, your cache uses some of its throughput to fetch it from backend storage.
42
44
43
-
For cache storage, Azure HPC Cache manages which files are cached and preloaded to maximize cache hit rates. The cache contents are continuously assessed and files are moved to long-term storage when they are less frequently accessed. Choose a cache storage size that can comfortably hold the active set of working files with additional space for metadata and other overhead.
45
+
Azure HPC Cache manages which files are cached and preloaded to maximize cache hit rates. The cache contents are continuously assessed and files are moved to long-term storage when they are less frequently accessed. Choose a cache storage size that can comfortably hold the active set of working files with additional space for metadata and other overhead.
44
46
45
47

46
48
@@ -58,7 +60,7 @@ You can define up to ten different storage targets.
58
60
59
61
Step-by-step instructions to add a storage target are included in [Add storage targets](hpc-cache-add-storage.md). The procedure is different for Blob storage or for NFS exports.
60
62
61
-
Here are some tips:
63
+
Here are some tips:
62
64
63
65
* For both types of storage, you must specify how to find the back-end storage system (either an NFS address or a Blob container name) and the client-facing namespace path.
64
66
@@ -68,7 +70,7 @@ Here are some tips:
68
70
69
71
## Add resource tags (optional)
70
72
71
-
The **Tags** page lets you add [resource tags](https://go.microsoft.com/fwlink/?linkid=873112) to your Azure HPC Cache.
73
+
The **Tags** page lets you add [resource tags](https://go.microsoft.com/fwlink/?linkid=873112) to your Azure HPC Cache instance.
Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-ingest.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,21 +4,21 @@ description: How to populate Azure Blob storage for use with Azure HPC Cache
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: conceptual
7
-
ms.date: 09/18/2019
7
+
ms.date: 09/24/2019
8
8
ms.author: v-erkell
9
9
---
10
10
11
-
# Move data to Azure Blob storage for Azure HPC Cache (preview)
11
+
# Move data to Azure Blob storage
12
12
13
-
If your workflow includes moving data to Azure Blob storage, make sure you are using an efficient strategy to copy your data through the Azure HPC Cache.
13
+
If your workflow includes moving data to Azure Blob storage, make sure you are using an efficient strategy. You can either preload data in a new Blob container before defining it as a storage target, or add the container and then copy your data using Azure HPC Cache.
14
14
15
15
This article explains the best ways to move data to Blob storage for use with Azure HPC Cache.
16
16
17
17
Keep these facts in mind:
18
18
19
19
* Azure HPC Cache uses a specialized storage format to organize data in Blob storage. This is why a Blob storage target must either be a new, empty container, or a Blob container that was previously used for Azure HPC Cache data. ([Avere vFXT for Azure](https://azure.microsoft.com/services/storage/avere-vfxt/) also uses this cloud filesystem.)
20
20
21
-
* Copying data through the Azure HPC Cache is best when you use multiple clients and parallel operations. A simple copy command from one client will move data slowly.
21
+
* Copying data through the Azure HPC Cache to a back-end storage target is more efficient when you use multiple clients and parallel operations. A simple copy command from one client will move data slowly.
22
22
23
23
A Python-based utility is available to load content into a Blob storage container. Read [Pre-load data in Blob storage](#pre-load-data-in-blob-storage-with-clfsload) to learn more.
24
24
@@ -36,7 +36,7 @@ Detailed information is included in the Avere CLFSLoad distribution, which is av
36
36
37
37
A general overview of the process:
38
38
39
-
1. Prepare a Linux system (VM or physical) with Python version 3.6 or later. (Python 3.7 is recommended for better performance.)
39
+
1. Prepare a Linux system (VM or physical) with Python version 3.6 or later. Python 3.7 is recommended for better performance.
40
40
1. Install the Avere-CLFSLoad software on the Linux system.
41
41
1. Execute the transfer from the Linux command line.
42
42
@@ -45,7 +45,7 @@ The Avere CLFSLoad utility needs the following information:
45
45
* The storage account ID that contains your Blob storage container
46
46
* The name of the empty Blob storage container
47
47
* A shared access signature (SAS) token that allows the utility to write to the container
48
-
* A local path to the data source - either a local directory that contains the data to copy, or a local path to a mounted remote system with the data.
48
+
* A local path to the data source - either a local directory that contains the data to copy, or a local path to a mounted remote system with the data
49
49
50
50
<!-- The requirements are explained in detail in the [Avere CLFSLoad readme](https://aka.ms/avere-clfsload). -->
Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-mount.md
+10-6Lines changed: 10 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,22 +4,26 @@ description: How to connect clients to an Azure HPC Cache service
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: conceptual
7
-
ms.date: 09/12/2019
7
+
ms.date: 09/24/2019
8
8
ms.author: v-erkell
9
9
---
10
10
11
11
# Mount the Azure HPC Cache (preview)
12
12
13
13
After the cache is created, NFS clients can access it with a simple mount command.
14
14
15
-
Use the mount addresses listed on the cache overview page and the virtual namespace path that you set when you created the storage target.
15
+
The mount command is made up of two elements:
16
+
17
+
* One of the cache's mount addresses (listed on the cache overview page)
18
+
* The virtual namespace path that you set when you created the storage target
16
19
17
20

18
21
19
22
> [!NOTE]
20
-
> The cache mount addresses correspond to network interfaces inside the cache's subnet. These NICs appear in the resource group with names ending in `-cluster-nic-` and a number. Do not alter or delete these interfaces, or the cache will become unavailable.
23
+
> The cache mount addresses correspond to network interfaces inside the cache's subnet. In a resource group, these NICs are listed with names ending in `-cluster-nic-` and a number. Do not alter or delete these interfaces, or the cache will become unavailable.
24
+
25
+
The virtual namespace paths are shown in the **Storage targets** page. Click an individual storage target name to see its details, including aggregated namespace paths associated with it.
21
26
22
-
The virtual namespace paths are shown in the **Storage targets** page. Click the target name to see details, including the aggregated namespace path (or paths) associated with it.
23
27

24
28
25
29
## Mount command syntax
@@ -28,15 +32,15 @@ Use a mount command like the following:
28
32
29
33
> sudo mount *cache_mount_address*:/*namespace_path**local_path* {*options*}
30
34
31
-
Example:
35
+
Example:
32
36
33
37
```
34
38
root@test-client:/tmp# mkdir hpccache
35
39
root@test-client:/tmp# sudo mount 10.0.0.28:/blob-demo-0722 ./hpccache/ -orw,tcp,mountproto=tcp,vers3,hard,intr
36
40
root@test-client:/tmp#
37
41
```
38
42
39
-
After this command succeeds, the contents of the storage export should be visible in the ``hpccache`` directory on the client.
43
+
After this command succeeds, the contents of the storage export should be visible in the ``hpccache`` directory on the client.
40
44
41
45
> [!NOTE]
42
46
> Your clients must be able to access the virtual network and subnet that houses your cache. For example, create client VMs within the same virtual network, or use an endpoint, gateway, or other solution in the virtual network for access from outside. Remember that nothing else can be hosted inside the cache's subnet.
Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-namespace.md
+10-6Lines changed: 10 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,19 +4,21 @@ description: How to create an Azure HPC Cache instance
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: conceptual
7
-
ms.date: 09/06/2019
7
+
ms.date: 09/24/2019
8
8
ms.author: v-erkell
9
9
---
10
10
11
11
# Plan the aggregated namespace
12
12
13
13
Azure HPC Cache (preview) allows clients to access a variety of storage systems through a virtual namespace that hides the details of the back-end storage system.
14
14
15
-
When you add a storage target, you set the client-facing filepath. Client machines mount this filepath. You can change the storage target associated with that path. For example, you could replace a hardware storage system with cloud storage without needing to rewrite client-facing procedures.
15
+
When you add a storage target, you set the client-facing filepath. Client machines mount this filepath and can make file read requests to the cache instead of mounting the storage system directly.
16
+
17
+
Because Azure HPC Cache manages this virtual filesystem, you can change the storage target without changing the client-facing path. For example, you could replace a hardware storage system with cloud storage without needing to rewrite client-facing procedures.
16
18
17
19
## Aggregated namespace example
18
20
19
-
Plan your aggregated namespace so that client machines can conveniently reach the information they need, and administrators and workflow engineers can easily distinguish the paths.
21
+
Plan your aggregated namespace so that client machines can conveniently reach the information they need, and so that administrators and workflow engineers can easily distinguish the paths.
20
22
21
23
For example, consider a system where an Azure HPC Cache instance is being used to process data stored in Azure Blob. The analysis requires template files that are stored in an on-premises datacenter.
22
24
@@ -35,20 +37,22 @@ The data to be analyzed has been copied to an Azure Blob storage container named
35
37
36
38
To allow easy access through the cache, consider creating storage targets with these virtual namespace paths:
|*IP address or hostname*| /goldline/templates | acme2017/sku798 | /templates/sku798 |
49
53
|*IP address or hostname*| /goldline/templates | acme2017/sku980 | /templates/sku980 |
50
54
51
-
A client application can mount the cache and easily access the aggregated namespace filepaths /source, /templates/sku798, and /templates/sku980.
55
+
A client application can mount the cache and easily access the aggregated namespace filepaths ``/source``, ``/templates/sku798``, and ``/templates/sku980``.
0 commit comments