Skip to content

Commit dcb9123

Browse files
liljack17liljack17
authored
Merge pull request #219152 from sipastak/dbd-metadata-2
DBD metadata
2 parents 9393b0b + a779e9c commit dcb9123

7 files changed

+257
-116
lines changed

‎articles/databox/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,8 @@
161161
href: data-box-disk-limits.md
162162
- name: Review security
163163
href: data-box-disk-security.md
164+
- name: Transfer ACLs and metadata
165+
href: data-box-disk-file-acls-preservation.md
164166
- name: FAQ
165167
href: data-box-disk-faq.yml
166168
- name: How to

‎articles/databox/data-box-deploy-copy-data.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ author: alkohli
77
ms.service: databox
88
ms.subservice: pod
99
ms.topic: tutorial
10-
ms.date: 08/26/2022
10+
ms.date: 11/18/2022
1111
ms.author: alkohli
1212

1313
# Customer intent: As an IT admin, I need to be able to copy data to Data Box to upload on-premises data from my server onto Azure.

‎articles/databox/data-box-disk-deploy-copy-data.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ author: alkohli
77
ms.service: databox
88
ms.subservice: disk
99
ms.topic: tutorial
10-
ms.date: 10/12/2022
10+
ms.date: 11/18/2022
1111
ms.author: alkohli
1212

1313
# Customer intent: As an IT admin, I need to be able to order Data Box Disk to upload on-premises data from my server onto Azure.
@@ -53,6 +53,7 @@ Review the following considerations before you copy the data to the disks:
5353

5454
- It is your responsibility to ensure that you copy the data to folders that correspond to the appropriate data format. For instance, copy the block blob data to the folder for block blobs. If the data format does not match the appropriate folder (storage type), then at a later step, the data upload to Azure fails.
5555
- While copying data, ensure that the data size conforms to the size limits described in the [Azure storage and Data Box Disk limits](data-box-disk-limits.md).
56+
- If you want to preserve metadata (ACLs, timestamps, and file attributes) when transferring data to Azure Files, follow the guidance in [Preserving file ACLs, attributes, and timestamps with Azure Data Box Disk](data-box-disk-file-acls-preservation.md).
5657
- If data that is being uploaded by Data Box Disk is concurrently uploaded by other applications outside of Data Box Disk, this could result in upload job failures and data corruption.
5758

5859
> [!IMPORTANT]

‎articles/databox/data-box-disk-file-acls-preservation.md

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
---
2+
title: Preserving file ACLs, attributes, and timestamps with Azure Data Box disk
3+
description: ACLs, timestamps, and attributes preserved during data copy to Azure Data Box Disk. Copying metadata with Windows and Linux data copy tools.
4+
services: databox
5+
author: alkohli
6+
7+
ms.service: databox
8+
ms.subservice: pod
9+
ms.topic: conceptual
10+
ms.date: 12/22/2022
11+
ms.author: alkohli
12+
---
13+
14+
# Preserving file ACLs, attributes, and timestamps with Azure Data Box disk
15+
16+
Azure Data Box Disk lets you preserve access control lists (ACLs), timestamps, and file attributes when sending data to Azure. This article describes the metadata that you can transfer when copying data to Data Box Disk to upload it to Azure Files.
17+
18+
## Transferred metadata
19+
20+
ACLs, timestamps, and file attributes are the metadata that is transferred when the data from Data Box Disk is uploaded to Azure Files. In this article, ACLs, timestamps, and file attributes are referred to collectively as *metadata*.
21+
22+
The metadata can be copied with Windows data copy tools. Metadata isn't preserved when transferring data to blob storage.
23+
24+
The subsequent sections of the article discuss in detail as to how the timestamps, file attributes, and ACLs are transferred when the data from Data Box Disk is uploaded to Azure Files.
25+
26+
[!INCLUDE [data-box-transferred-metadata](../../includes/data-box-transferred-metadata.md)]
27+
28+
## ACLs
29+
30+
Depending on the transfer method used and whether you're using a Windows or Linux client, some or all discretionary and default access control lists (ACLs) on files and folders may be transferred during the data copy to Azure Files.
31+
32+
> [!NOTE]
33+
> Files with ACLs containing conditional access control entry (ACE) strings are not copied. This is a known issue. To work around this, copy these files to the Azure Files share manually by mounting the share and then using a copy tool that supports copying ACLs.
34+
35+
## Copying data and metadata
36+
37+
To transfer the ACLs, timestamps, and attributes for your data, use the following procedures to copy data into the Data Box.
38+
39+
### Windows data copy tool
40+
41+
To copy data to your Data Box Disk, use a file copy tool such as `robocopy`. The following sample command copies all files and directories, transferring metadata along with the data.
42+
43+
```console
44+
robocopy <Source> <Target> * /copyall /e /dcopy:DAT /B /r:3 /w:60 /is /nfl /ndl /np /MT:32 or 64 /fft /log+:<LogFile>
45+
```
46+
47+
where
48+
49+
|Option |Description |
50+
|------------------- | ----- |
51+
|`/copyall` |Copies all attributes.|
52+
|`/e` |Copies subdirectories, including empty directories. |
53+
|`/dcopy:DAT` |Copies data, attributes, and timestamps. Note: The /dcopy:DAT option must be used to transfer `CreationTime` on directories. |
54+
|`/B` |Copies files in Backup mode. |
55+
|`/r:3` |Specifies 3 retries on failed copies. |
56+
|`/w:60` |Specifies a wait time of 60 seconds between retries. |
57+
|`/is` |Includes the same files. |
58+
|`/nfl` |Does not log file names. |
59+
|`/ndl` |Does not log directory names. |
60+
|`/np` |Does not display progress of the copying operation. |
61+
|`/MT:32 or 64` |Uses multithreading, with 32 or 64 threads. |
62+
|`/fft` |Reduces time stamp granularity for any file system. |
63+
|`/log+:<LogFile>` |Appends the output to the existing log file.|
64+
65+
For more information on these `robocopy` parameters, see [Tutorial: Copy data to Azure Data Box via SMB](./data-box-deploy-copy-data.md)
66+
67+
> [!NOTE]
68+
> If you use `/copyall` to copy your data, the source ACLs on directories and files are transferred to Azure Files. If you only had read-access on your source data and could not modify the source data, you'll have read-access only on the data in the Data Box Disk. Use `/copyall` only if you intend to copy all the ACLs on the directories and files along with the data.
69+
70+
#### Use robocopy to list, copy, modify files on Data Box disk
71+
72+
Here are some of the common scenarios you'll use when copying data using `robocopy`.
73+
74+
- **Copy only data to Data Box Disk, no ACLs on directories and files**
75+
76+
Use the `/dcopy:DAT` option to only copy data, attributes, timestamps. ACLs on directories and files are not copied.
77+
78+
- **Copy data and ACLs on directories and files to Data Box Disk**
79+
80+
Use `/copyall` to copy all the source data including all the ACLs on directories and files.
81+
82+
- **List the filesystem on Data Box Disk using robocopy**
83+
84+
Use this command to list directory contents:
85+
86+
`robocopy <source-dir> NULL /l /s /xx /njh /njs /fp /B`
87+
88+
Note that the File Explorer doesn't allow you to list these files.
89+
90+
- **Copy or delete folders and files on Data Box Disk**
91+
92+
Use this command to copy a single file:
93+
94+
`robocopy <source-dir> <destination-dir> <file-name> /B`
95+
96+
Use this command to delete a single file:
97+
98+
`robocopy <source-dir> <destination-dir> <file-name> /purge /B`
99+
100+
In the above command, the `<source-dir>` should not have the file: `<file-name>`. Then, the above command syncs the destination with the source, resulting in the removal of the file from the destination.
101+
102+
Note that the File Explorer may not allow you to perform the above operations.
103+
104+
For more information, see [Using robocopy commands](/windows-server/administration/windows-commands/robocopy).
105+
106+
### Linux data copy tools
107+
108+
Transferring metadata in Linux is a two-step process. First, you copy the source data using a tool such as `rsync`, which does not copy metadata. After you copy the data, you can copy the metadata using a tool such as `smbcacls` or `cifsacl`.
109+
110+
The following sample commands do the first step, copying the data using `rsync`.
111+
112+
```console
113+
cp -aR /etc /opt/
114+
rsync -avP /etc /opt (-a copies a directory)
115+
```
116+
117+
## Next steps
118+
119+
- [Copy data to Azure Data Box Disk](data-box-disk-deploy-copy-data.md)

‎articles/databox/data-box-file-acls-preservation.md

Lines changed: 3 additions & 114 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ author: alkohli
77
ms.service: databox
88
ms.subservice: pod
99
ms.topic: conceptual
10-
ms.date: 09/12/2022
10+
ms.date: 11/18/2022
1111
ms.author: alkohli
1212
---
1313

@@ -23,37 +23,8 @@ The metadata can be copied with Windows and Linux data copy tools. Metadata isn'
2323

2424
The subsequent sections of the article discuss in detail as to how the timestamps, file attributes, and ACLs are transferred when the data from Data Box is uploaded to Azure Files.
2525

26-
## Timestamps
26+
[!INCLUDE [data-box-transferred-metadata](../../includes/data-box-transferred-metadata.md)]
2727

28-
The following timestamps are transferred:
29-
- CreationTime
30-
- LastWriteTime
31-
32-
The following timestamp isn't transferred:
33-
- LastAccessTime
34-
35-
## File attributes
36-
37-
File attributes on both files and directories are transferred unless otherwise noted.
38-
39-
The following file attributes are transferred:
40-
- FILE_ATTRIBUTE_READONLY (file only)
41-
- FILE_ATTRIBUTE_HIDDEN
42-
- FILE_ATTRIBUTE_SYSTEM
43-
- FILE_ATTRIBUTE_DIRECTORY (directory only)
44-
- FILE_ATTRIBUTE_ARCHIVE
45-
- FILE_ATTRIBUTE_TEMPORARY (file only)
46-
- FILE_ATTRIBUTE_NO_SCRUB_DATA
47-
48-
The following file attributes aren't transferred:
49-
- FILE_ATTRIBUTE_OFFLINE
50-
- FILE_ATTRIBUTE_NOT_CONTENT_INDEXED
51-
52-
Read-only attributes on directories aren't transferred.
53-
54-
## Alternate data streams and extended attributes
55-
56-
[Alternate data streams](/openspecs/windows_protocols/ms-fscc/e2b19412-a925-4360-b009-86e3b8a020c8) and extended attributes are not supported in Azure Files, page blob, or block blob storage, so they are not transferred when copying data.
5728

5829
## ACLs
5930

@@ -121,89 +92,7 @@ The following default ACLs are transferred:
12192

12293
- Security descriptors with these properties: DACL, Owner, Group, SACL
12394

124-
## Copying data and metadata
125-
126-
To transfer the ACLs, timestamps, and attributes for your data, use the following procedures to copy data into the Data Box.
127-
128-
### Windows data copy tool
129-
130-
To copy data to your Data Box via SMB, use an SMB-compatible file copy tool such as `robocopy`. The following sample command copies all files and directories, transferring metadata along with the data.
131-
132-
When using the `/copyall` or `/dcopy:DAT` option, make sure the required Backup Operator privileges aren't disabled. For more information, see [Use the local web UI to administer your Data Box and Data Box Heavy](./data-box-local-web-ui-admin.md).
133-
134-
```console
135-
robocopy <Source> <Target> * /copyall /e /dcopy:DAT /B /r:3 /w:60 /is /nfl /ndl /np /MT:32 or 64 /fft /log+:<LogFile>
136-
```
137-
138-
where
139-
140-
|Option |Description |
141-
|------------------- | ----- |
142-
|`/copyall` |Copies all attributes.|
143-
|`/e` |Copies subdirectories, including empty directories. |
144-
|`/dcopy:DAT` |Copies data, attributes, and timestamps. Note: The /dcopy:DAT option must be used to transfer `CreationTime` on directories. |
145-
|`/B` |Copies files in Backup mode. |
146-
|`/r:3` |Specifies 3 retries on failed copies. |
147-
|`/w:60` |Specifies a wait time of 60 seconds between retries. |
148-
|`/is` |Includes the same files. |
149-
|`/nfl` |Does not log file names. |
150-
|`/ndl` |Does not log directory names. |
151-
|`/np` |Does not display progress of the copying operation. |
152-
|`/MT:32 or 64` |Uses multithreading, with 32 or 64 threads. |
153-
|`/fft` |Reduces time stamp granularity for any file system. |
154-
|`/log+:<LogFile>` |Appends the output to the existing log file.|
155-
156-
For more information on these `robocopy` parameters, see [Tutorial: Copy data to Azure Data Box via SMB](./data-box-deploy-copy-data.md)
157-
158-
> [!NOTE]
159-
> If you use `/copyall` to copy your data, the source ACLs on directories and files are transferred to Azure Files. If you only had read-access on your source data and could not modify the source data, you'll have read-access only on the data in the Data Box. Use `/copyall` only if you intend to copy all the ACLs on the directories and files along with the data.
160-
161-
#### Use robocopy to list, copy, modify files on Data Box
162-
163-
Here are some of the common scenarios you'll use when copying data using `robocopy`.
164-
165-
- **Copy only data to Data Box, no ACLs on directories and files**
166-
167-
Use the `/dcopy:DAT` option to only copy data, attributes, timestamps. ACLs on directories and files are not copied.
168-
169-
- **Copy data and ACLs on directories and files to Data Box**
170-
171-
Use `/copyall` to copy all the source data including all the ACLs on directories and files.
172-
173-
- **List the filesystem on Data Box using robocopy**
174-
175-
Use this command to list directory contents:
176-
177-
`robocopy <source-dir> NULL /l /s /xx /njh /njs /fp /B`
178-
179-
Note that the File Explorer doesn't allow you to list these files.
180-
181-
- **Copy or delete folders and files on Data Box**
182-
183-
Use this command to copy a single file:
184-
185-
`robocopy <source-dir> <destination-dir> <file-name> /B`
186-
187-
Use this command to delete a single file:
188-
189-
`robocopy <source-dir> <destination-dir> <file-name> /purge /B`
190-
191-
In the above command, the `<source-dir>` should not have the file: `<file-name>`. Then, the above command syncs the destination with the source, resulting in the removal of the file from the destination.
192-
193-
Note that the File Explorer may not allow you to perform the above operations.
194-
195-
For more information, see [Using robocopy commands](/windows-server/administration/windows-commands/robocopy).
196-
197-
### Linux data copy tools
198-
199-
Transferring metadata in Linux is a two-step process. First, you copy the source data using a tool such as `rsync`, which does not copy metadata. After you copy the data, you can copy the metadata using a tool such as `smbcacls` or `cifsacl`.
200-
201-
The following sample commands do the first step, copying the data using `rsync`.
202-
203-
```console
204-
cp -aR /etc /opt/
205-
rsync -avP /etc /opt (-a copies a directory)
206-
```
95+
[!INCLUDE [data-box-copy-data-and-metadata](../../includes/data-box-copy-data-and-metadata.md)]
20796

20897
## Next steps
20998

‎includes/data-box-copy-data-and-metadata.md

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
author: sipastak
3+
ms.service: databox
4+
ms.topic: include
5+
ms.date: 11/18/2022
6+
ms.author: sipastak
7+
---
8+
9+
## Copying data and metadata
10+
11+
To transfer the ACLs, timestamps, and attributes for your data, use the following procedures to copy data into the Data Box.
12+
13+
### Windows data copy tool
14+
15+
To copy data to your Data Box via SMB, use an SMB-compatible file copy tool such as `robocopy`. The following sample command copies all files and directories, transferring metadata along with the data.
16+
17+
When using the `/copyall` or `/dcopy:DAT` option, make sure the required Backup Operator privileges aren't disabled. For more information, see [Use the local web UI to administer your Data Box and Data Box Heavy](/azure/databox/data-box-local-web-ui-admin).
18+
19+
```console
20+
robocopy <Source> <Target> * /copyall /e /dcopy:DAT /B /r:3 /w:60 /is /nfl /ndl /np /MT:32 or 64 /fft /log+:<LogFile>
21+
```
22+
23+
where
24+
25+
|Option |Description |
26+
|------------------- | ----- |
27+
|`/copyall` |Copies all attributes.|
28+
|`/e` |Copies subdirectories, including empty directories. |
29+
|`/dcopy:DAT` |Copies data, attributes, and timestamps. Note: The /dcopy:DAT option must be used to transfer `CreationTime` on directories. |
30+
|`/B` |Copies files in Backup mode. |
31+
|`/r:3` |Specifies 3 retries on failed copies. |
32+
|`/w:60` |Specifies a wait time of 60 seconds between retries. |
33+
|`/is` |Includes the same files. |
34+
|`/nfl` |Does not log file names. |
35+
|`/ndl` |Does not log directory names. |
36+
|`/np` |Does not display progress of the copying operation. |
37+
|`/MT:32 or 64` |Uses multithreading, with 32 or 64 threads. |
38+
|`/fft` |Reduces time stamp granularity for any file system. |
39+
|`/log+:<LogFile>` |Appends the output to the existing log file.|
40+
41+
For more information on these `robocopy` parameters, see [Tutorial: Copy data to Azure Data Box via SMB](/azure/databox/data-box-deploy-copy-data)
42+
43+
> [!NOTE]
44+
> If you use `/copyall` to copy your data, the source ACLs on directories and files are transferred to Azure Files. If you only had read-access on your source data and could not modify the source data, you'll have read-access only on the data in the Data Box. Use `/copyall` only if you intend to copy all the ACLs on the directories and files along with the data.
45+
46+
#### Use robocopy to list, copy, modify files on Data Box
47+
48+
Here are some of the common scenarios you'll use when copying data using `robocopy`.
49+
50+
- **Copy only data to Data Box, no ACLs on directories and files**
51+
52+
Use the `/dcopy:DAT` option to only copy data, attributes, timestamps. ACLs on directories and files are not copied.
53+
54+
- **Copy data and ACLs on directories and files to Data Box**
55+
56+
Use `/copyall` to copy all the source data including all the ACLs on directories and files.
57+
58+
- **List the filesystem on Data Box using robocopy**
59+
60+
Use this command to list directory contents:
61+
62+
`robocopy <source-dir> NULL /l /s /xx /njh /njs /fp /B`
63+
64+
Note that the File Explorer doesn't allow you to list these files.
65+
66+
- **Copy or delete folders and files on Data Box**
67+
68+
Use this command to copy a single file:
69+
70+
`robocopy <source-dir> <destination-dir> <file-name> /B`
71+
72+
Use this command to delete a single file:
73+
74+
`robocopy <source-dir> <destination-dir> <file-name> /purge /B`
75+
76+
In the above command, the `<source-dir>` should not have the file: `<file-name>`. Then, the above command syncs the destination with the source, resulting in the removal of the file from the destination.
77+
78+
Note that the File Explorer may not allow you to perform the above operations.
79+
80+
For more information, see [Using robocopy commands](/windows-server/administration/windows-commands/robocopy).
81+
82+
### Linux data copy tools
83+
84+
Transferring metadata in Linux is a two-step process. First, you copy the source data using a tool such as `rsync`, which does not copy metadata. After you copy the data, you can copy the metadata using a tool such as `smbcacls` or `cifsacl`.
85+
86+
The following sample commands do the first step, copying the data using `rsync`.
87+
88+
```console
89+
cp -aR /etc /opt/
90+
rsync -avP /etc /opt (-a copies a directory)
91+
```

0 commit comments

Comments
 (0)