You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Merge pull request ceph#59311 from soumyakoduri/wip-skoduri-cloud-restore
rgw/cloudtier: initial MVP of the feature RestoreObject from cloud
Reviewed-by: Daniel Gryniewicz <[email protected]>
Reviewed-by: Matt Benjamin <[email protected]>
[`cloud-transition`](https://docs.ceph.com/en/latest/radosgw/cloud-transition) feature enables data transition to a remote cloud service as part of Lifecycle Configuration via Storage Classes. However the transition is unidirectional; data cannot be transitioned back from the remote zone.
6
+
7
+
The `cloud-restore` feature enables restoration of those transitioned objects from the remote cloud S3 endpoints back into RGW.
8
+
9
+
The objects can be restored either by using S3 `restore-object` CLI or via `read-through`. The restored copies can be either temporary or permanent.
10
+
11
+
## S3 restore-object CLI
12
+
13
+
The goal here is to implement minimal functionality of [`S3RestoreObject`](https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html) API so that users can restore the cloud transitioned objects.
14
+
15
+
```sh
16
+
aws s3api restore-object \
17
+
--bucket <value> \
18
+
--key <value> ( can be object name or *for Bulk restore) \
19
+
[--version-id <value>] \
20
+
--restore-request (structure) {
21
+
// for temporary restore
22
+
{ "Days": integer, }
23
+
// if Days not provided, it will be considered as permanent copy
24
+
}
25
+
```
26
+
27
+
This CLI may be extended in future to include custom parameters (like target-bucket/storage-class etc) specific to RGW.
28
+
29
+
## read-through
30
+
31
+
As per the cloud-transition feature functionality, the cloud-transitioned objects cannot be read. `GET` on those objects fails with ‘InvalidObjectState’ error.
32
+
33
+
But using this restore feature, transitioned objects can be restored and read. New tier-config options `allow_read_through` and `read_through_restore_days` are added for the same. Only when `allow_read_through` is enabled, `GET` on the transitioned objects will restore the objects from the S3 endpoint.
34
+
35
+
Note: The object copy restored via `readthrough` is temporary and is retained only for the duration of `read_through_restore_days`.
36
+
37
+
## Design
38
+
39
+
* Similar to cloud-transition feature, this feature currently works for**only s3 compatible cloud endpoint**.
40
+
* This feature works for only **cloud-transitioned objects**. In order to validate this, `retain_head_object` option should be set to true so that the object’s `HEAD` object can be verified before restoring the object.
41
+
42
+
***Request flow:**
43
+
* Once the `HEAD` object is verified, its cloudtier storage class config details are fetched.
44
+
Note: Incase the cloudtier storage-class is deleted/updated, the object may not be restored.
45
+
* RestoreStatus for the `HEAD` object is marked `RestoreAlreadyInProgress`
46
+
* Object Restore is done asynchronously by issuing either S3 `GET` or S3 `RESTORE` request to the remote endpoint.
47
+
* Once the object is restored, RestoreStaus is updated as `CloudRestored` and RestoreType is set to either `Temporary` or `Permanent`.
48
+
* Incase the operation fails, RestoreStatus is marked as `RestoreFailed`.
49
+
50
+
***New attrs:** Below are the new attrs being added
*`user.rgw.restore-expiry-date`: <Expiration time incase of temporary copies>
55
+
*`user.rgw.cloudtier_storage_class`: <CloudTier storage class used incase of temporarily restored copies>
56
+
57
+
```cpp
58
+
enum RGWRestoreStatus : uint8_t {
59
+
None = 0,
60
+
RestoreAlreadyInProgress = 1,
61
+
CloudRestored = 2,
62
+
RestoreFailed = 3
63
+
};
64
+
enum class RGWRestoreType : uint8_t {
65
+
None = 0,
66
+
Temporary = 1,
67
+
Permanent = 2
68
+
};
69
+
```
70
+
71
+
***Response:**
72
+
*`S3 restore-object CLI` returns SUCCESS - either the 200 OK or 202 Accepted status code.
73
+
* If the object is not previously restored, then RGW returns 202 Accepted in the response.
74
+
* If the object is previously restored, RGW returns 200 OK in the response.
75
+
* Special errors:
76
+
Code: RestoreAlreadyInProgress ( Cause: Object restore is already in progress.)
77
+
Code: ObjectNotFound (if Object is not found in cloud endpoint)
78
+
Code: I/O error (for any other I/O errors during restore)
79
+
*`GET request` continues to return an ‘InvalidObjectState’ error till the object is successfully restored.
80
+
* S3 head-object can be used to verify if the restore is still in progress.
81
+
* Once the object is restored, GET will return the object data.
82
+
83
+
***StorageClass**: By default, the objects are restored to `STANDARD` storage class. However, as per [AWS S3 Restore](https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html) the storage-class remains the same for restored objects. Hence for the temporary copies, the `x-amz-storage-class` returned contains original cloudtier storage-class.
84
+
* Note: A new tier-config option may be added to selectthe storage-class to restore the objects to.
85
+
86
+
***mtime**: If the restored object is temporary, object is still marked `RGWObj::CloudTiered` and mtime is not changed i.e, still set to transition time. But incase the object is permanent copy, it is marked `RGWObj::Main` and mtime is updated to the restore time (now()).
87
+
88
+
***Lifecycle**:
89
+
*`Temporary` copies are not subjected to any further transition to the cloud. However (as is the case with cloud-transitioned objects) they can be deleted via regular LC expiration rules or via external S3 Delete request.
90
+
*`Permanent` copies are treated as any regular objects and are subjected to any LC rules applicable.
91
+
92
+
***Replication**: The restored objects (both temporary and permanent) are also replicated like regular objects and will be deleted across the zones post expiration.
93
+
94
+
***VersionedObjects**: In case of versioning, if any object is cloud-transitioned, it would have been non-current. Post restore too, the same non-current object will be updated with the downloaded data and its HEAD object will be updated accordingly as the case with regular objects.
95
+
96
+
***Temporary Object Expiry**: This is done via Object Expirer
97
+
* When the object is restored as temporary, `user.rgw.expiry-date` is set accordingly and `delete_at` attr is also updated with the same value.
98
+
* This object is then added to the list used by `ObjectExpirer`.
99
+
*`LC` worker thread is used to scan through that list and post expiry, resets the objects back to cloud-transitioned state i.e,
100
+
* HEAD object with size=0
101
+
* new attrs removed
102
+
*`delete_at` reset
103
+
* Note: A new RGW option `rgw_restore_debug_interval` is added, which when set will be considered as `Days` value (similar to `rgw_lc_debug_interval`).
104
+
105
+
***FAILED Restore**: In case the restore operation fails,
106
+
* The HEAD object will be updated accordingly.. i.e, Storage-class is reset to the original cloud-tier storage class
107
+
* All the new attrs added will be removed , except for`user.rgw.restore-status` which will be updated as `RestoreFailed`
108
+
109
+
***Check Restore Progress**: Users can issue S3 `head-object` request to check if the restore is done or still in progress for any object.
110
+
111
+
***RGW down/restarts** - Since the restore operation is asynchronous, we need to keep track of the objects being restored. In case RGW is down/restarts, this data will be used to retrigger on-going restore requests or do appropriate cleanup for the failed requests.
112
+
113
+
***Compression** - If the placement-target to which the objects are being restored to has compression enabled, the data will be compressed accordingly (bug2294512)
114
+
115
+
***Encryption** - If the restored object is encrypted, the old sse-related xattrs/keys from the HEAD stub will be copied back into object metadata (bug2294512)
116
+
117
+
***Delete cloud object post restore** - Once the object is successfully restored, the object at the remote endpoint is still retained. However we could choose to delete it for permanent restored copies by adding new tier-config option.
118
+
119
+
## Future work
120
+
121
+
***Bulk Restore**: In the case of BulkRestore, some of the objects may not be restored. User needs to manually cross-check the objects to check the objects restored or InProgress.
122
+
123
+
***Admin CLIs**: Admin debug commands will be provided to start, check the status and cancel the restore operations.
0 commit comments