Skip to content

Commit 94016c7

Browse files
yuvalifyuvalif
committed
rgw/logging: support object metadata changes in journal mode
Fixes: https://tracker.ceph.com/issues/71255 Signed-off-by: Yuval Lifshitz <[email protected]>
1 parent 56d7e88 commit 94016c7

File tree

5 files changed

+120
-5
lines changed

5 files changed

+120
-5
lines changed

‎doc/radosgw/bucket_logging.rst‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ log data can be used to monitor bucket activity, detect unauthorized
1111
access, get insights into the bucket usage and use the logs as a journal for bucket changes.
1212
The log records are stored in objects in a separate bucket and can be analyzed later.
1313
Logging configuration is done at the bucket level and can be enabled or disabled at any time.
14-
The log bucket can accumulate logs from multiple buckets. It is recommended to configured
14+
The log bucket can accumulate logs from multiple buckets. It is recommended to configured
1515
a different "prefix" for each bucket, so that the logs of different buckets will be stored
1616
in different objects in the log bucket.
1717

@@ -50,12 +50,38 @@ This means that there are the logging operation may fail, with no indication to
5050

5151
Journal
5252
```````
53-
If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed.
54-
This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client.
55-
An exception to the above are "multi/delete" log records: if writing these log records fail, the operation continues and may still be successful.
53+
If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed.
54+
This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client
55+
Some exceptions exists to that rule, the "Fails Operation" columns in the table below indicate by "No" which operations will not fail even if logging failed.
5656
Journal mode supports filtering out records based on matches of the prefixes and suffixes of the logged object keys. Regular-expression matching can also be used on these to create filters.
5757
Note that it may happen that the log records were successfully written, but the bucket operation failed, since the logs are written.
5858

59+
The following operation are supported in journal mode:
60+
61+
+-------------------------------+-------------------------------------+-----------------+
62+
| Operation | Operation Name | Fails Operation |
63+
+===============================+=====================================+=================+
64+
| ``PutObject`` | ``REST.PUT.OBJECT`` | Yes |
65+
+-------------------------------+-------------------------------------+-----------------+
66+
| ``DeleteObject`` | ``REST.DELETE.OBJECT`` | No |
67+
+-------------------------------+-------------------------------------+-----------------+
68+
| ``DeleteObjects`` | ``REST.POST.DELETE_MULTI_OBJECT`` | No |
69+
+-------------------------------+-------------------------------------+-----------------+
70+
| ``CompleteMultipartUpload`` | ``REST.POST.UPLOAD`` | Yes |
71+
+-------------------------------+-------------------------------------+-----------------+
72+
| ``CopyObject`` | ``REST.PUT.OBJECT`` | Yes |
73+
+-------------------------------+-------------------------------------+-----------------+
74+
| ``PutObjectAcl`` | ``REST.PUT.ACL`` | Yes |
75+
+-------------------------------+-------------------------------------+-----------------+
76+
| ``PutObjectLegalHold`` | ``REST.PUT.LEGAL_HOLD`` | Yes |
77+
+-------------------------------+-------------------------------------+-----------------+
78+
| ``PutObjectRetention`` | ``REST.PUT.RETENTION`` | Yes |
79+
+-------------------------------+-------------------------------------+-----------------+
80+
| ``PutObjectTagging`` | ``REST.PUT.OBJECT_TAGGING`` | Yes |
81+
+-------------------------------+-------------------------------------+-----------------+
82+
| ``DeleteObjectTagging`` | ``REST.DELETE.OBJECT_TAGGING`` | No |
83+
+-------------------------------+-------------------------------------+-----------------+
84+
5985

6086
Bucket Logging Policy
6187
---------------------

‎doc/radosgw/s3/bucketops.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -788,7 +788,7 @@ Parameters are XML encoded in the body of the request, in the following format:
788788
| ``LoggingType`` | String | The type of logging. Valid values are: | No |
789789
| | | ``Standard`` (default) all bucket operations are logged after being perfomed. | |
790790
| | | The log record will contain all fields. | |
791-
| | | ``Journal`` only PUT, COPY, MULTI/DELETE and MPU operations are logged. | |
791+
| | | ``Journal`` only operations that modify and object are logged. | |
792792
| | | Will record the minimum subset of fields in the log record that is needed | |
793793
| | | for journaling. | |
794794
+-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+

‎src/rgw/rgw_op.cc‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1295,6 +1295,25 @@ void RGWPutObjTags::execute(optional_yield y)
12951295
return;
12961296
}
12971297

1298+
op_ret = s->object->get_obj_attrs(y, this);
1299+
if (op_ret < 0) {
1300+
ldpp_dout(this, 0) << "ERROR: failed to get obj attrs, obj=" << s->object
1301+
<< " ret=" << op_ret << dendl;
1302+
return;
1303+
}
1304+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
1305+
op_ret = rgw::bucketlogging::log_record(driver,
1306+
rgw::bucketlogging::LoggingType::Journal,
1307+
s->object.get(),
1308+
s,
1309+
canonical_name(),
1310+
etag,
1311+
s->object->get_size(),
1312+
this, y, false, false);
1313+
if (op_ret < 0) {
1314+
return;
1315+
}
1316+
12981317
s->object->set_atomic(true);
12991318
op_ret = s->object->modify_obj_attrs(RGW_ATTR_TAGS, tags_bl, y, this);
13001319
if (op_ret == -ECANCELED){
@@ -1329,6 +1348,25 @@ void RGWDeleteObjTags::execute(optional_yield y)
13291348
if (rgw::sal::Object::empty(s->object.get()))
13301349
return;
13311350

1351+
op_ret = s->object->get_obj_attrs(y, this);
1352+
if (op_ret < 0) {
1353+
ldpp_dout(this, 0) << "ERROR: failed to get obj attrs, obj=" << s->object
1354+
<< " ret=" << op_ret << dendl;
1355+
return;
1356+
}
1357+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
1358+
op_ret = rgw::bucketlogging::log_record(driver,
1359+
rgw::bucketlogging::LoggingType::Journal,
1360+
s->object.get(),
1361+
s,
1362+
canonical_name(),
1363+
etag,
1364+
s->object->get_size(),
1365+
this, y, false, false);
1366+
if (op_ret < 0) {
1367+
return;
1368+
}
1369+
13321370
op_ret = s->object->delete_obj_attrs(this, RGW_ATTR_TAGS, y);
13331371
}
13341372

@@ -6317,6 +6355,19 @@ void RGWPutACLs::execute(optional_yield y)
63176355
return;
63186356
}
63196357

6358+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
6359+
op_ret = rgw::bucketlogging::log_record(driver,
6360+
rgw::bucketlogging::LoggingType::Journal,
6361+
s->object.get(),
6362+
s,
6363+
canonical_name(),
6364+
etag,
6365+
s->object->get_size(),
6366+
this, y, false, false);
6367+
if (op_ret < 0) {
6368+
return;
6369+
}
6370+
63206371
bufferlist bl;
63216372
new_policy.encode(bl);
63226373
map<string, bufferlist> attrs;
@@ -8991,6 +9042,19 @@ void RGWPutObjRetention::execute(optional_yield y)
89919042
}
89929043
}
89939044

9045+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
9046+
op_ret = rgw::bucketlogging::log_record(driver,
9047+
rgw::bucketlogging::LoggingType::Journal,
9048+
s->object.get(),
9049+
s,
9050+
canonical_name(),
9051+
etag,
9052+
s->object->get_size(),
9053+
this, y, false, false);
9054+
if (op_ret < 0) {
9055+
return;
9056+
}
9057+
89949058
op_ret = s->object->modify_obj_attrs(RGW_ATTR_OBJECT_RETENTION, bl, s->yield, this);
89959059

89969060
return;
@@ -9093,6 +9157,26 @@ void RGWPutObjLegalHold::execute(optional_yield y) {
90939157
op_ret = -ERR_MALFORMED_XML;
90949158
return;
90959159
}
9160+
9161+
op_ret = s->object->get_obj_attrs(y, this);
9162+
if (op_ret < 0) {
9163+
ldpp_dout(this, 0) << "ERROR: failed to get obj attrs, obj=" << s->object
9164+
<< " ret=" << op_ret << dendl;
9165+
return;
9166+
}
9167+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
9168+
op_ret = rgw::bucketlogging::log_record(driver,
9169+
rgw::bucketlogging::LoggingType::Journal,
9170+
s->object.get(),
9171+
s,
9172+
canonical_name(),
9173+
etag,
9174+
s->object->get_size(),
9175+
this, y, false, false);
9176+
if (op_ret < 0) {
9177+
return;
9178+
}
9179+
90969180
bufferlist bl;
90979181
obj_legal_hold.encode(bl);
90989182
//if instance is empty, we should modify the latest object

‎src/rgw/rgw_rest.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -580,24 +580,28 @@ class RGWPutObjRetention_ObjStore : public RGWPutObjRetention {
580580
public:
581581
RGWPutObjRetention_ObjStore() = default;
582582
~RGWPutObjRetention_ObjStore() override = default;
583+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
583584
};
584585

585586
class RGWGetObjRetention_ObjStore : public RGWGetObjRetention {
586587
public:
587588
RGWGetObjRetention_ObjStore() = default;
588589
~RGWGetObjRetention_ObjStore() = default;
590+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
589591
};
590592

591593
class RGWPutObjLegalHold_ObjStore : public RGWPutObjLegalHold {
592594
public:
593595
RGWPutObjLegalHold_ObjStore() = default;
594596
~RGWPutObjLegalHold_ObjStore() override = default;
595597
int get_params(optional_yield y) override;
598+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
596599
};
597600

598601
class RGWGetObjLegalHold_ObjStore : public RGWGetObjLegalHold {
599602
public:
600603
RGWGetObjLegalHold_ObjStore() = default;
604+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
601605
~RGWGetObjLegalHold_ObjStore() = default;
602606
};
603607

‎src/rgw/rgw_rest_s3.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ class RGWDeleteObjTags_ObjStore_S3 : public RGWDeleteObjTags
8383
public:
8484
~RGWDeleteObjTags_ObjStore_S3() override {}
8585
void send_response() override;
86+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
8687
};
8788

8889
class RGWGetBucketTags_ObjStore_S3 : public RGWGetBucketTags_ObjStore

0 commit comments

Comments
 (0)