Skip to content

Commit 7c89819

Browse files
yuvalifyuvalif
authored
Merge pull request ceph#63240 from yuvalif/wip-yuval-71255
rgw/logging: support object metadata changes in journal mode
2 parents 6854557 + 94016c7 commit 7c89819

File tree

5 files changed

+120
-5
lines changed

5 files changed

+120
-5
lines changed

‎doc/radosgw/bucket_logging.rst‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ log data can be used to monitor bucket activity, detect unauthorized
1111
access, get insights into the bucket usage and use the logs as a journal for bucket changes.
1212
The log records are stored in objects in a separate bucket and can be analyzed later.
1313
Logging configuration is done at the bucket level and can be enabled or disabled at any time.
14-
The log bucket can accumulate logs from multiple buckets. It is recommended to configured
14+
The log bucket can accumulate logs from multiple buckets. It is recommended to configured
1515
a different "prefix" for each bucket, so that the logs of different buckets will be stored
1616
in different objects in the log bucket.
1717

@@ -50,12 +50,38 @@ This means that there are the logging operation may fail, with no indication to
5050

5151
Journal
5252
```````
53-
If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed.
54-
This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client.
55-
An exception to the above are "multi/delete" log records: if writing these log records fail, the operation continues and may still be successful.
53+
If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed.
54+
This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client
55+
Some exceptions exists to that rule, the "Fails Operation" columns in the table below indicate by "No" which operations will not fail even if logging failed.
5656
Journal mode supports filtering out records based on matches of the prefixes and suffixes of the logged object keys. Regular-expression matching can also be used on these to create filters.
5757
Note that it may happen that the log records were successfully written, but the bucket operation failed, since the logs are written.
5858

59+
The following operation are supported in journal mode:
60+
61+
+-------------------------------+-------------------------------------+-----------------+
62+
| Operation | Operation Name | Fails Operation |
63+
+===============================+=====================================+=================+
64+
| ``PutObject`` | ``REST.PUT.OBJECT`` | Yes |
65+
+-------------------------------+-------------------------------------+-----------------+
66+
| ``DeleteObject`` | ``REST.DELETE.OBJECT`` | No |
67+
+-------------------------------+-------------------------------------+-----------------+
68+
| ``DeleteObjects`` | ``REST.POST.DELETE_MULTI_OBJECT`` | No |
69+
+-------------------------------+-------------------------------------+-----------------+
70+
| ``CompleteMultipartUpload`` | ``REST.POST.UPLOAD`` | Yes |
71+
+-------------------------------+-------------------------------------+-----------------+
72+
| ``CopyObject`` | ``REST.PUT.OBJECT`` | Yes |
73+
+-------------------------------+-------------------------------------+-----------------+
74+
| ``PutObjectAcl`` | ``REST.PUT.ACL`` | Yes |
75+
+-------------------------------+-------------------------------------+-----------------+
76+
| ``PutObjectLegalHold`` | ``REST.PUT.LEGAL_HOLD`` | Yes |
77+
+-------------------------------+-------------------------------------+-----------------+
78+
| ``PutObjectRetention`` | ``REST.PUT.RETENTION`` | Yes |
79+
+-------------------------------+-------------------------------------+-----------------+
80+
| ``PutObjectTagging`` | ``REST.PUT.OBJECT_TAGGING`` | Yes |
81+
+-------------------------------+-------------------------------------+-----------------+
82+
| ``DeleteObjectTagging`` | ``REST.DELETE.OBJECT_TAGGING`` | No |
83+
+-------------------------------+-------------------------------------+-----------------+
84+
5985

6086
Bucket Logging Policy
6187
---------------------

‎doc/radosgw/s3/bucketops.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -790,7 +790,7 @@ Parameters are XML encoded in the body of the request, in the following format:
790790
| ``LoggingType`` | String | The type of logging. Valid values are: | No |
791791
| | | ``Standard`` (default) all bucket operations are logged after being perfomed. | |
792792
| | | The log record will contain all fields. | |
793-
| | | ``Journal`` only PUT, COPY, MULTI/DELETE and MPU operations are logged. | |
793+
| | | ``Journal`` only operations that modify and object are logged. | |
794794
| | | Will record the minimum subset of fields in the log record that is needed | |
795795
| | | for journaling. | |
796796
+-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+

‎src/rgw/rgw_op.cc‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1295,6 +1295,25 @@ void RGWPutObjTags::execute(optional_yield y)
12951295
return;
12961296
}
12971297

1298+
op_ret = s->object->get_obj_attrs(y, this);
1299+
if (op_ret < 0) {
1300+
ldpp_dout(this, 0) << "ERROR: failed to get obj attrs, obj=" << s->object
1301+
<< " ret=" << op_ret << dendl;
1302+
return;
1303+
}
1304+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
1305+
op_ret = rgw::bucketlogging::log_record(driver,
1306+
rgw::bucketlogging::LoggingType::Journal,
1307+
s->object.get(),
1308+
s,
1309+
canonical_name(),
1310+
etag,
1311+
s->object->get_size(),
1312+
this, y, false, false);
1313+
if (op_ret < 0) {
1314+
return;
1315+
}
1316+
12981317
s->object->set_atomic(true);
12991318
op_ret = s->object->modify_obj_attrs(RGW_ATTR_TAGS, tags_bl, y, this);
13001319
if (op_ret == -ECANCELED){
@@ -1329,6 +1348,25 @@ void RGWDeleteObjTags::execute(optional_yield y)
13291348
if (rgw::sal::Object::empty(s->object.get()))
13301349
return;
13311350

1351+
op_ret = s->object->get_obj_attrs(y, this);
1352+
if (op_ret < 0) {
1353+
ldpp_dout(this, 0) << "ERROR: failed to get obj attrs, obj=" << s->object
1354+
<< " ret=" << op_ret << dendl;
1355+
return;
1356+
}
1357+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
1358+
op_ret = rgw::bucketlogging::log_record(driver,
1359+
rgw::bucketlogging::LoggingType::Journal,
1360+
s->object.get(),
1361+
s,
1362+
canonical_name(),
1363+
etag,
1364+
s->object->get_size(),
1365+
this, y, false, false);
1366+
if (op_ret < 0) {
1367+
return;
1368+
}
1369+
13321370
op_ret = s->object->delete_obj_attrs(this, RGW_ATTR_TAGS, y);
13331371
}
13341372

@@ -6332,6 +6370,19 @@ void RGWPutACLs::execute(optional_yield y)
63326370
return;
63336371
}
63346372

6373+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
6374+
op_ret = rgw::bucketlogging::log_record(driver,
6375+
rgw::bucketlogging::LoggingType::Journal,
6376+
s->object.get(),
6377+
s,
6378+
canonical_name(),
6379+
etag,
6380+
s->object->get_size(),
6381+
this, y, false, false);
6382+
if (op_ret < 0) {
6383+
return;
6384+
}
6385+
63356386
bufferlist bl;
63366387
new_policy.encode(bl);
63376388
map<string, bufferlist> attrs;
@@ -9006,6 +9057,19 @@ void RGWPutObjRetention::execute(optional_yield y)
90069057
}
90079058
}
90089059

9060+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
9061+
op_ret = rgw::bucketlogging::log_record(driver,
9062+
rgw::bucketlogging::LoggingType::Journal,
9063+
s->object.get(),
9064+
s,
9065+
canonical_name(),
9066+
etag,
9067+
s->object->get_size(),
9068+
this, y, false, false);
9069+
if (op_ret < 0) {
9070+
return;
9071+
}
9072+
90099073
op_ret = s->object->modify_obj_attrs(RGW_ATTR_OBJECT_RETENTION, bl, s->yield, this);
90109074

90119075
return;
@@ -9108,6 +9172,26 @@ void RGWPutObjLegalHold::execute(optional_yield y) {
91089172
op_ret = -ERR_MALFORMED_XML;
91099173
return;
91109174
}
9175+
9176+
op_ret = s->object->get_obj_attrs(y, this);
9177+
if (op_ret < 0) {
9178+
ldpp_dout(this, 0) << "ERROR: failed to get obj attrs, obj=" << s->object
9179+
<< " ret=" << op_ret << dendl;
9180+
return;
9181+
}
9182+
const auto etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
9183+
op_ret = rgw::bucketlogging::log_record(driver,
9184+
rgw::bucketlogging::LoggingType::Journal,
9185+
s->object.get(),
9186+
s,
9187+
canonical_name(),
9188+
etag,
9189+
s->object->get_size(),
9190+
this, y, false, false);
9191+
if (op_ret < 0) {
9192+
return;
9193+
}
9194+
91119195
bufferlist bl;
91129196
obj_legal_hold.encode(bl);
91139197
//if instance is empty, we should modify the latest object

‎src/rgw/rgw_rest.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -583,24 +583,28 @@ class RGWPutObjRetention_ObjStore : public RGWPutObjRetention {
583583
public:
584584
RGWPutObjRetention_ObjStore() = default;
585585
~RGWPutObjRetention_ObjStore() override = default;
586+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
586587
};
587588

588589
class RGWGetObjRetention_ObjStore : public RGWGetObjRetention {
589590
public:
590591
RGWGetObjRetention_ObjStore() = default;
591592
~RGWGetObjRetention_ObjStore() = default;
593+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
592594
};
593595

594596
class RGWPutObjLegalHold_ObjStore : public RGWPutObjLegalHold {
595597
public:
596598
RGWPutObjLegalHold_ObjStore() = default;
597599
~RGWPutObjLegalHold_ObjStore() override = default;
598600
int get_params(optional_yield y) override;
601+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
599602
};
600603

601604
class RGWGetObjLegalHold_ObjStore : public RGWGetObjLegalHold {
602605
public:
603606
RGWGetObjLegalHold_ObjStore() = default;
607+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
604608
~RGWGetObjLegalHold_ObjStore() = default;
605609
};
606610

‎src/rgw/rgw_rest_s3.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ class RGWDeleteObjTags_ObjStore_S3 : public RGWDeleteObjTags
8383
public:
8484
~RGWDeleteObjTags_ObjStore_S3() override {}
8585
void send_response() override;
86+
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
8687
};
8788

8889
class RGWGetBucketTags_ObjStore_S3 : public RGWGetBucketTags_ObjStore

0 commit comments

Comments
 (0)