You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: doc/radosgw/bucket_logging.rst
+68-45Lines changed: 68 additions & 45 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,60 +6,74 @@ Bucket Logging
6
6
7
7
.. contents::
8
8
9
-
Bucket logging provides a mechanism for logging all access to a bucket. The
10
-
log data can be used to monitor bucket activity, detect unauthorized
11
-
access, get insights into the bucket usage and use the logs as a journal for bucket changes.
12
-
The log records are stored in objects in a separate bucket and can be analyzed later.
13
-
Logging configuration is done at the bucket level and can be enabled or disabled at any time.
14
-
The log bucket can accumulate logs from multiple buckets. It is recommended to configured
15
-
a different "prefix" for each bucket, so that the logs of different buckets will be stored
16
-
in different objects in the log bucket.
9
+
Bucket logging provides a mechanism for logging all access to a bucket. The log
10
+
data can be used to monitor bucket activity, detect unauthorized access, get
11
+
insights into bucket usage, and use the logs as a journal for bucket
12
+
changes. The log records are stored in objects in a separate bucket and can be
13
+
analyzed later. Logging configuration is done at the bucket level and can be
14
+
enabled or disabled at any time. The log bucket can accumulate logs from
15
+
multiple buckets. It is recommended to configure a different "prefix" for each
16
+
bucket, so that the logs of different buckets will be stored in different
17
+
objects in the log bucket.
17
18
18
19
.. note::
19
20
20
21
- The log bucket must be created before enabling logging on a bucket
21
22
- The log bucket cannot be the same as the bucket being logged
22
23
- The log bucket cannot have logging enabled on it
23
-
- The log bucket cannot have any encryption set on in (including SSE-S3 with AES-256)
24
+
- The log bucket cannot have any encryption set on it (including SSE-S3
25
+
with AES-256)
24
26
- The log bucket cannot have any compression set on it
25
27
- The log bucket must not have RequestPayer enabled
26
-
- Source and log bucket must be in the same zonegroup
27
-
- Source and log buckets may belong to different accounts (with proper bucket policy set)
28
+
- Source and log buckets must be in the same zonegroup
29
+
- Source and log buckets may belong to different accounts (with proper
30
+
bucket policy set)
28
31
- The log bucket may have object lock enabled with default retention period
29
-
- The 16 byte unique ID part of the log object name is a lexicographically ordered random string,
30
-
that consists of a 10 bytes counter, and a 6 bytes random alphanumeric string.
31
-
or a random, alphanumeric string if the counter is not available
32
+
- The 16-byte unique ID part of the log object name is a lexicographically
33
+
ordered random string that consists of a 10-byte counter and a 6-byte
34
+
random alphanumeric string (or a random alphanumeric string if the
35
+
counter is not available)
32
36
33
37
34
38
.. toctree::
35
39
:maxdepth:1
36
40
37
41
Logging Reliability
38
42
-------------------
39
-
For performance reasons, even though the log records are written to persistent storage, the log object will
40
-
appear in the log bucket only after some configurable amount of time (or if the maximum object size of 128MB is reached).
41
-
This time (in seconds) could be set per source bucket via a Ceph extension to the REST API,
42
-
or globally via the `rgw_bucket_logging_obj_roll_time` configuration option. If not set, the default time is 5 minutes.
43
-
Adding a log object to the log bucket is done "lazily", meaning, that if no more records are written to the object, it may
44
-
remain outside of the log bucket even after the configured time has passed.
45
-
To counter that, you can flush all logging objects on a given source bucket to log them,
46
-
regardless if enough time passed or if no more records are written to the object.
47
-
Flushing will happen automatically when logging is disabled on a bucket, its logging configuration is changed, or the bucket is deleted.
43
+
For performance reasons, even though the log records are written to persistent
44
+
storage, the log object will appear in the log bucket only after some
45
+
configurable amount of time (or if the maximum object size of 128MB is
46
+
reached). This time (in seconds) can be set per source bucket via a Ceph
47
+
extension to the REST API, or globally via the
48
+
`rgw_bucket_logging_obj_roll_time` configuration option. If not set, the
49
+
default time is 5 minutes. Adding a log object to the log bucket is done
50
+
"lazily", meaning that if no more records are written to the object, it may
51
+
remain outside of the log bucket even after the configured time has passed. To
52
+
counter that, you can flush all logging objects on a given source bucket to log
53
+
them, regardless if enough time passed or if no more records are written to the
54
+
object. Flushing will happen automatically when logging is disabled on a
55
+
bucket, or its logging configuration is changed, or the bucket is deleted.
48
56
49
57
Standard
50
58
````````
51
-
If logging type is set to "Standard" (the default) the log records are written to the log bucket after the bucket operation is completed.
52
-
This means that there are the logging operation may fail, with no indication to he client.
59
+
If the logging type is set to "Standard" (the default) the log records are
60
+
written to the log bucket after the bucket operation is completed. This means
61
+
that the logging operation may fail with no indication to the client.
53
62
54
63
Journal
55
64
```````
56
-
If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed.
57
-
This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client
58
-
Some exceptions exists to that rule, the "Fails Operation" columns in the table below indicate by "No" which operations will not fail even if logging failed.
59
-
Journal mode supports filtering out records based on matches of the prefixes and suffixes of the logged object keys. Regular-expression matching can also be used on these to create filters.
60
-
Note that it may happen that the log records were successfully written, but the bucket operation failed, since the logs are written.
61
-
62
-
The following operation are supported in journal mode:
65
+
If the logging type is set to "Journal", the records are written to the log
66
+
bucket before the bucket operation is completed. This means that if the logging
67
+
action fails, the operation is not executed and an error is returned to the
68
+
client. Some exceptions to that rule exist: the "Fails Operation" columns in
69
+
the table below indicate by "No" which operations will not fail even if logging
70
+
failed. Journal mode supports filtering out records based on matches of the
71
+
prefixes and suffixes of the logged object keys. Regular-expression matching
72
+
can also be used on these to create filters. Note that it may happen that the
73
+
log records were successfully written but the bucket operation failed, since
74
+
the logs are written.
75
+
76
+
The following operations are supported in journal mode:
@@ -87,16 +101,21 @@ The following operation are supported in journal mode:
87
101
88
102
Multisite
89
103
`````````
90
-
In a multi-zone deployment, each zone will use its own log object before the log object is added to the log bucket.
91
-
After the log object is added to the log bucket (e.g after being flushed) it will be replicated to other zones.
92
-
This means that for a given time period there could be more than one log object holding relevant log records.
104
+
In a multi-zone deployment, each zone uses its own log object before the
105
+
log object is added to the log bucket. After the log object is added to the
106
+
log bucket (that is, after being flushed) it is replicated to other zones.
107
+
This means that for a given time period there can be more than one log object
108
+
holding relevant log records.
93
109
94
110
Bucket Logging Policy
95
111
---------------------
96
-
On the source bucket, only its owner is allowed to enable or disable bucket logging.
97
-
For a bucket to be used as a log bucket, it must have bucket policy that allows that (even if the source bucket and the log bucket are owned by the same user or account).
98
-
The bucket policy must allow the `s3:PutObject` action for the log bucket, to be performed by the `logging.s3.amazonaws.com` service principal.
99
-
It should also specify the source bucket and account that are expected to write logs to it. For example:
112
+
On the source bucket, only its owner is allowed to enable or disable bucket
113
+
logging. For a bucket to be used as a log bucket, it must have a bucket policy
114
+
that allows that (even if the source bucket and the log bucket are owned by the
115
+
same user or account). The bucket policy must allow the `s3:PutObject` action
116
+
for the log bucket, to be performed by the `logging.s3.amazonaws.com` service
117
+
principal. The bucket policy should also specify the source bucket and account
118
+
that are expected to write logs to it. For example:
100
119
101
120
::
102
121
@@ -126,9 +145,11 @@ It should also specify the source bucket and account that are expected to write
126
145
127
146
Bucket Logging Quota
128
147
--------------------
129
-
Bucket and user quota are applied on the log bucket. Quota is checked every time a log record is written,
130
-
and updated when the log object is added to the log bucket. In "Journal" mode, if the quota is exceeded, the logging operation will fail
131
-
and as a result the bucket operation will also fail. In "Standard" mode, the logging operation will be skipped, but the bucket operation will continue.
148
+
Bucket and user quota are applied on the log bucket. Quota is checked every
149
+
time a log record is written, and is updated when the log object is added to
150
+
the log bucket. In "Journal" mode, if the quota is exceeded, the logging
151
+
operation fails and as a result the bucket operation also fails. In "Standard"
152
+
mode, the logging operation is skipped, but the bucket operation continues.
132
153
133
154
134
155
Bucket Logging REST API
@@ -170,14 +191,16 @@ For example:
170
191
Log Records
171
192
~~~~~~~~~~~
172
193
173
-
The log records are space separated string columns and have the following possible formats:
194
+
The log records are space-separated string columns and have the following
195
+
possible formats:
174
196
175
197
Journal
176
198
```````
177
-
minimum amount of data used for journaling bucket changes (this is a Ceph extension).
199
+
minimum amount of data used for journaling bucket changes (this is a Ceph
200
+
extension).
178
201
179
202
- bucket owner (or dash if empty)
180
-
- bucket name (or dash if empty). in the format: ``[tenant:]<bucket name>``
203
+
- bucket name (or dash if empty), in the format: ``[tenant:]<bucket name>``
181
204
- time in the following format: ``[day/month/year:hour:minute:second timezone]``
182
205
- object key (or dash if empty)
183
206
- operation in the following format: ``WEBSITE/REST.<HTTP method>.<resource>``
0 commit comments