You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: crowdsec-docs/docs/log_processor/data_sources/s3.md
+7-1Lines changed: 7 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,6 +14,7 @@ To monitor a S3 bucket detecting new objects from a SQS queue:
14
14
source: s3
15
15
polling_method: sqs
16
16
sqs_name: test-sqs-s3-acquis
17
+
use_time_machine: true
17
18
labels:
18
19
type: foo
19
20
```
@@ -24,10 +25,15 @@ source: s3
24
25
polling_method: list
25
26
bucket_name: my_bucket
26
27
polling_interval: 30
28
+
use_time_machine: true
27
29
labels:
28
30
type: foo
29
31
```
30
32
33
+
:::warning
34
+
It is **strongly recommended** to set `use_time_machine: true` when using the S3 data source. Since files from S3 are not read in real time, the parser must rely on the timestamps within the log lines themselves to process events accurately.
35
+
:::
36
+
31
37
:::warning
32
38
The `list` polling method is mostly intended for testing purposes, and its usage is not recommended in production.
33
39
It won't work well with moderately big buckets (tens of thousands of files), as the listing operation is slow.
@@ -127,4 +133,4 @@ You can specify the `log_level` parameter to change the log level for the acquis
0 commit comments