Skip to content

Commit 6f7745e

Browse files
enhance: Document and update examples in s3 (#816)
1 parent eb2a5d7 commit 6f7745e

File tree

1 file changed

+7
-1
lines changed
  • crowdsec-docs/docs/log_processor/data_sources

1 file changed

+7
-1
lines changed

crowdsec-docs/docs/log_processor/data_sources/s3.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ To monitor a S3 bucket detecting new objects from a SQS queue:
1414
source: s3
1515
polling_method: sqs
1616
sqs_name: test-sqs-s3-acquis
17+
use_time_machine: true
1718
labels:
1819
type: foo
1920
```
@@ -24,10 +25,15 @@ source: s3
2425
polling_method: list
2526
bucket_name: my_bucket
2627
polling_interval: 30
28+
use_time_machine: true
2729
labels:
2830
type: foo
2931
```
3032
33+
:::warning
34+
It is **strongly recommended** to set `use_time_machine: true` when using the S3 data source. Since files from S3 are not read in real time, the parser must rely on the timestamps within the log lines themselves to process events accurately.
35+
:::
36+
3137
:::warning
3238
The `list` polling method is mostly intended for testing purposes, and its usage is not recommended in production.
3339
It won't work well with moderately big buckets (tens of thousands of files), as the listing operation is slow.
@@ -127,4 +133,4 @@ You can specify the `log_level` parameter to change the log level for the acquis
127133
crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log?log_level=debug
128134
```
129135

130-
AWS SDK behaviour can be configured with the standard AWS environment variables.
136+
AWS SDK behaviour can be configured with the standard AWS environment variables.

0 commit comments

Comments
 (0)