Skip to content

Fix timestamp precision mismatch causing S3 input to skip backup and delete operations#60

Open
yongkyun wants to merge 1 commit intologstash-plugins:mainfrom
yongkyun:time-comparison-fix
Open

Fix timestamp precision mismatch causing S3 input to skip backup and delete operations#60
yongkyun wants to merge 1 commit intologstash-plugins:mainfrom
yongkyun:time-comparison-fix

Conversation

@yongkyun
Copy link
Contributor

@yongkyun yongkyun commented Dec 9, 2025

Summary

This PR fixes an issue where the S3 input plugin fails to delete processed objects when using certain S3-compatible storage services (e.g., Cloudflare R2).
The problem is caused by a mismatch in timestamp precision between values returned by different APIs, which makes the delete condition fail.

Cause

  • Some S3-compatible services such as Cloudflare R2 return the object’s last_modified timestamp with millisecond precision when using the ListObjectsV2 API.
  • AWS S3 also returns last_modified with millisecond precision via ListObjectsV2, but in practice all millisecond values are 0.
  • Inside the process_log(queue, log) function, when backing up and deleting files, the plugin compares:
    • the log.last_modified timestamp (with millisecond precision, coming from the list operation), and
    • the object’s last_modified timestamp obtained via the HeadObject API (which effectively has second-level precision).
  • Because of this difference in precision between the two timestamps, the millisecond part often does not match, causing the comparison to fail.
  • As a result, the plugin never proceeds to back up or delete the file, and processed objects remain in the bucket.

What this PR changes

  • Normalize both timestamps by converting them to integer seconds (to_i) before comparing:

elsif log.last_modified.to_i <= sincedb_time.to_i
if object.last_modified.to_i == log.last_modified.to_i

Related issues

Closes #45

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 input does not delete files

1 participant