LOGC-42: Replace timestamp by startTime in log ordering and offset tracking

[dvasilas]

Problem

Log objects were being sorted by the internal Timestamp field (end time), but this field isn't written to the log output. The visible field in the logs is StartTime (AWS "Time" field). This made it impossible for readers to verify chronological ordering by examining the logs - the visible timestamps may not always match the actual sort order.

FIx

• Changed log-courier to sort by StartTime instead of Timestamp.
• Updated offsets to use StartTime instead of Timestamp.

Timestamp is no longer used by log-courier and can be removed from the access_logs table.

[jonathan-gramain]

Indentation looks off, maybe a round of go fmt is advisable

[fredmnl]

Should we consider this proposal from a discussion we had a while ago. We'd have a sortTS that would closely mimic StartTime except when insertedAt is more than X later than the StartTime (late delivery). We keep the guarantee that if you filter on sortTS is older than X, you will not get any new event inserted in there. You also get sorting done when the view materializes. I wonder if this will make the query more efficient too (especially the big one that looks for the bucket that have logs to ship).

Feel free to discuss it with me if you think this is worth it

[dvasilas]

@fredmnl Thanks for reminding me of this!
I did a bit of thinking and wrote a proposal here: https://scality.atlassian.net/wiki/spaces/OS/pages/3712614425/Bucket+server+access+logging+Idempotent+Batch+Processing+for+Log-Courier.
Let's discuss it.

I don't think we actually need sortTs, but I may be wrong around late-arriving objects.

I'd prefer to do this work in a separate PR (and merge this one).
Given the time constraints we have (we need end-to-end tests in log-courier -> bucket logging Integration tests this sprint), I'd prioritize it after we have Integration tests.
But it is a good time to have the discussion and maybe agree on the approach. If ClickHouse schema changes are needed it's better to do them before RC.

[codecov]

Codecov Report

❌ Patch coverage is 98.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 73.20%. Comparing base (be4e14b) to head (a3a5000).
⚠️ Report is 31 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/logcourier/logobject.go	92.85%	0 Missing and 1 partial ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
pkg/logcourier/batchfinder.go	94.78% <100.00%> (ø)
pkg/logcourier/logfetch.go	91.30% <100.00%> (-0.13%)	⬇️
pkg/logcourier/offset.go	91.07% <100.00%> (ø)
pkg/logcourier/offset_buffer.go	93.98% <ø> (ø)
pkg/logcourier/processor.go	79.38% <100.00%> (ø)
pkg/s3/client.go	95.55% <ø> (ø)
pkg/testutil/clickhouse.go	71.30% <100.00%> (-0.68%)	⬇️
pkg/logcourier/logobject.go	93.25% <92.85%> (-0.15%)	⬇️

@@            Coverage Diff             @@
##             main      #80      +/-   ##
==========================================
- Coverage   73.35%   73.20%   -0.15%     
==========================================
  Files          21       21              
  Lines        1625     1620       -5     
==========================================
- Hits         1192     1186       -6     
- Misses        347      348       +1     
  Partials       86       86              

Flag	Coverage Δ
unit	73.20% <98.00%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fredmnl]

I"ll check out the Confluence doc. I'm ok with merging this as is