feat(log-ingestor): Submit compression jobs by metadata IDs (addresses #2018)

[jonathan-imanu]

Description

HOLD OFF ON REVIEWING UNTIL #2082 IS MERGED.

This PR is the builds off of #2082 and completes the proposal in #2018.

In #2082, we added support for a new input that contains the ingestor_job_id & metadata_ids so the scheduler can directly query the ingested_s3_object_metadata table for the metadata. This input was unused when #2082 was merged since thelog-ingestor was still submitting S3InputConfig.

This PR updates the log-ingestor to submit the new S3ObjectMetadataInputConfig. To achieve this, this PR:

• updates the buffer to hold a list of metadata ids as opposed to complete metadata objects
• introduces a new type CompressionBufferEntry to associate object size with metadata_id and updates interfaces of listener and submitter to use this type
• updates the unit tests of log-ingestor

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• task tests:rust-all
• Workflows
• TODO: Manual One-time verification of full flow

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2e72891e-51a4-42de-b15d-13611cab4931

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}