Support durable log forwarding in seqcli

[KodrAus]

It would be beneficial to merge Seq Forwarder into seqcli. This would let us share a lot of the log shipping functionality they have in common while enhancing existing seqcli commands related to ingestion with bookmarking from Seq Forwarder. It will simplify how we deploy Seq Forwarder, making its functionality available to seqcli users on Windows or Docker on any platform already supported by seqcli.

The CLI may look something like:

seqcli forwarder run --listen "http://localhost:5341" --storage "/seqcli-logs"

And on Windows, some additional commands for managing installation as a Windows service:

seqcli forwarder install
seqcli forwarder start
seqcli forwarder stop
seqcli forwarder remove

As part of this, we'd like to replace the LMDB-based disk buffer with something bespoke that's easier to manage and doesn't rely on the host to install libraries to support it. The transition from LMDB to the new format will be handled in Seq Forwarder, so existing users can let their old buffers age out before migrating to seqcli.

We also need to overhaul how secrets are managed by seqcli to better support secure deployments on non-Windows platforms.

This work effectively deprecates the Seq Forwarder repository, which will enter maintenance mode.

[KodrAus]

Apologies to anybody currently watching this repository and getting swamped by our barely coherent broken PRs to a feature/forwarder branch 😅

We're currently working towards getting an MVP of this into the tree. Once it's in reasonable shape we'll write up some more details about what's there and how it all works.

[CrossBound]

I use Seq Forwarder heavily in a production environment. I see from datalust/seq-forwarder#75 that the old repo is moving here, but it looks like no work has been done since 2024?? I would really like to have a modernized version of the fowarder to work with, specifically running as a container in a K8s environment. Is there anything I can do to help move this migration along? I was thinking I might create a PR in the forwarder repo but if that is a dead project it sounds like that won't help any.

[CrossBound]

Just to expand on this... currently I'm trying to get tracing working in our environment using SerilogTracing, but the trace information is not being passed through properly when it goes through the forwarder.

[nblumhardt]

Thanks for the nudge, @CrossBound. Unfortunately work on this has been stalled for a little while due to 2024's long excursion into clustering, but I've just merged the latest dev changes into feature/forwarder and PR'd the change here: #402

We're due to sync up again on the schedule for completing this work, and might be able to identify a quicker path forwards than the (fairly extensive) one we were on. I'll aim to post an update here tomorrow :-)

[nblumhardt]

Hi @CrossBound; we're planning to merge #403 shortly, which will get the bones of seqcli forwarder onto dev behind a --pre feature flag.

Once that happens, we'll be able to divide and conquer to tidy things up and nail down the initial feature set - there should be a few opportunities to dig in at that point, if you're keen. After the merge I'll post a (fairly long 😅) TODO list here. Cheers!

[CrossBound]

Thank you. I'd like to help get this to the finish line if I can. Where do you have the list of things that need to be done before basic forwarding can be usable?

[liammclennan]

It is done.

[nblumhardt]

Hi @CrossBound, we could definitely use a hand, thanks! 🙏

The setup instructions are in:

https://github.com/datalust/seqcli?tab=readme-ov-file#store-and-forward-ingestion-proxy-preview

Right now, if by "over the finish line" we're talking wide production deployment, the main priority is getting experience with the new codebase in the field, and feedback on how it holds up.

As @liammclennan suggests, the basic functionality is done 🙂, but removing the --pre flag will depend on us building some confidence in it, which we need wider input to achieve.