Thoughts on failure reporting in ServiceGroup:run()

[Jerry-Carter]

A confession from a noob: I did not use ServiceLifecycle in the intended manner. "Mea culpa, mea culpa, mea maxima culpa", but hear me out...

My application has two services: GoodService and BadService. The former does everything expected by ServiceLifecycle. The latter ignores cancellation and shutdown requests. The code is quite vanilla and follows the Readme.

let serviceGroup = ServiceGroup(services: [GoodService(), BadService()], gracefulShutdownSignals: [.sigterm], logger: logger)

do {
    logger.info("Starting Server on port \(port) with pid \(pid ?? "unspecified")")
    try await serviceGroup.run()
} catch {
    logger.error("Exception while running: \(error.localizedDescription)")
}

Scenario: BadService throws an exception: ServiceLifecycle catches the exception and cancels GoodService. The exception is rethrown and my code catches it. ✅

Scenario: GoodService throws an exception: ServiceLifecycle catches the exception and tries to cancel BadService, which ignores the cancellation. The ServiceGroup continues to wait indefinitely. No errors are logged or exceptions thrown. The server continue to run. ❎❎

Obviously, the second case would benefit from using ServiceGroupConfiguration and setting the maximumCancellationDuration and maximumGracefulShutdownDuration. It might be worth calling attention to the existence of the configuration in either the readme or "How to adopt ServiceLifecycle in ____" articles.

Scenario: With durations specifed, GoodService throws an exception: Now the server exits with "Fatal error: Cancellation took longer than allowed by the configuration." which is better, but the original exception is lost so important information for debugging is lost. ❎

Scenario: I have two good services and both throw an exception: From my read of the code, only one exception is passed to the application. Again, potentially valuable debugging information is lost. ❎

Assuming everything I've said above is correct or mostly correct, let me bravely attempt to suggest a correction. A logger is already being passed to ServiceGroup. May I suggest that

• Upon entering run(), a message be logged with the number of services (perhaps .info). This is optional as it could be done instead by the application author.
• Upon cancellation, a message be logged indicating that cancellation has been requested (perhaps .info). Again optional if triggered by the application author. But if ServiceGroup is initiating the cancellation, I'd like to know why.
• Upon encountering an exception, that exception should be logged immediately with a message that the other members of the group are being cancelled (perhaps .error)
• If cancellation takes a long time (say a minute), a message be logged indicating which services are still running. This could repeat or not. Simply having it could be very helpful. (perhaps .notify)

[czechboy0]

Did you check with .debug logging? IIRC SSL actually does produce pretty detailed logs, as long as you bump your logger's log level.

[Jerry-Carter]

Let me start by complimenting the debug level logging from ServiceGroup within ServiceLifecycle. The messages are clear and thorough. I appreciate the ability to set metadata key names. Let me also acknowledge that logging is a tricky subject and that one user’s requirements for ServiceGroup will probably not match those of a second user. My initial comment was well intentioned but perhaps a bit shortsighted given the many situations where ServiceLifecycle might be employed. I agree with the guidance for libraries in the Log Level document. I believe that there is room for improving ServiceLifecycle; I will attempt to offer a more refined proposal.

In development, .trace or .debug may be used regularly, but, in production, .info or higher is the norm (at least from my experience). Most library logging must disappears below .info, but there are certain key events that may require attention from the operations team. It is largely reasonable to place the burden on the user of the library as the user is best positioned to evaluate the significance of key events in the broader application context. But this does not mean that the actual log messages best originate in code written by the library user. The library may have more information available internally than the user of that library or at an earlier point in time.

Proposal:

1. Make the log levels configurable in ServiceGroupConfiguration for key events (akin to the LoggingConfiguration.Keys) with a default value of .debug for backward compatibility. Details in next section.
2. Add content to LoggingConfiguration.md detailing the LoggingConfiguration.Keys and the new LoggingConfiguration.LogLevels.
3. Add content to ServiceConfiguration.md. Specifically, add discuss the maximumGracefulShutdownDuration & maximumCancellationDuration parameters and mention that the logging configuration is adjustable and is detailed in LoggingConfiguration.md.

Key Events:
These four events would, I believe, benefit from adjustable log levels:

1. ServiceGroupStart. The application could log a message before calling ServiceGroup:run(). That stated, the application logging cannot be regarded as authoritative; ServiceGroup best knows what configuration it’s actually using.
   2025-07-04T18:06:09-0400 debug OtherServe : sl-cancellationSignals=[] sl-gracefulShutdownSignals=[SIGTERM] sl-services=[MyServer.BadService, MyServer.GoodService] [ServiceLifecycle] Starting service lifecycle
2. TerminationAfterFinish. “Service finished unexpectedly. Cancelling group.” / “Service finished. Gracefully shutting down group.” The cancellation / graceful termination process can take time. This allows the operations team to start responding immediately. These two cases could be split.
3. TerminationAfterError. “Service threw error. Cancelling group.” / “Service threw error. Shutting down group.” The cancellation / graceful termination process can take time. This allows the operations team to start responding immediately. Again, these two cases could be split.
4. TerminationTimeout. “Cancellation took longer than allowed by the configuration.” This is the last message that passes through swift-log before fatalError().

Additional Comments:
Adding this capability to ServiceGroupConfiguration allows the library user to adjust the log levels for each ServiceGroup. If the group is key to the project, the log levels can be set to .critical or .error. If the group is not core, then .notice or .info may be appropriate. By keeping the default value at .debug, there should be no impact on existing users of ServiceLifecycle.

If this direction seems reasonable, I am happy to do this work and generate a pull request.

[czechboy0]

Asking each library in the ecosystem to allow customizing the level at which it emits logging is one way, but I believe that Swift Log already has a solution for this: provide a custom LogHandler (the "backend"), which makes its own decision about increasing/decreasing log levels, and wrap your existing log handler.

In your case, you could write:

struct LevelAdjustingHandler<Upstream: LogHandler>: LogHandler {
    var upstream: Upstream

    func log(
        level: Logger.Level,
        message: Logger.Message,
        metadata: Logger.Metadata?,
        source: String,
        file: String,
        function: String,
        line: UInt
    ) {
        var newLevel = level
        // here inspect source, file, function, line - whichever ones are useful to identify the critical code
        // and change newLevel to the more severe one
        upstream.log(
            level: newLevel,
            message: message,
            metadata: metadata,
            source: source,
            file: file,
            function: function,
            line: line
        )
    }
}

And then bootstrap LevelAdjustingHandler wrapping your original log handler.

I believe this allows you to customize the log level of any messages in any library, without the libraries needing to change. My understanding is that this is how Swift Log was designed to do it: libraries emit "naively", log handlers is where the application-specific, smart logic is provided.

[czechboy0]

Separately, if there are events that aren't logged today that you need, those can certainly be added, but still at the debug/trace level, and you can escalate them in your custom log handler.

[Jerry-Carter]

One of the first things we did after encountering swift-log was to create a custom LogHandler to route messages for developers, for the operation team, and for alert generation. It's a useful and nearly essential element of swift-log. I have a pair of objections to placing additional logic into the LogHandler.

First, logging operations are generally high throughput functions. Logging is to be encouraged and log functions are expected to be highly efficient to minimize the performance impact. Every additional bit of logic that must be calculated in this layer erodes that promise.

Second, looking for specific message structures introduces fragility. An innocent change to swift-service-lifecycle can break logic that depends on details of the old messages. The most interesting events tend to be some of the hardest to test. As a result, code for reporting issues can break silently.

As an observation rather than an objection, this goes against my understanding of the spirit of swift-log. Currently the bootstrap process follows the factory method pattern to allow specific LogHander implementations to be created by Logger without knowing the specifics of the LogHandler. LoggingSystem.bootstrap is called once early on and every subsequent Logger uses the same LogHandler implementation. It is possible to create a LogHandler specific to ServiceGroup to minimize the associated processing and then unit test that LogHandler. This will work, but does not seem to be the normal way.

Finally, let me close with language from the Swift Log Level document: "Some libraries and situations may not be entirely clear with regard to what log level is “best” for them. In such situations, it sometimes is worth it to allow the end-users of the library to be able to configure the levels of specific groups of messages."

[czechboy0]

Yeah it's completely reasonable to offer configuring the log level. It's also reasonable for libraries to not take on that extra maintenance burden brought by that customizability. For example - should Vapor offer a struct with one property for each location where it logs, to offer this level of flexibility? Your points are sound - any one of those locations could be crucial for an operations team to build an alert around.

My view is that the text, metadata, and level is up to the library. And which log events are important is up to the application. While I agree the brittleness argument is valid, I suspect it's a fact of life that how a library logs is not part of the API contract and it's free to completely change that even in patch versions. Extrapolating from that, I'd put any logic important for the application to function on the side of the application, rather than to try to push it up into each library. Because I suspect you'll never be able to fully remove your custom log handlers (unless the ecosystem as a whole makes all logging customizable in each library's API), so the question is why you'd prefer to have two ways of doing the same thing (tweaking the log level of events emitted by a dependency).

All in all, I don't think individual logging events should be part of ServiceLifecycle's API, as it removes flexibility and ability to evolve internal logging. That seems quite limiting. But again, I emphasize with the reasons for this ask, just wanted to share that this path might be quite difficult, and investing into specialized log handlers might be easier overall.

[Jerry-Carter]

I respect that position. Not surprisingly, perhaps, but I feel that I've made much the same argument when I was the one developing the library. I will admit, there is something of a slippery slope here. Once you make a few log messages configurable, feature requests get filed to make others available. If too many messages are configurable, it can become virtually impossible to make changes without breaking someone's . The only defense is to have a firm idea of how the library will be used and what very small and generally defined set of events are significant. For instance, the AWS Swift interface, Soto, did a good job by settling on just two configuration options: requestLogLevel and errorLogLevel. But I recognized that one can't always be so precise.

At this point, I think we've beaten this issue to death. I have no objection to closing it.

P.S. Open source projects have a wonderful escape clause. You can always tell prospective users: "if it doesn't meet your needs, just fork the code".