Proposal: EmbeddedServer Extension API (here for custom client connection services)

[mutech]

Background

I need to extend the NATS server functionality for a project of mine. I require support for UNIX domain sockets and a relatively complex permission model allowing me to map UNIX permissions to NATS world.

A first prototype used in-process connections to implement UDS support, which was easy enough. But I could not implement my permission model that way, because public APIs don't support that (at least I didn't find a way to do it).

My next attempt was to create a fork of nats-server and implement the functionality I needed there. This turned out to be a rather invasive change set, that has little to no chances to be merged into upstream. The burden of maintaining such a change that only benefits a narrow user base (me) is not appealing.

The Problem

The problem is that NATS is not a library, it's a product. It supports embedding a NATS server into a Go application, but only in the form of embedding a complete und mostly unmodified NATS server. The API has little or no support for extending or modifying the server capabilities.

This is not a problem for most NATS users and no immediate problem for the NATS project, because it primarily serves its users requirements. But it is a problem in that it makes expanding the scope of NATS very difficult. Being an established product with a large user base increases the risk of every added feature and every public API surface. Each change needs to be tested and maintained, an existing API can not easily be changed or removed.

Extending or changing NATS functionality becomes incredibly hard, risky and costly and that in turn stifles innovation and makes community contributions to NATS unattractive, because it's difficult to get changes merged, the more difficult the more ambitious the change is.

The Proposed Solution

I propose to add a new embedding API to NATS, that is isolated from the current API and is entirely an experimental opt-in feature. It follows these principles:

• Restrict changes to existing NATS functionality to the absolute minimum.
• Additional functionality is provided as optional extension API in a separate directory (probably in server package to avoid being blocked by private APIs)
• Where necessary, existing functionality is replicated, this too is only done when there is no better alternative
• Private functionality is only exposed in dedicated wrapper APIs to avoid bleeding implementation details into the new API or making them part of the general public API.

EmbeddedServer API

The main access point of the new API is an EmbeddedServer object, that is based on the current Server object that can already be used to embed NATS into a Go application.

The Start() methods of both are semantically equivalent, but the embedded server supports a service manager structure, that allows a more fine grained controll over startup, shutdown and reload operations. NATS services are categorized as follows:

Category	Service
Observability Services	ProfilingService
	MonitoringService
Account Services	ResolverService
	SystemAccountService
Feature Services	JetStreamService
	OCSPService
Server Connection Services	GatewayService
	LeafNodeAcceptorService
	LeafNodeSolicitorService
	RouteService
Client Connection Services	TCPAcceptorService
	WebsocketService
	MQTTService
	e.g. CustomUDSService

The goal of restructuring the life cycle management of these services is to allow applications to add new services of various types.

The scope of my project only requires adding client connection services to support UNIX domain sockets, so I will focus on these. The startup sequence also does not fully correspond to the categories and their order here, things are complicated. But the basic idea is to provide extension points for various service types and provide them with the API functionality they need.

The restricted focus on client connections shows however, that this solution can be progressively extended to other service types with a similar approach. Once this proves not to have major downsides (e.g. in terms of performance or stability), the main line code can be refactored to use this pattern, making core feature extensions easier.

Client Connection Services

Using the API

Creating a new client connection service would look like this:

// main.go of the embedding application
func main() {
	// Assume 'opts' is a standard NATS server Options struct,
	// loaded from a config file or set programmatically.
	opts := &server.Options{ /* ... */ }

	// 1. Create a pre-configured, embeddable server instance.
	//    NewEmbeddedServer populates it with all the standard services
	//    (TCP, JetStream, etc.) based on the options.
	s, err := server.NewEmbeddedServer(opts)
	if err != nil {
		log.Fatalf("Error creating embedded server: %v", err)
	}

	// 2. Get the specific API needed for the custom service.
	clientAPI := s.ClientConnectionServiceAPI()

	// 3. Create the custom service, injecting its API dependency.
	mySvc := NewCustomClientConnectionService(clientAPI)

	// 4. Add the custom service to the appropriate group.
	s.AddClientConnectionService(mySvc)

	// 5. Start the entire composed server. This will start all default
	//    services and our custom UDS service in the correct order.
	if err := s.Start(); err != nil {
		log.Fatalf("Error starting server: %v", err)
	}

	s.WaitForShutdown()
}

The client connection service would need to implement the interface of that service type and use the provided client-API to integrate with NATS server.

The client API would need to provide functionality to allow a new connection service to:

• Add accounts, users and permissions (typically at service start). Remove them again (if service stop/restart would be supported)
• Create a new client structure and initialize it appropriately
• Perform proper authentication and authorization for new client connections, using auth primitives currently not exposed in NATS API.
• Update server statistics/information (to be analyzed what's needed)
• Update info block information (per server instance, per service, maybe per connection)
• ...

To determine what functionality is required by the API, it should in theory be possible to reimplement all existing client types using this interface and as POC to also implement the current functionality of UDS support in my fork.

Implementing the API

The implementation would need to replicate the code in server.Start() and decompose it to implement managed services. This can be done progressively, for example to support custom client services, the start method would add an extension point where new services would be started. Analog implementations would be provided for server.Shutdown(), server.Reload() and possibly other methods that would have to handle additional client connection services (and later maybe other service types).

The service type specific client API that is provided custom services would need to be integrated into core server functionality. In a first implementation, this would mostly copy existing server code and expose it in different granularities as needed to support custom clients. If this proves to be equivalent or better, it can then be integrated back into mainline code.

Benefits and Risks

This approach would make it possible to develop and test third party modules. If they prove to be valuable (adoption/robustness/performance), they can be easily incorporated and otherwise be used by interested parties out of band. This would encourage more community participation that would not burden the core team.

It would also be easier for the core team to develop new extensions if there is a defined and eventually supported interface.

Since the EmbeddedServer uses mostly the same API interface as the Server interface, all tests using that interface can be applied to an extended server. This makes it relatively easy to validate the correctness of an extended server.

This leaves only the risk, that the API integration itself damages the mainline functionality. This cannot be helped, but it should be manageable. Here it would be important to get some support from the core team.

All other risks would be related to people using the extension API. It being experimental and opt-in, should mitigate these risks.

In a way that's similar to the approach taken on the client side with Orbit.

Where to go from here

I believe this is "the right thing to do" for the NATS server, because it provides a lot of flexibility for a small risk. However, I can see that there is no immediate return of investment. At best you might get a more active "plugin ecosystem" and maybe fewer barriers for developing new experimental features. Can this be sold to product management? I don't know.

For me, the problem is that my use case is not aligned with that of most NATS users. At the same time, NATS is the product that best matches my use case. I don't see any viable alternative to NATS, other than to roll my own solution. However, maintaining a NATS fork in the long term or worse implementing a custom solution is not sustainable for me. This would doom my project.

What I can offer is this:

• I can implement a solution matching this proposal. I would design the API for client connection services such that - in principle - the various existing client connection services could be straightforwardly migrated to use the API. I won't do that migration, but I would ensure that the API design would not make it especially hard.
• I would try to get the API integration right, following the principles I outlined
• I would use this to implement my UNIX domain socket and permission model using the API.

However, this is a major undertaking for me. I would like to know if this is interesting enough to at least have a realistic chance to eventually be merged, if implemented properly.