Replies: 8 comments 6 replies
-
|
Some compelling use cases here. |
Beta Was this translation helpful? Give feedback.
-
|
I'm creating a prototype for my homelab, using an embedded nats server with domain socket support patched in (via in-process connection, needs the private For the client side, I need a cli with domain socket support, that probably needs to be forked (mainly to support shell interfaces to system-nats). That's little more then checking a URL for unix: scheme. The system-nats would then use an account (either standalone or from network/org/site) and authenticate processes based on the UID or things like their systemd user sessions or units (PID inspection). NATS' permission model is a nice fit for the system security boundary mappable into a site. I think integrating this should work without additional patching. The ability to exchange FDs with domain sockets is quite interesting, to provide some streaming options or shared memory. But that's probably too much of a paradigm shift to consider in native NATS. Would be great not to have to patch in domain socket support. |
Beta Was this translation helpful? Give feedback.
-
|
A quick hack (domain socks + peer-creds with unpriv user running embedded nats): socket=/tmp/sysnatsd.sock
# Start Server
sysnatsd/sysnatsd &
sysnatsd_pid=$!
sleep 1
chmod 0666 $socket
# Show socket
echo; echo -n "Socket: "
ls -lh $socket
echo
# client connection
client_out="$(printf 'CONNECT {}\r\nPING\r\n' | sudo -u nobody nc -w1 -U $socket)"
kill $sysnatsd_pid
sleep 1
echo; echo "Client Output:"
echo "$client_out" |
Beta Was this translation helpful? Give feedback.
-
|
@mutech you might want to re-open the discussion, if youre able |
Beta Was this translation helpful? Give feedback.
-
|
For my use case, the NATS security model (as I understand it - not very well) is problematic, specifically the authorization part of it. The identity part is fine. A system-nats provides identities based on local UID. These can be signed to use in other NATs contexts (where the system-nats account is trusted). On a local system, services come and go. New services can be deployed. NATS authorization happens when a client connection is established. So every time permissions change, all connections need to be restarted or (if that's supported) the server has to update the permissions. At the very least, this creates a window where existing clients operate with outdated permissions. System services are not NATS application clients, they are not designed to work with nats. Instead, they will use adapters that would then need to implement a retry logic. They can't easily determine if a disconnect represents a network outage or is just a permission update. Periods where a permission revocation is pending can be critical for some services. The from my perspective ideal solution would be if messages would contain a trusted header containing the authenticated identity of the sender. This would allow recipients to perform their own authorization, independent of NATS authorization as an opt in feature (only available on system-nats). This would be quite easy to implement, if there was a hook in the embedded NATS API, that would allow a server instance to rewrite headers (e.g. remove existing ID headers, because they would be untrusted/unauthenticated and then add a local header). System-nats would act as authority for clients connected to a local UDS connection. For remote servers, it would distinguish if it trusts ID headers (e.g a peer system-nats) or just uses an ID representing the remote connection. This would create a nice security boundary, where a system-nats instance would use standard NATS permissions, provide trustworthy local identities, configure system local NATS permissions and additionally leave it up to services to perform their own authorization (using the local system identities and optionally verified remote identities. With the current API, this is hard, because there is no entry point to hook into to rewrite or add headers. Relying on native NATS permissions would require all kinds of hacks and create a lot of friction. All of that is based on my shallow understanding of NATS security and the API, so please correct me if I'm wrong. I'm thinking of an API extension (embedded) looking like this: type MessageTransformer interface {
// Iterator - interface for inspections/mutations for whatever use case
Headers() iter.Seq[HeaderEntry]
// Add/modify header (identity for dynamic authorization on service-client side)
SetHeader(key, value string)
GetHeader(key) string
// Reject message entirely (dynamic permissions on server side, payload validation, etc.)
Reject(err error)
// Read-only
Payload() []byte
Client() Client // i injected uid/pid/gid in conn/remoteaddess in my PoC
}
type HeaderEntry interface {
Key() string
Value() string
Delete()
ReplaceValue(newValue string)
}
// to be used like this:
func identityInterceptor(msg MessageTransformer) {
// Strip untrusted headers
for h := range msg.Headers() {
if h.Key() == "X-Identity" {
h.Delete() // safe - scoped to this entry
}
}
// Inject trusted identity
msg.SetHeader("X-Identity", fmt.Sprintf("uid=%d", msg.Client().UID()))
// Or reject entirely
if msg.Client().UID() == 1234 {
msg.Reject(errors.New("not allowed"))
}
}This would need to be registered to be called when a message enters the node/system-nats. This might result in a realloc, which might generally be a high price, but for my use case it would be negligable. Using this interface, the actually rewriting of the message (-headers) could be deferred, assuming that the actual message content is opaque to NATS itself. I assume that a message will anyways evantually be copied somewhere, so some scatter/gather mechanism could make the edit almost zero cost. Using the transformer interface would avoid having to parse byte[] (internals) in API client code and defer parsing to when/if it's actually needed for the transform. This should be efficient. This could also be used to fix the INBOX snooping/injection issue, that is actually a real problem if clients are inherently suspicious to be malicious actors (random processes connecting to socket). Here we would need to look for reply headers and add temporary perms for subscribers of the topic to publish to the reply-to-target. That would not require any client participation and could be used to restrict inboxes server-side only. This could be done without having to edit static permissions (which is hard), using the same interceptor to allow only responses matching a previous request. (Default perms for INBOX.> would be allow, interceptor would discard all messages not matching active requests, so snooping/injecting INBOX.> should reliably fail for all messages an observer has no legitimate interest.). This should also work for remote messaging, not sure though. If it doesn't, remote traffic can be handled separately, but this is entirely up to server API client. |
Beta Was this translation helpful? Give feedback.
-
|
Another interesting use case of server side header transformation is that this can enable save exchange of FDs between clients connected to domain sockets. A client can request exchange of FDs via headers. The server can obtain an FD from one client's domain socket and pass it to another client. If the server can inspect headers, clients and the server can establish communication protocols and side channels easily and implement security boundaries (as described above) to protect the channels. The benefit of looping such negotiations through a nats instance would be observability and auditability and of course convenience. Since this is too use case specific to add it to core NATS, this API extension would make it relatively easy to add to specialized embedded servers implementing such features. |
Beta Was this translation helpful? Give feedback.
-
|
For the side-project driving my interest in these features, I could just create a couple of Nix-Package overlays (I'm using nixos in my homelab), patching NATS. If that gets to messy I could fork nats-server (UDC+Header rewriting) and the nats-cli (UDC). Some client APIs might already support UDC via URLs (if they can unix:///...). My problem is that I probably will not have the capacity to maintain it for upstream changes, at least not reliably. That's bad for a security sensitive application. I believe there is a lot of value in these propositions:
I can see no downside (other than the implementation cost, which I believe is limited). |
Beta Was this translation helpful? Give feedback.
-
|
Here a proposal for how to implement the UNIX Domain Socket part of my idea properly (@derekcollison feeback on this would be welcome. I'm going to implement that for my project, but it would be great if this would eventually end up upstream, would like to do it in a way that's acceptable for you guys). UNIX Domain SocketsImplementation Changes
Identity and UsernameIdentity (canonical, for audit/logging):
Username (NATS
Username derivation:
Peercred MatchingSyntax:
Extension point: Permission Accumulation
ConfigurationWhen user contains a uid-query and RationaleThis would allow clients connecting via UDS to act similar to clients using user/password or TLS credentials. Clients could still use JWT/NKEY authentication overwriting these local configs (if I understand the auth procedure correctly). As a result, the UNIX permission model could be fully integrated in a system local scope. If the local nats instance has an account, it can act as signatory for local processes in connected NATS meshes (would need additional config). The query syntax and extension point would make it easy to integrate things like kerberos, pam, etc. Nats running as unprivileged user can delegate verifications requiring caps to external processes (that could integrate as NATS services), example: The syntax is analog to TLS subject names. Multiple rule matches and negatives make it possible to easily add deny rules (e.g. "uid!=0,gid:name!=wheel" -> deny privileged stuff). Rule matching is efficient (if c.isUds && rule ~ '[ugp]id[:=]') is no overhead, so there should be no noticable penalty for settings not using this. |
Beta Was this translation helpful? Give feedback.
-
|
Here a proposal regarding identity headers. Again, I would appreciate feedback a lot. I'm a bit uncertain where and how to implement that (if this is interesting for you @derekcollison). I changed my mind about general header rewriting. This is actually the only application for touching headers on the server-side I could think of: Identity Headers for Incoming MessagesAll incoming messages will (see Implementation/Configuration) receive a server generated message header PurposeNATS permissions are have two properties that make them difficult to use in certain situations:
The proposed solution is for a NATS server to prepend an identity header to each incoming message, adding In the absense of other NATS server side mechanisms prepending message headers, this is semantically like If that degree of transparency is not generally desired, a server can strip or filter this chain or replace Servers can prevent untrusted (in regards to ID chain) clients from forging identities by stripping Header Name and FormatPreliminary header name: Server AccountThis identifies a server by a NATS account or a public key, that uniquely identifies the NATS instance and TypeThis specifies the authentication type of the identity. This determines the format and semantics of Client Identity
Optional SignatureIt might be desirable to let a server generating identity headers sign them with their account key (if Reasons not to sign:
ImplementationInjection Point Candidates
Caching of Identities
Configuration
|
Beta Was this translation helpful? Give feedback.
-
|
TCP/UDS "bench" (16 cores, i7):
Another run with in-proc (the results are very consistent for TCP/UDS): Total ops/s:
Per client ops/s:
Using this: func pingPong(b *testing.B, c net.Conn, n int) time.Duration {
buf := make([]byte, 4096)
ping := []byte("PING\r\n")
start := time.Now()
for i := 0; i < n; i++ {
c.Write(ping)
c.SetReadDeadline(time.Now())
_, err := c.Read(buf)
if err != nil {
continue
}
for {
c.SetReadDeadline(time.Now())
_, err = c.Read(buf)
if err != nil {
break
}
}
}
elapsed := time.Since(start)
c.SetReadDeadline(time.Time{})
c.Read(buf)
return elapsed
}
...
func Benchmark_Ping_10(b *testing.B) {
b.Run("TCP", func(b *testing.B) {
s := runServerTCP(8423)
defer s.Shutdown()
benchPing(b, dialTCP(8423), 10)
})
b.Run("UDS", func(b *testing.B) {
s := runServerUDS("/tmp/nats-bench.sock")
defer s.Shutdown()
benchPing(b, dialUDS("/tmp/nats-bench.sock"), 10)
})
}I was surprized, that UDS is faster than inproc for one client. I thought that uses pipes (should be same as UDS), but learned it doesn't. My interpretation:
|
Beta Was this translation helpful? Give feedback.
-
|
Here is a first implementation (domain sockets + auth): https://github.com/mutech/nats-server |
Beta Was this translation helpful? Give feedback.
-
|
My concern here is not necessarily with the idea of supporting UNIX domain sockets as they are relatively easy to support, but rather how quickly this grows arms and legs in order to support additional auth modes, header rewriting etc. This seems like a significant set of changes to the security model that have an ongoing maintenance cost for us. |
Beta Was this translation helpful? Give feedback.
-
|
I can see that. While working on the patches I was surprised how fast that went all over the place. The UDS support itself (as an additional transport) seems to be interesting in regards to performance, but I don't believe that's a big advantage for core nats. The win comes from the peer credentials and secret-free authentication. I think the better way to implement something like this would be as separate app that embeds NATS, but that can't be done with the current code, since there are no extension points for that. The advantage of extending by embedding would be that NATS would not suffer the maintenance burden for a feature that's not in your customers mainstream. The header rewriting is a separate topic/project that I believe would solve a couple of problems outside my domain (enable RPC-style services to do additional authz, solve the inbox stuff for replies without client prefix stunts). I mixed that into this discussion, because it's a use case of mine, but it's really a separate issue. Would you be interested in brainstorming about the feasibility of creating such extension points? I think I can maintain the current level of additions for my use case as a separate fork, but keeping this maintained in the long term is a bit heavy (considering this is just a side project for me). PS: Was an interesting journey to read your code. At first I was a bit shocked, then I understood more of what you're doing. I envy you a bit for working on this project full time, must be fun! Thanks for this great product! |
Beta Was this translation helpful? Give feedback.
-
|
I understand that my approach is not attractive for you guys, and i actually agree, considering that my use case is too far away from the main stream. The main reasons why I cannot implement my features using an embedded approach are:
Currently, NATS has a monolithic start/shutdown and config-reload procedure. This makes it impossible to have a staged startup. One of the things I need for my use case is to start NATS during early boot, before the network is up, and serve clients with reduced capabilities. Once the network is up, I need to start networked modules and transition into full featured mode. I thought about replicating the start/shutdown/reload functionality in a separate module in a fork, using a dependency-aware service manager. This would not add any features, it would only modularize the life cycle management and transform dependencies implied by the code into explicit ones. Given such an architecture, the UDS and permission model extensions I wrote here could be easy to implement using an embedded server, simply by adding a new service (UDS client listener). I could also use a staggered start up procedure for my use case. This could be an experimental API, not affecting the rest of NATS, that could be tested for equivalence with the current architecture. If it turns out to be robust and not having significant downsides, NATS can eventually migrate to that architecture. I believe it would have major benefits, one of which would be that it's much easier to understand explicit dependencies than implicit assumptions in code. It would make it much easier to add integrations for other messaging protocols (like mqtt). And of course, it would make it much easier for people like me to create custom solutions that require a deeper integration than what embedding NATS can offer today. I'm closing this, seems to be the wrong approach. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
(see https://github.com/mutech/nats-server for a preliminary implementation)
NATS seems to be a revolutionary technology that - if my understanding is correct - should be THE hype. I'm surprised it is not, though it gets quite a lot of attention.
I have two use cases in mind for which NATS would be a complete game changer for my project (-ideas):
1. Linux "system-nats"
This would start early in the boot process providing a UNIX domain socket to which clients can connect. The nats would serve various components to do IPC, communicate state, etc.
The reason why this should be or even has to use a UNIX domain socket, is not so much the possible performance characteristics, but that the connection provides the UID and PID of the client. The UID allows the system-nats to authenticate the client reliably. The PID allows nats to do further inspections of the clients credibility (has it been started as a systemd unit via PPID chain, is it a setuid process, etc.).
Security
This is huge, because the system-nats could act as identity broker, provisioning credentials to local processes. It could implement things like:
This would allow the use of arbitrary system-local mechanisms to ensure identity without requiring each client to manage their secrets. This client side secret management is mostly useless, since the protection of secrets is almost exclusively based on file system permissions and user identities. This mechanism would eliminate all attack vectors based on proper FS setup, it would remove the need for complex secret deployment for apps using this mechanism and centralize secret deployment for apps relying on traditional mechanisms.
Bridging authentication to the world
Once a local client can seamlessly connect to a local nats server, the local identity can be translated to externals, either via connectivity of system-nats to other servers or system-nats could even pass authenticated connections to clients (the server could open a connection to a service, authenticate (obtain vault tokens, JWT, certs, etc.) and pass the connection to the client through the unix socket.
The system-nats would become the ultimate authority for establishing system verified identity to the outer world. This is consistent with the fact that UID + system meta data is all the security a non-interactive process on a Linux process can have. This is better than just file system access, because it adds all the additional checks available.
The downside is the complexity added to the verification process and the privileges required to perform the checks. This is not trivial, but so is local secret provisioning.
Observability
Once any process on a local system can interact with system-nats without complex authentication procedures, nats becomes very accessible. Any service, process, script can log typed events and update state. Logs can be fed to nats (for RT alerting, workflows, forwarding to central logging, etc.).
Instead of logging to prometheus or a log stash, define alert queries and doing all that dance, you just observe events and can act on them anywhere, on the local system or in the network. You don't need to setup a specific log/metrics collector, you just subscribe to system logs and feed them into whatever system without having to install software on each system (using nats transport).
All the NATS advantages
I have another 20 uses for this, that all rely on the frictionless authentication provided by unix sockets. I leave that part out because it's not relevant. But the relevant effect is that this approach would bridge all the wonderful potential of a distributed communication platform to the local system and integrating that system into the mesh.
This low friction approach is important, because if you need to authenticate each service manually, the mechanism just isn't attractive.
2. User-nats
Users are not local, they live outside of computers, smart phones and web browser sessions, but they have representations in login sessions connected to applications. NATS could be a wonderful bridge connecting these sessions.
It does not take much imagination to come up with use cases, if my smart watch and phone, my browser session, my login session could communicate with each other.
I am using Linux workstations most of the time. I find it incredible annoying when I have to tap my yubikey ten times when running an ansible-playbook. It's idiotic when I'm logged into a remote system via ssh where a stale desktop session and a confirmation dialog is opening on the GUI that I don't see and my shell hangs.
For me, the UNIX domain socket is important here too, because I don't want to have to type a password or deploy a secret on every host. If I can connect to a local (user-session-) nats, I get reliable system-level authentication (that can determine if my process is running in a remote ssh-session) and for critical operations, this can query my yubikey, request confirmation on my watch or phone.
This is not only convenient, but it adds security.
UNIX domain sockets
I believe this is a major game changer. Can we please have them supported?
Beta Was this translation helpful? Give feedback.
All reactions