`nats auth account ls` panics: invalid memory address or nil pointer dereference

[pobk]

Observed behavior

running nats auth account ls fails with a panic:

$ nats auth account ls
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x28 pc=0x103576688]

goroutine 1 [running]:
github.com/synadia-io/jwt-auth-builder%2ego.KeyFromNkey({0x0, 0x0}, {0x140004991ff, 0x1, 0x1})
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/key.go:44 +0x68
github.com/synadia-io/jwt-auth-builder.go/providers/nsc.(*NscProvider).loadUser(0x1400068a4e0?, {0x103e2b420, 0x1400068a4e0}, {{0x140001a6870, 0x6}}, {0x140001a6d70, 0x3}, {0x14000134318, 0x3})
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/providers/nsc/nsc.go:204 +0x120
github.com/synadia-io/jwt-auth-builder.go/providers/nsc.(*NscProvider).loadUsers(0x140002d7740, {0x103e2b420, 0x1400068a4e0}, {{0x140001a6870?, 0x1400004d308?}}, {0x140001a6d70, 0x3})
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/providers/nsc/nsc.go:178 +0x138
github.com/synadia-io/jwt-auth-builder.go/providers/nsc.(*NscProvider).loadAccount(0x140002d7740, {0x103e2b420, 0x1400068a4e0}, {{0x140001a6870, 0x6}}, {0x140001a6d70, 0x3})
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/providers/nsc/nsc.go:159 +0x394
github.com/synadia-io/jwt-auth-builder.go/providers/nsc.(*NscProvider).loadAccounts(0x140002d7740, {0x103e2b420, 0x1400068a4e0}, {{0x140001a6870?, 0x140003a6e80?}})
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/providers/nsc/nsc.go:123 +0xf4
github.com/synadia-io/jwt-auth-builder.go/providers/nsc.(*NscProvider).loadOperator(0x140002d7740, {0x103e2b420, 0x1400068a4e0})
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/providers/nsc/nsc.go:106 +0x1d0
github.com/synadia-io/jwt-auth-builder.go/providers/nsc.(*NscProvider).Load(0x140002d7740)
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/providers/nsc/nsc.go:54 +0x114
github.com/synadia-io/jwt-auth-builder%2ego.NewAuthWithOptions({0x103e183f0, 0x140002d7740}, 0x140003a6dc0?)
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/auth.go:45 +0xc8
github.com/synadia-io/jwt-auth-builder%2ego.NewAuth(...)
	/Users/[REDACTED]/go/pkg/mod/github.com/synadia-io/jwt-auth-builder.go@v0.0.9/auth.go:31
github.com/nats-io/natscli/internal/auth.GetAuthBuilder()
	/Users/[REDACTED]/go/pkg/mod/github.com/nats-io/natscli@v0.3.1/internal/auth/auth.go:53 +0xd4
github.com/nats-io/natscli/internal/auth.SelectOperator({0x0, 0x0}, 0x1, 0x1)
	/Users/[REDACTED]/go/pkg/mod/github.com/nats-io/natscli@v0.3.1/internal/auth/auth.go:111 +0x6c
github.com/nats-io/natscli/cli.(*authAccountCommand).selectOperator(0x140001cfc08, 0x0?)
	/Users/[REDACTED]/go/pkg/mod/github.com/nats-io/natscli@v0.3.1/cli/auth_account_command.go:332 +0x30
github.com/nats-io/natscli/cli.(*authAccountCommand).lsAction(0x140001cfc08, 0x103436a60?)
	/Users/[REDACTED]/go/pkg/mod/github.com/nats-io/natscli@v0.3.1/cli/auth_account_command.go:785 +0x2c
github.com/choria-io/fisk.(*actionMixin).applyActions(...)
	/Users/[REDACTED]/go/pkg/mod/github.com/choria-io/fisk@v0.7.2/actions.go:28
github.com/choria-io/fisk.(*Application).applyActions(0x14000001cc0?, 0x1400011e6c0)
	/Users/[REDACTED]/go/pkg/mod/github.com/choria-io/fisk@v0.7.2/app.go:812 +0xe4
github.com/choria-io/fisk.(*Application).execute(0x14000001cc0, 0x1400011e6c0, {0x1400015ea80, 0x3, 0x4})
	/Users/[REDACTED]/go/pkg/mod/github.com/choria-io/fisk@v0.7.2/app.go:613 +0x50
github.com/choria-io/fisk.(*Application).Parse(0x14000001cc0, {0x140001d6010?, 0x1400004db78?, 0x3a?})
	/Users/[REDACTED]/go/pkg/mod/github.com/choria-io/fisk@v0.7.2/app.go:276 +0x108
github.com/choria-io/fisk.(*Application).MustParseWithUsage(0x14000001cc0, {0x140001d6010, 0x3, 0x3})
	/Users/[REDACTED]/go/pkg/mod/github.com/choria-io/fisk@v0.7.2/app.go:878 +0x34
main.main()
	/Users/[REDACTED]/go/pkg/mod/github.com/nats-io/natscli@v0.3.1/nats/main.go:87 +0x2a94

Expected behavior

Expecting a list of accounts.

Server and client version

# nats server info
Server information for nats-1 (NDVCT3F2HY4J4UOPCTEL26X4OW3K3BVQQWCZIWVNEI4DWMRYQIM6DKXX)

Process Details:

                          Version: 2.12.4
                       Git Commit: 34894c1
                       Go Version: go1.25.6
                       Start Time: 2026-01-30 13:56:56
          Configuration Load Time: 2026-01-30 13:56:56
             Configuration Digest: sha256:6979bdd778302a9f5eff9cb2fe66eeb71a436c4db0056a0ff328218658ea4bc0
                           Uptime: 25m26s

Connection Details:

                    Auth Required: true
                     TLS Required: false
                             Host: 0.0.0.0:4222
                      Client URLs: 192.168.148.2:4222
                                   192.168.148.3:4222
                                   192.168.148.4:4222

# nats --version
v0.3.1

The nats.conf i'm using on the cluster is. Adjustments as appropriate for each node.

include ./common.conf

server_name: nats-1

cluster {
  name: nats-cluster
  port: 6222

  routes = [
    nats://nats-2:6222
    nats://nats-3:6222
  ]
}

common.conf:

# Common NATS server options shared by all nodes.

include ./resolver.conf

# Client & monitoring ports (published differently per container).
port: 4222
http: 8222

# JetStream storage. Each node has its own /data volume.
jetstream {
  store_dir: /data/jetstream
  max_mem_store: 512MB
  max_file_store: 10GB
}

# Optional: keep logs readable in docker logs
# (Disable/adjust as you like.)
debug: false
trace: false
logtime: true

resolver.conf:

# Operator named POC
operator: [snip]
# System Account named SYS
system_account: ADL3LXN{...snip...]PCZFCWMBQEB

# configuration of the nats based resolver
resolver {
    type: full
    # Directory in which the account jwt will be stored
    dir: '/jwt'
    # In order to support jwt deletion, set to true
    # If the resolver type is full delete will rename the jwt.
    # This is to allow manual restoration in case of inadvertent deletion.
    # To restore a jwt, remove the added suffix .delete and restart or send a reload signal.
    # To free up storage you must manually delete files with the suffix .delete.
    allow_delete: true
    # Interval at which a nats-server with a nats based account resolver will compare
    # it's state with one random nats based account resolver in the cluster and if needed, 
    # exchange jwt and converge on the same set of jwt.
    interval: "2m"
    # Timeout for lookup requests in case an account does not exist locally.
    timeout: "1.9s"
}


# Preload the nats based resolver with the system account jwt.
# This is not necessary but avoids a bootstrapping system account. 
# This only applies to the system account. Therefore other account jwt are not included here.
# To populate the resolver:
# 1) make sure that your operator has the account server URL pointing at your nats servers.
#    The url must start with: "nats://" 
#    nsc edit operator --account-jwt-server-url nats://localhost:4222
# 2) push your accounts using: nsc push --all
#    The argument to push -u is optional if your account server url is set as described.
# 3) to prune accounts use: nsc push --prune 
#    In order to enable prune you must set above allow_delete to true
# Later changes to the system account take precedence over the system account jwt listed here.
resolver_preload: {
	ADL3LXN{...snip...]PCZFCWMBQEB: [snip],
}

docker-compose.yml:

version: "3.9"

x-nats-common: &nats-common
  image: nats:2.12.4-alpine
  command: ["-c", "/etc/nats/nats.conf"]
  restart: unless-stopped
  networks:
    nats:
      aliases: []  # overridden per service
  volumes:
    - ./nats:/etc/nats

services:
  nats-1:
    <<: *nats-common
    container_name: nats-1
    hostname: nats-1
    networks:
      nats:
        aliases: ["nats-1"]
    volumes:
      - ./nats:/etc/nats
      - ./nats/nats-1.conf:/etc/nats/nats.conf
      - ./data/nats-1/jetstream:/data
      - ./data/nats-1/jwt:/jwt
    ports:
      - "4222:4222"
      - "8222:8222"

  nats-2:
    <<: *nats-common
    container_name: nats-2
    hostname: nats-2
    networks:
      nats:
        aliases: ["nats-2"]
    volumes:
      - ./nats:/etc/nats
      - ./nats/nats-2.conf:/etc/nats/nats.conf
      - ./data/nats-2/jetstream:/data
      - ./data/nats-2/jwt:/jwt
    ports:
      - "4223:4222"
      - "8223:8222"

  nats-3:
    <<: *nats-common
    container_name: nats-3
    hostname: nats-3
    networks:
      nats:
        aliases: ["nats-3"]
    volumes:
      - ./nats:/etc/nats
      - ./nats/nats-3.conf:/etc/nats/nats.conf
      - ./data/nats-3/jetstream:/data
      - ./data/nats-3/jwt:/jwt
    ports:
      - "4224:4222"
      - "8224:8222"

networks:
  nats:
    name: poc-nats

Host environment

macbook running Apple M4
macOS Tahoe 26.2

NATS server operating on OrbStack running docker images.

Steps to reproduce

Run nats --creds=./creds/sys.creds auth account list

[ripienaar]

@aricart any idea what this might be?

Happens from this code:

	return ab.NewAuth(nsc.NewNscProvider(filepath.Join(storeDir, "stores"), filepath.Join(storeDir, "keys")))

Guessing some missing keys in the store or something?

[pobk]

@aricart any idea what this might be?

Happens from this code:

	return ab.NewAuth(nsc.NewNscProvider(filepath.Join(storeDir, "stores"), filepath.Join(storeDir, "keys")))

Guessing some missing keys in the store or something?

I'll re-init my nsc store and re-try this. It's a dev instance, so no loss.

[aricart]

so it's panicking on getting a public key.
this is really weird, the error happens when trying to get the public key, but the trick is we read and found the file...

Can you repeat this? - and with the nsc environment can you do an nsc list keys -A - take a look to see if the users all have keys stored. I would have this expected to die on parsing the key, but it didn't.

[aricart]

The NSC store returns nil, nil (not found) - not an error. The authbuilder "manages the keys" so conflict in ideology there.

they key is missing - is this keystore actively used by the humans?

[pobk]

Can you repeat this? - and with the nsc environment can you do an nsc list keys -A - take a look to see if the users all have keys stored. I would have this expected to die on parsing the key, but it didn't.

nsc list keys -A returns stored keys for all users in my test environment... I've not re-initialised it yet, though

[pobk]

they key is missing - is this keystore actively used by the humans?

No, it's just one of my dev environments... which, given I've got about 9 of them for different projects could just be confused/screwed.

[aricart]

@pobk all good - it is ok for the keys to go missing, the panic however is not ok - so the next build will close it here.