[INF] JetStream cluster no metadata leader

[thedengerzone]

First this is my configuration, each server is placed inside a docker container and run on a VM :

`server_name: NATS-1
listen=4222

jetstream {
    store_dir: '/nats/jetstream/data'
    max_file_store: '1024M'
}

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "ADMIN"
      }
    ]
  }
}

cluster {
  name: jetstream

  # host/port for inbound route connections from other server
  listen: 0.0.0.0:4223

  # This server establishes routes with these server.
  # This server solicits new routes and Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above.
  routes = [
    nats://10.10.10.2:4223, nats://10.10.10.3:4223
  ]

}

#LOGGING

  trace: true
  debug: true
~

server_name: NATS-2
listen=4222

jetstream {
    store_dir: '/nats/jetstream/data'
    max_file_store: '1024M'
}

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "ADMIN"
      }
    ]
  }
}

cluster {
  name: jetstream

  # host/port for inbound route connections from other server
  listen: 0.0.0.0:4223

  # This server establishes routes with these server.
  # This server solicits new routes and Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above.
  routes = [
    nats://10.10.10.1:4223, nats://10.10.10.3:4223
  ]

}

#LOGGING

  trace: true
  debug: true
~


server_name: NATS-3
listen=4222

jetstream {
    store_dir: '/nats/jetstream/data'
    max_file_store: '1024M'
}

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "ADMIN"
      }
    ]
  }
}

cluster {
  name: jetstream

  # host/port for inbound route connections from other server
  listen: 0.0.0.0:4223

  # This server establishes routes with these server.
  # This server solicits new routes and Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above.
  routes = [
    nats://10.10.10.1:4223, nats://10.10.10.2:4223
  ]

}

#LOGGING

  trace: true
  debug: true
~

`

Logs from 1 server:

`2024-01-17 22:19:59.215 stderr [16] 2024/01/17 22:19:59.215728 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteRequest &{term:8637 lastTerm:15 lastIndex:86 candidate:7xP00awI reply:$NRG.R.2iWNKnz4}
2024-01-17 22:19:59.215 stderr [16] 2024/01/17 22:19:59.215767 [DBG] RAFT [u5tD5e5R - _meta_] Sending a voteResponse &{term:8636 peer:u5tD5e5R granted:true} -> "$NRG.R.2iWNKnz4"
2024-01-17 22:19:59.751 stderr [16] 2024/01/17 22:19:59.751030 [DBG] Checking JetStream cluster state
2024-01-17 22:20:00.751 stderr [16] 2024/01/17 22:20:00.751391 [DBG] Checking JetStream cluster state
2024-01-17 22:20:01.751 stderr [16] 2024/01/17 22:20:01.750939 [DBG] Checking JetStream cluster state
2024-01-17 22:20:02.750 stderr [16] 2024/01/17 22:20:02.750676 [DBG] Checking JetStream cluster state
2024-01-17 22:20:03.575 stderr [16] 2024/01/17 22:20:03.575603 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteRequest &{term:8638 lastTerm:15 lastIndex:86 candidate:7xP00awI reply:$NRG.R.2iWNKnz4}
2024-01-17 22:20:03.575 stderr [16] 2024/01/17 22:20:03.575697 [DBG] RAFT [u5tD5e5R - _meta_] Sending a voteResponse &{term:8637 peer:u5tD5e5R granted:true} -> "$NRG.R.2iWNKnz4"
2024-01-17 22:20:03.751 stderr [16] 2024/01/17 22:20:03.751032 [DBG] Checking JetStream cluster state
2024-01-17 22:20:04.751 stderr [16] 2024/01/17 22:20:04.751418 [DBG] Checking JetStream cluster state
2024-01-17 22:20:05.751 stderr [16] 2024/01/17 22:20:05.751277 [DBG] Checking JetStream cluster state
2024-01-17 22:20:06.751 stderr [16] 2024/01/17 22:20:06.751466 [DBG] Checking JetStream cluster state
2024-01-17 22:20:07.750 stderr [16] 2024/01/17 22:20:07.750664 [DBG] Checking JetStream cluster state
2024-01-17 22:20:08.750 stderr [16] 2024/01/17 22:20:08.750545 [DBG] Checking JetStream cluster state
2024-01-17 22:20:09.751 stderr [16] 2024/01/17 22:20:09.750946 [DBG] Checking JetStream cluster state
2024-01-17 22:20:09.788 stderr [16] 2024/01/17 22:20:09.788250 [DBG] RAFT [u5tD5e5R - _meta_] Switching to candidate
2024-01-17 22:20:09.788 stderr [16] 2024/01/17 22:20:09.788285 [DBG] RAFT [u5tD5e5R - _meta_] Sending out voteRequest {term:8639 lastTerm:15 lastIndex:86 candidate:u5tD5e5R reply:}
2024-01-17 22:20:09.789 stderr [16] 2024/01/17 22:20:09.789535 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteResponse &{term:8638 peer:7xP00awI granted:true}
2024-01-17 22:20:09.789 stderr [16] 2024/01/17 22:20:09.789884 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteResponse &{term:8638 peer:bmL0UAXo granted:true}
2024-01-17 22:20:10.751 stderr [16] 2024/01/17 22:20:10.751222 [DBG] Checking JetStream cluster state
2024-01-17 22:20:11.751 stderr [16] 2024/01/17 22:20:11.751137 [DBG] Checking JetStream cluster state
2024-01-17 22:20:12.751 stderr [16] 2024/01/17 22:20:12.750943 [DBG] Checking JetStream cluster state
2024-01-17 22:20:13.192 stderr [16] 2024/01/17 22:20:13.192683 [DBG] IP ADRESS - rid:36 - Router Ping Timer
2024-01-17 22:20:13.192 stderr [16] 2024/01/17 22:20:13.192690 [DBG] IP ADRESS - rid:38 - Router Ping Timer
2024-01-17 22:20:13.359 stderr [16] 2024/01/17 22:20:13.359658 [DBG] IP ADRESS - rid:35 - Router Ping Timer
2024-01-17 22:20:13.359 stderr [16] 2024/01/17 22:20:13.359673 [DBG] IP ADRESS- rid:34 - Router Ping Timer
2024-01-17 22:20:13.751 stderr [16] 2024/01/17 22:20:13.750985 [DBG] Checking JetStream cluster state
2024-01-17 22:20:14.357 stderr [16] 2024/01/17 22:20:14.357036 [DBG] RAFT [u5tD5e5R - _meta_] Sending out voteRequest {term:8640 lastTerm:15 lastIndex:86 candidate:u5tD5e5R reply:}
2024-01-17 22:20:14.357 stderr [16] 2024/01/17 22:20:14.357088 [INF] JetStream cluster no metadata leader
2024-01-17 22:20:14.358 stderr [16] 2024/01/17 22:20:14.358173 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteResponse &{term:8639 peer:7xP00awI granted:true}
2024-01-17 22:20:14.358 stderr [16] 2024/01/17 22:20:14.358822 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteResponse &{term:8639 peer:bmL0UAXo granted:true}
2024-01-17 22:20:14.751 stderr [16] 2024/01/17 22:20:14.751092 [DBG] Checking JetStream cluster state
2024-01-17 22:20:15.181 stderr [16] 2024/01/17 22:20:15.181168 [DBG]IP ADRESS- rid:26 - Router Ping Timer
2024-01-17 22:20:15.221 stderr [16] 2024/01/17 22:20:15.221217 [DBG] IP ADRESS - rid:27 - Router Ping Timer
2024-01-17 22:20:15.271 stderr [16] 2024/01/17 22:20:15.271053 [DBG] IP ADRESS- rid:28 - Router Ping Timer
2024-01-17 22:20:15.282 stderr [16] 2024/01/17 22:20:15.281983 [DBG] IP ADRESS - rid:29 - Router Ping Timer
2024-01-17 22:20:15.750 stderr [16] 2024/01/17 22:20:15.750854 [DBG] Checking JetStream cluster state
2024-01-17 22:20:16.750 stderr [16] 2024/01/17 22:20:16.750658 [DBG] Checking JetStream cluster state
2024-01-17 22:20:17.751 stderr [16] 2024/01/17 22:20:17.751366 [DBG] Checking JetStream cluster state
2024-01-17 22:20:18.751 stderr [16] 2024/01/17 22:20:18.750890 [DBG] Checking JetStream cluster state
2024-01-17 22:20:19.751 stderr [16] 2024/01/17 22:20:19.751094 [DBG] Checking JetStream cluster state
2024-01-17 22:20:19.886 stderr [16] 2024/01/17 22:20:19.886827 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteRequest &{term:8641 lastTerm:15 lastIndex:86 candidate:bmL0UAXo reply:$NRG.R.sqSZh2ZJ}
2024-01-17 22:20:19.887 stderr [16] 2024/01/17 22:20:19.886867 [DBG] RAFT [u5tD5e5R - _meta_] Stepping down from candidate, detected higher term: 8641 vs 8640
2024-01-17 22:20:19.887 stderr [16] 2024/01/17 22:20:19.886878 [DBG] RAFT [u5tD5e5R - _meta_] Sending a voteResponse &{term:8640 peer:u5tD5e5R granted:true} -> "$NRG.R.sqSZh2ZJ"
2024-01-17 22:20:19.887 stderr [16] 2024/01/17 22:20:19.886886 [DBG] RAFT [u5tD5e5R - _meta_] Switching to follower
2024-01-17 22:20:20.750 stderr [16] 2024/01/17 22:20:20.750572 [DBG] Checking JetStream cluster state
2024-01-17 22:20:21.751 stderr [16] 2024/01/17 22:20:21.750934 [DBG] Checking JetStream cluster state
2024-01-17 22:20:22.751 stderr [16] 2024/01/17 22:20:22.751529 [DBG] Checking JetStream cluster state
2024-01-17 22:20:23.751 stderr [16] 2024/01/17 22:20:23.751380 [DBG] Checking JetStream cluster state
2024-01-17 22:20:24.750 stderr [16] 2024/01/17 22:20:24.750483 [DBG] Checking JetStream cluster state
2024-01-17 22:20:25.751 stderr [16] 2024/01/17 22:20:25.750931 [DBG] Checking JetStream cluster state
2024-01-17 22:20:25.961 stderr [16] 2024/01/17 22:20:25.961286 [DBG] RAFT [u5tD5e5R - _meta_] Received a voteRequest &{term:8642 lastTerm:15 lastIndex:86 candidate:7xP00awI reply:$NRG.R.2iWNKnz4}
2024-01-17 22:20:25.961 stderr [16] 2024/01/17 22:20:25.961494 [DBG] RAFT [u5tD5e5R - _meta_] Sending a voteResponse &{term:8641 peer:u5tD5e5R granted:true} -> "$NRG.R.2iWNKnz4"
2024-01-17 22:20:26.750 stderr [16] 2024/01/17 22:20:26.750805 [DBG] Checking JetStream cluster state
2024-01-17 22:20:27.751 stderr [16] 2024/01/17 22:20:27.751231 [DBG] Checking JetStream cluster state
2024-01-17 22:20:28.751 stderr [16] 2024/01/17 22:20:28.751298 [DBG] Checking JetStream cluster state
2024-01-17 22:20:29.750 stderr [16] 2024/01/17 22:20:29.750832 [DBG] Checking JetStream cluster state`

It seems servers are communicating and they are sending voteRequests and voteResponses but i leader is never assigned.

[thedengerzone]

Reason why things were failing was due to some misconfiguration that was stored into docker volume. After deploying again data saved in docker volume was taken and it seems problem was somewhere there(even if we had correct configuration). After removing the volume and assigning it again everything worked as expected