[Q&A] AWS networking of 2.7.1 FLARE project - client can't connect to server

[atriaybagur]

Python version (python3 -V)

3.12

NVFlare version (python3 -m pip list | grep "nvflare")

2.7.1

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 22.04

Have you successfully run any of the following examples?

• hello-numpy with simulator
• hello-pt with simulator
• hello-numpy with POC
• hello-pt with POC

Please describe your question

Hello 👋 I am trying to set up a FLARE network in AWS and I am facing issues with clients (on one EC2 instance) not being able to connect to server (on another EC2 instance).

This is my nvflare 2.7.1 provisioning script https://github.com/londonaicentre/flip-fl-base/blob/66-feature-fl-port-consolidation/net-1_project_stag.yml#L21

I have an Amazon Web Services (AWS) network load balancer (NLB) with domain address fl.stag.flip.aicentre.co.uk https://github.com/londonaicentre/FLIP/blob/4-feature-fl-port-consolidation/deploy/providers/AWS/main.tf#L352

I have a 'central' EC2 instance which runs the FL API container and the FL server container in 2 separate Docker containers within the same network, flip-fl-api-net-1 and fl-server-net-1 are their internal Docker addresses respectively: https://github.com/londonaicentre/FLIP/blob/4-feature-fl-port-consolidation/deploy/compose.production.nvflare.yml

I also have a 'site' EC2 instance which runs an FL client container https://github.com/londonaicentre/FLIP/blob/4-feature-fl-port-consolidation/trust/compose_trust.production.nvflare.yml#L18

The errors I am getting are the following:

FL API container logs

  File "/app/fl_api/startup/session_manager.py", line 47, in create_fl_session
    session.try_connect(get_settings().TIMEOUT_SESSION_CONNECT)
  File "/app/.venv/lib/python3.12/site-packages/nvflare/fuel/flare_api/flare_api.py", line 117, in try_connect
    self.api.connect(timeout)
  File "/app/.venv/lib/python3.12/site-packages/nvflare/fuel/hci/client/api.py", line 431, in connect
    token, token_signature, ssid, token_verifier = authenticator.authenticate(
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/nvflare/private/fed/authenticator.py", line 210, in authenticate
    raise FLCommunicationError(f"cannot connect to server for {self.timeout} seconds")
nvflare.apis.fl_exception.FLCommunicationError: cannot connect to server for 20.0 seconds

FL server container logs (no errors)

❯ docker logs fl-server-net-1
🔧 Generating log_config.json from template...
✅ log_config.json written to /app/local/log_config.json
[entrypoint] Starting FL server with:
  LOG_LEVEL=DEBUG
WORKSPACE set to /app/startup/..
PYTHONPATH is /local/custom:
start fl because of no pid.fl
new pid 22
2026-03-12 18:16:44,829 - ServerDeployer - INFO - server heartbeat timeout set to 600
2026-03-12 18:16:44,833 - ConfigFactory - DEBUG - search file basename:'comm_config', search dir = /app/startup/..
2026-03-12 18:16:46,759 - ConfigService - DEBUG - got var max_message_size from env
2026-03-12 18:16:46,760 - CoreCell - DEBUG - connection secure: True
2026-03-12 18:16:46,761 - CoreCell - DEBUG - server: max_msg_size=2144337904
2026-03-12 18:16:46,761 - CoreCell - DEBUG - Creating Cell: server
2026-03-12 18:16:46,762 - ConfigService - DEBUG - got var heartbeat_interval from env
2026-03-12 18:16:46,763 - ConfigService - DEBUG - got var backbone_conn_gen from env
2026-03-12 18:16:46,763 - ConfigService - DEBUG - got var internal_conn_scheme from env
2026-03-12 18:16:46,764 - ConfigService - DEBUG - got var allow_adhoc_conns from env
2026-03-12 18:16:46,764 - ConfigService - DEBUG - got var adhoc_conn_scheme from env
2026-03-12 18:16:46,765 - ConnectorManager - DEBUG - internal scheme=tcp, resources={'host': 'localhost', <DriverParams.CONNECTION_SECURITY: 'connection_security'>: 'clear'}
2026-03-12 18:16:46,766 - ConnectorManager - DEBUG - adhoc scheme=tcp, resources={}
2026-03-12 18:16:47,042 - ConfigService - DEBUG - got var streaming_chunk_size from env
2026-03-12 18:16:47,043 - Cell - DEBUG - __getattr__: args=(), kwargs={}
2026-03-12 18:16:47,043 - Cell - DEBUG - calling core_cell start
2026-03-12 18:16:47,043 - CoreCell - INFO - server: creating listener on grpc://0:8002
2026-03-12 18:16:47,044 - CoreCell - DEBUG - server: trying create ext listener: url=grpc://0:8002
2026-03-12 18:16:48,783 - driver_manager - DEBUG - Driver AioHttpDriver is registered for http
2026-03-12 18:16:48,784 - driver_manager - DEBUG - Driver AioHttpDriver is registered for https
2026-03-12 18:16:48,785 - driver_manager - DEBUG - Driver AioHttpDriver is registered for ws
2026-03-12 18:16:48,785 - driver_manager - DEBUG - Driver AioHttpDriver is registered for wss
2026-03-12 18:16:49,108 - ConfigService - DEBUG - got var use_aio_grpc from env
2026-03-12 18:16:49,110 - driver_manager - DEBUG - Driver AioGrpcDriver is registered for agrpc
2026-03-12 18:16:49,111 - driver_manager - DEBUG - Driver AioGrpcDriver is registered for agrpcs
2026-03-12 18:16:49,113 - ConfigService - DEBUG - got var use_aio_grpc from env
2026-03-12 18:16:49,113 - driver_manager - DEBUG - Driver GrpcDriver is registered for grpc
2026-03-12 18:16:49,114 - driver_manager - DEBUG - Driver GrpcDriver is registered for grpcs
2026-03-12 18:16:49,124 - driver_manager - DEBUG - Driver TcpDriver is registered for tcp
2026-03-12 18:16:49,124 - driver_manager - DEBUG - Driver TcpDriver is registered for stcp
2026-03-12 18:16:49,136 - driver_manager - DEBUG - Driver AioTcpDriver is registered for atcp
2026-03-12 18:16:49,137 - driver_manager - DEBUG - Driver AioTcpDriver is registered for satcp
2026-03-12 18:16:49,140 - ConfigService - DEBUG - got var comm_driver_path from env
2026-03-12 18:16:49,141 - GrpcDriver - DEBUG - GRPC Config: max_workers=100, options=[('grpc.max_send_message_length', 2145386496), ('grpc.max_receive_message_length', 2145386496)]
2026-03-12 18:16:49,143 - conn_manager - DEBUG - Connector [CH00001 PASSIVE grpc://0:8002] is created
2026-03-12 18:16:49,144 - CoreCell - INFO - server: created backbone external listener for grpc://0:8002
2026-03-12 18:16:49,144 - ConnectorManager - DEBUG - 23: Try start_listener Listener resources: {'secure': False, 'host': 'localhost', <DriverParams.CONNECTION_SECURITY: 'connection_security'>: 'clear'}
2026-03-12 18:16:49,145 - ConfigService - DEBUG - got var comm_driver_path from env
2026-03-12 18:16:49,148 - conn_manager - DEBUG - Connector [CH00002 PASSIVE tcp://0:5716] is created
2026-03-12 18:16:49,149 - conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:5716] is starting
2026-03-12 18:16:49,150 - ConnectorManager - DEBUG - 23: ############ dynamic listener at tcp://localhost:5716
2026-03-12 18:16:49,650 - CoreCell - INFO - server: created backbone internal listener for tcp://localhost:5716
2026-03-12 18:16:49,651 - conn_manager - INFO - Connector [CH00001 PASSIVE grpc://0:8002] is starting
2026-03-12 18:16:49,695 - communicator - DEBUG - Communicator for local endpoint: server is started
2026-03-12 18:16:49,714 - FederatedServer - DEBUG - trying to remove dead clients .......
2026-03-12 18:16:49,726 - ServerEngine - DEBUG - initialize_comm called!
2026-03-12 18:16:49,755 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'aux_communication', 'topic': '*', 'cb': <bound method ServerEngine._handle_aux_message of <nvflare.private.fed.server.server_engine.ServerEngine object at 0x7f6b6216c200>>}
2026-03-12 18:16:49,755 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 18:16:49,756 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2026-03-12 18:16:49,759 - Server - DEBUG - SERVER: connector params: {'url': 'grpc://0:8002', 'scheme': 'grpc', 'host': '', 'port': '8002', 'path': '', 'params': '', 'query': '', 'frag': '', 'ca_cert': '/app/startup/../startup/rootCA.pem', 'server_cert': '/app/startup/../startup/server.crt', 'server_key': '/app/startup/../startup/server.key', 'connection_security': 'mtls', <DriverParams.SECURE: 'secure'>: True}
2026-03-12 18:16:49,760 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'task', 'topic': 'challenge', 'cb': <bound method FederatedServer.client_challenge of <nvflare.private.fed.server.fed_server.FederatedServer object at 0x7f6b669f5970>>}
2026-03-12 18:16:49,761 - Cell - DEBUG - calling core_cell register_request_cb
2026-03-12 18:16:49,761 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'task', 'topic': 'register', 'cb': <bound method FederatedServer.register_client of <nvflare.private.fed.server.fed_server.FederatedServer object at 0x7f6b669f5970>>}
2026-03-12 18:16:49,761 - Cell - DEBUG - calling core_cell register_request_cb
2026-03-12 18:16:49,762 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'task', 'topic': 'quit', 'cb': <bound method FederatedServer.quit_client of <nvflare.private.fed.server.fed_server.FederatedServer object at 0x7f6b669f5970>>}
2026-03-12 18:16:49,763 - Cell - DEBUG - calling core_cell register_request_cb
2026-03-12 18:16:49,763 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'task', 'topic': 'heart_beat', 'cb': <bound method FederatedServer.client_heartbeat of <nvflare.private.fed.server.fed_server.FederatedServer object at 0x7f6b669f5970>>}
2026-03-12 18:16:49,764 - Cell - DEBUG - calling core_cell register_request_cb
2026-03-12 18:16:49,764 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'task', 'topic': 'report_job_failure', 'cb': <bound method FederatedServer.process_job_failure of <nvflare.private.fed.server.fed_server.FederatedServer object at 0x7f6b669f5970>>}
2026-03-12 18:16:49,765 - Cell - DEBUG - calling core_cell register_request_cb
2026-03-12 18:16:49,765 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'server_parent_listener', 'topic': '*', 'cb': <bound method FederatedServer._listen_command of <nvflare.private.fed.server.fed_server.FederatedServer object at 0x7f6b669f5970>>}
2026-03-12 18:16:49,765 - Cell - DEBUG - calling core_cell register_request_cb
2026-03-12 18:16:49,766 - ConfigService - DEBUG - got var max_reg_duration from env
2026-03-12 18:16:49,766 - FederatedServer - INFO - max_reg_duration=60.0
2026-03-12 18:16:49,767 - ConfigService - DEBUG - got var max_concurrent_streaming_sessions from env
2026-03-12 18:16:49,768 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Request
2026-03-12 18:16:49,768 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Abort
2026-03-12 18:16:49,769 - ServerEngine - DEBUG - set_run_manager is called
2026-03-12 18:16:49,770 - AuxRunner - INFO - registered aux handler for topic fed.event
2026-03-12 18:16:49,771 - ObjectStreamer - DEBUG - registered processor_factory: channel='log_streaming' topic='*' _ChunkConsumerFactory
2026-03-12 18:16:49,772 - ServerDeployer - INFO - deployed FLARE Server.
2026-03-12 18:16:49,780 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'hci_channel', 'topic': '*', 'cb': <bound method AdminServer._process_admin_request of <nvflare.private.fed.server.admin.FedAdminServer object at 0x7f6b5f0a74a0>>}
2026-03-12 18:16:49,781 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 18:16:49,781 - Cell - INFO - Register blob CB for channel='hci_channel', topic='*'
2026-03-12 18:16:49,781 - Server - INFO - added secure port at 0.0.0.0:8002
2026-03-12 18:16:49,784 - ObjectStreamer - DEBUG - registered processor_factory: channel='hci.upload' topic='*' _ChunkConsumerFactory
2026-03-12 18:16:49,784 - hci - INFO - Admin Server is started
2026-03-12 18:16:49,784 - root - INFO - Server started
2026-03-12 18:16:50,268 - ServerState - DEBUG - handle_sd_callback Got SP: sp.name='fl.stag.flip.aicentre.co.uk' sp.fl_port='8002' sp.primary=True self.host='fl.stag.flip.aicentre.co.uk' self.service_port='8002'
2026-03-12 18:16:50,268 - ServerState - INFO - Got the primary sp: fl.stag.flip.aicentre.co.uk fl_port: 8002 SSID: ebc6125d-0a56-4688-9b08-355fe9e4d61a. Turning to hot.
2026-03-12 18:16:50,270 - SimpleJobDefManager - DEBUG - [identity=server, run=?] - objects to scan: 0
2026-03-12 18:16:54,786 - FederatedServer - DEBUG - trying to remove dead clients .......
2026-03-12 18:16:59,800 - FederatedServer - DEBUG - trying to remove dead clients .......
2026-03-12 18:17:04,803 - FederatedServer - DEBUG - trying to remove dead clients .......
2026-03-12 18:17:09,807 - FederatedServer - DEBUG - trying to remove dead clients .......
2026-03-12 18:17:14,810 - FederatedServer - DEBUG - trying to remove dead clients .......
2026-03-12 18:17:19,814 - FederatedServer - DEBUG - trying to remove dead clients .......

FL client container logs:

❯ docker logs trust1-fl-client-net-1-1 | head -n 100
🔧 Generating resources.json from template...
✅ resources.json written to /app/local/resources.json
🔧 Generating log_config.json from template...
✅ log_config.json written to /app/local/log_config.json
[entrypoint] Cleaning up stale processes and files...
[entrypoint] Starting FL client with:
  NUM_AVAILABLE_GPUS=0
  MEMORY_PER_GPU_IN_GIB=0
  LOG_LEVEL=DEBUG
WORKSPACE set to /app/startup/..
PYTHONPATH is /local/custom:
start fl because of no pid.fl
new pid 23
2026-03-12 18:21:53,586 - FederatedClient - DEBUG - got relay_conn_props: {}
2026-03-12 18:21:53,587 - FederatedClient - DEBUG - me='CP': my_fqcn='Trust_1' root_url='grpc://fl.stag.flip.aicentre.co.uk:8002' parent_url=None
2026-03-12 18:21:53,587 - ConfigFactory - DEBUG - search file basename:'comm_config', search dir = /app/startup/..
2026-03-12 18:21:54,326 - ConfigService - DEBUG - got var max_message_size from env
2026-03-12 18:21:54,326 - CoreCell - DEBUG - connection secure: True
2026-03-12 18:21:54,327 - CoreCell - DEBUG - Trust_1: max_msg_size=2144337904
2026-03-12 18:21:54,327 - CoreCell - DEBUG - Creating Cell: Trust_1
2026-03-12 18:21:54,327 - ConfigService - DEBUG - got var heartbeat_interval from env
2026-03-12 18:21:54,328 - ConfigService - DEBUG - got var backbone_conn_gen from env
2026-03-12 18:21:54,328 - ConfigService - DEBUG - got var internal_conn_scheme from env
2026-03-12 18:21:54,328 - ConfigService - DEBUG - got var allow_adhoc_conns from env
2026-03-12 18:21:54,328 - ConfigService - DEBUG - got var adhoc_conn_scheme from env
2026-03-12 18:21:54,329 - ConnectorManager - DEBUG - internal scheme=tcp, resources={'host': 'localhost', <DriverParams.CONNECTION_SECURITY: 'connection_security'>: 'clear'}
2026-03-12 18:21:54,329 - ConnectorManager - DEBUG - adhoc scheme=tcp, resources={}
2026-03-12 18:21:54,414 - ConfigService - DEBUG - got var streaming_chunk_size from env
2026-03-12 18:21:54,415 - Cell - DEBUG - __getattr__: args=(), kwargs={}
2026-03-12 18:21:54,415 - Cell - DEBUG - calling core_cell start
2026-03-12 18:21:54,415 - CoreCell - DEBUG - Trust_1: creating connector to grpc://fl.stag.flip.aicentre.co.uk:8002
2026-03-12 18:21:54,587 - ConfigService - DEBUG - got var job_query_timeout from env
2026-03-12 18:21:54,590 - ConfigService - DEBUG - got var max_concurrent_streaming_sessions from env
2026-03-12 18:21:54,591 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Request
2026-03-12 18:21:54,592 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Abort
2026-03-12 18:21:54,593 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'aux_communication', 'topic': '*', 'cb': <bound method ClientEngine._handle_aux_message of <nvflare.private.fed.client.client_engine.ClientEngine object at 0x7bfee81602c0>>}
2026-03-12 18:21:54,593 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 18:21:54,593 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2026-03-12 18:21:54,594 - AuxRunner - INFO - registered aux handler for topic fed.event
2026-03-12 18:21:54,599 - Communicator - INFO - Waiting for the client cell to be created.
2026-03-12 18:21:54,922 - driver_manager - DEBUG - Driver AioHttpDriver is registered for http
2026-03-12 18:21:54,922 - driver_manager - DEBUG - Driver AioHttpDriver is registered for https
2026-03-12 18:21:54,923 - driver_manager - DEBUG - Driver AioHttpDriver is registered for ws
2026-03-12 18:21:54,923 - driver_manager - DEBUG - Driver AioHttpDriver is registered for wss
2026-03-12 18:21:55,024 - ConfigService - DEBUG - got var use_aio_grpc from env
2026-03-12 18:21:55,024 - driver_manager - DEBUG - Driver AioGrpcDriver is registered for agrpc
2026-03-12 18:21:55,024 - driver_manager - DEBUG - Driver AioGrpcDriver is registered for agrpcs
2026-03-12 18:21:55,025 - ConfigService - DEBUG - got var use_aio_grpc from env
2026-03-12 18:21:55,025 - driver_manager - DEBUG - Driver GrpcDriver is registered for grpc
2026-03-12 18:21:55,025 - driver_manager - DEBUG - Driver GrpcDriver is registered for grpcs
2026-03-12 18:21:55,029 - driver_manager - DEBUG - Driver TcpDriver is registered for tcp
2026-03-12 18:21:55,029 - driver_manager - DEBUG - Driver TcpDriver is registered for stcp
2026-03-12 18:21:55,033 - driver_manager - DEBUG - Driver AioTcpDriver is registered for atcp
2026-03-12 18:21:55,033 - driver_manager - DEBUG - Driver AioTcpDriver is registered for satcp
2026-03-12 18:21:55,034 - ConfigService - DEBUG - got var comm_driver_path from env
2026-03-12 18:21:55,034 - GrpcDriver - DEBUG - GRPC Config: max_workers=100, options=[('grpc.max_send_message_length', 2145386496), ('grpc.max_receive_message_length', 2145386496)]
2026-03-12 18:21:55,034 - conn_manager - DEBUG - Connector [CH00001 ACTIVE grpc://fl.stag.flip.aicentre.co.uk:8002] is created
2026-03-12 18:21:55,035 - CoreCell - INFO - Trust_1: created backbone external connector to grpc://fl.stag.flip.aicentre.co.uk:8002
2026-03-12 18:21:55,035 - ConnectorManager - DEBUG - 24: Try start_listener Listener resources: {'secure': False, 'host': 'localhost', <DriverParams.CONNECTION_SECURITY: 'connection_security'>: 'clear'}
2026-03-12 18:21:55,036 - ConfigService - DEBUG - got var comm_driver_path from env
2026-03-12 18:21:55,036 - conn_manager - DEBUG - Connector [CH00002 PASSIVE tcp://0:1623] is created
2026-03-12 18:21:55,036 - conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:1623] is starting
2026-03-12 18:21:55,037 - ConnectorManager - DEBUG - 24: ############ dynamic listener at tcp://localhost:1623
2026-03-12 18:21:55,100 - Communicator - INFO - Waiting for the client cell to be created.
2026-03-12 18:21:55,538 - CoreCell - INFO - Trust_1: created backbone internal listener for tcp://localhost:1623
2026-03-12 18:21:55,538 - conn_manager - INFO - Connector [CH00001 ACTIVE grpc://fl.stag.flip.aicentre.co.uk:8002] is starting
2026-03-12 18:21:55,539 - GrpcDriver - DEBUG - CLIENT: trying connect ...
2026-03-12 18:21:55,539 - communicator - DEBUG - Communicator for local endpoint: Trust_1 is started
2026-03-12 18:21:55,540 - GrpcDriver - DEBUG - CLIENT: creating secure channel
2026-03-12 18:21:55,540 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'server_command', 'topic': 'get_task', 'cb': <bound method Communicator._process_get_task of <nvflare.private.fed.client.communicator.Communicator object at 0x7bfee9dc2240>>}
2026-03-12 18:21:55,541 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 18:21:55,541 - Cell - INFO - Register blob CB for channel='server_command', topic='get_task'
2026-03-12 18:21:55,541 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'server_command', 'topic': 'submit_update', 'cb': <bound method Communicator._process_submit_result of <nvflare.private.fed.client.communicator.Communicator object at 0x7bfee9dc2240>>}
2026-03-12 18:21:55,541 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 18:21:55,542 - Cell - INFO - Register blob CB for channel='server_command', topic='submit_update'
2026-03-12 18:21:55,542 - FederatedClient - INFO - Wait for engine to be created.
2026-03-12 18:21:55,567 - GrpcDriver - INFO - created secure channel at fl.stag.flip.aicentre.co.uk:8002
2026-03-12 18:21:55,567 - GrpcDriver - DEBUG - CLIENT: got stub
2026-03-12 18:21:55,568 - conn_manager - INFO - Connection [CN00002 N/A => fl.stag.flip.aicentre.co.uk:8002] is created: PID: 24
2026-03-12 18:21:55,568 - sfm_conn - DEBUG - Sending frame: Prefix(length=71, header_len=0, type=4, reserved=0, flags=0, app_id=0, stream_id=1, sequence=1) on [CN00002 N/A => fl.stag.flip.aicentre.co.uk:8002]
2026-03-12 18:21:55,568 - StreamConnection - DEBUG - CLIENT: queued frame #1
2026-03-12 18:21:55,569 - base_driver - DEBUG - Connection created: GrpcDriver:[CN00002 N/A => fl.stag.flip.aicentre.co.uk:8002]
2026-03-12 18:21:55,569 - GrpcDriver - DEBUG - CLIENT: added connection
2026-03-12 18:21:55,570 - StreamConnection - DEBUG - CLIENT: generate_output in thread Thread-4 (_run)
2026-03-12 18:21:55,570 - StreamConnection - DEBUG - CLIENT: started read_loop in thread conn_mgr_1
2026-03-12 18:21:55,571 - StreamConnection - DEBUG - CLIENT: outgoing frame #1
2026-03-12 18:21:55,606 - Cell - DEBUG - __getattr__: args=(), kwargs={'target': 'server', 'channel': 'task', 'topic': 'challenge', 'request': <nvflare.fuel.f3.message.Message object at 0x7bfee1d707a0>, 'timeout': 30.0, 'optional': True}
2026-03-12 18:21:55,607 - Cell - DEBUG - calling core_cell send_request
2026-03-12 18:21:55,607 - CoreCell - DEBUG - Trust_1: sending request task:challenge to server
2026-03-12 18:21:55,608 - CoreCell - DEBUG - Trust_1: broadcasting to ['server'] ...
2026-03-12 18:21:55,608 - CoreCell - DEBUG - Trust_1: finding path to server
2026-03-12 18:21:55,608 - CoreCell - DEBUG - Trust_1: trying path ['server'] ...
2026-03-12 18:21:55,608 - CoreCell - DEBUG - Trust_1: no CellAgent for server
2026-03-12 18:21:55,609 - CoreCell - DEBUG - [ME=Trust_1 O=? D=server F=? T=? CH=task TP=challenge SCH=? STP=? SEQ=?] no connection to server
2026-03-12 18:21:55,609 - CoreCell - DEBUG - [ME=Trust_1 O=Trust_1 D=server F=Trust_1 T=server CH=task TP=challenge SCH=? STP=? SEQ=?] cannot send to 'server': target_unreachable
2026-03-12 18:21:55,609 - CoreCell - DEBUG - released waiter on REQ 42ca07d1-87f5-4c6d-beb8-9a7258d49bc5
2026-03-12 18:21:55,610 - Authenticator - DEBUG - challenge result: target_unreachable 
2026-03-12 18:21:55,610 - Authenticator - INFO - re-challenge after 2.0 seconds
2026-03-12 18:21:57,611 - Cell - DEBUG - __getattr__: args=(), kwargs={'target': 'server', 'channel': 'task', 'topic': 'challenge', 'request': <nvflare.fuel.f3.message.Message object at 0x7bfee1d9f440>, 'timeout': 30.0, 'optional': True}
2026-03-12 18:21:57,612 - Cell - DEBUG - calling core_cell send_request

Note that if I change host_names: in the provisioning file to be:

host_names: [fl.stag.flip.aicentre.co.uk, fl-server-net-1]

then the FL API successfully connects to FL server. But the FL client seems to be trying to connect to FL server using the internal Docker name fl-server-net-1:8002 (which is not accessible from another EC2 instance) instead of the actual domain name fl.stag.flip.aicentre.co.uk:

2026-03-12 17:43:32,406 - GrpcDriver - DEBUG - CLIENT: trying connect ... 
2026-03-12 17:43:32,406 - GrpcDriver - DEBUG - CLIENT: creating secure channel 
2026-03-12 17:43:32,408 - GrpcDriver - INFO - created secure channel at fl-server-net-1:8002 
2026-03-12 17:43:32,408 - GrpcDriver - DEBUG - CLIENT: got stub 
2026-03-12 17:43:32,408 - conn_manager - INFO - Connection [CN00015 N/A => fl-server-net-1:8002] is created: PID: 23 2026-03-12 17:43:32,409 - sfm_conn - DEBUG - Sending frame: Prefix(length=71, header_len=0, type=4, reserved=0, flags=0, app_id=0, stream_id=1, sequence=1) on [CN00015 N/A => fl-server-net-1:8002] 
2026-03-12 17:43:32,409 - StreamConnection - DEBUG - CLIENT: queued frame #14 
2026-03-12 17:43:32,409 - base_driver - DEBUG - Connection created: GrpcDriver:[CN00015 N/A => fl-server-net-1:8002] 2026-03-12 17:43:32,409 - GrpcDriver - DEBUG - CLIENT: added connection 
2026-03-12 17:43:32,410 - StreamConnection - DEBUG - CLIENT: generate_output in thread Thread-30 (_run) 2026-03-12 17:43:32,410 - StreamConnection - DEBUG - CLIENT: started read_loop in thread conn_mgr_1 
2026-03-12 17:43:32,411 - StreamConnection - DEBUG - CLIENT: outgoing frame #14 
2026-03-12 17:43:32,413 - StreamConnection - DEBUG - CLIENT: exception <class 'grpc._channel._MultiThreadedRendezvous'> in read_loop 
2026-03-12 17:43:32,413 - StreamConnection - DEBUG - CLIENT: closing queue 
2026-03-12 17:43:32,414 - StreamConnection - DEBUG - CLIENT in conn_mgr_1: done read_loop
2026-03-12 17:43:32,414 - base_driver - DEBUG - Connection removed: GrpcDriver:[CN00015 N/A => fl-server-net-1:8002] 2026-03-12 17:43:32,414 - conn_manager - INFO - Connection [CN00015 Not Connected] is closed PID: 23 
2026-03-12 17:43:32,414 - conn_manager - DEBUG - Connection CN00015 is closed before SFM handshake 
2026-03-12 17:43:32,415 - GrpcDriver - INFO - CLIENT: finished connection [CN00015 Not Connected]
2026-03-12 17:43:32,415 - conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://fl-server-net-1:8002] in 10 seconds

I've tried many combinations with little luck so any pointers would be greatly appreciated!! 🙏

If it comes down to the AWS setup I am using, please could you share working examples of AWS networking setups?

Thank you for your time !!

Alex

[atriaybagur]

Update: with connect_to in provisioning script https://github.com/londonaicentre/flip-fl-base/blob/66-feature-fl-port-consolidation/net-1_project_stag.yml#L21 I can get the FL API container to connect to FL server, but the FL client still can't connect:

FL API container logs:

❯ docker logs flip-fl-api-net-1 
🚢 Starting API in normal mode... 
INFO:     Will watch for changes in these directories: ['/app/fl_api']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [10] using WatchFiles
INFO:     Started server process [12]
INFO:     Waiting for application startup.
INFO:     Running FL startup initialization...
INFO:     Admin directory set to: /app/admin
INFO:     Default GPU resource_spec for each job: 0 GPUs, 0 GiB per GPU
INFO:     Using LOG_LEVEL: INFO
INFO:     Upload directory set to: /app/admin/transfer
INFO:     Download directory set to: /app/admin/transfer
INFO:     FL session initialized successfully.
INFO:     Application startup complete.
Connecting to FLARE ...
INFO:     10.0.1.117:9372 - "GET / HTTP/1.1" 200 OK

FL server container logs:

❯ docker logs fl-server-net-1 
🔧 Generating log_config.json from template...
✅ log_config.json written to /app/local/log_config.json
[entrypoint] Starting FL server with:
  LOG_LEVEL=INFO
WORKSPACE set to /app/startup/..
PYTHONPATH is /local/custom:
start fl because of no pid.fl
new pid 23
2026-03-12 20:12:37,556 - ServerDeployer - INFO - server heartbeat timeout set to 600
2026-03-12 20:12:38,359 - CoreCell - INFO - server: creating listener on grpc://0:8002
2026-03-12 20:12:38,930 - CoreCell - INFO - server: created backbone external listener for grpc://0:8002
2026-03-12 20:12:38,931 - conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:63167] is starting
2026-03-12 20:12:39,432 - CoreCell - INFO - server: created backbone internal listener for tcp://localhost:63167
2026-03-12 20:12:39,433 - conn_manager - INFO - Connector [CH00001 PASSIVE grpc://0:8002] is starting
2026-03-12 20:12:39,434 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2026-03-12 20:12:39,435 - FederatedServer - INFO - max_reg_duration=60.0
2026-03-12 20:12:39,448 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Request
2026-03-12 20:12:39,457 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Abort
2026-03-12 20:12:39,460 - AuxRunner - INFO - registered aux handler for topic fed.event
2026-03-12 20:12:39,461 - Server - INFO - added secure port at 0.0.0.0:8002
2026-03-12 20:12:39,462 - ServerDeployer - INFO - deployed FLARE Server.
2026-03-12 20:12:39,466 - Cell - INFO - Register blob CB for channel='hci_channel', topic='*'
2026-03-12 20:12:39,467 - hci - INFO - Admin Server is started
2026-03-12 20:12:39,467 - root - INFO - Server started
2026-03-12 20:12:39,948 - ServerState - INFO - Got the primary sp: fl.stag.flip.aicentre.co.uk fl_port: 8002 SSID: ebc6125d-0a56-4688-9b08-355fe9e4d61a. Turning to hot.
2026-03-12 20:12:42,874 - conn_manager - INFO - Connection [CN00002 0.0.0.0:8002 <= ipv4:172.22.0.5:44218 SSL admin@nvidia.com] is created: PID: 25
2026-03-12 20:12:43,916 - ClientManager - INFO - Client: New admin admin@nvidia.com@172.22.0.5 joined. Sent token: 5b18e212-ebe3-4df1-ba18-359b8efa1e7c.  Total clients: 0
2026-03-12 20:12:44,009 - LoginModule - INFO - Created user session for admin@nvidia.com

FL client container logs (debug log mode)

❯ docker logs trust1-fl-client-net-1-1 
🔧 Generating resources.json from template...
✅ resources.json written to /app/local/resources.json
🔧 Generating log_config.json from template...
✅ log_config.json written to /app/local/log_config.json
[entrypoint] Cleaning up stale processes and files...
[entrypoint] Starting FL client with:
  NUM_AVAILABLE_GPUS=0
  MEMORY_PER_GPU_IN_GIB=0
  LOG_LEVEL=DEBUG
WORKSPACE set to /app/startup/..
PYTHONPATH is /local/custom:
start fl because of no pid.fl
new pid 23
2026-03-12 20:18:32,862 - FederatedClient - DEBUG - got relay_conn_props: {}
2026-03-12 20:18:32,863 - FederatedClient - DEBUG - me='CP': my_fqcn='Trust_1' root_url='grpc://fl.stag.flip.aicentre.co.uk:8002' parent_url=None
2026-03-12 20:18:32,863 - ConfigFactory - DEBUG - search file basename:'comm_config', search dir = /app/startup/..
2026-03-12 20:18:33,571 - ConfigService - DEBUG - got var max_message_size from env
2026-03-12 20:18:33,571 - CoreCell - DEBUG - connection secure: True
2026-03-12 20:18:33,572 - CoreCell - DEBUG - Trust_1: max_msg_size=2144337904
2026-03-12 20:18:33,572 - CoreCell - DEBUG - Creating Cell: Trust_1
2026-03-12 20:18:33,573 - ConfigService - DEBUG - got var heartbeat_interval from env
2026-03-12 20:18:33,573 - ConfigService - DEBUG - got var backbone_conn_gen from env
2026-03-12 20:18:33,573 - ConfigService - DEBUG - got var internal_conn_scheme from env
2026-03-12 20:18:33,573 - ConfigService - DEBUG - got var allow_adhoc_conns from env
2026-03-12 20:18:33,574 - ConfigService - DEBUG - got var adhoc_conn_scheme from env
2026-03-12 20:18:33,574 - ConnectorManager - DEBUG - internal scheme=tcp, resources={'host': 'localhost', <DriverParams.CONNECTION_SECURITY: 'connection_security'>: 'clear'}
2026-03-12 20:18:33,574 - ConnectorManager - DEBUG - adhoc scheme=tcp, resources={}
2026-03-12 20:18:33,659 - ConfigService - DEBUG - got var streaming_chunk_size from env
2026-03-12 20:18:33,660 - Cell - DEBUG - __getattr__: args=(), kwargs={}
2026-03-12 20:18:33,660 - Cell - DEBUG - calling core_cell start
2026-03-12 20:18:33,661 - CoreCell - DEBUG - Trust_1: creating connector to grpc://fl.stag.flip.aicentre.co.uk:8002
2026-03-12 20:18:33,864 - ConfigService - DEBUG - got var job_query_timeout from env
2026-03-12 20:18:33,871 - ConfigService - DEBUG - got var max_concurrent_streaming_sessions from env
2026-03-12 20:18:33,873 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Request
2026-03-12 20:18:33,873 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Abort
2026-03-12 20:18:33,873 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'aux_communication', 'topic': '*', 'cb': <bound method ClientEngine._handle_aux_message of <nvflare.private.fed.client.client_engine.ClientEngine object at 0x7ef15ad661b0>>}
2026-03-12 20:18:33,874 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 20:18:33,875 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2026-03-12 20:18:33,875 - AuxRunner - INFO - registered aux handler for topic fed.event
2026-03-12 20:18:33,876 - Communicator - INFO - Waiting for the client cell to be created.
2026-03-12 20:18:34,142 - driver_manager - DEBUG - Driver AioHttpDriver is registered for http
2026-03-12 20:18:34,143 - driver_manager - DEBUG - Driver AioHttpDriver is registered for https
2026-03-12 20:18:34,143 - driver_manager - DEBUG - Driver AioHttpDriver is registered for ws
2026-03-12 20:18:34,143 - driver_manager - DEBUG - Driver AioHttpDriver is registered for wss
2026-03-12 20:18:34,239 - ConfigService - DEBUG - got var use_aio_grpc from env
2026-03-12 20:18:34,239 - driver_manager - DEBUG - Driver AioGrpcDriver is registered for agrpc
2026-03-12 20:18:34,240 - driver_manager - DEBUG - Driver AioGrpcDriver is registered for agrpcs
2026-03-12 20:18:34,240 - ConfigService - DEBUG - got var use_aio_grpc from env
2026-03-12 20:18:34,240 - driver_manager - DEBUG - Driver GrpcDriver is registered for grpc
2026-03-12 20:18:34,240 - driver_manager - DEBUG - Driver GrpcDriver is registered for grpcs
2026-03-12 20:18:34,244 - driver_manager - DEBUG - Driver TcpDriver is registered for tcp
2026-03-12 20:18:34,244 - driver_manager - DEBUG - Driver TcpDriver is registered for stcp
2026-03-12 20:18:34,248 - driver_manager - DEBUG - Driver AioTcpDriver is registered for atcp
2026-03-12 20:18:34,248 - driver_manager - DEBUG - Driver AioTcpDriver is registered for satcp
2026-03-12 20:18:34,249 - ConfigService - DEBUG - got var comm_driver_path from env
2026-03-12 20:18:34,249 - GrpcDriver - DEBUG - GRPC Config: max_workers=100, options=[('grpc.max_send_message_length', 2145386496), ('grpc.max_receive_message_length', 2145386496)]
2026-03-12 20:18:34,249 - conn_manager - DEBUG - Connector [CH00001 ACTIVE grpc://fl.stag.flip.aicentre.co.uk:8002] is created
2026-03-12 20:18:34,250 - CoreCell - INFO - Trust_1: created backbone external connector to grpc://fl.stag.flip.aicentre.co.uk:8002
2026-03-12 20:18:34,250 - ConnectorManager - DEBUG - 24: Try start_listener Listener resources: {'secure': False, 'host': 'localhost', <DriverParams.CONNECTION_SECURITY: 'connection_security'>: 'clear'}
2026-03-12 20:18:34,251 - ConfigService - DEBUG - got var comm_driver_path from env
2026-03-12 20:18:34,251 - conn_manager - DEBUG - Connector [CH00002 PASSIVE tcp://0:58308] is created
2026-03-12 20:18:34,251 - conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:58308] is starting
2026-03-12 20:18:34,252 - ConnectorManager - DEBUG - 24: ############ dynamic listener at tcp://localhost:58308
2026-03-12 20:18:34,386 - Communicator - INFO - Waiting for the client cell to be created.
2026-03-12 20:18:34,753 - CoreCell - INFO - Trust_1: created backbone internal listener for tcp://localhost:58308
2026-03-12 20:18:34,753 - conn_manager - INFO - Connector [CH00001 ACTIVE grpc://fl.stag.flip.aicentre.co.uk:8002] is starting
2026-03-12 20:18:34,754 - GrpcDriver - DEBUG - CLIENT: trying connect ...
2026-03-12 20:18:34,754 - GrpcDriver - DEBUG - CLIENT: creating secure channel
2026-03-12 20:18:34,755 - communicator - DEBUG - Communicator for local endpoint: Trust_1 is started
2026-03-12 20:18:34,755 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'server_command', 'topic': 'get_task', 'cb': <bound method Communicator._process_get_task of <nvflare.private.fed.client.communicator.Communicator object at 0x7ef15cfd3260>>}
2026-03-12 20:18:34,756 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 20:18:34,756 - Cell - INFO - Register blob CB for channel='server_command', topic='get_task'
2026-03-12 20:18:34,757 - Cell - DEBUG - __getattr__: args=(), kwargs={'channel': 'server_command', 'topic': 'submit_update', 'cb': <bound method Communicator._process_submit_result of <nvflare.private.fed.client.communicator.Communicator object at 0x7ef15cfd3260>>}
2026-03-12 20:18:34,776 - Cell - DEBUG - calling cell register_request_cb
2026-03-12 20:18:34,776 - Cell - INFO - Register blob CB for channel='server_command', topic='submit_update'
2026-03-12 20:18:34,777 - FederatedClient - INFO - Wait for engine to be created.
2026-03-12 20:18:34,786 - GrpcDriver - INFO - created secure channel at fl.stag.flip.aicentre.co.uk:8002
2026-03-12 20:18:34,787 - GrpcDriver - DEBUG - CLIENT: got stub
2026-03-12 20:18:34,787 - conn_manager - INFO - Connection [CN00002 N/A => fl.stag.flip.aicentre.co.uk:8002] is created: PID: 24
2026-03-12 20:18:34,788 - sfm_conn - DEBUG - Sending frame: Prefix(length=71, header_len=0, type=4, reserved=0, flags=0, app_id=0, stream_id=1, sequence=1) on [CN00002 N/A => fl.stag.flip.aicentre.co.uk:8002]
2026-03-12 20:18:34,788 - StreamConnection - DEBUG - CLIENT: queued frame #1
2026-03-12 20:18:34,788 - base_driver - DEBUG - Connection created: GrpcDriver:[CN00002 N/A => fl.stag.flip.aicentre.co.uk:8002]
2026-03-12 20:18:34,789 - GrpcDriver - DEBUG - CLIENT: added connection
2026-03-12 20:18:34,789 - StreamConnection - DEBUG - CLIENT: generate_output in thread Thread-4 (_run)
2026-03-12 20:18:34,790 - StreamConnection - DEBUG - CLIENT: started read_loop in thread conn_mgr_1
2026-03-12 20:18:34,790 - StreamConnection - DEBUG - CLIENT: outgoing frame #1
2026-03-12 20:18:34,893 - Cell - DEBUG - __getattr__: args=(), kwargs={'target': 'server', 'channel': 'task', 'topic': 'challenge', 'request': <nvflare.fuel.f3.message.Message object at 0x7ef1549cce60>, 'timeout': 30.0, 'optional': True}
2026-03-12 20:18:34,893 - Cell - DEBUG - calling core_cell send_request
2026-03-12 20:18:34,893 - CoreCell - DEBUG - Trust_1: sending request task:challenge to server
2026-03-12 20:18:34,894 - CoreCell - DEBUG - Trust_1: broadcasting to ['server'] ...
2026-03-12 20:18:34,894 - CoreCell - DEBUG - Trust_1: finding path to server
2026-03-12 20:18:34,894 - CoreCell - DEBUG - Trust_1: trying path ['server'] ...
2026-03-12 20:18:34,895 - CoreCell - DEBUG - Trust_1: no CellAgent for server
2026-03-12 20:18:34,895 - CoreCell - DEBUG - [ME=Trust_1 O=? D=server F=? T=? CH=task TP=challenge SCH=? STP=? SEQ=?] no connection to server
2026-03-12 20:18:34,895 - CoreCell - DEBUG - [ME=Trust_1 O=Trust_1 D=server F=Trust_1 T=server CH=task TP=challenge SCH=? STP=? SEQ=?] cannot send to 'server': target_unreachable
2026-03-12 20:18:34,895 - CoreCell - DEBUG - released waiter on REQ 187f2d83-2d22-484f-bf47-62cbae01e669
2026-03-12 20:18:34,896 - Authenticator - DEBUG - challenge result: target_unreachable 
2026-03-12 20:18:34,896 - Authenticator - INFO - re-challenge after 2.0 seconds
2026-03-12 20:18:36,896 - Cell - DEBUG - __getattr__: args=(), kwargs={'target': 'server', 'channel': 'task', 'topic': 'challenge', 'request': <nvflare.fuel.f3.message.Message object at 0x7ef1549cce60>, 'timeout': 30.0, 'optional': True}
2026-03-12 20:18:36,897 - Cell - DEBUG - calling core_cell send_request
2026-03-12 20:18:36,897 - CoreCell - DEBUG - Trust_1: sending request task:challenge to server
2026-03-12 20:18:36,897 - CoreCell - DEBUG - Trust_1: broadcasting to ['server'] ...
2026-03-12 20:18:36,897 - CoreCell - DEBUG - Trust_1: finding path to server
2026-03-12 20:18:36,898 - CoreCell - DEBUG - Trust_1: trying path ['server'] ...
2026-03-12 20:18:36,898 - CoreCell - DEBUG - Trust_1: no CellAgent for server
2026-03-12 20:18:36,898 - CoreCell - DEBUG - [ME=Trust_1 O=? D=server F=? T=? CH=task TP=challenge SCH=? STP=? SEQ=?] no connection to server
2026-03-12 20:18:36,899 - CoreCell - DEBUG - [ME=Trust_1 O=Trust_1 D=server F=Trust_1 T=server CH=task TP=challenge SCH=? STP=? SEQ=?] cannot send to 'server': target_unreachable
2026-03-12 20:18:36,899 - CoreCell - DEBUG - released waiter on REQ 0de24463-25ca-4942-a295-60201b680ab5
2026-03-12 20:18:36,899 - Authenticator - DEBUG - challenge result: target_unreachable 
2026-03-12 20:18:36,899 - Authenticator - INFO - re-challenge after 2.0 seconds
2026-03-12 20:18:38,900 - Cell - DEBUG - __getattr__: args=(), kwargs={'target': 'server', 'channel': 'task', 'topic': 'challenge', 'request': <nvflare.fuel.f3.message.Message object at 0x7ef1549cc050>, 'timeout': 30.0, 'optional': True}
2026-03-12 20:18:38,900 - Cell - DEBUG - calling core_cell send_request
2026-03-12 20:18:38,901 - CoreCell - DEBUG - Trust_1: sending request task:challenge to server
2026-03-12 20:18:38,901 - CoreCell - DEBUG - Trust_1: broadcasting to ['server'] ...
2026-03-12 20:18:38,901 - CoreCell - DEBUG - Trust_1: finding path to server
2026-03-12 20:18:38,901 - CoreCell - DEBUG - Trust_1: trying path ['server'] ...
2026-03-12 20:18:38,902 - CoreCell - DEBUG - Trust_1: no CellAgent for server
2026-03-12 20:18:38,902 - CoreCell - DEBUG - [ME=Trust_1 O=? D=server F=? T=? CH=task TP=challenge SCH=? STP=? SEQ=?] no connection to server
2026-03-12 20:18:38,902 - CoreCell - DEBUG - [ME=Trust_1 O=Trust_1 D=server F=Trust_1 T=server CH=task TP=challenge SCH=? STP=? SEQ=?] cannot send to 'server': target_unreachable
2026-03-12 20:18:38,902 - CoreCell - DEBUG - released waiter on REQ d8634c43-1ce4-4cb8-9791-c7d93cd432fa
2026-03-12 20:18:38,902 - Authenticator - DEBUG - challenge result: target_unreachable 
2026-03-12 20:18:38,903 - Authenticator - INFO - re-challenge after 2.0 seconds
2026-03-12 20:18:39,755 - sfm_conn - DEBUG - Trying to send heartbeat before SFM Endpoint is established
2026-03-12 20:18:39,755 - heartbeat_monitor - DEBUG - Heartbeat sent to connection: [CN00002 N/A => fl.stag.flip.aicentre.co.uk:8002]
2026-03-12 20:18:40,903 - Cell - DEBUG - __getattr__: args=(), kwargs={'target': 'server', 'channel': 'task', 'topic': 'challenge', 'request': <nvflare.fuel.f3.message.Message object at 0x7ef1549f0770>, 'timeout': 30.0, 'optional': True}
2026-03-12 20:18:40,904 - Cell - DEBUG - calling core_cell send_request
2026-03-12 20:18:40,904 - CoreCell - DEBUG - Trust_1: sending request task:challenge to server
2026-03-12 20:18:40,904 - CoreCell - DEBUG - Trust_1: broadcasting to ['server'] ...
2026-03-12 20:18:40,904 - CoreCell - DEBUG - Trust_1: finding path to server
2026-03-12 20:18:40,905 - CoreCell - DEBUG - Trust_1: trying path ['server'] ...
2026-03-12 20:18:40,905 - CoreCell - DEBUG - Trust_1: no CellAgent for server
2026-03-12 20:18:40,905 - CoreCell - DEBUG - [ME=Trust_1 O=? D=server F=? T=? CH=task TP=challenge SCH=? STP=? SEQ=?] no connection to server
2026-03-12 20:18:40,905 - CoreCell - DEBUG - [ME=Trust_1 O=Trust_1 D=server F=Trust_1 T=server CH=task TP=challenge SCH=? STP=? SEQ=?] cannot send to 'server': target_unreachable
2026-03-12 20:18:40,906 - CoreCell - DEBUG - released waiter on REQ 1075e21d-9dfd-4361-b443-544666d39394
2026-03-12 20:18:40,906 - Authenticator - DEBUG - challenge result: target_unreachable 
2026-03-12 20:18:40,906 - Authenticator - INFO - re-challenge after 2.0 seconds
2026-03-12 20:18:42,907 - Cell - DEBUG - __getattr__: args=(), kwargs={'target': 'server', 'channel': 'task', 'topic': 'challenge', 'request': <nvflare.fuel.f3.message.Message object at 0x7ef1549cfd40>, 'timeout': 30.0, 'optional': True}
2026-03-12 20:18:42,907 - Cell - DEBUG - calling core_cell send_request
2026-03-12 20:18:42,907 - CoreCell - DEBUG - Trust_1: sending request task:challenge to server
2026-03-12 20:18:42,908 - CoreCell - DEBUG - Trust_1: broadcasting to ['server'] ...
2026-03-12 20:18:42,908 - CoreCell - DEBUG - Trust_1: finding path to server
2026-03-12 20:18:42,908 - CoreCell - DEBUG - Trust_1: trying path ['server'] ...
2026-03-12 20:18:42,909 - CoreCell - DEBUG - Trust_1: no CellAgent for server
2026-03-12 20:18:42,909 - CoreCell - DEBUG - [ME=Trust_1 O=? D=server F=? T=? CH=task TP=challenge SCH=? STP=? SEQ=?] no connection to server
2026-03-12 20:18:42,909 - CoreCell - DEBUG - [ME=Trust_1 O=Trust_1 D=server F=Trust_1 T=server CH=task TP=challenge SCH=? STP=? SEQ=?] cannot send to 'server': target_unreachable
2026-03-12 20:18:42,909 - CoreCell - DEBUG - released waiter on REQ aba5f90d-b97f-49d7-9433-9060ce0d6aba
2026-03-12 20:18:42,910 - Authenticator - DEBUG - challenge result: target_unreachable 
2026-03-12 20:18:42,910 - Authenticator - INFO - re-challenge after 2.0 seconds