[Q&A] Failed to get engine and Login failed

[Plachido]

Python version (python3 -V)

3.10

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.1

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 22.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Hello!
I am working with two machines: one remote one (but still on premise) and a laptop running windows.
On the remote machine, I have run multiple examples of simulation and POC without any problems. However, I am struggling to configure a real world scenario.

As nvflare is not (yet) available for windows, I am using Ubuntu 22.04 on WSL1. I started off trying with WSL2, but I had some connectivity issues because I needed to use a VPN. The VPN is necessary because the laptop and the remote machine are on the same network this way.

Here is my project.yml

api_version: 3
name: secure_project
description: NVIDIA FLARE sample project yaml

participants:
  - name: server-fl
    type: server
    org: nvidia
    fed_learn_port: 8102
    admin_port: 8103
  - name: client1
    type: client
    org: nvidia
  - name: [email protected]
    type: admin
    org: nvidia
    role: project_admin

builders:
  - path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    args:
      template_file: master_template.yml
  - path: nvflare.lighter.impl.template.TemplateBuilder
  - path: nvflare.lighter.impl.static_file.StaticFileBuilder
    args:
      config_folder: config
      overseer_agent:
        path: nvflare.ha.dummy_overseer_agent.DummyOverseerAgent
        overseer_exists: false
        args:
           sp_end_point: 0.0.0.0:8102:8103
           heartbeat_interval: 6
  - path: nvflare.lighter.impl.he.HEBuilder
    args:
      poly_modulus_degree: 8192
      coeff_mod_bit_sizes: [60, 40, 40]
      scale_bits: 40
      scheme: CKKS
  - path: nvflare.lighter.impl.cert.CertBuilder
  - path: nvflare.lighter.impl.signature.SignatureBuilder

It's not yet very clear to me why I need the overseer, which is why I am sticking with the dummy one for the moment.
I have created, on both my machines, the entries for server-fl and client1 in /etc/hosts.

The server log is the following:

new pid 3628388
2024-07-25 18:57:44,833 - CoreCell - INFO - server: creating listener on grpc://0:8102
2024-07-25 18:57:44,841 - CoreCell - INFO - server: created backbone external listener for grpc://0:8102
2024-07-25 18:57:44,841 - ConnectorManager - INFO - 3628388: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-07-25 18:57:44,842 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:30085] is starting
2024-07-25 18:57:45,343 - CoreCell - INFO - server: created backbone internal listener for tcp://localhost:30085
2024-07-25 18:57:45,344 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 PASSIVE grpc://0:8102] is starting
deployed FL server trainer.
2024-07-25 18:57:45,361 - nvflare.fuel.f3.drivers.grpc_driver.Server - INFO - added secure port at 0.0.0.0:8102
2024-07-25 18:57:45,373 - nvflare.fuel.hci.server.hci - INFO - Starting Admin Server server-fl on Port 8103
2024-07-25 18:57:45,373 - root - INFO - Server started

It remains stuck like this indefinitly, since it seems like no connection arrives.
It's interesting that changing 0.0.0.0 in sp_end_point with "server-fl" results in an additional part in the log:

2024-07-25 18:36:23,106 - ServerState - INFO - Got the primary sp: server-fl fl_port: 8102 SSID: ebc6125d-0a56-4688-9b08-355fe9e4d61a. Turning to hot.

Client side, I have the following log:

2024-07-25 18:56:46,449 - CoreCell - INFO - client1: created backbone external connector to grpc://0.0.0.0:8102
2024-07-25 18:56:46,450 - ConnectorManager - INFO - 1389: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-07-25 18:56:46,453 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:29877] is starting
2024-07-25 18:56:46,955 - CoreCell - INFO - client1: created backbone internal listener for tcp://localhost:29877
2024-07-25 18:56:46,957 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE grpc://0.0.0.0:8102] is starting
2024-07-25 18:56:46,960 - FederatedClient - INFO - Wait for engine to be created.
2024-07-25 18:56:46,999 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:56:47,000 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 N/A => 0.0.0.0:8102] is created: PID: 1389
2024-07-25 18:56:47,008 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 1389
2024-07-25 18:56:47,008 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00002 Not Connected]
2024-07-25 18:56:48,028 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:56:48,029 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => 0.0.0.0:8102] is created: PID: 1389
2024-07-25 18:56:48,033 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 1389
2024-07-25 18:56:48,034 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-07-25 18:56:50,050 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:56:50,051 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 N/A => 0.0.0.0:8102] is created: PID: 1389
2024-07-25 18:56:50,057 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 Not Connected] is closed PID: 1389
2024-07-25 18:56:50,058 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00004 Not Connected]
2024-07-25 18:56:54,066 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:56:54,068 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 N/A => 0.0.0.0:8102] is created: PID: 1389
2024-07-25 18:56:54,072 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 Not Connected] is closed PID: 1389
2024-07-25 18:56:54,073 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00005 Not Connected]
2024-07-25 18:56:54,074 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 8 seconds
2024-07-25 18:57:02,091 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:02,092 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 N/A => 0.0.0.0:8102] is created: PID: 1389
2024-07-25 18:57:02,099 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 1389
2024-07-25 18:57:02,099 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00006 Not Connected]
2024-07-25 18:57:02,099 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:57:12,108 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:12,109 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 N/A => 0.0.0.0:8102] is created: PID: 1389
2024-07-25 18:57:12,114 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 Not Connected] is closed PID: 1389
2024-07-25 18:57:12,115 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00007 Not Connected]
2024-07-25 18:57:12,115 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:57:17,355 - MPM - ERROR - Execute exception: RuntimeError: Login failed.
2024-07-25 18:57:17,405 - MPM - ERROR - Traceback (most recent call last):
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/private/fed/app/client/client_train.py", line 120, in main
    raise RuntimeError("Login failed.")
RuntimeError: Login failed.

2024-07-25 18:57:18,993 - MPM - INFO - MPM: Good Bye!
2024-07-25 18:57:21,670 - CoreCell - INFO - client1: created backbone external connector to grpc://0.0.0.0:8102
2024-07-25 18:57:21,670 - ConnectorManager - INFO - 1566: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-07-25 18:57:21,672 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:26168] is starting
2024-07-25 18:57:22,174 - CoreCell - INFO - client1: created backbone internal listener for tcp://localhost:26168
2024-07-25 18:57:22,175 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE grpc://0.0.0.0:8102] is starting
2024-07-25 18:57:22,178 - FederatedClient - INFO - Wait for engine to be created.
2024-07-25 18:57:22,191 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:22,192 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 N/A => 0.0.0.0:8102] is created: PID: 1566
2024-07-25 18:57:22,197 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 1566
2024-07-25 18:57:22,198 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00002 Not Connected]
2024-07-25 18:57:23,221 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:23,223 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => 0.0.0.0:8102] is created: PID: 1566
2024-07-25 18:57:23,229 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 1566
2024-07-25 18:57:23,230 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-07-25 18:57:25,243 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:25,245 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 N/A => 0.0.0.0:8102] is created: PID: 1566
2024-07-25 18:57:25,251 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 Not Connected] is closed PID: 1566
2024-07-25 18:57:25,252 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00004 Not Connected]
2024-07-25 18:57:29,263 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:29,265 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 N/A => 0.0.0.0:8102] is created: PID: 1566
2024-07-25 18:57:29,271 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 Not Connected] is closed PID: 1566
2024-07-25 18:57:29,273 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00005 Not Connected]
2024-07-25 18:57:29,274 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 8 seconds
2024-07-25 18:57:37,297 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:37,299 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 N/A => 0.0.0.0:8102] is created: PID: 1566
2024-07-25 18:57:37,306 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 1566
2024-07-25 18:57:37,307 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00006 Not Connected]
2024-07-25 18:57:37,308 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:57:47,328 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:47,330 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 N/A => 0.0.0.0:8102] is created: PID: 1566
2024-07-25 18:57:47,334 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 Not Connected] is closed PID: 1566
2024-07-25 18:57:47,335 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00007 Not Connected]
2024-07-25 18:57:47,335 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:57:52,687 - MPM - ERROR - Execute exception: RuntimeError: Login failed.
2024-07-25 18:57:52,691 - MPM - ERROR - Traceback (most recent call last):
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/private/fed/app/client/client_train.py", line 120, in main
    raise RuntimeError("Login failed.")
RuntimeError: Login failed.

2024-07-25 18:57:54,211 - MPM - INFO - MPM: Good Bye!
2024-07-25 18:57:56,860 - CoreCell - INFO - client1: created backbone external connector to grpc://0.0.0.0:8102
2024-07-25 18:57:56,861 - ConnectorManager - INFO - 1747: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-07-25 18:57:56,863 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:13161] is starting
2024-07-25 18:57:57,365 - CoreCell - INFO - client1: created backbone internal listener for tcp://localhost:13161
2024-07-25 18:57:57,366 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE grpc://0.0.0.0:8102] is starting
2024-07-25 18:57:57,370 - FederatedClient - INFO - Wait for engine to be created.
2024-07-25 18:57:57,392 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:57,394 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 N/A => 0.0.0.0:8102] is created: PID: 1747
2024-07-25 18:57:57,406 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 1747
2024-07-25 18:57:57,407 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00002 Not Connected]
2024-07-25 18:57:58,422 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:57:58,423 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => 0.0.0.0:8102] is created: PID: 1747
2024-07-25 18:57:58,428 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 1747
2024-07-25 18:57:58,429 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-07-25 18:58:00,441 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:00,443 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 N/A => 0.0.0.0:8102] is created: PID: 1747
2024-07-25 18:58:00,448 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 Not Connected] is closed PID: 1747
2024-07-25 18:58:00,449 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00004 Not Connected]
2024-07-25 18:58:04,458 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:04,460 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 N/A => 0.0.0.0:8102] is created: PID: 1747
2024-07-25 18:58:04,465 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 Not Connected] is closed PID: 1747
2024-07-25 18:58:04,466 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00005 Not Connected]
2024-07-25 18:58:04,467 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 8 seconds
2024-07-25 18:58:12,482 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:12,484 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 N/A => 0.0.0.0:8102] is created: PID: 1747
2024-07-25 18:58:12,489 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 1747
2024-07-25 18:58:12,489 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00006 Not Connected]
2024-07-25 18:58:12,490 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:58:22,504 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:22,508 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 N/A => 0.0.0.0:8102] is created: PID: 1747
2024-07-25 18:58:22,520 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 Not Connected] is closed PID: 1747
2024-07-25 18:58:22,522 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00007 Not Connected]
2024-07-25 18:58:22,523 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:58:27,878 - MPM - ERROR - Execute exception: RuntimeError: Login failed.
2024-07-25 18:58:27,881 - MPM - ERROR - Traceback (most recent call last):
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/private/fed/app/client/client_train.py", line 120, in main
    raise RuntimeError("Login failed.")
RuntimeError: Login failed.

2024-07-25 18:58:29,401 - MPM - INFO - MPM: Good Bye!
2024-07-25 18:58:32,330 - CoreCell - INFO - client1: created backbone external connector to grpc://0.0.0.0:8102
2024-07-25 18:58:32,331 - ConnectorManager - INFO - 1928: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-07-25 18:58:32,333 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:48354] is starting
2024-07-25 18:58:32,835 - CoreCell - INFO - client1: created backbone internal listener for tcp://localhost:48354
2024-07-25 18:58:32,836 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE grpc://0.0.0.0:8102] is starting
2024-07-25 18:58:32,840 - FederatedClient - INFO - Wait for engine to be created.
2024-07-25 18:58:32,854 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:32,854 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 N/A => 0.0.0.0:8102] is created: PID: 1928
2024-07-25 18:58:32,859 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 1928
2024-07-25 18:58:32,859 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00002 Not Connected]
2024-07-25 18:58:33,874 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:33,876 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => 0.0.0.0:8102] is created: PID: 1928
2024-07-25 18:58:33,892 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 1928
2024-07-25 18:58:33,895 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-07-25 18:58:35,902 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:35,903 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 N/A => 0.0.0.0:8102] is created: PID: 1928
2024-07-25 18:58:35,908 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 Not Connected] is closed PID: 1928
2024-07-25 18:58:35,909 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00004 Not Connected]
2024-07-25 18:58:39,925 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:39,926 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 N/A => 0.0.0.0:8102] is created: PID: 1928
2024-07-25 18:58:39,932 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 Not Connected] is closed PID: 1928
2024-07-25 18:58:39,933 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00005 Not Connected]
2024-07-25 18:58:39,934 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 8 seconds
2024-07-25 18:58:47,959 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:47,960 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 N/A => 0.0.0.0:8102] is created: PID: 1928
2024-07-25 18:58:47,969 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 1928
2024-07-25 18:58:47,970 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00006 Not Connected]
2024-07-25 18:58:47,971 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:58:57,990 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:58:57,991 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 N/A => 0.0.0.0:8102] is created: PID: 1928
2024-07-25 18:58:57,996 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 Not Connected] is closed PID: 1928
2024-07-25 18:58:57,997 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00007 Not Connected]
2024-07-25 18:58:57,997 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:59:03,333 - MPM - ERROR - Execute exception: RuntimeError: Login failed.
2024-07-25 18:59:03,337 - MPM - ERROR - Traceback (most recent call last):
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/private/fed/app/client/client_train.py", line 120, in main
    raise RuntimeError("Login failed.")
RuntimeError: Login failed.

2024-07-25 18:59:04,872 - MPM - INFO - MPM: Good Bye!
2024-07-25 18:59:07,828 - CoreCell - INFO - client1: created backbone external connector to grpc://0.0.0.0:8102
2024-07-25 18:59:07,829 - ConnectorManager - INFO - 2105: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-07-25 18:59:07,830 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:4611] is starting
2024-07-25 18:59:08,332 - CoreCell - INFO - client1: created backbone internal listener for tcp://localhost:4611
2024-07-25 18:59:08,333 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE grpc://0.0.0.0:8102] is starting
2024-07-25 18:59:08,336 - FederatedClient - INFO - Wait for engine to be created.
2024-07-25 18:59:08,352 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:59:08,353 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 N/A => 0.0.0.0:8102] is created: PID: 2105
2024-07-25 18:59:08,361 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 2105
2024-07-25 18:59:08,362 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00002 Not Connected]
2024-07-25 18:59:09,374 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:59:09,375 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => 0.0.0.0:8102] is created: PID: 2105
2024-07-25 18:59:09,379 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 2105
2024-07-25 18:59:09,380 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-07-25 18:59:11,387 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:59:11,391 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 N/A => 0.0.0.0:8102] is created: PID: 2105
2024-07-25 18:59:11,397 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 Not Connected] is closed PID: 2105
2024-07-25 18:59:11,398 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00004 Not Connected]
2024-07-25 18:59:15,416 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:59:15,418 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 N/A => 0.0.0.0:8102] is created: PID: 2105
2024-07-25 18:59:15,424 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 Not Connected] is closed PID: 2105
2024-07-25 18:59:15,425 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00005 Not Connected]
2024-07-25 18:59:15,427 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 8 seconds
2024-07-25 18:59:23,447 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:59:23,448 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 N/A => 0.0.0.0:8102] is created: PID: 2105
2024-07-25 18:59:23,458 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 2105
2024-07-25 18:59:23,459 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00006 Not Connected]
2024-07-25 18:59:23,460 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:59:33,471 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at 0.0.0.0:8102
2024-07-25 18:59:33,473 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 N/A => 0.0.0.0:8102] is created: PID: 2105
2024-07-25 18:59:33,479 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 Not Connected] is closed PID: 2105
2024-07-25 18:59:33,480 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00007 Not Connected]
2024-07-25 18:59:33,481 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://0.0.0.0:8102] in 10 seconds
2024-07-25 18:59:38,812 - MPM - ERROR - Execute exception: RuntimeError: Login failed.
2024-07-25 18:59:38,816 - MPM - ERROR - Traceback (most recent call last):
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
  File "/home/client1/FL/nvflare-env/lib/python3.10/site-packages/nvflare/private/fed/app/client/client_train.py", line 120, in main
    raise RuntimeError("Login failed.")
RuntimeError: Login failed.

2024-07-25 18:59:40,370 - MPM - INFO - MPM: Good Bye!

I believe nothing is sent to the server at all. I tried monitoring incoming network traffic with the command
sudo tcpdump port 8102 and '(tcp-syn|tcp-ack)!=0' but nothing appears during the execution of the client startup script.

Notice that I am able to reach the server from the client and viceversa.

Any help is greatly appreciated!

[Plachido]

Reinstalling nvflare made the job

[YuanTingHsieh]

Thanks for providing your question and solution!

[JonasBoehringer]

Hi, I got the same problem but I'm just on client side and reinstalling nvflare didn't help :( . Did you reinstall it on server and client side? Please give me more Info, I would really appreciate.

[Plachido]

Hello @JonasBoehringer, if I remember correctly, I just reinstalled on the client machine and it worked fine.
Are you sure it's not just a connectivity issue? Are you able to reach the server from the client without nvflare?