[Q&A] default HA project.yml failed to run

[jeffydc]

Python version (python3 -V)

Python 3.12.11

NVFlare version (python3 -m pip list | grep "nvflare")

2.6.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

NixOS 25.05 (I am running inside venv so this shouldn't be an issue)

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I ran nvflare provision and choose 1 which is project.yml with HA mode and run nvflare provison again, but it exited with

Unable to handle command: provision due to: cannot add participant server2 as server - server already exists

I also tried nvflare poc prepare -i project.yml to test POC locally and it also exited with

project should only have one server, but 2 are provided: ['server1', 'server2']

Here is the generated project.yml, I didn't edit anything:

api_version: 3
name: example_project
description: NVIDIA FLARE sample project yaml file

participants:
  # Change the name of the overseer to the Fully Qualified Domain Name (FQDN)
  # of the overseer, for example: overseer.example.com.
  # Ensure that the FQDN is correctly mapped in the /etc/hosts file.
  - name: overseer
    type: overseer
    org: nvidia
    protocol: https
    api_root: /api/v1
    port: 8443
  # Change the name of the server (server1) to the Fully Qualified Domain Name
  # (FQDN) of the first server, for example: server1.example.com.
  # Ensure that the FQDN is correctly mapped in the /etc/hosts file.
  - name: server1
    type: server
    org: nvidia
    fed_learn_port: 8002
    admin_port: 8003
  # Change the name of the server (server2) to the Fully Qualified Domain Name
  # (FQDN) of the second server, for example: server2.example.com.
  # Ensure that the FQDN is correctly mapped in the /etc/hosts file.
  - name: server2
    type: server
    org: nvidia
    fed_learn_port: 8102
    admin_port: 8103
  - name: site-1
    type: client
    org: nvidia
    # listening_host will enable creating one pair of cert/private key for this client
    # so it can behave like a server for client api.  The value must be a hostname that
    # client api can reach via network.
    # listening_host: site-1-lh
  - name: site-2
    type: client
    org: nvidia
  - name: admin@nvidia.com
    type: admin
    org: nvidia
    role: project_admin

# The same methods in all builders are called in their order defined in builders section
builders:
  - path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    args:
      template_file:
        - master_template.yml
        - aws_template.yml
        - azure_template.yml
  - path: nvflare.lighter.impl.template.TemplateBuilder
  - path: nvflare.lighter.impl.docker.DockerBuilder
    args:
      base_image: python:3.8
      requirements_file: docker_compose_requirements.txt
  - path: nvflare.lighter.impl.helm_chart.HelmChartBuilder
    args:
      docker_image: localhost:32000/nvfl-min:0.0.1
  - path: nvflare.lighter.impl.static_file.StaticFileBuilder
    args:
      # config_folder can be set to inform NVIDIA FLARE where to get configuration
      config_folder: config

      # scheme for communication driver (currently supporting the default, grpc, only).
      # scheme: grpc

      # app_validator is used to verify if uploaded app has proper structures
      # if not set, no app_validator is included in fed_server.json
      # app_validator: PATH_TO_YOUR_OWN_APP_VALIDATOR

      # when docker_image is set to a docker image name, docker.sh will be generated on server/client/admin
      # docker_image:

      # download_job_url is set to http://download.server.com/ as default in fed_server.json.  You can override this
      # to different url.
      # download_job_url: http://download.server.com/

      overseer_agent:
        path: nvflare.ha.overseer_agent.HttpOverseerAgent
        # if overseer_exists is true, args here are ignored.  Provisioning
        #   tool will fill role, name and other local parameters automatically.
        # if overseer_exists is false, args in this section will be used.
        overseer_exists: true
        # args:
        #   sp_end_point: example1.com.8002:8003

  - path: nvflare.lighter.impl.cert.CertBuilder
  # - path: nvflare.lighter.impl.he.HEBuilder
  #   args:
  #     poly_modulus_degree: 8192
  #     coeff_mod_bit_sizes: [60, 40, 40]
  #     scale_bits: 40
  #     scheme: CKKS
  - path: nvflare.lighter.impl.signature.SignatureBuilder

Is this a bug or am I doing something wrong?

Thanks!

[YuanTingHsieh]

Hi @ydcjeff , thanks for the issue!

Our POC mode's purpose is to start the whole system in one machine and user can play with it.
So it DOES NOT support the "HA" project.yml.

And the other thing is we are deprecating this HA in 2.7.
So please choose 2.

We will be updating our documentation and code as well.
Thank you!

[jeffydc]

Thanks for your reply. So when using v2.6 and HA is required, what should we use instead?

[chesterxgchen]

No, using v2.6, HA is not required. Currently a dummy overseer is used.

The HA feature is depreciated, we will remove from the project.yml configuration in the next release (soon)

[jeffydc]

Thanks for your reply. And sorry that my question wasn't clear.
I am trying to ask if my project needs high availability requirement, what should I use instead?

[chesterxgchen]

Since we are depreciating this feature, we have been gradually disabling the feature. Lets try to use alternative ways

One possible alternative :

• if you can use a load balance with two FL servers, one is standby FL server. You can probably setup in such a way, that the 2nd FL server will active when 1st one is no longer available. During switch, the internal state will be lost, you might have to resume from last check-point.
  (FLARE HA will keep and transfer the state)

[chesterxgchen]

The feature is actually disabled is the main branch, not only deprecated. We just need to update the documentation

[chesterxgchen]

Looks like in release 2.6.2, the feature is broken due to the refactoring as well.

https://github.com/NVIDIA/NVFlare/blob/2.6/nvflare/lighter/entity.py#L464

[or9327]

In version 2.7, can we consider that HttpOverseer (not the dummy overseer) has been deprecated? Also, does that mean it's a structure where snapshots cannot be saved either?

[YuanTingHsieh]

yes it has been deprecated, we are expecting to propose a new architecture to handle High Availability