|
| 1 | +# Hello PyTorch execution environments |
| 2 | + |
| 3 | +This advanced continuation moves the [Hello PyTorch](../../hello-world/hello-pt/README.md) application from local |
| 4 | +simulation to a local POC federation and then to a provisioned production system. It reuses the beginner example's |
| 5 | +actual `client.py`, `model.py`, and `prepare_data.py`; only the Recipe options and execution environment change. |
| 6 | +Keep this directory in a full NVFlare checkout alongside `examples/hello-world/hello-pt`; copying this directory alone |
| 7 | +does not include the shared application. |
| 8 | + |
| 9 | +Complete the beginner simulation first so you already understand its training, evaluation, and artifacts: |
| 10 | + |
| 11 | +```bash |
| 12 | +cd examples/hello-world/hello-pt |
| 13 | +python job.py |
| 14 | +``` |
| 15 | + |
| 16 | +Then enter this directory and install the matching NVFlare 2.10 dependencies: |
| 17 | + |
| 18 | +```bash |
| 19 | +cd ../../advanced/hello-pt-environments |
| 20 | +python -m pip install -r requirements.txt |
| 21 | +``` |
| 22 | + |
| 23 | +## Compare the three environments |
| 24 | + |
| 25 | +| Stage | What changes | Command | |
| 26 | +| --- | --- | --- | |
| 27 | +| Simulation | Server and clients run through the local simulator. | `python job.py` | |
| 28 | +| POC | `PocEnv` provisions, starts, and stops a local system for this job. | `python job.py --env poc` | |
| 29 | +| Production | `ProdEnv` connects to an already-running system through an authorized admin startup kit. | `python job.py --env prod --startup-kit <admin-kit> --username <admin-identity>` | |
| 30 | + |
| 31 | +The Recipe, model, client script, data code, and local training loop stay the same across all three stages. With the |
| 32 | +fixed seeds, local runs currently report 75% final accuracy on site-1 and 77% on site-2; these are observations, |
| 33 | +not benchmark claims or exact CI thresholds. Premerge CI checks that simulation and POC produce identical per-site |
| 34 | +final accuracies. Unit tests cover production argument handling, environment construction, and job export; they do |
| 35 | +not connect to a live production federation. |
| 36 | + |
| 37 | +## Run a job-scoped local POC |
| 38 | + |
| 39 | +```bash |
| 40 | +python job.py --env poc |
| 41 | +``` |
| 42 | + |
| 43 | +`PocEnv` provisions a local federation, starts separate server and client processes, submits the same application, |
| 44 | +downloads the result, and stops the services. It needs permission to start processes and bind the standard NVFlare |
| 45 | +POC ports. The POC lifecycle belongs to this invocation, so provisioning and process startup occur for every job and |
| 46 | +make this deliberately slower than simulation. |
| 47 | + |
| 48 | +Each invocation uses a unique workspace beside the configured CLI POC workspace, with a `.recipe-<UUID>` suffix. |
| 49 | +On success, the command prints and retains that workspace so the result and service logs remain available across |
| 50 | +later runs. The configured CLI workspace and earlier Recipe results are preserved. Stop any running CLI POC with |
| 51 | +`nvflare poc stop` before starting this example, since the services still need the same local ports. |
| 52 | + |
| 53 | +If a job returns a downloaded result but its status is unsuccessful or unavailable, the command stops the services, |
| 54 | +retains the result and workspace, prints the result path and `poc_console.log` locations, and exits nonzero. This keeps |
| 55 | +the client errors available for diagnosis. Monitoring errors or interruptions before a result is obtained trigger |
| 56 | +service shutdown and workspace cleanup. `PocEnv` handles deployment failures itself. If shutdown cannot be verified, |
| 57 | +it preserves the workspace and reports recovery instructions. |
| 58 | + |
| 59 | +If `NVFLARE_HOME` is set, POC admin transfers use `$NVFLARE_HOME/examples`, so downloaded results can be outside the |
| 60 | +retained POC workspace. The printed result path is authoritative. To keep downloads inside the run workspace, |
| 61 | +unset `NVFLARE_HOME` before running this example. Removing a retained workspace does not remove externally downloaded |
| 62 | +results. |
| 63 | + |
| 64 | +### Stop an interrupted run and remove retained artifacts |
| 65 | + |
| 66 | +An interrupted Recipe run has its own workspace. Target that exact path to stop its services; an ordinary |
| 67 | +`nvflare poc stop` targets the separate CLI workspace. For example, substitute the workspace printed by your run: |
| 68 | + |
| 69 | +```bash |
| 70 | +NVFLARE_POC_WORKSPACE="/tmp/nvflare/poc.recipe-<UUID>" nvflare poc stop |
| 71 | +``` |
| 72 | + |
| 73 | +If the process was killed before it printed the workspace, locate the run's `.recipe-<UUID>` directory beside the |
| 74 | +configured CLI POC workspace and inspect its service logs. After confirming its services have stopped and saving any |
| 75 | +artifacts you need, remove that specific directory: |
| 76 | + |
| 77 | +```bash |
| 78 | +rm -rf "/tmp/nvflare/poc.recipe-<UUID>" |
| 79 | +``` |
| 80 | + |
| 81 | +Remove any downloaded result outside that workspace separately, using the printed result path. Each invocation |
| 82 | +retains its own directory, so repeat this for the particular old runs you no longer need. |
| 83 | + |
| 84 | +## Connect to an existing production system |
| 85 | + |
| 86 | +A production submission requires a running provisioned NVFlare system, network connectivity, and an authorized admin |
| 87 | +startup kit. `--username` must match the identity represented by that kit; it defaults to `admin@nvidia.com`. |
| 88 | + |
| 89 | +```bash |
| 90 | +python job.py --env prod \ |
| 91 | + --startup-kit /path/to/admin/startup-kit \ |
| 92 | + --username researcher@example.com |
| 93 | +``` |
| 94 | + |
| 95 | +This integrated Recipe path constructs the job, submits it, waits for completion, downloads the result, and prints |
| 96 | +its location. `ProdEnv` does not start or stop the provisioned system; the server and clients must already be running |
| 97 | +and ready for connections. |
| 98 | + |
| 99 | +## Export for CLI-managed submission |
| 100 | + |
| 101 | +Export the environment-independent job when you want to inspect or edit its generated configuration and submit it to |
| 102 | +an already-running POC or production system with the NVFlare CLI: |
| 103 | + |
| 104 | +```bash |
| 105 | +python job.py --export --export-dir /tmp/nvflare/jobs |
| 106 | +``` |
| 107 | + |
| 108 | +For a reusable local POC, prepare it once, start it, and submit as many jobs as needed before stopping it: |
| 109 | + |
| 110 | +```bash |
| 111 | +nvflare poc prepare -n 2 |
| 112 | +nvflare poc start |
| 113 | +nvflare job submit -j /tmp/nvflare/jobs/hello-pt |
| 114 | + |
| 115 | +# Replace JOB_ID with the ID printed by the submit command. |
| 116 | +nvflare job monitor JOB_ID |
| 117 | +nvflare job download JOB_ID -o /tmp/nvflare/hello-pt-results |
| 118 | + |
| 119 | +# Keep the POC running for more jobs, then stop it when finished. |
| 120 | +nvflare poc stop |
| 121 | +``` |
| 122 | + |
| 123 | +For production, the provisioned system must already be running. Register and activate its admin startup kit, then use |
| 124 | +the same job commands: |
| 125 | + |
| 126 | +```bash |
| 127 | +nvflare config add hello-pt-admin /path/to/admin/startup-kit |
| 128 | +nvflare config use hello-pt-admin |
| 129 | +nvflare job submit -j /tmp/nvflare/jobs/hello-pt |
| 130 | + |
| 131 | +# Replace JOB_ID with the ID printed by the submit command. |
| 132 | +nvflare job monitor JOB_ID |
| 133 | +nvflare job download JOB_ID -o /tmp/nvflare/hello-pt-results |
| 134 | +``` |
| 135 | + |
| 136 | +Unlike `PocEnv`, `nvflare job submit` does not own the system lifecycle: it assumes the selected POC or production |
| 137 | +system is already running and leaves it running after the job. The command itself returns without waiting. The active |
| 138 | +`nvflare config` selection supplies the startup kit and its admin identity to subsequent CLI commands; the Recipe |
| 139 | +script's `--username` option is only for the integrated `python job.py --env prod` path. |
| 140 | + |
| 141 | +Export alone verifies construction of the deployable application. It does not prove connectivity, authorization, or |
| 142 | +successful execution on an external system. See the [Job CLI guide](../../../docs/user_guide/nvflare_cli/job_cli.rst) |
| 143 | +for startup-kit selection and job lifecycle commands, and the |
| 144 | +[deployment guide](../../../docs/user_guide/admin_guide/deployment/index.rst) for provisioning and production |
| 145 | +operations. |
| 146 | + |
| 147 | +## Advanced Recipe controls |
| 148 | + |
| 149 | +Run `python job.py --help` for all example and Recipe export options. The most useful combinations are: |
| 150 | + |
| 151 | +```bash |
| 152 | +# Persist client metrics through a server-side TensorBoard receiver. |
| 153 | +python -m pip install tensorboard |
| 154 | +python job.py --experiment_tracking tensorboard |
| 155 | + |
| 156 | +# Evaluate the final client models as well as the server models. |
| 157 | +python job.py --evaluation cross-site |
| 158 | + |
| 159 | +# Run the shared client script out of process and stream its logs. |
| 160 | +python job.py --launch_external_process --enable_log_streaming |
| 161 | + |
| 162 | +# Periodically release client model parameters and run garbage collection. |
| 163 | +python job.py --client_memory_gc_rounds 1 |
| 164 | + |
| 165 | +# Override selected local-training controls; omitted values remain client-owned defaults. |
| 166 | +python job.py --epochs 2 --batch_size 16 --learning_rate 0.05 --num_workers 0 |
| 167 | +``` |
| 168 | + |
| 169 | +The shared client defaults to one local epoch, batch size 32, and no data-loader worker processes. Its SGD learning |
| 170 | +rate is 0.1 for synthetic images and 0.01 for CIFAR-10 unless `--learning_rate` overrides it. |
| 171 | + |
| 172 | +`--evaluation none` produces a plain FedAvg job without post-training model evaluation. The default remains |
| 173 | +`--evaluation final`, matching the beginner quickstart. `--evaluation cross-site` additionally collects each client's |
| 174 | +latest local model and evaluates all submitted client and server models. |
| 175 | + |
| 176 | +## Optional CIFAR-10 path |
| 177 | + |
| 178 | +All simulated or local POC clients run on the same machine, so they can share one cache and the same logical CIFAR-10 |
| 179 | +datasets. Prepare both splits once before any clients start, and pass the same client-local path to the job: |
| 180 | + |
| 181 | +```bash |
| 182 | +python ../../hello-world/hello-pt/prepare_data.py --data_root "/data/cifar cache" |
| 183 | +python job.py --dataset cifar10 --data_root "/data/cifar cache" |
| 184 | +``` |
| 185 | + |
| 186 | +Each client checks for missing or empty cache files when loading data and reports the preparation command in its |
| 187 | +error log. This happens after simulation or POC starts; `job.py` does not validate the cache, and exports do not |
| 188 | +require local data. Failed POC jobs retain their results and service logs as described above. Use an absolute |
| 189 | +client-local `--data_root` path to avoid depending on a client process's working directory. Clients open the cache |
| 190 | +with downloads disabled, so concurrent processes do not race while writing it. The client checks file presence and |
| 191 | +nonzero size; torchvision performs its own integrity checks when loading. This option is useful for experimentation |
| 192 | +but is not a federated data partition. |
| 193 | + |
| 194 | +For production, `--data_root` is a path on each client—not on the admin machine running `job.py`. Every site operator |
| 195 | +must prepare CIFAR-10 at that same local path before the job is submitted. Running `prepare_data.py` beside the admin |
| 196 | +startup kit does not populate remote clients. Use the synthetic default unless client-side data has been staged. |
0 commit comments