Skip to content

Commit 32a88fe

Browse files
holgerrothholgerroth
authored
[2.7] Redesign Job-level Authorization Example (#4074)
Fixes # . ### Description Redesign job-level-authorization example to utilize Recipe and ProdEnv. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Quick tests passed locally by running `./runtest.sh`. - [ ] In-line docstrings updated. - [ ] Documentation updated.
1 parent 6cdf953 commit 32a88fe

File tree

11 files changed

+283
-44
lines changed

11 files changed

+283
-44
lines changed

‎examples/advanced/job-level-authorization/README.md‎

Lines changed: 37 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ source nvflare-env/bin/activate
1616
```
1717
2. Install NVFlare
1818
```
19-
pip install nvflare
19+
pip install -r requirements.txt
2020
```
2121
3. The example is part of the NVFlare source code. The source code can be obtained like this,
2222
```
@@ -39,6 +39,13 @@ All the startup kits will be generated in this folder,
3939
/tmp/nvflare/poc/job-level-authorization/prod_00
4040
```
4141

42+
**Important**: The `setup.sh` script performs the following operations:
43+
1. Removes the workspace folder (if it exists) and regenerates the POC environment
44+
2. Prepares the POC deployment with the specified configuration
45+
3. **Overwrites site_a's security settings** by copying the custom security handler from `security/site_a/*` to `/tmp/nvflare/poc/job-level-authorization/prod_00/site_a/local`
46+
47+
This custom security configuration installs the `CustomSecurityHandler` that enforces job-level authorization on site_a, blocking jobs named "FL-Demo-Job2" while allowing all other jobs.
48+
4249
Note that the "workspace" folder is removed every time `setup.sh` is run. Please do not save customized files in this folder.
4350

4451
### Starting NVFlare
@@ -48,33 +55,51 @@ This script will start up the server and 2 clients,
4855
nvflare poc start
4956
```
5057

51-
### Logging with Admin Console
58+
### Submitting Jobs to ProdEnv
5259

53-
For example, to login as the `[email protected]` user:
60+
Here, we treat the created POC environment as a production environemnt running in the background.
61+
You can submit jobs programmatically using the Recipe API with `ProdEnv`. Two example scripts are provided:
5462

63+
**job1.py** - Submits a job named "hello-numpy" (**ALLOWED by site_a**):
5564
```
56-
cd /tmp/nvflare/poc/job-level-authorization/prod_00/[email protected]
57-
./startup/fl_admin.sh
65+
python job1.py
5866
```
5967

60-
At the prompt, enter the user email `[email protected]`
68+
**job2.py** - Submits a job named "FL-Demo-Job2" (**BLOCKED by site_a**):
69+
```
70+
python job2.py
71+
```
6172

62-
The setup.sh has copied the jobs folder to the workspace folder.
63-
So jobs can be submitted like this, type the following command in the admin console:
73+
Both scripts use `ProdEnv` to connect to the production deployment and submit jobs via the Flare API. The jobs demonstrate how site_a's `CustomSecurityHandler` enforces authorization based on job name:
74+
- Job 1 with name "hello-numpy" will be accepted by both site_a and site_b
75+
- Job 2 with name "FL-Demo-Job2" will be rejected by site_a but accepted by site_b
6476

77+
You can customize the startup kit location and username using command-line arguments:
6578
```
66-
submit_job ../../job1
67-
submit_job ../../job2
79+
python job1.py --startup_kit_location /path/to/startup_kit --username [email protected]
6880
```
6981

7082
## Participants
7183

7284
### Site
7385
* `server1`: NVFlare server
74-
* `site_a`: Site_a has a CustomSecurityHandler set up which does not allow the job "FL Demo Job1" to run. Any other named jobs will be able to deploy and run on site_a.
86+
* `site_a`: Site_a has a CustomSecurityHandler set up which does not allow the job "FL-Demo-Job2" to run. Any other named jobs will be able to deploy and run on site_a.
7587
* `site_b`: Site_b does not have the extra security handling codes. It allows any job to be deployed and run.
7688

7789
### Jobs
7890

7991
* job1: The job is called `hello-numpy`. site_a will allow this job to run.
80-
* job2: The job is called `FL Demo Job1`. site_a will block this job to run.
92+
* job2: The job is called `FL-Demo-Job2`. site_a will block this job to run.
93+
94+
### Output
95+
96+
For job1, you will see successful completion with training on both clients (site_a & site_b).
97+
98+
For job2, you will see an output like this in the POC log messages:
99+
100+
```
101+
2026-01-30 12:41:51,006 - site_security - ERROR - Authorization failed. Reason: Job 'FL-Demo-Job2' BLOCKED by site_a's CustomSecurityHandler - not authorized to execute: check_resources
102+
2026-01-30 12:41:51,008 - ServerEngine - ERROR - Client reply error: Job 'FL-Demo-Job2' BLOCKED by site_a's CustomSecurityHandler - not authorized to execute: check_resources
103+
```
104+
Only site_b will execute the training run.
105+

‎examples/advanced/job-level-authorization/client.py‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
"""
15+
client side training scripts
16+
"""
17+
18+
import argparse
19+
20+
import numpy as np
21+
22+
import nvflare.client as flare
23+
from nvflare.app_common.np.constants import NPConstants
24+
from nvflare.client.tracking import SummaryWriter
25+
26+
27+
def train(input_numpy_array: np.ndarray) -> np.ndarray:
28+
"""Mock training of the model by adding 1 to the input numpy array.
29+
In a real application, this would be actual model training.
30+
"""
31+
return input_numpy_array + 1
32+
33+
34+
def evaluate(input_numpy_array: np.ndarray) -> dict:
35+
"""Mock evaluation of the model by returning the mean of the input numpy array.
36+
In a real application, this would be actual model evaluation.
37+
"""
38+
return {"weight_mean": np.mean(input_numpy_array)}
39+
40+
41+
def main():
42+
parser = argparse.ArgumentParser()
43+
parser.add_argument("--update_type", choices=["full", "diff"], default="full")
44+
args = parser.parse_args()
45+
46+
# Initialize FLARE
47+
flare.init()
48+
sys_info = flare.system_info()
49+
client_name = sys_info["site_name"]
50+
51+
print(f"Client {client_name} initialized")
52+
53+
# Initialize summary writer for tracking
54+
summary_writer = SummaryWriter()
55+
56+
while flare.is_running():
57+
# Receive model from server
58+
input_model = flare.receive()
59+
print(f"Client {client_name}, current_round={input_model.current_round}")
60+
61+
# Get model parameters
62+
input_np_arr = input_model.params[NPConstants.NUMPY_KEY]
63+
print(f"Received weights: {input_np_arr}")
64+
65+
new_params = train(input_np_arr)
66+
67+
# Evaluate the model
68+
metrics = evaluate(new_params)
69+
print(f"Client {client_name} evaluation metrics: {metrics}")
70+
71+
# Log metrics to summary writer
72+
global_step = input_model.current_round
73+
summary_writer.add_scalar(tag="weight_mean", scalar=metrics["weight_mean"], global_step=global_step)
74+
75+
print(f"Client {client_name} finished training for round {input_model.current_round}")
76+
if args.update_type == "diff":
77+
params_to_send = new_params - input_np_arr
78+
params_type = flare.ParamsType.DIFF
79+
else:
80+
params_to_send = new_params
81+
params_type = flare.ParamsType.FULL
82+
83+
# Send updated model back to server
84+
print(f"Sending weights: {params_to_send}")
85+
output_model = flare.FLModel(
86+
params={NPConstants.NUMPY_KEY: params_to_send},
87+
params_type=params_type,
88+
metrics=metrics,
89+
current_round=input_model.current_round,
90+
)
91+
92+
flare.send(output_model)
93+
94+
95+
if __name__ == "__main__":
96+
main()

‎examples/advanced/job-level-authorization/job1.py‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
"""
15+
This job demonstrates job-level authorization with ProdEnv.
16+
Job1 has the name "hello-numpy" which is ALLOWED by site_a's CustomSecurityHandler.
17+
"""
18+
import argparse
19+
20+
from nvflare.app_common.np.recipes.fedavg import NumpyFedAvgRecipe
21+
from nvflare.recipe import ProdEnv
22+
23+
24+
def define_parser():
25+
parser = argparse.ArgumentParser()
26+
parser.add_argument(
27+
"--startup_kit_location",
28+
type=str,
29+
default="/tmp/nvflare/poc/job-level-authorization/prod_00/[email protected]",
30+
help="Path to the admin startup kit directory",
31+
)
32+
parser.add_argument("--username", type=str, default="[email protected]", help="Username for authentication")
33+
34+
return parser.parse_args()
35+
36+
37+
def main():
38+
args = define_parser()
39+
40+
# Create recipe with job name "hello-numpy" - this is ALLOWED by site_a
41+
recipe = NumpyFedAvgRecipe(
42+
name="hello-numpy", # This name is ALLOWED by site_a's security handler
43+
min_clients=1,
44+
num_rounds=1,
45+
initial_model=[[1, 2, 3], [4, 5, 6], [7, 8, 9]],
46+
train_script="client.py",
47+
)
48+
49+
print("Submitting job with name: 'hello-numpy' (ALLOWED by site_a)")
50+
print(f"Using startup kit at: {args.startup_kit_location}")
51+
print(f"Authenticating as: {args.username}")
52+
53+
# Use ProdEnv to submit to the production environment
54+
env = ProdEnv(startup_kit_location=args.startup_kit_location, username=args.username)
55+
56+
print("Submitting job to production environment...")
57+
run = recipe.execute(env)
58+
print()
59+
print("Result can be found in:", run.get_result())
60+
print("Job Status is:", run.get_status())
61+
print()
62+
print("SUCCESS: Job 'hello-numpy' should be ACCEPTED by site_a's security handler")
63+
64+
65+
if __name__ == "__main__":
66+
main()

‎examples/advanced/job-level-authorization/job2.py‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
"""
15+
This job demonstrates job-level authorization with ProdEnv.
16+
Job2 has the name "FL-Demo-Job2" which is BLOCKED by site_a's CustomSecurityHandler.
17+
18+
This script must be run from the job-level-authorization directory so that
19+
the client.py script can be found and included in the job package.
20+
"""
21+
import argparse
22+
23+
from nvflare.app_common.np.recipes.fedavg import NumpyFedAvgRecipe
24+
from nvflare.recipe import ProdEnv
25+
26+
27+
def define_parser():
28+
parser = argparse.ArgumentParser()
29+
parser.add_argument(
30+
"--startup_kit_location",
31+
type=str,
32+
default="/tmp/nvflare/poc/job-level-authorization/prod_00/[email protected]",
33+
help="Path to the admin startup kit directory",
34+
)
35+
parser.add_argument("--username", type=str, default="[email protected]", help="Username for authentication")
36+
37+
return parser.parse_args()
38+
39+
40+
def main():
41+
args = define_parser()
42+
43+
# Create recipe with job name "FL-Demo-Job2" - this is BLOCKED by site_a
44+
recipe = NumpyFedAvgRecipe(
45+
name="FL-Demo-Job2", # This name is BLOCKED by site_a's security handler
46+
min_clients=1,
47+
num_rounds=1,
48+
initial_model=[[1, 2, 3], [4, 5, 6], [7, 8, 9]],
49+
train_script="client.py",
50+
)
51+
52+
print("Submitting job with name: 'FL-Demo-Job2' (BLOCKED by site_a)")
53+
print(f"Using startup kit at: {args.startup_kit_location}")
54+
print(f"Authenticating as: {args.username}")
55+
56+
# Use ProdEnv to submit to the production environment
57+
env = ProdEnv(startup_kit_location=args.startup_kit_location, username=args.username)
58+
59+
print("Submitting job to production environment...")
60+
run = recipe.execute(env)
61+
print()
62+
print("Job Status is:", run.get_status())
63+
print("Result can be found in:", run.get_result())
64+
print()
65+
print("EXPECTED BEHAVIOR: Job 'FL-Demo-Job2' should be REJECTED by site_a's security handler")
66+
print("site_a will block this job, but site_b (without security handler) will accept it")
67+
68+
69+
if __name__ == "__main__":
70+
main()

‎examples/advanced/job-level-authorization/jobs/job1/meta.json‎

Lines changed: 0 additions & 10 deletions
This file was deleted.

‎examples/advanced/job-level-authorization/jobs/job2/meta.json‎

Lines changed: 0 additions & 11 deletions
This file was deleted.

‎examples/advanced/job-level-authorization/requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
nvflare~=2.5.0rc
1+
nvflare~=2.7.2rc

‎examples/advanced/job-level-authorization/security/site_a/custom/security_handler.py‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
22
#
33
# Licensed under the Apache License, Version 2.0 (the "License");
44
# you may not use this file except in compliance with the License.
@@ -33,8 +33,12 @@ def authorize(self, fl_ctx: FLContext) -> tuple[bool, str]:
3333
if command in ["check_resources"]:
3434
security_items = fl_ctx.get_prop(FLContextKey.SECURITY_ITEMS)
3535
job_meta = security_items.get(FLContextKey.JOB_META)
36-
if job_meta.get(JobMetaKey.JOB_NAME) == "FL Demo Job1":
37-
return False, f"Not authorized to execute: {command}"
36+
job_name = job_meta.get(JobMetaKey.JOB_NAME)
37+
if job_name == "FL-Demo-Job2":
38+
return (
39+
False,
40+
f"Job '{job_name}' BLOCKED by site_a's CustomSecurityHandler - not authorized to execute: {command}",
41+
)
3842
else:
3943
return True, ""
4044
else:

‎examples/advanced/job-level-authorization/security/site_a/resources.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,11 @@
1919
"path": "nvflare.app_common.resource_consumers.gpu_resource_consumer.GPUResourceConsumer",
2020
"args": {}
2121
},
22+
{
23+
"id": "process_launcher",
24+
"path": "nvflare.app_common.job_launcher.client_process_launcher.ClientProcessJobLauncher",
25+
"args": {}
26+
},
2227
{
2328
"id": "security_handler",
2429
"path": "security_handler.CustomSecurityHandler"

‎examples/advanced/job-level-authorization/setup.sh‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,4 @@ nvflare poc prepare -i project.yml -c site_a
77
WORKSPACE="/tmp/nvflare/poc/job-level-authorization/prod_00"
88
cp -r security/site_a/* $WORKSPACE/site_a/local
99

10-
for i in {1..2}
11-
do
12-
cp -r ../../hello-world/hello-numpy $WORKSPACE/job$i
13-
cp -r jobs/job$i/* $WORKSPACE/job$i
14-
done
15-
1610
echo Your workspace is "$WORKSPACE"

0 commit comments

Comments
 (0)