diff --git a/examples/apps/LICENSE b/examples/apps/LICENSE new file mode 100644 index 00000000..f49a4e16 --- /dev/null +++ b/examples/apps/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/examples/apps/README.md b/examples/apps/README.md new file mode 100644 index 00000000..33eb51a5 --- /dev/null +++ b/examples/apps/README.md @@ -0,0 +1,194 @@ +# MONAI Application Package (MAP) for sample nnunet model + +This README describes the process of converting the [CCHMC Pediatric Airway Segmentation nnUnet model] into a MONAI Application Package (MAP). + +## Convert nnUNet checkpoints to MONAI compatible models + +The `convert_nnunet_ckpts.py` script simplifies the process of converting nnUNet model checkpoints to MONAI bundle format. This conversion is necessary to use nnUNet models within MONAI applications and the MONAI Deploy ecosystem. + +## Example model checkpoints + +Sample nnunet model checkpoints for a UTE MRI airway segmentation in NICU patients are available here + +https://drive.google.com/drive/folders/1lRs-IoLR47M_WFyZmuCaROJULtyPdkLm?usp=drive_link + +### Prerequisites + +Before running the conversion script, ensure that: +1. You have trained nnUNet models available +2. The nnUNet environment variables are set or you can provide them as arguments +3. Python environment with required dependencies is set up (my_app/requirements.txt) + +### Basic Usage + +The script can be executed with the following command: + +```bash +python convert_nnunet_ckpts.py --dataset_name_or_id DATASET_ID --MAP_root OUTPUT_DIR --nnUNet_results RESULTS_PATH +``` + +The RESULTS_PATH should have "inference_information.json" file created by nnunetv2 automatically, as the conversion relies on this to figure out the best model configuration and convert those for the MAP. + +### Command-line Arguments + +| Argument | Description | Required | Default | +|----------|-------------|----------|---------| +| `--dataset_name_or_id` | Name or ID of the nnUNet dataset to convert | Yes | N/A | +| `--MAP_root` | Output directory for the converted MONAI bundle | No | Current directory | +| `--nnUNet_results` | Path to nnUNet results directory with trained models | Yes | Uses environment variable if set | + +#### Example + +Convert dataset with ID 4 to models directory: + +```bash +python convert_nnunet_ckpts.py \ + --dataset_name_or_id 4 \ + --MAP_root "." \ + --nnUNet_results "/path/to/nnunet/models" +``` + +#### Output Structure + +The conversion creates a MONAI bundle with the following structure in the specified `MAP_root` directory: + +``` +MAP_root/ +└── models/ + ├── jsonpkls/ + │ ├── dataset.json # Dataset configuration + │ ├── plans.json # Model planning information + │ ├── postprocessing.pkl # Optional postprocessing configuration + ├── 3d_fullres/ # Model configuration (if present) + │ ├── nnunet_checkpoint.pth + │ └── fold_X/ # Each fold's model weights + │ └── best_model.pt + ├── 3d_lowres/ # Model configuration (if present) + └── 3d_cascade_fullres/ # Model configuration (if present) +``` + +This bundle structure is compatible with MONAI inference tools and the MONAI Deploy application ecosystem. + + +## Setting Up Environment +Instructions regarding installation of MONAI Deploy App SDK and details of the necessary system requirements can be found on the MONAI Deploy App SDK [GitHub Repository](https://github.com/Project-MONAI/monai-deploy-app-sdk) and [docs](https://docs.monai.io/projects/monai-deploy-app-sdk/en/latest/getting_started/installing_app_sdk.html). Instructions on how to create a virtual environment and install other dependencies can be found in the MONAI Deploy App SDK docs under the Creating a Segmentation App Consuming a MONAI Bundle [example](https://docs.monai.io/projects/monai-deploy-app-sdk/en/latest/getting_started/tutorials/monai_bundle_app.html). + +Per MONAI, MONAI Deploy App SDK is required to be run in a Linux environment, specifically Ubuntu 22.04 on X86-64, as this is the only X86 platform that the underlying Holoscan SDK has been tested to support as of now. This project uses Poetry for dependency management, which simplifies setting up the environment with all required dependencies. + +### System Requirements +- **Operating System:** Linux (Ubuntu 22.04 recommended) +- **Architecture:** x86_64 +- **GPU:** NVIDIA GPU (recommended for inference) +- **Python:** 3.10 or newer (project requires >=3.10,<3.13) + + + +## Executing Model Bundle Pythonically +Prior to MAP building, the exported model bundle can be executed pythonically via the command line. + +Within the main directory of this downloaded repository, create a `.env` file. MONAI recommends the following `.env` structure and naming conventions: + +```env +HOLOSCAN_INPUT_PATH=${PWD}/input +HOLOSCAN_MODEL_PATH=${PWD}/models +HOLOSCAN_OUTPUT_PATH=${PWD}/output +``` + +Load in the environment variables: + +``` +source .env +``` + +If already specified, remove the directory specified by the `HOLOSCAN_OUTPUT_PATH` environment variable: + +``` +rm -rf $HOLOSCAN_OUTPUT_PATH +``` + +Execute the model bundle pythonically via the command line; the directory specified by the `HOLOSCAN_INPUT_PATH` environment variable should be created and populated with a DICOM series for testing by the user. The model bundle file should be populated within the `/model` folder to match the recommended `HOLOSCAN_MODEL_PATH` value. `HOLOSCAN_INPUT_PATH`, `HOLOSCAN_OUTPUT_PATH`, and `HOLOSCAN_MODEL_PATH` default values can be amended by updating the `.env` file appropriately. + +``` +python my_app -i "$HOLOSCAN_INPUT_PATH" -o "$HOLOSCAN_OUTPUT_PATH" -m "$HOLOSCAN_MODEL_PATH" +``` + +## Building the MAP +It is recommended that the NVIDIA Clara Holoscan base image is pulled prior to building the MAP. If this base image is not pulled prior to MAP building, it will be done so automically during the build process, which will increase the build time from around 1/2 minutes to around 10/15 minutes. Ensure the base image matches the Holoscan SDK version being used in your environment (e.g. if you are using Holoscan SDK v3.2.0, replace `${holoscan-version}` with `v3.2.0`). + +``` +docker pull nvcr.io/nvidia/clara-holoscan/holoscan:${holoscan-version}-dgpu +``` + +Execute the following command to build the MAP Docker image based on the supported NVIDIA Clara Holoscan base image. During MAP building, a Docker container based on the `moby/buildkit` Docker image will be spun up; this container (Docker BuildKit builder `holoscan_app_builder`) facilitates the MAP build. + +``` +monai-deploy package my_app -m $HOLOSCAN_MODEL_PATH -c my_app/app.yaml -t ${tag_prefix}:${image_version} --platform x86_64 -l DEBUG +``` + +As of August 2024, a new error may appear during the MAP build related to the Dockerfile, where `monai-deploy-app-sdk` v0 (which does not exist) is attempted to be installed: + +```bash +Dockerfile:78 +-------------------- + 76 | + 77 | # Install MONAI Deploy from PyPI org + 78 | >>> RUN pip install monai-deploy-app-sdk==0 + 79 | + 80 | +-------------------- +``` + +If you encounter this error, you can specify the MONAI Deploy App SDK version via `--sdk-version` directly in the build command (`3.0.0`, for example). The base image for the MAP build can also be specified via `--base-image`: + +``` +monai-deploy package my_app -m $HOLOSCAN_MODEL_PATH -c my_app/app.yaml -t ${tag_prefix}:${image_version} --platform x86_64 --base-image ${base_image} --sdk-version ${version} -l DEBUG +``` + +If using Docker Desktop, the MAP should now appear in the "Images" tab as `${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version}`. You can also confirm MAP creation in the CLI by executing this command: + +``` +docker image ls | grep ${tag_prefix} +``` + +## Display and Extract MAP Contents +There are a few commands that can be executed in the command line to view MAP contents. + +To display some basic MAP manifests, use the `show` command. The following command will run and subsequently remove a MAP Docker container; the `show` command will display informaiton about the MAP-associated `app.json` and `pkg.json` files as command line outputs. + +``` +docker run --rm ${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version} show +``` + +MAP manifests and other contents can also be extracted into a specific host folder using the `extract` command. + +The host folder used to store the extracted MAP contents must be created by the host, not by Docker upon running the MAP as a container. This is most applicable when MAP contents are extracted more than once; the export folder must be deleted and recreated in this case. + +``` +rm -rf `pwd`/export && mkdir -p `pwd`/export +``` + +After creating the folder for export, executing the following command will run and subsequently remove a MAP Docker container. + +``` +docker run --rm -v `pwd`/export/:/var/run/holoscan/export/ ${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version} extract +``` + +The `extract` command will extract MAP contents to the `/export` folder, organized as follows: +- `app` folder, which contains of the all the files present in `my_app` +- `config` folder, which contains the MAP manifests (`app.json`, `pkg.json`, and `app.yaml`) +- `models` folder, which contains the model bundle used to created the MAP + +## Executing MAP Locally via the MONAI Application Runner (MAR) +The generated MAP can be tested locally using the MONAI Application Runner (MAR). + +First, clear the contents of the output directory: + +``` +rm -rf $HOLOSCAN_OUTPUT_PATH +``` + +Then, the MAP can be executed locally via the MAR command line utility; input and output directories must be specified: + +``` +monai-deploy run -i $HOLOSCAN_INPUT_PATH -o $HOLOSCAN_OUTPUT_PATH ${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version} +``` diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/LICENSE b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/LICENSE new file mode 100644 index 00000000..f49a4e16 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/README.md b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/README.md new file mode 100644 index 00000000..33eb51a5 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/README.md @@ -0,0 +1,194 @@ +# MONAI Application Package (MAP) for sample nnunet model + +This README describes the process of converting the [CCHMC Pediatric Airway Segmentation nnUnet model] into a MONAI Application Package (MAP). + +## Convert nnUNet checkpoints to MONAI compatible models + +The `convert_nnunet_ckpts.py` script simplifies the process of converting nnUNet model checkpoints to MONAI bundle format. This conversion is necessary to use nnUNet models within MONAI applications and the MONAI Deploy ecosystem. + +## Example model checkpoints + +Sample nnunet model checkpoints for a UTE MRI airway segmentation in NICU patients are available here + +https://drive.google.com/drive/folders/1lRs-IoLR47M_WFyZmuCaROJULtyPdkLm?usp=drive_link + +### Prerequisites + +Before running the conversion script, ensure that: +1. You have trained nnUNet models available +2. The nnUNet environment variables are set or you can provide them as arguments +3. Python environment with required dependencies is set up (my_app/requirements.txt) + +### Basic Usage + +The script can be executed with the following command: + +```bash +python convert_nnunet_ckpts.py --dataset_name_or_id DATASET_ID --MAP_root OUTPUT_DIR --nnUNet_results RESULTS_PATH +``` + +The RESULTS_PATH should have "inference_information.json" file created by nnunetv2 automatically, as the conversion relies on this to figure out the best model configuration and convert those for the MAP. + +### Command-line Arguments + +| Argument | Description | Required | Default | +|----------|-------------|----------|---------| +| `--dataset_name_or_id` | Name or ID of the nnUNet dataset to convert | Yes | N/A | +| `--MAP_root` | Output directory for the converted MONAI bundle | No | Current directory | +| `--nnUNet_results` | Path to nnUNet results directory with trained models | Yes | Uses environment variable if set | + +#### Example + +Convert dataset with ID 4 to models directory: + +```bash +python convert_nnunet_ckpts.py \ + --dataset_name_or_id 4 \ + --MAP_root "." \ + --nnUNet_results "/path/to/nnunet/models" +``` + +#### Output Structure + +The conversion creates a MONAI bundle with the following structure in the specified `MAP_root` directory: + +``` +MAP_root/ +└── models/ + ├── jsonpkls/ + │ ├── dataset.json # Dataset configuration + │ ├── plans.json # Model planning information + │ ├── postprocessing.pkl # Optional postprocessing configuration + ├── 3d_fullres/ # Model configuration (if present) + │ ├── nnunet_checkpoint.pth + │ └── fold_X/ # Each fold's model weights + │ └── best_model.pt + ├── 3d_lowres/ # Model configuration (if present) + └── 3d_cascade_fullres/ # Model configuration (if present) +``` + +This bundle structure is compatible with MONAI inference tools and the MONAI Deploy application ecosystem. + + +## Setting Up Environment +Instructions regarding installation of MONAI Deploy App SDK and details of the necessary system requirements can be found on the MONAI Deploy App SDK [GitHub Repository](https://github.com/Project-MONAI/monai-deploy-app-sdk) and [docs](https://docs.monai.io/projects/monai-deploy-app-sdk/en/latest/getting_started/installing_app_sdk.html). Instructions on how to create a virtual environment and install other dependencies can be found in the MONAI Deploy App SDK docs under the Creating a Segmentation App Consuming a MONAI Bundle [example](https://docs.monai.io/projects/monai-deploy-app-sdk/en/latest/getting_started/tutorials/monai_bundle_app.html). + +Per MONAI, MONAI Deploy App SDK is required to be run in a Linux environment, specifically Ubuntu 22.04 on X86-64, as this is the only X86 platform that the underlying Holoscan SDK has been tested to support as of now. This project uses Poetry for dependency management, which simplifies setting up the environment with all required dependencies. + +### System Requirements +- **Operating System:** Linux (Ubuntu 22.04 recommended) +- **Architecture:** x86_64 +- **GPU:** NVIDIA GPU (recommended for inference) +- **Python:** 3.10 or newer (project requires >=3.10,<3.13) + + + +## Executing Model Bundle Pythonically +Prior to MAP building, the exported model bundle can be executed pythonically via the command line. + +Within the main directory of this downloaded repository, create a `.env` file. MONAI recommends the following `.env` structure and naming conventions: + +```env +HOLOSCAN_INPUT_PATH=${PWD}/input +HOLOSCAN_MODEL_PATH=${PWD}/models +HOLOSCAN_OUTPUT_PATH=${PWD}/output +``` + +Load in the environment variables: + +``` +source .env +``` + +If already specified, remove the directory specified by the `HOLOSCAN_OUTPUT_PATH` environment variable: + +``` +rm -rf $HOLOSCAN_OUTPUT_PATH +``` + +Execute the model bundle pythonically via the command line; the directory specified by the `HOLOSCAN_INPUT_PATH` environment variable should be created and populated with a DICOM series for testing by the user. The model bundle file should be populated within the `/model` folder to match the recommended `HOLOSCAN_MODEL_PATH` value. `HOLOSCAN_INPUT_PATH`, `HOLOSCAN_OUTPUT_PATH`, and `HOLOSCAN_MODEL_PATH` default values can be amended by updating the `.env` file appropriately. + +``` +python my_app -i "$HOLOSCAN_INPUT_PATH" -o "$HOLOSCAN_OUTPUT_PATH" -m "$HOLOSCAN_MODEL_PATH" +``` + +## Building the MAP +It is recommended that the NVIDIA Clara Holoscan base image is pulled prior to building the MAP. If this base image is not pulled prior to MAP building, it will be done so automically during the build process, which will increase the build time from around 1/2 minutes to around 10/15 minutes. Ensure the base image matches the Holoscan SDK version being used in your environment (e.g. if you are using Holoscan SDK v3.2.0, replace `${holoscan-version}` with `v3.2.0`). + +``` +docker pull nvcr.io/nvidia/clara-holoscan/holoscan:${holoscan-version}-dgpu +``` + +Execute the following command to build the MAP Docker image based on the supported NVIDIA Clara Holoscan base image. During MAP building, a Docker container based on the `moby/buildkit` Docker image will be spun up; this container (Docker BuildKit builder `holoscan_app_builder`) facilitates the MAP build. + +``` +monai-deploy package my_app -m $HOLOSCAN_MODEL_PATH -c my_app/app.yaml -t ${tag_prefix}:${image_version} --platform x86_64 -l DEBUG +``` + +As of August 2024, a new error may appear during the MAP build related to the Dockerfile, where `monai-deploy-app-sdk` v0 (which does not exist) is attempted to be installed: + +```bash +Dockerfile:78 +-------------------- + 76 | + 77 | # Install MONAI Deploy from PyPI org + 78 | >>> RUN pip install monai-deploy-app-sdk==0 + 79 | + 80 | +-------------------- +``` + +If you encounter this error, you can specify the MONAI Deploy App SDK version via `--sdk-version` directly in the build command (`3.0.0`, for example). The base image for the MAP build can also be specified via `--base-image`: + +``` +monai-deploy package my_app -m $HOLOSCAN_MODEL_PATH -c my_app/app.yaml -t ${tag_prefix}:${image_version} --platform x86_64 --base-image ${base_image} --sdk-version ${version} -l DEBUG +``` + +If using Docker Desktop, the MAP should now appear in the "Images" tab as `${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version}`. You can also confirm MAP creation in the CLI by executing this command: + +``` +docker image ls | grep ${tag_prefix} +``` + +## Display and Extract MAP Contents +There are a few commands that can be executed in the command line to view MAP contents. + +To display some basic MAP manifests, use the `show` command. The following command will run and subsequently remove a MAP Docker container; the `show` command will display informaiton about the MAP-associated `app.json` and `pkg.json` files as command line outputs. + +``` +docker run --rm ${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version} show +``` + +MAP manifests and other contents can also be extracted into a specific host folder using the `extract` command. + +The host folder used to store the extracted MAP contents must be created by the host, not by Docker upon running the MAP as a container. This is most applicable when MAP contents are extracted more than once; the export folder must be deleted and recreated in this case. + +``` +rm -rf `pwd`/export && mkdir -p `pwd`/export +``` + +After creating the folder for export, executing the following command will run and subsequently remove a MAP Docker container. + +``` +docker run --rm -v `pwd`/export/:/var/run/holoscan/export/ ${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version} extract +``` + +The `extract` command will extract MAP contents to the `/export` folder, organized as follows: +- `app` folder, which contains of the all the files present in `my_app` +- `config` folder, which contains the MAP manifests (`app.json`, `pkg.json`, and `app.yaml`) +- `models` folder, which contains the model bundle used to created the MAP + +## Executing MAP Locally via the MONAI Application Runner (MAR) +The generated MAP can be tested locally using the MONAI Application Runner (MAR). + +First, clear the contents of the output directory: + +``` +rm -rf $HOLOSCAN_OUTPUT_PATH +``` + +Then, the MAP can be executed locally via the MAR command line utility; input and output directories must be specified: + +``` +monai-deploy run -i $HOLOSCAN_INPUT_PATH -o $HOLOSCAN_OUTPUT_PATH ${tag_prefix}-x64-workstation-dgpu-linux-amd64:${image_version} +``` diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/convert_nnunet_ckpts.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/convert_nnunet_ckpts.py new file mode 100644 index 00000000..e1691e89 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/convert_nnunet_ckpts.py @@ -0,0 +1,103 @@ +# Copyright (c) MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Convert nnUNet checkpoints to MONAI bundle format. +This script follows the logic in the conversion notebook but imports from local apps.nnunet_bundle. +""" + +import argparse +import os +import sys + +# Add the current directory to the path to find the local module +current_dir = os.path.dirname(os.path.abspath(__file__)) +if current_dir not in sys.path: + sys.path.insert(0, current_dir) + +# Try importing from local apps.nnunet_bundle instead of from MONAI +try: + from my_app.nnunet_bundle import convert_best_nnunet_to_monai_bundle +except ImportError: + # If local import fails, try to find the module in alternate locations + try: + from monai.apps.nnunet_bundle import convert_best_nnunet_to_monai_bundle + except ImportError: + print( + "Error: Could not import convert_best_nnunet_to_monai_bundle from my_app.nnunet_bundle or apps.nnunet_bundle" + ) + print("Please ensure that nnunet_bundle.py is properly installed in your project.") + sys.exit(1) + + +def parse_args(): + parser = argparse.ArgumentParser(description="Convert nnUNet checkpoints to MONAI bundle format.") + parser.add_argument( + "--dataset_name_or_id", type=str, required=True, help="The name or ID of the dataset to convert." + ) + parser.add_argument( + "--MAP_root", + type=str, + default=os.getcwd(), + help="The root directory where the Medical Application Package (MAP) will be created. Defaults to current directory.", + ) + + parser.add_argument( + "--nnUNet_results", + type=str, + required=False, + default=None, + help="Path to nnUNet results directory with trained models.", + ) + return parser.parse_args() + + +def main(): + args = parse_args() + + # Create the nnUNet config dictionary + nnunet_config = { + "dataset_name_or_id": args.dataset_name_or_id, + } + + # Create the MAP root directory + map_root = args.MAP_root + os.makedirs(map_root, exist_ok=True) + + # Set nnUNet environment variables if provided + if args.nnUNet_results: + os.environ["nnUNet_results"] = args.nnUNet_results + print(f"Set nnUNet_results to: {args.nnUNet_results}") + + # Check if required environment variables are set + required_env_vars = ["nnUNet_results"] + missing_vars = [var for var in required_env_vars if var not in os.environ] + + if missing_vars: + print(f"Error: The following required nnUNet environment variables are not set: {', '.join(missing_vars)}") + print("Please provide them as arguments or set them in your environment before running this script.") + sys.exit(1) + + print(f"Converting nnUNet checkpoints for dataset {nnunet_config['dataset_name_or_id']} to MONAI bundle format...") + print(f"MAP will be created at: {map_root}") + print(f" nnUNet_results: {os.environ.get('nnUNet_results')}") + + # Convert the nnUNet checkpoints to MONAI bundle format + try: + convert_best_nnunet_to_monai_bundle(nnunet_config, map_root) + print(f"Successfully converted nnUNet checkpoints to MONAI bundle at: {map_root}/models") + except Exception as e: + print(f"Error converting nnUNet checkpoints: {e}") + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/development_notes.md b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/development_notes.md new file mode 100644 index 00000000..f1e91b99 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/development_notes.md @@ -0,0 +1,62 @@ +# Development Notes + +## Implementation Notes for nnUNet MAP + + +* Initial Tests show volume and Dice agreement with Bundle, need to do more thorough testing. + +1. For each model configuration the output gets written to .npz file by nnunet inference functions. + +2. These file paths are then used by the EnsembleProbabilities Transform function to create the final output. + +3. If nnunet postprocessing is used, use the largest connected component transform in the MAP. There could be minor differences in the implementation, will do thorough analysis later. + +3. Need to better understand the use of "context" in compute and compute_impl as input arguments. + +4. Investigate keeping the probabilities in the memory, to help with speedup. + +5. Need to investigate the current traceability provisions in the operators implemented. + + +## Implementation Details + +### Testing Strategy + +Tests should be conducted to: +1. Compare MAP output with native nnUNet output +2. Measure performance (time, memory usage) +3. Validate with various input formats and sizes +4. Test error handling and edge cases + + +### nnUNet Integration + +The current implementation relies on the nnUNet's native inference approach which outputs intermediate .npz files for each model configuration. While this works, it introduces file I/O overhead which could potentially be optimized. + +### Ensemble Prediction Flow + +1. Multiple nnUNet models (3d_fullres, 3d_lowres, 3d_cascade_fullres) are loaded +2. Each model performs inference separately +3. Results are written to temporary .npz files +4. EnsembleProbabilitiesToSegmentation transform reads these files +5. Final segmentation is produced by combining results + +### Potential Optimizations + +- Keep probability maps in memory instead of writing to disk +- Parallelize model inference where applicable +- Streamline the ensemble computation process + +### Context Usage + +The `context` parameter in `compute` and `compute_impl` functions appears to be used for storing and retrieving models. Further investigation is needed to fully understand how this context is managed and whether it's being used optimally. + +### Traceability + +Current traceability in the operators may need improvement. Consider adding: + +- More detailed logging +- Performance metrics +- Input/output validation steps +- Error handling with informative messages + diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/__init__.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/__init__.py new file mode 100644 index 00000000..06014cc7 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/__init__.py @@ -0,0 +1,29 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# __init__.py is used to initialize a Python package +# ensures that the directory __init__.py resides in is included at the start of the sys.path +# this is useful when you want to import modules from this directory, even if it’s not the +# directory where your Python script is running. + +# give access to operating system and Python interpreter +import os +import sys + +# grab absolute path of directory containing __init__.py +_current_dir = os.path.abspath(os.path.dirname(__file__)) + +# if sys.path is not the same as the directory containing the __init__.py file +if sys.path and os.path.abspath(sys.path[0]) != _current_dir: + # insert directory containing __init__.py file at the beginning of sys.path + sys.path.insert(0, _current_dir) +# delete variable +del _current_dir diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/__main__.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/__main__.py new file mode 100644 index 00000000..afd32bef --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/__main__.py @@ -0,0 +1,26 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# __main__.py is needed for MONAI Application Packager to detect the main app code (app.py) when +# app.py is executed in the application folder path +# e.g., python my_app + +import logging + +# import UTEAirwayNNUnetApp class from app.py +from app import UTEAirwayNNUnetApp + +# if __main__.py is being run directly +if __name__ == "__main__": + logging.info(f"Begin {__name__}") + # create and run an instance of UTEAirwayNNUnetApp + UTEAirwayNNUnetApp().run() + logging.info(f"End {__name__}") diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/app.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/app.py new file mode 100644 index 00000000..0161c647 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/app.py @@ -0,0 +1,257 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from pathlib import Path + +# custom DICOMSCWriterOperator (Secondary Capture) +from dicom_sc_writer_operator import DICOMSCWriterOperator + +# custom DICOMSeriesSelectorOperator +from dicom_series_selector_operator import DICOMSeriesSelectorOperator + +# custom inference operator +from nnunet_seg_operator import NNUnetSegOperator + +# required for setting SegmentDescription attributes +# direct import as this is not part of App SDK package +from pydicom.sr.codedict import codes + +from monai.deploy.conditions import CountCondition +from monai.deploy.core import Application +from monai.deploy.operators.dicom_data_loader_operator import DICOMDataLoaderOperator +from monai.deploy.operators.dicom_seg_writer_operator import DICOMSegmentationWriterOperator, SegmentDescription +from monai.deploy.operators.dicom_series_to_volume_operator import DICOMSeriesToVolumeOperator +from monai.deploy.operators.dicom_text_sr_writer_operator import DICOMTextSRWriterOperator, EquipmentInfo, ModelInfo + + +# inherit new Application class instance, AIAbdomenSegApp, from MONAI Application base class +# base class provides support for chaining up operators and executing application +class UTEAirwayNNUnetApp(Application): + """Demonstrates inference with nnU-Net ensemble models for airway segmentation. + + This application loads a set of DICOM instances, selects the appropriate series, converts the series to + 3D volume image, performs inference with the NNUnetSegOperator, including pre-processing + and post-processing, saves a DICOM SEG (airway contour), a DICOM Secondary Capture (airway contour overlay), + and a DICOM SR (airway volume). + + Pertinent MONAI Bundle: + This MAP is designed to work with a MONAI bundle compatible with nnU-Net. + """ + + def __init__(self, *args, **kwargs): + """Creates an application instance.""" + self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__)) + super().__init__(*args, **kwargs) + + def run(self, *args, **kwargs): + # this method calls the base class to run; can be omitted if simply calling through + self._logger.info(f"Begin {self.run.__name__}") + super().run(*args, **kwargs) + self._logger.info(f"End {self.run.__name__}") + + # use compose method to instantiate operators and connect them to form a Directed Acyclic Graph (DAG) + def compose(self): + """Creates the app specific operators and chain them up in the processing DAG.""" + + logging.info(f"Begin {self.compose.__name__}") + + # use Commandline options over environment variables to init context + app_context = Application.init_app_context(self.argv) + app_input_path = Path(app_context.input_path) + app_output_path = Path(app_context.output_path) + model_path = Path(app_context.model_path) + + # Temporary bug fix for MAP execution where model path copy is messed up - need fix to app-sdk package function + # Check if the model_path has a subfolder named 'models' and set model_path to that subfolder if it exists + models_subfolder = model_path / "models" + if models_subfolder.exists() and models_subfolder.is_dir(): + self._logger.info(f"Found 'models' subfolder in {model_path}. Setting model_path to {models_subfolder}") + model_path = models_subfolder + + # create the custom operator(s) as well as SDK built-in operator(s) + # DICOM Data Loader op + study_loader_op = DICOMDataLoaderOperator( + self, CountCondition(self, 1), input_folder=app_input_path, name="study_loader_op" + ) + + # custom DICOM Series Selector op + # all_matched and sort_by_sop_instance_count = True; want all series that meet the selection criteria + # to be matched, and SOP sorting + series_selector_op = DICOMSeriesSelectorOperator( + self, rules=Sample_Rules_Text, all_matched=True, sort_by_sop_instance_count=True, name="series_selector_op" + ) + + # DICOM Series to Volume op + series_to_vol_op = DICOMSeriesToVolumeOperator(self, name="series_to_vol_op") + + # custom inference op + # output_labels specifies which of the organ segmentations are desired in the DICOM SEG, DICOM SC, and DICOM SR outputs + # 1 = airway + output_labels = [1] + nnunet_seg_op = NNUnetSegOperator( + self, + app_context=app_context, + model_path=model_path, + output_folder=app_output_path, + output_labels=output_labels, + name="nnunet_seg_op", + ) + + # create DICOM Seg writer providing the required segment description for each segment with + # the actual algorithm and the pertinent organ/tissue; the segment_label, algorithm_name, + # and algorithm_version are of DICOM VR LO type, limited to 64 chars + # https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html + + # general algorithm information + _algorithm_name = "UTE_nnunet_airway" + _algorithm_family = codes.DCM.ArtificialIntelligence + _algorithm_version = "1.0.0" + + segment_descriptions = [ + SegmentDescription( + segment_label="Airway", + segmented_property_category=codes.SCT.BodyStructure, + segmented_property_type=codes.SCT.TracheaAndBronchus, + algorithm_name=_algorithm_name, + algorithm_family=_algorithm_family, + algorithm_version=_algorithm_version, + ), + ] + + # model info is algorithm information + my_model_info = ModelInfo( + creator="UTE", # institution name + name=_algorithm_name, # algorithm name + version=_algorithm_version, # algorithm version + uid="1.0.0", # MAP version + ) + + # equipment info is MONAI Deploy App SDK information + my_equipment = EquipmentInfo( + manufacturer="The MONAI Consortium", + manufacturer_model="MONAI Deploy App SDK", + software_version_number="3.0.0", # MONAI Deploy App SDK version + ) + + # custom tags - add AlgorithmName for monitoring purposes + custom_tags_seg = { + "SeriesDescription": "AI Generated DICOM SEG; Not for Clinical Use.", + "AlgorithmName": f"{my_model_info.name}:{my_model_info.version}:{my_model_info.uid}", + } + custom_tags_sr = { + "SeriesDescription": "AI Generated DICOM SR; Not for Clinical Use.", + "AlgorithmName": f"{my_model_info.name}:{my_model_info.version}:{my_model_info.uid}", + } + custom_tags_sc = { + "SeriesDescription": "AI Generated DICOM Secondary Capture; Not for Clinical Use.", + "AlgorithmName": f"{my_model_info.name}:{my_model_info.version}:{my_model_info.uid}", + } + + # DICOM SEG Writer op writes content from segment_descriptions to output DICOM images as DICOM tags + dicom_seg_writer = DICOMSegmentationWriterOperator( + self, + segment_descriptions=segment_descriptions, + model_info=my_model_info, + custom_tags=custom_tags_seg, + # store DICOM SEG in SEG subdirectory; necessary for routing in CCHMC MDE workflow definition + output_folder=app_output_path / "SEG", + # omit_empty_frames is a default parameteter (type bool) of DICOMSegmentationWriterOperator + # dictates whether or not to omit frames that contain no segmented pixels from the output segmentation + # default value is True; changed to False to ensure input and output DICOM series #'s match + omit_empty_frames=False, + name="dicom_seg_writer", + ) + + # DICOM SR Writer op + dicom_sr_writer = DICOMTextSRWriterOperator( + self, + # copy_tags is a default parameteter (type bool) of DICOMTextSRWriterOperator; default value is True + # dictates whether or not to copy DICOM attributes from the selected DICOM series + # changed to True to copy DICOM attributes so DICOM SR has same Study UID + copy_tags=True, + model_info=my_model_info, + equipment_info=my_equipment, + custom_tags=custom_tags_sr, + # store DICOM SR in SR subdirectory; necessary for routing in CCHMC MDE workflow definition + output_folder=app_output_path / "SR", + ) + + # custom DICOM SC Writer op + dicom_sc_writer = DICOMSCWriterOperator( + self, + model_info=my_model_info, + equipment_info=my_equipment, + custom_tags=custom_tags_sc, + # store DICOM SC in SC subdirectory; necessary for routing in CCHMC MDE workflow definition + output_folder=app_output_path / "SC", + ) + + # create the processing pipeline, by specifying the source and destination operators, and + # ensuring the output from the former matches the input of the latter, in both name and type + # instantiate and connect operators using self.add_flow(); specify current operator, next operator, and tuple to match I/O + self.add_flow(study_loader_op, series_selector_op, {("dicom_study_list", "dicom_study_list")}) + self.add_flow( + series_selector_op, series_to_vol_op, {("study_selected_series_list", "study_selected_series_list")} + ) + self.add_flow(series_to_vol_op, nnunet_seg_op, {("image", "image")}) + + # note below the dicom_seg_writer, dicom_sr_writer, and dicom_sc_writer each require two inputs, + # each coming from a source operator + + # DICOM SEG + self.add_flow( + series_selector_op, dicom_seg_writer, {("study_selected_series_list", "study_selected_series_list")} + ) + self.add_flow(nnunet_seg_op, dicom_seg_writer, {("seg_image", "seg_image")}) + + # DICOM SR + self.add_flow( + series_selector_op, dicom_sr_writer, {("study_selected_series_list", "study_selected_series_list")} + ) + self.add_flow(nnunet_seg_op, dicom_sr_writer, {("result_text", "text")}) + + # DICOM SC + self.add_flow( + series_selector_op, dicom_sc_writer, {("study_selected_series_list", "study_selected_series_list")} + ) + self.add_flow(nnunet_seg_op, dicom_sc_writer, {("dicom_sc_dir", "dicom_sc_dir")}) + + logging.info(f"End {self.compose.__name__}") + + +# series selection rule in JSON, which selects for Axial T2 MR series: +# StudyDescription (Type 3): matches any value +# Modality (Type 1): matches "MR" value (case-insensitive); filters out non-MR modalities +# ImageOrientationPatient (Type 1): matches Axial orientations; filters out Sagittal and Coronal orientations +# MRAcquisitionType (Type 2): matches "2D" value (case-insensitive); filters out 3D acquisitions +# RepetitionTime (Type 2C): matches values greater than 1200; filters for T2 acquisitions +# EchoTime (Type 2): matches values bewtween 75 and 100 (inclusive); filters out SSH series +# EchoTrainLength (Type 2): matches values less than 50; filters out SSH series +# FlipAngle (Type 3): matches values greater than 75; filters for T2 acquisitions +# all valid series will be selected; downstream operators only perform inference and write outputs for 1st selected series +# please see more detail in DICOMSeriesSelectorOperator + +Sample_Rules_Text = """ +""" + +# if executing application code using python interpreter: +if __name__ == "__main__": + # creates the app and test it standalone; when running is this mode, please note the following: + # -m , for model file path + # -i , for input DICOM MR series folder + # -o , for the output folder, default $PWD/output + # e.g. + # monai-deploy exec app.py -i input -m model/ls_swinunetr_FT.pt + # + logging.info(f"Begin {__name__}") + UTEAirwayNNUnetApp().run() + logging.info(f"End {__name__}") diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/app.yaml b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/app.yaml new file mode 100644 index 00000000..7833d614 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/app.yaml @@ -0,0 +1,34 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--- + +# app.yaml is a configuration file that specifies MAP settings +# used by MONAI App SDK to understand how to run our app in a MAP and what resources it needs + +# specifies high-level information about our app +application: + title: MONAI Deploy App Package - CCHMC Pediatric Airway Segmentation using nnUNet + description: This application segments the airway from a MRI scan using a nnUNet model trained + version: 0.0.1 + inputFormats: ["file"] + outputFormats: ["file"] + +# specifies the resources our app needs to run +# per MONAI docs (https://docs.monai.io/projects/monai-deploy-app-sdk/en/latest/developing_with_sdk/executing_packaged_app_locally.html) +# MAR does not validate all of the resource requirements embedded in the MAP to ensure they are met in host system +# e.g, MAR will throw an error if gpu requirement is not met on host system; however, gpuMemory parameter doesn't appear to be validated +resources: + cpu: 4 + gpu: 1 + memory: 4Gi + # during MAP execution, for an input DICOM Series of 72 instances, GPU usage peaks at just under 8100 MiB ~= 8.5 GB ~= 7.9 Gi + gpuMemory: 8Gi + \ No newline at end of file diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/dicom_sc_writer_operator.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/dicom_sc_writer_operator.py new file mode 100644 index 00000000..9479485e --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/dicom_sc_writer_operator.py @@ -0,0 +1,253 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import os +from pathlib import Path +from typing import Dict, Optional, Union + +import pydicom + +from monai.deploy.core import Fragment, Operator, OperatorSpec +from monai.deploy.core.domain.dicom_series import DICOMSeries +from monai.deploy.core.domain.dicom_series_selection import StudySelectedSeries +from monai.deploy.operators.dicom_utils import EquipmentInfo, ModelInfo, write_common_modules +from monai.deploy.utils.importutil import optional_import +from monai.deploy.utils.version import get_sdk_semver + +dcmread, _ = optional_import("pydicom", name="dcmread") +dcmwrite, _ = optional_import("pydicom.filewriter", name="dcmwrite") +generate_uid, _ = optional_import("pydicom.uid", name="generate_uid") +ImplicitVRLittleEndian, _ = optional_import("pydicom.uid", name="ImplicitVRLittleEndian") +Dataset, _ = optional_import("pydicom.dataset", name="Dataset") +FileDataset, _ = optional_import("pydicom.dataset", name="FileDataset") +Sequence, _ = optional_import("pydicom.sequence", name="Sequence") + + +class DICOMSCWriterOperator(Operator): + """Class to write a new DICOM Secondary Capture (DICOM SC) instance with source DICOM Series metadata included. + + Named inputs: + dicom_sc_dir: file path of temporary DICOM SC (w/o source DICOM Series metadata). + study_selected_series_list: DICOM Series for copying metadata from. + + Named output: + None. + + File output: + New, updated DICOM SC file (with source DICOM Series metadata) in the provided output folder. + """ + + # file extension for the generated DICOM Part 10 file + DCM_EXTENSION = ".dcm" + # the default output folder for saving the generated DICOM instance file + # DEFAULT_OUTPUT_FOLDER = Path(os.path.join(os.path.dirname(__file__))) / "output" + DEFAULT_OUTPUT_FOLDER = Path.cwd() / "output" + + def __init__( + self, + fragment: Fragment, + *args, + output_folder: Union[str, Path], + model_info: ModelInfo, + equipment_info: Optional[EquipmentInfo] = None, + custom_tags: Optional[Dict[str, str]] = None, + **kwargs, + ): + """Class to write a new DICOM Secondary Capture (DICOM SC) instance with source DICOM Series metadata. + + Args: + output_folder (str or Path): The folder for saving the generated DICOM SC instance file. + model_info (ModelInfo): Object encapsulating model creator, name, version and UID. + equipment_info (EquipmentInfo, optional): Object encapsulating info for DICOM Equipment Module. + Defaults to None. + custom_tags (Dict[str, str], optional): Dictionary for setting custom DICOM tags using Keywords and str values only. + Defaults to None. + + Raises: + ValueError: If result cannot be found either in memory or from file. + """ + + self._logger = logging.getLogger(f"{__name__}.{type(self).__name__}") + + # need to init the output folder until the execution context supports dynamic FS path + # not trying to create the folder to avoid exception on init + self.output_folder = Path(output_folder) if output_folder else DICOMSCWriterOperator.DEFAULT_OUTPUT_FOLDER + self.input_name_sc_dir = "dicom_sc_dir" + self.input_name_study_series = "study_selected_series_list" + + # for copying DICOM attributes from a provided DICOMSeries + # required input for write_common_modules; will always be True for this implementation + self.copy_tags = True + + self.model_info = model_info if model_info else ModelInfo() + self.equipment_info = equipment_info if equipment_info else EquipmentInfo() + self.custom_tags = custom_tags + + # set own Modality and SOP Class UID + # Standard SOP Classes: https://dicom.nema.org/dicom/2013/output/chtml/part04/sect_B.5.html + # Modality, e.g., + # "OT" for PDF + # "SR" for Structured Report. + # Media Storage SOP Class UID, e.g., + # "1.2.840.10008.5.1.4.1.1.88.11" for Basic Text SR Storage + # "1.2.840.10008.5.1.4.1.1.104.1" for Encapsulated PDF Storage, + # "1.2.840.10008.5.1.4.1.1.88.34" for Comprehensive 3D SR IOD + # "1.2.840.10008.5.1.4.1.1.66.4" for Segmentation Storage + self.modality_type = "OT" # OT Modality for Secondary Capture + self.sop_class_uid = ( + "1.2.840.10008.5.1.4.1.1.7.4" # SOP Class UID for Multi-frame True Color Secondary Capture Image Storage + ) + # custom OverlayImageLabeld post-processing transform creates an RBG overlay + + # equipment version may be different from contributing equipment version + try: + self.software_version_number = get_sdk_semver() # SDK Version + except Exception: + self.software_version_number = "" + self.operators_name = f"AI Algorithm {self.model_info.name}" + + super().__init__(fragment, *args, **kwargs) + + def setup(self, spec: OperatorSpec): + """Set up the named input(s), and output(s) if applicable. + + This operator does not have an output for the next operator, rather file output only. + + Args: + spec (OperatorSpec): The Operator specification for inputs and outputs etc. + """ + + spec.input(self.input_name_sc_dir) + spec.input(self.input_name_study_series) + + def compute(self, op_input, op_output, context): + """Performs computation for this operator and handles I/O. + + For now, only a single result content is supported, which could be in memory or an accessible file. + The DICOM Series used during inference is required (and copy_tags is hardcoded to True). + + When there are multiple selected series in the input, the first series' containing study will + be used for retrieving DICOM Study module attributes, e.g. StudyInstanceUID. + + Raises: + NotADirectoryError: When temporary DICOM SC path is not a directory. + FileNotFoundError: When result object not in the input, and result file not found either. + ValueError: Content object and file path not in the inputs, or no DICOM series provided. + IOError: If the input content is blank. + """ + + # receive the temporary DICOM SC file path and study selected series list + dicom_sc_dir = Path(op_input.receive(self.input_name_sc_dir)) + if not dicom_sc_dir: + raise IOError("Temporary DICOM SC path is read but blank.") + if not dicom_sc_dir.is_dir(): + raise NotADirectoryError(f"Provided temporary DICOM SC path is not a directory: {dicom_sc_dir}") + self._logger.info(f"Received temporary DICOM SC path: {dicom_sc_dir}") + + study_selected_series_list = op_input.receive(self.input_name_study_series) + if not study_selected_series_list or len(study_selected_series_list) < 1: + raise ValueError("Missing input, list of 'StudySelectedSeries'.") + + # retrieve the DICOM Series used during inference in order to grab appropriate study/series level tags + # this will be the 1st Series in study_selected_series_list + dicom_series = None + for study_selected_series in study_selected_series_list: + if not isinstance(study_selected_series, StudySelectedSeries): + raise ValueError(f"Element in input is not expected type, {StudySelectedSeries}.") + selected_series = study_selected_series.selected_series[0] + dicom_series = selected_series.series + break + + # log basic DICOM metadata for the retrieved DICOM Series + self._logger.debug(f"Dicom Series: {dicom_series}") + + # the output folder should come from the execution context when it is supported + self.output_folder.mkdir(parents=True, exist_ok=True) + + # write the new DICOM SC instance + self.write(dicom_sc_dir, dicom_series, self.output_folder) + + def write(self, dicom_sc_dir, dicom_series: DICOMSeries, output_dir: Path): + """Writes a new, updated DICOM SC instance and deletes the temporary DICOM SC instance. + The new, updated DICOM SC instance is the temporary DICOM SC instance with source + DICOM Series metadata copied. + + Args: + dicom_sc_dir: temporary DICOM SC file path. + dicom_series (DICOMSeries): DICOMSeries object encapsulating the original series. + + Returns: + None + + File output: + New, updated DICOM SC file (with source DICOM Series metadata) in the provided output folder. + """ + + if not isinstance(output_dir, Path): + raise ValueError("output_dir is not a valid Path.") + + output_dir.mkdir(parents=True, exist_ok=True) # just in case + + # find the temporary DICOM SC file in the directory; there should only be one .dcm file present + dicom_files = list(dicom_sc_dir.glob("*.dcm")) + dicom_sc_file = dicom_files[0] + + # load the temporary DICOM SC file using pydicom + dicom_sc_dataset = pydicom.dcmread(dicom_sc_file) + self._logger.info(f"Loaded temporary DICOM SC file: {dicom_sc_file}") + + # use write_common_modules to copy metadata from dicom_series + # this will copy metadata and return an updated Dataset + ds = write_common_modules( + dicom_series, + self.copy_tags, # always True for this implementation + self.modality_type, + self.sop_class_uid, + self.model_info, + self.equipment_info, + ) + + # Secondary Capture specific tags + ds.ImageType = ["DERIVED", "SECONDARY"] + + # for now, only allow str Keywords and str value + if self.custom_tags: + for k, v in self.custom_tags.items(): + if isinstance(k, str) and isinstance(v, str): + try: + ds.update({k: v}) + except Exception as ex: + # best effort for now + logging.warning(f"Tag {k} was not written, due to {ex}") + + # merge the copied metadata into the loaded temporary DICOM SC file (dicom_sc_dataset) + for tag, value in ds.items(): + dicom_sc_dataset[tag] = value + + # save the updated DICOM SC file to the output folder + # instance file name is the same as the new SOP instance UID + output_file_path = self.output_folder.joinpath( + f"{dicom_sc_dataset.SOPInstanceUID}{DICOMSCWriterOperator.DCM_EXTENSION}" + ) + dicom_sc_dataset.save_as(output_file_path) + self._logger.info(f"Saved updated DICOM SC file at: {output_file_path}") + + # remove the temporary DICOM SC file + os.remove(dicom_sc_file) + self._logger.info(f"Removed temporary DICOM SC file: {dicom_sc_file}") + + # check if the temp directory is empty, then delete it + if not any(dicom_sc_dir.iterdir()): + os.rmdir(dicom_sc_dir) + self._logger.info(f"Removed temporary directory: {dicom_sc_dir}") + else: + self._logger.warning(f"Temporary directory {dicom_sc_dir} is not empty, skipping removal.") diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/dicom_series_selector_operator.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/dicom_series_selector_operator.py new file mode 100644 index 00000000..9249b4d9 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/dicom_series_selector_operator.py @@ -0,0 +1,629 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import numbers +import re +from json import loads as json_loads +from typing import List + +import numpy as np + +from monai.deploy.core import ConditionType, Fragment, Operator, OperatorSpec +from monai.deploy.core.domain.dicom_series import DICOMSeries +from monai.deploy.core.domain.dicom_series_selection import SelectedSeries, StudySelectedSeries +from monai.deploy.core.domain.dicom_study import DICOMStudy + + +class DICOMSeriesSelectorOperator(Operator): + """This operator selects a list of DICOM Series in a DICOM Study for a given set of selection rules. + + Named input: + dicom_study_list: A list of DICOMStudy objects. + + Named output: + study_selected_series_list: A list of StudySelectedSeries objects. Downstream receiver optional. + + This class can be considered a base class, and a derived class can override the 'filter' function with + custom logic. + + In its default implementation, this class + 1. selects a series or all matched series within the scope of a study in a list of studies + 2. uses rules defined in JSON string, see below for details + 3. supports DICOM Study and Series module attribute matching + 4. supports multiple named selections, in the scope of each DICOM study + 5. outputs a list of StudySelectedSeries objects, as well as a flat list of SelectedSeries (to be deprecated) + + The selection rules are defined in JSON, + 1. attribute "selections" value is a list of selections + 2. each selection has a "name", and its "conditions" value is a list of matching criteria + 3. each condition uses the implicit equal operator; in addition, the following are supported: + - regex, relational, and range matching for float and int types + - regex matching for str type + - inclusion and exclusion matching for set type + - image orientation check for the ImageOrientationPatient tag + 4. DICOM attribute keywords are used, and only for those defined as DICOMStudy and DICOMSeries properties + + An example selection rules: + { + "selections": [ + { + "name": "CT Series 1", + "conditions": { + "StudyDescription": "(?i)^Spleen", + "Modality": "(?i)CT", + "SeriesDescription": "(?i)^No series description|(.*?)" + } + }, + { + "name": "CT Series 2", + "conditions": { + "Modality": "CT", + "BodyPartExamined": "Abdomen", + "SeriesDescription" : "Not to be matched. For illustration only." + } + }, + { + "name": "CT Series 3", + "conditions": { + "StudyDescription": "(.*?)", + "Modality": "(?i)CT", + "ImageType": ["PRIMARY", "ORIGINAL", "AXIAL"], + "SliceThickness": [3, 5] + } + }, + { + "name": "CT Series 4", + "conditions": { + "StudyDescription": "(.*?)", + "Modality": "(?i)CT", + "ImageOrientationPatient": "Axial", + "SliceThickness": [2, ">"] + } + }, + { + "name": "CT Series 5", + "conditions": { + "StudyDescription": "(.*?)", + "Modality": "(?i)CT", + "ImageType": ["PRIMARY", "!SECONDARY"] + } + } + ] + } + """ + + def __init__( + self, + fragment: Fragment, + *args, + rules: str = "", + all_matched: bool = False, + sort_by_sop_instance_count: bool = False, + **kwargs, + ) -> None: + """Instantiate an instance. + + Args: + fragment (Fragment): An instance of the Application class which is derived from Fragment. + rules (Text): Selection rules in JSON string. + all_matched (bool): Gets all matched series in a study. Defaults to False for first match only. + sort_by_sop_instance_count (bool): If all_matched = True and multiple series are matched, sorts the matched series in + descending SOP instance count (i.e. the first Series in the returned List[StudySelectedSeries] will have the highest # + of DICOM images); Defaults to False for no sorting. + """ + + # Delay loading the rules as JSON string till compute time. + self._rules_json_str = rules if rules and rules.strip() else None + self._all_matched = all_matched # all_matched + self._sort_by_sop_instance_count = sort_by_sop_instance_count # sort_by_sop_instance_count + self.input_name_study_list = "dicom_study_list" + self.output_name_selected_series = "study_selected_series_list" + + super().__init__(fragment, *args, **kwargs) + + def setup(self, spec: OperatorSpec): + spec.input(self.input_name_study_list) + spec.output(self.output_name_selected_series).condition(ConditionType.NONE) # Receiver optional + + # Can use the config file to alter the selection rules per app run + # spec.param("selection_rules") + + def compute(self, op_input, op_output, context): + """Performs computation for this operator.""" + + dicom_study_list = op_input.receive(self.input_name_study_list) + selection_rules = self._load_rules() if self._rules_json_str else None + study_selected_series = self.filter( + selection_rules, dicom_study_list, self._all_matched, self._sort_by_sop_instance_count + ) + + # Log Series Description and Series Instance UID of the first selected DICOM Series (i.e. the one to be used for inference) + if study_selected_series and len(study_selected_series) > 0: + inference_study = study_selected_series[0] + if inference_study.selected_series and len(inference_study.selected_series) > 0: + inference_series = inference_study.selected_series[0].series + logging.info("Series Selection finalized") + logging.info( + f"Series Description of selected DICOM Series for inference: {inference_series.SeriesDescription}" + ) + logging.info( + f"Series Instance UID of selected DICOM Series for inference: {inference_series.SeriesInstanceUID}" + ) + + op_output.emit(study_selected_series, self.output_name_selected_series) + + def filter( + self, selection_rules, dicom_study_list, all_matched: bool = False, sort_by_sop_instance_count: bool = False + ) -> List[StudySelectedSeries]: + """Selects the series with the given matching rules. + + If rules object is None, all series will be returned with series instance UID as the selection name. + + Supported matching logic: + Float + Int: exact matching, relational matching, range matching, and regex matching + String: matches case insensitive, if fails then tries RegEx search + String array (set): inclusive and exclusive (via !) matching as subsets, case insensitive + ImageOrientationPatient tag: image orientation (Axial, Coronal, Sagittal) matching + + Args: + selection_rules (object): JSON object containing the matching rules. + dicom_study_list (list): A list of DICOMStudy objects. + all_matched (bool): Gets all matched series in a study. Defaults to False for first match only. + sort_by_sop_instance_count (bool): If all_matched = True and multiple series are matched, sorts the matched series in + descending SOP instance count (i.e. the first Series in the returned List[StudySelectedSeries] will have the highest # + of DICOM images); Defaults to False for no sorting. + + Returns: + list: A list of objects of type StudySelectedSeries. + + Raises: + ValueError: If the selection_rules object does not contain "selections" attribute. + """ + + if not dicom_study_list or len(dicom_study_list) < 1: + return [] + + if not selection_rules: + # Return all series if no selection rules are supplied + logging.warn("No selection rules given; select all series.") + return self._select_all_series(dicom_study_list) + + selections = selection_rules.get("selections", None) # TODO type is not json now. + # If missing selections in the rules then it is an error. + if not selections: + raise ValueError('Expected "selections" not found in the rules.') + + study_selected_series_list = [] # List of StudySelectedSeries objects + + for study in dicom_study_list: + study_selected_series = StudySelectedSeries(study) + for selection in selections: + # Get the selection name. Blank name will be handled by the SelectedSeries + selection_name = selection.get("name", "").strip() + logging.info(f"Finding series for Selection named: {selection_name}") + + # Skip if no selection conditions are provided. + conditions = selection.get("conditions", None) + if not conditions: + continue + + # Select only the first series that matches the conditions, list of one + series_list = self._select_series(conditions, study, all_matched, sort_by_sop_instance_count) + if series_list and len(series_list) > 0: + for series in series_list: + selected_series = SelectedSeries(selection_name, series, None) # No Image obj yet. + study_selected_series.add_selected_series(selected_series) + + if len(study_selected_series.selected_series) > 0: + study_selected_series_list.append(study_selected_series) + + return study_selected_series_list + + def _load_rules(self): + return json_loads(self._rules_json_str) if self._rules_json_str else None + + def _select_all_series(self, dicom_study_list: List[DICOMStudy]) -> List[StudySelectedSeries]: + """Select all series in studies + + Returns: + list: list of StudySelectedSeries objects + """ + + study_selected_series_list = [] + for study in dicom_study_list: + logging.info(f"Working on study, instance UID: {study.StudyInstanceUID}") + study_selected_series = StudySelectedSeries(study) + for series in study.get_all_series(): + logging.info(f"Working on series, instance UID: {str(series.SeriesInstanceUID)}") + selected_series = SelectedSeries("", series, None) # No selection name or Image obj. + study_selected_series.add_selected_series(selected_series) + study_selected_series_list.append(study_selected_series) + return study_selected_series_list + + def _select_series( + self, attributes: dict, study: DICOMStudy, all_matched=False, sort_by_sop_instance_count=False + ) -> List[DICOMSeries]: + """Finds series whose attributes match the given attributes. + + Args: + attributes (dict): Dictionary of attributes for matching + all_matched (bool): Gets all matched series in a study. Defaults to False for first match only. + sort_by_sop_instance_count (bool): If all_matched = True and multiple series are matched, sorts the matched series in + descending SOP instance count (i.e. the first Series in the returned List[StudySelectedSeries] will have the highest # + of DICOM images); Defaults to False for no sorting. + + Returns: + List of DICOMSeries. At most one element if all_matched is False. + + Raises: + NotImplementedError: If the value_to_match type is not supported for matching or unsupported PatientPosition value. + """ + assert isinstance(attributes, dict), '"attributes" must be a dict.' + + logging.info(f"Searching study, : {study.StudyInstanceUID}\n # of series: {len(study.get_all_series())}") + study_attr = self._get_instance_properties(study) + + found_series = [] + for series in study.get_all_series(): + logging.info(f"Working on series, instance UID: {series.SeriesInstanceUID}") + + # Combine Study and current Series properties for matching + series_attr = self._get_instance_properties(series) + series_attr.update(study_attr) + + matched = True + # Simple matching on attribute value + for key, value_to_match in attributes.items(): + logging.info(f" On attribute: {key!r} to match value: {value_to_match!r}") + # Ignore None + if not value_to_match: + continue + # Try getting the attribute value from Study and current Series prop dict + attr_value = series_attr.get(key, None) + logging.info(f" Series attribute {key} value: {attr_value}") + + # If not found, try the best at the native instance level for string VR + # This is mainly for attributes like ImageType + if not attr_value: + try: + # Can use some enhancements, especially multi-value where VM > 1 + elem = series.get_sop_instances()[0].get_native_sop_instance()[key] + if elem.VM > 1: + attr_value = [elem.repval] # repval: str representation of the element’s value + else: + attr_value = elem.value # element's value + + logging.info(f" Instance level attribute {key} value: {attr_value}") + series_attr.update({key: attr_value}) + except Exception: + logging.info(f" Attribute {key} not at instance level either") + + if not attr_value: + logging.info(f" Missing attribute: {key!r}") + matched = False + # Image orientation check + elif key == "ImageOrientationPatient": + patient_position = series_attr.get("PatientPosition") + if patient_position is None: + raise NotImplementedError( + "PatientPosition tag absent; value required for image orientation calculation" + ) + if patient_position not in ("HFP", "HFS", "HFDL", "HFDR", "FFP", "FFS", "FFDL", "FFDR"): + raise NotImplementedError(f"No support for PatientPosition value {patient_position}") + matched = self._match_image_orientation(value_to_match, attr_value) + elif isinstance(attr_value, (float, int)): + matched = self._match_numeric_condition(value_to_match, attr_value) + elif isinstance(attr_value, str): + matched = attr_value.casefold() == (value_to_match.casefold()) + if not matched: + # For str, also try RegEx search to check for a match anywhere in the string + # unless the user constrains it in the expression. + if re.search(value_to_match, attr_value, re.IGNORECASE): + matched = True + elif isinstance(attr_value, list): + # Assume multi value string attributes + meta_data_list = str(attr_value).lower() + if isinstance(value_to_match, list): + value_set = {str(element).lower() for element in value_to_match} + # split inclusion and exclusion matches using ! indicator + include_terms = {v for v in value_set if not v.startswith("!")} + exclude_terms = {v[1:] for v in value_set if v.startswith("!")} + matched = all(term in meta_data_list for term in include_terms) and all( + term not in meta_data_list for term in exclude_terms + ) + elif isinstance(value_to_match, (str, numbers.Number)): + v = str(value_to_match).lower() + # ! indicates exclusion match + if v.startswith("!"): + matched = v[1:] not in meta_data_list + else: + matched = v in meta_data_list + else: + raise NotImplementedError( + f"No support for matching condition {value_to_match} (type: {type(value_to_match)})" + ) + + if not matched: + logging.info("This series does not match the selection conditions") + break + + if matched: + logging.info(f"Selected Series, UID: {series.SeriesInstanceUID}") + found_series.append(series) + + if not all_matched: + return found_series + + # If sorting indicated and multiple series found, sort series in descending SOP instance count + if sort_by_sop_instance_count and len(found_series) > 1: + logging.info( + "Multiple series matched the selection criteria; choosing series with the highest number of DICOM images." + ) + found_series.sort(key=lambda x: len(x.get_sop_instances()), reverse=True) + + return found_series + + def _match_numeric_condition(self, value_to_match, attr_value): + """ + Helper method to match numeric conditions, supporting relational, inclusive range, regex, and exact match checks. + + Supported formats: + - [val, ">"]: match if attr_value > val + - [val, ">="]: match if attr_value >= val + - [val, "<"]: match if attr_value < val + - [val, "<="]: match if attr_value <= val + - [val, "!="]: match if attr_value != val + - [min_val, max_val]: inclusive range check + - "regex": regular expression match + - number: exact match + + Args: + value_to_match (Union[list, str, int, float]): The condition to match against. + attr_value (Union[int, float]): The attribute value from the series. + + Returns: + bool: True if the attribute value matches the condition, else False. + + Raises: + NotImplementedError: If the value_to_match condition is not supported for numeric matching. + """ + + if isinstance(value_to_match, list): + # Relational operator check: >, >=, <, <=, != + if len(value_to_match) == 2 and isinstance(value_to_match[1], str): + val = float(value_to_match[0]) + op = value_to_match[1] + if op == ">": + return attr_value > val + elif op == ">=": + return attr_value >= val + elif op == "<": + return attr_value < val + elif op == "<=": + return attr_value <= val + elif op == "!=": + return attr_value != val + else: + raise NotImplementedError( + f"Unsupported relational operator {op!r} in numeric condition. Must be one of: '>', '>=', '<', '<=', '!='" + ) + + # Inclusive range check + elif len(value_to_match) == 2 and all(isinstance(v, (int, float)) for v in value_to_match): + return value_to_match[0] <= attr_value <= value_to_match[1] + + else: + raise NotImplementedError(f"No support for numeric matching condition {value_to_match}") + + # Regular expression match + elif isinstance(value_to_match, str): + return bool(re.fullmatch(value_to_match, str(attr_value))) + + # Exact numeric match + elif isinstance(value_to_match, (int, float)): + return value_to_match == attr_value + + else: + raise NotImplementedError(f"No support for numeric matching on this type: {type(value_to_match)}") + + def _match_image_orientation(self, value_to_match, attr_value): + """ + Helper method to calculate and match the image orientation using the ImageOrientationPatient tag. + The following PatientPosition values are supported and have been tested: + - "HFP" + - "HFS" + - "HFDL" + - "HFDR" + - "FFP" + - "FFS" + - "FFDL" + - "FFDR" + + Supported image orientation inputs for matching (case-insensitive): + - "Axial" + - "Coronal" + - "Sagittal" + + Args: + value_to_match (str): The image orientation condition to match against. + attr_value (List[str]): Raw ImageOrientationPatient tag value from the series. + + Returns: + bool: True if the computed orientation matches the expected orientation, else False. + + Raises: + ValueError: If the expected orientation is invalid or the normal vector cannot be computed. + """ + + # Validate image orientation to match input + value_to_match = value_to_match.strip().lower().capitalize() + allowed_orientations = {"Axial", "Coronal", "Sagittal"} + if value_to_match not in allowed_orientations: + raise ValueError(f"Invalid orientation string {value_to_match!r}. Must be one of: {allowed_orientations}") + + # Format ImageOrientationPatient tag value as an array and grab row and column cosines + iop_str = attr_value[0].strip("[]") + iop = [float(x.strip()) for x in iop_str.split(",")] + row_cosines = np.array(iop[:3], dtype=np.float64) + col_cosines = np.array(iop[3:], dtype=np.float64) + + # Validate DICOM constraints (normal row and column cosines + should be orthogonal) + # Throw warnings if tolerance exceeded + tolerance = 1e-4 + row_norm = np.linalg.norm(row_cosines) + col_norm = np.linalg.norm(col_cosines) + dot_product = np.dot(row_cosines, col_cosines) + + if abs(row_norm - 1.0) > tolerance: + logging.warn(f"Row direction cosine normal is {row_norm}, deviates from 1 by more than {tolerance}") + if abs(col_norm - 1.0) > tolerance: + logging.warn(f"Column direction cosine normal is {col_norm}, deviates from 1 by more than {tolerance}") + if abs(dot_product) > tolerance: + logging.warn(f"Row and Column cosines are not orthogonal: dot product = {dot_product}") + + # Normalize row and column vectors + row_cosines /= np.linalg.norm(row_cosines) + col_cosines /= np.linalg.norm(col_cosines) + + # Compute and validate slice normal + normal = np.cross(row_cosines, col_cosines) + if np.linalg.norm(normal) == 0: + raise ValueError("Invalid normal vector computed from IOP") + + # Normalize the slice normal + normal /= np.linalg.norm(normal) + + # Identify the dominant image orientation + axis_labels = ["Sagittal", "Coronal", "Axial"] + major_axis = np.argmax(np.abs(normal)) + computed_orientation = axis_labels[major_axis] + + logging.info(f" Computed orientation from ImageOrientationPatient value: {computed_orientation}") + + return bool(computed_orientation == value_to_match) + + @staticmethod + def _get_instance_properties(obj: object): + if not obj: + return {} + else: + return {x: getattr(obj, x, None) for x in type(obj).__dict__ if isinstance(type(obj).__dict__[x], property)} + + +# Module functions +# Helper function to get console output of the selection content when testing the script +def _print_instance_properties(obj: object, pre_fix: str = "", print_val=True): + print(f"{pre_fix}Instance of {type(obj)}") + for attribute in [x for x in type(obj).__dict__ if isinstance(type(obj).__dict__[x], property)]: + attr_val = getattr(obj, attribute, None) + print(f"{pre_fix} {attribute}: {type(attr_val)} {attr_val if print_val else ''}") + + +def test(): + from pathlib import Path + + from monai.deploy.operators.dicom_data_loader_operator import DICOMDataLoaderOperator + + current_file_dir = Path(__file__).parent.resolve() + data_path = current_file_dir.joinpath("../../../inputs/spleen_ct/dcm").absolute() + + fragment = Fragment() + loader = DICOMDataLoaderOperator(fragment, name="loader_op") + selector = DICOMSeriesSelectorOperator(fragment, name="selector_op") + study_list = loader.load_data_to_studies(data_path) + sample_selection_rule = json_loads(Sample_Rules_Text) + print(f"Selection rules in JSON:\n{sample_selection_rule}") + study_selected_series_list = selector.filter(sample_selection_rule, study_list) + + for sss_obj in study_selected_series_list: + _print_instance_properties(sss_obj, pre_fix="", print_val=False) + study = sss_obj.study + pre_fix = " " + print(f"{pre_fix}==== Details of the study ====") + _print_instance_properties(study, pre_fix, print_val=False) + print(f"{pre_fix}==============================") + + # The following commented code block accesses and prints the flat list of all selected series. + # for ss_obj in sss_obj.selected_series: + # pre_fix = " " + # _print_instance_properties(ss_obj, pre_fix, print_val=False) + # pre_fix = " " + # print(f"{pre_fix}==== Details of the series ====") + # _print_instance_properties(ss_obj, pre_fix) + # print(f"{pre_fix}===============================") + + # The following block uses hierarchical grouping by selection name, and prints the list of series for each. + for selection_name, ss_list in sss_obj.series_by_selection_name.items(): + pre_fix = " " + print(f"{pre_fix}Selection name: {selection_name}") + for ss_obj in ss_list: + pre_fix = " " + _print_instance_properties(ss_obj, pre_fix, print_val=False) + print(f"{pre_fix}==== Details of the series ====") + _print_instance_properties(ss_obj, pre_fix) + print(f"{pre_fix}===============================") + + print(f" A total of {len(sss_obj.selected_series)} series selected for study {study.StudyInstanceUID}") + + +# Sample rule used for testing +Sample_Rules_Text = """ +{ + "selections": [ + { + "name": "CT Series 1", + "conditions": { + "StudyDescription": "(?i)^Spleen", + "Modality": "(?i)CT", + "SeriesDescription": "(?i)^No series description|(.*?)" + } + }, + { + "name": "CT Series 2", + "conditions": { + "Modality": "CT", + "BodyPartExamined": "Abdomen", + "SeriesDescription" : "Not to be matched" + } + }, + { + "name": "CT Series 3", + "conditions": { + "StudyDescription": "(.*?)", + "Modality": "(?i)CT", + "ImageType": ["PRIMARY", "ORIGINAL", "AXIAL"], + "SliceThickness": [3, 5] + } + }, + { + "name": "CT Series 4", + "conditions": { + "StudyDescription": "(.*?)", + "Modality": "(?i)MR", + "ImageOrientationPatient": "Axial", + "SliceThickness": [2, ">"] + } + }, + { + "name": "CT Series 5", + "conditions": { + "StudyDescription": "(.*?)", + "Modality": "(?i)CT", + "ImageType": ["PRIMARY", "!SECONDARY"] + } + } + ] +} +""" + +if __name__ == "__main__": + test() diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/nnunet_bundle.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/nnunet_bundle.py new file mode 100644 index 00000000..6e24b7a3 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/nnunet_bundle.py @@ -0,0 +1,1001 @@ +# Copyright (c) MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from __future__ import annotations + +import os +import shutil +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple, Union + +import numpy as np +import torch +from torch.backends import cudnn + +from monai.data.meta_tensor import MetaTensor +from monai.utils import optional_import + +join, _ = optional_import("batchgenerators.utilities.file_and_folder_operations", name="join") +load_json, _ = optional_import("batchgenerators.utilities.file_and_folder_operations", name="load_json") +nnunet_predictor_cls, _ = optional_import("nnunetv2.inference.predict_from_raw_data", name="nnUNetPredictor") + +__all__ = [ + "get_nnunet_trainer", + "get_nnunet_monai_predictor", + "get_network_from_nnunet_plans", + "convert_nnunet_to_monai_bundle", + "convert_monai_bundle_to_nnunet", + "ModelnnUNetWrapper", + "EnsembleProbabilitiesToSegmentation", +] + +# Constants +NNUNET_CHECKPOINT_FILENAME = "nnunet_checkpoint.pth" +PLANS_JSON_FILENAME = "plans.json" +DATASET_JSON_FILENAME = "dataset.json" + + +# Convert a single nnUNet model checkpoint to MONAI bundle format +# The function saves the converted model checkpoint and configuration files in the specified bundle root folder. +def convert_nnunet_to_monai_bundle(nnunet_config: Dict[str, Any], bundle_root_folder: str, fold: int = 0) -> None: + """ + Convert nnUNet model checkpoints and configuration to MONAI bundle format. + + Parameters + ---------- + nnunet_config : dict + Configuration dictionary for nnUNet, containing keys such as 'dataset_name_or_id', 'nnunet_configuration', + 'nnunet_trainer', and 'nnunet_plans'. + bundle_root_folder : str + Root folder where the MONAI bundle will be saved. + fold : int, optional + Fold number of the nnUNet model to be converted, by default 0. + + Returns + ------- + None + """ + + nnunet_trainer = "nnUNetTrainer" + nnunet_plans = "nnUNetPlans" + nnunet_configuration = "3d_fullres" + + if "nnunet_trainer" in nnunet_config: + nnunet_trainer = nnunet_config["nnunet_trainer"] + + if "nnunet_plans" in nnunet_config: + nnunet_plans = nnunet_config["nnunet_plans"] + + if "nnunet_configuration" in nnunet_config: + nnunet_configuration = nnunet_config["nnunet_configuration"] + + from nnunetv2.utilities.dataset_name_id_conversion import maybe_convert_to_dataset_name + + dataset_name = maybe_convert_to_dataset_name(nnunet_config["dataset_name_or_id"]) + nnunet_model_folder = Path(os.environ["nnUNet_results"]).joinpath( + dataset_name, f"{nnunet_trainer}__{nnunet_plans}__{nnunet_configuration}" + ) + + nnunet_checkpoint_final = torch.load( + Path(nnunet_model_folder).joinpath(f"fold_{fold}", "checkpoint_final.pth"), weights_only=False + ) + nnunet_checkpoint_best = torch.load( + Path(nnunet_model_folder).joinpath(f"fold_{fold}", "checkpoint_best.pth"), weights_only=False + ) + + nnunet_checkpoint = {} + nnunet_checkpoint["inference_allowed_mirroring_axes"] = nnunet_checkpoint_final["inference_allowed_mirroring_axes"] + nnunet_checkpoint["init_args"] = nnunet_checkpoint_final["init_args"] + nnunet_checkpoint["trainer_name"] = nnunet_checkpoint_final["trainer_name"] + + Path(bundle_root_folder).joinpath("models", nnunet_configuration).mkdir(parents=True, exist_ok=True) + + torch.save( + nnunet_checkpoint, Path(bundle_root_folder).joinpath("models", nnunet_configuration, NNUNET_CHECKPOINT_FILENAME) + ) + + Path(bundle_root_folder).joinpath("models", nnunet_configuration, f"fold_{fold}").mkdir(parents=True, exist_ok=True) + # This might not be needed, comment it out for now + # monai_last_checkpoint = {} + # monai_last_checkpoint["network_weights"] = nnunet_checkpoint_final["network_weights"] + # torch.save( + # monai_last_checkpoint, + # Path(bundle_root_folder).joinpath("models", nnunet_configuration, f"fold_{fold}", "model.pt") + # ) + + monai_best_checkpoint = {} + monai_best_checkpoint["network_weights"] = nnunet_checkpoint_best["network_weights"] + torch.save( + monai_best_checkpoint, + Path(bundle_root_folder).joinpath("models", nnunet_configuration, f"fold_{fold}", "best_model.pt"), + ) + + if not os.path.exists(os.path.join(bundle_root_folder, "models", "jsonpkls", PLANS_JSON_FILENAME)): + shutil.copy( + Path(nnunet_model_folder).joinpath(PLANS_JSON_FILENAME), + Path(bundle_root_folder).joinpath("models", "jsonpkls", PLANS_JSON_FILENAME), + ) + + if not os.path.exists(os.path.join(bundle_root_folder, "models", "jsonpkls", DATASET_JSON_FILENAME)): + shutil.copy( + Path(nnunet_model_folder).joinpath(DATASET_JSON_FILENAME), + Path(bundle_root_folder).joinpath("models", "jsonpkls", DATASET_JSON_FILENAME), + ) + + +# A function to convert all nnunet models (configs and folds) to MONAI bundle format. +# The function iterates through all folds and configurations, converting each model to the specified bundle format. +# The number of folds, configurations, plans and dataset.json will be parsed from the nnunet folder +def convert_best_nnunet_to_monai_bundle( + nnunet_config: Dict[str, Any], bundle_root_folder: str, inference_info_file: str = "inference_information.json" +) -> None: + """ + Convert all nnUNet models (configs and folds) to MONAI bundle format. + + Parameters + ---------- + nnunet_config : dict + Configuration dictionary for nnUNet. Expected keys are: + - "dataset_name_or_id": str, name or ID of the dataset. + - "nnunet_configuration": str, configuration name. + - "nnunet_trainer": str, optional, name of the nnU-Net trainer (default is "nnUNetTrainer"). + - "nnunet_plans": str, optional, name of the nnU-Net plans (default is "nnUNetPlans"). + bundle_root_folder : str + Path to the root folder of the MONAI bundle. + inference_info : str, optional + Path to the inference information file (default is "inference_information.json"). + + Returns + ------- + None + """ + from nnunetv2.utilities.dataset_name_id_conversion import maybe_convert_to_dataset_name + + dataset_name = nnunet_config["dataset_name_or_id"] + + inference_info_path = Path(os.environ["nnUNet_results"]).joinpath( + maybe_convert_to_dataset_name(dataset_name), inference_info_file + ) + + if not os.path.exists(inference_info_path): + raise FileNotFoundError(f"Inference information file not found: {inference_info_path}") + inference_info = load_json(inference_info_path) + + # Get the best model or ensemble from the inference information + if "best_model_or_ensemble" not in inference_info: + raise KeyError(f"Key 'best_model_or_ensemble' not found in inference information file: {inference_info_path}") + best_model_dict = inference_info["best_model_or_ensemble"] + + # Get the folds information + if "folds" not in inference_info: + raise KeyError(f"Key 'folds' not found in inference information file: {inference_info_path}") + folds = inference_info["folds"] # list of folds + + cascade_3d_fullres = False + for model_dict in best_model_dict["selected_model_or_models"]: + if model_dict["configuration"] == "3d_cascade_fullres": + cascade_3d_fullres = True + + print("Converting model: ", model_dict["configuration"]) + nnunet_model_folder = Path(os.environ["nnUNet_results"]).joinpath( + maybe_convert_to_dataset_name(dataset_name), + f"{model_dict['trainer']}__{model_dict['plans_identifier']}__{model_dict['configuration']}", + ) + nnunet_config["nnunet_configuration"] = model_dict["configuration"] + nnunet_config["nnunet_trainer"] = model_dict["trainer"] + nnunet_config["nnunet_plans"] = model_dict["plans_identifier"] + + if not os.path.exists(nnunet_model_folder): + raise FileNotFoundError(f"Model folder not found: {nnunet_model_folder}") + + for fold in folds: + print("Converting fold: ", fold, " of model: ", model_dict["configuration"]) + convert_nnunet_to_monai_bundle(nnunet_config, bundle_root_folder, fold) + + # IF model is a cascade model, 3d_lowres is also needed + if cascade_3d_fullres: + # check if 3d_lowres is already in the bundle + if not os.path.exists(os.path.join(bundle_root_folder, "models", "3d_lowres")): + # copy the 3d_lowres model folder from nnunet results + nnunet_model_folder = Path(os.environ["nnUNet_results"]).joinpath( + maybe_convert_to_dataset_name(dataset_name), + f"{model_dict['trainer']}__{model_dict['plans_identifier']}__3d_lowres", + ) + if not os.path.exists(nnunet_model_folder): + raise FileNotFoundError(f"Model folder not found: {nnunet_model_folder}") + # copy the 3d_lowres model folder to the bundle root folder + nnunet_config["nnunet_configuration"] = "3d_lowres" + nnunet_config["nnunet_trainer"] = best_model_dict["selected_model_or_models"][-1][ + "trainer" + ] # Using the same trainer as the cascade model + nnunet_config["nnunet_plans"] = best_model_dict["selected_model_or_models"][-1][ + "plans_identifier" + ] # Using the same plans id as the cascade model + for fold in folds: + print("Converting fold: ", fold, " of model: ", "3d_lowres") + convert_nnunet_to_monai_bundle(nnunet_config, bundle_root_folder, fold) + + # Finally if postprocessing is needed (for ensemble models) + if "postprocessing_file" in best_model_dict: + postprocessing_file_path = best_model_dict["postprocessing_file"] + if not os.path.exists(postprocessing_file_path): + raise FileNotFoundError(f"Postprocessing file not found: {postprocessing_file_path}") + shutil.copy(postprocessing_file_path, Path(bundle_root_folder).joinpath("models", "postprocessing.pkl")) + + +def convert_monai_bundle_to_nnunet(nnunet_config: dict, bundle_root_folder: str, fold: int = 0) -> None: + """ + Convert a MONAI bundle to nnU-Net format. + + Parameters + ---------- + nnunet_config : dict + Configuration dictionary for nnU-Net. Expected keys are: + - "dataset_name_or_id": str, name or ID of the dataset. + - "nnunet_trainer": str, optional, name of the nnU-Net trainer (default is "nnUNetTrainer"). + - "nnunet_plans": str, optional, name of the nnU-Net plans (default is "nnUNetPlans"). + bundle_root_folder : str + Path to the root folder of the MONAI bundle. + fold : int, optional + Fold number for cross-validation (default is 0). + + Returns + ------- + None + """ + from odict import odict + + nnunet_trainer: str = "nnUNetTrainer" + nnunet_plans: str = "nnUNetPlans" + + if "nnunet_trainer" in nnunet_config: + nnunet_trainer = nnunet_config["nnunet_trainer"] + + if "nnunet_plans" in nnunet_config: + nnunet_plans = nnunet_config["nnunet_plans"] + + from nnunetv2.training.logging.nnunet_logger import nnUNetLogger + from nnunetv2.utilities.dataset_name_id_conversion import maybe_convert_to_dataset_name + + def subfiles( + folder: Union[str, Path], prefix: Optional[str] = None, suffix: Optional[str] = None, sort: bool = True + ) -> list[str]: + res = [ + i.name + for i in Path(folder).iterdir() + if i.is_file() + and (prefix is None or i.name.startswith(prefix)) + and (suffix is None or i.name.endswith(suffix)) + ] + if sort: + res.sort() + return res + + nnunet_model_folder: Path = Path(os.environ["nnUNet_results"]).joinpath( + maybe_convert_to_dataset_name(nnunet_config["dataset_name_or_id"]), + f"{nnunet_trainer}__{nnunet_plans}__3d_fullres", + ) + + nnunet_preprocess_model_folder: Path = Path(os.environ["nnUNet_preprocessed"]).joinpath( + maybe_convert_to_dataset_name(nnunet_config["dataset_name_or_id"]) + ) + + Path(nnunet_model_folder).joinpath(f"fold_{fold}").mkdir(parents=True, exist_ok=True) + + nnunet_checkpoint: dict = torch.load( + f"{bundle_root_folder}/models/{NNUNET_CHECKPOINT_FILENAME}", weights_only=False + ) + latest_checkpoints: list[str] = subfiles( + Path(bundle_root_folder).joinpath("models", f"fold_{fold}"), prefix="checkpoint_epoch", sort=True + ) + epochs: list[int] = [] + for latest_checkpoint in latest_checkpoints: + epochs.append(int(latest_checkpoint[len("checkpoint_epoch=") : -len(".pt")])) + + epochs.sort() + final_epoch: int = epochs[-1] + monai_last_checkpoint: dict = torch.load( + f"{bundle_root_folder}/models/fold_{fold}/checkpoint_epoch={final_epoch}.pt", weights_only=False + ) + + best_checkpoints: list[str] = subfiles( + Path(bundle_root_folder).joinpath("models", f"fold_{fold}"), prefix="checkpoint_key_metric", sort=True + ) + key_metrics: list[str] = [] + for best_checkpoint in best_checkpoints: + key_metrics.append(str(best_checkpoint[len("checkpoint_key_metric=") : -len(".pt")])) + + key_metrics.sort() + best_key_metric: str = key_metrics[-1] + monai_best_checkpoint: dict = torch.load( + f"{bundle_root_folder}/models/fold_{fold}/checkpoint_key_metric={best_key_metric}.pt", weights_only=False + ) + + if "optimizer_state" in monai_last_checkpoint: + nnunet_checkpoint["optimizer_state"] = monai_last_checkpoint["optimizer_state"] + + nnunet_checkpoint["network_weights"] = odict() + + for key in monai_last_checkpoint["network_weights"]: + nnunet_checkpoint["network_weights"][key] = monai_last_checkpoint["network_weights"][key] + + nnunet_checkpoint["current_epoch"] = final_epoch + nnunet_checkpoint["logging"] = nnUNetLogger().get_checkpoint() + nnunet_checkpoint["_best_ema"] = 0 + nnunet_checkpoint["grad_scaler_state"] = None + + torch.save(nnunet_checkpoint, Path(nnunet_model_folder).joinpath(f"fold_{fold}", "checkpoint_final.pth")) + + nnunet_checkpoint["network_weights"] = odict() + + if "optimizer_state" in monai_last_checkpoint: + nnunet_checkpoint["optimizer_state"] = monai_best_checkpoint["optimizer_state"] + + for key in monai_best_checkpoint["network_weights"]: + nnunet_checkpoint["network_weights"][key] = monai_best_checkpoint["network_weights"][key] + + torch.save(nnunet_checkpoint, Path(nnunet_model_folder).joinpath(f"fold_{fold}", "checkpoint_best.pth")) + + if not os.path.exists(os.path.join(nnunet_model_folder, DATASET_JSON_FILENAME)): + shutil.copy(f"{bundle_root_folder}/models/jsonpkls/{DATASET_JSON_FILENAME}", nnunet_model_folder) + if not os.path.exists(os.path.join(nnunet_model_folder, PLANS_JSON_FILENAME)): + shutil.copy(f"{bundle_root_folder}/models/jsonpkls/{PLANS_JSON_FILENAME}", nnunet_model_folder) + if not os.path.exists(os.path.join(nnunet_model_folder, "dataset_fingerprint.json")): + shutil.copy(f"{nnunet_preprocess_model_folder}/dataset_fingerprint.json", nnunet_model_folder) + if not os.path.exists(os.path.join(nnunet_model_folder, NNUNET_CHECKPOINT_FILENAME)): + shutil.copy(f"{bundle_root_folder}/models/{NNUNET_CHECKPOINT_FILENAME}", nnunet_model_folder) + + +# This function loads a nnUNet network from the provided plans and dataset files. +# It initializes the network architecture and loads the model weights if a checkpoint is provided. +def get_network_from_nnunet_plans( + plans_file: str, + dataset_file: str, + configuration: str, + model_ckpt: Optional[str] = None, + model_key_in_ckpt: str = "model", +) -> Union[torch.nn.Module, Any]: + """ + Load and initialize a nnUNet network based on nnUNet plans and configuration. + + Parameters + ---------- + plans_file : str + Path to the JSON file containing the nnUNet plans. + dataset_file : str + Path to the JSON file containing the dataset information. + configuration : str + The configuration name to be used from the plans. + model_ckpt : Optional[str], optional + Path to the model checkpoint file. If None, the network is returned without loading weights (default is None). + model_key_in_ckpt : str, optional + The key in the checkpoint file that contains the model state dictionary (default is "model"). + + Returns + ------- + network : torch.nn.Module + The initialized neural network, with weights loaded if `model_ckpt` is provided. + """ + from batchgenerators.utilities.file_and_folder_operations import load_json + from nnunetv2.utilities.get_network_from_plans import get_network_from_plans + from nnunetv2.utilities.label_handling.label_handling import determine_num_input_channels + from nnunetv2.utilities.plans_handling.plans_handler import PlansManager + + plans = load_json(plans_file) + dataset_json = load_json(dataset_file) + + plans_manager = PlansManager(plans) + configuration_manager = plans_manager.get_configuration(configuration) + num_input_channels = determine_num_input_channels(plans_manager, configuration_manager, dataset_json) + label_manager = plans_manager.get_label_manager(dataset_json) + + enable_deep_supervision = True + + network = get_network_from_plans( + configuration_manager.network_arch_class_name, + configuration_manager.network_arch_init_kwargs, + configuration_manager.network_arch_init_kwargs_req_import, + num_input_channels, + label_manager.num_segmentation_heads, + allow_init=True, + deep_supervision=enable_deep_supervision, + ) + + if model_ckpt is None: + return network + else: + state_dict = torch.load(model_ckpt, weights_only=False) + network.load_state_dict(state_dict[model_key_in_ckpt]) + return network + + +def get_nnunet_trainer( + dataset_name_or_id: Union[str, int], + configuration: str, + fold: Union[int, str], + trainer_class_name: str = "nnUNetTrainer", + plans_identifier: str = "nnUNetPlans", + use_compressed_data: bool = False, + continue_training: bool = False, + only_run_validation: bool = False, + disable_checkpointing: bool = False, + device: str = "cuda", + pretrained_model: Optional[str] = None, +) -> Any: # type: ignore + """ + Get the nnUNet trainer instance based on the provided configuration. + The returned nnUNet trainer can be used to initialize the SupervisedTrainer for training, including the network, + optimizer, loss function, DataLoader, etc. + + Example:: + + from monai.apps import SupervisedTrainer + from monai.bundle.nnunet import get_nnunet_trainer + + dataset_name_or_id = 'Task009_Spleen' + fold = 0 + configuration = '3d_fullres' + nnunet_trainer = get_nnunet_trainer(dataset_name_or_id, configuration, fold) + + trainer = SupervisedTrainer( + device=nnunet_trainer.device, + max_epochs=nnunet_trainer.num_epochs, + train_data_loader=nnunet_trainer.dataloader_train, + network=nnunet_trainer.network, + optimizer=nnunet_trainer.optimizer, + loss_function=nnunet_trainer.loss_function, + epoch_length=nnunet_trainer.num_iterations_per_epoch, + ) + + Parameters + ---------- + dataset_name_or_id : Union[str, int] + The name or ID of the dataset to be used. + configuration : str + The configuration name for the training. + fold : Union[int, str] + The fold number or 'all' for cross-validation. + trainer_class_name : str, optional + The class name of the trainer to be used. Default is 'nnUNetTrainer'. + For a complete list of supported trainers, check: + https://github.com/MIC-DKFZ/nnUNet/tree/master/nnunetv2/training/nnUNetTrainer/variants + plans_identifier : str, optional + Identifier for the plans to be used. Default is 'nnUNetPlans'. + use_compressed_data : bool, optional + Whether to use compressed data. Default is False. + continue_training : bool, optional + Whether to continue training from a checkpoint. Default is False. + only_run_validation : bool, optional + Whether to only run validation. Default is False. + disable_checkpointing : bool, optional + Whether to disable checkpointing. Default is False. + device : str, optional + The device to be used for training. Default is 'cuda'. + pretrained_model : Optional[str], optional + Path to the pretrained model file. + + Returns + ------- + nnunet_trainer : object + The nnUNet trainer instance. + """ + # From nnUNet/nnunetv2/run/run_training.py#run_training + if isinstance(fold, str): + if fold != "all": + try: + fold = int(fold) + except ValueError as e: + print( + f'Unable to convert given value for fold to int: {fold}. fold must bei either "all" or an integer!' + ) + raise e + + from nnunetv2.run.run_training import get_trainer_from_args, maybe_load_checkpoint + + nnunet_trainer = get_trainer_from_args( + str(dataset_name_or_id), + configuration, + fold, + trainer_class_name, + plans_identifier, + device=torch.device(device), + ) + if disable_checkpointing: + nnunet_trainer.disable_checkpointing = disable_checkpointing + + assert not (continue_training and only_run_validation), "Cannot set --c and --val flag at the same time. Dummy." + + maybe_load_checkpoint(nnunet_trainer, continue_training, only_run_validation) + nnunet_trainer.on_train_start() # Added to Initialize Trainer + if torch.cuda.is_available(): + cudnn.deterministic = False + cudnn.benchmark = True + + if pretrained_model is not None: + state_dict = torch.load(pretrained_model, weights_only=False) + if "network_weights" in state_dict: + nnunet_trainer.network._orig_mod.load_state_dict(state_dict["network_weights"]) + return nnunet_trainer + + +def get_nnunet_monai_predictor( + model_folder: Union[str, Path], + model_name: str = "model.pt", + dataset_json: Optional[Dict[Any, Any]] = None, + plans: Optional[Dict[Any, Any]] = None, + nnunet_config: Optional[Dict[Any, Any]] = None, + save_probabilities: bool = False, + save_files: bool = False, + use_folds: Optional[Union[int, str]] = None, +) -> ModelnnUNetWrapper: + """ + Initializes and returns a `nnUNetMONAIModelWrapper` containing the corresponding `nnUNetPredictor`. + The model folder should contain the following files, created during training: + + - dataset.json: from the nnUNet results folder + - plans.json: from the nnUNet results folder + - nnunet_checkpoint.pth: The nnUNet checkpoint file, containing the nnUNet training configuration + - model.pt: The checkpoint file containing the model weights. + + The returned wrapper object can be used for inference with MONAI framework: + Example:: + + from monai.bundle.nnunet import get_nnunet_monai_predictor + + model_folder = 'path/to/monai_bundle/model' + model_name = 'model.pt' + wrapper = get_nnunet_monai_predictor(model_folder, model_name) + + # Perform inference + input_data = ... + output = wrapper(input_data) + + + Parameters + ---------- + model_folder : Union[str, Path] + The folder where the model is stored. + model_name : str, optional + The name of the model file, by default "model.pt". + dataset_json : dict, optional + The dataset JSON file containing dataset information. + plans : dict, optional + The plans JSON file containing model configuration. + nnunet_config : dict, optional + The nnUNet configuration dictionary containing model parameters. + + Returns + ------- + ModelnnUNetWrapper + A wrapper object that contains the nnUNetPredictor and the loaded model. + """ + + from nnunetv2.inference.predict_from_raw_data import nnUNetPredictor + + predictor = nnUNetPredictor( + tile_step_size=0.5, + use_gaussian=True, + use_mirroring=True, + device=torch.device("cuda", 0), + verbose=True, + verbose_preprocessing=False, + allow_tqdm=True, + ) + # initializes the network architecture, loads the checkpoint + print("nnunet_predictor: Model Folder: ", model_folder) + print("nnunet_predictor: Model name: ", model_name) + print("nnunet_predictor: use_folds: ", use_folds) + wrapper = ModelnnUNetWrapper( + predictor, + model_folder=model_folder, + checkpoint_name=model_name, + dataset_json=dataset_json, + plans=plans, + nnunet_config=nnunet_config, + save_probabilities=save_probabilities, + save_files=save_files, + use_folds=use_folds, + ) + return wrapper + + +def get_nnunet_monai_predictors_for_ensemble( + model_list: List[Any], + model_path: Union[str, Path], + model_name: str = "model.pt", + use_folds: Optional[Union[int, str]] = None, +) -> Tuple[ModelnnUNetWrapper, ...]: + network_list = [] + for model_config in model_list: + model_folder = Path(model_path).joinpath(model_config) + network_list.append( + get_nnunet_monai_predictor( + model_folder=model_folder, + model_name=model_name, + save_probabilities=True, + save_files=True, + use_folds=use_folds, + ) + ) + return tuple(network_list) + + +from nnunetv2.ensembling.ensemble import average_probabilities +from nnunetv2.utilities.plans_handling.plans_handler import PlansManager + +from monai.config import KeysCollection +from monai.transforms import MapTransform + + +class EnsembleProbabilitiesToSegmentation(MapTransform): + """ + MONAI transform that loads .npz probability files from metadata['saved_file'] for a given key, + averages them, and converts to final segmentation using nnU-Net's LabelManager. + Returns a MetaTensor segmentation result (instead of saving to disk). + """ + + def __init__( + self, + keys: KeysCollection, + dataset_json_path: str, + plans_json_path: str, + allow_missing_keys: bool = False, + output_key: str = "pred", + ): + super().__init__(keys, allow_missing_keys) + + # Load required nnU-Net configs + self.plans_manager = PlansManager(plans_json_path) + self.dataset_json = self._load_json(dataset_json_path) + self.label_manager = self.plans_manager.get_label_manager(self.dataset_json) + self.output_key = output_key + + def _load_json(self, path: str) -> Dict[Any, Any]: + import json + + with open(path, "r") as f: + result = json.load(f) + return dict(result) # Ensure return type matches annotation + + def __call__(self, data: Dict[Any, Any]) -> Dict[Any, Any]: + d = dict(data) + all_files = [] + for key in self.keys: + meta = d[key].meta if isinstance(d[key], MetaTensor) else d.get("meta", {}) + saved_file = meta.get("saved_file", None) + + # Support multiple files for ensemble + if isinstance(saved_file, str): + saved_file = [saved_file] + elif not isinstance(saved_file, list): + raise ValueError(f"'saved_file' in meta must be str or List[str], got {type(saved_file)}") + + for f in saved_file: + if not os.path.exists(f): + raise FileNotFoundError(f"Probability file not found: {f}") + all_files.append(f) + + print("All files to average: ", all_files) + # Step 1: average probabilities + avg_probs = average_probabilities(all_files) + + # Step 2: convert to segmentation + segmentation = self.label_manager.convert_logits_to_segmentation(avg_probs) # shape: (H, W, D) + + # Step 3: wrap as MetaTensor and attach meta + seg_tensor = MetaTensor(segmentation[None].astype(np.uint8)) # add channel dim + seg_tensor.meta = dict(meta) + + # Replace the key or store in new key + d[self.output_key] = seg_tensor + return d + + +class ModelnnUNetWrapper(torch.nn.Module): + """ + A wrapper class for nnUNet model integration with MONAI framework. + The wrapper can be use to integrate the nnUNet Bundle within MONAI framework for inference. + + Parameters + ---------- + predictor : nnUNetPredictor + The nnUNet predictor object used for inference. + model_folder : Union[str, Path] + The folder path where the model and related files are stored. + model_name : str, optional + The name of the model file, by default "model.pt". + dataset_json : dict, optional + The dataset JSON file containing dataset information. + plans : dict, optional + The plans JSON file containing model configuration. + nnunet_config : dict, optional + The nnUNet configuration dictionary containing model parameters. + + Attributes + ---------- + predictor : nnUNetPredictor + The nnUNet predictor object used for inference. + network_weights : torch.nn.Module + The network weights of the model. + + Notes + ----- + This class integrates nnUNet model with MONAI framework by loading necessary configurations, + restoring network architecture, and setting up the predictor for inference. + """ + + def __init__( + self, + predictor: Any, # nnUNetPredictor type, but using Any to avoid import issues + model_folder: Union[str, Path], + checkpoint_name: Optional[str] = None, + dataset_json: Optional[Dict[Any, Any]] = None, + plans: Optional[Dict[Any, Any]] = None, + nnunet_config: Optional[Dict[Any, Any]] = None, + save_probabilities: bool = False, + save_files: bool = False, + tmp_dir: str = "tmp", + use_folds: Optional[Union[int, str, Tuple[Union[int, str], ...], List[Union[int, str]]]] = None, + ): + + super().__init__() + self.predictor = predictor + + if not checkpoint_name: + raise ValueError("Model name is required. Please provide a valid model name.") + + self.tmp_dir = tmp_dir + self.save_probabilities = save_probabilities + self.save_files = save_files + + # Set up model paths + model_training_output_dir = model_folder + model_parent_dir = Path(model_training_output_dir).parent + + # Import required modules + from nnunetv2.utilities.plans_handling.plans_handler import PlansManager + + # Load dataset and plans if not provided + if dataset_json is None: + dataset_json = load_json(join(Path(model_parent_dir), "jsonpkls", DATASET_JSON_FILENAME)) + if plans is None: + plans = load_json(join(Path(model_parent_dir), "jsonpkls", PLANS_JSON_FILENAME)) + + plans_manager = PlansManager(plans) + parameters = [] + + # Get configuration from nnunet_checkpoint.pth or provided config + if nnunet_config is None: + checkpoint_path = join(Path(model_training_output_dir), NNUNET_CHECKPOINT_FILENAME) + if not os.path.exists(checkpoint_path): + raise ValueError( + f"Checkpoint file not found at {checkpoint_path}. Please ensure the model is trained and the checkpoint exists." + ) + + checkpoint = torch.load(checkpoint_path, weights_only=False, map_location=torch.device("cpu")) + trainer_name = checkpoint["trainer_name"] + configuration_name = checkpoint["init_args"]["configuration"] + inference_allowed_mirroring_axes = ( + checkpoint["inference_allowed_mirroring_axes"] + if "inference_allowed_mirroring_axes" in checkpoint.keys() + else None + ) + else: + trainer_name = nnunet_config["trainer_name"] + configuration_name = nnunet_config["configuration"] + inference_allowed_mirroring_axes = nnunet_config["inference_allowed_mirroring_axes"] + + # Store configuration name + self.configuration_name = configuration_name + + # Handle folds + if isinstance(use_folds, str) or isinstance(use_folds, int): + use_folds = [use_folds] + + if use_folds is None: + use_folds = self.predictor.auto_detect_available_folds(model_training_output_dir, checkpoint_name) + + # Ensure use_folds is always iterable + if not isinstance(use_folds, (list, tuple)): + use_folds = [use_folds] + + # Load model parameters from each fold + for f in use_folds: + f = int(f) if f != "all" else f + fold_checkpoint_path = join(model_training_output_dir, f"fold_{f}", checkpoint_name) + monai_checkpoint = torch.load(fold_checkpoint_path, map_location=torch.device("cpu"), weights_only=False) + + if "network_weights" in monai_checkpoint.keys(): + parameters.append(monai_checkpoint["network_weights"]) + else: + parameters.append(monai_checkpoint) + + # Get configuration manager and setup network + configuration_manager = plans_manager.get_configuration(configuration_name) + + # Import required nnUNet modules + import nnunetv2 + from nnunetv2.utilities.find_class_by_name import recursive_find_python_class + from nnunetv2.utilities.label_handling.label_handling import determine_num_input_channels + + # Determine input channels and find trainer class + num_input_channels = determine_num_input_channels(plans_manager, configuration_manager, dataset_json) + trainer_class = recursive_find_python_class( + join(nnunetv2.__path__[0], "training", "nnUNetTrainer"), trainer_name, "nnunetv2.training.nnUNetTrainer" + ) + + if trainer_class is None: + raise RuntimeError(f"Unable to locate trainer class {trainer_name} in nnunetv2.training.nnUNetTrainer.") + + # Build network architecture + network = trainer_class.build_network_architecture( + configuration_manager.network_arch_class_name, + configuration_manager.network_arch_init_kwargs, + configuration_manager.network_arch_init_kwargs_req_import, + num_input_channels, + plans_manager.get_label_manager(dataset_json).num_segmentation_heads, + enable_deep_supervision=False, + ) + + # Configure predictor with all required settings + predictor.plans_manager = plans_manager + predictor.configuration_manager = configuration_manager + predictor.list_of_parameters = parameters + predictor.network = network + predictor.dataset_json = dataset_json + predictor.trainer_name = trainer_name + predictor.allowed_mirroring_axes = inference_allowed_mirroring_axes + predictor.label_manager = plans_manager.get_label_manager(dataset_json) + + # Store network weights reference + self.network_weights = self.predictor.network + + def forward(self, x: MetaTensor) -> MetaTensor: + """ + Forward pass for the nnUNet model. + + Args: + x (MetaTensor): Input tensor for inference. + + Returns: + MetaTensor: The output tensor with the same metadata as the input. + + Raises: + TypeError: If the input is not a MetaTensor. + """ + if not isinstance(x, MetaTensor): + raise TypeError("Input must be a MetaTensor.") + + # Extract spatial shape from input + spatial_shape = list(x.shape[-3:]) # [H, W, D] or [X, Y, Z] + + # Get spacing information from metadata + properties_or_list_of_properties = {} + + if "pixdim" in x.meta: + # Get spacing from pixdim + if x.meta["pixdim"].ndim == 1: + properties_or_list_of_properties["spacing"] = x.meta["pixdim"][1:4].tolist() + else: + properties_or_list_of_properties["spacing"] = x.meta["pixdim"][0][1:4].numpy().tolist() + + elif "affine" in x.meta: + # Get spacing from affine matrix + affine = x.meta["affine"][0].cpu().numpy() if x.meta["affine"].ndim == 3 else x.meta["affine"].cpu().numpy() + spacing = np.array( + [ + np.sqrt(np.sum(affine[:3, 0] ** 2)), + np.sqrt(np.sum(affine[:3, 1] ** 2)), + np.sqrt(np.sum(affine[:3, 2] ** 2)), + ] + ) + properties_or_list_of_properties["spacing"] = spacing + else: + # Default spacing if no metadata available + properties_or_list_of_properties["spacing"] = [1.0, 1.0, 1.0] + + # Add spatial shape to properties + properties_or_list_of_properties["spatial_shape"] = spatial_shape + + # Convert input tensor to numpy array + image_or_list_of_images = x.cpu().numpy()[0, :] + + # Setup output file path if saving enabled + outfile = None + if self.save_files: + # Get original filename from metadata + infile = x.meta["filename_or_obj"] + if isinstance(infile, list): + infile = infile[0] + + # Create output path + outfile_name = os.path.basename(infile).split(".")[0] + outfolder = Path(self.tmp_dir).joinpath(self.configuration_name) + os.makedirs(outfolder, exist_ok=True) + outfile = str(Path(outfolder).joinpath(outfile_name)) + + # Extract 4x4 affine matrix for SimpleITK compatibility + if "affine" in x.meta: + # Get affine matrix with proper shape + if x.meta["affine"].shape == (1, 4, 4): + affine = x.meta["affine"][0].cpu().numpy() + elif x.meta["affine"].shape == (4, 4): + affine = x.meta["affine"].cpu().numpy() + else: + raise ValueError(f"Unexpected affine shape: {x.meta['affine'].shape}") + + # Calculate spacing, origin and direction + spacing_tuple = tuple(float(np.linalg.norm(affine[:3, i])) for i in range(3)) + origin = tuple(float(v) for v in affine[:3, 3]) + direction_matrix = affine[:3, :3] / np.array(spacing_tuple) + direction = tuple(direction_matrix.flatten().round(6)) + + # Add to properties dict for SimpleITK + properties_or_list_of_properties["sitk_stuff"] = { + "spacing": spacing_tuple, + "origin": origin, + "direction": direction, + } + # Handle cascade models by loading segmentation from previous stage + previous_segmentation = None + if self.configuration_name == "3d_cascade_fullres": + # For cascade models, we need the lowres prediction + lowres_predictions_folder = os.path.join(self.tmp_dir, "3d_lowres") + + if outfile: + seg_file = os.path.join(lowres_predictions_folder, outfile_name + ".nii.gz") + # Load the lowres segmentation from file + rw = self.predictor.plans_manager.image_reader_writer_class() + previous_segmentation, _ = rw.read_seg(seg_file) + + if previous_segmentation is None: + raise ValueError("Failed to load previous segmentation for cascade model.") + else: + raise ValueError("Output file name is required for 3d_cascade_fullres configuration.") + + # Run prediction using nnUNet predictor + prediction_output = self.predictor.predict_from_list_of_npy_arrays( + image_or_list_of_images, + previous_segmentation, + properties_or_list_of_properties, + save_probabilities=self.save_probabilities, + truncated_ofname=outfile, + num_processes=2, + num_processes_segmentation_export=2, + ) + + # Process prediction output based on save_files setting + if not self.save_files: + # Return the prediction output directly + out_tensors = [] + for out in prediction_output: + # Add batch and channel dimensions + out_tensors.append(torch.from_numpy(np.expand_dims(np.expand_dims(out, 0), 0))) + # Concatenate along batch dimension + out_tensor = torch.cat(out_tensors, 0) + + return MetaTensor(out_tensor, meta=x.meta) + else: + # Return a placeholder tensor with file path in metadata + if outfile is None: + raise ValueError("Output file path is None when save_files is True") + saved_path = outfile + ".npz" + if not os.path.exists(saved_path): + raise FileNotFoundError(f"Expected saved file not found: {saved_path}") + + # Create placeholder tensor with same spatial dimensions + shape = properties_or_list_of_properties["spatial_shape"] + dummy_tensor = torch.zeros((1, 1, *shape), dtype=torch.float32) + + # Create metadata with file path + meta_with_filepath = dict(x.meta) + meta_with_filepath["saved_file"] = saved_path + + return MetaTensor(dummy_tensor, meta=meta_with_filepath) diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/nnunet_seg_operator.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/nnunet_seg_operator.py new file mode 100644 index 00000000..4d000dd1 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/nnunet_seg_operator.py @@ -0,0 +1,421 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from pathlib import Path +from typing import Dict, List, Optional + +import torch +from numpy import int16, uint8 + +# Import custom transforms +from post_transforms import CalculateVolumeFromMaskd, ExtractVolumeToTextd, LabelToContourd, OverlayImageLabeld + +# Import from MONAI deploy +from monai.deploy.utils.importutil import optional_import + +Dataset, _ = optional_import("monai.data", name="Dataset") +DataLoader, _ = optional_import("monai.data", name="DataLoader") +import os + +# Try importing from local version first, then fall back to MONAI if not available +# This approach works regardless of how the file is imported (as module or script) +import sys + +# Add current directory to path to ensure the local module is found +current_dir = os.path.dirname(os.path.abspath(__file__)) +if current_dir not in sys.path: + sys.path.insert(0, current_dir) + +# Import from local nnunet_bundle module +from nnunet_bundle import EnsembleProbabilitiesToSegmentation, get_nnunet_monai_predictors_for_ensemble + +from monai.deploy.core import AppContext, Fragment, Model, Operator, OperatorSpec +from monai.deploy.operators.monai_seg_inference_operator import InMemImageReader + +# Import MONAI transforms +from monai.transforms import Compose, KeepLargestConnectedComponentd, Lambdad, LoadImaged, SaveImaged, Transposed + +DEFAULT_OUTPUT_FOLDER = Path.cwd() / "output" + + +class NNUnetSegOperator(Operator): + """ + Operator that performs segmentation inference with nnU-Net ensemble models. + + This operator loads and runs multiple nnU-Net models in an ensemble fashion, + processes the results, and outputs segmentation masks, volume measurements, + and visualization overlays. + """ + + def __init__( + self, + fragment: Fragment, + *args, + app_context: AppContext, + model_path: Path, + output_folder: Path = DEFAULT_OUTPUT_FOLDER, + output_labels: Optional[List[int]] = None, + model_list: Optional[List[str]] = None, + model_name: str = "best_model.pt", + save_probabilities: bool = False, + save_files: bool = False, + **kwargs, + ): + """ + Initialize the nnU-Net segmentation operator. + + Args: + fragment: The fragment this operator belongs to + app_context: The application context + model_path: Path to the nnU-Net model directory + output_folder: Directory to save output files + output_labels: List of label indices to include in outputs + model_list: List of nnU-Net model types to use in ensemble + model_name: Name of the model checkpoint file + save_probabilities: Whether to save probability maps + save_files: Whether to save intermediate files + """ + # Initialize logger + self._logger = logging.getLogger(f"{__name__}.{type(self).__name__}") + + # Set up data keys + self._input_dataset_key = "image" + self._pred_dataset_key = "pred" + + # Model configuration + self.model_path = self._find_model_file_path(model_path) + self.model_list = model_list or ["3d_fullres", "3d_lowres", "3d_cascade_fullres"] + self.model_name = model_name + self.save_probabilities = save_probabilities + self.save_files = save_files + self.prediction_keys = [f"pred_{model}" for model in self.model_list] + + # Output configuration + self.output_folder = output_folder if output_folder is not None else DEFAULT_OUTPUT_FOLDER + self.output_folder.mkdir(parents=True, exist_ok=True) + self.output_labels = output_labels if output_labels is not None else [1] + + # Store app context + self.app_context = app_context + + # I/O names for operator + self.input_name_image = "image" + self.output_name_seg = "seg_image" + self.output_name_text = "result_text" + self.output_name_sc_path = "dicom_sc_dir" + + # Call parent constructor + super().__init__(fragment, *args, **kwargs) + + def _find_model_file_path(self, model_path: Path) -> Path: + """ + Validates and returns the model directory path. + + Args: + model_path: Path to the model directory + + Returns: + Validated Path object to the model directory + + Raises: + ValueError: If model_path is invalid or doesn't exist + """ + # When executing as MAP, model_path is typically a directory (/opt/holoscan/models) + # nnU-Net expects a directory structure with model subdirectories + if not model_path: + raise ValueError("Model path not provided") + + if not model_path.is_dir(): + raise ValueError(f"Model path should be a directory, got: {model_path}") + + return model_path + + def _load_nnunet_models(self): + """ + Loads nnU-Net ensemble models using MONAI's nnU-Net bundle functionality + and registers them in the app_context. + + Raises: + RuntimeError: If model loading fails + """ + # Determine device based on availability + _device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + self._logger.info(f"Loading nnU-Net ensemble models from: {self.model_path} on {_device}") + + try: + # Get nnU-Net ensemble predictors (returns tuple of ModelnnUNetWrapper objects) + network_def = get_nnunet_monai_predictors_for_ensemble( + model_list=self.model_list, model_path=str(self.model_path), model_name=self.model_name + ) + + # Move models to device and set to evaluation mode + ensemble_predictors = [] + for predictor in network_def: + predictor.to(_device) + predictor.eval() + ensemble_predictors.append(predictor) + + # Create a MONAI Model object to encapsulate the ensemble + loaded_model = Model(self.model_path, name="nnunet_ensemble") + loaded_model.predictor = ensemble_predictors + + # Register the loaded Model object in the application context + self.app_context.models = loaded_model + + self._logger.info(f"Successfully loaded {len(ensemble_predictors)} nnU-Net models: {self.model_list}") + + except Exception as e: + self._logger.error(f"Failed to load nnU-Net models: {str(e)}") + raise + + def setup(self, spec: OperatorSpec): + """ + Sets up the operator by configuring input and output specifications. + + Args: + spec: The operator specification to configure + """ + # Define input - expects a DICOM image + spec.input(self.input_name_image) + + # Define outputs: + # 1. Segmentation output (for DICOM SEG) + spec.output(self.output_name_seg) + + # 2. Measurement results text (for DICOM SR) + spec.output(self.output_name_text) + + # 3. Directory path for visualization overlays (for DICOM SC) + spec.output(self.output_name_sc_path) + + def _convert_dicom_metadata_datatype(self, metadata: Dict) -> Dict: + """ + Converts pydicom-specific metadata types to corresponding native Python types. + + This addresses an issue with pydicom types in metadata for images converted from DICOM series. + Reference issue: https://github.com/Project-MONAI/monai-deploy-app-sdk/issues/185 + + Args: + metadata: Dictionary containing image metadata + + Returns: + Dictionary with converted metadata types + """ + if not metadata: + return metadata + + # Convert known metadata attributes to appropriate Python types + known_conversions = {"SeriesInstanceUID": str, "row_pixel_spacing": float, "col_pixel_spacing": float} + + for key, conversion_func in known_conversions.items(): + if key in metadata: + try: + metadata[key] = conversion_func(metadata[key]) + except Exception: + self._logger.warning(f"Failed to convert {key} to {conversion_func.__name__}") + + # Log converted metadata at debug level + if self._logger.isEnabledFor(logging.DEBUG): + self._logger.debug("Converted Image object metadata:") + for k, v in metadata.items(): + self._logger.debug(f"{k}: {v}, type {type(v)}") + + return metadata + + def compute(self, op_input, op_output, context): + """ + Main compute method that processes input, runs inference, and emits outputs. + """ + # Get input image + input_image = op_input.receive(self.input_name_image) + if not input_image: + raise ValueError("Input image is not found.") + + # Load nnU-Net ensemble models + self._logger.info("Loading nnU-Net ensemble models") + self._load_nnunet_models() + + # Perform inference using our custom implementation + data_dict = self.compute_impl(input_image, context)[0] + + # Squeeze the batch dimension + data_dict[self._pred_dataset_key] = data_dict[self._pred_dataset_key].squeeze(0) + data_dict[self._input_dataset_key] = data_dict[self._input_dataset_key].squeeze(0) + + # Squeeze the batch dimension of affine meta data + data_dict[self._pred_dataset_key].affine = data_dict[self._pred_dataset_key].affine.squeeze(0) + data_dict[self._input_dataset_key].affine = data_dict[self._input_dataset_key].affine.squeeze(0) + + # Log shape information + self._logger.info(f"Segmentation prediction shape: {data_dict[self._pred_dataset_key].shape}") + self._logger.info(f"Segmentation image shape: {data_dict[self._input_dataset_key].shape}") + + # Get post transforms for MAP outputs + post_transforms = self.post_process_stage2() + + # Apply postprocessing transforms for MAP outputs + data_dict = post_transforms(data_dict) + + self._logger.info( + f"Segmentation prediction shape after post processing: {data_dict[self._pred_dataset_key].shape}" + ) + + # DICOM SEG output + op_output.emit(data_dict[self._pred_dataset_key].squeeze(0).numpy().astype(uint8), self.output_name_seg) + + # DICOM SR output - extract result text + result_text = self.get_result_text_from_transforms(post_transforms) + if not result_text: + raise ValueError("Result text could not be generated.") + + self._logger.info(f"Calculated Organ Volumes: {result_text}") + op_output.emit(result_text, self.output_name_text) + + # DICOM SC output + dicom_sc_dir = self.output_folder / "temp" + self._logger.info(f"Temporary DICOM SC saved at: {dicom_sc_dir}") + op_output.emit(dicom_sc_dir, self.output_name_sc_path) + + def pre_process(self, img_reader) -> Compose: + """Composes transforms for preprocessing the input image before predicting on nnU-Net models.""" + my_key = self._input_dataset_key + + return Compose( + [ + LoadImaged(keys=my_key, reader=img_reader, ensure_channel_first=True), + Transposed(keys=my_key, indices=[0, 3, 2, 1]), + ] + ) + + def compute_impl(self, input_image, context) -> List[Dict]: + """ + Performs the actual nnU-Net ensemble inference using ModelnnUNetWrapper. + This function handles the complete inference pipeline including preprocessing, + ensemble prediction, and postprocessing. + """ + + if not input_image: + raise ValueError("Input is None.") + + # Need to try to convert the data type of a few metadata attributes. + # input_img_metadata = self._convert_dicom_metadata_datatype(input_image.metadata()) + # Need to give a name to the image as in-mem Image obj has no name. + img_name = "Img_in_context" + + # This operator gets an in-memory Image object, so a specialized ImageReader is needed. + _reader = InMemImageReader(input_image) + + # Apply preprocessing transforms + pre_transforms = self.pre_process(_reader) + + # Create data dictionary + data_dict = {self._input_dataset_key: img_name} + + # Create dataset and dataloader + dataset = Dataset(data=[data_dict], transform=pre_transforms) + dataloader = DataLoader(dataset, batch_size=1, shuffle=False, num_workers=0) + + out_dict = [] + for d in dataloader: + preprocessed_image = d[self._input_dataset_key] + self._logger.info(f"Input shape: {preprocessed_image.shape}") + + # Get the loaded ensemble models from app context + if not hasattr(self.app_context, "models") or self.app_context.models is None: + raise RuntimeError("nnU-Net models not loaded. Call _load_nnunet_models first.") + + ensemble_predictors = self.app_context.models.predictor + + # Perform ensemble inference + self._logger.info("Running nnU-Net ensemble inference...") + + for i, predictor in enumerate(ensemble_predictors): + model_key = self.prediction_keys[i] + self._logger.info(f"Running inference with model: {model_key}") + + # Run inference with individual model + prediction = predictor(preprocessed_image) + d[model_key] = prediction + + self._logger.info("Inference complete, applying postprocessing...") + + # Apply postprocessing transforms (includes ensemble combination) + post_transforms1 = self.post_process_stage1() + d = post_transforms1(d) + out_dict.append(d) + return out_dict + + def post_process_stage1(self) -> Compose: + """Composes transforms for postprocessing the nnU-Net prediction results.""" + pred_key = self._pred_dataset_key + return Compose( + [ + # nnU-Net ensemble post-processing + EnsembleProbabilitiesToSegmentation( + keys=self.prediction_keys, + dataset_json_path=str(self.model_path / "jsonpkls/dataset.json"), + plans_json_path=str(self.model_path / "jsonpkls/plans.json"), + output_key=pred_key, + ), + # Add batch dimension to final prediction + Lambdad(keys=[pred_key], func=lambda x: x.unsqueeze(0)), + # Transpose dimensions back to original format + Transposed(keys=[self._input_dataset_key, pred_key], indices=(0, 1, 4, 3, 2)), + ] + ) + + def post_process_stage2(self) -> Compose: + """Composes transforms for postprocessing MAP outputs""" + pred_key = self._pred_dataset_key + + # Define labels for the segmentation output + labels = {"background": 0, "airway": 1} + + return Compose( + [ + # Keep only largest connected component for each label + KeepLargestConnectedComponentd(keys=pred_key, applied_labels=[1]), + # Calculate volume from segmentation mask + CalculateVolumeFromMaskd(keys=pred_key, label_names=labels), + # Extract volume data to text format + ExtractVolumeToTextd( + keys=[pred_key + "_volumes"], label_names=labels, output_labels=self.output_labels + ), + # Convert labels to contours + LabelToContourd(keys=pred_key, output_labels=self.output_labels), + # Create overlay of image and contours + OverlayImageLabeld(image_key=self._input_dataset_key, label_key=pred_key, overlay_key="overlay"), + # Save overlays as DICOM SC + SaveImaged( + keys="overlay", + output_ext=".dcm", + output_dir=self.output_folder / "temp", + separate_folder=False, + output_dtype=int16, + ), + ] + ) + + def get_result_text_from_transforms(self, post_transforms: Compose) -> str: + """ + Extracts result_text from the ExtractVolumeToTextd transform in the transform pipeline. + + Args: + post_transforms: Composed transforms that include ExtractVolumeToTextd + + Returns: + The extracted result text or empty string if not found + """ + for transform in post_transforms.transforms: + if isinstance(transform, ExtractVolumeToTextd): + return str(transform.result_text) # Ensure return type is str + return "" diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/post_transforms.py b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/post_transforms.py new file mode 100644 index 00000000..a53de889 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/post_transforms.py @@ -0,0 +1,390 @@ +# Copyright 2021-2025 MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json +import logging +import os +from typing import List, Optional + +import matplotlib.cm as cm +import numpy as np + +from monai.config import KeysCollection +from monai.data import MetaTensor +from monai.transforms import LabelToContour, MapTransform + + +# Calculate segmentation volumes in ml +class CalculateVolumeFromMaskd(MapTransform): + """ + Dictionary-based transform to calculate the volume of predicted organ masks. + + Args: + keys (list): The keys corresponding to the predicted organ masks in the dictionary. + label_names (list): The list of organ names corresponding to the masks. + """ + + def __init__(self, keys, label_names): + self._logger = logging.getLogger(f"{__name__}.{type(self).__name__}") + super().__init__(keys) + self.label_names = label_names + + def __call__(self, data): + # Initialize a dictionary to store the volumes of each organ + pred_volumes = {} + + for key in self.keys: + for label_name in self.label_names.keys(): + # self._logger.info('Key: ', key, ' organ_name: ', label_name) + if label_name != "background": + # Get the predicted mask from the dictionary + pred_mask = data[key] + # Calculate the voxel size in cubic millimeters (voxel size should be in the metadata) + # Assuming the metadata contains 'spatial_shape' with voxel dimensions in mm + if hasattr(pred_mask, "affine"): + # voxel size + # Ensure the affine matrix is collapsed to shape (4, 4) + affine_matrix = np.squeeze(pred_mask.affine) + if affine_matrix.shape != (4, 4): + raise ValueError(f"Affine matrix must have shape (4, 4), but got {affine_matrix.shape}") + + # Calculate voxel size + voxel_size = np.abs(np.linalg.det(affine_matrix[:3, :3])) + # print(f"Voxel Size (mm³): {voxel_size}") + else: + raise ValueError("Affine transformation matrix with voxel spacing information is required.") + + # Calculate the volume in cubic millimeters + label_volume_mm3 = np.sum(pred_mask == self.label_names[label_name]) * voxel_size + + # Convert to milliliters (1 ml = 1000 mm^3) + label_volume_ml = label_volume_mm3 / 1000.0 + + # Store the result in the pred_volumes dictionary + # convert to int - radiologists prefer whole number with no decimals + pred_volumes[label_name] = int(round(label_volume_ml, 2)) + + # Add the calculated volumes to the data dictionary + key_name = key + "_volumes" + + data[key_name] = pred_volumes + # self._logger.info('pred_volumes: ', pred_volumes) + return data + + +class LabelToContourd(MapTransform): + def __init__(self, keys: KeysCollection, output_labels: list, allow_missing_keys: bool = False): + + self._logger = logging.getLogger(f"{__name__}.{type(self).__name__}") + super().__init__(keys, allow_missing_keys) + + self.output_labels = output_labels + + def __call__(self, data): + d = dict(data) + for key in self.keys: + label_image = d[key] + assert isinstance(label_image, MetaTensor), "Input image must be a MetaTensor." + + # Initialize the contour image with the same shape as the label image + contour_image = np.zeros_like(label_image.cpu().numpy()) + + if label_image.ndim == 4: # Check if the label image is 4D with a channel dimension + # Process each 2D slice independently along the last axis (z-axis) + for i in range(label_image.shape[-1]): + slice_image = label_image[:, :, :, i].cpu().numpy() + + # Extract unique labels excluding background (assumed to be 0) + unique_labels = np.unique(slice_image) + unique_labels = unique_labels[unique_labels != 0] + + slice_contour = np.zeros_like(slice_image) + + # Generate contours for each label in the slice + for label in unique_labels: + # skip contour generation for labels that are not in output_labels + if label not in self.output_labels: + continue + + # Create a binary mask for the current label + binary_mask = np.zeros_like(slice_image) + binary_mask[slice_image == label] = 1.0 + + # Apply LabelToContour to the 2D slice (replace this with actual contour logic) + binary_mask = binary_mask.astype(np.float32) # Convert to float32 for LabelToContour + thick_edges = LabelToContour()(binary_mask) + + # Convert the edges back to binary mask + thick_edges = (thick_edges > 0).astype(np.uint8) + + # Assign the label value to the contour image at the edge positions + slice_contour[thick_edges > 0] = label + + # Stack the processed slice back into the 4D contour image + contour_image[:, :, :, i] = slice_contour + else: + # If the label image is not 4D, process it directly + slice_image = label_image.cpu().numpy() + unique_labels = np.unique(slice_image) + unique_labels = unique_labels[unique_labels != 0] + + for label in unique_labels: + binary_mask = np.zeros_like(slice_image) + binary_mask[slice_image == label] = 1.0 + + thick_edges = LabelToContour()(binary_mask) + contour_image[thick_edges > 0] = label + + # Convert the contour image back to a MetaTensor with the original metadata + contour_image_meta = MetaTensor(contour_image, meta=label_image.meta) # , affine=label_image.affine) + + # Store the contour MetaTensor in the output dictionary + d[key] = contour_image_meta + + return d + + +class OverlayImageLabeld(MapTransform): + def __init__( + self, + image_key: KeysCollection, + label_key: str, + overlay_key: str = "overlay", + alpha: float = 0.7, + allow_missing_keys: bool = False, + ): + + self._logger = logging.getLogger(f"{__name__}.{type(self).__name__}") + super().__init__(image_key, allow_missing_keys) + + self.image_key = image_key + self.label_key = label_key + self.overlay_key = overlay_key + self.alpha = alpha + self.jet_colormap = cm.get_cmap("jet", 256) # Get the Jet colormap with 256 discrete colors + + def apply_jet_colormap(self, label_volume): + """ + Apply the Jet colormap to a 3D label volume using matplotlib's colormap. + """ + assert label_volume.ndim == 3, "Label volume should have 3 dimensions (H, W, D) after removing channel." + + label_volume_normalized = (label_volume / label_volume.max()) * 255.0 + label_volume_uint8 = label_volume_normalized.astype(np.uint8) + + # Apply the colormap to each label + label_rgb = self.jet_colormap(label_volume_uint8)[:, :, :, :3] # Only take the RGB channels + + label_rgb = (label_rgb * 255).astype(np.uint8) + # Rearrange axes to get (3, H, W, D) + label_rgb = np.transpose(label_rgb, (3, 0, 1, 2)) + + assert label_rgb.shape == ( + 3, + *label_volume.shape, + ), f"Label RGB shape should be (3,H, W, D) but got {label_rgb.shape}" + + return label_rgb + + def convert_to_rgb(self, image_volume): + """ + Convert a single-channel grayscale 3D image to an RGB 3D image. + """ + assert image_volume.ndim == 3, "Image volume should have 3 dimensions (H, W, D) after removing channel." + + image_volume_normalized = (image_volume - image_volume.min()) / (image_volume.max() - image_volume.min()) + image_rgb = np.stack([image_volume_normalized] * 3, axis=0) + image_rgb = (image_rgb * 255).astype(np.uint8) + + assert image_rgb.shape == ( + 3, + *image_volume.shape, + ), f"Image RGB shape should be (3,H, W, D) but got {image_rgb.shape}" + + return image_rgb + + def _create_overlay(self, image_volume, label_volume): + # Convert the image volume and label volume to RGB + image_rgb = self.convert_to_rgb(image_volume) + label_rgb = self.apply_jet_colormap(label_volume) + + # Create an alpha-blended overlay + overlay = image_rgb.copy() + mask = label_volume > 0 + + # Apply the overlay where the mask is present + for i in range(3): # For each color channel + overlay[i, mask] = (self.alpha * label_rgb[i, mask] + (1 - self.alpha) * overlay[i, mask]).astype(np.uint8) + + assert ( + overlay.shape == image_rgb.shape + ), f"Overlay shape should match image RGB shape: {overlay.shape} vs {image_rgb.shape}" + + return overlay + + def __call__(self, data): + d = dict(data) + + # Get the image and label tensors + image = d[self.image_key] # Expecting shape (1, H, W, D) + label = d[self.label_key] # Expecting shape (1, H, W, D) + + # # uncomment when running pipeline with mask (non-contour) outputs, i.e. LabelToContourd transform absent + # if image.device.type == "cuda": + # image = image.cpu() + # d[self.image_key] = image + # if label.device.type == "cuda": + # label = label.cpu() + # d[self.label_key] = label + # # ----------------------- + + # Ensure that the input has the correct dimensions + assert image.shape[0] == 1 and label.shape[0] == 1, "Image and label must have a channel dimension of 1." + assert image.shape == label.shape, f"Image and label must have the same shape: {image.shape} vs {label.shape}" + + # Remove the channel dimension for processing + image_volume = image[0] # Shape: (H, W, D) + label_volume = label[0] # Shape: (H, W, D) + + # Convert to 3D overlay + overlay = self._create_overlay(image_volume, label_volume) + + # Add the channel dimension back + # d[self.overlay_key] = np.expand_dims(overlay, axis=0) # Shape: (1, H, W, D, 3) + d[self.overlay_key] = MetaTensor(overlay, meta=label.meta, affine=label.affine) # Shape: (3, H, W, D) + + # Assert the final output shape + # assert d[self.overlay_key].shape == (1, *image_volume.shape, 3), \ + # f"Final overlay shape should be (1, H, W, D, 3) but got {d[self.overlay_key].shape}" + + assert d[self.overlay_key].shape == ( + 3, + *image_volume.shape, + ), f"Final overlay shape should be (3, H, W, D) but got {d[self.overlay_key].shape}" + + # Log the overlay creation (debugging) + self._logger.info(f"Overlay created with shape: {overlay.shape}") + # self._logger.info(f"Dictionary keys: {d.keys()}") + + # self._logger.info('overlay_image shape: ', d[self.overlay_key].shape) + return d + + +class SaveData(MapTransform): + """ + Save the output dictionary into JSON files. + + The name of the saved file will be `{key}_{output_postfix}.json`. + + Args: + keys: keys of the corresponding items to be saved in the dictionary. + output_dir: directory to save the output files. + output_postfix: a string appended to all output file names, default is `data`. + separate_folder: whether to save each file in a separate folder. Default is `True`. + print_log: whether to print logs when saving. Default is `True`. + """ + + def __init__( + self, + keys: KeysCollection, + namekey: str = "image", + output_dir: str = "./", + output_postfix: str = "data", + separate_folder: bool = False, + print_log: bool = True, + allow_missing_keys: bool = False, + ): + self._logger = logging.getLogger(f"{__name__}.{type(self).__name__}") + super().__init__(keys, allow_missing_keys) + self.output_dir = output_dir + self.output_postfix = output_postfix + self.separate_folder = separate_folder + self.print_log = print_log + self.namekey = namekey + + def __call__(self, data): + d = dict(data) + image_name = os.path.basename(d[self.namekey].meta["filename_or_obj"]).split(".")[0] + for key in self.keys: + # Get the data + output_data = d[key] + + # Determine the file name + file_name = f"{image_name}_{self.output_postfix}.json" + if self.separate_folder: + file_path = os.path.join(self.output_dir, image_name, file_name) + os.makedirs(os.path.dirname(file_path), exist_ok=True) + else: + file_path = os.path.join(self.output_dir, file_name) + + # Save the dictionary as a JSON file + with open(file_path, "w") as f: + json.dump(output_data, f) + + if self.print_log: + self._logger.info(f"Saved data to {file_path}") + + return d + + +# custom transform (not in original post_transforms.py in bundle): +class ExtractVolumeToTextd(MapTransform): + """ + Custom transform to extract volume information from the segmentation results and format it as a textual summary. + Filters organ volumes based on output_labels for DICOM SR write, while including all organs for MongoDB write. + The upstream CalculateVolumeFromMaskd transform calculates organ volumes and stores them in the dictionary + under the pred_key + '_volumes' key. The input dictionary is outputted unchanged as to not affect downstream operators. + + Args: + keys: keys of the corresponding items to be saved in the dictionary. + label_names: dictionary mapping organ names to their corresponding label indices. + output_labels: list of target label indices for organs to include in the DICOM SR output. + """ + + def __init__( + self, + keys: KeysCollection, + label_names: dict, + output_labels: List[int], + allow_missing_keys: bool = False, + ): + self._logger = logging.getLogger(f"{__name__}.{type(self).__name__}") + super().__init__(keys, allow_missing_keys) + + self.label_names = label_names + self.output_labels = output_labels + + self.result_text: Optional[str] = None + + def __call__(self, data): + d = dict(data) + # use the first key in `keys` to access the volume data (e.g., pred_key + '_volumes') + volumes_key = self.keys[0] + organ_volumes = d.get(volumes_key, None) + + if organ_volumes is None: + raise ValueError(f"Volume data not found for key {volumes_key}.") + + # create the volume text output + volume_text = [] + + # loop through calculated organ volumes + for organ, volume in organ_volumes.items(): + # if the organ's label index is in output_labels + label_index = self.label_names.get(organ, None) + if label_index in self.output_labels: + # append organ volume for DICOM SR entry + volume_text.append(f"{organ.capitalize()} Volume: {volume} mL") + + self.result_text = "\n".join(volume_text) + + # not adding result_text to dictionary; return dictionary unchanged as to not affect downstream operators + return d diff --git a/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/requirements.txt b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/requirements.txt new file mode 100644 index 00000000..d3035e41 --- /dev/null +++ b/examples/apps/cchmc_nnunet_fifteen_ckpt_app/my_app/requirements.txt @@ -0,0 +1,37 @@ +# requirements.txt file specifies dependencies our Python project needs to run + +# install MONAI and necessary image processing packages (base list pulled from MONAI Bundle Spleen Seg App example) +# based on CCHMC Ped Abd MRI MONAI Bundle dependencies: +# monai, numpy, nibabel versions upgraded +# pytorch-ignite and fire dependencies added +# python 3.9 is required to install specified pytorch and monai-deploy-app-sdk versions +# einops optional dependency needed for DAE model workflow +monai[einops]==1.3.0 +torch>=1.12.0 +pytorch-ignite==0.4.11 +fire==0.4.0 +numpy>=1.24,<2.0 +nibabel==4.0.1 +# pydicom v3.0.0 removed pydicom._storage_sopclass_uids; don't meet or exceed this version +pydicom>=2.3.0,<3.0.0 +# pylibjpeg for processing compressed DICOM pixel data +pylibjpeg[all] +highdicom>=0.18.2 +itk>=5.3.0 +SimpleITK>=2.0.0 +scikit-image>=0.17.2 +Pillow>=8.0.0 +numpy-stl>=2.12.0 +trimesh>=3.8.11 +matplotlib>=3.7.2 +setuptools>=75.8.0 # for pkg_resources + +# MONAI Deploy App SDK package installation +# includes Holoscan SDK and CLI ~=3.0 +monai-deploy-app-sdk==3.0.0 + +# fine control over holoscan and holoscan-cli versions +holoscan==3.2.0 +holoscan-cli==3.2.0 +nvflare>=2.6.2,<3.0.0 +nnunetv2>=2.6.2,<3.0.0 \ No newline at end of file diff --git a/examples/apps/convert_nnunet_ckpts.py b/examples/apps/convert_nnunet_ckpts.py new file mode 100644 index 00000000..e1691e89 --- /dev/null +++ b/examples/apps/convert_nnunet_ckpts.py @@ -0,0 +1,103 @@ +# Copyright (c) MONAI Consortium +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Convert nnUNet checkpoints to MONAI bundle format. +This script follows the logic in the conversion notebook but imports from local apps.nnunet_bundle. +""" + +import argparse +import os +import sys + +# Add the current directory to the path to find the local module +current_dir = os.path.dirname(os.path.abspath(__file__)) +if current_dir not in sys.path: + sys.path.insert(0, current_dir) + +# Try importing from local apps.nnunet_bundle instead of from MONAI +try: + from my_app.nnunet_bundle import convert_best_nnunet_to_monai_bundle +except ImportError: + # If local import fails, try to find the module in alternate locations + try: + from monai.apps.nnunet_bundle import convert_best_nnunet_to_monai_bundle + except ImportError: + print( + "Error: Could not import convert_best_nnunet_to_monai_bundle from my_app.nnunet_bundle or apps.nnunet_bundle" + ) + print("Please ensure that nnunet_bundle.py is properly installed in your project.") + sys.exit(1) + + +def parse_args(): + parser = argparse.ArgumentParser(description="Convert nnUNet checkpoints to MONAI bundle format.") + parser.add_argument( + "--dataset_name_or_id", type=str, required=True, help="The name or ID of the dataset to convert." + ) + parser.add_argument( + "--MAP_root", + type=str, + default=os.getcwd(), + help="The root directory where the Medical Application Package (MAP) will be created. Defaults to current directory.", + ) + + parser.add_argument( + "--nnUNet_results", + type=str, + required=False, + default=None, + help="Path to nnUNet results directory with trained models.", + ) + return parser.parse_args() + + +def main(): + args = parse_args() + + # Create the nnUNet config dictionary + nnunet_config = { + "dataset_name_or_id": args.dataset_name_or_id, + } + + # Create the MAP root directory + map_root = args.MAP_root + os.makedirs(map_root, exist_ok=True) + + # Set nnUNet environment variables if provided + if args.nnUNet_results: + os.environ["nnUNet_results"] = args.nnUNet_results + print(f"Set nnUNet_results to: {args.nnUNet_results}") + + # Check if required environment variables are set + required_env_vars = ["nnUNet_results"] + missing_vars = [var for var in required_env_vars if var not in os.environ] + + if missing_vars: + print(f"Error: The following required nnUNet environment variables are not set: {', '.join(missing_vars)}") + print("Please provide them as arguments or set them in your environment before running this script.") + sys.exit(1) + + print(f"Converting nnUNet checkpoints for dataset {nnunet_config['dataset_name_or_id']} to MONAI bundle format...") + print(f"MAP will be created at: {map_root}") + print(f" nnUNet_results: {os.environ.get('nnUNet_results')}") + + # Convert the nnUNet checkpoints to MONAI bundle format + try: + convert_best_nnunet_to_monai_bundle(nnunet_config, map_root) + print(f"Successfully converted nnUNet checkpoints to MONAI bundle at: {map_root}/models") + except Exception as e: + print(f"Error converting nnUNet checkpoints: {e}") + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/examples/apps/development_notes.md b/examples/apps/development_notes.md new file mode 100644 index 00000000..f1e91b99 --- /dev/null +++ b/examples/apps/development_notes.md @@ -0,0 +1,62 @@ +# Development Notes + +## Implementation Notes for nnUNet MAP + + +* Initial Tests show volume and Dice agreement with Bundle, need to do more thorough testing. + +1. For each model configuration the output gets written to .npz file by nnunet inference functions. + +2. These file paths are then used by the EnsembleProbabilities Transform function to create the final output. + +3. If nnunet postprocessing is used, use the largest connected component transform in the MAP. There could be minor differences in the implementation, will do thorough analysis later. + +3. Need to better understand the use of "context" in compute and compute_impl as input arguments. + +4. Investigate keeping the probabilities in the memory, to help with speedup. + +5. Need to investigate the current traceability provisions in the operators implemented. + + +## Implementation Details + +### Testing Strategy + +Tests should be conducted to: +1. Compare MAP output with native nnUNet output +2. Measure performance (time, memory usage) +3. Validate with various input formats and sizes +4. Test error handling and edge cases + + +### nnUNet Integration + +The current implementation relies on the nnUNet's native inference approach which outputs intermediate .npz files for each model configuration. While this works, it introduces file I/O overhead which could potentially be optimized. + +### Ensemble Prediction Flow + +1. Multiple nnUNet models (3d_fullres, 3d_lowres, 3d_cascade_fullres) are loaded +2. Each model performs inference separately +3. Results are written to temporary .npz files +4. EnsembleProbabilitiesToSegmentation transform reads these files +5. Final segmentation is produced by combining results + +### Potential Optimizations + +- Keep probability maps in memory instead of writing to disk +- Parallelize model inference where applicable +- Streamline the ensemble computation process + +### Context Usage + +The `context` parameter in `compute` and `compute_impl` functions appears to be used for storing and retrieving models. Further investigation is needed to fully understand how this context is managed and whether it's being used optimally. + +### Traceability + +Current traceability in the operators may need improvement. Consider adding: + +- More detailed logging +- Performance metrics +- Input/output validation steps +- Error handling with informative messages + diff --git a/monai/deploy/operators/dicom_seg_writer_operator.py b/monai/deploy/operators/dicom_seg_writer_operator.py index e96490c1..21b3a51f 100644 --- a/monai/deploy/operators/dicom_seg_writer_operator.py +++ b/monai/deploy/operators/dicom_seg_writer_operator.py @@ -341,8 +341,8 @@ def create_dicom_seg(self, image: np.ndarray, dicom_series: DICOMSeries, output_ # Adding a few tags that are not in the Dataset # Also try to set the custom tags that are of string type dt_now = datetime.datetime.now() - seg.SeriesDate = dt_now.strftime("%Y%m%d") - seg.SeriesTime = dt_now.strftime("%H%M%S") + seg.SeriesDate = dt_now.strftime("%Y%m%d") # type: ignore[assignment] + seg.SeriesTime = dt_now.strftime("%H%M%S") # type: ignore[assignment] seg.TimezoneOffsetFromUTC = ( dt_now.astimezone().isoformat()[-6:].replace(":", "") ) # '2022-09-27T22:36:20.143857-07:00'