Merge branch 'main' into pstjohn/changed-files

Author: pstjohn

.github/pull_request_template.md

@@ -1,45 +1,51 @@
 ### Description
+
 <!-- Provide a detailed description of the changes in this PR -->
 
 ### Type of changes
+
 <!-- Mark the relevant option with an [x] -->
 
-- [ ]  Bug fix (non-breaking change which fixes an issue)
-- [ ]  New feature (non-breaking change which adds functionality)
-- [ ]  Refactor
-- [ ]  Documentation update
-- [ ]  Other (please describe):
+- [ ] Bug fix (non-breaking change which fixes an issue)
+- [ ] New feature (non-breaking change which adds functionality)
+- [ ] Refactor
+- [ ] Documentation update
+- [ ] Other (please describe):
 
 ### CI Pipeline Configuration
+
 Configure CI behavior by applying the relevant labels:
 
 - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests
 - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest
 - [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing
 
-> [!NOTE]
+> \[!NOTE\]
 > By default, the notebooks validation tests are skipped unless explicitly enabled.
 
 #### Authorizing CI Runs
 
 We use [copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation) to manage authorization of CI
 runs on NVIDIA's compute resources.
 
-* If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
+- If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
   automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
-* If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
+- If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
   `/ok to test` comment on the pull request to trigger CI. This will need to be done for each new commit.
 
 ### Usage
+
 <!--- How does a user interact with the changed code -->
+
 ```python
-TODO: Add code snippet
+# TODO: Add code snippet
 ```
 
 ### Pre-submit Checklist
+
 <!--- Ensure all items are completed before submitting -->
 
- - [ ] I have tested these changes locally
- - [ ] I have updated the documentation accordingly
- - [ ] I have added/updated tests as needed
- - [ ] All existing tests pass successfully
+- [ ] I have tested these changes locally
+- [ ] I have updated the documentation accordingly
+- [ ] I have added/updated tests as needed
+- [ ] All existing tests pass successfully

.github/workflows/internal_tools.yml

@@ -0,0 +1 @@
+number = true        # options: {false, true}

.mdformat.toml

@@ -7,12 +7,21 @@ repos:
       - id: check-yaml
         exclude: "mkdocs.yml"
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.10
+    rev: v0.12.8
     hooks:
       - id: ruff
         # 1. Attempt to automatically fix any lint issues.
         args: ["--fix"]
       - id: ruff-format
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 0.7.17             # Use the latest stable version
+    hooks:
+      - id: mdformat
+        additional_dependencies:
+        - mdformat-tables
+        - mdformat-gfm
+        - mdformat-black
+        - mdformat-frontmatter
   - repo: https://github.com/Yelp/detect-secrets
     rev: v1.5.0
     hooks:
@@ -27,7 +36,7 @@ repos:
     hooks:
       - id: license-header-check
         name: Run license-check script
-        entry: python internal/infra-bionemo/src/infra_bionemo/license_check.py -c scripts -c sub-packages -c docs -c internal --license-header ./license_header --modify
+        entry: python ci/scripts/license_check.py -c sub-packages -c docs -c ci --license-header ./license_header --modify
         language: python
         additional_dependencies: ["click==8.1.7"]
         pass_filenames: false

.pre-commit-config.yaml

@@ -64,7 +64,7 @@ EOF
 ## Drop this when pytorch images ship the fixed commit.
 ARG TE_TAG=9d4e11eaa508383e35b510dc338e58b09c30be73
 
-COPY ./patches/te.patch /tmp/te.patch
+COPY ./docker_build_patches/te.patch /tmp/te.patch
 RUN git clone --recurse-submodules https://github.com/NVIDIA/TransformerEngine.git /tmp/TransformerEngine && \
     cd /tmp/TransformerEngine && \
     git checkout --recurse-submodules ${TE_TAG} && \
@@ -354,7 +354,6 @@ FROM bionemo2-base AS release
 RUN mkdir -p /workspace/bionemo2/.cache/
 
 COPY VERSION .
-COPY ./scripts ./scripts
 COPY ./README.md ./
 # Copy over folders so that the image can run tests in a self-contained fashion.
 COPY ./ci/scripts ./ci/scripts

Dockerfile

@@ -21,7 +21,6 @@ The `bionemo-framework` is organized into independently installable namespace pa
 `sub-packages/` directory. Please refer to [PEP 420 – Implicit Namespace Packages](https://peps.python.org/pep-0420/)
 for details.
 
-
 ## Documentation Resources
 
 - **Official Documentation:** For user guides, API references, and troubleshooting, visit our [official documentation](https://docs.nvidia.com/bionemo-framework/latest/).
@@ -62,7 +61,6 @@ git submodule update --init --recursive
 
 Different branches of the repo can have different pinned versions of these third-party submodules. Ensure submodules are automatically updated after switching branches or pulling updates by configuring git with:
 
-
 ```bash
 git config submodule.recurse true
 ```
@@ -72,21 +70,19 @@ You will have to run the full `git submodule update --init --recursive` command
 
 #### Build the Docker Image Locally
 
-
 With a locally cloned repository and initialized submodules, build the BioNeMo container using:
 
 ```bash
 docker buildx build . -t my-container-tag
 ```
 
-
 #### VSCode Devcontainer for Interactive Debugging
 
 We distribute a [development container](https://devcontainers.github.io/) configuration for vscode
 (`.devcontainer/devcontainer.json`) that simplifies the process of local testing and development. Opening the
 bionemo-framework folder with VSCode should prompt you to re-open the folder inside the devcontainer environment.
 
-> [!NOTE]
+> \[!NOTE\]
 > The first time you launch the devcontainer, it may take a long time to build the image. Building the image locally
 > (using the command shown above) will ensure that most of the layers are present in the local docker cache.
 

README.md

@@ -7,15 +7,16 @@ If you need to report a security issue, please use the appropriate contact point
 ## Reporting Potential Security Vulnerability in an NVIDIA Product
 
 To report a potential security vulnerability in any NVIDIA product:
+
 - Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html)
 - E-Mail: psirt@nvidia.com
-    - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key)
-    - Please include the following information:
-        - Product/Driver name and version/branch that contains the vulnerability
-        - Type of vulnerability (code execution, denial of service, buffer overflow, etc.)
-        - Instructions to reproduce the vulnerability
-        - Proof-of-concept or exploit code
-        - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability
+  - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key)
+  - Please include the following information:
+    - Product/Driver name and version/branch that contains the vulnerability
+    - Type of vulnerability (code execution, denial of service, buffer overflow, etc.)
+    - Instructions to reproduce the vulnerability
+    - Proof-of-concept or exploit code
+    - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability
 
 While NVIDIA currently does not have a bug bounty program, we do offer acknowledgement when an externally reported security issue is addressed under our coordinated vulnerability disclosure policy. Please visit our [Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) policies page for more information.
 

SECURITY.md

@@ -0,0 +1,22 @@
+This is an overview of how to release bionemo sub-packages.
+
+01. The code should be in a sub-directory of `bionemo-framework/sub-packages`. The package should be named bionemo-\<package_name>. For an example of the directory structure, see https://github.com/NVIDIA/bionemo-framework/tree/main/sub-packages/bionemo-scdl.
+    The directory should contain:
+    - a `pyproject.toml` file with the dependencies
+    - a `README.md`
+    - a `LICENSE` file
+    - a `VERSION` file
+    - the source code should be in src/bionemo/package-name.
+    - the test should be in tests/bionemo/package-name. The test directory structure should be the same as the source code directory structure.
+02. Create some tests that can be run in a notebook within the package or as a small python script that verifies that the package is correctly installed. These can be re-purposed for QA test plan.
+03. In the VERSION file in the root of the sub-package, set the package version. Currently, the sub-package versions are independent of the overall BioNeMo version. An ideal approach is to specify the bionemo sub-package versions. That the package depends on. This may create issues. For example, an issue could arise if the latest version of your sub-package depends on the newest bionemo-core, but the latest pushed version of bionemo-core does not have these changes. It may be necessary to update bionemo-core then, but before updating another package, it should be tested and its authors should be consulted.
+04. Make sure that the directory dist doesn’t exist or is empty.
+05. Run `python -m build .`
+06. Create a test-pypi and pypi account if you don’t have one at: https://test.pypi.org/ and https://pypi.org/
+07. Upload to test-pypi with:
+    `twine upload --repository-url https://test.pypi.org/legacy/ dist/* --non-interactive -u $TWINE_USERNAME -p $TWINE_PASSWORD`
+08. In a clean python environment, download the package from test-pypi:
+    `pip install --index-url https://test.pypi.org/simple/ --no-deps package-name`
+09. Run the code/notebooks from step 3.
+10. If everything looks good, upload it to the actual pypi repository: `twine upload  dist/* --non-interactive -u $TWINE_USERNAME -p $TWINE_PASSWORD --verbose`
+11. Run steps 7 and 8 with pypi instead of test-pypi.

ci/Pypi_publish.md

@@ -14,6 +14,24 @@
 # limitations under the License.
 
 
+#! /usr/bin/env python3
+
+# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: LicenseRef-Apache2
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
 import ast
 import sys
 from dataclasses import dataclass