NVIDIA NeMo Evaluator 0.1.74

nemo-evaluator-v0.1.74

ci: Fix integration test by avoid writing to read-only test directory…

NVIDIA NeMo Evaluator Launcher 0.1.75

nemo-evaluator-launcher-v0.1.75

ci: Fix integration test by avoid writing to read-only test directory…

NVIDIA NeMo Evaluator 0.1.73

fix(slurm): node_array undefined (#671)

## Summary

When running the launcher on Slurm with `deployment.type: none`, the
generated sbatch script could fail at runtime with:

- `line N: nodes_array[0]: unbound variable`

This was triggered by `set -u` (nounset) and an unconditional
`--nodelist ${nodes_array[0]}` in the evaluation client `srun`.

## Impact

- **Configs affected**: any Slurm run with `deployment.type=none` (e.g.,
“target-only” evaluation).
- **Failure mode**: sbatch script exits before launching the evaluation
client.
- **Where observed**: Slurm job log (`slurm_script` / `slurm-%A.log`).



## Direct cause

- The sbatch script enables:
  - `set -u` (treat unset variables as an error)
- The evaluation client `srun` was emitted as:
  - `srun ... --nodelist ${nodes_array[0]} ...`
- `nodes_array` was only defined inside the deployment block (`if
cfg.deployment.type != "none": ...`).
- Therefore, for `deployment.type=none`, `nodes_array` was undefined and
`${nodes_array[0]}` crashed under nounset.

## Secondary risks (also addressed)

Even when deployment is enabled, `${nodes_array[0]}` can still fail if:

- `$SLURM_JOB_NODELIST` is unset/empty (non-standard environment) or
only `$SLURM_NODELIST` is present.
- `scontrol` is unavailable on the node or not in `PATH`.
- `scontrol show hostnames ...` returns an empty list.

Any of these can result in an empty/unset array index under `set -u`.

## Solution

### Approach

Introduce a **single, always-defined** “node pinning” variable for
single-node sruns:

- `PRIMARY_NODE`

This is resolved at runtime in the sbatch script with safe fallbacks:

1. Prefer `SLURM_JOB_NODELIST`
2. Fallback to `SLURM_NODELIST`
3. Fallback to local `hostname`

---------

Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>

NVIDIA NeMo Evaluator 0.1.72

fix: restore support for running tasks not listed in FDF (#667)

We have improved our validation in the spirit of failing early. However,
this lead to unwanted side effect - we've lost support for running tasks
not listed in FDF with `harness.task` syntax. Calling evaluation with
this syntax was resulting in
`nemo_evaluator.core.utils.MisconfigurationError: Unknown evaluation
xxx`

It stopped working because:
* we run validation (everything passes here)
* then we prepare the config, extracting `task` from `harness.task` and
using in as evaluation `type`
* we run 2nd validation and it fails because we no longer use
`harness.task` syntax and there's no evaluation called
`task` in FDF

This PR uses `harness.task` as `type` to make sure it's always valid +
adds test verifying custom task support. It also removes one redundant
validation

---------

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

NVIDIA NeMo Evaluator Launcher 0.1.74

fix(slurm): node_array undefined (#671)

## Summary

When running the launcher on Slurm with `deployment.type: none`, the
generated sbatch script could fail at runtime with:

- `line N: nodes_array[0]: unbound variable`

This was triggered by `set -u` (nounset) and an unconditional
`--nodelist ${nodes_array[0]}` in the evaluation client `srun`.

## Impact

- **Configs affected**: any Slurm run with `deployment.type=none` (e.g.,
“target-only” evaluation).
- **Failure mode**: sbatch script exits before launching the evaluation
client.
- **Where observed**: Slurm job log (`slurm_script` / `slurm-%A.log`).



## Direct cause

- The sbatch script enables:
  - `set -u` (treat unset variables as an error)
- The evaluation client `srun` was emitted as:
  - `srun ... --nodelist ${nodes_array[0]} ...`
- `nodes_array` was only defined inside the deployment block (`if
cfg.deployment.type != "none": ...`).
- Therefore, for `deployment.type=none`, `nodes_array` was undefined and
`${nodes_array[0]}` crashed under nounset.

## Secondary risks (also addressed)

Even when deployment is enabled, `${nodes_array[0]}` can still fail if:

- `$SLURM_JOB_NODELIST` is unset/empty (non-standard environment) or
only `$SLURM_NODELIST` is present.
- `scontrol` is unavailable on the node or not in `PATH`.
- `scontrol show hostnames ...` returns an empty list.

Any of these can result in an empty/unset array index under `set -u`.

## Solution

### Approach

Introduce a **single, always-defined** “node pinning” variable for
single-node sruns:

- `PRIMARY_NODE`

This is resolved at runtime in the sbatch script with safe fallbacks:

1. Prefer `SLURM_JOB_NODELIST`
2. Fallback to `SLURM_NODELIST`
3. Fallback to local `hostname`

---------

Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>

NVIDIA NeMo Evaluator 0.1.71

nemo-evaluator-v0.1.71

Merge branch 'deploy-release/dc6f9f1b-6f4c-4cb0-85c8-60ba14007ca5'

NVIDIA NeMo Evaluator Launcher 0.1.73

nemo-evaluator-launcher-v0.1.73

Merge branch 'deploy-release/dc6f9f1b-6f4c-4cb0-85c8-60ba14007ca5'

NVIDIA NeMo Evaluator 0.1.70

feat(exporters): put the NEL config in the MLflow parameters (#653)

Signed-off-by: Piotr Januszewski <pjanuszewski@nvidia.com>

NVIDIA NeMo Evaluator Launcher 0.1.72

feat(exporters): put the NEL config in the MLflow parameters (#653)

Signed-off-by: Piotr Januszewski <pjanuszewski@nvidia.com>

NVIDIA NeMo Evaluator 0.1.69

ci: Add `release-docs` workflow (#645)

This workflow will be launched manually via the `Actions` tab - it is
called `Release docs`.

When launching it, we can select the branch of the docs we want to
deploy via the dropdown menu. When deploying `main`, we will upload the
docs to the `latest` directory of the webserver. If it's a versioned
tag, it will strip off the version prefix `v` and upload it into the
version folder of the webserver (e.g. `0.2.0`).

---------

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>