opendatahub-io · BSynRedhat · Aug 11, 2025 · Aug 8, 2025 · Aug 8, 2025 · Aug 8, 2025
diff --git a/.DS_Store b/.DS_Store
diff --git a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc
@@ -0,0 +1,15 @@
+:_module-type: ASSEMBLY
+
+ifdef::context[:parent-context: {context}]
+[id="enabling-external-resource-access-for-lmeval-jobs_{context}"]
+= Enabling external resource access for LMEval jobs
+
+[role='_abstract']
+LMEval jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, an `LMEvalJob` can be configured to automatically download them.
+
+Follow the steps below to enable online access and remote code execution for LMEval jobs. Choose to update these settings by using either the CLI or in the console. Enable one or both settings according to your needs.
+
+include::modules/updating-lmeval-job-configuration-using-the-cli.adoc[leveloffset=+1]
+
+include::modules/updating-lmeval-job-configuration-using-the-web-console.adoc[leveloffset=+1]
+
diff --git a/assemblies/evaluating-large-language-models.adoc b/assemblies/evaluating-large-language-models.adoc
@@ -17,6 +17,8 @@ The following sections show you how to create an `LMEvalJob` custom resource (CR
 
 include::modules/setting-up-lmeval.adoc[leveloffset=+1]
 
+include::./enabling-external-resource-access-for-lmeval-jobs.adoc[leveloffset=+1]
+
 include::modules/lmeval-evaluation-job.adoc[leveloffset=+1]
 
 include::modules/lmeval-evaluation-job-properties.adoc[leveloffset=+1]

diff --git a/modules/setting-up-lmeval.adoc b/modules/setting-up-lmeval.adoc
@@ -16,7 +16,6 @@ The following information explains how to create an `LMEvalJob` custom resource
 
 ifdef::upstream[]
 [NOTE]
-
 --
 LM-Eval is only available in the latest community builds. To use LM-Eval on {productname-long}, ensure that you use ODH 2.20 or later versions and add the following `devFlag` to your `DataScienceCluster` resource:
 [source]
@@ -46,7 +45,7 @@ You can configure the following properties for LM-Eval:
 
 | `lmes-detect-device`
 | `true/false`
-| Detect if there are GPUs available and assign a value for `--device argument` for LM Evaluation Harness. If GPUs are available, the value is `cuda`. If there are no GPUs available, the value is `cpu`.
+| Detect if there are GPUs available and assign a value for the `--device argument` for LM Evaluation Harness. If GPUs are available, the value is `cuda`. If there are no GPUs available, the value is `cpu`.
 
 | `lmes-pod-image`
 | `quay.io/trustyai/ta-lmes-job:latest`
@@ -73,56 +72,18 @@ You can configure the following properties for LM-Eval:
 | The interval to check the job pod for an evaluation job.
 
 | `lmes-allow-online`
-| true
+| `false`
 | Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. 
 
-| `lmes-code-execution`
-| true
+| `lmes-allow-code-execution`
+| `false`
 | Determines whether LMEval jobs can set the `trust remote code` mode to `on`.
 
 |===
 
 After updating the settings in the `ConfigMap`, restart the operator to apply the new values.
 
-
-// Notes on the 'allow online' setting and patches 
-
-// upstream - allowOnline setting is enabled; end-user can disable if they wish.
-ifdef::upstream[]
-[IMPORTANT]
---
-The `allowOnline` setting is enabled by default in {productname-long}. Using `allowOnline` gives the job permissions to automatically download artifacts from external sources. Change this setting to `false` if you do not want LM-Eval to access external sources.
---
-endif::[]
-
-// downstream - allowOnline is disabled; end-user needs include a patch in the ConfigMap in order to enable it.
-
-ifndef::upstream[]
 [IMPORTANT]
 --
-The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. This disallows job permissions to automatically download artifacts from external sources. 
+The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. LMEval jobs do not allow internet access or permission to run any externally downloaded code unless explicitly enabled.
 --
-
-.Enabling allowOnline and allowCodeExecution modes
-
-You can set your `LMEvalJob` to access online content by updating and patching the TrustyAI operator `ConfigMap`. 
-
-. In the `ConfigMap`, change the `opendatahub.io/managed` setting to `'false'`. 
-
-. Patch the TrustyAI operator `ConfigMap` with the following `kubectl` code: 
-+
-[source]
-----
- kubectl patch configmap trustyai-service-operator-config -n redhat-ods-applications \
---type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}'
-----
-+
-. Restart the TrustyAI operator with the following `kubectl` command: 
-+
-[source]
-----
-kubectl rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications
-----
-
-endif::[]
-
diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc
@@ -0,0 +1,111 @@
+:_module-type: PROCEDURE
+
+ifdef::context[:parent-context: {context}]
+
+[id="updating-lmeval-job-configuration-using-the-cli_{context}"]
+= Updating LMEval Job configuration using the CLI
+
+[role='_abstract']
+Follow these steps to enable online access (`allowOnline`) or remote code execution (`allowCodeExecution`) or both modes using the CLI for LMEval jobs. 
+
+[IMPORTANT]
+====
+Enabling online access or code execution involves a security risk. Only use these configurations if you trust the source(s).
+====
+
+.Prerequisites
+
+* You have cluster administrator privileges for your {productname-long} cluster.
+
+ifndef::upstream[]
+* You have downloaded and installed the {productname-short} command-line interface (CLI). See link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-latest-version}/html/cli_tools/openshift-cli-oc#installing-openshift-cli[Installing the OpenShift CLI^].
+endif::[]
+
+.Procedure
+
+. Add or set the `opendatahub.io/managed` annotation to `false` to allow user changes to the `trustyai-service-operator-config` ConfigMap: 
++
+[source,sh]
+----
+oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \
+--type merge -p '{"metadata": {"annotations": {"opendatahub.io/managed": "false"}}}'
+----
++
+[NOTE]
+--
+If the `opendatahub.io/managed` annotation does not exist, the default value is `true`.
+--
++
+.trustyai-service-operator-config annotation keys for LMEval job access
+[cols="3,2,5", options="header"]
+|===
+| Annotation Key
+| Value
+| Effect
+
+| `opendatahub.io/managed`
+| `true`
+| User changes are not allowed and will be automatically reverted. 
+
+| `opendatahub.io/managed`
+| `false`
+| Allows the user to manage the TrustyAI operator.
+|===
+. Update the settings for the TrustyAI operator to enable external connectivity or remote code execution, or both:
++
+[source,sh]
+----
+oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \
+--type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}'
+----
++
+.trustyai-service-operator-config configuration keys for LMEval job access
+[cols="3,2,5", options="header"]
+|===
+| ConfigMap Key
+| Default
+| Description
+
+| `lmes-allow-online`
+| `false`
+| Enables internet access for LMEval jobs at the cluster level.
+
+| `lmes-allow-code-execution`
+| `false`
+| Allows execution of code downloaded alongside datasets or tokenizers.
+|===
+. Restart the TrustyAI operator to apply the configuration changes:
++
+[source,sh]
+----
+oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications
+----
+. Enable online access or code execution or both on specific `LMEvalJob` resources: 
+
+.. For LMEval jobs that already exist, apply the following patch: 
++
+[source,sh]
+----
+oc patch lmevaljobs example-lmeval -n <user-or-target-namespace> \
+  --type merge -p '{"spec":{"allowOnline":true,"allowCodeExecution":true}}'
+oc get lmevaljobs example-lmeval -n <user-or-target-namespace> \
+  -o jsonpath='{.spec.allowOnline}{"\n"}{.spec.allowCodeExecution}{"\n"}'
+----
+
+.. For new LMEval jobs, apply a YAML that includes the following fields: 
++
+[source,yaml]
+----
+apiVersion: trustyai.opendatahub.io/v1alpha1
+kind: LMEvalJob
+metadata:
+  name: example-lmeval
+  namespace: <your-namespace>
+spec:
+  allowOnline: true
+  allowCodeExecution: true
+
+----
+
+
+
diff --git a/modules/updating-lmeval-job-configuration-using-the-web-console.adoc b/modules/updating-lmeval-job-configuration-using-the-web-console.adoc
@@ -0,0 +1,82 @@
+:_module-type: PROCEDURE
+
+ifdef::context[:parent-context: {context}]
+[id="updating-lmeval-job-configuration-using-the-web-console_{context}"]
+= Updating LMEval job configuration using the web console
+
+[role='_abstract']
+Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes through the {productname-short} web console for LMEval jobs. 
+
+[IMPORTANT]
+====
+Enabling online access or code execution involves a security risk. Only use these configurations if you trust the source(s).
+====
+
+.Prerequisites
+
+* You have cluster administrator privileges for your {productname-long} cluster.
+
+.Procedure
+. Navigate to *Workloads → ConfigMaps*.
+
+. Select the *`redhat-ods-applications`* namespace.
+
+. Open the *`trustyai-service-operator-config`* ConfigMap.
+
+. Under *Annotations*, add:
++
+[source, yaml]
+----
+opendatahub.io/managed: "false"
+----
+. Under *Data*, add or update:
++
+[source, yaml]
+----
+lmes-allow-online: "true"
+lmes-allow-code-execution: "true"
+----
+. Save the ConfigMap.
+
+. Navigate to *Workloads → Deployments*.
+
+. Select the *`redhat-ods-applications`* namespace.
+
+. Locate *`trustyai-service-operator-controller-manager`*.
+
+. Click the *Actions* menu and select *Restart rollout*.
++
+[NOTE]
+--
+Each LMEval job must explicitly be set to allow online access and code execution.
+--
+. Ensure that the following fields are set to `true` to enable online access and code execution for this job when writing your `LMEvalJob` custom resource:
++
+[source,yaml]
+----
+apiVersion: trustyai.opendatahub.io/v1alpha1
+kind: LMEvalJob
+metadata:
+  name: example-lmeval
+spec:
+  allowOnline: true
+  allowCodeExecution: true
+
+----
+
+
+.Configuration keys for LMEvalJob custom resource
+[cols="3,2,5", options="header"]
+|===
+| Field
+| Default
+| Description
+
+| `spec.allowOnline`
+| `false`
+| Enables this job to access the internet (e.g., to download datasets or tokenizers).
+
+| `spec.allowCodeExecution`
+| `false`
+| Allows this job to run code included with downloaded resources.
+|===