From a1612fa1df0e559ac695d3968bb212065390320c Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Fri, 8 Aug 2025 14:49:02 +0100 Subject: [PATCH 01/11] Update to LMEval allowOnline setting instructions --- .DS_Store | Bin 14340 -> 0 bytes ...ernal-resource-access-for-lmeval-jobs.adoc | 55 ++++++++++++ .../evaluating-large-language-models.adoc | 2 + modules/setting-up-lmeval.adoc | 28 +----- ...meval-job-configuration-using-the-cli.adoc | 83 ++++++++++++++++++ ...l-job-configuration-using-the-console.adoc | 82 +++++++++++++++++ 6 files changed, 224 insertions(+), 26 deletions(-) delete mode 100644 .DS_Store create mode 100644 assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc create mode 100644 modules/updating-lmeval-job-configuration-using-the-cli.adoc create mode 100644 modules/updating-lmeval-job-configuration-using-the-console.adoc diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index 46322d8e5a07bdb04eb3647ec37306c435bf1fdd..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 14340 zcmeHN3v?V+8NUBCO=p>HGii6rrcIM=gMDtxZr-J|Y&QulO=!|Q+9Z91eN2;)%}&_a zB(0%ZqTBTd|8KNZa>>AR>yE4(t zsLsj?kHQQTW}q+wg&8Q!Kw$>n>cUSa=p@pfCf48OY86-5>mT(=cp=`+3N= z4t!7(o|aP+o?RG6KY-==jWBG4`+3NkVjI>JL*5h&7Xx^6%=gpk7>12-KM#3x0=zi^ z8J!vVQOM;HCFxR&ZjE3|xDTuY7)r}P22 zG#*XHN0ZTvvObyYQs1XwUWp!0YsqAXrmM7@zTS+oCqv6}ihBEXEuCp%o%O2NKd&W^ z>Pn}AovLF>Gj^?e9AgC6p;_mbFMQAP3u{-eYueD-ezZg`o$dCx=WZKQbv2e$ddAg6 zW~eW!$CXqM0%g|+gP`R0pqmDoEPQ?D9TGkbYo6+XX9 zJ}mA?;QwE)C1}e8uV3N2Koqyr{HC;mPoC{w;#=mDqhhy?1La)=+^bx&?&`i$O&Lu? z?ix`}xO$S&n3CkZ)w>%-xz)98jL$XCT;pCVmO{YQi-gHAzPHY^!@h_pKPm2_|J@eV zGwmZv%6zM^?^03h*N38MrF}f5q|FwaM3Jo@=F=uE2{oe`g%!TdqS!|}Tc1c^KeOf1 zl8!At-+(NusWBB98C*S1%*aPXQLHjop(?zbN7*b^g@Zih!tH(msV2>&ja)&}WP;p4 zZY6h;2g&2)3G#FDEAkTgBY6V=1egxRAb|(Ga6b58AuNX#uo70m8fbC*WrI415vphA+XF;V^s?z6IZb@4`>vIQ$Zx zgWtica1vepgr6i%zqGdZ_u2a|Iq5l#X1 zmCc)9zQA4CgNmT+ajK@NvhoZn^5Riid*LE?r92%~g(nv;!DKP22~VyFR94CYY9~(y zLzQlM8tR^lwXF+a5{$&^i`;Ix2(^%xt-A!1gl+&l-PDXdQJp$d>3){HNM0tdl7FIf z&w@E9(@P)-^>8uD@@D9OF6f3Hl;V_oS~QcDeDKGm-oNCs;pK}JcE z63Vf0-OwcRh^a}(Oi*$V!;xruJEe)cH7yx4o|%ig`S-+@!tA8H70>NV$O*bsHK_S zo~_$@`}%Q@c7o(thv__~f-5gDOSrd~TJF?-8!!tVkpnY@B43%;n+0VAXpuphRNSHk&Ut{jDr}fcwjDZ))H(JPL+&7nkWFM486kVgL2@(sEF$+|@(`l; zkI6Ce9HRG0@)~)ayg~i}MKBXeK!#G72lHV8R3LuWLLKgS5om=r-1FLT$J@zvyeRAj z4H2B~c6%9qe+aIF8{rnX6+VSK-51~x+z0o=1Bl*yXZr=5fM3Il@K^YkFhdXpmoQtH zBY1_m!ZKmGP$OI{vztCE;qqSr~Ye}-9b)|%8jl05Wp!M4-@;@L)$TQ>wnyV+tzhN4PXr}sM zF$7S+*FhMXn3+m}y^}%vPBc`b4BC^1v6_Yn*vAakYv2G}kI+s9{*z4LZ-=`Kqm={u z*R#<66LCP97bS~ggpbppF_ z_TvC{TBC5CJ2Oy@uTn%-;kms^G8QxIVROqnYi5>jVxqpGA>gm64^cB~b#*Y{udS(f zSXl@7)JKX*uz3{L!RjKwkbj@!AR20H1DHO$TpHPq%?fzz>b^A66s8hI<&R)&c8;mYK# zXI-WK*Y7udk}2v=ZeypaYg;|eL+9-?XCmuwjeD5Ca)2>2)O@=G?u5@FBSQ_g$B=LQJ{*Cc!BKb~ z{>-?>za!T;T_{GbQDz>?Vqu98M7D9AumQCNudGSe$J6Fh=*3uGwMP30Rl zZa|;qoYkgWk7a#U{@zA@QL3mkIR>YGa^6uy;y)nIU>sq&RCR%fdLmCZl1rCivQ;kT z$?}z}MAS)HU`eUGy2gc?Cr>kyD;qEgJRz;Q*o8VM7ag5nHqW=NyaGKI!(AyaZ*IYE zo|I2bF1|Sd&+nZqg=%O-@8lBZpR_O+ocbr+4X6G|9Fhi^PQ8<>nSVmPlY{7=aCe;g zC)_)^2kwQh!h=Ywe+?c%f&U&7j!3L?fz66>n<>CNgg;Cr##VT*J9O>yFLNU-ugF^s zvQyQy!~0G2HS+f9fOiq$d=H}FFp?ckGV&dM8Z&S%}YDHg8_}fIKG_ zjbAyKraUqpRBGu+a#YElRa%ZVXFKY3Dv4c_S>1a_@c7d+nZf$nkXDq`_QAB8Xy3hC zS2Depcwf$GY2OZZ44YAgFH7y#raY>(9lrMVMbqhMYEX&n-N{ZJ+fxxOlhKBA*<$wJ zT_e#r9+y^K@+#Nps&I!HS;`|av77@Ag;}PUvwwA`w3Om`M}kE#7Zx!JaV_p&8xf8>akm-*72eC+XLpCg>S~~6EdAA zgVa%Nnfm*Gd!s_CFaw1d_`jL~C~EF%j^L2Yqf6)L=(l!$ysyBU*7-Kx^ujo6f)DmN y9)|35JWSBf@fukr5+QVpz diff --git a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc new file mode 100644 index 000000000..7da358a2c --- /dev/null +++ b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc @@ -0,0 +1,55 @@ +:_module-type: ASSEMBLY + +ifdef::context[:parent-context: {context}] + +[id="enabling-external-resource-access-for-lmeval-jobs_{context}"] += Enabling external resource access for LMEval Jobs + +[role='_abstract'] +LMEval Jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, `LMEvalJob` can be configured to automatically download them. + +Follow the steps below to enable online access and remote code execution for LMEval Jobs. Choose to update these settings by using either the CLI or in the console. + +include::modules/updating-lmeval-job-configuration-using-the-cli.adoc[leveloffset=+1] + +include::modules/updating-lmeval-job-configuration-using-the-console.adoc[leveloffset=+1] + + + + + + + + + +// Incl NOTE ? Opt in to online/code access per LMEvalJob + +// . Option 1/2: Enabling allowOnline and allowCodeExecution modes in your cluster + + + + +//// +In an `LMEvalJob`, there are two configuration options that might require access to external resources: the task dataset and model tokenizer. +You can set your `LMEvalJob` to access online content by updating and patching the TrustyAI operator `ConfigMap`. + +. In the `ConfigMap`, change the `opendatahub.io/managed` setting to `'false'`. + +. Patch the TrustyAI operator `ConfigMap` with the following code: ++ +[source] +---- + oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ +--type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' +---- ++ +. Restart the TrustyAI operator with the following command: ++ +[source] +---- +oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications +---- + +//// + + diff --git a/assemblies/evaluating-large-language-models.adoc b/assemblies/evaluating-large-language-models.adoc index f070bbf4a..33c91e89b 100644 --- a/assemblies/evaluating-large-language-models.adoc +++ b/assemblies/evaluating-large-language-models.adoc @@ -17,6 +17,8 @@ The following sections show you how to create an `LMEvalJob` custom resource (CR include::modules/setting-up-lmeval.adoc[leveloffset=+1] +include::modules/enabling-external-resource-access-for-lmeval-jobs.adoc[leveloffset=+1] + include::modules/lmeval-evaluation-job.adoc[leveloffset=+1] include::modules/lmeval-evaluation-job-properties.adoc[leveloffset=+1] diff --git a/modules/setting-up-lmeval.adoc b/modules/setting-up-lmeval.adoc index 8061d4246..cbbd26103 100644 --- a/modules/setting-up-lmeval.adoc +++ b/modules/setting-up-lmeval.adoc @@ -84,7 +84,6 @@ You can configure the following properties for LM-Eval: After updating the settings in the `ConfigMap`, restart the operator to apply the new values. - // Notes on the 'allow online' setting and patches // upstream - allowOnline setting is enabled; end-user can disable if they wish. @@ -100,29 +99,6 @@ endif::[] ifndef::upstream[] [IMPORTANT] -- -The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. This disallows job permissions to automatically download artifacts from external sources. +The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. LMEval jobs do not have internet access or permission to run any externally downloaded code unless explicitly enabled. -- - -.Enabling allowOnline and allowCodeExecution modes - -You can set your `LMEvalJob` to access online content by updating and patching the TrustyAI operator `ConfigMap`. - -. In the `ConfigMap`, change the `opendatahub.io/managed` setting to `'false'`. - -. Patch the TrustyAI operator `ConfigMap` with the following `kubectl` code: -+ -[source] ----- - kubectl patch configmap trustyai-service-operator-config -n redhat-ods-applications \ ---type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' ----- -+ -. Restart the TrustyAI operator with the following `kubectl` command: -+ -[source] ----- -kubectl rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ----- - -endif::[] - +endif::[] \ No newline at end of file diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc new file mode 100644 index 000000000..cee4610bb --- /dev/null +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -0,0 +1,83 @@ +:_module-type: PROCEDURE + +ifdef::context[:parent-context: {context}] + +[id="updating-lmeval-job-configuration-using-the-cli_{context}"] += Updating LMEval Job configuration using the CLI + +[role='_abstract'] +Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes in the CLI for LMEval Jobs. + +[IMPORTANT] +==== +Enabling online access and code execution involves a security risk. Only use this configuration if you trust the source(s). +==== + +.Prerequisites + +* You have cluster administrator privileges for your {productname-long} cluster. + +ifndef::upstream[] +* You have downloaded and installed the OpenShift command-line interface (CLI). See link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-latest-version}/html/cli_tools/openshift-cli-oc#installing-openshift-cli[Installing the OpenShift CLI^]. +endif::[] + +.Procedure + +. Add or set the `opendatahub.io/managed` annotation to `false` to allow user changes to the `trustyai-service-operator-config` ConfigMap: ++ +[source,sh] +---- +oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ +--type merge -p '{"metadata": {"annotations": {"opendatahub.io/managed": "false"}}}' +---- ++ +.Annotation keys for LMEval job access +[cols="3,2,5",options="header"] +|=== +| Annotation Key +| Value +| Effect + +| `opendatahub.io/managed` +| `true` +| User changes are not allowed and will be automatically reverted. + +| `opendatahub.io/managed` +| `false` +| Allows the user to manage the TrustyAI operator. +|=== ++ +[NOTE] +-- +If the `opendatahub.io/managed` annotation does not exist, the default value is `true`. +-- +. Update the settings for the TrustyAI operator to enable external connectivity and remote code execution: ++ +[source,sh] +---- +oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ +--type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' +---- ++ +.Configuration keys for LMEval job access +[cols="3,2,5", options="header"] +|=== +| ConfigMap Key +| Default +| Description + +| `lmes-allow-online` +| `false` +| Enables internet access for LMEval jobs at the cluster level. + +| `lmes-allow-code-execution` +| `false` +| Allows execution of code downloaded alongside datasets or tokenizers. +|=== + +. Restart the TrustyAI operator to apply the configuration changes: + +[source,sh] +---- +oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications +---- diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc new file mode 100644 index 000000000..5d241664a --- /dev/null +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -0,0 +1,82 @@ +:_module-type: PROCEDURE + +ifdef::context[:parent-context: {context}] + +[id="updating-lmeval-job-configuration-using-the-console_{context}"] += Updating LMEval Job configuration using the console + +[role='_abstract'] +Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes through the {productname-short} console for LMEval Jobs. + +[IMPORTANT] +==== +Enabling online access and code execution involves a security risk. Only use this configuration if you trust the source(s). +==== + +.Prerequisites + +* You have cluster administrator privileges for your {productname-long} cluster. + +.Procedure +. Navigate to *Workloads → ConfigMaps*. + +. Select the *`redhat-ods-applications`* namespace. + +. Open the *`trustyai-service-operator-config`* ConfigMap. + +. Under *Annotations*, add: ++ +[source, sh] +---- +opendatahub.io/managed: false +---- +. Under *Data*, add or update: ++ +[source, sh] +---- +lmes-allow-online: true +lmes-allow-code-execution: true +---- +. Save the ConfigMap. + +. Navigate to *Workloads → Deployments*. + +. Select the *`redhat-ods-applications`* namespace. + +. Locate *`trustyai-service-operator-controller-manager`*. + +. Click the *Actions* menu and select *Restart rollout*. ++ +[NOTE] +-- +Each LMEval Job must explicitly be set to allow online access and code execution. +-- +. Add the following fields to the `LMEvalJob` custom resource to enable online access in the LMEvalJob custom resource: + +[source,yaml] +---- +apiVersion: trustyai.opendatahub.io/v1alpha1 +kind: LMEvalJob +metadata: + name: example-lmeval +spec: + allowOnline: true + allowCodeExecution: true + +---- + +.Configuration keys for LMEval job access +[cols="3,2,5", options="header"] +|=== +| Field +| Default +| Description + +| `spec.allowOnline` +| `false` +| Enables this job to access the internet (e.g., to download datasets or tokenizers). + +| `spec.allowCodeExecution` +| `false` +| Allows this job to run code included with downloaded resources. +|=== From 13f3d443bf5ef7eac879091662d29f47647dbb7e Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Fri, 8 Aug 2025 15:06:52 +0100 Subject: [PATCH 02/11] Updated files --- ...ernal-resource-access-for-lmeval-jobs.adoc | 40 ------------------- .../evaluating-large-language-models.adoc | 2 +- ...meval-job-configuration-using-the-cli.adoc | 11 +++-- ...l-job-configuration-using-the-console.adoc | 3 +- 4 files changed, 7 insertions(+), 49 deletions(-) diff --git a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc index 7da358a2c..1a482c6f0 100644 --- a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc +++ b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc @@ -1,7 +1,6 @@ :_module-type: ASSEMBLY ifdef::context[:parent-context: {context}] - [id="enabling-external-resource-access-for-lmeval-jobs_{context}"] = Enabling external resource access for LMEval Jobs @@ -14,42 +13,3 @@ include::modules/updating-lmeval-job-configuration-using-the-cli.adoc[leveloffse include::modules/updating-lmeval-job-configuration-using-the-console.adoc[leveloffset=+1] - - - - - - - - -// Incl NOTE ? Opt in to online/code access per LMEvalJob - -// . Option 1/2: Enabling allowOnline and allowCodeExecution modes in your cluster - - - - -//// -In an `LMEvalJob`, there are two configuration options that might require access to external resources: the task dataset and model tokenizer. -You can set your `LMEvalJob` to access online content by updating and patching the TrustyAI operator `ConfigMap`. - -. In the `ConfigMap`, change the `opendatahub.io/managed` setting to `'false'`. - -. Patch the TrustyAI operator `ConfigMap` with the following code: -+ -[source] ----- - oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ ---type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' ----- -+ -. Restart the TrustyAI operator with the following command: -+ -[source] ----- -oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ----- - -//// - - diff --git a/assemblies/evaluating-large-language-models.adoc b/assemblies/evaluating-large-language-models.adoc index 33c91e89b..4aaae79e2 100644 --- a/assemblies/evaluating-large-language-models.adoc +++ b/assemblies/evaluating-large-language-models.adoc @@ -17,7 +17,7 @@ The following sections show you how to create an `LMEvalJob` custom resource (CR include::modules/setting-up-lmeval.adoc[leveloffset=+1] -include::modules/enabling-external-resource-access-for-lmeval-jobs.adoc[leveloffset=+1] +include::./enabling-external-resource-access-for-lmeval-jobs.adoc[leveloffset=+1] include::modules/lmeval-evaluation-job.adoc[leveloffset=+1] diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index cee4610bb..05cacc0d5 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -10,7 +10,7 @@ Follow these steps to enable online access (`allowOnline`) and remote code execu [IMPORTANT] ==== -Enabling online access and code execution involves a security risk. Only use this configuration if you trust the source(s). +Enabling online access and code execution involves a security risk. Only use these configurations if you trust the source(s). ==== .Prerequisites @@ -31,6 +31,10 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ --type merge -p '{"metadata": {"annotations": {"opendatahub.io/managed": "false"}}}' ---- + +[NOTE] +-- +If the `opendatahub.io/managed` annotation does not exist, the default value is `true`. +-- .Annotation keys for LMEval job access [cols="3,2,5",options="header"] |=== @@ -46,11 +50,6 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ | `false` | Allows the user to manage the TrustyAI operator. |=== -+ -[NOTE] --- -If the `opendatahub.io/managed` annotation does not exist, the default value is `true`. --- . Update the settings for the TrustyAI operator to enable external connectivity and remote code execution: + [source,sh] diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index 5d241664a..a9cc0bd5a 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -1,7 +1,6 @@ :_module-type: PROCEDURE ifdef::context[:parent-context: {context}] - [id="updating-lmeval-job-configuration-using-the-console_{context}"] = Updating LMEval Job configuration using the console @@ -10,7 +9,7 @@ Follow these steps to enable online access (`allowOnline`) and remote code execu [IMPORTANT] ==== -Enabling online access and code execution involves a security risk. Only use this configuration if you trust the source(s). +Enabling online access and code execution involves a security risk. Only use these configurations if you trust the source(s). ==== .Prerequisites From 5eeabc897a989f546aa169b78f619078a6986f9c Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Fri, 8 Aug 2025 15:17:22 +0100 Subject: [PATCH 03/11] Updates to layout --- modules/updating-lmeval-job-configuration-using-the-cli.adoc | 3 ++- .../updating-lmeval-job-configuration-using-the-console.adoc | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index 05cacc0d5..d675bfaf0 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -35,8 +35,9 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ -- If the `opendatahub.io/managed` annotation does not exist, the default value is `true`. -- ++ .Annotation keys for LMEval job access -[cols="3,2,5",options="header"] +[cols="3,2,5", options="header"] |=== | Annotation Key | Value diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index a9cc0bd5a..3ee6b1470 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -64,7 +64,7 @@ spec: ---- -.Configuration keys for LMEval job access +.Configuration keys for LMEval Job access [cols="3,2,5", options="header"] |=== | Field From 1e21b6feddc254badab8828cb1f469ad2b7142c3 Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Fri, 8 Aug 2025 15:30:34 +0100 Subject: [PATCH 04/11] Further small updates --- ...ernal-resource-access-for-lmeval-jobs.adoc | 2 +- modules/setting-up-lmeval.adoc | 2 +- ...meval-job-configuration-using-the-cli.adoc | 21 +++++++++++++++---- ...l-job-configuration-using-the-console.adoc | 2 +- 4 files changed, 20 insertions(+), 7 deletions(-) diff --git a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc index 1a482c6f0..8bd244d3a 100644 --- a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc +++ b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc @@ -7,7 +7,7 @@ ifdef::context[:parent-context: {context}] [role='_abstract'] LMEval Jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, `LMEvalJob` can be configured to automatically download them. -Follow the steps below to enable online access and remote code execution for LMEval Jobs. Choose to update these settings by using either the CLI or in the console. +Follow the steps below to enable online access and remote code execution for LMEval Jobs. Choose to update these settings by using either the CLI or in the console. Enable one or both settings according to your needs. include::modules/updating-lmeval-job-configuration-using-the-cli.adoc[leveloffset=+1] diff --git a/modules/setting-up-lmeval.adoc b/modules/setting-up-lmeval.adoc index cbbd26103..e5cd54384 100644 --- a/modules/setting-up-lmeval.adoc +++ b/modules/setting-up-lmeval.adoc @@ -76,7 +76,7 @@ You can configure the following properties for LM-Eval: | true | Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. -| `lmes-code-execution` +| `lmes-allow-code-execution` | true | Determines whether LMEval jobs can set the `trust remote code` mode to `on`. diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index d675bfaf0..ed6e9d7ff 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -6,7 +6,7 @@ ifdef::context[:parent-context: {context}] = Updating LMEval Job configuration using the CLI [role='_abstract'] -Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes in the CLI for LMEval Jobs. +Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes using the CLI for LMEval Jobs. [IMPORTANT] ==== @@ -51,7 +51,7 @@ If the `opendatahub.io/managed` annotation does not exist, the default value is | `false` | Allows the user to manage the TrustyAI operator. |=== -. Update the settings for the TrustyAI operator to enable external connectivity and remote code execution: +. Update the settings for the TrustyAI operator to enable external connectivity or remote code execution, or both: + [source,sh] ---- @@ -74,10 +74,23 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ | `false` | Allows execution of code downloaded alongside datasets or tokenizers. |=== - . Restart the TrustyAI operator to apply the configuration changes: - ++ +[source,sh] +---- +oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications +---- +. Set the flags on your `LMEvalJob` resource: ++ [source,sh] ---- oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ---- +. Enable online access and/or code execution on the specific `LMEvalJob`: ++ +[source,sh] +---- +# Patch an existing job (example name: example-lmeval) +oc patch lmevaljob example-lmeval -n \ + --type merge -p '{"spec":{"allowOnline":true,"allowCodeExecution":true}}' +---- diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index 3ee6b1470..1d140b166 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -50,7 +50,7 @@ lmes-allow-code-execution: true -- Each LMEval Job must explicitly be set to allow online access and code execution. -- -. Add the following fields to the `LMEvalJob` custom resource to enable online access in the LMEvalJob custom resource: +. Add the following fields to the `LMEvalJob` to enable online access and code execution for this job: [source,yaml] ---- From 5a212a3a496ae3ce84c9b4aea8a3325aa181b337 Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Fri, 8 Aug 2025 15:42:18 +0100 Subject: [PATCH 05/11] Updating acc to CodeRabbit suggestions --- ...ating-lmeval-job-configuration-using-the-cli.adoc | 12 +++++++----- ...g-lmeval-job-configuration-using-the-console.adoc | 12 ++++++------ 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index ed6e9d7ff..818ca5511 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -6,11 +6,11 @@ ifdef::context[:parent-context: {context}] = Updating LMEval Job configuration using the CLI [role='_abstract'] -Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes using the CLI for LMEval Jobs. +Follow these steps to enable online access (`allowOnline`) or remote code execution (`allowCodeExecution`) or both modes using the CLI for LMEval Jobs. [IMPORTANT] ==== -Enabling online access and code execution involves a security risk. Only use these configurations if you trust the source(s). +Enabling online access or code execution involves a security risk. Only use these configurations if you trust the source(s). ==== .Prerequisites @@ -86,11 +86,13 @@ oc rollout restart deployment trustyai-service-operator-controller-manager -n re ---- oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ---- -. Enable online access and/or code execution on the specific `LMEvalJob`: +. Enable online access or code execution or both on specific `LMEvalJob` resources: + [source,sh] ---- -# Patch an existing job (example name: example-lmeval) -oc patch lmevaljob example-lmeval -n \ +oc patch lmevaljobs example-lmeval -n \ --type merge -p '{"spec":{"allowOnline":true,"allowCodeExecution":true}}' + +oc get lmevaljobs example-lmeval -n \ + -o jsonpath='{.spec.allowOnline}{"\n"}{.spec.allowCodeExecution}{"\n"}' ---- diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index 1d140b166..8a542d30d 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -9,7 +9,7 @@ Follow these steps to enable online access (`allowOnline`) and remote code execu [IMPORTANT] ==== -Enabling online access and code execution involves a security risk. Only use these configurations if you trust the source(s). +Enabling online access or code execution involves a security risk. Only use these configurations if you trust the source(s). ==== .Prerequisites @@ -25,16 +25,16 @@ Enabling online access and code execution involves a security risk. Only use the . Under *Annotations*, add: + -[source, sh] +[source, yaml] ---- -opendatahub.io/managed: false +opendatahub.io/managed: "false" ---- . Under *Data*, add or update: + -[source, sh] +[source, yaml] ---- -lmes-allow-online: true -lmes-allow-code-execution: true +lmes-allow-online: "true" +lmes-allow-code-execution: "true" ---- . Save the ConfigMap. From a8506ce611eb16ccb78744c88edfbbd745ce0d69 Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Fri, 8 Aug 2025 15:46:40 +0100 Subject: [PATCH 06/11] further tweaks --- modules/updating-lmeval-job-configuration-using-the-cli.adoc | 4 ++-- .../updating-lmeval-job-configuration-using-the-console.adoc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index 818ca5511..ec0fc4bcd 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -36,7 +36,7 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ If the `opendatahub.io/managed` annotation does not exist, the default value is `true`. -- + -.Annotation keys for LMEval job access +.trustyai-service-operator-config annotation keys for LMEval job access [cols="3,2,5", options="header"] |=== | Annotation Key @@ -59,7 +59,7 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ --type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' ---- + -.Configuration keys for LMEval job access +.trustyai-service-operator-config configuration keys for LMEval job access [cols="3,2,5", options="header"] |=== | ConfigMap Key diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index 8a542d30d..af07e7346 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -64,7 +64,7 @@ spec: ---- -.Configuration keys for LMEval Job access +.Configuration keys for LMEvalJob custom resource [cols="3,2,5", options="header"] |=== | Field From 3ded1f0000f79ba6074636bd4a2759c08094683c Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Mon, 11 Aug 2025 15:21:15 +0100 Subject: [PATCH 07/11] Tweaks as a result of QE feedback --- modules/setting-up-lmeval.adoc | 15 --------- ...meval-job-configuration-using-the-cli.adoc | 31 +++++++++++++------ ...l-job-configuration-using-the-console.adoc | 4 +-- 3 files changed, 23 insertions(+), 27 deletions(-) diff --git a/modules/setting-up-lmeval.adoc b/modules/setting-up-lmeval.adoc index e5cd54384..351b86343 100644 --- a/modules/setting-up-lmeval.adoc +++ b/modules/setting-up-lmeval.adoc @@ -16,7 +16,6 @@ The following information explains how to create an `LMEvalJob` custom resource ifdef::upstream[] [NOTE] - -- LM-Eval is only available in the latest community builds. To use LM-Eval on {productname-long}, ensure that you use ODH 2.20 or later versions and add the following `devFlag` to your `DataScienceCluster` resource: [source] @@ -84,21 +83,7 @@ You can configure the following properties for LM-Eval: After updating the settings in the `ConfigMap`, restart the operator to apply the new values. -// Notes on the 'allow online' setting and patches - -// upstream - allowOnline setting is enabled; end-user can disable if they wish. -ifdef::upstream[] -[IMPORTANT] --- -The `allowOnline` setting is enabled by default in {productname-long}. Using `allowOnline` gives the job permissions to automatically download artifacts from external sources. Change this setting to `false` if you do not want LM-Eval to access external sources. --- -endif::[] - -// downstream - allowOnline is disabled; end-user needs include a patch in the ConfigMap in order to enable it. - -ifndef::upstream[] [IMPORTANT] -- The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. LMEval jobs do not have internet access or permission to run any externally downloaded code unless explicitly enabled. -- -endif::[] \ No newline at end of file diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index ec0fc4bcd..922301752 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -59,7 +59,7 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ --type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' ---- + -.trustyai-service-operator-config configuration keys for LMEval job access +.trustyai-service-operator-config configuration keys for LMEval job access [cols="3,2,5", options="header"] |=== | ConfigMap Key @@ -80,19 +80,30 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ ---- oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ---- -. Set the flags on your `LMEvalJob` resource: -+ -[source,sh] ----- -oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ----- . Enable online access or code execution or both on specific `LMEvalJob` resources: + +.. For LMEval Jobs that already exist, apply the following patch: + [source,sh] ---- -oc patch lmevaljobs example-lmeval -n \ +oc patch lmevaljobs example-lmeval -n \ --type merge -p '{"spec":{"allowOnline":true,"allowCodeExecution":true}}' - -oc get lmevaljobs example-lmeval -n \ +oc get lmevaljobs example-lmeval -n \ -o jsonpath='{.spec.allowOnline}{"\n"}{.spec.allowCodeExecution}{"\n"}' ---- + +.. For new LMEval Jobs, apply a YAML that includes the following fields: +[source,yaml] +---- +apiVersion: trustyai.opendatahub.io/v1alpha1 +kind: LMEvalJob +metadata: + name: example-lmeval +spec: + allowOnline: true + allowCodeExecution: true + +---- + + + diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index af07e7346..8cc788fdc 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -50,8 +50,8 @@ lmes-allow-code-execution: "true" -- Each LMEval Job must explicitly be set to allow online access and code execution. -- -. Add the following fields to the `LMEvalJob` to enable online access and code execution for this job: - +. +Ensure that the following fields are set to 'true' to enable online access and code execution for this job when writing your `LMEvalJob` custom resource: [source,yaml] ---- apiVersion: trustyai.opendatahub.io/v1alpha1 From 89d7ab75defb437293b11e69f32bcd40e3f91711 Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Mon, 11 Aug 2025 15:28:14 +0100 Subject: [PATCH 08/11] Updated the instances of Job --- .../enabling-external-resource-access-for-lmeval-jobs.adoc | 6 +++--- modules/setting-up-lmeval.adoc | 4 ++-- .../updating-lmeval-job-configuration-using-the-cli.adoc | 6 +++--- ...updating-lmeval-job-configuration-using-the-console.adoc | 6 +++--- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc index 8bd244d3a..2b6a60720 100644 --- a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc +++ b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc @@ -2,12 +2,12 @@ ifdef::context[:parent-context: {context}] [id="enabling-external-resource-access-for-lmeval-jobs_{context}"] -= Enabling external resource access for LMEval Jobs += Enabling external resource access for LMEval jobs [role='_abstract'] -LMEval Jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, `LMEvalJob` can be configured to automatically download them. +LMEval jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, `LMEvalJob` can be configured to automatically download them. -Follow the steps below to enable online access and remote code execution for LMEval Jobs. Choose to update these settings by using either the CLI or in the console. Enable one or both settings according to your needs. +Follow the steps below to enable online access and remote code execution for LMEval jobs. Choose to update these settings by using either the CLI or in the console. Enable one or both settings according to your needs. include::modules/updating-lmeval-job-configuration-using-the-cli.adoc[leveloffset=+1] diff --git a/modules/setting-up-lmeval.adoc b/modules/setting-up-lmeval.adoc index 351b86343..6fde11e5d 100644 --- a/modules/setting-up-lmeval.adoc +++ b/modules/setting-up-lmeval.adoc @@ -72,11 +72,11 @@ You can configure the following properties for LM-Eval: | The interval to check the job pod for an evaluation job. | `lmes-allow-online` -| true +| false | Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. | `lmes-allow-code-execution` -| true +| false | Determines whether LMEval jobs can set the `trust remote code` mode to `on`. |=== diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index 922301752..489a0901e 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -6,7 +6,7 @@ ifdef::context[:parent-context: {context}] = Updating LMEval Job configuration using the CLI [role='_abstract'] -Follow these steps to enable online access (`allowOnline`) or remote code execution (`allowCodeExecution`) or both modes using the CLI for LMEval Jobs. +Follow these steps to enable online access (`allowOnline`) or remote code execution (`allowCodeExecution`) or both modes using the CLI for LMEval jobs. [IMPORTANT] ==== @@ -82,7 +82,7 @@ oc rollout restart deployment trustyai-service-operator-controller-manager -n re ---- . Enable online access or code execution or both on specific `LMEvalJob` resources: -.. For LMEval Jobs that already exist, apply the following patch: +.. For LMEval jobs that already exist, apply the following patch: + [source,sh] ---- @@ -92,7 +92,7 @@ oc get lmevaljobs example-lmeval -n \ -o jsonpath='{.spec.allowOnline}{"\n"}{.spec.allowCodeExecution}{"\n"}' ---- -.. For new LMEval Jobs, apply a YAML that includes the following fields: +.. For new LMEval jobs, apply a YAML that includes the following fields: [source,yaml] ---- apiVersion: trustyai.opendatahub.io/v1alpha1 diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index 8cc788fdc..c6ac05c82 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -2,10 +2,10 @@ ifdef::context[:parent-context: {context}] [id="updating-lmeval-job-configuration-using-the-console_{context}"] -= Updating LMEval Job configuration using the console += Updating LMEval job configuration using the console [role='_abstract'] -Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes through the {productname-short} console for LMEval Jobs. +Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes through the {productname-short} console for LMEval jobs. [IMPORTANT] ==== @@ -48,7 +48,7 @@ lmes-allow-code-execution: "true" + [NOTE] -- -Each LMEval Job must explicitly be set to allow online access and code execution. +Each LMEval job must explicitly be set to allow online access and code execution. -- . Ensure that the following fields are set to 'true' to enable online access and code execution for this job when writing your `LMEvalJob` custom resource: From cbf7a6d57983840026a7b43a8ce0c3e232dd8322 Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Mon, 11 Aug 2025 15:50:11 +0100 Subject: [PATCH 09/11] further tweaks --- .../enabling-external-resource-access-for-lmeval-jobs.adoc | 2 +- modules/updating-lmeval-job-configuration-using-the-cli.adoc | 3 ++- .../updating-lmeval-job-configuration-using-the-console.adoc | 5 +++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc index 2b6a60720..a653f709d 100644 --- a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc +++ b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc @@ -5,7 +5,7 @@ ifdef::context[:parent-context: {context}] = Enabling external resource access for LMEval jobs [role='_abstract'] -LMEval jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, `LMEvalJob` can be configured to automatically download them. +LMEval jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, an `LMEvalJob` can be configured to automatically download them. Follow the steps below to enable online access and remote code execution for LMEval jobs. Choose to update these settings by using either the CLI or in the console. Enable one or both settings according to your needs. diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index 489a0901e..fdb3819ce 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -36,7 +36,7 @@ oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ If the `opendatahub.io/managed` annotation does not exist, the default value is `true`. -- + -.trustyai-service-operator-config annotation keys for LMEval job access +.trustyai-service-operator-config annotation keys for LMEval job access [cols="3,2,5", options="header"] |=== | Annotation Key @@ -93,6 +93,7 @@ oc get lmevaljobs example-lmeval -n \ ---- .. For new LMEval jobs, apply a YAML that includes the following fields: ++ [source,yaml] ---- apiVersion: trustyai.opendatahub.io/v1alpha1 diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-console.adoc index c6ac05c82..bf6a2ff24 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-console.adoc @@ -50,8 +50,8 @@ lmes-allow-code-execution: "true" -- Each LMEval job must explicitly be set to allow online access and code execution. -- -. -Ensure that the following fields are set to 'true' to enable online access and code execution for this job when writing your `LMEvalJob` custom resource: +. Ensure that the following fields are set to 'true' to enable online access and code execution for this job when writing your `LMEvalJob` custom resource: ++ [source,yaml] ---- apiVersion: trustyai.opendatahub.io/v1alpha1 @@ -64,6 +64,7 @@ spec: ---- + .Configuration keys for LMEvalJob custom resource [cols="3,2,5", options="header"] |=== From 1c15d99b179132cff85aa0036def5aebcc684b8f Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Mon, 11 Aug 2025 15:57:08 +0100 Subject: [PATCH 10/11] addressing codeRabbit comments for newly updated module setting-up-lmeval --- modules/setting-up-lmeval.adoc | 6 +++--- .../updating-lmeval-job-configuration-using-the-cli.adoc | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/modules/setting-up-lmeval.adoc b/modules/setting-up-lmeval.adoc index 6fde11e5d..0e2a198a6 100644 --- a/modules/setting-up-lmeval.adoc +++ b/modules/setting-up-lmeval.adoc @@ -45,7 +45,7 @@ You can configure the following properties for LM-Eval: | `lmes-detect-device` | `true/false` -| Detect if there are GPUs available and assign a value for `--device argument` for LM Evaluation Harness. If GPUs are available, the value is `cuda`. If there are no GPUs available, the value is `cpu`. +| Detect if there are GPUs available and assign a value for the `--device argument` for LM Evaluation Harness. If GPUs are available, the value is `cuda`. If there are no GPUs available, the value is `cpu`. | `lmes-pod-image` | `quay.io/trustyai/ta-lmes-job:latest` @@ -72,11 +72,11 @@ You can configure the following properties for LM-Eval: | The interval to check the job pod for an evaluation job. | `lmes-allow-online` -| false +| `false` | Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. | `lmes-allow-code-execution` -| false +| `false` | Determines whether LMEval jobs can set the `trust remote code` mode to `on`. |=== diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index fdb3819ce..754d5b80b 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -86,9 +86,9 @@ oc rollout restart deployment trustyai-service-operator-controller-manager -n re + [source,sh] ---- -oc patch lmevaljobs example-lmeval -n \ +oc patch lmevaljobs example-lmeval -n \ --type merge -p '{"spec":{"allowOnline":true,"allowCodeExecution":true}}' -oc get lmevaljobs example-lmeval -n \ +oc get lmevaljobs example-lmeval -n \ -o jsonpath='{.spec.allowOnline}{"\n"}{.spec.allowCodeExecution}{"\n"}' ---- @@ -100,6 +100,7 @@ apiVersion: trustyai.opendatahub.io/v1alpha1 kind: LMEvalJob metadata: name: example-lmeval + namespace: spec: allowOnline: true allowCodeExecution: true From 62a1f79dead9723792d0fb346205df817bbabf44 Mon Sep 17 00:00:00 2001 From: "B. Synnott" Date: Mon, 11 Aug 2025 16:44:18 +0100 Subject: [PATCH 11/11] UPdated to address peer review comments --- ...enabling-external-resource-access-for-lmeval-jobs.adoc | 2 +- modules/setting-up-lmeval.adoc | 2 +- .../updating-lmeval-job-configuration-using-the-cli.adoc | 2 +- ...g-lmeval-job-configuration-using-the-web-console.adoc} | 8 ++++---- 4 files changed, 7 insertions(+), 7 deletions(-) rename modules/{updating-lmeval-job-configuration-using-the-console.adoc => updating-lmeval-job-configuration-using-the-web-console.adoc} (88%) diff --git a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc index a653f709d..1414f5097 100644 --- a/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc +++ b/assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc @@ -11,5 +11,5 @@ Follow the steps below to enable online access and remote code execution for LME include::modules/updating-lmeval-job-configuration-using-the-cli.adoc[leveloffset=+1] -include::modules/updating-lmeval-job-configuration-using-the-console.adoc[leveloffset=+1] +include::modules/updating-lmeval-job-configuration-using-the-web-console.adoc[leveloffset=+1] diff --git a/modules/setting-up-lmeval.adoc b/modules/setting-up-lmeval.adoc index 0e2a198a6..242690c49 100644 --- a/modules/setting-up-lmeval.adoc +++ b/modules/setting-up-lmeval.adoc @@ -85,5 +85,5 @@ After updating the settings in the `ConfigMap`, restart the operator to apply th [IMPORTANT] -- -The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. LMEval jobs do not have internet access or permission to run any externally downloaded code unless explicitly enabled. +The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. LMEval jobs do not allow internet access or permission to run any externally downloaded code unless explicitly enabled. -- diff --git a/modules/updating-lmeval-job-configuration-using-the-cli.adoc b/modules/updating-lmeval-job-configuration-using-the-cli.adoc index 754d5b80b..f2d7f337d 100644 --- a/modules/updating-lmeval-job-configuration-using-the-cli.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-cli.adoc @@ -18,7 +18,7 @@ Enabling online access or code execution involves a security risk. Only use thes * You have cluster administrator privileges for your {productname-long} cluster. ifndef::upstream[] -* You have downloaded and installed the OpenShift command-line interface (CLI). See link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-latest-version}/html/cli_tools/openshift-cli-oc#installing-openshift-cli[Installing the OpenShift CLI^]. +* You have downloaded and installed the {productname-short} command-line interface (CLI). See link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-latest-version}/html/cli_tools/openshift-cli-oc#installing-openshift-cli[Installing the OpenShift CLI^]. endif::[] .Procedure diff --git a/modules/updating-lmeval-job-configuration-using-the-console.adoc b/modules/updating-lmeval-job-configuration-using-the-web-console.adoc similarity index 88% rename from modules/updating-lmeval-job-configuration-using-the-console.adoc rename to modules/updating-lmeval-job-configuration-using-the-web-console.adoc index bf6a2ff24..9f14e407c 100644 --- a/modules/updating-lmeval-job-configuration-using-the-console.adoc +++ b/modules/updating-lmeval-job-configuration-using-the-web-console.adoc @@ -1,11 +1,11 @@ :_module-type: PROCEDURE ifdef::context[:parent-context: {context}] -[id="updating-lmeval-job-configuration-using-the-console_{context}"] -= Updating LMEval job configuration using the console +[id="updating-lmeval-job-configuration-using-the-web-console_{context}"] += Updating LMEval job configuration using the web console [role='_abstract'] -Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes through the {productname-short} console for LMEval jobs. +Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes through the {productname-short} web console for LMEval jobs. [IMPORTANT] ==== @@ -50,7 +50,7 @@ lmes-allow-code-execution: "true" -- Each LMEval job must explicitly be set to allow online access and code execution. -- -. Ensure that the following fields are set to 'true' to enable online access and code execution for this job when writing your `LMEvalJob` custom resource: +. Ensure that the following fields are set to `true` to enable online access and code execution for this job when writing your `LMEvalJob` custom resource: + [source,yaml] ----