Skip to content

Commit bdaadd6

Browse files
BSynRedhatBSynRedhat
authored
ENG-28527 - UI for LMEval (#887)
* New module for using UI for LMEval * Updating a word * updates after peer review * updated after peer review 2 * Updated acc to peer review and QE advice
1 parent fe5800c commit bdaadd6

File tree

2 files changed

+143
-0
lines changed

2 files changed

+143
-0
lines changed

‎assemblies/evaluating-large-language-models.adoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ include::modules/lmeval-evaluation-job.adoc[leveloffset=+1]
2121

2222
include::modules/lmeval-evaluation-job-properties.adoc[leveloffset=+1]
2323

24+
include::modules/performing-model-evaluations-in-the-dashboard.adoc[leveloffset=+1]
25+
2426
include::./lmeval-scenarios.adoc[leveloffset=+1]
2527

2628

‎modules/performing-model-evaluations-in-the-dashboard.adoc

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
:_module-type: PROCEDURE
2+
3+
ifdef::context[:parent-context: {context}]
4+
[id="performing-model-evaluations-in-the-dashboard_{context}"]
5+
= Performing model evaluations in the dashboard
6+
7+
[role='_abstract']
8+
LM-Eval is a Language Model Evaluation as a Service (LM-Eval-aaS) feature integrated into the TrustyAI Operator. It offers a unified framework for testing generative language models across a wide variety of evaluation tasks.
9+
You can use LM-Eval through the {productname-long} dashboard or the command line interface (CLI).
10+
These instructions are for using the dashboard.
11+
12+
13+
ifndef::upstream[]
14+
[IMPORTANT]
15+
====
16+
ifdef::self-managed[]
17+
Model evaluation through the dashboard is currently available in {productname-long} {vernum} as a Technology Preview feature.
18+
endif::[]
19+
ifdef::cloud-service[]
20+
Model evaluation through the dashboard is currently available in {productname-long} as a Technology Preview feature.
21+
endif::[]
22+
Technology Preview features are not supported with {org-name} production service level agreements (SLAs) and might not be functionally complete.
23+
{org-name} does not recommend using them in production.
24+
These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
25+
For more information about the support scope of {org-name} Technology Preview features, see link:https://access.redhat.com/support/offerings/techpreview/[Technology Preview Features Support Scope].
26+
====
27+
endif::[]
28+
29+
30+
.Prerequisites
31+
32+
* You have logged in to {productname-long} with administrator privileges.
33+
34+
ifdef::upstream[]
35+
* You have enabled the TrustyAI component, as described in link:{odhdocshome}/monitoring-data-science-models/#enabling-trustyai-component_monitor[Enabling the TrustyAI component].
36+
endif::[]
37+
ifndef::upstream[]
38+
* You have enabled the TrustyAI component, as described in link:{rhoaidocshome}{default-format-url}/monitoring_data_science_models/configuring-trustyai_monitor#enabling-trustyai-component_monitor[Enabling the TrustyAI component].
39+
endif::[]
40+
41+
* You have created a data science project in {productname-short}.
42+
43+
* You have deployed an LLM model in your data science project.
44+
45+
ifdef::upstream[]
46+
[NOTE]
47+
--
48+
By default, the *Model evaluations* option is hidden from the dashboard navigation menu. To show the *Model evaluations* option in the dashboard, go to the `OdhDashboardConfig` custom resource (CR) in {productname-long} and set the `disableLMEval` value to `false`. For more information about enabling dashboard configuration options, see link:{odhdocshome}/managing-resources/#ref-dashboard-configuration-options_dashboard[Dashboard configuration options].
49+
--
50+
endif::[]
51+
ifndef::upstream[]
52+
[NOTE]
53+
--
54+
By default, the *Model evaluations* option is hidden from the dashboard navigation menu. To show the *Model evaluations* option in the dashboard, go to the go to the `OdhDashboardConfig` custom resource (CR) in {productname-long} and set the `disableLMEval` value to false. or more information about enabling dashboard configuration options, see link:{rhoaidocshome}{default-format-url}/managing_openshift_ai/customizing-the-dashboard#ref-dashboard-configuration-options_dashboard[Dashboard configuration options].
55+
--
56+
endif::[]
57+
58+
.Procedure
59+
60+
. In the dashboard, click *Models* > *Model evaluation runs*. The Model evaluation page appears. It contains:
61+
62+
.. A *Start evaluation run* button. If you have not run any previous evaluations, only this button appears.
63+
64+
.. A list of evaluations you have previously run, if any exist.
65+
66+
.. A *Project* dropdown option you can click to show the evaluations relating to one project instead of all projects.
67+
68+
.. A filter to sort your evaluations by model or evaluation name.
69+
70+
+
71+
The following table outlines the elements and functions of the evaluations list:
72+
73+
.Evaluations list components
74+
[cols="1,4"]
75+
|===
76+
| Property | Function
77+
78+
| Evaluation
79+
| The name of the evaluation.
80+
81+
| Model
82+
| The model that was used in the evaluation.
83+
84+
| Evaluated
85+
| The date and time when the evaluation was created.
86+
87+
| Status
88+
| The status of your evaluation: running, completed, or failed.
89+
90+
| More options icon
91+
| Click this icon to access the options to delete the evaluation, or download the evaluation log in JSON format.
92+
|===
93+
--
94+
--
95+
96+
[start=2]
97+
. From the *Project* dropdown menu, select the namespace of the project where you want to evaluate the model.
98+
99+
. Click the *Start evaluation run* button. The Model evaluation form appears.
100+
101+
. Fill in the details of the form. The model argument summary appears after you complete the form details:
102+
103+
.. *Model name*: Select a model from all the deployed LLMs in your project.
104+
105+
.. *Evaluation name*: Give your evaluation a unique name.
106+
107+
.. *Tasks*: Choose one or more evaluation tasks against which to measure your LLM. The 100 most common evaluation tasks are supported.
108+
109+
.. *Model type*: Choose the type of model based on the type of prompt-formatting you use:
110+
111+
... *Local-completion*: You assemble the entire prompt chain yourself. Use this when you want to evaluate models that take a plain text prompt and return a continuation.
112+
113+
... *Local-chat-completion*: The framework injects roles or templates automatically. Use this for models that simulate a conversation by taking a list of chat messages with roles like `user` and `assistant` and reply appropriately.
114+
115+
.. *Security settings*:
116+
117+
... *Available online*: Choose *enable* to allow your model to access the internet to download datasets.
118+
119+
... *Trust remote code*: Choose *enable* to allow your model to trust code from outside of the project namespace.
120+
+
121+
[NOTE]
122+
--
123+
The *Security settings* section is grayed out if the security option in global settings is set to `active`.
124+
--
125+
126+
+
127+
. Observe that a model argument summary appears as soon as you fill in the form details.
128+
129+
. Complete the tokenizer settings:
130+
131+
.. *Tokenized requests*: If set to `true`, the evaluation requests are broken down into tokens. If set to `false`, the evaluation dataset remains as raw text.
132+
133+
.. *Tokenizer*: Type the model's tokenizer URL that is required for the evaluations.
134+
135+
. Click *Evaluate*. The screen returns to the Model evaluation page of your project and your job appears in the evaluations list.
136+
+
137+
[NOTE]
138+
====
139+
* It can take time for your evaluation to complete, depending on factors including hardware support, model size, and the type of evaluation task(s). The status column reports the current status of the evaluation: _completed_, _running_, or _failed_.
140+
* If your evaluation fails, the evaluation pod logs in your cluster provide more information.
141+
====

0 commit comments

Comments
 (0)