Skip to content

Commit 767e87b

Browse files
Shivam6693Shivam6693
authored
Merge pull request #32 from ibm-ecosystem-engineering/framework_platform_option_ht
Framework platform option
2 parents 177de06 + ab7f89a commit 767e87b

File tree

6 files changed

+67
-17
lines changed

6 files changed

+67
-17
lines changed

‎Framework/README.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,8 @@ The following prerequisites are required to run the tester:
5454
judge_type = rag_eval_answer_similarity
5555

5656
[WML_CRED]
57+
wml_platform = saas
58+
wml_user = ''
5759
wml_url = https://us-south.ml.cloud.ibm.com
5860
api_key = ibm_cloud_api_key
5961
project_id = watsonx.ai_project_id
@@ -70,9 +72,11 @@ The following prerequisites are required to run the tester:
7072
1. `rag_eval_answer_similarity`
7173
2. `rag_eval_answer_rating`
7274
3. `multi_turn_eval`
73-
6. `wml_url`: you watsonx.ai url: https://<your_region>.ml.cloud.ibm.com
74-
7. `api_key`: your IBM Cloud apikey: <https://cloud.ibm.com/iam/apikeys>
75-
8. `project_id`: you watsonx.ai project id: watsonx.ai project's Manage tab (Project -> Manage -> General -> Details)
75+
6. `wml_platform`: There are two options available: `saas` or `onpremise`. If you're using the IBM Watsonx platform, choose `saas`, but if you're using the on-premise Watsonx platform on CP4D, select `onpremise`.
76+
7. `wml_url`: you watsonx.ai url: https://<your_region>.ml.cloud.ibm.com
77+
8. `wml_user`: wml user is required when you choose the platform `onpremise`
78+
9. `api_key`: your IBM Cloud apikey: <https://cloud.ibm.com/iam/apikeys>
79+
10. `project_id`: you watsonx.ai project id: watsonx.ai project's Manage tab (Project -> Manage -> General -> Details)
7680

7781
3. Run the following to evaluate.
7882

‎Framework/answer_rating.py‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,9 @@ def batch_llm_answer_rating(model_id, input_data):
4141

4242
# instantiate wml connection
4343
wml_credentials = {
44-
"url": "https://us-south.ml.cloud.ibm.com",
44+
"url": config['WML_CRED']['wml_url'],
4545
"apikey": config['WML_CRED']['api_key']
46-
}
46+
}
4747

4848
project_id = config['WML_CRED']['project_id']
4949

@@ -58,12 +58,26 @@ def batch_llm_answer_rating(model_id, input_data):
5858
"stop_sequences": ['}']
5959
}
6060

61-
# instatiate llm
62-
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
61+
platform = config['WML_CRED']['wml_platform']
62+
if platform == "saas":
63+
# instatiate llm
64+
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
65+
url=wml_credentials['url'],
66+
project_id=project_id,
67+
model_id=llm_model_id,
68+
params=generate_parameters_1)
69+
elif platform == "onpremise":
70+
wml_user = config['WML_CRED']['wml_user']
71+
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
6372
url=wml_credentials['url'],
64-
project_id=project_id,
6573
model_id=llm_model_id,
74+
username=wml_user,
75+
instance_id='openshift',
76+
project_id=project_id,
77+
version="5.0",
6678
params=generate_parameters_1)
79+
else:
80+
raise Exception("Please set a correct value in config.ini [WML_CRED][wml_platform], correct values are `onpremise` or `saas` ")
6781

6882
input_data['Grade'] = None
6983
input_data['Explanation'] = None

‎Framework/answer_similarity.py‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ def batch_llm_answer_similarity(model_id, input_data):
3737

3838
# instantiate wml connection
3939
wml_credentials = {
40-
"url": "https://us-south.ml.cloud.ibm.com",
40+
"url": config['WML_CRED']['wml_url'],
4141
"apikey": config['WML_CRED']['api_key']
4242
}
4343

@@ -54,13 +54,27 @@ def batch_llm_answer_similarity(model_id, input_data):
5454
"stop_sequences": ['}']
5555
}
5656

57-
# instatiate llm
58-
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
57+
platform = config['WML_CRED']['wml_platform']
58+
if platform == "saas":
59+
# instatiate llm
60+
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
61+
url=wml_credentials['url'],
62+
project_id=project_id,
63+
model_id=llm_model_id,
64+
params=generate_parameters_1)
65+
elif platform == "onpremise":
66+
wml_user = config['WML_CRED']['wml_user']
67+
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
5968
url=wml_credentials['url'],
60-
project_id=project_id,
6169
model_id=llm_model_id,
70+
username=wml_user,
71+
instance_id='openshift',
72+
project_id=project_id,
73+
version="5.0",
6274
params=generate_parameters_1)
63-
75+
else:
76+
raise Exception("Please set a correct value in config.ini [WML_CRED][wml_platform], correct values are `onpremise` or `saas` ")
77+
6478
input_data['Grade'] = None
6579
input_data['Explanation'] = None
6680

‎Framework/config.ini‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ output_file_name = input_path_and_filename
66
judge_type = judge_type
77

88
[WML_CRED]
9+
wml_platform = saas
10+
wml_user = watsonx.user
911
wml_url = watsonx.ai_url
1012
api_key = ibm_cloud_api_key
1113
project_id = watsonx.ai_project_id

‎Framework/multi_turn_eval.py‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ def batch_llm_multi_turn_eval(model_id, input_data):
9191

9292
# instantiate wml connection
9393
wml_credentials = {
94-
"url": "https://us-south.ml.cloud.ibm.com",
94+
"url": config['WML_CRED']['wml_url'],
9595
"apikey": config['WML_CRED']['api_key']
9696
}
9797

@@ -108,12 +108,26 @@ def batch_llm_multi_turn_eval(model_id, input_data):
108108
"stop_sequences": ['}']
109109
}
110110

111-
# instatiate llm
112-
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
111+
platform = config['WML_CRED']['wml_platform']
112+
if platform == "saas":
113+
# instatiate llm
114+
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
115+
url=wml_credentials['url'],
116+
project_id=project_id,
117+
model_id=llm_model_id,
118+
params=generate_parameters_1)
119+
elif platform == "onpremise":
120+
wml_user = config['WML_CRED']['wml_user']
121+
llm_model = WatsonxLLM(apikey=wml_credentials['apikey'],
113122
url=wml_credentials['url'],
114-
project_id=project_id,
115123
model_id=llm_model_id,
124+
username=wml_user,
125+
instance_id='openshift',
126+
project_id=project_id,
127+
version="5.0",
116128
params=generate_parameters_1)
129+
else:
130+
raise Exception("Please set a correct value in config.ini [WML_CRED][wml_platform], correct values are `onpremise` or `saas` ")
117131

118132
input_data['Grade'] = None
119133

‎Framework/sample_config.ini‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ output_file_name = Framework/data/output/sample_rag_answer_similarity_output.xls
66
judge_type = rag_eval_answer_similarity
77

88
[WML_CRED]
9+
wml_platform = saas
10+
wml_user = ''
911
wml_url = https://us-south.ml.cloud.ibm.com
1012
api_key = ibm_cloud_api_key
1113
project_id = watsonx.ai_project_id

0 commit comments

Comments
 (0)