Evaluation Script Fails Due to Invalid JSON Response from Authenticated Endpoint

[christopher-mierbach]

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

1. Clone the ai-rag-chat-evaluator repository.
2. Follow the setup instructions to install dependencies and configure the environment.
3. Run the evaluation script with the provided command:

   python -m scripts.evaluate --config=config.json
   

Any log messages given by the failure

@Myname ➜ /workspaces/ai-rag-chat-evaluator (main) $ python -m scripts evaluate --config=config.json
[17:57:29] INFO Running evaluation from config /workspaces/ai-rag-chat-evaluator/config.json
INFO Replaced results_dir in config with timestamp
INFO Using Azure OpenAI Service with Azure Developer CLI Credential
INFO Running evaluation using data from /workspaces/ai-rag-chat-evaluator/example_input/qa.jsonl
INFO Sending a test question to the target to ensure it is running...
ERROR Failed to send a test question to the target due to error:
Response from target https://MYBACKEND.azurewebsites.net/chat is not valid JSON:
Make sure that your configuration points at a chat endpoint that returns a single JSON object.
ERROR Evaluation was terminated early due to an error ⬆

Expected/desired behavior

The evaluation script should successfully communicate with the chat endpoint, and the evaluation should proceed without errors.

OS and Version?

Ubuntu 20.04 LTS

Versions

Python: 3.10
Scripts version: Latest from the main branch as of 17.07.2024

Mention any other details that might be useful

The target_url in my config.json points to https://MYBACKEND.azurewebsites.net/chat.
The chat endpoint should return a single JSON object but seems not to be in the expected format.
The application is deployed as per the instructions in the repository documentation.
The app at https://MYBACKEND.azurewebsites.net has enabled authentication which might be affecting the evaluation script.

[pamelafox]

Yeah, an authenticated endpoint is tricky. I know one developer got an approach working here:
Azure-Samples/azure-search-openai-demo#1578

I can try to make that an officially supported thing here.

[christopher-mierbach]

Thank you - I was able to reach the authenticated endpoint with the approach you linked.
In my view it would be a great thing to make this solution official!
I guess authenticated endpoints are a standard for most.

[DuboisABB]

+1. Same request here.

[pamelafox]

Matt just updated the login docs for azure-search-openai-demo to share another way to get a token:
https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/login_and_acl.md#programmatic-access-with-authentication

That only works if you disable built-in auth, however, and use MSAL SDK only. We haven't determined why it's no working with built-in auth yet, we're chatting with App Service team.

[christopher-mierbach]

Thanks for the heads up, @pamelafox!
I am currently in the middle of a customer project (the app uses app-built-in authentification) and we are about to start technical metric-based testing.
Please keep us up to date.

[DuboisABB]

I tried the newly documented approach to get a token (after upgrading to the latest commit and setting AZURE_DISABLE_APP_SERVICES_AUTHENTICATION to true).
This is what I'm getting

[11:46:42] WARNING  DefaultAzureCredential failed to retrieve a token from the included credentials.      chained.py:123
                    Attempted credentials:
                            EnvironmentCredential: EnvironmentCredential authentication unavailable.
                    Environment variables are not fully configured.
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to
                    troubleshoot this issue.
                            ManagedIdentityCredential: ManagedIdentityCredential authentication
                    unavailable, no response from the IMDS endpoint.
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication
                    unavailable. No accounts were found in the cache.
                            AzureCliCredential: ERROR: AADSTS500011: The resource principal named
                    api://d9d64278-xxx-0b06169a315e/access_as_user was not found in the tenant
                    named ABB. This can happen if the application has not been installed by the
                    administrator of the tenant or consented to by any user in the tenant. You might have
                    sent your authentication request to the wrong tenant. Trace ID:
                    7be1db17-xxx-ad14-0deb55cdc700 Correlation ID:
                    2e1bb066-xxx-8aeb-5e6a696466b9 Timestamp: 2024-08-28 15:46:40Z
                    Interactive authentication is needed. Please run:
                    az login --scope api://d9d64278-xxx-a560-0b06169a315e/access_as_user/.default

                            AzurePowerShellCredential: Az.Account module >= 2.2.0 is not installed
                            AzureDeveloperCliCredential:
                    {"type":"consoleMessage","timestamp":"2024-08-28T11:46:42.6244974-04:00","data":{"mes
                    sage":"\nERROR: fetching token: reauthentication required, run `azd auth login
                    --scope api://d9d64278-xxx-0b06169a315e/access_as_user` to log in\n"}}

I tried to run az login as instructed in the error message but then I get:
AADSTS500011: The resource principal named api://d9d64278-xxx-0b06169a315e/access_as_user was not found in the tenant named ABB. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant. Trace ID: 593010fb-xxx-add83a4f1f00 Correlation ID: 0940ebfc-xxx-0ae931bcd801 Timestamp: 2024-08-28 15:50:46Z

So I tried to remove access_as_user (I'm not sure it should be there). Different warning now:

[11:49:21] WARNING  DefaultAzureCredential failed to retrieve a token from the included credentials.      chained.py:123
                    Attempted credentials:
                            EnvironmentCredential: EnvironmentCredential authentication unavailable.
                    Environment variables are not fully configured.
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to
                    troubleshoot this issue.
                            ManagedIdentityCredential: ManagedIdentityCredential authentication
                    unavailable, no response from the IMDS endpoint.
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication
                    unavailable. No accounts were found in the cache.
                            AzureCliCredential: ERROR: AADSTS65001: The user or administrator has not
                    consented to use the application with ID '04b07795-xxx-02f9e1bf7b46' named
                    'Microsoft Azure CLI'. Send an interactive authorization request for this user and
                    resource. Trace ID: 044a0774-xxx-54a1f4841a00 Correlation ID:
                    d4597997-xxx-4ba450df9b9a Timestamp: 2024-08-28 15:49:19Z
                    Interactive authentication is needed. Please run:
                    az login --scope api://d9d64278-xxx-0b06169a315e/.default

                            AzurePowerShellCredential: Az.Account module >= 2.2.0 is not installed
                            AzureDeveloperCliCredential:
                    {"type":"consoleMessage","timestamp":"2024-08-28T11:49:21.9571004-04:00","data":{"mes
                    sage":"\nERROR: fetching token: failed to authenticate:\n(invalid_resource)
                    AADSTS500011: The resource principal named api:/ was not found in the tenant named
                    372ee9e0-xxx-c07073a91ecd. This can happen if the application has not been
                    installed by the administrator of the tenant or consented to by any user in the
                    tenant. You might have sent your authentication request to the wrong tenant. Trace
                    ID: a2cbc92b-xxx-3f4e98169c00 Correlation ID:
                    bd1ac5b0-xxx-a14b7c406540 Timestamp: 2024-08-28 15:49:21Z\n\n"}}

                    To mitigate this issue, please refer to the troubleshooting guidelines here at
                    https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.

Also hinting to do az login, which I do. Now I get a different error during az login:
AADSTS650057: Invalid resource. The client has requested access to a resource which is not listed in the requested permissions in the client's application registration. Client app ID: 04b07795-xxx-02f9e1bf7b46(Microsoft Azure CLI). Resource value from request: api://d9d64278-xxx-0b06169a315e. Resource app ID: d9d64278-xxx-a560-0b06169a315e. List of valid resources from app registration: . Trace ID: 4a4e91cd-xxx-280648dbd700 Correlation ID: acb7fd77-xxx-cb9af401f197 Timestamp: 2024-08-28 15:53:47Z

For reference, my server and client IDs are:
AZURE_CLIENT_APP_ID="a76a1345-xxx-b39d2a46a015"
AZURE_SERVER_APP_ID="d9d64278-xxx-0b06169a315e"

I'm running out of time to troubleshoot now but I still want to update on my progress. I will continue later today.

[DuboisABB]

I got the furthest using this method to get a token:

def get_access_token():
    server_app_id = os.getenv("AZURE_SERVER_APP_ID")
    server_app_secret = os.getenv("AZURE_SERVER_APP_SECRET")
    tenant_id = os.getenv("AZURE_TENANT_ID")
    authority = f"https://login.microsoftonline.com/{tenant_id}"
    scope = [f"api://{server_app_id}/.default"]    

    # Create a confidential client application
    app = ConfidentialClientApplication(
        client_id=server_app_id,
        client_credential=server_app_secret,
        authority=authority,
    )

    # Acquire a token for the given scope
    result = app.acquire_token_for_client(scopes=scope)

    if "access_token" in result:
        access_token = result["access_token"]
        print(f"Access Token: {access_token}")  
        return access_token
    else:
        print(f"Failed to get access token. Error: {result.get('error')}, Description: {result.get('error_description')}")  
        return -1

This is what I get:

[19:38:19] INFO     Running evaluation from config                                                       evaluate.py:320
                    C:\Programming\ai-rag-chat-evaluator\bochat_dontknows.config.json
           INFO     Using Azure OpenAI Service with API Key from AZURE_OPENAI_KEY                    service_setup.py:19
           INFO     Running evaluation using data from                                                   evaluate.py:182
                    C:\Programming\ai-rag-chat-evaluator\bochat_input\qa_dontknows.jsonl
           INFO     Getting token...                                                                     evaluate.py:188
Access Token: (REDACTED)
[19:38:20] INFO     Token successful.                                                                    evaluate.py:191
           INFO     Sending a test question to the target to ensure it is running...                     evaluate.py:196
           ERROR    Failed to send a test question to the target due to error:                           evaluate.py:215
                    Response from target https://bochat5.azurewebsites.net/chat is not valid JSON:


                    Make sure that your configuration points at a chat endpoint that returns a single
                    JSON object.

           ERROR    Evaluation was terminated early due to an error ⬆  

I'm getting an empty reply (effectively the same error as OP).

[mattgotteiner]

Thanks for tagging me, I'll try to take a look

[DuboisABB]

@mattgotteiner do you plan on working on this in the short term? It would be extremely useful to be able to do evaluation runs on a securely deployed application. Or any advice on changes required to make it work?

[raul-bermejo]

+1, echo @DuboisABB's interest of running evals on a securely deployed app (with mfa login + private endpoints)