Skip to content

Assertion message from JMESPathCheck is misleading when actual result evaluates to False. #29697

@tobiasb-ms

Description

@tobiasb-ms

Describe the bug

When running scenario tests for an extension I'm writing, with a check like this:

self.cmd(f"az some_command --output json", checks=[
    self.check("thing", True)
])

The exception when the actual value of thing is False is misleading:

>           raise JMESPathCheckAssertionError(self._query, self._expected_result, 'None',
                                              execution_result.output)
E           azure.cli.testsdk.exceptions.JMESPathCheckAssertionError: Query 'thing' doesn't yield expected value 'True', instead the actual value is 'None'. Data:
E           {
E             "otherThing": "value",
E             "thing": false
E           }

It would be better in this case to say "the actual value is False".

We encounter similar issues with empty strings, empty lists, empty dictionaries, the number 0 and generally anything that evaluates to False.

I've created a pull request with more information about why this happens and a proposed fix.

Related command

azdev test

Errors

      raise JMESPathCheckAssertionError(self._query, self._expected_result, 'None',
                                          execution_result.output)

E azure.cli.testsdk.exceptions.JMESPathCheckAssertionError: Query 'thing' doesn't yield expected value 'True', instead the actual value is 'None'. Data:
E {
E "otherThing": "value",
E "thing": false
E }


### Issue script & Debug output

I feel this is unnecessary because the error message makes it clear (and see linked pull request).

### Expected behavior

The error message should say the real value of the actual result -- In the provided example, that would be `'False'`.

### Environment Summary

az --version
azure-cli                         2.63.0

core                              2.63.0
telemetry                          1.1.0

Extensions:
aks-preview                      7.0.0b3
azure-devops                       1.0.1
<redacted>                        0.0.1 (dev) /workspace/code/<redacted>/<redacted>/src/<redacted>

Dependencies:
msal                              1.30.0
azure-mgmt-resource               23.1.1

Python location '/workspace/code/<redacted>/<redacted>/.venv/bin/python'
Extensions directory '/home/<redacted>/.azure/cliextensions'
Development extension sources:
    /workspace/code/<redacted>/<redacted>

Python (Linux) 3.9.19 (main, Jul 31 2024, 03:47:41)
[GCC 11.2.0]

Legal docs and information: aka.ms/AzureCliLegal


Your CLI is up-to-date.

### Additional context

_No response_

Metadata

Metadata

Assignees

Labels

Azure CLI TeamThe command of the issue is owned by Azure CLI teampotential-pruningquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions