[BUG] value_raw set to a variable

[clagio]

Describe the bug
I'm trying to set the value of value_raw to a variable instead of hardcoding it in the task, so I can reuse the same task for multiple rules. Unfortunately I get the error "Unsupported type. Field must be string". The same task with the value_raw explicitly set works fine. Seems to be an escaping issue, and I'm not 100% the issue is with the collection honestly.. but I'm banging my head on this so any help is appreciated.

Component Name
tribe29.checkmk.rule

Ansible Version

$ ansible --version
ansible 2.10.8
python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]

Checkmk Version

Checkmk Raw Edition 2.1.0p13

Collection Version

$ ansible-galaxy collection list
tribe29.checkmk   0.11.0

To Reproduce
Steps to reproduce the behavior:
Execute this task:

    - name: "Create a rule."
      tribe29.checkmk.rule:
        server_url: "{{ hostvars['checkmk'].checkmk_url }}"
        site: "{{checkmk_site}}"
        automation_user: "{{ hostvars['checkmk'].checkmk_user }}"
        automation_secret: "{{ hostvars['checkmk'].checkmk_secret }}"
        ruleset: "checkgroup_parameters:memory_percentage_used"
        rule:
            properties: {
                "comment": "Warning at 80%\nCritical at 90%\n",
                "description": "Allow higher memory usage",
                "disabled": false
            }
            value_raw: "{{ myvariable }}"
        state: "present"
      vars:
        - myvariable: "{'levels': (80.0, 90.0)}"

If I replace value_raw: "{{ myvariable }}" to value_raw: "{'levels': (80.0, 90.0)}" the task works fine.

Expected behavior
Rule to be created using the value in "myvariable"

Actual behavior
The following error is printed:

"msg": "Error calling API. HTTP code 400. Details: b'{\"title\": \"Bad Request\", \"status\": 400, \"detail\": \"These fields have problems: value_raw\", \"fields\": {\"value_raw\": [\"Unsupported type. Field must be string.\"]}}', "

Additional context
I suspect the issue is caused by the curly bracket required by the API which make ansible interpret the variable's value as a dictionary instead of a string. But I couldn't find a way to escape that and make it works with the API.

[muehlings]

Hello clagio,

can you please try to write the rule as dict(key:list)?
vars:
- myvariable: {'levels': [80,90]}

Tuples and ansible are kind of impossible and I suspect that the API call will be JSON-encoded anyways. JSON doesn't know tuples.

The debug output in ansible is the output of json.dumps() and thus the original type of the object isn't preserved. At the moment I have no 2.1 instance available for testing and 2.0-API does not provide the rules.

[clagio]

Hi, I tried removing the double quotes s you asked, but I get the same error "Unsupported type. Field must be string" but in this case I see from the debug info that all the output is mixed up.
there is a way to see the raw output in ansible before any transformation?

I get the same error also with a different rule (ruleset special_agents:aws) if I try to move the access_key_id in a variable`. As before, if I hardcode the value, it works fine:

value_raw: "{
              'access_key_id': '{{ hostvars['aws-rds'].aws_access_key }}',
              'assume_role': {},
              'global_services': {},
              'regions': ['eu-west-1'],
              'secret_access_key': ('password', 'xxxxxxxxxxxxxxxxxxxx'), 
              'services': {
                'rds': {'limits': True, 'selection': 'all'}
              }"

[muehlings]

Have you removed the "()" and replaced them with "[]", too? Ansible doesn't know the tuple-notation (tup1,tup2) of python. The line in YAML is then interpreted as string and mixed up.

Edit: Does not work. See next comment for solution.

[muehlings]

I set up a 2.1 instance, modified the source and achieved this:

  - name: "Create a rule."
    tribe29.checkmk.rule:
      server_url: "http://127.0.0.1/"
      site: "test"
      automation_user: "automation"
      automation_secret: "$SECRET"
      ruleset: "checkgroup_parameters:memory_percentage_used"
      rule:
          properties: {
              "comment": "Warning at 81%\nCritical at 91%\n",
              "description": "Allow higher memory usage",
              "disabled": false
          }
          value_raw: "{{ myvariable }}" #"{'levels': (80.0, 90.0)}"  #"{{ myvariable }}"
      state: "present"
    vars:
      - myvariable: {'levels': [81,91]}

=>

changed: [debian] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "automation_secret": "$SECRET",
            "automation_user": "automation",
            "rule": {
                "conditions": {
                    "host_labels": [],
                    "host_tags": [],
                    "service_labels": []
                },
                "folder": "/",
                "properties": {
                    "comment": "Warning at 81%\nCritical at 91%\n",
                    "description": "Allow higher memory usage",
                    "disabled": false
                },
                "value_raw": {
                    "levels": [
                        81,
                        91
                    ]
                }
            },
            "ruleset": "checkgroup_parameters:memory_percentage_used",
            "server_url": "http://127.0.0.1/",
            "site": "test",
            "state": "present",
            "validate_certs": true
        }
    },
    "msg": "Created rule"

Source modification:

def get_existing_rule(module, base_url, headers, ruleset, rule):
    # Get rules in ruleset
    rules = get_rules_in_ruleset(module, base_url, headers, ruleset)
    if rules is not None:
        # Loop through all rules
        for r in rules.get("value"):
            # Check if conditions, properties and values are the same
            if type(rule["value_raw"]) == str:
                pass
            else:
                # Input is YAML variable. Need to convert to string...
                rule["value_raw"] = str(rule["value_raw"]).replace('[','(').replace(']',')')
            if (
                sorted(r["extensions"]["conditions"]) == sorted(rule["conditions"])
                and sorted(r["extensions"]["properties"]) == sorted(rule["properties"])
                and sorted(r["extensions"]["value_raw"]) == sorted(rule["value_raw"])
            ):
                # If they are the same, return the ID
                return r["id"]
    return None

and

def create_rule(module, base_url, headers, ruleset, rule):
    api_endpoint = "/domain-types/rule/collections/all"

    params = {
        "ruleset": ruleset,
        "folder": rule["folder"],
        "properties": rule["properties"],
        "conditions": rule["conditions"],
    }

    if type(rule["value_raw"]) == str:
        params["value_raw"] = rule["value_raw"]
    else:
        # Input is YAML variable. Need to convert to string...
        params["value_raw"] = str(rule["value_raw"]).replace('[','(').replace(']',')')

    url = base_url + api_endpoint

    response, info = fetch_url(
        module, url, module.jsonify(params), headers=headers, method="POST"
    )

    if info["status"] != 200:
        exit_failed(
            module,
            "Error calling API. HTTP code %d. Details: %s, %s"
            % (info["status"], info["body"], module.jsonify(params)),
        )

Now I have two options:

1. Write value_raw directly as string => value_raw: "{'levels': (80.0, 90.0)}"
2. Define a YAML(!) variable and reference it in my playbook => value_raw: "{{ myvariable }}" with

    vars:
      - myvariable: {'levels': [81,91]}

My modifications construct the proper string for the API including replacement of the square brackets (=python/YAML list) with the round brackets (=python tuple).

[clagio]

thanks, I modified the source as you did, and indeed now the first task( myvariable: {'levels': [81,91]}) works, but I get an error with the following :

value_raw: "{
              'access_key_id': '{{ hostvars['aws-rds'].aws_access_key }}',
              'assume_role': {},
              'global_services': {},
              'regions': ['eu-west-1'],
              'secret_access_key': ('password', 'xxxxxxxxxxxxxxxxxxxx'), 
              'services': {
                'rds': {'limits': True, 'selection': 'all'}
              }"

I will have a look later at the sources to see if I can find why

[clagio]

I had a better look, your change replace all squared brackets with rounded brackets, which means this:
'regions': ['eu-west-1'], becomes 'regions': ('eu-west-1'), and it fails.
It's tricky because depending on the rule we may need a replace or not..

I have also an issue where it keep creating the rule even if already exists, but I haven't narrowed down the problem yet. Would make sense to use the description field to detect if the rule already exists instead of matching conditions/properties/value_raw?

[muehlings]

Yes, you are right.

The only clean way to get this resolved in my opinion is that the API of Checkmk should accept and deliver JSON and not a string which is "evaluated" into a python object. There was some discussion about that in #153
I don't know the reason for the decision to not modifying the API.

The second possibility will be to get rid of all tuples inside the rules. But this is a lot more work to do I think.

I have problems to update the rule without my modifications, too. If only a single character or value differs in "conditions", "properties" or "value_raw" a new rule is created.

[clagio]

I agree the cleanest way is to fix it at the API level..
I think the different parameters in the value_raw are handled differently by the API, for example:
'regions': ['eu-west-1', 'eu-west-2'], need to stay with the squared brackets (will fails with the rounded brackets )
'secret_access_key': ['password', 'xxxxxxxxxxxxxxxxxxxx'], need to be with rounded brackets or will fail.
But from my understanding there is no difference between the two. I can only suppose secret_access_key is handled different in the API.

Anyway I did some changes and now I have a rule.py that works in all my cases. It's not super clean, but I'm not sure there is a better way without modifying the API. I also changed so that the rule matching (if already exists) is done only comparing the description.

There are several problem trying to compare conditions, properties and value_raw. Value raw has all the problems with escaping chars, and since there are different possible parameters I suspect there is an high chance of mismatch.
Conditions seems to be problematic as well.. below the conditions returned by an existing rule and the one passed by the new rule. This mismatch doesn't happen for all the rules, not sure why, maybe depends on the ruleset type?

existing_rule - compare11:['host_labels', 'host_tags', 'service_labels']
new_rule - compare12:['host_labels']

It's not impossible to match against all the properties, but I think requires a lot of work to make it work for all the possible cases. I believe matching by description it's simple and effective.

I created a new function transform_value_raw, where I take the value_raw and change the brackets only for secret_access_key. Eventually new keys can be added if needed,

def transform_value_raw(value_raw): 
    if type(value_raw) == dict:
        concat_value_raw= "{"
        for key, value in value_raw.items():
            if key == "secret_access_key":
                concat_value_raw = concat_value_raw + "'" +str(key) + "': " +str(value).replace('[','(').replace(']',')')+ ", "
            else:
                if str(value).startswith('[') or str(value).startswith('{'):
                    concat_value_raw = concat_value_raw + "'" +str(key) + "': " +str(value)+ ", "
                else:
                    concat_value_raw = concat_value_raw + "'" +str(key) + "': '" +str(value)+ "', "
        concat_value_raw= concat_value_raw + "}"
        return concat_value_raw
    return value_raw

Modified the create_rule:

def create_rule(module, base_url, headers, ruleset, rule):
    api_endpoint = "/domain-types/rule/collections/all"
    
    value_raw = transform_value_raw(rule["value_raw"])

    params = {
        "ruleset": ruleset,
        "folder": rule["folder"],
        "properties": rule["properties"],
        "conditions": rule["conditions"],
        "value_raw": value_raw
    }
    params_json = json.dumps(params)

    url = base_url + api_endpoint

    response, info = fetch_url(
        module, url, params_json, headers=headers, method="POST"
    )

    if info["status"] != 200:
        exit_failed(
            module,
            "Error calling API. HTTP code %d. Details: %s, %s"
            % (info["status"], info["body"], module.jsonify(params)),
        )

Modified the get_existing_rule:

def get_existing_rule(module, base_url, headers, ruleset, rule):
    # Get rules in ruleset
    rules = get_rules_in_ruleset(module, base_url, headers, ruleset)
    if rules.get("extensions").get("found_rules") != 0 :
        # Loop through all rules
        for r in rules.get("value"):
            if ( r["extensions"]["properties"]["description"] == rule["properties"]["description"]):
                # If they have same description, return the ID
                return r["id"]
    return None

I'm happy to submit the change if is ok

[muehlings]

For all password fields I found out the following: You can just write the password as string without the ('password', 'YOURPASSWORD') notation:
/omd/sites/TEST/lib/python3/cmk/gui/plugins/wato/utils/init.py has a function to transform the string into the required tuple:
forth=lambda v: ("password", v) if not isinstance(v, tuple) else v

This is done automatically and removes at least one type of tuples you have to care about, if the API won't be changed in the future.

I try to find the relevant part in the API of https://github.com/tribe29/checkmk to make a PR there.

[clagio]

that's right! thanks, it's helpful at least for the passwords

[clagio]

Another tuple issue with the overall_tags:

'overall_tags': [('{{aws_tag_key}}', ['{{aws_tag_value}}'])],

The playbook runs fine and the rule is created but when I check the rule in checkmk the option "Restrict monitoring services by one of these AWS tags" is not flagged and I get the following message:

Unable to read current options of this rule. Falling back to default values. When saving this rule now, your previous settings will be overwritten. Problem was: Restrict monitoring services by one of these AWS tags: The datatype must be a tuple, but is list.

[msekania]

@muehlings,

Unfortunately string substitution does not solve the problem and creates other ones, because there are arrays (lists) too, not only tuples!
for example : expect_response: ['302', '403'] it should be an array exactly with [ , ] and not tuple with ( , ), otherwise it's again an error!

With this substitution for example

I agree though that API should deliver correctly formatted JSON.

Otherwise:

• one can also use [[ and ]] as an escape for tuple ( and ).
• additionally one can also solve the YAML to python string conversion for value_raw with jinja2 macro

[msekania]

Probably this is also helpful,

jinja2 template file with macros value_raw_gen.j2

{#- Generate value_raw for rule from yaml nested dictionary                                                                     -#}
{#- ########################################################################################################################### -#}
{%- macro generateline(elem)                                                                                                    -%}
{%-   if elem is mapping                                                                                                        -%}
{{-     '{' -}}
{%-      for el in elem | dict2items                                                                                            -%}
{%-        if el.value | lower != 'none'                                                                                        -%}
'{{          el.key }}': {{ generateline(el.value) }}
{%-        endif                                                                                                                -%}
{%-      endfor                                                                                                                 -%}
{{-     '}' -}},
{%-   elif ( elem is iterable ) and ( elem is not string )                                                                      -%}
{#-     check if it is real array, escaped, '[[]]' or just a substitute for tuple '[]'                                          -#}
{%-     if ( elem | count == 1 ) and ( elem[0] is not mapping ) and ( elem[0] is iterable ) and ( elem[0] is not string )       -%}
{{-       '(' -}}
{%-       for el in elem[0]                                                                                                     -%}
{{          generateline(el)[:(-1 if loop.last else none):] }}
{%-       endfor                                                                                                                -%}
{{-       ')' -}},
{%-     else                                                                                                                    -%}
{{-       '[' -}}
{%-       for el in elem                                                                                                        -%}
{{          generateline(el)[:(-1 if loop.last else none):] }}
{%-       endfor                                                                                                                -%}
{{-       ']' -}},
{%-     endif                                                                                                                   -%}
{%-   else                                                                                                                      -%}
{%-     if elem |int(-1) == -1                                                                                                  -%}
{%        if '"' in elem or "'" in elem                                                                                         -%}
{{-         elem }},
{%-       else                                                                                                                  -%}
{{          elem | to_json }},
{%-       endif                                                                                                                 -%}
{%-     else                                                                                                                    -%}
{{        elem }},
{%-     endif                                                                                                                   -%}
{%-   endif                                                                                                                     -%}
{%- endmacro                                                                                                                    -%}
{#- ########################################################################################################################### -#}
{#-                                                                                                                             -#}
{#- ########################################################################################################################### -#}
{%- if value_raw is defined                                                                                                     -%}
{{    generateline(value_raw)[:-1:] }}
{%- endif                                                                                                                       -%}

in ansible then

value_raw: "{{ lookup('template', 'value_raw_gen.j2') | string }}"