Should Filters that work on lists should check for null value?

[ravensorb]

I noticed that a there are a few filters that work on lists/collections that do not check to see if the value being passed in a value and defined:

Example:

jinja/src/jinja2/filters.py

Line 460 in e20a52e

for item in value:

Any thoughts on adding a little defensive coding to these filters?

Reason behind the request is that when chaining filters together using | it can be difficult to validate if a list contains elements before passing it to the next filter.

[ThiefMaster]

I think passing a None/Undefined where you expect a list is a bug, not something jinja should silently allow. Being empty is perfectly fine though. Typically an empty list returns another empty list after applying a filter on it..

[ravensorb]

Ok, So how would you handle a series of filters on a collection that could under some scenarios filter all values out? Also, wouldn't it be more useful for all filters to behaving in a consistent way (ex: do_map check value before processing)?

[ThiefMaster]

Filtering out values removes them, it does not make them None. And if all values are filtered out it's an empty list, not None. So changing stuff together is usually fine.

[ravensorb]

Here is a code snippet:

{%- set networks = app.containers | default(None) | map(attribute="networks", default="") | default(None) | map("first") | default([]) | unique(attribute='name') | list %}

Explanation:

• app.containers can contain 0 or more objects that may or may not have the attribute "networks" (if no networks needed the attribute will not be defined)
• networks is a nested object collection so I want to "collapse" the nested collection into a single top list which is what I am using "first" to do
• unique is used to make sure there are no duplicates in the list

In the above snippet, passing default(None) allows the map filter to not through an error. That same is not the case with unique which throws an exception if the list is empty.

Maybe I am thinking about this wrong?

[ravensorb]

Also -- if I change default(None) to default([]) I still get an exception when "unique" is executed.

[ThiefMaster]

Taking that snippet apart:

• app.containers - can this be undefined?
• default(None) - this makes no sense unless the answer above is "yes". and in that case you probably want [] to get an empty list in that case
• map(attribute="networks", default="") - the default makes no sense, networks seems to be a list, so you want [] as a default if there's nothing
• default(None) - makes no sense, since map never returns undefined
• map("first") - so you want the first item from each such list
• default([]) - if there was no item, this gives you an empty list. if it's a nested list that seems reasonable
• unique(attribute='name') - making the elements unique by name; seems reasonable again

[ravensorb]

Answers to your questions

• app.containers - yes app.it can be none (there are other potential filters applied to it earlier)
• default(none) - this allowed me to get the "map" call to work
• map(attribute="networks", default="") - this is needed as app.containers is a list of complex objects that can contain nested objects that default networks. As for the default value - you are correct, that again was me trying different things as I had it set to [] and also None to see what happened.
• default(none) - removing this on the result of map makes sense, that was me trying different things to get map("first") to work
• map("first") - this pulls back the 1st item in each of the iterator results from the map call (it seems to come back as a [ [ { object } ], [ { object } ] type result so I wanted the 1st item in each one to come back as single list
• default([]) - agree this should not be needed as the prior map call should return at least an empty list
• unique(attribute='name') <- this is what is actually failing as what is being passed in is an empty sequence (actual error is: jinja2.exceptions.UndefinedError: No first item, sequence was empty.)

[ThiefMaster]

The bug would be []|unique|list failing (please test in jinja, without ansible first).

However, after looking at the code again, that exception happens when you do first on an empty list - map('first') then gives you undefined for that one. So you need to filter out those undefineds (since getting undefined.name in unique will fail). try adding |reject| before the unique filter - that should filter out anything falsy, including those undefineds

[ravensorb]

No problem -- this is all jina (no ansible) as it is a config tool I am building

Here is the full call stack if it helps

  File "/home/ravensorb/source/github/ravensorb/src/scripts/utils/TemplateManager.py", line 237, in __generateTemplateOutput
    f.write(jtpl.render(app=app, container=container, args=self.__args, basePath=os.path.abspath(os.path.dirname(outFile))))
  File "/home/ravensorb/.local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/ravensorb/.local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/ravensorb/.local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "./.templates/config.yml.j2", line 230, in top-level template code
    {%- set networks = app.containers | default(None) | map(attribute="networks", default=[]) | map("first") | default([{}]) | unique(attribute='name') | list %}
  File "/home/ravensorb/.local/lib/python3.8/site-packages/jinja2/filters.py", line 1016, in do_list
    return list(value)
  File "/home/ravensorb/.local/lib/python3.8/site-packages/jinja2/filters.py", line 372, in do_unique
    key = getter(item)
  File "/home/ravensorb/.local/lib/python3.8/site-packages/jinja2/filters.py", line 74, in attrgetter
    item = environment.getitem(item, part)
  File "/home/ravensorb/.local/lib/python3.8/site-packages/jinja2/environment.py", line 452, in getitem
    return obj[argument]
jinja2.exceptions.UndefinedError: No first item, sequence was empty.

[ravensorb]

Modified the line to:

{%- set networks = app.containers | default(None) | map(attribute="networks", default=[]) | map("first") | default([]) | reject | unique(attribute='name') | list %}

Still getting the exception in "do_unique" -> attrgetter -> getitem method

[ravensorb]

Couple of more interesting things

Tested the following

1. {%- set networks = [] | map(attribute="network", default=[]) | map("first") | unique(attribute="name") | list %}
2. {%- set networks = app.containers | map(attribute="networks", default=[]) | list %}
3. {%- set networks = app.containers | map(attribute="networks", default=[]) | map("first") | list %}
4. {%- set networks = app.containers | map(attribute="networks", default=[]) | map("first") | unique(attribute="name") | list %}
5. {%- set networks = app.containers | map(attribute="networks", default=[]) | map("first") | reject | unique(attribute="name") | list %}

1, 2, and 3 all do NOT throw an exception
4 and 5 both throw exceptions

It is interesting that 1 does not throw an exception as the only difference is [] vs app.containers as the starting object which seems a bit odd

Also looking at the call stack and the code a little more closely -- it looks like the issue is actually in the "do_first" method which is being called by the map filter. Does that sound correct?

[ravensorb]

Here is another one that is interesting
The following displays [Undefined] which means the if statement is returning true even though the pprint statement is dumping "[Undefined]"

{%- set networkList = app.containers | map(attribute="networks") | map("first") | default([]) | list %}
{% if networkList and networkList | length > 0 %}
{{ networkList | pprint }}
{% else %}
nothing found
{% endif %}

The following displays nothing found

{%- set networkList = [] | map(attribute="networks") | map("first") | default([]) | list %}
{% if networkList | length > 0 %}
{{ networkList | pprint }}
{% else %}
nothing found
{% endif %}