Skip to content

Commit e833024

Browse files
ogenstadogenstad
authored
Merge pull request #327 from opsmill/pog-jinja2-refactor
Refactor handling of Jinja2 templates and add filters
2 parents 84b8f40 + 66ac470 commit e833024

File tree

31 files changed

+1223
-65
lines changed

31 files changed

+1223
-65
lines changed

‎.vale/styles/spelling-exceptions.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ namespace
8585
namespaces
8686
Nautobot
8787
Netbox
88+
Netutils
8889
Newsfragment
8990
Nornir
9091
npm

‎changelog/+5660f1dc.added.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Refactored management of Jinja2 templating to allow for filters within Infrahub. Builtin filters as well as those from Netutils are available.

‎docs/_templates/sdk_template_reference.j2‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
title: Python SDK Templating
3+
---
4+
Filters can be used when defining [computed attributes](https://docs.infrahub.app/guides/computed-attributes) or [Jinja2 Transforms](https://docs.infrahub.app/guides/jinja2-transform) within Infrahub.
5+
6+
## Builtin Jinja2 filters
7+
8+
The following filters are those that are [shipped with Jinja2](https://jinja.palletsprojects.com/en/stable/templates/#list-of-builtin-filters) and enabled within Infrahub. The trusted column indicates if the filter is allowed for use with Infrahub's computed attributes when the server is configured in strict mode.
9+
10+
<!-- vale off -->
11+
| Name | Trusted |
12+
|----------|----------|
13+
{% for filter in builtin %}
14+
| {{ filter.name }} | {% if filter.trusted %}✅{% else %}❌{% endif %} |
15+
{% endfor %}
16+
<!-- vale on -->
17+
18+
## Netutils filters
19+
20+
The following Jinja2 filters from <a href="https://netutils.readthedocs.io">Netutils</a> are included within Infrahub.
21+
<!-- vale off -->
22+
| Name | Trusted |
23+
|----------|----------|
24+
{% for filter in netutils %}
25+
| {{ filter.name }} | {% if filter.trusted %}✅{% else %}❌{% endif %} |
26+
{% endfor %}
27+
<!-- vale on -->

‎docs/docs/python-sdk/reference/templating.mdx‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
---
2+
title: Python SDK Templating
3+
---
4+
Filters can be used when defining [computed attributes](https://docs.infrahub.app/guides/computed-attributes) or [Jinja2 Transforms](https://docs.infrahub.app/guides/jinja2-transform) within Infrahub.
5+
6+
## Builtin Jinja2 filters
7+
8+
The following filters are those that are [shipped with Jinja2](https://jinja.palletsprojects.com/en/stable/templates/#list-of-builtin-filters) and enabled within Infrahub. The trusted column indicates if the filter is allowed for use with Infrahub's computed attributes when the server is configured in strict mode.
9+
10+
<!-- vale off -->
11+
| Name | Trusted |
12+
|----------|----------|
13+
| abs | ✅ |
14+
| attr | ❌ |
15+
| batch | ❌ |
16+
| capitalize | ✅ |
17+
| center | ✅ |
18+
| count | ✅ |
19+
| d | ✅ |
20+
| default | ✅ |
21+
| dictsort | ❌ |
22+
| e | ✅ |
23+
| escape | ✅ |
24+
| filesizeformat | ✅ |
25+
| first | ✅ |
26+
| float | ✅ |
27+
| forceescape | ✅ |
28+
| format | ✅ |
29+
| groupby | ❌ |
30+
| indent | ✅ |
31+
| int | ✅ |
32+
| items | ❌ |
33+
| join | ✅ |
34+
| last | ✅ |
35+
| length | ✅ |
36+
| list | ✅ |
37+
| lower | ✅ |
38+
| map | ❌ |
39+
| max | ✅ |
40+
| min | ✅ |
41+
| pprint | ❌ |
42+
| random | ❌ |
43+
| reject | ❌ |
44+
| rejectattr | ❌ |
45+
| replace | ✅ |
46+
| reverse | ✅ |
47+
| round | ✅ |
48+
| safe | ❌ |
49+
| select | ❌ |
50+
| selectattr | ❌ |
51+
| slice | ✅ |
52+
| sort | ❌ |
53+
| string | ✅ |
54+
| striptags | ✅ |
55+
| sum | ✅ |
56+
| title | ✅ |
57+
| tojson | ❌ |
58+
| trim | ✅ |
59+
| truncate | ✅ |
60+
| unique | ❌ |
61+
| upper | ✅ |
62+
| urlencode | ✅ |
63+
| urlize | ❌ |
64+
| wordcount | ✅ |
65+
| wordwrap | ✅ |
66+
| xmlattr | ❌ |
67+
<!-- vale on -->
68+
69+
## Netutils filters
70+
71+
The following Jinja2 filters from <a href="https://netutils.readthedocs.io">Netutils</a> are included within Infrahub.
72+
<!-- vale off -->
73+
| Name | Trusted |
74+
|----------|----------|
75+
| abbreviated_interface_name | ✅ |
76+
| abbreviated_interface_name_list | ✅ |
77+
| asn_to_int | ✅ |
78+
| bits_to_name | ✅ |
79+
| bytes_to_name | ✅ |
80+
| canonical_interface_name | ✅ |
81+
| canonical_interface_name_list | ✅ |
82+
| cidr_to_netmask | ✅ |
83+
| cidr_to_netmaskv6 | ✅ |
84+
| clean_config | ✅ |
85+
| compare_version_loose | ✅ |
86+
| compare_version_strict | ✅ |
87+
| config_compliance | ✅ |
88+
| config_section_not_parsed | ✅ |
89+
| delimiter_change | ✅ |
90+
| diff_network_config | ✅ |
91+
| feature_compliance | ✅ |
92+
| find_unordered_cfg_lines | ✅ |
93+
| fqdn_to_ip | ❌ |
94+
| get_all_host | ❌ |
95+
| get_broadcast_address | ✅ |
96+
| get_first_usable | ✅ |
97+
| get_ips_sorted | ✅ |
98+
| get_nist_urls | ✅ |
99+
| get_nist_vendor_platform_urls | ✅ |
100+
| get_oui | ✅ |
101+
| get_peer_ip | ✅ |
102+
| get_range_ips | ✅ |
103+
| get_upgrade_path | ✅ |
104+
| get_usable_range | ✅ |
105+
| hash_data | ✅ |
106+
| int_to_asdot | ✅ |
107+
| interface_range_compress | ✅ |
108+
| interface_range_expansion | ✅ |
109+
| ip_addition | ✅ |
110+
| ip_subtract | ✅ |
111+
| ip_to_bin | ✅ |
112+
| ip_to_hex | ✅ |
113+
| ipaddress_address | ✅ |
114+
| ipaddress_interface | ✅ |
115+
| ipaddress_network | ✅ |
116+
| is_classful | ✅ |
117+
| is_fqdn_resolvable | ❌ |
118+
| is_ip | ✅ |
119+
| is_ip_range | ✅ |
120+
| is_ip_within | ✅ |
121+
| is_netmask | ✅ |
122+
| is_network | ✅ |
123+
| is_reversible_wildcardmask | ✅ |
124+
| is_valid_mac | ✅ |
125+
| longest_prefix_match | ✅ |
126+
| mac_normalize | ✅ |
127+
| mac_to_format | ✅ |
128+
| mac_to_int | ✅ |
129+
| mac_type | ✅ |
130+
| name_to_bits | ✅ |
131+
| name_to_bytes | ✅ |
132+
| name_to_name | ✅ |
133+
| netmask_to_cidr | ✅ |
134+
| netmask_to_wildcardmask | ✅ |
135+
| normalise_delimiter_caret_c | ✅ |
136+
| paloalto_panos_brace_to_set | ✅ |
137+
| paloalto_panos_clean_newlines | ✅ |
138+
| regex_findall | ❌ |
139+
| regex_match | ❌ |
140+
| regex_search | ❌ |
141+
| regex_split | ❌ |
142+
| regex_sub | ❌ |
143+
| sanitize_config | ✅ |
144+
| section_config | ✅ |
145+
| sort_interface_list | ✅ |
146+
| split_interface | ✅ |
147+
| uptime_seconds_to_string | ✅ |
148+
| uptime_string_to_seconds | ✅ |
149+
| version_metadata | ✅ |
150+
| vlanconfig_to_list | ✅ |
151+
| vlanlist_to_config | ✅ |
152+
| wildcardmask_to_netmask | ✅ |
153+
<!-- vale on -->

‎docs/sidebars-python-sdk.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ const sidebars: SidebarsConfig = {
3838
label: 'Reference',
3939
items: [
4040
'reference/config',
41+
'reference/templating',
4142
],
4243
},
4344
],

‎infrahub_sdk/ctl/cli_commands.py‎

Lines changed: 32 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@
99
from pathlib import Path
1010
from typing import TYPE_CHECKING, Any, Callable, Optional
1111

12-
import jinja2
1312
import typer
1413
import ujson
1514
from rich.console import Console
@@ -18,7 +17,6 @@
1817
from rich.panel import Panel
1918
from rich.pretty import Pretty
2019
from rich.table import Table
21-
from rich.traceback import Traceback
2220

2321
from .. import __version__ as sdk_version
2422
from ..async_typer import AsyncTyper
@@ -31,7 +29,7 @@
3129
from ..ctl.generator import run as run_generator
3230
from ..ctl.menu import app as menu_app
3331
from ..ctl.object import app as object_app
34-
from ..ctl.render import list_jinja2_transforms
32+
from ..ctl.render import list_jinja2_transforms, print_template_errors
3533
from ..ctl.repository import app as repository_app
3634
from ..ctl.repository import get_repository_config
3735
from ..ctl.schema import app as schema_app
@@ -44,8 +42,9 @@
4442
)
4543
from ..ctl.validate import app as validate_app
4644
from ..exceptions import GraphQLError, ModuleImportError
47-
from ..jinja2 import identify_faulty_jinja_code
4845
from ..schema import MainSchemaTypesAll, SchemaRoot
46+
from ..template import Jinja2Template
47+
from ..template.exceptions import JinjaTemplateError
4948
from ..utils import get_branch, write_to_file
5049
from ..yaml import SchemaFile
5150
from .exporter import dump
@@ -168,43 +167,28 @@ async def run(
168167
raise typer.Abort(f"Unable to Load the method {method} in the Python script at {script}")
169168

170169
client = initialize_client(
171-
branch=branch, timeout=timeout, max_concurrent_execution=concurrent, identifier=module_name
170+
branch=branch,
171+
timeout=timeout,
172+
max_concurrent_execution=concurrent,
173+
identifier=module_name,
172174
)
173175
func = getattr(module, method)
174176
await func(client=client, log=log, branch=branch, **variables_dict)
175177

176178

177-
def render_jinja2_template(template_path: Path, variables: dict[str, str], data: dict[str, Any]) -> str:
178-
if not template_path.is_file():
179-
console.print(f"[red]Unable to locate the template at {template_path}")
180-
raise typer.Exit(1)
181-
182-
templateLoader = jinja2.FileSystemLoader(searchpath=".")
183-
templateEnv = jinja2.Environment(loader=templateLoader, trim_blocks=True, lstrip_blocks=True)
184-
template = templateEnv.get_template(str(template_path))
185-
179+
async def render_jinja2_template(template_path: Path, variables: dict[str, Any], data: dict[str, Any]) -> str:
180+
variables["data"] = data
181+
jinja_template = Jinja2Template(template=Path(template_path), template_directory=Path())
186182
try:
187-
rendered_tpl = template.render(**variables, data=data) # type: ignore[arg-type]
188-
except jinja2.TemplateSyntaxError as exc:
189-
console.print("[red]Syntax Error detected on the template")
190-
console.print(f"[yellow] {exc}")
191-
raise typer.Exit(1) from exc
192-
193-
except jinja2.UndefinedError as exc:
194-
console.print("[red]An error occurred while rendering the jinja template")
195-
traceback = Traceback(show_locals=False)
196-
errors = identify_faulty_jinja_code(traceback=traceback)
197-
for frame, syntax in errors:
198-
console.print(f"[yellow]{frame.filename} on line {frame.lineno}\n")
199-
console.print(syntax)
200-
console.print("")
201-
console.print(traceback.trace.stacks[0].exc_value)
183+
rendered_tpl = await jinja_template.render(variables=variables)
184+
except JinjaTemplateError as exc:
185+
print_template_errors(error=exc, console=console)
202186
raise typer.Exit(1) from exc
203187

204188
return rendered_tpl
205189

206190

207-
def _run_transform(
191+
async def _run_transform(
208192
query_name: str,
209193
variables: dict[str, Any],
210194
transform_func: Callable,
@@ -227,7 +211,11 @@ def _run_transform(
227211

228212
try:
229213
response = execute_graphql_query(
230-
query=query_name, variables_dict=variables, branch=branch, debug=debug, repository_config=repository_config
214+
query=query_name,
215+
variables_dict=variables,
216+
branch=branch,
217+
debug=debug,
218+
repository_config=repository_config,
231219
)
232220

233221
# TODO: response is a dict and can't be printed to the console in this way.
@@ -249,15 +237,15 @@ def _run_transform(
249237
raise typer.Abort()
250238

251239
if asyncio.iscoroutinefunction(transform_func):
252-
output = asyncio.run(transform_func(response))
240+
output = await transform_func(response)
253241
else:
254242
output = transform_func(response)
255243
return output
256244

257245

258246
@app.command(name="render")
259247
@catch_exception(console=console)
260-
def render(
248+
async def render(
261249
transform_name: str = typer.Argument(default="", help="Name of the Python transformation", show_default=False),
262250
variables: Optional[list[str]] = typer.Argument(
263251
None, help="Variables to pass along with the query. Format key=value key=value."
@@ -289,7 +277,7 @@ def render(
289277
transform_func = functools.partial(render_jinja2_template, transform_config.template_path, variables_dict)
290278

291279
# Query GQL and run the transform
292-
result = _run_transform(
280+
result = await _run_transform(
293281
query_name=transform_config.query,
294282
variables=variables_dict,
295283
transform_func=transform_func,
@@ -410,7 +398,10 @@ def version() -> None:
410398

411399
@app.command(name="info")
412400
@catch_exception(console=console)
413-
def info(detail: bool = typer.Option(False, help="Display detailed information."), _: str = CONFIG_PARAM) -> None: # noqa: PLR0915
401+
def info( # noqa: PLR0915
402+
detail: bool = typer.Option(False, help="Display detailed information."),
403+
_: str = CONFIG_PARAM,
404+
) -> None:
414405
"""Display the status of the Python SDK."""
415406

416407
info: dict[str, Any] = {
@@ -476,10 +467,14 @@ def info(detail: bool = typer.Option(False, help="Display detailed information."
476467
infrahub_info = Table(show_header=False, box=None)
477468
if info["user_info"]:
478469
infrahub_info.add_row("User:", info["user_info"]["AccountProfile"]["display_label"])
479-
infrahub_info.add_row("Description:", info["user_info"]["AccountProfile"]["description"]["value"])
470+
infrahub_info.add_row(
471+
"Description:",
472+
info["user_info"]["AccountProfile"]["description"]["value"],
473+
)
480474
infrahub_info.add_row("Status:", info["user_info"]["AccountProfile"]["status"]["label"])
481475
infrahub_info.add_row(
482-
"Number of Groups:", str(info["user_info"]["AccountProfile"]["member_of_groups"]["count"])
476+
"Number of Groups:",
477+
str(info["user_info"]["AccountProfile"]["member_of_groups"]["count"]),
483478
)
484479

485480
if groups := info["groups"]:

0 commit comments

Comments
 (0)