{{ 🧩 }} Add custom jinja filter for shuffling elements of your prompt

[johnnygreco]

doing something like below with a shuffle filter (which doesn't exist in standard Jinja, but can be implemented) would be helpful to drive diversity not just in Personas, but basically all use-cases we have today:

{% set items = [
    ('Name', first_name ~ ' ' ~ (middle_name if middle_name else '') ~ ' ' ~ last_name),
    ('Sex', sex),
    ('Age', age|string),
    ('Ethnic background', ethnic_background),
    ('Marital status', marital_status),
    ('Occupation', occupation),
    ('Location', city ~ ', ' ~ state ~ ', ' ~ county)
] %}

Based on a person with the following profile:
{% for label, value in items|shuffle %}
{% if value %}{{ label }}: {{ value }}
{% endfor %}

[eric-tramel]

Do you want to do it through Jinja, or do you think that there could be a way to handle direct, unparameterized/fixed operations on the dataset via the config builder? For instance, if a column is a list or a dictionary then doing

config_builder.shuffle(name="items")

could accomplish the same thing expediently without torturing the Jinja2?

I think Jinja filters are a fine thing to support, but it harms a bit of the discoverability & "agentability" of config definitions. For instance, Cursor or some future agent defining these steps is going to be less naturally aware of our specialized Jinja2 filters vs. the default set that are well represented online and in docs.

[johnnygreco]

This issue was ported from NMP based on a request from @3mei. In this particular case, I don't think a config operation is the way to go – that would add quite a bit of complexity to the final DD config. We also need to limit how much cooking happens in the config builder to play nice with the microservice (this has been a gotcha for us in the past). That said, I also worry about torturing the Jinja2, as you say. We'll want to keep any builtin custom filters simple and intuitive.

[eric-tramel]

I see, yes, it could be defined in other ways. But we've been using Jinja as a bit of an escape hatch for needed flexibility, but considering the context switching requirements and the challenges some users have with implementing Jinja and wondering what is / isn't supported, etc, I would really be in favor of almost anything other than piling more features into Jinja.

[johnnygreco]

lol, it's funny because you sound like me a year ago 😅 I totally agree with this instinct. Perhaps it's a slippery slope going down this path, as it opens the door to a wild west of jinja filters. Another option would be leveraging the prompt rendering recipes. Or maybe this particular use case is possible with list data types and standard jinja 🤔 Very open to explore other options.