Better Environment Variable workflow

[knolleary]

We use environment variables to make a flow more easily portable. However there are some gaps in the workflow that can make it quite awkward:

1. when creating a flow, not all properties can be set to an env var property as the node edit dialog doesn't provide a way to set one (eg boolean values).
2. when importing a flow that contains env vars, you do not get any help to know what env vars need to be set

I think there are things we could do to help here.

Some ideas for discussion:

Creating flows with env vars

1. Provide a way to edit a node's raw JSON configuration directly - would allow any property to be set to an env var identifier. The downside is you lose some of the protection of the node's edit dialog to ensure the configuration is self-consistent and valid.
2. Alternatively, add a step in the Export Flow process that allows any node property to be replaced with an env var. Downside here is you would have to do that every time you exported an updated version.

Importing a flow with env vars

1. When importing a flow, the import dialog could detect the presence of env vars and prompt the user to set some values - either on the containing Flow or as new Global env vars (see Global Environment Variable #76)

---

What else could we do to make env vars easier to work with?

[Steveorevo]

Not sure if I’m grasping the scenarios. is this for subflow import?

With regards to #1, “downside is you lose some of the protection of the node's edit dialog…”. That seems paramount. If we wanted to design a particular node’s property to be env variable enabled; couldn’t we have chosen such behavior from the start? I see that we can design nodes specifically with that intent or to keep it fixed to a narrow set of data types and/or abilities.

Perhaps that idea here is to introduce a new selector, an env variable with data type enforcement; as opposed to just an arbitrary env var?

A boolean’s simplicity, or a checkbox is straightforward. If the intent here is to make it an env var enabled one too; perhaps a new, subtle difference in the control’s appearance could be created. Like a thin drop down arrow to the right hand side of the checkbox that reveals a hidden field specifically for an env var entry input.

[knolleary]

is this for subflow import?

For someone importing any flow into the editor, there is currently no support given to help them know if what they have just imported requires env vars to be set - they have to rely on being told, or poking around everything they just imported to see if there are any env var references.

For someone exporting a flow who wants to do a last-minute substitution of local configuration with env vars, they have to currently manually edit the JSON. I think we can provide help with that.

If we wanted to design a particular node’s property to be env variable enabled; couldn’t we have chosen such behavior from the start

Yes, however there are 4000+ nodes in the community and the majority have given this no thought at all. So finding a solution that works in the general case is quite important.

[Steveorevo]

For "Importing a flow with environment variables"; that may be an ideal time to show a list of areas to be addressed; leveraging:

#79 (comment) "part of the url...fragment ends with /edit, [node's] edit dialog will be opened..."

[sammachin]

I'm not sure that opening up to modifying a nodes parameters directly in the JSON flow is a great idea, it feels like overstepping into the choices of the node-developer, we should be encouraging and making it easier for developers to use typed inputs (with env vars) in their nodes as best practice though.

[sammachin]

To elborate a little more on this my concern as a node developer would be the potential support overhead, if I have a field in my node that allows a user to select, Red, Green or Blue and the user now somehow manages to set this to Badger by modifying the JSON that could cause all kinds of strange errors

[HiroyasuNishiyama]

What else could we do to make env vars easier to work with?

Considering a large-scale flow, it seems to be difficult for developers to change node properties directly.
Instead, I think developers should explicitly use environment variables to build reusable flows.

Following features may be useful for that purpose:

• Specifying default value for environment variable like UNIX shell (e.g. ${VAR:=val}),
• Listing environment variables used within group or tab and adding,
• Adding comment and a flag that distinguishes environment variable used as parameter.

[dceejay]

We already have several tools around Node-RED that may be relevant like NRlint or more probably the Flow Inspector (on the flows site)... While not directly helpful for exporting - if the concept of the inspector was brought into the core editor (maybe as - About flow ???) - it could trawl the flow and produce a report that included the list of used nodes - non-core nodes - environment variables - subflows - any other potentially useful parameters . (web endpoints ? urls ?) etc

[Steve-Mcl]

Hi all, I would like to resurrect this design and move it forward please.

I am particularly interested in ability to select a config node in the env var parameters of a subflow as this would greatly enhance the ability to encapsulate functionality (set flows) where nodes have config choices.

This becomes especially important in supporting Dashboard type nodes in a subflow.

Supporting this capability would open doors to OEMs who create templates (sometimes called widgetize or dynamos in other HMI/SCADA packages)

As I see it, there are 2 parts to this:

1. Exposing the Config Select dropdown + edit button on the subflow instance
2. Providing a simple means of instructing the subflow template to use an ${env-var} instead of a fixed value

---

Part 1: I have already demonstrated part 1 locally in my dev env (short video to follow)

Part 2: requires further discussion and design but I see this similar to how @HiroyasuNishiyama does:

• Listing environment variables used within group or tab and adding,
• Adding comment and a flag that distinguishes environment variable used as parameter.

These 2 points align very closely with how I personally see this working.

1. Env Vars entered in the subflow are presented as a choice to the user when they chose a property to be supplied via env-var
2. The row is then flagged or highlighted as being overridden by an env-var

To make this clean, we might wish to consider introducing the feature via a new data-xyz element (in the form-row) that indicates to Node-RED this property is overridable. Think of it as a contract with the core that the users has explicitly stated they have implemented the form-row in such a way it would behave correctly should the element be chosen to be overridden?

Another consideration: Perhaps this is ONLY supported for form elements that are a typedInput or config-select. This gives us the ability (at the core) to control how the property is rendered / operated when set to be an env var?

---

I intend on "having a play" with this over the coming weeks to see how it feels. I will post ideas / questions / demos here in the hope I can get feedback and resolution.

Cheers all.

[Steve-Mcl]

Part 1: I have already demonstrated part 1 locally in my dev env (short video to follow)

As promised, this is the working demo I have regarding support for Configs in a subflow.

NR-EnvVar-part1.mp4

Comments (good and bad) most welcome.

[joepavitt]

SO good Steve ❤️

[Steve-Mcl]

And after some play around, a (very rough) visual demo of how we might support simplified choice/selection of Env Vars

NOTES:

• I have added the built-in RED.contextMenu to the first label of a form-row when the nodes defaults are scanned and linked up to the UI elements
• The drop down presented is build from the internal knowledge that the node is INSIDE a subflow template.
  • the entries are built from the parent subflow templates' env-vars defined by the user
• Upon selecting an entry from the dropdown, I hide all internal elements of the form-row EXCEPT the label (this will NOT suit all node edit panels since many nodes are loose with the standard - this is where we might need to introduce the feature via data- attributes or custom class names (still working that out))

NR-EnvVar-part2.mp4

[Steve-Mcl]

Additional notes

• In this demo, the simple list is built from the raw names (not the friendly UI labels) - I would definitely want to pick these up

Additional thoughts

Having had this swishing around my head, I have the additional thoughts:

• I realise this UX (kinda) forces the user to first enter Env Vars. We could have a menu item named New that inserts the new env var into the subflow config & enteres into the editing node.
• There is nothing to stop (or at minimum hint) what the expected value should be. For example, the SIZE is not really a single number but a CSV number (e.g. "1,12") under the hood.
  • Permitting the user to add a hint in the Env Var setup might be a useful addition
• There is no validation that the Env Var exists. For example, a user may enter ${Group_ID} instead of ${groupId}
• The list does not present Globally defined Env Vars.
  • Personally, I am ok with this as it encourages more portable designs
• We could also display the icon in the menu (the icon chosen by the user in the Env Var setup) to aid faster recognition
• As Right Click is not entirely touch friendly, we may wish to support Double Click and/or have permanent icon buttons that permits the selection & override of a property?

Missing bits:

Assuming we like this UX we need:

• a way of determining a property has been set to an env var so that when re-opening the node edit panel the correct UI is presented
• a way of remapping the new Env Var override element to the nodes default property

Reality Check

In reality, applying this pattern carte blanche, I can foresee issues in simply hiding the original content and adding another element for the Env Var value. In particuar, nodes that have reactive elements based on user selection. I am now leaning towards a new, well defined UI pattern that is deliberately designed to support the "drop list" Env Var selection UX. Any node that currently employs a free-text entry can be utilised with the existing ${xxx} syntax.

I would hope that, where possible, the core nodes would be updated to take advantage of this "quick builder" Env Var selection.

Stretch

in a group (or any node on a tab), would we want to offer the same level of support (Groups and Tags have Env Vars too!)

[joepavitt]

I realise this UX (kinda) forces the user to first enter Env Vars. We could have a menu item named New that inserts the new env var into the subflow config & enteres into the editing node.

I don't think that's necessarily a bad thing, as a first pass of this, it certainly isn't a blocker imo

The list does not present Globally defined Env Vars

Nor do I think it should, the scope here is about reusable subflows.

We could also display the icon in the menu (the icon chosen by the user in the Env Var setup) to aid faster recognition

Great idea

As Right Click is not entirely touch friendly, we may wish to support Double Click and/or have permanent icon buttons that permits the selection & override of a property?

I agree here, great for demonstration, but an icon of some kind would be preferred I think, but this may open concerns to a wider Node-RED sidebar aesthetics discussion

in a group (or any node on a tab), would we want to offer the same level of support (Groups and Tags have Env Vars too!)

This being a subflow-only feature as a first pass is plenty

[Steve-Mcl]

Progress on the design - I will write up later what that entails but here is a teaser video of how I imagine we can make the override by Env Var much more accessible and RAD for OEMs

I would greatly appreciate feedback on the progress (would really like to know I am not wasting my time) - thank yoooo. :)

rad-env-vars-in-subflow.mp4

[Steve-Mcl]

Bit confused by the two options here, does the bottom one effect the top?

Yes. But I have changed nothing in this area. Have a play with current node-red to see how these 2 interact.

[Steve-Mcl]

Rather than "Override", possible an "Inherit" label instead?

Hmmm. Inherit sounds a little bit sexier but it has other connotations. I think override is more accurate and explicit.

It could say "use env var" or something more friendly though.

[joepavitt]

Inherit sounds a little bit sexier but it has other connotations

What other connotations? It's being inherited from the parent/wrapping Subflow, so it's accurate?

Override feels like the opposite, that the node would "override" any value provided by the parent

[Steve-Mcl]

Override was chosen from the perspective of the node. As in "the nodes property is being overridden". Since the users action is, at that instant in time, operating on an existing node property.

I'm my mind saying "the node property is inherited" made me immediately think of OOO inheritance (like a class inherits a property from a parent class, not the value, the property itself).

But you make fair points & I am almost convinced, but i just don't feel comfortable with "inherit".

Happy to be inherited overridden if anyone else ever joins the thread 😉

[svdasein]

@Steve-Mcl hi - I'm just curious as to how this thing is progressing? It'd be an awesome addition to node-red! Any idea when it might make its way into a release?