Convert declarative config env var subtitution syntax to ABNF (#4448)

Fixes #4027.

@open-telemetry/configuration-maintainers please take a look!

Author: jack-berg

CHANGELOG.md

@@ -27,6 +27,9 @@ release.
 
 ### SDK Configuration
 
+- Convert declarative config env var substitution syntax to ABNF.
+  ([#4448](https://github.com/open-telemetry/opentelemetry-specification/pull/4448))
+
 ### Common
 
 ### Supplementary Guidelines

specification/configuration/data-model.md

@@ -53,27 +53,65 @@ YAML configuration files MUST use file extensions `.yaml` or `.yml`.
 
 #### Environment variable substitution
 
-Configuration files support environment variables substitution for references
-which match the following PCRE2 regular expression:
+Configuration files support environment variables substitution for references,
+defined using
+the [Augmented Backus-Naur Form](https://tools.ietf.org/html/rfc5234):
 
-```regexp
-\$\{(?:env:)?(?<ENV_NAME>[a-zA-Z_][a-zA-Z0-9_]*)(:-(?<DEFAULT_VALUE>[^\n]*))?\}
+```abnf
+SUBSTITUTION-REF = "${" ["env:"] ENV-NAME [":-" DEFAULT-VALUE] "}"; valid substitution reference
+INVALID-SUBSTITUTION-REF = "${" *(VCHAR-WSP-NO-RBRACE) "}"; invalid substitution reference
+
+ENV-NAME = (ALPHA / "_") *(ALPHA / DIGIT / "_"); the name of the variable to be substituted
+DEFAULT-VALUE = *(VCHAR-WSP-NO-RBRACE); any number of VCHAR-WSP-NO-RBRACE
+VCHAR-WSP-NO-RBRACE = %x21-7C / "~" / WSP; printable chars and whitespace, except }
+
+ALPHA = %x41-5A / %x61-7A; A-Z / a-z
+DIGIT = %x30-39 ; 0-9
 ```
 
-The `ENV_NAME` MUST start with an alphabetic or `_` character, and is followed
-by 0 or more alphanumeric or `_` characters.
+`SUBSTITUTION-REF` defines a valid environment variable substitution reference:
+
+* Must start with `${`
+* Optionally followed by `env:`
+* Must follow with `REF-NAME`, the name of the environment variable to be
+  substituted
+  * Must start with alphabetic or `_` character
+  * Must follow with any number of alphanumeric or `_` characters
+* Optionally followed by default value
+  * Must start with `:-`
+  * Must follow with `DEFAULT-VALUE`, any number of printable characters and
+    whitespace except `}`
+* Must follow with `}`
+
+`INVALID-SUBSTITUTION-REF` defines an invalid environment variable substitution
+reference:
+
+* Must start with `${`
+* Must follow with any number of printable characters and whitespace except `}`
+* Must follow with `}`
+
+For convenience, `SUBSTITUTION-REF` and `INVALID-SUBSTITUTION-REF` are expressed
+below as a PCRE2 regular expressions. Note that these expressions are
+non-normative.
+
+```regexp
+// SUBSTITUTION-REF
+\$\{(?:env:)?(?<ENV-NAME>[a-zA-Z_][a-zA-Z0-9_]*)(:-(?<DEFAULT-VALUE>[^\n]*))?\}
+
+// INVALID-SUBSTITUTION-REF
+\$\{(?<INVALID_IDENTIFIER>[^}]+)\}
+```
 
 For example, `${API_KEY}` and `${env:API_KEY}` are valid, while `${1API_KEY}`
 and `${API_$KEY}` are invalid.
 
 Environment variable substitution MUST only apply to scalar values. Mapping keys
 are not candidates for substitution.
 
-The `DEFAULT_VALUE` is an optional fallback value which is substituted
-if `ENV_NAME` is null, empty, or undefined. `DEFAULT_VALUE` consists of 0 or
-more non line break characters (i.e. any character except `\n`). If a referenced
-environment variable is not defined and does not have a `DEFAULT_VALUE`, it MUST
-be replaced with an empty value.
+The `DEFAULT-VALUE` component of `SUBSTITUTION-REF` is an optional fallback
+value which is substituted if `ENV-NAME` is null, empty, or undefined. If a
+referenced environment variable is not defined and does not have
+a `DEFAULT-VALUE`, it MUST be replaced with an empty value.
 
 The `$` character is an escape sequence, such that `$$` in the input is
 translated to a single `$` in the output. The resolved `$` from an escape
@@ -83,7 +121,7 @@ to `${API_KEY}`, and the value of the `API_KEY` environment variable is NOT
 substituted. See table below for more examples. In practice, this implies that
 parsers consume the input from left to right, iteratively identifying the next
 escape sequence, and matching the content since the prior escape sequence
-against the environment variable substitution regular expression.
+against `SUBSTITUTION-REF`.
 
 For example, the pseudocode for processing input `$${FOO} ${BAR} $${BAZ}`
 where `FOO=a, BAR=b, BAZ=c` would resemble:
@@ -98,14 +136,10 @@ where `FOO=a, BAR=b, BAZ=c` would resemble:
   against `input.substring(15+2, input.length)="{BAZ}"` => `"{BAZ}"` and append
   to output. Return output: `"${FOO} b ${BAZ}"`.
 
-When parsing a configuration file that contains a reference not matching
-the references regular expression but does match the following PCRE2
-regular expression, the parser MUST return an empty result (no partial
-results are allowed) and an error describing the parse failure to the user.
-
-```regexp
-\$\{(?<INVALID_IDENTIFIER>[^}]+)\}
-```
+When parsing a configure file that contains a reference not
+matching `SUBSTITUTION-REF` but matching `INVALID-SUBSTITUTION-REF`, the parser
+must return an empty result (no partial results are allowed) and an error
+describing the parse failure to the user.
 
 Node types MUST be interpreted after environment variable substitution takes
 place. This ensures the environment string representation of boolean, integer,