<unknown>:1: SyntaxWarning: invalid escape sequence '\$' (maybe rose or cylc-rose?)

[ColemanTom]

I'm not sure if this is the correct repo for this. Perhaps it should be rose?

If you have a workflow structure with

.
├── flow.cylc
├── opt
│   └── rose-suite-a.conf
└── rose-suite.conf

And opt/rose-suite.a.conf has a single entry

[template variables]
F='\${ABC:-\$VARIABLE}'

When validating the workflow like cylc validate -O a $PWD I get

<unknown>:1: SyntaxWarning: invalid escape sequence '\$'
Valid for cylc-8.6.0

This warning message does not appear when I use cylc-flow=8.5.2

cylc-flow-8.5.2-pyhb01cd2f_0.json
cylc-flow-base-8.5.2-pyh707e725_0.json
cylc-rose-1.6.0-pyh707e725_0.json
metomi-rose-2.5.0-pyh707e725_0.json
metomi-rose-base-2.5.0-pyh707e725_0.json

but does with this combination

cylc-flow-8.6.0-pyhb01cd2f_0.json
cylc-flow-base-8.6.0-pyh707e725_0.json
cylc-rose-1.7.0-pyh707e725_0.json
metomi-rose-2.6.0-pyh707e725_0.json
metomi-rose-base-2.6.0-pyh707e725_0.json

Running with --debug on, both combinations have the say ROSE_SUITE_VARIABLES result, so it doesn't seem to affect the running of a workflow.

# 8.6.0 combo
    + ROSE_SUITE_VARIABLES={'ROSE_VERSION': '2.6.0', 'ROSE_ORIG_HOST': 'host.bom.gov.au', 'F': '\\${ABC:-$VARIABLE}',

# 8.5.2 combo
    + ROSE_SUITE_VARIABLES={'ROSE_VERSION': '2.5.0', 'ROSE_ORIG_HOST': 'host.bom.gov.au', 'F': '\\${ABC:-$VARIABLE}',

[oliver-sanders]

I'm not sure if this is the correct repo for this. Perhaps it should be rose?

No problem, open the issue with your best guess and we'll shift it where it needs to go.

[oliver-sanders]

This is a warning that Python itself will raise whenever it encounters an unknown escape sequence. E.g:

$ cat mess.py 
FOO='\${ABC:-\$VARIABLE}'
$ PYTHONWARNINGS=all python mess.py
/var/tmp/tmp.DUUWiHGYQt/mess.py:1: DeprecationWarning: invalid escape sequence \$
  FOO='\${ABC:-\$VARIABLE}'

In this case, the warning is being raised by Python's literal_eval function when the variable is parsed from the rose-suite.conf file:

cylc-rose/cylc/rose/jinja2_parser.py

Line 280 in 9c22ea2

return python_literal_eval(value)

I have no idea why we have only started seeing this warning now, possibly a change to literal_eval in the core library? The warning is perfectly correct, this is an unknown escape sequence! However, it would appear to be harmless, Python is grumbling, but not erroring.

In a Python program, you can declare a literal string as a "raw" string which avoids the attempted escape sequence expansion:

$ cat mess.py 
FOO=r'\${ABC:-\$VARIABLE}'
$ PYTHONWARNINGS=all python mess.py

When reading from a file, you don't get that option, however, you can jump through some (bizzare) hoops the achieve the same ends:

with open('myfile', 'r') as myfile:
    string = myfile.read()
    raw_string codecs.escape_decode(bytes(string 'utf-8'))[0].decode('utf-8')

In theory, we could do this. However, this would enforce all template variables to become raw strings (we can have it one way, or the other, but not both). So if someone else was relying on escape sequences (e.g, if they wanted a newline in a string or have encoded a funky character), enforcing raw strings would break that.

So I'm not sure what we could / should do here?

We can stop it warning for this example, but doing so might break another example.