WIP

Author: rrthomas

Filtering.md

@@ -0,0 +1 @@
+Change the syntax of `$include` to: `$include(command,arg,...){file}` (and similar for `$paste`). The semantics are that `file` is found and expanded, then the result is fed to `command arg...`'s stdin. If you omit the `{file}`, then you are just running a command and expanding/pasting its output.

README.md

@@ -110,25 +110,21 @@ a file.
 
 Nancy expands a template file as follows:
 
-1. Scan the file for commands. Expand any arguments to the command, run
-   each command, and replace the command by the result.
-2. Output the result.
+1. Scan the text for commands. Expand any arguments to the command, run each
+   command, and replace the command by the result, eliding any final
+   newline. (This elision may look tricky, but it almost always does what
+   you want, and makes `$include` behave better in various contexts.)
+2. Output the resultant text.
 
 A command takes the form `$COMMAND` or `$COMMAND{ARGUMENT, ...}`.
 
-Nancy treats its input as 8-bit ASCII, but command names and other
-punctuation only use the 7-bit subset. This means that any text encoding
-that is a superset of 7-bit ASCII can be used, such as UTF-8.
-
 ### Built-in commands
 
 Nancy recognises these commands:
 
 * *`$include{FILE}`* Look up the given source file in the input tree (see
   below); read its contents, then expand them (that is, execute any commands
-  it contains) and return the result, eliding any final newline. (This
-  elision may look tricky, but it almost always does what you want, and
-  makes $include behave better in various contexts.)
+  it contains) and return the result.
 * *`$paste{FILE}`* Like `$include`, but does not expand its result before
   returning it.
 * *`$path`* Expands to the file currently being expanded, relative to the

README.nancy.md

@@ -2,7 +2,7 @@
 
 ![logo](logo/nancy-small.png) _logo by Silvia Polverini_
 
-$paste{/bin/sh,-c,PYTHONPATH=. python -m nancy --version | tail +2 | head -2 | sed -e 's/$/  /'}
+$paste(/bin/sh,-c,PYTHONPATH=. python -m nancy --version | tail +2 | head -2 | sed -e 's/$/  /')
 
 Nancy is a simple templating system that copies a file or directory, filling
 in templates as it goes. It has just one non-trivial construct:
@@ -35,7 +35,7 @@ $ pip install nancy
 ## Invocation
 
 ```
-$paste{/bin/sh,-c,PYTHONPATH=. python -m nancy --help | sed -e 's/usage: nancy/nancy/'}
+$paste(/bin/sh,-c,PYTHONPATH=. python -m nancy --help | sed -e 's/usage: nancy/nancy/')
 ```
 
 ## Operation <a name="operation"></a>
@@ -99,7 +99,11 @@ Nancy expands a template file as follows:
    each command, and replace the command by the result.
 2. Output the result.
 
-A command takes the form `\$COMMAND` or `\$COMMAND{ARGUMENT, ...}`.
+A command is written as its name prefixed with a dollar sign: `\$COMMAND`.
+Some commands take an argument, given in braces: `\$COMMAND{ARGUMENT}`.
+Finally, some commands also take an optional external command and arguments:
+`\$COMMAND(EXTERNAL-COMMAND, ARGUMENT, …){ARGUMENT}`; the argument in braces
+is optional in this case.
 
 Nancy treats its input as 8-bit ASCII, but command names and other
 punctuation only use the 7-bit subset. This means that any text encoding
@@ -152,27 +156,24 @@ worked example.
 ### Running other programs
 
 In addition to the rules given above, Nancy also allows `\$include` and
-`\$paste` to take their input from programs. This can be useful in a variety
-of ways: to insert the current date or time, to make a calculation, or to
-convert a file to a different format.
+`\$paste` to run external programs, whose output becomes the result of the
+command. If an input is given, it is supplied to the program’s standard
+input. This can be useful in a variety of ways: to insert the current date
+or time, to make a calculation, or to convert a file to a different format.
 
-Nancy can run a program in two ways:
+Nancy looks for programs in two ways:
 
-1. If a file found by an `\$include` or `\$paste` command has the “execute”
-   permission, it is run.
+1. Using the same rules as for finding an `\$include` or `\$paste` input,
+   Nancy looks for a file which has the “execute” permission.
 
 2. If no file of the given name can be found using the rules in the previous
    section, Nancy looks for an executable file on the user’s `PATH` (the
-   list of directories specified by the `PATH` environment variable). If one
-   is found, it is run.
-
-In either case, arguments may be passed to the program: use
-`\$include{FILE,ARGUMENT_1,ARGUMENT_2,…}`, or the equivalent for `\$paste`.
+   list of directories specified by the `PATH` environment variable).
 
 For example, to insert the current date:
 
 ```
-\$paste{date,+%Y-%m-%d}
+\$paste(date,+%Y-%m-%d)
 ```
 
 See the [date example](Cookbook.md#date-example) in the Cookbook for more
@@ -188,14 +189,13 @@ To prevent a comma from being interpreted as an argument separator, put a
 backslash in front of it:
 
 ```
-\$include{cat,I\, Robot.txt,3 Rules of Robotics.txt}
+\$include(cat,I\, Robot.txt,3 Rules of Robotics.txt)
 ```
 
-This will run the `\$include` command with the following arguments:
+This will run the `cat` command with the following arguments:
 
-1. `cat`
-2. `I, Robot.txt`
-3. `3 Rules of Robotics.txt`
+1. `I, Robot.txt`
+2. `3 Rules of Robotics.txt`
 
 Note that the filenames supplied to `cat` refer not to the input tree, but
 to the file system.

nancy/__init__.py

@@ -86,6 +86,32 @@ def find_object(obj: Path) -> Optional[Union[Path, list[os.DirEntry[str]]]]:
             return None
         return sorted(list(dirents.values()), key=lambda x: sorting_name(x.name))
 
+    def parse_arguments(
+        text: bytes, arg_start: int, opening: int, closing: int
+    ) -> tuple[list[bytes], int]:
+        args = []
+        depth = 1
+        next_index = arg_start + 1
+        while next_index < len(text):
+            if text[next_index] == closing:
+                depth -= 1
+                if depth == 0:
+                    args.append(text[arg_start + 1 : next_index])
+                    break
+            elif text[next_index] == opening:
+                depth += 1
+            elif (
+                depth == 1
+                and text[next_index] == ord(b",")
+                and text[next_index - 1] != ord(b"\\")
+            ):
+                args.append(text[arg_start + 1 : next_index])
+                arg_start = next_index
+            next_index += 1
+        if next_index == len(text):
+            raise ValueError(f"missing '{closing}'")
+        return args, next_index + 1
+
     def expand_bytes(
         text: bytes,
         base_file: Path,
@@ -149,14 +175,16 @@ def read_file(
 
                 # Set up macros
                 macros: dict[bytes, Callable[..., bytes]] = {}
-                macros[b"path"] = lambda _args: bytes(base_file)
-                macros[b"realpath"] = lambda _args: bytes(file_path)
+                macros[b"path"] = lambda _args, _external_args: bytes(base_file)
+                macros[b"realpath"] = lambda _args, _external_args: bytes(file_path)
                 macros[b"outputpath"] = (
-                    lambda _args: bytes(output_path) if output_path is not None else b""
+                    lambda _args, _external_args: bytes(output_path)
+                    if output_path is not None
+                    else b""
                 )
 
                 def get_included_file(
-                    command_name: str, args: list[bytes]
+                    command_name: str, args: list[bytes],external_args: list[bytes]
                 ) -> tuple[Optional[Path], bytes]:
                     debug(f"${command_name}{{{b','.join(args)}}}")
                     if len(args) < 1:
@@ -166,8 +194,8 @@ def get_included_file(
                     file = get_file(Path(os.fsdecode(args[0])))
                     return read_file(file, args[1:])
 
-                def include(args: list[bytes]) -> bytes:
-                    file, contents = get_included_file("include", args)
+                def include(args: list[bytes], external_args: list[bytes]) -> bytes:
+                    file, contents = get_included_file("include", args, external_args)
                     return strip_final_newline(
                         inner_expand(
                             contents, expand_stack + [file] if file is not None else []
@@ -182,19 +210,26 @@ def paste(args: list[bytes]) -> bytes:
 
                 macros[b"paste"] = paste
 
-                def do_macro(macro: bytes, args: list[bytes]) -> bytes:
-                    debug(f"do_macro {macro} {args}")
+                def expand_args(args: list[bytes]) -> list[bytes]:
                     expanded_args: list[bytes] = []
                     for a in args:
                         # Unescape escaped commas
                         debug(f"escaped arg {a}")
                         unescaped_arg = re.sub(rb"\\,", b",", a)
                         debug(f"unescaped arg {unescaped_arg}")
                         expanded_args.append(do_expand(unescaped_arg))
+                    return expanded_args
+
+                def do_macro(
+                    macro: bytes, args: list[bytes], external_args: list[bytes]
+                ) -> bytes:
+                    debug(f"do_macro {macro} {args} {external_args}")
+                    expanded_args = expand_args(args)
+                    expanded_external_args = expand_args(external_args)
                     if macro not in macros:
                         decoded_macro = macro.decode("iso-8859-1")
                         raise ValueError(f"no such macro '${decoded_macro}'")
-                    return macros[macro](expanded_args)
+                    return macros[macro](expanded_args, expanded_external_args)
 
                 debug("do_match")
                 startpos = 0
@@ -206,38 +241,23 @@ def do_macro(macro: bytes, args: list[bytes]) -> bytes:
                     debug(f"match: {res} {res.end()}")
                     escaped = res[1]
                     name = res[2]
-                    arg_start = res.end()
-                    startpos = arg_start
+                    startpos = res.end()
                     args = []
-                    # Parse arguments, respecting nested commands
-                    if arg_start < len(expanded) and expanded[arg_start] == ord(b"{"):
-                        depth = 1
-                        next_index = arg_start + 1
-                        while next_index < len(expanded):
-                            if expanded[next_index] == ord(b"}"):
-                                depth -= 1
-                                if depth == 0:
-                                    args.append(expanded[arg_start + 1 : next_index])
-                                    break
-                            elif expanded[next_index] == ord(b"{"):
-                                depth += 1
-                            elif (
-                                depth == 1
-                                and expanded[next_index] == ord(b",")
-                                and expanded[next_index - 1] != ord(b"\\")
-                            ):
-                                args.append(expanded[arg_start + 1 : next_index])
-                                arg_start = next_index
-                            next_index += 1
-                        if next_index == len(expanded):
-                            raise ValueError("missing close brace")
-                        startpos = next_index + 1
+                    # Parse external program arguments
+                    if startpos < len(expanded) and expanded[startpos] == ord(b"("):
+                        external_args, startpos = parse_arguments(
+                            expanded, startpos, ord("("), ord(")")
+                        )
+                    if startpos < len(expanded) and expanded[startpos] == ord(b"{"):
+                        args, startpos = parse_arguments(
+                            expanded, startpos, ord("{"), ord("}")
+                        )
                     if escaped != b"":
                         # Just remove the leading '\'
                         args_string = b"{" + b",".join(args) + b"}"
                         output = b"$" + name + (args_string if len(args) > 0 else b"")
                     else:
-                        output = do_macro(name, args)
+                        output = do_macro(name, args, external_args)
                     expanded = expanded[: res.start()] + output + expanded[startpos:]
                     # Update search position to restart matching after output of macro
                     startpos = res.start() + len(output)