Full architecture overhaul: bidirectional HCL2 ↔ JSON pipeline with typed rule classes (#203)

Author: kkozik-amplify

.coveragerc

@@ -1,9 +1,15 @@
 [run]
 branch = true
 omit =
-    hcl2/__main__.py
     hcl2/lark_parser.py
+    hcl2/version.py
+    hcl2/__main__.py
+    hcl2/__init__.py
+    hcl2/rules/__init__.py
+    cli/__init__.py
 
 [report]
 show_missing = true
-fail_under = 80
+fail_under = 95
+exclude_lines =
+    raise NotImplementedError

.github/ISSUE_TEMPLATE/hcl2-parsing-error.md

@@ -1,27 +1,31 @@
----
+______________________________________________________________________
+
 name: HCL2 parsing error
 about: Template for reporting a bug related to parsing HCL2 code
 title: ''
 labels: bug
 assignees: kkozik-amplify
 
----
+______________________________________________________________________
 
 **Describe the bug**
 
 A clear and concise description of what the bug is.
 
 **Software:**
- - OS: [macOS / Windows / Linux]
- - Python version (e.g. 3.9.21)
- - python-hcl2 version (e.g. 7.0.0)
+
+- OS: \[macOS / Windows / Linux\]
+- Python version (e.g. 3.9.21)
+- python-hcl2 version (e.g. 7.0.0)
 
 **Snippet of HCL2 code causing the unexpected behaviour:**
+
 ```terraform
 locals {
   foo = "bar"
 }
 ```
+
 **Expected behavior**
 
 A clear and concise description of what you expected to happen, e.g. python dictionary or JSON you expected to receive as a result of parsing.

.pre-commit-config.yaml

@@ -6,6 +6,7 @@ repos:
     rev: v4.3.0
     hooks:
       - id: trailing-whitespace
+        exclude: ^test/integration/(hcl2_reconstructed|specialized)/
       - id: end-of-file-fixer
       - id: check-added-large-files
       - id: no-commit-to-branch  # Prevent commits directly to master

README.md

@@ -1,5 +1,4 @@
 [![Codacy Badge](https://app.codacy.com/project/badge/Grade/2e2015f9297346cbaa788c46ab957827)](https://app.codacy.com/gh/amplify-education/python-hcl2/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
-[![Build Status](https://travis-ci.org/amplify-education/python-hcl2.svg?branch=master)](https://travis-ci.org/amplify-education/python-hcl2)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/amplify-education/python-hcl2/master/LICENSE)
 [![PyPI](https://img.shields.io/pypi/v/python-hcl2.svg)](https://pypi.org/project/python-hcl2/)
 [![Python Versions](https://img.shields.io/pypi/pyversions/python-hcl2.svg)](https://pypi.python.org/pypi/python-hcl2)
@@ -36,19 +35,58 @@ pip3 install python-hcl2
 
 ### Usage
 
+**HCL2 to Python dict:**
+
 ```python
 import hcl2
-with open('foo.tf', 'r') as file:
-    dict = hcl2.load(file)
+
+with open("main.tf") as f:
+    data = hcl2.load(f)
 ```
 
-### Parse Tree to HCL2 reconstruction
+**Python dict to HCL2:**
+
+```python
+import hcl2
+
+hcl_string = hcl2.dumps(data)
+
+with open("output.tf", "w") as f:
+    hcl2.dump(data, f)
+```
 
-With version 6.x the possibility of HCL2 reconstruction from the Lark Parse Tree and Python dictionaries directly was introduced.
+**Building HCL from scratch:**
 
-Documentation and an example of manipulating Lark Parse Tree and reconstructing it back into valid HCL2 can be found in [tree-to-hcl2-reconstruction.md](https://github.com/amplify-education/python-hcl2/blob/main/tree-to-hcl2-reconstruction.md) file.
+```python
+import hcl2
+
+doc = hcl2.Builder()
+res = doc.block("resource", labels=["aws_instance", "web"], ami="abc-123", instance_type="t2.micro")
+res.block("tags", Name="HelloWorld")
+
+hcl_string = hcl2.dumps(doc.build())
+```
+
+For the full API reference, option dataclasses, intermediate pipeline stages, and more examples
+see [docs/usage.md](https://github.com/amplify-education/python-hcl2/blob/main/docs/usage.md).
+
+### CLI Tools
+
+python-hcl2 ships two command-line converters:
+
+```sh
+# HCL2 → JSON
+hcl2tojson main.tf                     # prints JSON to stdout
+hcl2tojson main.tf output.json         # writes to file
+hcl2tojson terraform/ output/          # converts a directory
+
+# JSON → HCL2
+jsontohcl2 output.json                 # prints HCL2 to stdout
+jsontohcl2 output.json main.tf         # writes to file
+jsontohcl2 output/ terraform/          # converts a directory
+```
 
-More details about reconstruction implementation can be found in PRs #169 and #177.
+Both commands accept `-` as PATH to read from stdin. Run `hcl2tojson --help` or `jsontohcl2 --help` for the full list of flags.
 
 ## Building From Source
 
@@ -61,7 +99,7 @@ Running `tox` will automatically execute linters as well as the unit tests.
 
 You can also run them individually with the `-e` argument.
 
-For example, `tox -e py37-unit` will run the unit tests for python 3.7
+For example, `tox -e py310-unit` will run the unit tests for python 3.10
 
 To see all the available options, run `tox -l`.
 
@@ -81,21 +119,10 @@ You can reach us at <mailto:github@amplify.com>
 We welcome pull requests! For your pull request to be accepted smoothly, we suggest that you:
 
 - For any sizable change, first open a GitHub issue to discuss your idea.
-- Create a pull request.  Explain why you want to make the change and what it’s for.
+- Create a pull request. Explain why you want to make the change and what it's for.
 
-We’ll try to answer any PR’s promptly.
+We'll try to answer any PR's promptly.
 
 ## Limitations
 
-### Using inline expression as an object key
-
-- Object key can be an expression as long as it is wrapped in parentheses:
-  ```terraform
-  locals {
-    foo = "bar"
-    baz = {
-      (format("key_prefix_%s", local.foo)) : "value"
-      # format("key_prefix_%s", local.foo) : "value" this will fail
-    }
-  }
-  ```
+None that are known.

hcl2/py.typed cli/__init__.pyhcl2/py.typed renamed to cli/__init__.py

@@ -0,0 +1,132 @@
+"""``hcl2tojson`` CLI entry point — convert HCL2 files to JSON."""
+import argparse
+import json
+import os
+from typing import IO, Optional, TextIO
+
+from hcl2 import load
+from hcl2.utils import SerializationOptions
+from hcl2.version import __version__
+from .helpers import (
+    HCL_SKIPPABLE,
+    _convert_single_file,
+    _convert_directory,
+    _convert_stdin,
+)
+
+
+def _hcl_to_json(
+    in_file: TextIO,
+    out_file: IO,
+    options: SerializationOptions,
+    json_indent: Optional[int] = None,
+) -> None:
+    data = load(in_file, serialization_options=options)
+    json.dump(data, out_file, indent=json_indent)
+
+
+def main():
+    """The ``hcl2tojson`` console_scripts entry point."""
+    parser = argparse.ArgumentParser(
+        description="Convert HCL2 files to JSON",
+    )
+    parser.add_argument(
+        "-s", dest="skip", action="store_true", help="Skip un-parsable files"
+    )
+    parser.add_argument(
+        "PATH",
+        help='The file or directory to convert (use "-" for stdin)',
+    )
+    parser.add_argument(
+        "OUT_PATH",
+        nargs="?",
+        help="The path to write output to. Optional for single file (defaults to stdout)",
+    )
+    parser.add_argument("--version", action="version", version=__version__)
+
+    # SerializationOptions flags
+    parser.add_argument(
+        "--with-meta",
+        action="store_true",
+        help="Add meta parameters like __start_line__ and __end_line__",
+    )
+    parser.add_argument(
+        "--with-comments",
+        action="store_true",
+        help="Include comments in the output",
+    )
+    parser.add_argument(
+        "--wrap-objects",
+        action="store_true",
+        help="Wrap object values as an inline HCL2",
+    )
+    parser.add_argument(
+        "--wrap-tuples",
+        action="store_true",
+        help="Wrap tuple values an inline HCL2",
+    )
+    parser.add_argument(
+        "--no-explicit-blocks",
+        action="store_true",
+        help="Disable explicit block markers",
+    )
+    parser.add_argument(
+        "--no-preserve-heredocs",
+        action="store_true",
+        help="Convert heredocs to plain strings",
+    )
+    parser.add_argument(
+        "--force-parens",
+        action="store_true",
+        help="Force parentheses around all operations",
+    )
+    parser.add_argument(
+        "--no-preserve-scientific",
+        action="store_true",
+        help="Convert scientific notation to standard floats",
+    )
+
+    # JSON output formatting
+    parser.add_argument(
+        "--json-indent",
+        type=int,
+        default=2,
+        metavar="N",
+        help="JSON indentation width (default: 2)",
+    )
+
+    args = parser.parse_args()
+
+    options = SerializationOptions(
+        with_meta=args.with_meta,
+        with_comments=args.with_comments,
+        wrap_objects=args.wrap_objects,
+        wrap_tuples=args.wrap_tuples,
+        explicit_blocks=not args.no_explicit_blocks,
+        preserve_heredocs=not args.no_preserve_heredocs,
+        force_operation_parentheses=args.force_parens,
+        preserve_scientific_notation=not args.no_preserve_scientific,
+    )
+    json_indent = args.json_indent
+
+    def convert(in_file, out_file):
+        _hcl_to_json(in_file, out_file, options, json_indent=json_indent)
+
+    if args.PATH == "-":
+        _convert_stdin(convert)
+    elif os.path.isfile(args.PATH):
+        _convert_single_file(
+            args.PATH, args.OUT_PATH, convert, args.skip, HCL_SKIPPABLE
+        )
+    elif os.path.isdir(args.PATH):
+        _convert_directory(
+            args.PATH,
+            args.OUT_PATH,
+            convert,
+            args.skip,
+            HCL_SKIPPABLE,
+            in_extensions={".tf", ".hcl"},
+            out_extension=".json",
+        )
+    else:
+        raise RuntimeError(f"Invalid Path: {args.PATH}")

cli/hcl_to_json.py

@@ -0,0 +1,96 @@
+"""Shared file-conversion helpers for the HCL2 CLI commands."""
+import json
+import os
+import sys
+from typing import Callable, IO, Set, Tuple, Type
+
+from lark import UnexpectedCharacters, UnexpectedToken
+
+# Exceptions that can be skipped when -s is passed
+HCL_SKIPPABLE = (UnexpectedToken, UnexpectedCharacters, UnicodeDecodeError)
+JSON_SKIPPABLE = (json.JSONDecodeError, UnicodeDecodeError)
+
+
+def _convert_single_file(
+    in_path: str,
+    out_path: str,
+    convert_fn: Callable[[IO, IO], None],
+    skip: bool,
+    skippable: Tuple[Type[BaseException], ...],
+) -> None:
+    with open(in_path, "r", encoding="utf-8") as in_file:
+        print(in_path, file=sys.stderr, flush=True)
+        if out_path is not None:
+            try:
+                with open(out_path, "w", encoding="utf-8") as out_file:
+                    convert_fn(in_file, out_file)
+            except skippable:
+                if skip:
+                    if os.path.exists(out_path):
+                        os.remove(out_path)
+                    return
+                raise
+        else:
+            try:
+                convert_fn(in_file, sys.stdout)
+                sys.stdout.write("\n")
+            except skippable:
+                if skip:
+                    return
+                raise
+
+
+def _convert_directory(
+    in_path: str,
+    out_path: str,
+    convert_fn: Callable[[IO, IO], None],
+    skip: bool,
+    skippable: Tuple[Type[BaseException], ...],
+    in_extensions: Set[str],
+    out_extension: str,
+) -> None:
+    if out_path is None:
+        raise RuntimeError("Positional OUT_PATH parameter shouldn't be empty")
+    if not os.path.exists(out_path):
+        os.mkdir(out_path)
+
+    processed_files: set = set()
+    for current_dir, _, files in os.walk(in_path):
+        dir_prefix = os.path.commonpath([in_path, current_dir])
+        relative_current_dir = os.path.relpath(current_dir, dir_prefix)
+        current_out_path = os.path.normpath(
+            os.path.join(out_path, relative_current_dir)
+        )
+        if not os.path.exists(current_out_path):
+            os.mkdir(current_out_path)
+        for file_name in files:
+            _, ext = os.path.splitext(file_name)
+            if ext not in in_extensions:
+                continue
+
+            in_file_path = os.path.join(current_dir, file_name)
+            out_file_path = os.path.join(current_out_path, file_name)
+            out_file_path = os.path.splitext(out_file_path)[0] + out_extension
+
+            if in_file_path in processed_files or out_file_path in processed_files:
+                continue
+
+            processed_files.add(in_file_path)
+            processed_files.add(out_file_path)
+
+            with open(in_file_path, "r", encoding="utf-8") as in_file:
+                print(in_file_path, file=sys.stderr, flush=True)
+                try:
+                    with open(out_file_path, "w", encoding="utf-8") as out_file:
+                        convert_fn(in_file, out_file)
+                except skippable:
+                    if skip:
+                        if os.path.exists(out_file_path):
+                            os.remove(out_file_path)
+                        continue
+                    raise
+
+
+def _convert_stdin(convert_fn: Callable[[IO, IO], None]) -> None:
+    convert_fn(sys.stdin, sys.stdout)
+    sys.stdout.write("\n")