Merge branch 'develop' into tmp/1677620262/main

Author: aahung

.cfnlintrc.yaml

@@ -120,6 +120,7 @@ ignore_templates:
   - tests/translator/output/**/state_machine_with_schedule_dlq_retry_policy.json
   - tests/translator/output/**/globals_for_function.json  # RuntimeManagementConfig
   - tests/translator/output/**/function_with_runtime_config.json  # RuntimeManagementConfig
+  - tests/translator/output/**/managed_policies_minimal.json  # Intentionally has non-existent managed policy name
 ignore_checks:
   - E2531 # Deprecated runtime; not relevant for transform tests
   - W2531 # EOL runtime; not relevant for transform tests

.coveragerc

@@ -3,6 +3,7 @@ branch = True
 omit =
     # Schema tested by `make check-black`
     samtranslator/schema/*
+    samtranslator/internal/schema_source/*
 [report]
 exclude_lines =
     pragma: no cover

DEVELOPMENT_GUIDE.md

@@ -59,7 +59,7 @@ eval "$(pyenv virtualenv-init -)"
 We format our code using [Black](https://github.com/python/black) and verify the source code is black compliant
 during PR checks. Black will be installed automatically with `make init`.
 
-After installing, you can run our formatting through our Makefile by `make black` or integrating Black directly in your favorite IDE (instructions
+After installing, you can run our formatting through our Makefile by `make format` or integrating Black directly in your favorite IDE (instructions
 can be found [here](https://black.readthedocs.io/en/stable/editor_integration.html))
 
 ##### (Workaround) Integrating Black directly in your favorite IDE
@@ -158,11 +158,16 @@ Integration tests are covered in detail in the [INTEGRATION_TESTS.md file](INTEG
 
 ## Development guidelines
 
-1. **Do not resolve [intrinsic functions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html).** Adding [`AWS::LanguageExtensions`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-languageextension-transform.html) before the `AWS::Serverless-2016-10-31` transform resolves most of them (see https://github.com/aws/serverless-application-model/issues/2533). For new properties, use [`Property`](https://github.com/aws/serverless-application-model/blob/c5830b63857f52e540fec13b29f029458edc539a/samtranslator/model/__init__.py#L36-L45) or [`PassThroughProperty`](https://github.com/aws/serverless-application-model/blob/dd79f535500158baa8e367f081d6a12113497e45/samtranslator/model/__init__.py#L48-L56) instead of [`PropertyType`](https://github.com/aws/serverless-application-model/blob/c39c2807bbf327255de8abed8b8150b18c60f053/samtranslator/model/__init__.py#L13-L33).
+1. **Do not resolve [intrinsic functions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html).** Adding [`AWS::LanguageExtensions`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-languageextension-transform.html) before the `AWS::Serverless-2016-10-31` transform resolves most of them (see https://github.com/aws/serverless-application-model/issues/2533).
 2. **Do not break backward compatibility.** A specific SAM template should always transform into the same CloudFormation template. A template that has previously deployed successfully should continue to do so. Do not change logical IDs. Add opt-in properties for breaking changes. There are some exceptions, such as changes that do not impact resources (e.g. [`Metadata`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/metadata-section-structure.html)) or abstractions that can by design change over time.
 3. **Stick as close as possible to the underlying CloudFormation properties.** This includes both property names and values. This ensures we can pass values to CloudFormation and let it handle any intrinsic functions. In some cases, it also allows us to pass all properties as-is to a resource, which means customers can always use the newest properties, and we don’t spend effort maintaining a duplicate set of properties.
 4. **Only validate what’s necessary.** Do not validate properties if they’re passed directly to the underlying CloudFormation resource.
 5. **Add [type hints](https://peps.python.org/pep-0484/) to new code.** Strict typing was enabled in https://github.com/aws/serverless-application-model/pull/2558 by sprinkling [`# type: ignore`](https://peps.python.org/pep-0484/#compatibility-with-other-uses-of-function-annotations) across the existing code. Don't do that for new code. Avoid `# type: ignore`s at all cost. Instead, add types to new functions, and ideally add types to existing code it uses as well.
+6. **Do not use [`PropertyType`](https://github.com/aws/serverless-application-model/blob/c39c2807bbf327255de8abed8b8150b18c60f053/samtranslator/model/__init__.py#L13-L33) for new [`Resource`](https://github.com/aws/serverless-application-model/blob/13604cd2d9671cd6e774e5bfd610a03d82a08d76/samtranslator/model/__init__.py#L68) properties.**
+
+   For new properties of SAM resources, use [`Property`](https://github.com/aws/serverless-application-model/blob/c5830b63857f52e540fec13b29f029458edc539a/samtranslator/model/__init__.py#L36-L45) or [`PassThroughProperty`](https://github.com/aws/serverless-application-model/blob/dd79f535500158baa8e367f081d6a12113497e45/samtranslator/model/__init__.py#L48-L56) instead of [`PropertyType`](https://github.com/aws/serverless-application-model/blob/c39c2807bbf327255de8abed8b8150b18c60f053/samtranslator/model/__init__.py#L13-L33). This avoids [sneaky bugs](https://github.com/aws/serverless-application-model/pull/2495#discussion_r976715242) and ensures valid templates do not cause transform failures.
+
+   For new properties of CloudFormation resources, use [`GeneratedProperty`](https://github.com/aws/serverless-application-model/blob/79452f69bc1fcf918b8625c2f9005c74ab874801/samtranslator/model/__init__.py#L74-L82). It performs no runtime validation, reducing the risk of valid values causing transform failures.
 
 Code conventions
 ----------------

MANIFEST.in

@@ -8,6 +8,8 @@ recursive-include samtranslator/validator/sam_schema *.json
 include samtranslator/policy_templates_data/policy_templates.json
 include samtranslator/policy_templates_data/schema.json
 include samtranslator/model/connector_profiles/profiles.json
+include samtranslator/internal/data/aws_managed_policies.json
+include samtranslator/internal/schema_source/sam-docs.json
 include README.md
 include THIRD_PARTY_LICENSES
 

Makefile

@@ -17,23 +17,37 @@ test-cov-report:
 integ-test:
 	pytest --no-cov integration/
 
-black:
+format:
 	black setup.py samtranslator tests integration bin schema_source
+	bin/transform-test-error-json-format.py --write tests/translator/output/error_*.json
 	bin/json-format.py --write tests integration samtranslator/policy_templates_data
 	bin/yaml-format.py --write tests
 	bin/yaml-format.py --write integration --add-test-metadata
 
-black-check:
+black:
+	$(warning `make black` is deprecated, please use `make format`)
+	# sleep for 5 seconds so the message can be seen.
+	sleep 5
+	make format
+
+format-check:
 	# Checking latest schema was generated (run `make schema` if this fails)
 	mkdir -p .tmp
-	python -m schema_source.schema --sam-schema .tmp/sam.schema.json --cfn-schema schema_source/cloudformation.schema.json --unified-schema .tmp/schema.json
+	python -m samtranslator.internal.schema_source.schema --sam-schema .tmp/sam.schema.json --cfn-schema schema_source/cloudformation.schema.json --unified-schema .tmp/schema.json
 	diff -u schema_source/sam.schema.json .tmp/sam.schema.json
 	diff -u samtranslator/schema/schema.json .tmp/schema.json
 	black --check setup.py samtranslator tests integration bin schema_source
+	bin/transform-test-error-json-format.py --check tests/translator/output/error_*.json
 	bin/json-format.py --check tests integration samtranslator/policy_templates_data
 	bin/yaml-format.py --check tests
 	bin/yaml-format.py --check integration --add-test-metadata
 
+black-check:
+	$(warning `make black-check` is deprecated, please use `make format-check`)
+	# sleep for 5 seconds so the message can be seen.
+	sleep 5
+	make format-check
+
 lint:
 	ruff samtranslator bin schema_source integration tests
 	# mypy performs type check
@@ -60,14 +74,14 @@ fetch-schema-data:
 
 update-schema-data:
 	# Parse docs
-	bin/parse_docs.py .tmp/aws-sam-developer-guide/doc_source > schema_source/docs.json
+	bin/parse_docs.py .tmp/aws-sam-developer-guide/doc_source > samtranslator/internal/schema_source/sam-docs.json
 	bin/parse_docs.py --cfn .tmp/aws-cloudformation-user-guide/doc_source > schema_source/cloudformation-docs.json
 
 	# Add CloudFormation docs to CloudFormation schema
 	python bin/add_docs_cfn_schema.py --schema .tmp/cloudformation.schema.json --docs schema_source/cloudformation-docs.json > schema_source/cloudformation.schema.json
 
 schema:
-	python -m schema_source.schema --sam-schema schema_source/sam.schema.json --cfn-schema schema_source/cloudformation.schema.json --unified-schema samtranslator/schema/schema.json
+	python -m samtranslator.internal.schema_source.schema --sam-schema schema_source/sam.schema.json --cfn-schema schema_source/cloudformation.schema.json --unified-schema samtranslator/schema/schema.json
 
 # Update all schema data and schemas
 schema-all: fetch-schema-data update-schema-data schema
@@ -76,7 +90,7 @@ schema-all: fetch-schema-data update-schema-data schema
 dev: test
 
 # Verifications to run before sending a pull request
-pr: black-check lint init dev
+pr: format-check lint init dev
 
 clean:
 	rm -rf .tmp

bin/parse_docs.py

@@ -89,10 +89,10 @@ def main() -> None:
         if args.cfn and not re.match(r"^\w+::\w+::\w+( \w+)?$", title):
             continue
         page = title if args.cfn else path.stem
-        for name, description in parse(text):
+        for name, raw_description in parse(text):
             if page not in props:
                 props[page] = {}
-            description = remove_first_line(description)  # Remove property name; already in the schema title
+            description = remove_first_line(raw_description)  # Remove property name; already in the schema title
             description = fix_markdown_code_link(description)
             prefix = (
                 "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/"

bin/sam-translate.py

@@ -106,7 +106,7 @@ def transform_template(input_file_path, output_file_path):  # type: ignore[no-un
         sam_template = yaml_parse(f)  # type: ignore[no-untyped-call]
 
     try:
-        cloud_formation_template = transform(sam_template, {}, ManagedPolicyLoader(iam_client))  # type: ignore[no-untyped-call]
+        cloud_formation_template = transform(sam_template, {}, ManagedPolicyLoader(iam_client))
         cloud_formation_template_prettified = json.dumps(cloud_formation_template, indent=1)
 
         with open(output_file_path, "w") as f:

bin/transform-test-error-json-format.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+"""
+Transform test error JSON file formatter (without prettier).
+
+It makes error json easier to review by breaking down "errorMessage"
+into list of strings (delimiter: ". ").
+"""
+import os
+import sys
+
+from typing_extensions import Final
+
+my_path = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, my_path + "/..")
+
+import json
+from typing import Type
+
+from bin._file_formatter import FileFormatter
+
+
+class TransformTestErrorJSONFormatter(FileFormatter):
+    _ERROR_MESSAGE_KEY: Final[str] = "errorMessage"
+    _BREAKDOWN_ERROR_MESSAGE_KEY: Final[str] = "_autoGeneratedBreakdownErrorMessage"
+    _DELIMITER: Final[str] = ". "
+
+    @staticmethod
+    def description() -> str:
+        return "Transform test error JSON file formatter"
+
+    def format_str(self, input_str: str) -> str:
+        """
+        It makes error json easier to review by breaking down "errorMessage"
+        into list of strings (delimiter: ". ").
+        """
+        obj = json.loads(input_str)
+        error_message = obj.get(self._ERROR_MESSAGE_KEY)
+        if isinstance(error_message, str):
+            tokens = error_message.split(self._DELIMITER)
+            obj[self._BREAKDOWN_ERROR_MESSAGE_KEY] = [
+                token if index == len(tokens) - 1 else token + self._DELIMITER for index, token in enumerate(tokens)
+            ]
+        return json.dumps(obj, indent=2, sort_keys=True) + "\n"
+
+    @staticmethod
+    def decode_exception() -> Type[Exception]:
+        return json.JSONDecodeError
+
+    @staticmethod
+    def file_extension() -> str:
+        return ".json"
+
+
+if __name__ == "__main__":
+    TransformTestErrorJSONFormatter.main()

docs/globals.rst

@@ -81,6 +81,7 @@ Currently, the following resources and properties are being supported:
       Auth:
       Name:
       DefinitionUri:
+      MergeDefinitions:
       CacheClusterEnabled:
       CacheClusterSize:
       Variables:

integration/combination/test_connectors.py

@@ -26,6 +26,7 @@ def tearDown(self):
 
     @parameterized.expand(
         [
+            ("combination/connector_appsync_to_table",),
             ("combination/connector_function_to_function",),
             ("combination/connector_restapi_to_function",),
             ("combination/connector_httpapi_to_function",),