S3 tree optimizations (smithy-lang#2913)

* Improve BDD sifting (2x speed, more reduction)

- Add AdaptiveEffort to dynamically adjust optimization parameters based
  on observed improvement rates (increase effort when making progress,
  decrease when plateauing)
- Add block moves optimization that moves groups of dependent conditions
  together to escape local minima
- Add cost-based tie-breaking using BddCostEstimator when multiple
  positions have the same node count

* Implement rules engine ITE fn and S3 tree transform

This commit adds a new function to the rules engine, ite, that performs
an if-then-else check on a boolean expression without branching. By not
needing to branch in the decision tree, we avoid SSA transforms on
divergent branches which would create syntactically different but
semantically identical expressions that the BDD cannot deduplicate.

This commit also adds an S3-specific decision tree transform that
canonicalizes S3Express rules for better BDD compilation:

1. AZ extraction: Rewrites position-dependent substring operations to
   use a single split(Bucket, "--")[1] expression across all branches
2. URL canonicalization: Uses ITE to compute FIPS/DualStack URL segments,
   collapsing 4 URL variants into a single template with {_s3e_fips} and
   {_s3e_ds} placeholders
3. Auth scheme canonicalization: Uses ITE to select sigv4 vs
   sigv4-s3express based on DisableS3ExpressSessionAuth

The transform makes the rules tree ~30% larger but enables dramatic BDD
compression by making URL templates identical across FIPS/DualStack/auth
variants. Endpoints that previously appeared distinct now collapse into
single BDD results, reducing nodes and results by ~43%.

* Run type checking on BDD so type() works

* Improve some loops

* Attempt to fix the windows build

* Optimize rules further

Add nullable boolean transform.
Improve variable and expression sharing.
Add S3 BDD integration test.
Add TreeMapper to simplify transforms.

* Reduce SSA noise

* Refactor S3 tree rewriter into composable transforms

Split the monolithic S3TreeRewriter into three independent transforms:
- S3AzCanonicalizerTransform: substring → split for AZ extraction
- S3RegionUnifierTransform: unify Region/bucketArn#region/us-east-1
- S3ExpressEndpointTransform: canonicalize FIPS/DualStack/auth

Added region unification pass that rewrites all region references to
_effective_std_region (for aws-global → us-east-1 mapping) or
_effective_arn_region (for UseArnRegion logic). This enables additional
BDD sharing across endpoints with different region sources.

S3 BDD stats: 77 conditions, 97 results, 484 nodes after sifting.

* Address PR feedback on gradle and test case

* Add BDD coverage testing and unref condition tx

Author: mtdowling

.changes/next-release/feature-1c36b965bf2f51d85aa0171be6206ae2029137c4.json

@@ -0,0 +1,7 @@
+{
+  "type": "feature",
+  "description": "Implement rules engine ITE fn and S3 tree transform",
+  "pull_requests": [
+    "[#2903](https://github.com/smithy-lang/smithy/pull/2903)"
+  ]
+}

.changes/next-release/feature-5e4b03b74b1054ac2632fcae533d727eace1d26e.json

@@ -0,0 +1,7 @@
+{
+  "type": "feature",
+  "description": "Improve BDD sifting (2x speed, more reduction)",
+  "pull_requests": [
+    "[#2890](https://github.com/smithy-lang/smithy/pull/2890)"
+  ]
+}

docs/source-2.0/additional-specs/rules-engine/standard-library.rst

@@ -208,6 +208,103 @@ The following example uses ``isValidHostLabel`` to check if the value of the
     }
 
 
+.. _rules-engine-standard-library-ite:
+
+``ite`` function
+================
+
+Summary
+    An if-then-else function that returns one of two values based on a boolean condition.
+Argument types
+    * condition: ``bool``
+    * trueValue: ``T`` or ``option<T>``
+    * falseValue: ``T`` or ``option<T>``
+Return type
+    * ``ite(bool, T, T)`` → ``T`` (both non-optional, result is non-optional)
+    * ``ite(bool, T, option<T>)`` → ``option<T>`` (any optional makes result optional)
+    * ``ite(bool, option<T>, T)`` → ``option<T>`` (any optional makes result optional)
+    * ``ite(bool, option<T>, option<T>)`` → ``option<T>`` (both optional, result is optional)
+Since
+    1.1
+
+The ``ite`` (if-then-else) function evaluates a boolean condition and returns one of two values based on
+the result. If the condition is ``true``, it returns ``trueValue``; if ``false``, it returns ``falseValue``.
+This function is particularly useful for computing conditional values without branching in the rule tree, resulting
+in fewer result nodes, and enabling better BDD optimizations as a result of reduced fragmentation.
+
+.. important::
+    Both ``trueValue`` and ``falseValue`` must have the same base type ``T``. The result type follows
+    the "least upper bound" rule: if either branch is optional, the result is optional.
+
+The following example uses ``ite`` to compute a URL suffix based on whether FIPS is enabled:
+
+.. code-block:: json
+
+    {
+        "fn": "ite",
+        "argv": [
+            {"ref": "UseFIPS"},
+            "-fips",
+            ""
+        ],
+        "assign": "fipsSuffix"
+    }
+
+The following example uses ``ite`` with ``coalesce`` to handle an optional boolean parameter:
+
+.. code-block:: json
+
+    {
+        "fn": "ite",
+        "argv": [
+            {
+                "fn": "coalesce",
+                "argv": [
+                    {"ref": "DisableFeature"},
+                    false
+                ]
+            },
+            "disabled",
+            "enabled"
+        ],
+        "assign": "featureState"
+    }
+
+
+.. _rules-engine-standard-library-ite-examples:
+
+--------
+Examples
+--------
+
+The following table shows various inputs and their corresponding outputs for the ``ite`` function:
+
+.. list-table::
+    :header-rows: 1
+    :widths: 20 25 25 30
+
+    * - Condition
+      - True Value
+      - False Value
+      - Output
+    * - ``true``
+      - ``"-fips"``
+      - ``""``
+      - ``"-fips"``
+    * - ``false``
+      - ``"-fips"``
+      - ``""``
+      - ``""``
+    * - ``true``
+      - ``"sigv4"``
+      - ``"sigv4-s3express"``
+      - ``"sigv4"``
+    * - ``false``
+      - ``"sigv4"``
+      - ``"sigv4-s3express"``
+      - ``"sigv4-s3express"``
+
+
 .. _rules-engine-standard-library-not:
 
 ``not`` function

smithy-aws-endpoints/build.gradle.kts

@@ -11,10 +11,60 @@ description = "AWS specific components for managing endpoints in Smithy"
 extra["displayName"] = "Smithy :: AWS Endpoints Components"
 extra["moduleName"] = "software.amazon.smithy.aws.endpoints"
 
+// Custom configuration for S3 model - kept separate from test classpath to avoid
+// polluting other tests with S3 model discovery
+val s3Model: Configuration by configurations.creating
+
 dependencies {
     api(project(":smithy-aws-traits"))
     api(project(":smithy-diff"))
     api(project(":smithy-rules-engine"))
     api(project(":smithy-model"))
     api(project(":smithy-utils"))
+
+    s3Model("software.amazon.api.models:s3:1.0.11")
+}
+
+// Integration test source set for tests that require the S3 model
+// These tests require JDK 17+ due to the S3 model dependency
+sourceSets {
+    create("it") {
+        compileClasspath += sourceSets["main"].output + sourceSets["test"].output
+        runtimeClasspath += sourceSets["main"].output + sourceSets["test"].output
+    }
+}
+
+configurations["itImplementation"].extendsFrom(configurations["testImplementation"])
+configurations["itRuntimeOnly"].extendsFrom(configurations["testRuntimeOnly"])
+configurations["itImplementation"].extendsFrom(s3Model)
+
+// Configure IT source set to compile with JDK (17+) since the models it uses require it.
+tasks.named<JavaCompile>("compileItJava") {
+    sourceCompatibility = "17"
+    targetCompatibility = "17"
+}
+
+val integrationTest by tasks.registering(Test::class) {
+    description = "Runs integration tests that require external models like S3"
+    group = "verification"
+    testClassesDirs = sourceSets["it"].output.classesDirs
+    classpath = sourceSets["it"].runtimeClasspath
+    dependsOn(tasks.jar)
+    shouldRunAfter(tasks.test)
+
+    // Pass build directory to tests
+    systemProperty(
+        "buildDir",
+        layout.buildDirectory
+            .get()
+            .asFile.absolutePath,
+    )
+}
+
+tasks.test {
+    finalizedBy(integrationTest)
+}
+
+tasks.named("check") {
+    dependsOn(integrationTest)
 }

smithy-aws-endpoints/src/it/java/software/amazon/smithy/rulesengine/aws/language/functions/S3BddTest.java

@@ -0,0 +1,157 @@
+/*
+ * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package software.amazon.smithy.rulesengine.aws.language.functions;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import software.amazon.smithy.model.Model;
+import software.amazon.smithy.model.node.Node;
+import software.amazon.smithy.model.shapes.ModelSerializer;
+import software.amazon.smithy.model.shapes.ServiceShape;
+import software.amazon.smithy.model.shapes.ShapeId;
+import software.amazon.smithy.rulesengine.analysis.BddCoverageChecker;
+import software.amazon.smithy.rulesengine.aws.s3.S3TreeRewriter;
+import software.amazon.smithy.rulesengine.language.EndpointRuleSet;
+import software.amazon.smithy.rulesengine.language.evaluation.TestEvaluator;
+import software.amazon.smithy.rulesengine.logic.bdd.CostOptimization;
+import software.amazon.smithy.rulesengine.logic.bdd.SiftingOptimization;
+import software.amazon.smithy.rulesengine.logic.cfg.Cfg;
+import software.amazon.smithy.rulesengine.traits.EndpointBddTrait;
+import software.amazon.smithy.rulesengine.traits.EndpointRuleSetTrait;
+import software.amazon.smithy.rulesengine.traits.EndpointTestCase;
+import software.amazon.smithy.rulesengine.traits.EndpointTestsTrait;
+
+class S3BddTest {
+    private static final ShapeId S3_SERVICE_ID = ShapeId.from("com.amazonaws.s3#AmazonS3");
+    private static Model model;
+    private static ServiceShape s3Service;
+    private static EndpointRuleSet originalRules;
+    private static EndpointRuleSet rules;
+    private static List<EndpointTestCase> testCases;
+
+    @BeforeAll
+    static void loadS3Model() {
+        model = Model.assembler()
+                .discoverModels()
+                .assemble()
+                .unwrap();
+
+        s3Service = model.expectShape(S3_SERVICE_ID, ServiceShape.class);
+        originalRules = s3Service.expectTrait(EndpointRuleSetTrait.class).getEndpointRuleSet();
+        rules = S3TreeRewriter.transform(originalRules);
+        testCases = s3Service.expectTrait(EndpointTestsTrait.class).getTestCases();
+    }
+
+    @Test
+    void compileToBddWithOptimizations() {
+        // Verify transforms preserve semantics by running all test cases
+        for (EndpointTestCase testCase : testCases) {
+            TestEvaluator.evaluate(rules, testCase);
+        }
+
+        // Build CFG and compile to BDD
+        Cfg cfg = Cfg.from(rules);
+        EndpointBddTrait trait = EndpointBddTrait.from(cfg);
+
+        StringBuilder sb = new StringBuilder();
+        sb.append("\n=== BDD STATS ===\n");
+        sb.append("Conditions: ").append(trait.getConditions().size()).append("\n");
+        sb.append("Results: ").append(trait.getResults().size()).append("\n");
+        sb.append("Initial BDD nodes: ").append(trait.getBdd().getNodeCount()).append("\n");
+
+        // Apply sifting optimization
+        SiftingOptimization sifting = SiftingOptimization.builder().cfg(cfg).build();
+        EndpointBddTrait siftedTrait = sifting.apply(trait);
+        sb.append("After sifting - nodes: ").append(siftedTrait.getBdd().getNodeCount()).append("\n");
+
+        // Apply cost optimization
+        CostOptimization cost = CostOptimization.builder().cfg(cfg).build();
+        EndpointBddTrait optimizedTrait = cost.apply(siftedTrait);
+        sb.append("After cost opt - nodes: ").append(optimizedTrait.getBdd().getNodeCount()).append("\n");
+        System.out.println("Unreferenced BDD conditions before dead condition elimination: "
+                + new BddCoverageChecker(optimizedTrait).getUnreferencedConditions());
+
+        EndpointBddTrait finalizedTrait = optimizedTrait.removeUnreferencedConditions();
+        System.out.println("Unreferenced BDD conditions after dead condition elimination: "
+                + new BddCoverageChecker(optimizedTrait).getUnreferencedConditions());
+
+        // Print conditions for analysis
+        sb.append("\n=== CONDITIONS ===\n");
+        for (int i = 0; i < finalizedTrait.getConditions().size(); i++) {
+            sb.append(i).append(": ").append(finalizedTrait.getConditions().get(i)).append("\n");
+        }
+
+        // Print results (endpoints) for analysis
+        sb.append("\n=== RESULTS ===\n");
+        for (int i = 0; i < finalizedTrait.getResults().size(); i++) {
+            sb.append(i).append(": ").append(finalizedTrait.getResults().get(i)).append("\n");
+        }
+
+        System.out.println(sb);
+
+        // Verify transforms preserve semantics by running all test cases on the BDD -and- ensuring 100% coverage.
+        BddCoverageChecker coverageCheckerBdd = new BddCoverageChecker(finalizedTrait);
+        for (EndpointTestCase testCase : testCases) {
+            coverageCheckerBdd.evaluateTestCase(testCase);
+        }
+
+        if (coverageCheckerBdd.getConditionCoverage() < 100) {
+            throw new RuntimeException("Condition coverage < 100%: "
+                    + coverageCheckerBdd.getConditionCoverage()
+                    + " : " + coverageCheckerBdd.getUnevaluatedConditions());
+        }
+
+        if (coverageCheckerBdd.getResultCoverage() < 100) {
+            throw new RuntimeException("Result coverage < 100%: "
+                    + coverageCheckerBdd.getResultCoverage()
+                    + " : " + coverageCheckerBdd.getUnevaluatedResults());
+        }
+
+        // Write model with BDD trait to build directory
+        writeModelWithBddTrait(finalizedTrait);
+    }
+
+    private void writeModelWithBddTrait(EndpointBddTrait bddTrait) {
+        String buildDir = System.getProperty("buildDir");
+        if (buildDir == null) {
+            System.out.println("buildDir system property not set, skipping model output");
+            return;
+        }
+
+        // Create updated service with BDD trait instead of RuleSet trait
+        ServiceShape updatedService = s3Service.toBuilder()
+                .removeTrait(EndpointRuleSetTrait.ID)
+                .addTrait(bddTrait)
+                .build();
+
+        // Build updated model
+        Model updatedModel = model.toBuilder()
+                .removeShape(s3Service.getId())
+                .addShape(updatedService)
+                .build();
+
+        // Serialize to JSON
+        ModelSerializer serializer = ModelSerializer.builder().build();
+        String json = Node.prettyPrintJson(serializer.serialize(updatedModel));
+
+        // Write to build directory
+        Path outputPath = Paths.get(buildDir, "s3-bdd-model.json");
+        try {
+            Path parentDir = outputPath.getParent();
+            if (parentDir != null) {
+                Files.createDirectories(parentDir);
+            }
+            Files.writeString(outputPath, json);
+            System.out.println("Wrote S3 BDD model to: " + outputPath);
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to write S3 BDD model", e);
+        }
+    }
+}

smithy-aws-endpoints/src/it/java/software/amazon/smithy/rulesengine/aws/language/functions/S3TreeRewriterTest.java

@@ -0,0 +1,53 @@
+/*
+ * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package software.amazon.smithy.rulesengine.aws.language.functions;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+import java.util.List;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import software.amazon.smithy.model.Model;
+import software.amazon.smithy.model.shapes.ServiceShape;
+import software.amazon.smithy.model.shapes.ShapeId;
+import software.amazon.smithy.rulesengine.aws.s3.S3TreeRewriter;
+import software.amazon.smithy.rulesengine.language.EndpointRuleSet;
+import software.amazon.smithy.rulesengine.language.evaluation.TestEvaluator;
+import software.amazon.smithy.rulesengine.traits.EndpointRuleSetTrait;
+import software.amazon.smithy.rulesengine.traits.EndpointTestCase;
+import software.amazon.smithy.rulesengine.traits.EndpointTestsTrait;
+
+/**
+ * Runs the endpoint test cases against the transformed S3 model. We're fixed to a specific version for this test,
+ * but could periodically bump the version if needed.
+ */
+class S3TreeRewriterTest {
+    private static final ShapeId S3_SERVICE_ID = ShapeId.from("com.amazonaws.s3#AmazonS3");
+
+    private static EndpointRuleSet originalRules;
+    private static List<EndpointTestCase> testCases;
+
+    @BeforeAll
+    static void loadS3Model() {
+        Model model = Model.assembler()
+                .discoverModels()
+                .assemble()
+                .unwrap();
+
+        ServiceShape s3Service = model.expectShape(S3_SERVICE_ID, ServiceShape.class);
+        originalRules = s3Service.expectTrait(EndpointRuleSetTrait.class).getEndpointRuleSet();
+        testCases = s3Service.expectTrait(EndpointTestsTrait.class).getTestCases();
+    }
+
+    @Test
+    void transformPreservesEndpointTestSemantics() {
+        assertFalse(testCases.isEmpty(), "S3 model should have endpoint test cases");
+
+        EndpointRuleSet transformed = S3TreeRewriter.transform(originalRules);
+        for (EndpointTestCase testCase : testCases) {
+            TestEvaluator.evaluate(transformed, testCase);
+        }
+    }
+}