|
| 1 | +# Apigee Proxy Bundle Modifier & Validator |
| 2 | + |
| 3 | +This script assists in migrating Apigee Edge proxy bundles to Apigee X/Hybrid by allowing batch modification of policy XML files, ProxyEndpoint definitions, and TargetEndpoint definitions based on rules defined in a TOML configuration file. It uses the **`lxml` library** for modifications to better preserve XML formatting and can optionally validate the modified bundles against the Apigee API using Google Application Default Credentials (ADC). The final output is a detailed **Markdown report**. |
| 4 | + |
| 5 | +## Purpose |
| 6 | + |
| 7 | +During migration from Apigee Edge to Apigee X/Hybrid, certain policy configurations, endpoint settings, or target URLs might need adjustments for compatibility. This script automates the process of: |
| 8 | + |
| 9 | +1. Extracting an API proxy bundle (`.zip`). |
| 10 | +2. Parsing XML files within `apiproxy/policies/`, `apiproxy/proxies/`, and `apiproxy/targets/`. |
| 11 | +3. Modifying XML elements or attributes based on rules defined in a TOML file (using `lxml` and **XPath**). Supported operations include Set/Replace, Prefix, Suffix, and Conditional Prefix. |
| 12 | +4. Preserving original XML formatting (whitespace, comments, etc.) as much as possible during modification thanks to `lxml`. |
| 13 | +5. Generating detailed `diff` output only for files with actual content changes. |
| 14 | +6. Re-zipping the modified bundle structure. |
| 15 | +7. Optionally validating the modified bundle using the Apigee API (inferring the proxy name from the filename). |
| 16 | +8. Processing multiple bundles from an input directory to an output directory. |
| 17 | +9. Generating a comprehensive **Markdown report file** (`.md`) containing: |
| 18 | + * Overall processing summary counts. |
| 19 | + * A summary table of all processed bundles. |
| 20 | + * Detailed status per bundle, including validation results and error snippets. |
| 21 | + * Line-by-line diffs (using `difflib`) for all files that were modified. |
| 22 | + |
| 23 | +## Features |
| 24 | + |
| 25 | +* Modifies XML files (Policies, ProxyEndpoints, TargetEndpoints) within Apigee proxy bundles. |
| 26 | +* Uses the **`lxml` library** for robust XML parsing and modification, aiming to preserve formatting. |
| 27 | +* Uses a TOML file for defining modification rules using **XPath expressions**. |
| 28 | +* Supports multiple modification operations: Set/Replace (with wildcard 'from'), Prefix, Suffix, Conditional Prefix. |
| 29 | +* Processes all `.zip` files in a specified input directory to an output directory. |
| 30 | +* Generates accurate `diff` output highlighting content changes, minimizing formatting noise. |
| 31 | +* Optionally validates modified bundles against the Apigee API using Google ADC. |
| 32 | +* Infers the proxy name for validation from the bundle filename (requires consistent naming). |
| 33 | +* Generates a detailed **Markdown report file**. |
| 34 | +* Supports overwriting existing output files (`--overwrite`). |
| 35 | +* Verbose console logging option (`-v`) for real-time progress and debugging. |
| 36 | + |
| 37 | +## Prerequisites |
| 38 | + |
| 39 | +1. **Python:** Python 3.8 or higher recommended. |
| 40 | +2. **pip:** Python package installer. |
| 41 | +3. **Google Cloud SDK:** (`gcloud` command-line tool) installed and authenticated. |
| 42 | +4. **Application Default Credentials (ADC):** Configured for Google Cloud authentication. Run `gcloud auth application-default login` in your terminal and authenticate with a Google account that has permissions to validate Apigee APIs (e.g., Apigee API Admin role or custom role with `apigee.proxies.create`). |
| 43 | +5. **Required Python Libraries:** Install using pip. |
| 44 | + |
| 45 | +## Installation |
| 46 | + |
| 47 | +1. **Clone or Download:** Get the script file : `tools/apigee-proxy-modifier-validator/modify_proxies.py`. |
| 48 | +2. **Install Dependencies:** Open your terminal in the script's directory and run: |
| 49 | + ```bash |
| 50 | + pip install -r tools/apigee-proxy-modifier-validator/requirements.txt |
| 51 | + ``` |
| 52 | + |
| 53 | +## Configuration |
| 54 | + |
| 55 | +### 1. `policy.toml` Rules File (using XPath) |
| 56 | + |
| 57 | +This file defines *how* to modify the XML files. |
| 58 | + |
| 59 | +* Create a file (e.g., `policy_lxml.toml`). |
| 60 | +* Define rules based on **XPath expressions** to select target XML nodes. |
| 61 | +* The root tag of the XML file (e.g., `<ServiceCallout>`, `<ProxyEndpoint>`) determines which section of the TOML file applies. |
| 62 | + |
| 63 | +**Key TOML Fields per Rule:** |
| 64 | + |
| 65 | +* `[RootXmlElementTag]`: Section header matching the root XML tag. |
| 66 | +* `[RootXmlElementTag.RuleName]`: Unique name for the rule. |
| 67 | +* `xpath = "/path/to/element[@attr='val']"`: **Required.** An XPath 1.0 expression to select the target node(s). Use `.` for relative paths from root. |
| 68 | +* `target_type = "text" | "attribute"`: **Required.** What to modify (element's text or an attribute). |
| 69 | +* `attribute_name = "attr_name"`: **Required if `target_type="attribute"`.** The attribute name (no `@` prefix needed). |
| 70 | +* **Choose ONE Operation:** |
| 71 | + * `to = "new_value"`: **Required for Set/Replace.** Final value. |
| 72 | + * `from = "old_value"`: **Optional for Set/Replace & Conditional Prefix.** Value to replace or condition for prefixing. If omitted in Set/Replace, acts as a wildcard "set". |
| 73 | + * `prefix = "prefix_string"`: **Required for Prefix & Conditional Prefix.** String to prepend. |
| 74 | + * `suffix = "suffix_string"`: **Required for Suffix.** String to append. |
| 75 | +* **Exclusivity:** A rule cannot mix `prefix`, `suffix`, and `to`, *except* for `conditional_prefix` which requires both `prefix` and `from`. |
| 76 | +
|
| 77 | +**(See `policy_template_lxml.toml` or previous examples for details)** |
| 78 | +
|
| 79 | +### 2. Application Default Credentials (ADC) |
| 80 | +
|
| 81 | +Ensure ADC is set up before using `--validate`: |
| 82 | +
|
| 83 | +```bash |
| 84 | +gcloud auth application-default login |
| 85 | +``` |
| 86 | +Follow the browser prompts to authenticate. The account used needs Apigee permissions in the target Google Cloud project. |
| 87 | +
|
| 88 | +### Usage |
| 89 | +Run the script from your terminal: |
| 90 | +```bash |
| 91 | +python modify_proxies.py --input-dir <path/to/input_bundles> \ |
| 92 | + --output-dir <path/to/output_bundles> \ |
| 93 | + --config-path <path/to/policy_lxml.toml> \ |
| 94 | + --report-file <path/to/output_report.md> \ |
| 95 | + [options] |
| 96 | +``` |
| 97 | +
|
| 98 | +**Options:** |
| 99 | +Command-Line Arguments: |
| 100 | +* --input-dir PATH: Required. Path to the directory containing the source .zip proxy bundles. |
| 101 | +* --output-dir PATH: Required. Path to the directory where modified .zip bundles will be saved. Must be different from the input directory. |
| 102 | +* --config-path PATH: Required. Path to your .toml configuration file defining the modification rules. |
| 103 | +* --report-file PATH: Required. Path where the output Markdown report will be saved. |
| 104 | +* --validate: Optional. Enable validation of modified bundles using the Apigee API. Requires ADC to be configured. |
| 105 | +* --org ORG_ID: Required if --validate is used. Your Apigee Organization ID. |
| 106 | +* --overwrite: Optional. If specified, existing files in the output directory with the same name as a bundle being processed will be overwritten. By default, existing files are skipped. |
| 107 | +* -v, --verbose: Optional. Enable detailed debug logging output. |
| 108 | +
|
| 109 | +### Usage Examples |
| 110 | +
|
| 111 | +Modify all bundles in a directory (no validation): |
| 112 | +```bash |
| 113 | +python modify_proxies.py --input-dir ./edge_proxies \ |
| 114 | + --output-dir ./hybrid_proxies \ |
| 115 | + --config-path ./rules_lxml.toml \ |
| 116 | + --report-file modification_report.md |
| 117 | +``` |
| 118 | +
|
| 119 | +Modify and Validate bundles: |
| 120 | +
|
| 121 | +```bash |
| 122 | +python modify_proxies.py --input-dir ./edge_proxies \ |
| 123 | + --output-dir ./hybrid_proxies \ |
| 124 | + --config-path ./rules_lxml.toml \ |
| 125 | + --report-file validation_run.md \ |
| 126 | + --validate \ |
| 127 | + --org "your-apigee-project-id" |
| 128 | +``` |
| 129 | +
|
| 130 | +Modify and Validate with Overwrite and Verbose Logging: |
| 131 | +
|
| 132 | +```bash |
| 133 | +python modify_proxies.py -v --overwrite \ |
| 134 | + --input-dir /path/to/input \ |
| 135 | + --output-dir /path/to/output \ |
| 136 | + --config-path ./prod_rules_lxml.toml \ |
| 137 | + --report-file prod_run_report.md \ |
| 138 | + --validate --org "my-prod-org" |
| 139 | +``` |
| 140 | +
|
| 141 | +### Validation Details |
| 142 | +* Name Inference: Validation relies on inferring the proxy name from the .zip filename (removing suffixes like _revN, _vN). Ensure your filenames allow for correct inference. Adjust the infer_proxy_name function's regex if needed. |
| 143 | +* API Call vs. Content Validity: |
| 144 | + * Success (API Call OK): The API request returned HTTP 2xx. The bundle might still have internal issues not detected by this basic validation call structure. |
| 145 | + * Success (API Issues Found): The API request returned HTTP 2xx, but the response body contained keywords like "error" or "validationerrors", suggesting Apigee found problems within the bundle content. |
| 146 | + * Failed (...): Indicates issues with the API call itself (Auth, Network, HTTP error), name inference, file access, or an unknown script error during validation. |
| 147 | +* Check Logs: The "Validation Detail Snippet" provides a hint, but the full API response body and detailed error messages are only available in the script's console log output. Use -v for maximum detail. |
| 148 | +
|
| 149 | +### Output |
| 150 | +
|
| 151 | +1. Modified Bundles: .zip files containing the modified proxy structure are saved in the specified --output-dir. |
| 152 | +2. Console Logs: Real-time progress, warnings, errors, and API responses (especially with -v). |
| 153 | +3. Markdown Report (--report-file): A comprehensive report file containing: |
| 154 | + * Overall summary statistics. |
| 155 | + * A summary table showing the status for each processed bundle. |
| 156 | + * A detailed section for each bundle, including: |
| 157 | + * Modified status (Yes/No). |
| 158 | + * Validation result code (e.g., Success (API Call OK), Failed (Auth Error)). |
| 159 | + * A concise snippet extracted from the validation response, if relevant. |
| 160 | + * A list of modified files within that bundle, each followed by its diff output shown in a diff code block. |
| 161 | +
|
| 162 | +
|
| 163 | +### Important Notes & Caveats |
| 164 | +* Backup: Always back up your original Apigee Edge bundles before running modifications. |
| 165 | +* LXML Formatting: While lxml is good, minor formatting differences (e.g., self-closing tags, attribute order normalization) might still occur compared to the absolute original. Test thoroughly. |
| 166 | +* TOML Rules: Carefully define your policy.toml rules. Incorrect paths or filters will prevent modifications. Test rules on a single bundle first. |
| 167 | +* Validation Naming: Validation in directory mode depends heavily on accurate proxy name inference from filenames. Mismatched names will lead to validation failures. |
| 168 | +* Error Handling: The script includes basic error handling, but complex bundle structures or unexpected TOML configurations might cause issues. Review logs if failures occur. |
0 commit comments