This commit addresses a critical bug in `ipc-cli node init` that prevented libp2p from binding to network interfaces on cloud VMs (GCP, AWS, Azure). The fix ensures that `listen_addr` is set to `0.0.0.0` for proper binding, while `external_addresses` correctly advertises the public IP. This change restores functionality for parent finality voting and top-down message execution.

Changes include:
- Updated `ConnectionOverrideConfig` to include `external_addresses`.
- Modified port configuration logic to use `0.0.0.0` for `listen_addr`.
- Enhanced documentation in `CHANGELOG.md` and `node-init.md` to reflect these changes.
- Added tests to verify the correct configuration behavior.

Existing deployments may need to reinitialize or manually update their configurations to apply this fix.

This commit introduces a new `listen-ip` field in the P2P configuration, allowing advanced users to specify a custom IP address for binding services, while maintaining the default of `0.0.0.0` for maximum compatibility. This enhancement addresses previous limitations in binding on cloud VMs and improves flexibility for complex network setups.

Changes include:
- Updated `P2pConfig` structure to include the `listen-ip` field.
- Adjusted port configuration logic to utilize the `listen-ip` for binding.
- Enhanced documentation in `CHANGELOG.md` and `node-init.md` to reflect the new configuration options and usage examples.
- Added tests to ensure correct behavior of the new `listen-ip` functionality.

This update is fully backward compatible and does not require changes to existing configurations.

…ality issue

This commit updates the subnet configuration by changing the validator power from 1 to 3 and modifying the subnet ID to ensure compatibility with the latest deployment requirements. Additionally, a new markdown file is introduced to document the 16-hour lookback issue affecting parent finality on the Glif Calibration testnet, outlining the problem, root cause, and proposed solutions.

Changes include:
- Updated `ipc-subnet-config.yml` with new subnet ID and validator power.
- Added `PARENT-FINALITY-16H-LOOKBACK-ISSUE.md` to provide detailed insights into the parent finality issue and potential workarounds.

These updates aim to enhance the reliability and documentation of the IPC subnet management process.

…inality progress

This commit introduces a new `watch-finality` command to the IPC subnet manager, enabling users to monitor parent finality progress in real-time. The command supports continuous monitoring, target epoch tracking, and customizable refresh intervals.

Changes include:
- Added `cmd_watch_finality()` function in `ipc-subnet-manager.sh`.
- Updated usage documentation to include examples for the new command.
- Implemented `watch_parent_finality()` function in `lib/health.sh` for monitoring logic.
- Created `WATCH-FINALITY-FEATURE.md` to document usage, output, and potential use cases.

These enhancements improve the monitoring capabilities of the IPC subnet manager, facilitating better tracking of parent finality and subnet health.

…onitoring

This commit adds a new `watch-blocks` command to the IPC subnet manager, enabling users to monitor block production in real-time. The command supports continuous monitoring, target height tracking, and customizable refresh intervals.

Changes include:
- Implemented `cmd_watch_blocks()` function in `ipc-subnet-manager.sh`.
- Added `watch_block_production()` function in `lib/health.sh` for monitoring logic.
- Updated usage documentation with examples for the new command.
- Created `WATCH-BLOCKS-FEATURE.md` to document usage, output, and potential use cases.
- Adjusted `ipc-subnet-config.yml` to optimize block production settings.

These enhancements improve the monitoring capabilities of the IPC subnet manager, facilitating better tracking of block production and overall subnet health.

This commit introduces an extensive "Advanced Performance Tuning Guide" to optimize IPC subnet performance, detailing configuration changes and expected impacts on consensus timeouts, block production, and network performance. Additionally, a new script, `apply-advanced-tuning.sh`, is added to automate the application of these optimizations to existing nodes without reinitialization.

Changes include:
- Created `ADVANCED-TUNING-GUIDE.md` with detailed tuning parameters and expected performance improvements.
- Added `apply-advanced-tuning.sh` script for seamless configuration updates across validators.
- Updated `ipc-subnet-config.yml` with optimized settings for faster block production and parent finality.
- Introduced `OPTIMIZATION-SUMMARY.md` and `PERFORMANCE-OPTIMIZATION-RESULTS.md` to document performance improvements and configurations.
- Enhanced `TUNING-QUICK-REF.md` for quick access to tuning actions and parameters.

These enhancements significantly improve the performance and reliability of the IPC subnet, making it competitive with leading blockchain networks.

This commit introduces a comprehensive solution to address the broadcasting error encountered by validators due to incorrect address configuration. The changes include:

- Added `BOTTOMUP-CHECKPOINT-FIX.md` to document the problem, root cause, and the necessary fix for validator configurations.
- Created `fix-bottomup-checkpoint.sh` script to automate the process of disabling bottom-up checkpointing for federated subnets and updating validator configurations.
- Updated `lib/config.sh` to set the default validator key kind to "ethereum" for EVM-based subnets, preventing future issues.

These enhancements ensure that bottom-up checkpointing is operational and that validators are correctly configured for EVM compatibility, improving overall subnet reliability.

This commit adds a comprehensive live monitoring dashboard to the IPC subnet manager, enabling real-time tracking of various metrics and error categorization. Key changes include:

- Created `lib/dashboard.sh` for core dashboard functionality, including metrics collection and UI rendering.
- Added `cmd_dashboard()` function to `ipc-subnet-manager.sh` for command integration.
- Developed multiple documentation files detailing dashboard features, implementation, and quick reference guides.
- Enhanced error handling and formatting in the dashboard display for improved user experience.

These enhancements significantly improve the monitoring capabilities of the IPC subnet manager, providing users with a unified view of subnet health and activity.

This commit introduces a new `BottomUpSettings` struct to manage bottom-up checkpointing configurations, including an option to enable or disable the feature. Key changes include:

- Added `BottomUpSettings` struct with a default enabled state.
- Updated `IpcSettings` to include a configuration for bottom-up checkpointing.
- Enhanced `BottomUpManager` to accept a flag indicating whether bottom-up checkpointing is enabled.
- Implemented logic to conditionally execute bottom-up checkpointing based on the new settings.

These enhancements provide greater flexibility in managing checkpointing behavior within the IPC subnet, improving overall system reliability.

…t management

This commit introduces a comprehensive "Consensus Recovery Guide" and a "Diagnostic Tools Summary" to assist users in diagnosing and recovering from consensus issues within IPC subnets. Key changes include:

- Added `CONSENSUS-RECOVERY-GUIDE.md` detailing steps for diagnosing and resolving consensus problems, including commands for checking consensus and voting status.
- Introduced `DIAGNOSTIC-TOOLS-SUMMARY.md` outlining new commands like `consensus-status` and `voting-status`, enhancing the ability to monitor validator health and participation.
- Updated `ipc-subnet-manager.sh` to integrate new diagnostic commands.
- Enhanced `lib/health.sh` with functions to display consensus and voting statuses, improving operational visibility.

These enhancements significantly improve the operational capabilities of the IPC subnet manager, enabling targeted recovery actions without data loss and fostering better understanding of consensus dynamics.

…sting

This commit introduces several new scripts to enhance the IPC subnet manager's functionality. Key changes include:

- Added `enable-gateway-ports.sh` to enable GatewayPorts on remote VMs for SSH reverse tunneling.
- Introduced `setup-anvil-tunnels.sh` to establish SSH tunnels from local Anvil to remote validator nodes, allowing access to Anvil running on localhost.
- Created `test-anvil-connection.sh` to verify Anvil connectivity from remote VMs through the established SSH tunnels.
- Updated `ipc-subnet-config.yml` with new configuration settings for improved local and remote RPC endpoints.

These enhancements significantly improve the operational capabilities of the IPC subnet manager, facilitating better connectivity and management of validator nodes.

This commit introduces a new script, `debug-relayer-error.sh`, designed to assist in diagnosing issues related to checkpoint submission failures in the IPC subnet manager. Key features include:

- A series of connectivity checks to ensure the Anvil RPC is accessible.
- Validation of the existence of the Gateway and Subnet Actor contracts.
- Checks for the last bottom-up checkpoint height and subnet activity status.
- Recommendations for common issues encountered during relayer operations.

Additionally, new documentation files, including `FIXES-SUMMARY.md`, `IPC-CONFIG-ORDER-FIX.md`, and `RELAYER-UPDATE-SUMMARY.md`, have been added to summarize recent fixes and updates related to relayer connectivity and configuration management.

These enhancements significantly improve the operational capabilities of the IPC subnet manager, providing users with tools to effectively troubleshoot and resolve relayer-related issues.

This commit introduces a new documentation file, `INSTALL-SYSTEMD-FIX.md`, detailing fixes for common issues encountered during the installation of systemd services in the IPC subnet manager. Key changes include:

- Resolved installation issues where services were only installed on the first validator due to arithmetic expansion errors.
- Ensured the relayer service is installed correctly when requested.
- Added initialization for the `SCRIPT_DIR` variable in service generation functions to prevent template file access issues.
- Included steps to unmask services on affected validators before installation.

Additionally, improvements were made to the `ipc-subnet-manager.sh` and `lib/health.sh` scripts to enhance error handling and logging during the installation process.

These enhancements significantly improve the reliability and usability of the IPC subnet manager's systemd service installation process.

This commit updates the `ipc-subnet-config.yml` with new subnet IDs and contract addresses for improved configuration accuracy. Additionally, it introduces a `--debug` option in the `ipc-subnet-manager.sh` script to enable verbose logging during initialization and error handling, enhancing the debugging process. A new `RELAYER-AND-RESOLVER-FIX.md` documentation file is added, detailing fixes for relayer configuration issues and invalid resolver paths, ensuring better operational reliability.

… configuration improvements

This commit introduces a new command, `update-binaries`, to the `ipc-subnet-manager.sh` script, allowing users to pull the latest code, build, and install binaries on all validators. The command supports specifying a git branch for updates. Additionally, the `ipc-subnet-config.yml` file has been updated with new paths for the IPC repository, and several contract addresses have been modified for improved configuration accuracy. These enhancements streamline the process of maintaining validator binaries and ensure better operational reliability.

This commit adds functionality to convert the validator key to an Ethereum address using fendermint within the `show_subnet_info` function of `lib/health.sh`. It logs the converted address if successful, or warns if the conversion fails. This enhancement improves the visibility of validator information and aids in debugging by providing relevant Ethereum addresses alongside public keys.

This commit introduces a new script, `estimate-gas.sh`, designed to estimate gas usage for transactions between Ethereum addresses. The script utilizes JSON RPC to fetch gas estimates and provides a breakdown of costs at various gas prices. It also includes a recommendation for gas with a 20% buffer, enhancing the operational capabilities of the IPC subnet manager by aiding users in transaction cost planning.

This commit adds a newline at the end of the `estimate-gas.sh` script to ensure consistency with coding standards and improve readability. This minor adjustment helps maintain a clean file structure in the project.

This commit introduces a complete ELK (Elasticsearch, Logstash, Kibana) stack for aggregating logs from IPC validator nodes. Key components include:

- Docker Compose configuration for orchestrating the ELK stack.
- Elasticsearch for log storage and search capabilities.
- Logstash for processing and parsing logs from validators.
- Kibana for visualizing logs and creating dashboards.
- Grafana for alternative visualization options.

Additionally, comprehensive documentation is provided, including setup guides, troubleshooting tips, and monitoring instructions, ensuring a robust logging infrastructure for IPC validators.

This commit refactors the `get_chain_id` function in `lib/health.sh` to replace SSH commands with the `exec_on_host` function for executing remote commands. This change enhances consistency and simplifies the process of querying the Ethereum chain ID via JSON-RPC, improving the overall reliability of the health check functionality in the IPC subnet manager.

This commit updates the Logstash configuration in `ipc-logs.conf` to extract the hostname before cleanup, allowing for the use of a new field `validator_hostname` in the index name. This change improves the organization of logs by ensuring that the index is named consistently based on the validator's hostname, enhancing log management and retrieval.

This commit updates the `draw_dashboard` function in `dashboard.sh` to calculate the expected number of peers based on the count of validators, excluding the self-validator. This change enhances the accuracy of the network health status displayed in the dashboard, improving overall monitoring capabilities.

This commit updates the `fetch_metrics` function in `dashboard.sh` to include the fetching of the mempool maximum size from the CometBFT configuration. The maximum size is now dynamically set if not already defined, improving the accuracy of mempool metrics displayed in the dashboard. Additionally, the default value for `mempool_max` is adjusted to align with this change, enhancing overall monitoring capabilities.

This commit updates the `monitor-parent-finality-simple.sh` script to enhance the method of extracting finality information from logs. The previous use of `grep -P` has been replaced with a combination of `grep` and `sed` for better portability. This change ensures more reliable parsing of log entries, improving the accuracy of finality reporting in the monitoring process.

…er function

This commit updates the `set_federated_power` function in `lib/health.sh` to dynamically determine the `--from` address for transactions based on the primary validator's private key. If the address is not specified in the configuration, it derives the address from known Anvil accounts, improving flexibility and reducing configuration errors. Additionally, it logs the address being used for transactions, enhancing visibility during execution.

This commit modifies the `bottomup_enabled` method in `lib.rs` to return true by default when the bottom-up configuration is not specified. This change aligns with the intended default behavior of enabling bottom-up checkpointing, enhancing the clarity and consistency of the settings implementation.

…t manager path

This commit modifies the `elk-manager.sh` script to allow the IPC subnet manager configuration path to be set via an environment variable, enhancing flexibility. It updates the filebeat status check to use this variable, providing clearer error messages and guidance for users. Additionally, it improves logging to indicate the configuration file being used, streamlining the management of IPC subnet configurations.

Replace map_or with is_none_or as suggested by clippy lint.
This is more idiomatic and clearer in intent.

This commit addresses multiple SSH-related problems in the IPC manager when running in local mode. It replaces direct SSH calls with an abstraction layer function, ensuring commands execute locally without attempting SSH connections to localhost. Key functions affected include node management and subnet deployment, enhancing the overall functionality and reliability of the IPC manager in local environments. Additionally, new documentation files have been created to detail the fixes and verification steps.

This commit introduces a complete summary of fixes for SSH-related issues in the IPC manager, enabling full functionality in local mode on macOS. Key changes include the replacement of direct SSH calls with an abstraction layer, restoration of the `deploy_subnet` function, and updates to port checking logic for macOS compatibility. Additionally, new documentation files have been created to detail the fixes, verification steps, and technical changes, ensuring a smoother developer experience and improved command reliability in local environments.

This commit introduces a dedicated configuration option for subnet chain IDs in the IPC manager, addressing issues with chain ID collisions between parent and subnet networks. Key changes include the addition of a `chain_id` field in `ipc-subnet-config-local.yml`, updates to the `deploy_subnet()` function to utilize this configuration, and the creation of a Python utility for calculating chain IDs. These enhancements improve clarity, security, and usability for developers working with IPC subnets.