Research and Integrate Wazuh Agent Upgrade Functionality via Dashboard Upgrade Button

[t-desmond]

📋 Description

Investigate how the Wazuh Agent upgrade process works when triggered through the Wazuh Dashboard “Upgrade” button, understand why it might fail, and determine how to replicate or integrate this functionality into our custom implementation.

Currently, attempts to upgrade agents via the dashboard button appear to fail silently or incompletely. We need to:

• Identify the underlying mechanism the dashboard uses to trigger upgrades.
• Understand possible failure points (permissions, connectivity, configuration, version mismatches, etc.).
• Document steps to reproduce the issue.
• Propose and design a custom integration approach that replicates or improves the upgrade process programmatically.

---

🎯 Goals / Acceptance Criteria

• Research how the Wazuh Dashboard triggers agent upgrades (API calls, backend scripts, etc.).
• Identify dependencies or permissions required for the upgrade to succeed.
• Document potential reasons why the dashboard “Upgrade” button might fail (e.g., agent offline, Wazuh Manager configuration, endpoint OS compatibility, etc.).
• Provide a working proof of concept or API script that can trigger an agent upgrade.
• Add documentation on configuration or environment requirements.

---

🧠 Research Notes / Resources

• Wazuh documentation on [Agent upgrade management](https://documentation.wazuh.com/current/user-manual/agents/agent-upgrade.html)
• Wazuh API reference: PUT /agents/upgrade
• Logs to check: /var/ossec/logs/api.log, /var/ossec/logs/ossec.log

---

🐛 Possible Failure Causes to Investigate

• Missing or incorrect API credentials between dashboard and Wazuh manager.
• Agents running older versions that do not support in-place upgrades.
• Endpoint OS or package manager issues (e.g., yum/apt failures).
• Network connectivity or firewall restrictions.
• Misconfigured wazuh-manager or wazuh-api permissions.
• SSL/TLS certificate or token mismatches.

---

🧾 Deliverables

1. Research findings document (summary + root cause analysis).
2. Prototype or script demonstrating custom upgrade trigger.

[t-desmond]

Found potential causes of failures, implementing possible integration to current implementation

[t-desmond]

Implemented custum WPK package.. Facing usues with signature verification

[t-desmond]

Same timeout issue here

Possible fixes:

• Increase chunck size as described here
• Increase timeout as described here

[t-desmond]

Facing 2 issues:

2025/10/21 07:17:20 wazuh-authd: INFO: New connection from 10.42.0.1
2025/10/21 07:17:20 wazuh-authd: ERROR: SSL handshake failed for socket=9: error:0A0000C7:SSL routines::peer did not return a certificate

and

bash-5.2# ./agent_upgrade -a 002

Upgrading...

Failed upgrades:
	Agent 002 status: Send lock restart error

[t-desmond]

Wazuh Agent Upgrade Performance Test Report

Date: 2025-10-22
Environment: Wazuh Helm Cluster
Test Objective: Evaluate upgrade time with different <chunk_size> values.

---

Test Configuration

• <task_timeout>: 1 hour
• Upgrade module: agent-upgrade
• Chunk sizes tested: 4096, 8192, 16,384, 32,768, 40,000 bytes
• Default / baseline: no chunk size configured

---

Test Results

Chunk Size (bytes)	Observed Upgrade Time	Notes
Default (32768)	53 seconds	Baseline performance
4096	3 min 26 sec (206 seconds)	Logs indicate successful task dispatch and completion for agent ID 5
8192	51 seconds	Slight improvement over default
16,384	1 min 56 sec (116 seconds)	Logs indicate successful task dispatch and completion for agent ID 2
40,000	2 min 36 sec (156 seconds)	Logs indicate successful task dispatch and completion for agent ID 3

Note: Upgrade time may vary depending on network conditions, system load, and agent ID.

---

Observations

• Very small chunk sizes (e.g., 4096) increase total upgrade time due to more packets and overhead.
• Medium chunk sizes (8192–16,384 bytes) balance speed and reliability on stable LAN networks.
• Very large chunk sizes (40,000 bytes) may slightly increase time due to TCP segmentation/buffering.
• All upgrades completed successfully; task-manager logs confirmed "Success" for each agent.

[t-desmond]

Wazuh Agent Upgrade Performance Test Report

Date: 2025-10-22
Environment: Wazuh Helm Cluster
Test Objective: Evaluate upgrade time with different <chunk_size> values and <task_timeout> configurations.

---

Test Configuration

• <task_timeout> values: 5, 10, 15, and 20 minutes
• Upgrade module: agent-upgrade
• Chunk sizes tested: 4096, 8192, 16384, 32768 (default), and 40000 bytes
• All times recorded in seconds

---

Test Results (seconds)

Chunk Size (bytes)	Timeout: 5 min	Timeout: 10 min	Timeout: 15 min	Timeout: 20 min
Default (32768)	61.5	58	54	64.5
4096	64.7	55	58	56
8192	56	64	64	61
16,384	63	60	63	67
40,000	62	58	60	57.7

---

Procedure

1. Configure <chunk_size> and <task_timeout> in the agent-upgrade module
2. Initiate upgrade via the Wazuh dashboard
3. Monitor upgrade status through the dashboard interface
4. Repeat each test scenario 3–4 times

---

Observations

• Timeout scenarios could not be reproduced during testing
  • Most likely due to the agent and manager being located on the same network, minimizing latency
• The only encountered issue was a recurring “Send lock file error”, related to file transmission
• Every tested upgrade completed successfully
• Performance was stable, with upgrade durations consistently within ~55–65 seconds across all configurations

[MarantosGeorge]

Trying to test reducing network bandwidth in order to increase upgrade time.

[t-desmond]

Wazuh Agent Upgrade Chunk Size vs Timeout – Conclusion

Test Summary

• Tested chunk sizes: 4 KB, 8 KB, 16 KB, 32 KB (default), 40 KB
• Additional tests included 60,000 KB (maximum configurable) and 64 bytes (minimum configurable) chunk sizes for comparison
• Network conditions: both limited (1 Mbit, high latency) and full bandwidth
• Observed upgrade times: 54 s – 67 s (unlimited bandwidth), ~15 min (limited bandwidth)
• Each test repeated across multiple timeout settings: 5, 10, 15, 20 min

---

Additional Results

🧩 Chunk Size	Agent	Start Time	End Time	Duration
60,000 KB (Max)	034	09:46:34	09:47:32	58 seconds
	033	09:46:28	09:47:23	55 seconds
Average Duration (60,000 KB)				56.5 seconds
64 bytes (Min)	036	10:22:14	10:23:20	1 min 6 sec (66 s)
	035	10:22:07	10:23:10	1 min 3 sec (63 s)
Average Duration (64 bytes)				64.5 seconds

---

Findings

1. No clear trend with chunk size

   • Even with extreme chunk sizes (from 64 bytes to 60,000 KB), upgrade times remained within a narrow range (≈55–65 seconds) under fast network conditions.
   • Both on slow and fast networks, changing chunk size had minimal impact on total upgrade time.
   • Small chunks increase protocol overhead, large chunks increase per-chunk processing; effects roughly cancel.

2. Upgrade time dominated by other factors

   • CPU, disk I/O, encryption, protocol processing, and network jitter contribute more than chunk size.
   • On poor networks, bandwidth and latency dominate performance.

3. Implications for configuration

   • Chunk size: 32 KB – 64 KB recommended (safe, reasonable tradeoff).
   • Timeout: set based on observed median + margin
     • ~20–25 min for slow networks
     • ~2 min for fast networks

---

Conclusion

Even when testing extremes from 64 bytes (minimum) to 60,000 KB (maximum), chunk size had little effect on Wazuh agent upgrade performance. Configuring a reasonable chunk size and setting a timeout based on observed network and system conditions ensures reliable upgrades without wasted optimization effort.

[t-desmond]

MTN Cameroon has lowest performance according to Ookla
Upload/Download speeds can be found here

Facing issue with keeping a fixed bandwidth: