Merge pull request #360 from Ratio1/develop

Develop

Author: cristibleotiu

AGENTS.md

@@ -3,11 +3,18 @@
 This file is the durable operating manual for future agents working in `/edge_node`.
 It has two goals:
 1. Stable reference information that should remain useful across sessions.
-2. Append-only memory of important discoveries, decisions, and changes.
+2. Curated high-signal memory of critical/fundamental changes and horizontal project insights.
 
 ## Hard Rules
-- Treat this file as append-only memory for the `Memory Log` section.
-- Never delete or rewrite prior memory entries.
+- Treat the `Memory Log` as a high-signal ledger, not a full activity history.
+- Log only:
+  - Critical/fundamental changes with architectural, security, operational, or reliability impact.
+  - Horizontal insights that affect multiple subsystems, onboarding, deployment, or runbook safety.
+- Do not log:
+  - Minor docs edits, section reordering, wording tweaks, formatting, or cosmetic refactors.
+  - Narrow/local changes without material behavioral or operational impact.
+- Keep the `Memory Log` append-only for qualifying entries.
+- Cleanup/removal of ballast entries is allowed only during explicit curation requested by project owners.
 - If an older entry is wrong, add a new correction entry that references the old entry ID.
 - Use UTC timestamps in ISO-8601 format: `YYYY-MM-DDTHH:MM:SSZ`.
 - Keep shell examples copy-pasteable.
@@ -110,39 +117,35 @@ BUILDER must:
 - Refine the change if needed.
 - List verification commands run and observed results (pass/fail + short evidence).
 
-### Step 4: Log It
-Append a `Memory Log` entry with:
+### Step 4: Log It (Critical-Only)
+Append a `Memory Log` entry only when the change/insight is critical or fundamental.
+Each entry must include:
 - Timestamp and entry ID.
 - Summary of change and decision.
+- Why this is critical/horizontal.
 - CRITIC findings summary.
 - Verification commands and outcomes.
 - If correction: `Correction of: <entry_id>`.
 
-## Memory Log (append-only)
+## Memory Log (critical-only; append-only for qualifying entries)
 
 Entry format:
 - `ID`: `ML-YYYYMMDD-###`
 - `Timestamp`: UTC ISO-8601
 - `Type`: discovery | decision | change | correction
 - `Summary`:
+- `Criticality`:
 - `Details`:
 - `Verification`:
 - `Links`:
 
 ---
 
-- ID: `ML-20260211-001`
-- Timestamp: `2026-02-11T09:13:34Z`
-- Type: `discovery`
-- Summary: Repo-wide docs/ops audit performed to ground AGENTS/README rewrite.
-- Details: Confirmed runtime entrypoints (`device.py`, `constants.py`), operational scripts (`cmds/`), compose variants (`docker-compose/`), and deployment artifacts (`docker/`, `k8s/`).
-- Verification: `pwd && ls -la`; `find . -maxdepth 2 -type d | sort`; `find extensions -type f`; `find plugins -type f`
-- Links: `device.py`, `constants.py`, `docker-compose/debug-docker-compose.yaml`, `cmds/get_node_info`
-
 - ID: `ML-20260211-002`
 - Timestamp: `2026-02-11T09:13:34Z`
 - Type: `discovery`
 - Summary: Found operational mismatches that can break onboarding.
+- Criticality: Cross-cutting operations/onboarding risk across local dev, compose, and k8s paths.
 - Details: `debug.sh` builds `local_node` while debug compose expects `local_edge_node`; `docker-compose/debug_start.bat` references missing `Dockerfile_dev`; multiple `k8s/` naming/namespace/PVC path mismatches exist.
 - Verification: `rg -n "local_edge_node|local_node" -S`; `sed -n '1,120p' debug.sh`; `sed -n '1,160p' docker-compose/debug_start.bat`; `sed -n '1,220p' k8s/README.md`; `sed -n '1,220p' k8s/edgenode-deploy.yaml`; `sed -n '1,220p' k8s/edgenode-sa.yaml`; `sed -n '1,220p' k8s/edgenode-storage.yaml`
 - Links: `debug.sh`, `docker-compose/debug-docker-compose.yaml`, `docker-compose/debug_start.bat`, `k8s/README.md`, `k8s/edgenode-deploy.yaml`
@@ -151,46 +154,16 @@ Entry format:
 - Timestamp: `2026-02-11T09:13:34Z`
 - Type: `change`
 - Summary: Replaced prior short AGENTS guidance with durable long-term memory structure and mandatory BUILDER-CRITIC loop.
+- Criticality: Foundation process change governing agent behavior and decision quality.
 - Details: Added stable sections for run/test, repo map, conventions, pitfalls; established append-only log protocol with correction semantics.
 - Verification: `sed -n '1,260p' AGENTS.md`
 - Links: `AGENTS.md`
 
-- ID: `ML-20260211-004`
-- Timestamp: `2026-02-11T09:19:14Z`
-- Type: `change`
-- Summary: Rewrote `README.md` to prioritize operator usability and ordered sections as requested.
-- Details: Added explicit Need/Objective/Purpose, moved all practical usage content under `Usability & Features` (quickstart/examples/config/outputs/troubleshooting), and placed architecture/modules/deps/testing/security under `Technical Details`.
-- Verification: `git diff -- README.md`; `sed -n '1,320p' README.md`
-- Links: `README.md`
-
-- ID: `ML-20260211-005`
-- Timestamp: `2026-02-11T09:19:14Z`
-- Type: `discovery`
-- Summary: Verification run identified environment/tooling limits and current test signal.
-- Details: In this workspace, `docker` exists but neither `docker compose` plugin nor `docker-compose` binary is available; compose syntax could not be validated by execution. `python3 -m unittest discover -s plugins -p "*test*.py"` returns `Ran 0 tests`. Focused RedMesh suite runs but currently fails (`34` tests run, `1` failure, `3` errors with missing `service_info`/`web_tests_info` keys).
-- Verification: `command -v docker || true; command -v python3 || true`; `docker compose -f docker-compose/debug-docker-compose.yaml config`; `docker-compose -f docker-compose/debug-docker-compose.yaml config`; `python3 -m unittest discover -s plugins -p "*test*.py"`; `python3 -m unittest extensions.business.cybersec.red_mesh.test_redmesh`
-- Links: `README.md`, `AGENTS.md`, `extensions/business/cybersec/red_mesh/test_redmesh.py`
-
-- ID: `ML-20260211-006`
-- Timestamp: `2026-02-11T09:20:06Z`
-- Type: `change`
-- Summary: Added Compose command compatibility note (`docker-compose` and Compose v2 `docker compose`) in stable docs.
-- Details: Kept primary examples aligned to repo scripts (`docker-compose`) while explicitly documenting the Compose v2 equivalent to reduce operator ambiguity.
-- Verification: `rg -n "docker-compose|docker compose" AGENTS.md README.md`
-- Links: `AGENTS.md`, `README.md`
-
-- ID: `ML-20260211-007`
-- Timestamp: `2026-02-11T12:27:43Z`
+- ID: `ML-20260212-009`
+- Timestamp: `2026-02-12T14:32:58Z`
 - Type: `change`
-- Summary: Restored README citation section with the original BibTeX entries and improved end-of-doc discoverability.
-- Details: BUILDER Intent: restore dropped `## Citation` content exactly and place it before `## License` so researchers can find it quickly. Change scope: `README.md`, `AGENTS.md`. Assumptions: prior citations were correct and should be preserved verbatim. CRITIC findings: risk of accidental BibTeX drift or malformed markdown fence during reinsertion. BUILDER response: copied entries verbatim from provided/previous content, used explicit `bibtex` fenced blocks, and validated marker strings and line placement.
-- Verification: `rg -n "^## Citation|@misc\\{Ratio1EdgeNode|@inproceedings\\{Damian2025CSCS|@misc\\{Damian2025arXiv" README.md` (pass: section + all three entries found); `nl -ba README.md | sed -n '165,260p'` (pass: citation block rendered between Related Repositories and License)
-- Links: `README.md`, `AGENTS.md`
-
-- ID: `ML-20260211-008`
-- Timestamp: `2026-02-11T12:30:01Z`
-- Type: `change`
-- Summary: Restored the previously removed `Contact` and `Project Financing Disclaimer` sections in README.
-- Details: BUILDER Intent: reinsert the exact prior sections requested by user while keeping the recently restored citation block intact. Change scope: `README.md`, `AGENTS.md`. Assumptions: old section wording remained authoritative and should be restored verbatim. CRITIC findings: risk of introducing wording drift or placing sections in a confusing position near document end. BUILDER response: copied text directly from pre-rewrite README snapshot and inserted it immediately before `## License` to keep end-of-doc informational/legal sections grouped.
-- Verification: `rg -n "^## Contact|^## Project Financing Disclaimer|support@ratio1.ai|SMIS 143488|SMIS 156084" README.md` (pass: both headings and key markers found); `nl -ba README.md | sed -n '210,280p'` (pass: sections rendered with expected paragraphs and order)
-- Links: `README.md`, `AGENTS.md`
+- Summary: Re-scoped AGENTS memory policy to critical-only logging and pruned prior ballast entries.
+- Criticality: Fundamental governance change for long-term agent memory quality and signal-to-noise control.
+- Details: Updated Hard Rules and BUILDER-CRITIC Step 4 to enforce critical/horizontal-only logging; removed non-critical historical entries (`ML-20260211-001`, `ML-20260211-004`, `ML-20260211-005`, `ML-20260211-006`, `ML-20260211-007`, `ML-20260211-008`) per owner request.
+- Verification: `rg -n "critical-only|ballast|Criticality|ML-20260211-00[1245678]|ML-20260212-009" AGENTS.md`; `sed -n '1,260p' AGENTS.md`
+- Links: `AGENTS.md`

extensions/business/container_apps/container_app_runner.py

@@ -3081,6 +3081,10 @@ def _restart_container(self, stop_reason=None):
     self._stop_container_and_save_logs_to_disk()
     self.__reset_vars()
 
+    # Reset chainstore response for restart cycle
+    self.reset_chainstore_response()
+    self.set_plugin_ready(False)
+
     # Restore preserved state (reset_vars clears it)
     self._consecutive_failures = preserved_failures
     self._last_successful_start = preserved_last_success

extensions/business/deeploy/deeploy_manager_api.py

@@ -37,6 +37,7 @@
   'SUPRESS_LOGS_AFTER_INTERVAL' : 300,
   'WARMUP_DELAY' : 300,
   'PIPELINES_CHECK_DELAY' : 300,
+  'MIN_ETH_BALANCE' : 0.00005,
 
   'VALIDATION_RULES': {
     **BasePlugin.CONFIG['VALIDATION_RULES'],
@@ -74,8 +75,47 @@ def on_init(self):
     )
     self.__warmup_start_time = self.time()
     self.__last_pipelines_check_time = 0
+    if not self.__check_eth_balance():
+      self.P(
+        f"Shutting down tunnel engine for {self.__class__.__name__} due to insufficient ETH balance "
+        f"on {my_eth_address}. Please top up and restart the node.",
+        color='r', boxed=True
+      )
+      self.maybe_stop_tunnel_engine()
     return
 
+  def __check_eth_balance(self):
+    """
+    Check if the oracle has enough ETH to cover gas fees for web3 transactions.
+    Returns True if balance is sufficient, False otherwise.
+    """
+    try:
+      eth_address = self.bc.eth_address
+      balances = self.bc.get_addresses_balances([eth_address])
+      eth_balance = balances.get(eth_address, {}).get("ethBalance", 0)
+      if eth_balance < self.cfg_min_eth_balance:
+        self.P(
+          f"Insufficient ETH balance for oracle {eth_address}: "
+          f"{eth_balance:.6f} ETH < {self.cfg_min_eth_balance} ETH minimum.",
+          color='r'
+        )
+        return False
+      return True
+    except Exception as e:
+      self.P(f"Failed to check ETH balance: {e}", color='r')
+      return False
+
+  def __ensure_eth_balance(self):
+    """
+    Check ETH balance and raise ValueError if insufficient for web3 transactions.
+    Called at the top of mutating endpoints.
+    """
+    if not self.__check_eth_balance():
+      raise ValueError(
+        f"{DEEPLOY_ERRORS.GENERIC}: Oracle {self.bc.eth_address} does not have enough ETH "
+        f"to cover gas fees. Please top up the address and retry."
+      )
+
   def __handle_error(self, exc, request, extra_error_code=DEEPLOY_ERRORS.GENERIC):
     """
     Handle the error and return a response.
@@ -164,6 +204,7 @@ def _process_pipeline_request(
         The response dictionary
     """
     try:
+      self.__ensure_eth_balance()
       sender, inputs = self.deeploy_verify_and_get_inputs(request)
       normalized_request = self._normalize_plugins_input(self.deepcopy(request))
       if DEEPLOY_KEYS.PLUGINS in normalized_request:
@@ -622,13 +663,14 @@ def scale_up_job_workers(self,
         A dictionary with the result of the operation
     """
     try:
-      sender, inputs = self.deeploy_verify_and_get_inputs(request)   
+      self.__ensure_eth_balance()
+      sender, inputs = self.deeploy_verify_and_get_inputs(request)
       auth_result = self.deeploy_get_auth_result(inputs)
       job_id = inputs.get(DEEPLOY_KEYS.JOB_ID, None)
       if not job_id:
         msg = f"{DEEPLOY_ERRORS.REQUEST13}: Job ID is required."
         raise ValueError(msg)
-      
+
       is_confirmable_job = inputs.chainstore_response
 
       # check payment
@@ -659,13 +701,12 @@ def scale_up_job_workers(self,
       nodes = list(cstore_response["node"] for cstore_response in dct_status.values())
       self.Pd(f"Nodes to confirm: {self.json_dumps(nodes, indent=2)}")
 
-      self._submit_bc_job_confirmation(str_status=str_status, 
-                                       dct_status=dct_status, 
-                                       nodes=nodes, 
-                                       job_id=job_id, 
+      self._submit_bc_job_confirmation(str_status=str_status,
+                                       dct_status=dct_status,
+                                       nodes=nodes,
+                                       job_id=job_id,
                                        is_confirmable_job=is_confirmable_job)
 
-
       return_request = request.get(DEEPLOY_KEYS.RETURN_REQUEST, False)
       if return_request:
         dct_request = self.deepcopy(request)
@@ -715,6 +756,7 @@ def delete_pipeline(self,
         A dictionary with the result of the operation
     """
     try:
+      self.__ensure_eth_balance()
       self.Pd(f"Called Deeploy delete_pipeline endpoint")
       sender, inputs = self.deeploy_verify_and_get_inputs(request)
       auth_result = self.deeploy_get_auth_result(inputs)
@@ -774,6 +816,7 @@ def send_instance_command(self,
         A dictionary with the result of the operation
     """
     try:
+      self.__ensure_eth_balance()
       self.Pd(f"Called Deeploy send_instance_command endpoint")
       sender, inputs = self.deeploy_verify_and_get_inputs(request)
       auth_result = self.deeploy_get_auth_result(inputs)
@@ -827,6 +870,7 @@ def send_app_command(self,
         A dictionary with the result of the operation
     """
     try:
+      self.__ensure_eth_balance()
       self.Pd(f"Called Deeploy send_app_command endpoint")
       sender, inputs = self.deeploy_verify_and_get_inputs(request)
       auth_result = self.deeploy_get_auth_result(inputs)

extensions/business/deeploy/deeploy_mixin.py

@@ -2454,6 +2454,11 @@ def _get_online_apps(self, owner=None, target_nodes=None, job_id=None, project_i
             continue
           filtered_result[node][app_name] = app_data
       result = filtered_result
+
+    for node, apps in result.items():
+      node_alias = self.netmon.network_node_eeid(node)
+      for _, app_data in apps.items():
+        app_data["node_alias"] = node_alias
     return result
 
   # TODO: REMOVE THIS, once instance_id is coming from ui for instances that have to be updated

extensions/business/edge_inference_api/base_inference_api.py

@@ -85,6 +85,8 @@
   "REQUEST_TIMEOUT": 600,  # 10 minutes
   "SAVE_PERIOD": 300,  # 5 minutes
 
+  "LOG_REQUESTS_STATUS_EVERY_SECONDS": 5,  # log pending request status every 5 seconds
+
   "REQUEST_TTL_SECONDS": 60 * 60 * 2,  # keep historical results for 2 hours
   "RATE_LIMIT_PER_MINUTE": 5,
   "AUTH_TOKEN_ENV": "INFERENCE_API_TOKEN",
@@ -129,6 +131,7 @@ def on_init(self):
       self.P(err_msg)
       raise ValueError(err_msg)
     # endif AI_ENGINE not specified
+    self._request_last_log_time: Dict[str, float] = {}
     self._requests: Dict[str, Dict[str, Any]] = {}
     self._api_errors: Dict[str, Dict[str, Any]] = {}
     # TODO: add inference metrics tracking (latency, tokens, etc)
@@ -569,7 +572,11 @@ def solve_postponed_request(self, request_id: str):
         Request result when completed or failed, or a PostponedRequest for pending work.
       """
       if request_id in self._requests:
-        self.Pd(f"Checking status of request ID {request_id}...")
+        last_logged_status = self._request_last_log_time.get(request_id, 0)
+        if (self.time() - last_logged_status) > self.cfg_log_requests_status_every_seconds:
+          self.Pd(f"Checking status of request ID {request_id}...")
+          self._request_last_log_time[request_id] = self.time()
+        # endif logging status
         request_data = self._requests[request_id]
 
         self.maybe_mark_request_timeout(request_id=request_id, request_data=request_data)