[IMP] jinja_to_qweb: avoid high memory use by lxml/libxml2

upg-2994884

Avoid `MemoryError` and/or killed process because of `malloc()` failing within
`lxml` / `libxml2`.

Debugging this by determining the size of involved datastructures through means
of `sys.getsizeof()` showed that the overly large memory consumption has 3
different sources:
1. During conversion: The global variable `templates_to_check` grows to
   hundreds of MiB after the various calls to `upgrade_jinja_fields()` by
   upgrade scripts.
2. During conversion: The call to `cr.dictfetchall()` to gather all
   templates(fields) that are to be converted, already consumes hundreds of
   MiB.
3. At the start of function `verify_upgraded_jinja_fields()`, the process is
   at ~1.5GiB because of (1) and (2). While iterating over all the templates in
   `templates_to_check`, no significant amount of memory is allocated on top of
   this *by python datastructures*. But, with each call to
   `is_converted_template_valid()`, the size of the process increases until it
   hits the RLIMIT. This function calls into `lxml` multiple times, suggesting
   that the memory is allocated in `malloc()` calls in the `lxml` and/or
   `libxml2` C library/ies, evading python's memory accounting and garbage
   collection.
   Internet research shows that `lxml` has a long history of different memory
   leaks in C code, plus some caching mechanism *across documents* that could
   be responsible[^1]. More recent versions of the module seem to have been
   improved, but still we're stuck with old versions.

This patch solves / works around (3) by running the function body of
`is_converted_template_valid()` in a subprocess that is killed immediately
after and thus no additional memory is ever allocated by `lxml` in the main
process.

[^1]: https://benbernardblog.com/tracking-down-a-freaky-python-memory-leak-part-2/

closes #288

Signed-off-by: Christophe Simonis (chs) <[email protected]>

Author: cawo-odoo

src/util/jinja_to_qweb.py

@@ -3,6 +3,7 @@
 import html
 import logging
 import re
+from multiprocessing import Process, Queue
 
 import babel
 import lxml
@@ -544,32 +545,42 @@ def verify_upgraded_jinja_fields(cr):
 
 
 def is_converted_template_valid(env, template_before, template_after, model_name, record_id, engine="inline_template"):
-    render_before = None
-    with contextlib.suppress(Exception):
-        render_before = _render_template_jinja(env, template_before, model_name, record_id)
-
-    render_after = None
-    if render_before is not None:
-        try:
-            with mute_logger("odoo.addons.mail.models.mail_render_mixin"):
-                render_after = env["mail.render.mixin"]._render_template(
-                    template_after, model_name, [record_id], engine=engine
-                )[record_id]
-        except Exception:
-            pass
-
-    # post process qweb render to remove comments from the rendered jinja in
-    # order to avoid false negative because qweb never render comments.
-    if render_before and render_after and engine == "qweb":
-        element_before = lxml.html.fragment_fromstring(render_before, create_parent="div")
-        for comment_element in element_before.xpath("//comment()"):
-            comment_element.getparent().remove(comment_element)
-        render_before = lxml.html.tostring(element_before, encoding="unicode")
-        render_after = lxml.html.tostring(
-            lxml.html.fragment_fromstring(render_after, create_parent="div"), encoding="unicode"
-        )
+    def callback(q):
+        render_before = None
+        with contextlib.suppress(Exception):
+            render_before = _render_template_jinja(env, template_before, model_name, record_id)
+
+        render_after = None
+        if render_before is not None:
+            try:
+                with mute_logger("odoo.addons.mail.models.mail_render_mixin"):
+                    render_after = env["mail.render.mixin"]._render_template(
+                        template_after, model_name, [record_id], engine=engine
+                    )[record_id]
+            except Exception:
+                pass
+
+        # post process qweb render to remove comments from the rendered jinja in
+        # order to avoid false negative because qweb never render comments.
+        if render_before and render_after and engine == "qweb":
+            element_before = lxml.html.fragment_fromstring(render_before, create_parent="div")
+            for comment_element in element_before.xpath("//comment()"):
+                comment_element.getparent().remove(comment_element)
+            render_before = lxml.html.tostring(element_before, encoding="unicode")
+            render_after = lxml.html.tostring(
+                lxml.html.fragment_fromstring(render_after, create_parent="div"), encoding="unicode"
+            )
+
+        q.put(render_before is not None and render_before == render_after)
 
-    return render_before is not None and render_before == render_after
+    # to avoid memory leaks in external C libraries (lxml/libxml2), process in a forked child
+    queue = Queue()
+    proc = Process(target=callback, args=[queue])
+    proc.start()
+    res = queue.get(timeout=60)
+    if proc.is_alive():
+        proc.kill()
+    return res
 
 
 # jinja render