[FIX] jinja_to_web: fetch original templates in chunks

upg-2994884

Avoid `MemoryError` and/or killed process because of `malloc()` failing within
`lxml` / `libxml2`.

Debugging this by determining the size of involved datastructures through means
of `sys.getsizeof()` showed that the overly large memory consumption has 3
different sources:
1. During conversion: The global variable `templates_to_check` grows to
   hundreds of MiB after the various calls to `upgrade_jinja_fields()` by
   upgrade scripts.
2. During conversion: The call to `cr.dictfetchall()` to gather all
   templates(fields) that are to be converted, already consumes hundreds of
   MiB.
3. At the start of function `verify_upgraded_jinja_fields()`, the process is
   at ~1.5GiB because of (1) and (2). While iterating over all the templates in
   `templates_to_check`, no significant amount of memory is allocated on top of
   this *by python datastructures*. But, with each call to
   `is_converted_template_valid()`, the size of the process increases until it
   hits the RLIMIT. This function calls into `lxml` multiple times, suggesting
   that the memory is allocated in `malloc()` calls in the `lxml` and/or
   `libxml2` C library/ies, evading python's memory accounting and garbage
   collection.
   Internet research shows that `lxml` has a long history of different memory
   leaks in C code, plus some caching mechanism *across documents* that could
   be responsible[^1]. More recent versions of the module seem to have been
   improved, but still we're stuck with old versions.

This patch takes care of (2) by replacing the use of `cr.dictfetchall()` by a
loop over a named cursor with an itersize of 100.

[^1]: https://benbernardblog.com/tracking-down-a-freaky-python-memory-leak-part-2/

Part-of: #288
Signed-off-by: Christophe Simonis (chs) <[email protected]>

Author: cawo-odoo

src/util/jinja_to_qweb.py

@@ -290,14 +290,15 @@ def upgrade_jinja_fields(
     model = model_of_table(cr, table_name)
 
     cr.commit()  # ease the processing for PG
-    cr.execute(
+    ncr = named_cursor(cr, 100)
+    ncr.execute(
         f"""
         SELECT id, {name_field}, {sql_fields}
           FROM {table_name}
          WHERE {sql_where_fields}
         """
     )
-    for data in cr.dictfetchall():
+    for data in ncr.iterdict():
         _logger.info("process %s(%s) %s", table_name, data["id"], data[name_field])
 
         # only for mailing.mailing
@@ -352,6 +353,7 @@ def upgrade_jinja_fields(
                 """,
                 field_values + [data["id"]],
             )
+    ncr.close()
 
     if not table_exists(cr, "ir_translation"):
         return