Zend Memory Manager chapter

Author: Julien Pauli

Book/php7/internal_types.rst

@@ -13,7 +13,6 @@ Contents:
     internal_types/zvals.rst
     internal_types/strings.rst
     internal_types/zend_resources.rst
-    
-..
     internal_types/hashtables.rst
+..
     internal_types/objects/classes_and_objects.rst  

Book/php7/internal_types/hashtables.rst

@@ -0,0 +1,4 @@
+HashTables
+==========
+
+

Book/php7/internal_types/zvals/basic_structure.rst

@@ -13,8 +13,8 @@ the type can change during the life of a zval, so if the zval previously stored
 later point in time.
 
 The type is stored as an integer tag (an unsigned int). It can be one of several values. Some values correspond to the eight
-types available in PHP, others are used for internal engine purpose only. These values are referred to using constants of the form ``IS_TYPE``. E.g. ``IS_NULL``
-corresponds to the null type and ``IS_STRING`` corresponds to the string type.
+types available in PHP, others are used for internal engine purpose only. These values are referred to using constants 
+of the form ``IS_TYPE``. E.g. ``IS_NULL`` corresponds to the null type and ``IS_STRING`` corresponds to the string type.
 
 The actual value is stored in a union, which is defined as follows::
 
@@ -60,7 +60,9 @@ Secondly, ``zend_long`` represents an abstraction of the platform long, so whate
 ``zend_long`` weights 4 bytes on 32bit platforms and 8 bytes on 64bit ones.
 
 In addition to that, you may use macros related to longs, ``SIZEOF_ZEND_LONG`` or ``ZEND_LONG_MAX`` f.e.
-See Zend/zend_long.h in source code for more informations.
+See 
+`Zend/zend_long.h <https://github.com/php/php-src/blob/c3b910370c5c92007c3e3579024490345cb7f9a7/Zend/zend_long.h>`_
+in source code for more informations.
 
 The ``double`` type used to store floating point numbers is (typically) an 8-byte value following the IEEE-754
 specification. The details of this format won't be discussed here, but you should at least be aware of the fact that
@@ -74,7 +76,7 @@ The remaining four types will only be mentioned here quickly and discussed in gr
 
 Strings (``IS_STRING``) are stored in a ``zend_string`` structure, i.e. they consist of a ``char *`` string
 and an ``size_t`` length. You will find more informations about the ``zend_string`` structure and its dedicated API
-into the :doc:`/php7/zend_strings` chapter.
+into the :doc:`string <../strings>` chapter.
 
 Arrays use the ``IS_ARRAY`` type tag and are stored in the ``zend_array *arr`` member. How the ``HashTable`` structure
 works will be discussed in the :doc:`/hashtables` chapter.
@@ -120,14 +122,14 @@ internal use-case only. The zval structure has been thought to be very flexible,
 virtually any type of data of interest, and not only the PHP specific types we just reviewed above.
 
 The special ``IS_UNDEF`` type has a special meaning. That means "This zval contains no data of interest, do not access
-any data field from it". This is used for :doc:`/php7/zvals/memory_management` purposes. If you see an ``IS_UNDEF`` zval,
+any data field from it". This is used for :doc:`zvals/memory_management` purposes. If you see an ``IS_UNDEF`` zval,
 that means that it is of no special type and contains no valid information.
 
 The ``zend_refcounted *counted`` field is very tricky to understand. Basically, that field serve as a header for any
-other reference-countable type. This part is detailed into the :doc:`/zvals/memory_management` chapter.
+other reference-countable type. This part is detailed into the :doc:`zvals/memory_management` chapter.
 
 The ``zend_reference *ref`` is used to represent a PHP reference. The ``IS_REFERENCE`` type flag is then used.
-Here as well, we dedicated a chapter to such a concept, have a look at the :doc:`/php7/zvals/memory_management` chapter.
+Here as well, we dedicated a chapter to such a concept, have a look at the :doc:`zvals/memory_management` chapter.
 
 The ``zend_ast_ref *ast`` is used when you manipulate the AST from the compiler. The PHP compilation is detailed into
 the :doc:`/php7/compiler` chapter.

Book/php7/memory_management.rst

@@ -1,6 +1,29 @@
 Memory management
 =================
 
+C programmers usually have to deal with memory by hand. With dynamic memory, the programmer allocates memory when 
+needing and frees it when finished. Failing to free dynamic memory leads to a "memory leak", which may or may not be a 
+bad thing. In the case of PHP, as the process could live for a virtually infinite amount of time, creating a memory leak 
+will be dramatic. In any situation, leaking memory really translates to poorly and badly designed programs that cannot 
+be trusted.
+Memory leaking is easy to understand. You ask the OS to book some part of the main machine memory for you, but you never 
+tell it to release it back for other processes usage : you are not alone on the machine, other processes need some 
+memory, and the OS itself as well.
+
+Also, in C, memory areas are clearly bound. Reading or writing before or after the bounds is a very nasty operation.
+That will lead for sure to a crash, or worse an exploitable security issue. There are no magical things like 
+auto-resizeable areas with the C language. You must clearly tell the machine (and the CPU) what you want it to do. No 
+guess, no magic, no automation of any kind (like garbage collection).
+
+PHP's got a very specific memory model, and provides its own layer over the traditionnal libc's dynamic memory 
+allocator. This layer is called **Zend Memory Manager**.
+
+This chapter explains you what Zend Memory Manager is, how you must use it, and what you must/must not do with it.
+After that, we'll quickly introduce you to specific tools used in the C area to debug dynamic memory.
+
+.. note:: If you need, please get some (possibly strong) knowledge about C memory allocation classes (static vs 
+          dynamic memory), and about libc's allocator.
+
 Contents:
 
 .. toctree::

Book/php7/memory_management/zend_memory_manager.rst

@@ -1,3 +1,221 @@
 Zend Memory Manager
 ===================
 
+Zend Memory Manager, often abbreviated as ZendMM or ZMM, is a C layer that aims to provide abilities to allocate and 
+release dynamic **request-bound** memory.
+
+Note the "request-bound" in the above sentence.
+
+ZendMM is not just a classical layer over libc's dynamic memory allocator, mainly represented by the couple API calls 
+``malloc()/free()``. ZendMM is about request-bound memory that PHP must allocate while treating a request.
+
+The two main kind of dynamic memory pools in PHP
+************************************************
+
+PHP is a share-nothing architecture. Well, not at 100%. Let us explain.
+
+.. note:: You may need to read :doc:`the PHP lifecycle chapter <../extensions_design/php_lifecycle>` before continuing 
+          here, you'll get additionnal informations about the different steps and cycles that can be drawn from PHP 
+          lifetime.
+
+PHP can treat several (dozen, thousands ?) of requests into the same process. By default, PHP will forget anything it 
+knows of the current request, when that later finishes.
+
+"Forgetting" things translates to freeing any dynamic buffer that got allocated while treating a request. That means 
+that when in the process of treating a request, one must not allocate dynamic memory using traditionnal libc calls.
+Doing that is perfectly valid, but you give a chance to forget to free such a buffer.
+
+ZendMM comes with an API that substitute to libc's dynamic allocator, by copying its API. When in the process of 
+treating a request, the programmer must use that API instead of libc's allocator.
+
+For example, when PHP treats a request, it will parse PHP files. Those ones will lead to functions and classes 
+declarations, for example. When the compiler comes to compile the PHP files, it will allocate some dynamic memory to 
+store classes and functions it discovers. But, at the end of the request, PHP will forget about those latter. By 
+default, PHP forgets *a very huge number* of informations from one request to another.
+
+There exists however some pretty rare informations you need to persist across several requests. But that's uncommon.
+
+What could be kept unchanged throught requests ? What we call **persistent** objects. Once more let us insist : those 
+are rare cases. For example, the current PHP executable path won't change from requests to requests. That latter 
+information is allocated permanently, that means it is allocated with a traditionnal libc's ``malloc()`` call.
+
+What else ? Some strings. For example, the "_SERVER" string will be reused from request to request, as every request 
+will create the ``$_SERVER`` PHP array. So the "_SERVER" string itself can be permanently allocated, because it will be 
+allocated once
+
+What you must remember:
+
+* There exists two kinds of dynamic memory allocations while programming PHP (or extensions):
+    * Request-bound dynamic allocations
+    * Permanent dynamic allocations
+
+* Request-bound dynamic memory allocations
+    * Must only be performed when PHP is treating a request (not before, nor after)
+    * Can only be performed using the ZendMM dynamic memory allocation API
+    * Are very common, basically 95% of your dynamic allocations will be request-bound
+    * Are tracked by ZendMM, and you'll be informed about bad usage of the memory area, or if you leak
+
+* Permanent dynamic memory allocations
+    * Should not be performed while PHP is treating a request (not forbidden, but a bad idea)
+    * Are not tracked by ZendMM, and you won't be informed about bad usage of the memory area, or if you leak
+    * Should be pretty rare in an extension
+
+Also, keep in mind that all PHP source code has been based on such a memory level. Thus, many internal structures get 
+allocated using the Zend Memory Manager. Most of them got a "persistent" API call, which when used, lead to 
+traditionnal libc allocation.
+
+Here is a request-bound allocated :doc:`zend_string <../internal_types/strings/zend_strings>`::
+
+    zend_string *foo = zend_string_init("foo", strlen("foo"), 0);
+
+And here is the persistent allocated one::
+
+    zend_string *foo = zend_string_init("foo", strlen("foo"), 1);
+
+Same for :doc:`HashTable <../internal_types/hashtables>`. Request-bound allocated one::
+
+    zend_array ar;
+    zend_hash_init(&ar, 8, NULL, NULL, 0);
+
+Persistent allocated one::
+
+    zend_array ar;
+    zend_hash_init(&ar, 8, NULL, NULL, 1);
+
+It is always the same in all the different Zend APIs. Usually, it is weither a "0" to pass as last parameter to mean 
+"I want this structure to be allocated using ZendMM, so request-bound", or "1" meaning "I want this structure to get 
+allocated bypassing ZendMM and using a traditionnal libc's ``malloc()`` call".
+
+Obviously, those structures provide an API that remembers how it did allocate the structure, to use the right 
+deallocation function when destroyed. Hence in such a code::
+
+    zend_string_release(foo);
+    zend_hash_destroy(&ar);
+
+The API knows whether those structures were allocated using request-bound allocation, or permanent one, and in the 
+first case will use ``efree()`` to release it, and in the second case libc's ``free()``.
+
+Zend Memory Manager API
+***********************
+
+The API is located into 
+`Zend/zend_alloc.h <https://github.com/php/php-src/blob/c3b910370c5c92007c3e3579024490345cb7f9a7/Zend/zend_alloc.h>`_
+
+The API calls are mainly C macros and not functions, so get prepared if you debug them and want to look at how they 
+work. Those calls copy libc's calls, they usually add an "e" in the function name; So you should not be lost, and there 
+is not many things to detail about the API.
+
+Basically what you'll use most are ``emalloc(size_t)`` and ``efree(void *)``.
+
+You are also provided with ``ecalloc(size_t nmemb, size_t size)`` that allocates ``nmemb`` of individual size ``size``, 
+and zeroes the area. If you are a strong C programmer with experience, you should know that whenever possible, it is 
+better to use ``ecalloc()`` over ``emalloc()`` as ``ecalloc()`` will zero out the memory area which could help a lot in 
+pointer bug detection. Remember that ``emalloc()`` works basically like the libc ``malloc()``: it will look for a big 
+enough area in different pools, and return you the best fit. So you may be given a recycled pointer which points to 
+garbage.
+
+Then comes ``safe_emalloc(size_t nmemb, size_t size, size_t offset)``, which is an ``emalloc(size * nmemb + offset)`` 
+but that does check against overflows for you. You should use this API call if the numbers you must provide come from an 
+untrusted source, like the userland.
+
+About string facilities, ``estrdup(char *)`` and ``estrndup(char *, size_t len)`` allow to duplicate strings or binary 
+strings.
+
+Whatever happens, pointers returned by ZendMM must be freed using ZendMM, aka ``efree()`` call and 
+**not libc's free()**.
+
+Zend Memory Manager debugging shields
+*************************************
+
+ZendMM provides the following abilities:
+
+* Memory consumption management.
+* Memory leak tracking.
+* Buffer overflows or underflows.
+
+Memory consumption management
+-----------------------------
+
+ZendMM is the layer behind the PHP userland "memory_limit" feature. Every single byte allocated using the ZendMM layer 
+is counted and added. When the INI's *memory_limit* is reached, you know what happens.
+That also mean that any allocation you perform via ZendMM is reflected in the ``memory_get_usage()`` call from PHP 
+userland.
+
+As an extension developper, this is a good thing, because it helps mastering the PHP process' heap size.
+
+If a memory limit error is launched, the engine will bail out from the current code position to a catch block, and will 
+terminate smoothly. But there is no chance it goes back to the location in your code where the limit blew up.
+You must be prepared to that.
+
+That means that in theory, ZendMM cannot return a NULL pointer to you. If the allocation fails from the OS, or if the 
+allocation generates a memory limit error, the code will run into a catch block and won't return to you allocation call.
+
+If for any reason you need to bypass that protection, you must then use a traditionnal libc call, like ``malloc()``. 
+Take care however and know what you do. It may happen that you need to allocate lots of memory and could blow up the PHP 
+*memory_limit* if using ZendMM. Thus use another allocator (like libc) but take care: your extension will grow the 
+current process heap size. That cannot be seen using ``memory_get_usage()`` in PHP, but by analyzing the current heap 
+with the OS facilities (like */proc/{pid}/maps*)
+
+.. note:: If you need to fully disable ZendMM, you can launch PHP with the ``USE_ZEND_ALLOC=0`` env var. This way, every 
+          call to the ZendMM API (like ``emalloc()``) will be directed to a libc call, and ZendMM will be disabled.
+          This is especially useful when :doc:`debugging memory <./memory_debugging>`.
+
+Memory leak tracking
+--------------------
+
+Remember the main ZendMM rules: it starts when a request starts, it then expects you call its API when in need of 
+dynamic memory as you are treating a request. When the current request ends, ZendMM shuts down.
+
+By shutting down, it will browse every of its active pointer, and if using 
+:doc:`a debug build<../build_system/building_php>` of PHP, it will warn you about memory leaking.
+
+Let's be clear here: if at the end of the current request ZendMM finds some active memory blocks, that means those are 
+leaking. There should not be any active memory block living onto ZendMM heap at the end of the request, as anyone who 
+allocated some should have freed them.
+
+If you forget to free blocks, they will all get displayed on *stderr*. This process of memory leak reporting only works 
+in the following conditions:
+
+* You are using :doc:`a debug build<../build_system/building_php>` of PHP
+* You have report_memleaks=On in php.ini (default)
+
+Here is an example of a simple leak into an extension::
+
+    PHP_RINIT_FUNCTION(example)
+    {
+        void *foo = emalloc(128);
+    }
+
+When launching PHP with that extension activated, on a debug build, that generates on stderr::
+
+    [Fri Jun 9 16:04:59 2017]  Script:  '/tmp/foobar.php'
+    /path/to/extension/file.c(123) : Freeing 0x00007fffeee65000 (128 bytes), script=/tmp/foobar.php
+    === Total 1 memory leaks detected ===
+
+Those lines are generated when the Zend Memory Manager shuts down, that is at the end of each treated request.
+
+Beware however:
+
+* Obviously ZendMM doesn't know anything about persistent allocations, or allocations that were perform in another way 
+  than using it. Hence, ZendMM can only warn you about allocations it is aware of, every traditionnal libc allocation 
+  won't be reported in here.
+* If PHP shuts down in an incorrect maner (what we call an unclean shutdown), ZendMM will report tons of leaks. This is 
+  because when incorrectly shutdown, the engine uses a longjmp() call to a catch block, preventing every code that cleans 
+  memory to fire-in. Thus, many leaks get reported. This happens especially after a call to PHP's exit()/die(), or if a 
+  fatal error gets triggered in some critical parts of PHP.
+* If you use a non-debug build of PHP, nothing shows, ZendMM is dumb.
+
+What you must remember is that ZendMM leak tracking is a nice bonus tool to have, but it does not replace a 
+:doc:`true C memory debugger <./memory_debugging>`.
+
+
+Buffer overflows or underflows
+------------------------------
+
+Zend Memory Manager engine
+**************************
+
+ZendMM substitutes to libc's API by providing a very similar one. That API should only be used when treating requests.
+
+ZendMM encapsulates libc's allocator, and like this later, it asks for memory, arange the memory areas, sticks header 
+and canary blocks against it, and gives you back the buffer you asked.