|
| 1 | +Troubleshooting |
| 2 | +=============== |
| 3 | +This is a collection of problems with ``pyfakefs`` and possible solutions. |
| 4 | +It will be expanded continuously based on issues and problems found by users. |
| 5 | + |
| 6 | +Modules not working with pyfakefs |
| 7 | +--------------------------------- |
| 8 | + |
| 9 | +Modules may not work with ``pyfakefs`` for several reasons. ``pyfakefs`` |
| 10 | +works by patching some file system related modules and functions, specifically: |
| 11 | + |
| 12 | +- most file system related functions in the ``os`` and ``os.path`` modules |
| 13 | +- the ``pathlib`` module |
| 14 | +- the build-in ``open`` function and ``io.open`` |
| 15 | +- ``shutil.disk_usage`` |
| 16 | + |
| 17 | +Other file system related modules work with ``pyfakefs``, because they use |
| 18 | +exclusively these patched functions, specifically ``shutil`` (except for |
| 19 | +``disk_usage``), ``tempfile``, ``glob`` and ``zipfile``. |
| 20 | + |
| 21 | +A module may not work with ``pyfakefs`` because of one of the following |
| 22 | +reasons: |
| 23 | + |
| 24 | +- It uses a file system related function of the mentioned modules that is |
| 25 | + not or not correctly patched. Mostly these are functions that are seldom |
| 26 | + used, but may be used in Python libraries (this has happened for example |
| 27 | + with a changed implementation of ``shutil`` in Python 3.7). Generally, |
| 28 | + these shall be handled in issues and we are happy to fix them. |
| 29 | +- It uses file system related functions in a way that will not be patched |
| 30 | + automatically. This is the case for functions that are executed while |
| 31 | + reading a module. This case and a possibility to make them work is |
| 32 | + documented above under ``modules_to_reload``. |
| 33 | +- It uses OS specific file system functions not contained in the Python |
| 34 | + libraries. These will not work out of the box, and we generally will not |
| 35 | + support them in ``pyfakefs``. If these functions are used in isolated |
| 36 | + functions or classes, they may be patched by using the ``modules_to_patch`` |
| 37 | + parameter (see the example for file locks in Django above), or by using |
| 38 | + ``unittest.patch`` if you don't need to simulate the functions. We |
| 39 | + added some of these patches to ``pyfakefs``, so that they are applied |
| 40 | + automatically (currently done for some ``pandas`` and ``Django`` |
| 41 | + functionality). |
| 42 | +- It uses C libraries to access the file system. There is no way no make |
| 43 | + such a module work with ``pyfakefs``--if you want to use it, you |
| 44 | + have to patch the whole module. In some cases, a library implemented in |
| 45 | + Python with a similar interface already exists. An example is ``lxml``, |
| 46 | + which can be substituted with ``ElementTree`` in most cases for testing. |
| 47 | + |
| 48 | +A list of Python modules that are known to not work correctly with |
| 49 | +``pyfakefs`` will be collected here: |
| 50 | + |
| 51 | +`multiprocessing`_ (build-in) |
| 52 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 53 | +This module has several issues (related to points 1 and 3 above). |
| 54 | +Currently there are no plans to fix this, but this may change in case of |
| 55 | +sufficient demand. |
| 56 | + |
| 57 | +`subprocess`_ (build-in) |
| 58 | +~~~~~~~~~~~~~~~~~~~~~~~~ |
| 59 | +This has very similar problems to ``multiprocessing`` and cannot be used with |
| 60 | +``pyfakefs`` to start a process. ``subprocess`` can either be mocked, if |
| 61 | +the process is not needed for the test, or patching can be paused to start |
| 62 | +a process if needed, and resumed afterwards |
| 63 | +(see `this issue <https://github.com/jmcgeheeiv/pyfakefs/issues/447>`__). |
| 64 | + |
| 65 | +Modules that rely on ``subprocess`` or ``multiprocessing`` |
| 66 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 67 | +This includes a number of modules that need to start other executables to |
| 68 | +function correctly. Examples that have shown this problem include `GitPython`_ |
| 69 | +and `plumbum`_. |
| 70 | + |
| 71 | +The `Pillow`_ image library |
| 72 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 73 | +This library partly works with ``pyfakefs``, but it is known to not work at |
| 74 | +least if writing JPEG files |
| 75 | +(see `this issue <https://github.com/jmcgeheeiv/pyfakefs/issues/529>`__) |
| 76 | + |
| 77 | +`pandas`_ - the Python data analysis library |
| 78 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 79 | +This uses its own internal file system access written in C, thus much of |
| 80 | +``pandas`` will not work with ``pyfakefs`` out of the box. Having said that, |
| 81 | +``pyfakefs`` patches ``pandas`` to use standard file-system access instead, |
| 82 | +so that many of the ``read_xxx`` functions, including ``read_csv`` and |
| 83 | +``read_excel``, as well as some writer functions, do work with the fake file |
| 84 | +system. If you use only these functions, ``pyfakefs`` should work fine with |
| 85 | +``pandas``. |
| 86 | + |
| 87 | +`xlrd`_ |
| 88 | +~~~~~~~ |
| 89 | +This libary is used by ``pandas`` to read Excel files in the `.xls` format, and |
| 90 | +can also be used stand-alone. Similar to ``pandas``, it is by default patched |
| 91 | +by ``pyfakefs`` to use normal file system functions that can be patched. |
| 92 | + |
| 93 | +`openpyxl`_ |
| 94 | +~~~~~~~~~~~ |
| 95 | +This is another library that reads and writes Excel files, and is also |
| 96 | +used by ``pandas`` if installed. ``openpyxl`` uses ``lxml`` for some file-system |
| 97 | +access if it is installed - in this case ``pyfakefs`` will not able to patch |
| 98 | +it correctly (``lxml`` uses C functions for file system access). It will `not` |
| 99 | +use ``lxml`` however, if the environment variable ``OPENPYXL_LXML` is set to |
| 100 | +"False" (or anything other than "True"), so if you set this variable `before` |
| 101 | +running the tests, it can work fine with ``pyfakefs``. |
| 102 | + |
| 103 | +Please write a new issue, if you are not sure if a module can be handled, or |
| 104 | +how to do it. |
| 105 | + |
| 106 | +Pyfakefs behaves differently than the real filesystem |
| 107 | +----------------------------------------------------- |
| 108 | +There are at least the following kinds of deviations from the actual behavior: |
| 109 | + |
| 110 | +- unwanted deviations that we didn't notice--if you find any of these, please |
| 111 | + write an issue and will try to fix it |
| 112 | +- behavior that depends on different OS versions and editions--as mentioned |
| 113 | + in :ref:`limitations`, ``pyfakefs`` uses the TravisCI systems as reference |
| 114 | + system and will not replicate all system-specific behavior |
| 115 | +- behavior that depends on low-level OS functionality that ``pyfakefs`` is not |
| 116 | + able to emulate; examples are the ``fcntl.ioctl`` and ``fcntl.fcntl`` |
| 117 | + functions that are patched to do nothing |
| 118 | + |
| 119 | +The test code tries to access files in the real filesystem |
| 120 | +---------------------------------------------------------- |
| 121 | +The loading of the actual Python code from the real filesystem does not use |
| 122 | +the filesystem functions that ``pyfakefs`` patches, but in some cases it may |
| 123 | +access other files in the packages. An example is loading timezone information |
| 124 | +from configuration files. In these cases, you have to map the respective files |
| 125 | +or directories from the real into the fake filesystem as described in |
| 126 | +:ref:`real_fs_access`. |
| 127 | + |
| 128 | + |
| 129 | +OS temporary directories |
| 130 | +------------------------ |
| 131 | +Tests relying on a completely empty file system on test start will fail. |
| 132 | +As ``pyfakefs`` does not fake the ``tempfile`` module (as described above), |
| 133 | +a temporary directory is required to ensure ``tempfile`` works correctly, |
| 134 | +e.g., that ``tempfile.gettempdir()`` will return a valid value. This |
| 135 | +means that any newly created fake file system will always have either a |
| 136 | +directory named ``/tmp`` when running on Linux or Unix systems, |
| 137 | +``/var/folders/<hash>/T`` when running on MacOs, or |
| 138 | +``C:\Users\<user>\AppData\Local\Temp`` on Windows. |
| 139 | + |
| 140 | +User rights |
| 141 | +----------- |
| 142 | +If you run ``pyfakefs`` tests as root (this happens by default if run in a |
| 143 | +docker container), ``pyfakefs`` also behaves as a root user, for example can |
| 144 | +write to write-protected files. This may not be the expected behavior, and |
| 145 | +can be changed. |
| 146 | +``Pyfakefs`` has a rudimentary concept of user rights, which differentiates |
| 147 | +between root user (with the user id 0) and any other user. By default, |
| 148 | +``pyfakefs`` assumes the user id of the current user, but you can change |
| 149 | +that using ``fake_filesystem.set_uid()`` in your setup. This allows to run |
| 150 | +tests as non-root user in a root user environment and vice verse. |
| 151 | +Another possibility to run tests as non-root user in a root user environment |
| 152 | +is the convenience argument :ref:`allow_root_user`. |
| 153 | + |
| 154 | +.. _usage_with_mock_open: |
| 155 | + |
| 156 | +Pyfakefs and mock_open |
| 157 | +---------------------- |
| 158 | +If you patch ``open`` using ``mock_open`` before the initialization of |
| 159 | +``pyfakefs``, it will not work properly, because the ``pyfakefs`` |
| 160 | +initialization relies on ``open`` working correctly. |
| 161 | +Generally, you should not need ``mock_open`` if using ``pyfakefs``, because you |
| 162 | +always can create the files with the needed content using ``create_file``. |
| 163 | +This is true for patching any filesystem functions - avoid patching them |
| 164 | +while working with ``pyfakefs``. |
| 165 | +If you still want to use ``mock_open``, make sure it is only used while |
| 166 | +patching is in progress. For example, if you are using ``pytest`` with the |
| 167 | +``mocker`` fixture used to patch ``open``, make sure that the ``fs`` fixture is |
| 168 | +passed before the ``mocker`` fixture to ensure this. |
| 169 | + |
| 170 | +.. _`multiprocessing`: https://docs.python.org/3/library/multiprocessing.html |
| 171 | +.. _`subprocess`: https://docs.python.org/3/library/subprocess.html |
| 172 | +.. _`GitPython`: https://pypi.org/project/GitPython/ |
| 173 | +.. _`plumbum`: https://pypi.org/project/plumbum/ |
| 174 | +.. _`Pillow`: https://pypi.org/project/Pillow/ |
| 175 | +.. _`pandas`: https://pypi.org/project/pandas/ |
| 176 | +.. _`xlrd`: https://pypi.org/project/xlrd/ |
| 177 | +.. _`openpyxl`: https://pypi.org/project/openpyxl/ |
0 commit comments