The temporary file is not accessible and is not being deleted

[jrthornham]

I am using the jitcdde module in a fairly large system of ODE's and getting the following errors when running my code: "PermissionError: [WinError 5] Access is denied: ". I ran the jitcdde.test() and got the same error. I installed jitcdde using pip install jitcdde. I have run the MCVE in an administrator cmd and powershell. I have also tried in an anaconda environment. I can still run integrations, plot, and save the results but it leaves the temporary file behind and closes with the same WinError 5. Here is the MCVE:

import jitcdde
jitcdde.test()

The output is:

cl : Command line warning D9002 : ignoring unknown option '-fopenmp'
jitced.c
LINK : warning LNK4044: unrecognized option '/fopenmp'; ignored
   Creating library C:\Users\me\AppData\Local\Temp\jitcxde_3xeacy8s\Release\Users\me\AppData\Local\Temp\jitcxde_3xeacy8s\jitced.cp312-win_amd64.lib and object C:\Users\me\AppData\Local\Temp\jitcxde_3xeacy8s\Release\Users\me\AppData\Local\Temp\jitcxde_3xeacy8s\jitced.cp312-win_amd64.exp
Generating code
Finished generating code
Traceback (most recent call last):
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\shutil.py", line 634, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\me\\AppData\\Local\\Temp\\jitcxde_3xeacy8s\\jitced.cp312-win_amd64.pyd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\weakref.py", line 666, in _exitfunc
    f()
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\tempfile.py", line 935, in _cleanup
    cls._rmtree(name, ignore_errors=ignore_errors)
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\tempfile.py", line 930, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\shutil.py", line 808, in rmtree
    return _rmtree_unsafe(path, onexc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\shutil.py", line 636, in _rmtree_unsafe
    onexc(os.unlink, fullname, err)
  File "C:\Users\me\AppData\Local\Programs\Python\Python312\Lib\tempfile.py", line 905, in onexc
    _os.unlink(path)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\me\\AppData\\Local\\Temp\\jitcxde_3xeacy8s\\jitced.cp312-win_amd64.pyd'
C:\Users\me\AppData\Roaming\Python\Python312\site-packages\jitcxde_common\_jitcxde.py:366: UserWarning: Could not delete temporary directory because of the following error:
[WinError 5] Access is denied: 'C:\\Users\\me\\AppData\\Local\\Temp\\jitcxde_3xeacy8s\\jitced.cp312-win_amd64.pyd'
  warn(f"Could not delete temporary directory because of the following error:\n{error}")

The python version is 3.12.1
The jitcdde version is 1.8.1

I have also used jitcdde before on another computer and had no problems at all. I guess I have likely installed something wrong or am not running as an administrator somehow but I have no idea how to fix the issue since the exact same code is working fine on another laptop where the only obvious difference is that it is running python 3.10.4.

Thank you for any help!

[Wrzlprmft]

Hmm, that’s going to be tricky.

Am I understanding correctly that this only affects clean-up and the actual integrations and so on work fine?

[jrthornham]

That's correct. The program runs, allows me to save and plot data, then exits with the error right at the end.

[mwappner]

Not a solution, but a workaround in the meantime might be changing the location of the tempfiles? I have to do this when I run a bunch of systems in parallel because of some issues with the server where I run them:

# import warnings
import os
import tempfile

# TEMP DIR
tempdir = tempfile.TemporaryDirectory(dir=Path(__file__).parent, prefix='jitcdde_tmp_')
tempfile.tempdir = tempdir.name
os.environ['TMPDIR'] = tempdir.name

[jrthornham]

I wouldn't mind this work around at all but I gave it a go and still getting the same errors as before. I switched out Path(__file__).parent with C:\\Users\\me\\Documents\\Research. No luck though.

[Wrzlprmft]

Sorry that this took so long.

I have not tried to reproduce this problem on account of the hassle of preparing a suitable Windows machine. However, I looked into the respective code, which suggests the following:

• The problem is caused by Python trying to delete a temporary directory and Windows won’t let it, probably because files in the directory are still opened by Python itself (because the JiTCDDE compiles a Python module in that temporary directory and then loads it).
• This entire situation is nothing new and JiTC*DE’s destructor even catches exceptions during the cleanup of the temporary directory (i.e., within the temporary directory’s destructor) and forwards them as warnings – which should have addressed your issue. However, due to recent improvements to Python’s garbage collector (I guess), the temporary directory’s destructor is called separately before JiTC*DE’s destructor is called – where no safeguards are in place.

I tried to address this by using tempfile.TemporaryDirectory(…, ignore_cleanup_errors=True) on Windows. Please try to use the most recent version of JiTC*DE common (directly from Github) to see whether this fixes your problem. Mind that this may cause temporary directories to pile up, which can be a problem of its own.

If it doesn’t, one thing to try is to call JiTCDDE’s destructor explicitly at the end of your program or when you don’t need the instance anymore and wrap that in a try–except clause, i.e.:

try:
    del DDE # alternatively: DDE.__del__()
except PermissionError:
    pass