After "make install" the __pycache__ directories are only readable by root

[harri-pasanen]

Bug report

Bug description:

Building python on RHEL9 with

mkdir tmp; export TMPDIR=$(pwd)/tmp
tar xf /tmp/Python-3.11.10.tar.xz
cd Python-3.11.10/
./configure --enable-optimizations --with-lto
make -s -j
sudo make install

results in a usable python. However only root can read the __pycache__ files, example:

ls -l /usr/local/lib/python3.11/__pycache__

...
-rw-r-----. 1 root root 29675 Oct 28 18:26 zipimport.cpython-311.opt-1.pyc
-rw-r-----. 1 root root 25988 Oct 28 18:26 zipimport.cpython-311.opt-2.pyc
-rw-r-----. 1 root root 29793 Oct 28 18:26 zipimport.cpython-311.pyc

Note that the pycache folder itself is also only readable by root.

To fix this manually one can do

sudo find /usr/local -type d -name '__pycache__' -exec chmod -R a+rX {} \;

but I would have expected make install to do it.

CPython versions tested on:

3.11

Operating systems tested on:

Linux

[harri-pasanen]

Another observation, make install seems to add some site-packages, but those also are only readable by root:

$ ls -l  /usr/local/lib/python3.11/site-packages/
total 12
drwxr-x---. 3 root root   63 Oct 28 19:11 _distutils_hack
-rw-r-----. 1 root root  151 Oct 28 19:11 distutils-precedence.pth
drwxr-x---. 5 root root  136 Oct 28 19:11 pip
drwxr-x---. 2 root root  166 Oct 28 19:11 pip-24.0.dist-info
drwxr-x---. 5 root root   73 Oct 28 19:11 pkg_resources
-rw-r--r--. 1 root root  119 Oct 28 19:11 README.txt
drwxr-x---. 8 root root 4096 Oct 28 19:11 setuptools
drwxr-x---. 2 root root  143 Oct 28 19:11 setuptools-65.5.0.dist-info

[tengyangzh]

@tomasr8 Do you think this needs to be improved? If so, I think I can try to fix it.

[tomasr8]

I checked on an Ubuntu 22.04 with python 3.10 and __pycache__ has drwxr-xr-x while the files inside have -rw-r--r-- so they are readable. I am not sure if this is something make install should set.

cc @erlend-aasland as the build expert

[tengyangzh]

@erlend-aasland @harri-pasanen
After I started a RHEL9 container using Docker, I conducted an experiment. I used a regular account, but I executed sudo make install in the final step. The results showed that the file permissions in the paths /usr/local/lib/python3.11/site-packages/ and /usr/local/lib/python3.11/pycache are correct.

[zty@scc-a6k Python-3.11.10]$ sudo make installls -l  /usr/local/lib/python3.11/site-packages/
make: *** No rule to make target 'installls'.  Stop.
[zty@scc-a6k Python-3.11.10]$ ls -l  /usr/local/lib/python3.11/site-packages/
total 32
drwxr-xr-x 3 root root 4096 Nov  5 04:00 _distutils_hack
-rw-r--r-- 1 root root  151 Nov  5 04:00 distutils-precedence.pth
drwxr-xr-x 5 root root 4096 Nov  5 04:00 pip
drwxr-xr-x 2 root root 4096 Nov  5 04:00 pip-24.0.dist-info
drwxr-xr-x 5 root root 4096 Nov  5 04:00 pkg_resources
-rw-r--r-- 1 root root  119 Nov  5 04:00 README.txt
drwxr-xr-x 8 root root 4096 Nov  5 04:00 setuptools
drwxr-xr-x 2 root root 4096 Nov  5 04:00 setuptools-65.5.0.dist-info

[zty@scc-a6k Python-3.11.10]$ ls -l /usr/local/lib/python3.11/__pycache__
total 17192
-rw-r--r-- 1 root root   9044 Nov  5 04:00 abc.cpython-311.opt-1.pyc
-rw-r--r-- 1 root root   5844 Nov  5 04:00 abc.cpython-311.opt-2.pyc
-rw-r--r-- 1 root root   9044 Nov  5 04:00 abc.cpython-311.pyc
-rw-r--r-- 1 root root  45512 Nov  5 04:00 aifc.cpython-311.opt-1.pyc
-rw-r--r-- 1 root root  40305 Nov  5 04:00 aifc.cpython-311.opt-2.pyc
-rw-r--r-- 1 root root  45512 Nov  5 04:00 aifc.cpython-311.pyc
-rw-r--r-- 1 root root   4370 Nov  5 04:00 _aix_support.cpython-311.opt-1.pyc
-rw-r--r-- 1 root root   3037 Nov  5 04:00 _aix_support.cpython-311.opt-2.pyc
-rw-r--r-- 1 root root   4370 Nov  5 04:00 _aix_support.cpython-311.pyc
-rw-r--r-- 1 root root   1260 Nov  5 04:00 antigravity.cpython-311.opt-1.pyc
-rw-r--r-- 1 root root   1123 Nov  5 04:00 antigravity.cpython-311.opt-2.pyc
-rw-r--r-- 1 root root   1260 Nov  5 04:00 antigravity.cpython-311.pyc
-rw-r--r-- 1 root root 113695 Nov  5 04:00 argparse.cpython-311.opt-1.pyc
-rw-r--r-- 1 root root 103992 Nov  5 04:00 argparse.cpython-311.opt-2.pyc
-rw-r--r-- 1 root root 113986 Nov  5 04:00 argparse.cpython-311.pyc

[erlend-aasland]

The install modes can be found in Makefile.pre.in:

cpython/Makefile.pre.in

Lines 256 to 260 in 4a0d574

	# Modes for directories, executables and data files created by the
	# install process. Default to user-only-writable for all file types.
	DIRMODE= 755
	EXEMODE= 755
	FILEMODE= 644

This seems to be an issue with the environment, and not CPython itself. Moreover, Python 3.11 does not accept bug-fixes; please try a newer release (3.12 still accepts bug fixes, and 3.13 was recently released).

I'm inclined to close this as invalid.

[harri-pasanen]

The Docker experiment seems to have a fault:

@erlend-aasland @harri-pasanen After I started a RHEL9 container using Docker, I conducted an experiment...

[zty@scc-a6k Python-3.11.10]$ sudo make installls -l  /usr/local/lib/python3.11/site-packages/
make: *** No rule to make target 'installls'.  Stop.

Our VMs are not out of the box RH9, but I still wonder how the environment could cause this.
I'll check with python 3.13 build.

[harri-pasanen]

I tested with python 3.13.0 and I get exactly the same result. The problem seems to be my umask, which is 0027
This appears to carry over to install even if it is done with "sudo make install"
Makefile.pre.in modes are probably just used for the /usr/bin/install, but the "Compiling step" does not respect those.

How to reproduce the issue, building without optimization to make it faster.

umask 027
tar xf /tmp/Python-3.13.0.tar.xz
cd Python-3.13.0/
./configure
make -s -j
sudo make install
ls -l /usr/local/lib/python3.13/__pycache__/

will fail with:
ls: cannot open directory '/usr/local/lib/python3.13/pycache/': Permission denied
](ls: cannot open directory '/usr/local/lib/python3.13/pycache/': Permission denied)

[erlend-aasland]

Makefile.pre.in modes are probably just used for the /usr/bin/install, but the "Compiling step" does not respect those.

I'm don't understand what you mean by the "Compiling step", but the modes in the Makefile are not actually used for all cases. You are right that the bug is in our build system; I'll try to cook up a patch. Feel free to beat me to it, if you want to take a stab at it.

[erlend-aasland]

[...] but the "Compiling step" does not respect those.

Ah, you mean Lib/compileall.py?