Module source files not found when cross-compiling

[bill9889]

BPO	22699
Nosy	@doko42, @ned-deily, @xdegaye, @zooba, @moreati, @wscullin, @isuruf, @iliis
PRs	bpo-22699: Allow compiling on debian/ubuntu with a different compiler #28397

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-10-22.15:49:06.114>
labels = ['build', '3.8', '3.9']
title = 'Module source files not found when cross-compiling'
updated_at = <Date 2021-09-16.20:52:16.007>
user = 'https://bugs.python.org/bill9889'

bugs.python.org fields:

activity = <Date 2021-09-16.20:52:16.007>
actor = 'isuruf'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build', 'Cross-Build']
creation = <Date 2014-10-22.15:49:06.114>
creator = 'bill9889'
dependencies = []
files = []
hgrepos = []
issue_num = 22699
keywords = ['patch']
message_count = 16.0
messages = ['229828', '248441', '350532', '364574', '364575', '364577', '364635', '364658', '364662', '364683', '364685', '364686', '364687', '364693', '364706', '364715']
nosy_count = 9.0
nosy_names = ['doko', 'ned.deily', 'xdegaye', 'steve.dower', 'Alex.Willmer', 'bill9889', 'wscullin', 'isuruf', 'iliis']
pr_nums = ['28397']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue22699'
versions = ['Python 3.8', 'Python 3.9']

[bill9889]

Who knows to cross-compile Python 3.4?

[wscullin]

I thought this was originally a help request and was going to re-direct this. Cross-compile with 3.4.3 and later seems broken.

Procedure followed:

wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc1.tgz
tar -xf Python-3.5.0rc1.tgz
mkdir buildpowerpc64-linux-gnu
cd buildpowerpc64-linux-gnu
../Python-3.5.0rc1/configure \
--disable-shared \
--prefix=/local/soft/python/3.5.0rc1/powerpc64-linux-gnu/gcc-4.4.7
make
make install

# now for the actual cross-compile build

cd ..
mkdir buildpowerpc64-bgq-linux
export PYTHON_FOR_BUILD=/local/soft/python/3.5.0rc1/powerpc64-linux-gnu/gcc-4.4.7/bin/python3.5
../Python-3.5.0rc1/configure \
--host=powerpc64-bgq-linux \
--build=powerpc64-linux-gnu \
--disable-ipv6 \
--disable-shared \
ac_cv_pthread_system_supported=yes \
ac_cv_file__dev_ptmx=no \
ac_cv_file__dev_ptc=no \
ac_cv_big_endian_double=yes
make

which succeeds in building a cross-compiled interpreter, then fails to build modules as setup.py gets the srcdir wrong:

[wscullin@vestalac1 buildpowerpc64-bgq-linux]$ make
running build
running build_ext
building '_struct' extension
powerpc64-bgq-linux-gcc -fPIC -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I../Python-3.5.0rc1/Include -I/local/soft/python/3.5.0rc1/powerpc64-linux-gnu
/gcc-4.4.7/include -I. -IInclude -I/usr/local/include -I/local/soft/python/3.5.0rc1/powerpc64-linux-gnu/gcc-4.4.7/include/python3.5m -c _struct.c -o build/temp.linux-ppc64-3.5/_struct.o
powerpc64-bgq-linux-gcc: _struct.c: No such file or directory
powerpc64-bgq-linux-gcc: no input files

[iliis]

Has there been any update on this? I've run into this issue trying to cross-compile python for Android. I've tried 3.7.4, 3.8 and current master, both in-source and out of source builds, all ending with the modules failing to compile due to wrong paths.

...
CC='/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang' LDSHARED='/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang -shared -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libatomic.a -static-libstdc++ -Wl,--build-id -Wl,--warn-shared-textrel -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -Wl,-z,noexecstack -Wl,--gc-sections -lc++ -lm -lgcc -ldl -lc -lgcc -ldl -latomic -lm -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libatomic.a -static-libstdc++ -Wl,--build-id -Wl,--warn-shared-textrel -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -Wl,-z,noexecstack -Wl,--gc-sections -lc++ -lm -lgcc -ldl -lc -lgcc -ldl -latomic -lm ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' _PYTHON_PROJECT_BASE=/home/samuel/android/cpython _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/samuel/android/cpython/build/lib.linux-aarch64-3.7:./Lib _PYTHON_SYSCONFIGDATA_NAME=sysconfigdata_m_linux python3 ./setup.py build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_struct' extension
/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall --target=aarch64-none-linux-android26 --gcc-toolchain=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -fno-addrsig -Wa,--noexecstack -Wformat -Werror=format-security -O3 -fPIE --target=aarch64-none-linux-android26 --gcc-toolchain=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -fno-addrsig -Wa,--noexecstack -Wformat -Werror=format-security -O3 -fPIE -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include -I. -I/home/samuel/android/cpython_build/host/venv/include -I/home/samuel/android/cpython_build/host/install/include/python3.7m -c _struct.c -o build/temp.linux-aarch64-3.7/_struct.o
clang: error: no such file or directory: '_struct.c'
clang: error: no input files
...
Failed to build these modules:
_asyncio _bisect _blake2
_bz2 _codecs_cn _codecs_hk
_codecs_iso2022 _codecs_jp _codecs_kr
_codecs_tw _contextvars _crypt
_csv _ctypes _ctypes_test
_datetime _decimal _heapq
_json _lsprof _md5
_multibytecodec _multiprocessing _opcode
_pickle _posixsubprocess _queue
_random _sha1 _sha256
_sha3 _sha512 _socket
_struct _testbuffer _testcapi
_testimportmultiple _testmultiphase _uuid
_xxtestfuzz array audioop
binascii cmath fcntl
grp math mmap
ossaudiodev parser pyexpat
resource select spwd
syslog termios unicodedata
xxlimited
...

[zooba]

I'm now hitting this too, on nearly all the compiled modules in the stdlib. One example:

building '_codecs_jp' extension
x86_64-my-linux-gcc -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse --sysroot=/opt/my-generic/0.0.1-dev/sysroots/core2-64-my-linux -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I../Include/internal -I. -IObjects -IInclude -IPython -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/usr/include/python3.8 -c cjkcodecs/_codecs_jp.c -o build/temp.linux-x86_64-3.8/cjkcodecs/_codecs_jp.o
x86_64-my-linux-gcc: error: cjkcodecs/_codecs_jp.c: No such file or directory
x86_64-my-linux-gcc: fatal error: no input files
compilation terminated.

I think the working directory is incorrect at some stage - note that the first two -I would make sense from Modules, but the next three do not.

[zooba]

Trimmed a bit from setup.py here, but I think this is the cause:

    def build_extensions(self):
        self.srcdir = sysconfig.get_config_var('srcdir')
        self.srcdir = os.path.abspath(self.srcdir)

    [SNIP]

        # Fix up the autodetected modules, prefixing all the source files
        # with Modules/.
        moddirlist = [os.path.join(self.srcdir, 'Modules')]

In my build, I've set PYTHON_FOR_BUILD=python3.8, and the contents of moddirlist is '/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/Modules'. So it's picking the wrong source directory.

I replaced the second line of the function above with:
self.srcdir = os.path.dirname(os.path.abspath(file))

Which made things a little better (I can launch my build now), but all the -I/usr/* options need to be replaced as well or there are no dynamically loaded modules.

[zooba]

Perhaps sysconfig needs an overhaul to be able to handle _PYTHON_PROJECT_BASE properly? I see so many values in there that are taken from the running interpreter (sys.prefix, for example) that ought to come from the cross-compilation target. Wouldn't surprise me if the host's CFLAGS are leaking through and adding those extra includes.

[zooba]

So ultimately this is a major sysconfig design flaw, and it likely requires a rewrite.

The mix of information sources between _sysconfigdata (even when overriding the name and PYTHONPATH) and the Makefile, and the order in which you can rely them to be loaded (you can't) and the fact that INSTALL_SCHEMES depends on _neither_ (it only uses the running interpreter's sys module) means you can't possibly get a coherent view of the settings for anything other than the running interpreter.

Since we need something else for cross-compiling, we should either fix sysconfig so that it's possible to override the source of information reliably, or we avoid using it completely for building the CPython extension modules. The former would likely help third-party cross-compilation more.

I'm not sure who needs to be in on this, so I'll post to python-dev before going any further.