Skip to content

Commit eee52d4

Browse files
indygregindygreg
committed
Initial commit
0 parents  commit eee52d4

34 files changed

+3988
-0
lines changed

‎.hgignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
build/
2+
dist/
3+
venv/

‎LICENSE‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Copyright (c) 2018 to present, Gregory Szorc
2+
All rights reserved.
3+
4+
Redistribution and use in source and binary forms, with or without modification,
5+
are permitted provided that the following conditions are met:
6+
7+
1. Redistributions of source code must retain the above copyright notice, this
8+
list of conditions and the following disclaimer.
9+
10+
2. Redistributions in binary form must reproduce the above copyright notice,
11+
this list of conditions and the following disclaimer in the documentation
12+
and/or other materials provided with the distribution.
13+
14+
3. Neither the name of the copyright holder nor the names of its contributors
15+
may be used to endorse or promote products derived from this software without
16+
specific prior written permission.
17+
18+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
19+
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
20+
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
21+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
22+
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
23+
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
24+
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
25+
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
26+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
27+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎README.rst‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
========================
2+
Python Standalone Builds
3+
========================
4+
5+
This project contains code for building Python distributions that are
6+
self-contained and highly portable (the binaries can be executed
7+
on most target machines).
8+
9+
The intended audience of this project are people wanting to produce
10+
applications that embed Python in a larger executable. The artifacts
11+
that this project produces make it easier to build highly-portable
12+
applications containing Python.
13+
14+
Most consumers of this project can bypass the building of artifacts
15+
and consume the pre-built binaries produced from it.
16+
17+
Project Status
18+
==============
19+
20+
The project can be considered alpha quality. It is still in a heavy state
21+
of flux.
22+
23+
Currently, it produces a nearly full-featured CPython distribution for
24+
Linux that is fully statically linked with the exception of some very
25+
common system libraries.
26+
27+
Planned features include:
28+
29+
* Support for Windows
30+
* Support for macOS
31+
* Static/dynamic linking toggles for dependencies
32+
33+
Instructions
34+
============
35+
36+
To build a Python distribution for Linux x64::
37+
38+
$ ./build-linux.py
39+
40+
Requirements
41+
============
42+
43+
Linux
44+
-----
45+
46+
The host system must be 64-bit. A Python 3.5+ interpreter must be
47+
available. The execution environment must have access to a Docker
48+
daemon (all build operations are performed in Docker containers for
49+
isolation from the host system).
50+
51+
How It Works
52+
============
53+
54+
The first thing the ``build-*`` scripts do is bootstrap an environment
55+
for building Python. On Linux, a base Docker image based on a deterministic
56+
snapshot of Debian Wheezy is created. A modern binutils and GCC are built
57+
in this environment. That modern GCC is then used to build a modern Clang.
58+
Clang is then used to build all of Python's dependencies (openssl, ncurses,
59+
readline, sqlite, etc). Finally, Python itself is built.
60+
61+
Python is built in such a way that extensions are statically linked
62+
against their dependencies. e.g. instead of the ``sqlite3`` Python
63+
extension having a run-time dependency against ``libsqlite3.so``, the
64+
SQLite symbols are statically inlined into the Python extension object
65+
file.
66+
67+
From the built Python, we produce an archive containing the raw Python
68+
distribution (as if you had run ``make install``) as well as other files
69+
useful for downstream consumers.
70+
71+
Setup.local Hackery
72+
-------------------
73+
74+
Python's build system reads the ``Modules/Setup`` and ``Modules/Setup.local``
75+
files to influence how C extensions are built. By default, many extensions
76+
have no entry in these files and the ``setup.py`` script performs work
77+
to compile these extensions. (``setup.py`` looks for headers, libraries,
78+
etc, and sets up the proper compiler flags.)
79+
80+
``setup.py`` doesn't provide a lot of flexibility and relies on a lot
81+
of default behavior in ``distutils`` as well as other inline code in
82+
``setup.py``. This default behavior is often undesirable for our
83+
desired outcome of producing a standalone Python distribution.
84+
85+
Since the build environment is mostly deterministic and since we have
86+
special requirements, we generate a custom ``Setup.local`` file that
87+
builds C extensions in a specific manner. The undesirable behavior of
88+
``setup.py`` is bypassed and the Python C extensions are compiled just
89+
the way we want.
90+
91+
Linux Runtime Requirements
92+
==========================
93+
94+
The produced Linux binaries have minimal references to shared
95+
libraries and thus can be executed on most Linux systems.
96+
97+
The following shared libraries are referenced:
98+
99+
* linux-vdso.so.1
100+
* libpthread.so.0
101+
* libdl.so.2 (required by ctypes extension)
102+
* libutil.so.1
103+
* librt.so.1
104+
* libnsl.so.1 (required by nis extension)
105+
* libcrypt.so.1 (required by crypt extension)
106+
* libm.so.6
107+
* libc.so.6
108+
* ld-linux-x86-64.so.2
109+
110+
Licensing
111+
=========
112+
113+
Python and its various dependencies are governed by varied software use
114+
licenses. This impacts the rights and requirements of downstream consumers.
115+
116+
The ``python-licenses.rst`` file contained in this repository and produced
117+
artifacts summarizes the licenses of various components.
118+
119+
Most licenses are fairly permissive. Notable exceptions to this are GDBM and
120+
readline, which are both licensed under GPL Version 3.
121+
122+
**It is important to understand the licensing requirements when integrating
123+
the output of this project into derived works.**

‎build-linux.py‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/usr/bin/env python3
2+
# This Source Code Form is subject to the terms of the Mozilla Public
3+
# License, v. 2.0. If a copy of the MPL was not distributed with this
4+
# file, You can obtain one at https://mozilla.org/MPL/2.0/.
5+
6+
import datetime
7+
import os
8+
import pathlib
9+
import subprocess
10+
import sys
11+
import venv
12+
13+
14+
ROOT = pathlib.Path(os.path.abspath(__file__)).parent
15+
BUILD = ROOT / 'build'
16+
DIST = ROOT / 'dist'
17+
VENV = BUILD / 'venv'
18+
PIP = VENV / 'bin' / 'pip'
19+
PYTHON = VENV / 'bin' / 'python'
20+
REQUIREMENTS = ROOT / 'requirements.txt'
21+
MAKE_DIR = ROOT / 'cpython-linux'
22+
23+
24+
def bootstrap():
25+
BUILD.mkdir(exist_ok=True)
26+
DIST.mkdir(exist_ok=True)
27+
28+
venv.create(VENV, with_pip=True)
29+
30+
subprocess.run([str(PIP), 'install', '-r', str(REQUIREMENTS)],
31+
check=True)
32+
33+
os.environ['PYBUILD_BOOTSTRAPPED'] = '1'
34+
os.environ['PATH'] = '%s:%s' % (str(VENV / 'bin'), os.environ['PATH'])
35+
os.environ['PYTHONPATH'] = str(ROOT)
36+
subprocess.run([str(PYTHON), __file__], check=True)
37+
38+
39+
def run():
40+
import zstandard
41+
from pythonbuild.downloads import DOWNLOADS
42+
43+
now = datetime.datetime.utcnow()
44+
45+
subprocess.run(['make'],
46+
cwd=str(MAKE_DIR), check=True)
47+
48+
source_path = BUILD / 'cpython-linux64.tar'
49+
dest_path = DIST / ('cpython-%s-linux64-%s.tar.zst' % (
50+
DOWNLOADS['cpython-3.7']['version'], now.strftime('%Y%m%dT%H%M')))
51+
52+
print('compressing Python archive to %s' % dest_path)
53+
with source_path.open('rb') as ifh, dest_path.open('wb') as ofh:
54+
cctx = zstandard.ZstdCompressor(level=15)
55+
cctx.copy_stream(ifh, ofh, source_path.stat().st_size)
56+
57+
58+
if __name__ == '__main__':
59+
try:
60+
if 'PYBUILD_BOOTSTRAPPED' not in os.environ:
61+
bootstrap()
62+
else:
63+
run()
64+
except subprocess.CalledProcessError as e:
65+
sys.exit(e.returncode)

‎cpython-linux/Makefile‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
ROOT := $(abspath $(CURDIR)/..)
2+
HERE := $(ROOT)/cpython-linux
3+
OUTDIR := $(ROOT)/build
4+
5+
BUILD := $(HERE)/build.py
6+
NULL :=
7+
8+
COMMON_DEPENDS := \
9+
# $(BUILD) \
10+
$(NULL)
11+
12+
PLATFORM := linux64
13+
14+
TOOLCHAIN_DEPENDS := \
15+
$(OUTDIR)/binutils-linux64.tar \
16+
$(OUTDIR)/gcc-linux64.tar \
17+
$(OUTDIR)/clang-linux64.tar \
18+
$(NULL)
19+
20+
default: $(OUTDIR)/cpython-linux64.tar
21+
22+
$(OUTDIR)/image-%.tar: $(HERE)/%.Dockerfile $(COMMON_DEPENDS)
23+
$(BUILD) image-$*
24+
25+
$(OUTDIR)/binutils-linux64.tar: $(OUTDIR)/image-gcc.tar $(HERE)/build-binutils.sh
26+
$(BUILD) binutils
27+
28+
$(OUTDIR)/gcc-linux64.tar: $(OUTDIR)/binutils-linux64.tar $(HERE)/build-gcc.sh
29+
$(BUILD) gcc
30+
31+
$(OUTDIR)/clang-linux64.tar: $(OUTDIR)/binutils-linux64.tar $(OUTDIR)/gcc-linux64.tar $(OUTDIR)/image-clang.tar $(HERE)/build-clang.sh
32+
$(BUILD) clang
33+
34+
$(OUTDIR)/bzip2-%.tar: $(OUTDIR)/image-build.tar $(TOOLCHAIN_DEPENDS) $(HERE)/build-bzip2.sh
35+
$(BUILD) --platform $* bzip2
36+
37+
$(OUTDIR)/gdbm-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-gdbm.sh
38+
$(BUILD) --platform $* gdbm
39+
40+
$(OUTDIR)/libffi-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-libffi.sh
41+
$(BUILD) --platform $* libffi
42+
43+
$(OUTDIR)/ncurses-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-ncurses.sh
44+
$(BUILD) --platform $* ncurses
45+
46+
$(OUTDIR)/openssl-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-openssl.sh
47+
$(BUILD) --platform $* openssl
48+
49+
$(OUTDIR)/readline-%.tar: $(TOOLCHAIN_DEPENDS) $(OUTDIR)/ncurses-$(PLATFORM).tar $(HERE)/build-readline.sh
50+
$(BUILD) --platform $* readline
51+
52+
$(OUTDIR)/sqlite-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-sqlite.sh
53+
$(BUILD) --platform $(PLATFORM) sqlite
54+
55+
$(OUTDIR)/tcltk-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-tcltk.sh
56+
$(BUILD) --platform $(PLATFORM) tcltk
57+
58+
$(OUTDIR)/uuid-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-uuid.sh
59+
$(BUILD) --platform $(PLATFORM) uuid
60+
61+
$(OUTDIR)/xz-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-xz.sh
62+
$(BUILD) --platform $(PLATFORM) xz
63+
64+
$(OUTDIR)/zlib-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-zlib.sh
65+
$(BUILD) --platform $(PLATFORM) zlib
66+
67+
PYTHON_DEPENDS := \
68+
$(OUTDIR)/bzip2-$(PLATFORM).tar \
69+
$(OUTDIR)/gdbm-$(PLATFORM).tar \
70+
$(OUTDIR)/libffi-$(PLATFORM).tar \
71+
$(OUTDIR)/ncurses-$(PLATFORM).tar \
72+
$(OUTDIR)/openssl-$(PLATFORM).tar \
73+
$(OUTDIR)/readline-$(PLATFORM).tar \
74+
$(OUTDIR)/sqlite-$(PLATFORM).tar \
75+
$(OUTDIR)/uuid-$(PLATFORM).tar \
76+
$(OUTDIR)/xz-$(PLATFORM).tar \
77+
$(OUTDIR)/zlib-$(PLATFORM).tar \
78+
$(HERE)/static-modules \
79+
$(NULL)
80+
81+
$(OUTDIR)/cpython-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-cpython.sh $(PYTHON_DEPENDS)
82+
$(BUILD) --platform $(PLATFORM) cpython

‎cpython-linux/README.rst‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
This project builds CPython for Linux in a mostly deterministic and
2+
reproducible manner. The resulting Python build is mostly self-contained
3+
and the binaries are capable of running on many Linux distributions.
4+
5+
The produced binaries perform minimal loading of shared libraries.
6+
The required shared libraries are:
7+
8+
* linux-vdso.so.1
9+
* libpthread.so.0
10+
* libdl.so.2 (required by ctypes extension)
11+
* libutil.so.1
12+
* librt.so.1
13+
* libnsl.so.1 (required by nis extension)
14+
* libcrypt.so.1 (required by crypt extension)
15+
* libm.so.6
16+
* libc.so.6
17+
* ld-linux-x86-64.so.2
18+
19+
These shared libraries should be present on most modern Linux distros.

‎cpython-linux/base.Dockerfile‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Debian Wheezy.
2+
FROM debian@sha256:37103c15605251b2e35b70a3214af626a55cff39abbaadccd01ff828ee7005e0
3+
MAINTAINER Gregory Szorc <[email protected]>
4+
5+
RUN groupadd -g 1000 build && \
6+
useradd -u 1000 -g 1000 -d /build -s /bin/bash -m build && \
7+
mkdir /tools && \
8+
chown -R build:build /build /tools
9+
10+
ENV HOME=/build \
11+
SHELL=/bin/bash \
12+
USER=build \
13+
LOGNAME=build \
14+
HOSTNAME=builder \
15+
DEBIAN_FRONTEND=noninteractive
16+
17+
CMD ["/bin/bash", "--login"]
18+
WORKDIR '/build'
19+
20+
RUN for s in debian_wheezy debian_wheezy-updates debian_wheezy-backports debian-security_wheezy/updates; do \
21+
echo "deb http://snapshot.debian.org/archive/${s%_*}/20181129T234109Z/ ${s#*_} main"; \
22+
done > /etc/apt/sources.list && \
23+
( echo 'quiet "true";'; \
24+
echo 'APT::Get::Assume-Yes "true";'; \
25+
echo 'APT::Install-Recommends "false";'; \
26+
echo 'Acquire::Check-Valid-Until "false";'; \
27+
echo 'Acquire::Retries "5";'; \
28+
) > /etc/apt/apt.conf.d/99cpython-portable
29+
30+
RUN apt-get update

‎cpython-linux/build-binutils.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/usr/bin/env bash
2+
# This Source Code Form is subject to the terms of the Mozilla Public
3+
# License, v. 2.0. If a copy of the MPL was not distributed with this
4+
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
5+
6+
set -e
7+
8+
cd /build
9+
10+
tar -xf binutils-${BINUTILS_VERSION}.tar.xz
11+
mkdir binutils-objdir
12+
pushd binutils-objdir
13+
14+
../binutils-${BINUTILS_VERSION}/configure \
15+
--build=x86_64-unknown-linux-gnu \
16+
--prefix=/tools/host \
17+
--enable-plugins \
18+
--disable-nls \
19+
--with-sysroot=/
20+
21+
make -j `nproc`
22+
make install -j `nproc` DESTDIR=/build/out
23+
24+
popd

0 commit comments

Comments
 (0)