Skip to content

Commit f411932

Browse files
csegarragonzcsegarragonz
authored
Cross-compile ImageMagick (#97)
* chore(gh): add imagemagick as submodule * feat: add task to cross-compile imagemagick * chore(gh): bump code version * feat: imagemagick build working * feat: cross-compiled imagemagick runs as an executable in faasm (still does not run succesfully) * feat: cross-compile libpng and zlib for imagemagick with png delegate * feat(docker): build and install zlib libpng and imagemagick as part of the image build * feat: build new cross-compiled libraries in gha tests * chore(gh): bump code version to install latest libraries * fix: fix libpng task and remove clean flag from docker build. also update libpng third-party dep * fix: remove space in dockerfile * fix: point wasi libc to the right branch * fix(gha): remove unnecessary flags from tests * fix(ffmpeg): explicitely disable zlib in ffmpeg (autodetect feature made it fail now that we actually build a version of zlib) * fix(gha): don't check for changes in git diff in the third-party dir
1 parent 956f84c commit f411932

File tree

16 files changed

+279
-12
lines changed

16 files changed

+279
-12
lines changed

‎.env‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
SYSROOT_VERSION=0.1.7
2-
SYSROOT_CLI_IMAGE=faasm/cpp-sysroot:0.1.7
1+
SYSROOT_VERSION=0.1.8
2+
SYSROOT_CLI_IMAGE=faasm/cpp-sysroot:0.1.8
33
COMPOSE_PROJECT_NAME=cpp-dev

‎.github/workflows/tests.yml‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jobs:
1212
if: github.event.pull_request.draft == false
1313
runs-on: ubuntu-20.04
1414
container:
15-
image: faasm/cpp-sysroot:0.1.7
15+
image: faasm/cpp-sysroot:0.1.8
1616
defaults:
1717
run:
1818
working-directory: /code/cpp
@@ -28,6 +28,12 @@ jobs:
2828
run: git submodule update --init -f third-party/libffi
2929
- name: "Update FFmpeg submodule"
3030
run: git submodule update --init -f third-party/FFmpeg
31+
- name: "Update ImageMagick submodule"
32+
run: git submodule update --init -f third-party/ImageMagick
33+
- name: "Update libpng submodule"
34+
run: git submodule update --init -f third-party/libpng
35+
- name: "Update zlib submodule"
36+
run: git submodule update --init -f third-party/zlib
3137
- name: "Update wasi-libc submodule"
3238
run: git submodule update --init -f third-party/wasi-libc
3339
# --- Build libraries to wasm ---
@@ -47,6 +53,12 @@ jobs:
4753
run: ./bin/inv_wrapper.sh libemscripten
4854
- name: "Build FFmpeg"
4955
run: ./bin/inv_wrapper.sh ffmpeg
56+
- name: "Build zlib"
57+
run: ./bin/inv_wrapper.sh zlib
58+
- name: "Build libpng"
59+
run: ./bin/inv_wrapper.sh libpng
60+
- name: "Build ImageMagick"
61+
run: ./bin/inv_wrapper.sh imagemagick
5062
# --- Build functions to wasm ---
5163
- name: "Build the functions"
5264
run: ./bin/inv_wrapper.sh func.local --clean
@@ -64,4 +76,4 @@ jobs:
6476
- name: "Run C/C++ formatting"
6577
run: ./bin/run_clang_format.sh
6678
- name: "Check no formatting changes"
67-
run: git diff --exit-code
79+
run: git diff --exit-code -- :^third-party

‎.gitmodules‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,15 @@
2020
path = third-party/FFmpeg
2121
url = https://github.com/faasm/FFmpeg.git
2222
branch = faasm
23+
[submodule "third-party/ImageMagick"]
24+
path = third-party/ImageMagick
25+
url = https://github.com/faasm/ImageMagick.git
26+
branch = faasm
27+
[submodule "third-party/libpng"]
28+
path = third-party/libpng
29+
url = https://github.com/faasm/libpng.git
30+
branch = faasm
31+
[submodule "third-party/zlib"]
32+
path = third-party/zlib
33+
url = https://github.com/faasm/zlib.git
34+
branch = faasm

‎VERSION‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.1.7
1+
0.1.8

‎docker/cpp-sysroot.dockerfile‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,10 @@ RUN mkdir -p /code \
2323
&& git submodule update --init -f third-party/faasm-clapack \
2424
&& git submodule update --init -f third-party/libffi \
2525
&& git submodule update --init -f third-party/wasi-libc \
26-
&& git submodule update --init -f third-party/FFmpeg
26+
&& git submodule update --init -f third-party/FFmpeg \
27+
&& git submodule update --init -f third-party/zlib \
28+
&& git submodule update --init -f third-party/libpng \
29+
&& git submodule update --init -f third-party/ImageMagick
2730

2831
# Install the faasmtools Python lib
2932
RUN cd /code/cpp \
@@ -45,13 +48,18 @@ RUN cd /code/cpp \
4548
libfaasmpi --native --shared \
4649
# Install toolchain files
4750
&& inv install \
48-
# Build ported third-pary WASM libraries
51+
# Build ported third-pary WASM libraries (libc first as it is needed in the
52+
# others)
4953
&& inv \
54+
libc \
5055
clapack \
5156
clapack --clean --shared \
5257
ffmpeg \
53-
libc \
5458
libffi \
59+
# To build imagemagick, we need to build zlib and libpng
60+
zlib \
61+
libpng \
62+
imagemagick \
5563
# Build Faasm WASM libraries
5664
&& inv \
5765
libemscripten \

‎faasmtools/build.py‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,10 @@
3535

3636
# CFLAGS
3737
WASM_CFLAGS = [
38+
"-O3 -mno-atomics",
39+
"--sysroot={}".format(WASM_SYSROOT),
3840
"-m32",
39-
"-DCONFIG_32",
4041
"-DANSI",
41-
"-O3",
42-
"--sysroot={}".format(WASM_SYSROOT),
4342
"-D__faasm",
4443
]
4544

@@ -54,14 +53,29 @@
5453
]
5554
WASM_CXXFLAGS_SHARED = WASM_CFLAGS_SHARED
5655

56+
# ----------
5757
# LDFLAGS
58+
# ----------
59+
60+
# Flags for static libraries
5861
WASM_LDFLAGS = [
5962
"-static",
6063
"-Xlinker --no-gc-sections",
6164
"-Xlinker --stack-first",
6265
"-Xlinker --no-check-features",
6366
]
6467

68+
# Flags for executables
69+
WASM_EXE_LDFLAGS = [
70+
"-Xlinker --stack-first",
71+
"-Xlinker --export=__heap_base",
72+
"-Xlinker --export=__data_end",
73+
"-Xlinker --export=__wasm_call_ctors",
74+
"-Xlinker --max-memory=4294901760",
75+
"-Wl,-z,stack-size=4194304 -Wl",
76+
]
77+
78+
# Flags for shared libraries
6579
WASM_LDFLAGS_SHARED = [
6680
"-nostdlib",
6781
"-nostdlib++",

‎sysroot_extras/libc.imports‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ mkdtemp
143143
tmpfile
144144
memfd_create
145145
msync
146+
sendfile
146147
tempnam
147148

148149
# Non-WASI timing

‎tasks/__init__.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
from . import ffmpeg
66
from . import func
77
from . import git
8+
from . import imagemagick
89
from . import install
910
from . import libc
1011
from . import libemscripten
@@ -13,13 +14,16 @@
1314
from . import libfaasmpi
1415
from . import libfake
1516
from . import libffi
17+
from . import libpng
18+
from . import zlib
1619

1720
ns = Collection(
1821
clapack,
1922
docker,
2023
ffmpeg,
2124
func,
2225
git,
26+
imagemagick,
2327
install,
2428
libc,
2529
libemscripten,
@@ -28,4 +32,6 @@
2832
libfaasmpi,
2933
libfake,
3034
libffi,
35+
libpng,
36+
zlib,
3137
)

‎tasks/ffmpeg.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ def ffmpeg(ctx, clean=False):
4646
"--disable-stripping",
4747
"--disable-programs",
4848
"--disable-doc",
49+
"--disable-zlib",
4950
"--extra-cflags='{}'".format(" ".join(WASM_CFLAGS)),
5051
"--extra-cxxflags='{}'".format(" ".join(WASM_CFLAGS)),
5152
"--extra-ldflags='{}'".format(" ".join(WASM_LDFLAGS)),

‎tasks/imagemagick.py‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
from faasmtools.env import (
2+
THIRD_PARTY_DIR,
3+
)
4+
from faasmtools.build import (
5+
WASM_AR,
6+
WASM_CC,
7+
WASM_CFLAGS,
8+
WASM_CXX,
9+
WASM_EXE_LDFLAGS,
10+
WASM_HOST,
11+
WASM_LD,
12+
WASM_NM,
13+
WASM_RANLIB,
14+
WASM_SYSROOT,
15+
)
16+
from faasmtools.compile_util import wasm_copy_upload
17+
from invoke import task
18+
from os.path import join
19+
from subprocess import run
20+
21+
22+
@task(default=True)
23+
def imagemagick(ctx, clean=False):
24+
"""
25+
Compile and install ImageMagick
26+
"""
27+
imagemagick_dir = join(THIRD_PARTY_DIR, "ImageMagick")
28+
29+
if clean:
30+
run("make clean", shell=True, cwd=imagemagick_dir, check=True)
31+
32+
# List of flags inspired from the github project:
33+
# https://github.com/KnicKnic/WASM-ImageMagick
34+
# For all the configure options, see:
35+
# https://github.com/ImageMagick/ImageMagick/blob/main/Install-unix.txt
36+
configure_cmd = [
37+
"./configure",
38+
"CC={}".format(WASM_CC),
39+
"CXX={}".format(WASM_CXX),
40+
"CFLAGS='{}'".format(" ".join(WASM_CFLAGS)),
41+
"CXXFLAGS='{}'".format(" ".join(WASM_CFLAGS)),
42+
"LD={}".format(WASM_LD),
43+
"LDFLAGS='{}'".format(" ".join(WASM_EXE_LDFLAGS)),
44+
"CXXLDFLAGS='{}'".format(" ".join(WASM_EXE_LDFLAGS)),
45+
"AR={}".format(WASM_AR),
46+
"RANLIB={}".format(WASM_RANLIB),
47+
"NM={}".format(WASM_NM),
48+
"PKG_CONFIG_PATH={}".format(join(THIRD_PARTY_DIR, "libpng")),
49+
"--prefix={}".format(WASM_SYSROOT),
50+
"--disable-largefile",
51+
"--disable-openmp",
52+
"--disable-shared",
53+
"--host={}".format(WASM_HOST),
54+
"--with-png=yes",
55+
"--enable-delegate-build",
56+
"--without-bzlib",
57+
"--without-dps",
58+
"--without-djvu",
59+
"--without-fftw",
60+
"--without-flif",
61+
"--without-fpx",
62+
"--without-fontconfig",
63+
"--without-freetype",
64+
"--without-gslib",
65+
"--without-gvc",
66+
"--without-heic",
67+
"--without-jbig",
68+
"--without-lcms",
69+
"--without-lqr",
70+
"--without-magick-plus-plus",
71+
"--without-openexr",
72+
"--without-openjp2",
73+
"--without-pango",
74+
"--without-perl",
75+
"--without-raqm",
76+
"--without-raw",
77+
"--without-rsvg",
78+
"--without-threads",
79+
"--without-webp",
80+
"--without-wmf",
81+
"--without-x",
82+
"--without-xml",
83+
]
84+
85+
configure_cmd = " ".join(configure_cmd)
86+
run(configure_cmd, shell=True, cwd=imagemagick_dir, check=True)
87+
88+
run("make -j", shell=True, cwd=imagemagick_dir, check=True)
89+
90+
# Instead of installing ImageMagick, we copy the self-contained binary
91+
# (magick) to /usr/local/faasm/wasm/imagemagick/main/function.wasm
92+
wasm_copy_upload(
93+
"imagemagick", "main", join(imagemagick_dir, "utilities", "magick")
94+
)

0 commit comments

Comments
 (0)