Update Android libraries and scripts, support Android 15 and 16

Library changes
---------------

For Android (and Emscripten), replace libcrypto and libssl provided by OpenSSL with those provided by BoringSSL, as the former does not support using Android's system certificate store. This previously required embedding a certificate store separately in the APK, limiting the lifetime of the app until the certificate expires. With BoringSSL we can set `CURLOPT_CAPATH` for curl handles to use the Android system certificate store directly.

Upgrade/downgrade libraries to fixed versions used on other platforms for parity, or latest release versions when more desirable or necessary for Android:

- libpng: unspecified --> v1.6.43
- curl: curl-8_8_0 (unchanged)
- freetype: unspecified --> VER-2-13-2
- SDL: unspecified --> release-2.32.10
- ogg: unspecified --> v1.3.5
- opus: unspecified --> v1.5.2
- opusfile: unspecified --> v0.12
- sqlite3: amalgamation-3360000 --> amalgamation-3460000

Remove unused precompiled zlib library, which is not necessary for Android because the bundled `src/engine/external/zlib` can be compiled. It is only necessary for the Emscripten build because `src/engine/external/zlib` does not compile, but Emscripten libraries are not currently included in the ddnet-libs repository. Change the version of zlib used for the Emscripten build to v1.3.1.2 to fix build failures.

Remove unused `libSDL2main.a` library, which is not necessary for Android.

Support for Android 15 and 16
-----------------------------

Bump versions of Android NDK, build tools, command line tools and Gradle to support [16 KiB pages sizes](https://developer.android.com/guide/practices/page-sizes), which is required to support Android 15 (API 35).

Compile libraries with the correct flags for Android API 24 - 36 support. Fix various Android-specific CFLAGS not being set correctly when building some libraries. See the Android CMake toolchain files and https://android.googlesource.com/platform/ndk/+/master/docs/BuildSystemMaintainers.md for documentation of the flags being used. The only deviation from the default flags is that we build the libraries and the client without debug information (in Release mode), as the automatic stripping of debug information during the Gradle build does not work. This reduces the final APK size in Release mode by around 1/3 or 60 MiB.

Bump target/compile API level to 36 to support Android 16. Set `android:enableOnBackInvokedCallback="false"` in the manifest to disable the [predictive back gesture](https://developer.android.com/guide/navigation/custom-back/predictive-back-gesture) and keep the legacy behavior of the back button.

Fix Gradle deprecation warnings with latest Gradle version, by using `=` for property assignment in `build.gradle`.

Towards reproducible builds
---------------------------

Pin specific release version or commits for all libraries instead of building the default branch.

Make the library build reproducible by removing the embedded absolute build paths of the libraries using the `-ffile-prefix-map` flag. Also separately remove the absolute paths of the Android NDK and the Emscripten SDK which were being embedded in libraries and in the Android and Emscripten client, respectively.

Remove embedded paths of the build folder in addition to the source folder, as some compilers (Android) also seem to embed it separately for generated source files.

Order entries in Android asset integrity index file lexicographically instead of using non-deterministic file enumeration order.

Explicitly set `ANDROID_HOME` in the CI, as it was already defined and preinstalled Android tools were used instead.

Use fixed Rust default version for Android build in CI.

Overall, this should make the Android build almost reproducible if the specified versions of Android NDK and the JDK are used. Still not reproducible is the Vulkan shader compilation/optimization and something minor about the client binary compilation.

Streamline library building process
-----------------------------------

- Add log messages.
- Add error message when building for Android if the Android NDK could not be found. Avoid different warning when building for Emscripten and not having the Android NDK installed.
- Specify the correct libpng include and library paths when building libfreetype to enable PNG support in the latter.
- Disable protocol support in libcurl for all protocols except those enabled on other platforms: HTTPS, HTTP, FTPS, FTP, FILE.
- Use ninja for building CMake-based libraries to reduce building time.
- Specify exact build targets when building CMake-based libraries to avoid building unused extra tools.
- Avoid building zlib for Android, as it can be build from `src/engine/external/zlib` and only needs to be build separately for Emscripten.
- Avoid rebuilding the sqlite3 and opusfile libraries if they are unchanged by generating minimal Makefiles.
- Avoid copying build script files to the build folders of individual libraries, instead invoke the scripts within the build folders with absolute paths.
- Avoid copying libopus and libogg files when building libopusfile, instead reference the files in the existing build folders.

Improve documentation
---------------------

- Add missing instruction to install libtool to the README. Without libtool the initial project configuration of libopusfile fails, although it seems to work on the second attempt anyway.
- Add instructions for updating the Android SDK and Gradle versions to the README.
- Add instructions to delete old Android/Emscripten-specific libraries files before copying new files to ddnet-libs, to avoid old files being left behind.
- Explain how to perform a clean build of the libraries.

Various refactoring
-------------------

- Add `set -e` to all scripts to enable strict error handling and always exit on errors. Remove `|| exit 1` that are made unnecessary by this.
- Add shellcheck source directives to comments so that shellcheck can correctly check the sourced shell scripts, instead of disabling the check. The paths in the source directives are relative to project root, the folder where shellcheck is executed.
- Consolidate common flags and functions for Android and Emscripten building in the `_build_common.sh` file.
- Consistently use `ON` for CMake settings instead of `1` or `TRUE`.
- Use local variables when possible.

Author: Robyt3

.github/workflows/build-libraries-android.yml

@@ -0,0 +1,49 @@
+name: Build Android libraries
+
+# TODO: remove all except workflow_dispatch
+on:
+  push:
+    branches-ignore:
+      - gh-readonly-queue/**
+      - master
+  pull_request:
+  merge_group:
+  workflow_dispatch:
+
+jobs:
+  build-libraries-android:
+    runs-on: ubuntu-latest
+    env:
+      CARGO_HTTP_MULTIPLEXING: false
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+      with:
+        submodules: recursive
+
+    - name: Prepare Linux
+      run: |
+        sudo apt-get update -y
+        sudo apt-get install cmake glslang-tools libtool ninja-build p7zip-full
+        scripts/android/download_android_sdk.sh /home/runner
+        yes | latest/bin/sdkmanager --licenses
+
+    - name: Build Android libraries
+      run: |
+        scripts/compile_libs/gen_libs.sh build-android-libs android
+
+    - name: Check reproduciblity
+      run: |
+        find build-android-libs/ddnet-libs -type f -print0 | while IFS= read -r -d $'\0' new_file; do
+          rel_path=${new_file#build-android-libs/ddnet-libs/}
+          existing_file="ddnet-libs/$rel_path"
+          if [ -f "$existing_file" ]; then
+            diff -q "$new_file" "$existing_file" || true
+          fi
+        done
+
+    - name: Upload Artifacts
+      uses: actions/upload-artifact@v4
+      with:
+        name: ddnet-libs-android
+        path: build-android-libs/ddnet-libs

.github/workflows/build.yml

@@ -256,35 +256,14 @@ jobs:
     - name: Prepare Linux
       run: |
         sudo apt-get update -y
-        sudo apt-get install cmake ninja-build openjdk-21-jdk p7zip-full curl glslang-tools openssl
+        sudo apt-get install cmake ninja-build openjdk-21-jdk p7zip-full glslang-tools openssl
+        rustup default 1.92.0
         cargo install cargo-ndk
         rustup target add armv7-linux-androideabi
         rustup target add i686-linux-android
         rustup target add aarch64-linux-android
         rustup target add x86_64-linux-android
-        mkdir ~/Android
-        cd ~/Android
-        mkdir Sdk
-        cd Sdk
-        mkdir ndk
-        cd ndk
-        wget --quiet https://dl.google.com/android/repository/android-ndk-r26d-linux.zip
-        unzip android-ndk-r26d-linux.zip
-        rm android-ndk-r26d-linux.zip
-        cd ~/Android/Sdk
-        mkdir build-tools
-        cd build-tools
-        wget --quiet https://dl.google.com/android/repository/build-tools_r30.0.3-linux.zip
-        unzip build-tools_r30.0.3-linux.zip
-        rm build-tools_r30.0.3-linux.zip
-        mv android-11 30.0.3
-        cd ~/Android/Sdk
-        mkdir cmdline-tools
-        cd cmdline-tools
-        wget --quiet https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
-        unzip commandlinetools-linux-11076708_latest.zip
-        rm commandlinetools-linux-11076708_latest.zip
-        mv cmdline-tools latest
+        scripts/android/download_android_sdk.sh /home/runner
         yes | latest/bin/sdkmanager --licenses
 
     - name: Cache Rust dependencies
@@ -302,10 +281,16 @@ jobs:
       run: |
         export TW_KEY_PW="$(openssl rand -base64 32)"
         keytool -genkey -v -keystore "$TW_KEY_NAME" -keyalg RSA -keysize 2048 -validity 10000 -alias "$TW_KEY_ALIAS" -storepass "$TW_KEY_PW" -dname "CN=DDNet CI, OU=DDNet, O=DDNet"
-        mkdir build-android
+        export ANDROID_HOME="/home/runner/Android/Sdk"
+        unset ANDROID_SDK_ROOT
         scripts/android/cmake_android.sh all DDNet org.ddnet.client Release build-android
         mkdir artifacts
         mv build-android/DDNet.apk artifacts
+        # TODO: remove testing
+        mkdir artifacts/testing
+        cp build-android/build_arch/arm64/checksummed_extra.txt artifacts/testing/checksummed_extra.txt
+        cp build-android/build_arch/arm64/checksummed_files.txt artifacts/testing/checksummed_files.txt
+        cp -R build-android/build_arch/arm64/src artifacts/testing
 
     - name: Upload Artifacts
       uses: actions/upload-artifact@v4

CMakeLists.txt

@@ -307,6 +307,7 @@ if(NOT MSVC AND NOT HAIKU)
 
   if(NOT CMAKE_BUILD_TYPE STREQUAL Debug AND NOT DEV)
     add_cxx_compiler_flag_if_supported(OUR_FLAGS -ffile-prefix-map=${CMAKE_SOURCE_DIR}=/${CMAKE_PROJECT_NAME})
+    add_cxx_compiler_flag_if_supported(OUR_FLAGS -ffile-prefix-map=${PROJECT_BINARY_DIR}=/${CMAKE_PROJECT_NAME})
   endif()
 
   if(TARGET_OS STREQUAL "mac")
@@ -689,8 +690,8 @@ endif()
 if(CLIENT AND NOT(SDL2_FOUND))
   message(SEND_ERROR "You must install SDL2 to compile the ${CMAKE_PROJECT_NAME} client")
 endif()
-if(TARGET_OS STREQUAL "android" AND NOT(CRYPTO_FOUND))
-  message(SEND_ERROR "You must install OpenSSL to compile ${CMAKE_PROJECT_NAME}")
+if((TARGET_OS STREQUAL "android" OR TARGET_OS STREQUAL "emscripten") AND NOT(CRYPTO_FOUND))
+  message(SEND_ERROR "You must install BoringSSL to compile ${CMAKE_PROJECT_NAME} for Android/Emscripten")
 endif()
 if(NOT(GTEST_FOUND))
   if(DOWNLOAD_GTEST)

README.md

@@ -272,22 +272,30 @@ Install Emscripten cross-compilers on a modern linux distro. Follow the [instruc
 If you need to compile the ddnet-libs for WebAssembly, simply call
 
 ```sh
-# <directory to build in> should be a directory outside of the project's source directory
-scripts/compile_libs/gen_libs.sh "<directory to build in>" webasm
+scripts/compile_libs/gen_libs.sh build-webasm-libs webasm
 ```
 
 from the project's source directory. It will automatically create a directory called `ddnet-libs` in your build directory.
-You can then manually merge this directory with the one in the ddnet source directory.
+You can then merge this directory with the one in the ddnet source directory:
+
+```sh
+find ddnet-libs -type d -name 'webasm' -exec rm -r {} + -prune
+cp -r build-webasm-libs/ddnet-libs/. ddnet-libs/
+```
 
 Run `rustup target add wasm32-unknown-emscripten` to install the WASM target for compiling Rust.
 
-Then run `emcmake cmake .. -G "Unix Makefiles" -DVIDEORECORDER=OFF -DVULKAN=OFF -DSERVER=OFF -DTOOLS=OFF -DPREFER_BUNDLED_LIBS=ON` in your build directory to configure followed by `cmake --build . -j8` to build. For testing it is highly recommended to build in debug mode by also passing the argument `-DCMAKE_BUILD_TYPE=Debug` when invoking `emcmake cmake`, as this speeds up the build process and adds debug information as well as additional checks. Note that using the Ninja build system with Emscripten is not currently possible due to [CMake issue 16395](https://gitlab.kitware.com/cmake/cmake/-/issues/16395).
+Create a new directory to build the client in.
+Then run `emcmake cmake .. -G "Unix Makefiles" -DVIDEORECORDER=OFF -DVULKAN=OFF -DSERVER=OFF -DTOOLS=OFF -DPREFER_BUNDLED_LIBS=ON` in your build directory to configure followed by `cmake --build . -j8` to build.
+For testing it is highly recommended to build in debug mode by also passing the argument `-DCMAKE_BUILD_TYPE=Debug` when invoking `emcmake cmake`, as this speeds up the build process and adds debug information as well as additional checks.
+Note that using the Ninja build system with Emscripten is not currently possible due to [CMake issue 16395](https://gitlab.kitware.com/cmake/cmake/-/issues/16395).
 
-To test the compiled code locally, just use `emrun --browser firefox DDNet.html`
+To test the compiled code locally, just use `emrun --browser firefox DDNet.html`.
 
-To host the compiled .html file copy all `.data`, `.html`, `.js`, `.wasm` files to the web server. See `other/emscripten/minimal.html` for a minimal HTML example. You can also run `other/emscripten/server.py` to host a minimal server for testing using Python without needing to install Emscripten.
+To host the compiled .html file copy all `.data`, `.html`, `.js`, `.wasm` files to the web server.
+See `other/emscripten/minimal.html` for a minimal HTML example. You can also run `other/emscripten/server.py` to host a minimal server for testing using Python without needing to install Emscripten.
 
-Then enable cross origin policies. Example for apache2 on debian based distros:
+Enable cross-origin policies when using a proper web server. Example for apache2 on Debian-based distros:
 
 ```sh
 sudo a2enmod header

cmake/FindCrypto.cmake

@@ -8,37 +8,30 @@ if(NOT PREFER_BUNDLED_LIBS)
   endif()
 endif()
 
-if(PREFER_BUNDLED_LIBS AND (TARGET_OS STREQUAL "android" OR CMAKE_SYSTEM_NAME STREQUAL "Emscripten"))
-  set_extra_dirs_lib(CRYPTO openssl)
-  find_library(CRYPTO_LIBRARY1
+if(PREFER_BUNDLED_LIBS AND (TARGET_OS STREQUAL "android" OR TARGET_OS STREQUAL "emscripten"))
+  set_extra_dirs_lib(CRYPTO boringssl)
+  find_library(CRYPTO_LIBRARY
     NAMES crypto
     HINTS ${HINTS_CRYPTO_LIBDIR} ${PC_CRYPTO_LIBDIR} ${PC_CRYPTO_LIBRARY_DIRS}
     PATHS ${PATHS_CRYPTO_LIBDIR}
     ${CROSSCOMPILING_NO_CMAKE_SYSTEM_PATH}
   )
-  find_library(CRYPTO_LIBRARY2
+  find_library(SSL_LIBRARY
     NAMES ssl
     HINTS ${HINTS_CRYPTO_LIBDIR} ${PC_CRYPTO_LIBDIR} ${PC_CRYPTO_LIBRARY_DIRS}
     PATHS ${PATHS_CRYPTO_LIBDIR}
     ${CROSSCOMPILING_NO_CMAKE_SYSTEM_PATH}
   )
-  set(CRYPTO_LIBRARY ${CRYPTO_LIBRARY1} ${CRYPTO_LIBRARY2})
+  set(CRYPTO_LIBRARY ${CRYPTO_LIBRARY} ${SSL_LIBRARY})
 
   set(CMAKE_FIND_FRAMEWORK FIRST)
-  set_extra_dirs_include(CRYPTO openssl "${CRYPTO_LIBRARY}")
-  find_path(CRYPTO_INCLUDEDIR1 openssl/opensslconf.h
+  set_extra_dirs_include(CRYPTO boringssl "${CRYPTO_LIBRARY}")
+  find_path(CRYPTO_INCLUDEDIR openssl/ssl.h openssl/base.h openssl/hkdf.h openssl/opensslconf.h
     PATH_SUFFIXES CRYPTO
     HINTS ${HINTS_CRYPTO_INCLUDEDIR} ${PC_CRYPTO_INCLUDEDIR} ${PC_CRYPTO_INCLUDE_DIRS}
     PATHS ${PATHS_CRYPTO_INCLUDEDIR}
     ${CROSSCOMPILING_NO_CMAKE_SYSTEM_PATH}
   )
-  find_path(CRYPTO_INCLUDEDIR2 openssl/configuration.h
-    PATH_SUFFIXES CRYPTO
-    HINTS ${HINTS_CRYPTO_INCLUDEDIR} ${PC_CRYPTO_INCLUDEDIR} ${PC_CRYPTO_INCLUDE_DIRS}
-    PATHS ${PATHS_CRYPTO_INCLUDEDIR}
-    ${CROSSCOMPILING_NO_CMAKE_SYSTEM_PATH}
-  )
-  set(CRYPTO_INCLUDEDIR ${CRYPTO_INCLUDEDIR1} ${CRYPTO_INCLUDEDIR2})
 endif()
 
 include(FindPackageHandleStandardArgs)
@@ -47,6 +40,7 @@ find_package_handle_standard_args(Crypto DEFAULT_MSG CRYPTO_LIBRARY CRYPTO_INCLU
 mark_as_advanced(CRYPTO_LIBRARY CRYPTO_INCLUDEDIR)
 
 if(CRYPTO_FOUND)
+  is_bundled(CRYPTO_BUNDLED "${CRYPTO_LIBRARY}")
   set(CRYPTO_LIBRARIES ${CRYPTO_LIBRARY})
   set(CRYPTO_INCLUDE_DIRS ${CRYPTO_INCLUDEDIR})
 endif()

cmake/FindZLIB.cmake

@@ -8,7 +8,7 @@ if(NOT PREFER_BUNDLED_LIBS)
   endif()
 endif()
 
-if(CMAKE_SYSTEM_NAME STREQUAL "Emscripten")
+if(TARGET_OS STREQUAL "emscripten")
   set_extra_dirs_lib(ZLIB zlib)
   find_library(ZLIB_LIBRARY
     NAMES z

cmake/toolchains/Emscripten.toolchain

@@ -1,3 +1,7 @@
+if(NOT DEFINED ENV{EMSDK})
+  message(FATAL_ERROR "EMSDK environment variable must specify path of Emscripten SDK")
+endif()
+
 set(WASM_CXX_FLAGS "")
 set(WASM_LINKER_FLAGS "")
 
@@ -54,6 +58,8 @@ if(CMAKE_BUILD_TYPE STREQUAL "Debug")
 else()
 	# Will drastically reduce code size but also increases compile and link time significantly.
 	set(WASM_CXX_FLAGS "${WASM_CXX_FLAGS} -O3")
+	# Remove absolute paths of Emscripten SDK from binary.
+	set(WASM_CXX_FLAGS "${WASM_CXX_FLAGS} -ffile-prefix-map=$ENV{EMSDK}=EMSDK")
 	set(WASM_LINKER_FLAGS "${WASM_LINKER_FLAGS} -flto")
 endif()