Make APK builds reproducible more reliably

[thecockatiel]

related to #9210
opened since it needs its own issue

currently the .pyc files in private.tar embeds the full path to files, which makes the builds not reproducible when compiled on different paths/environments, which should not be happening

[SomberNight]

currently the .pyc files in private.tar embeds the full path to files

That might be fixed by patching python-for-android here:
https://github.com/SomberNight/python-for-android/blob/5df11be775eadcc19774e5bb118ee407330b8d83/pythonforandroid/recipes/python3/__init__.py#L351
https://github.com/SomberNight/python-for-android/blob/5df11be775eadcc19774e5bb118ee407330b8d83/pythonforandroid/bootstraps/common/build/build.py#L218

See https://docs.python.org/3/library/compileall.html#cmdoption-compileall-s
If we are lucky, we can just add the -s flag.

note: the -s flag was added in python 3.9. The latest release still uses python 3.8, but git master uses 3.10 for android.

---

Are the .pyc files the only difference? In the other issue you also mentioned something about the .so files.

[thecockatiel]

Finally I was able to reproduce the build by using the exact same paths as docker (/home/user/wspace/electrum) on the gitlab runner
and yes, the .so files are affected by the same reason

I'm glad I was able to pinpoint the issue after over 100+ builds :D

[thecockatiel]

I'm currently in the process of making the needed steps more compact for creating the f-droid MR, so that it gets accepted faster and easier to review (it takes a couple more builds to ensure things don't break)

[thecockatiel]

hmm, can you try patching it on your side to see if it strips the unnecessary paths ? (each build takes ~2h on gitlab and +4h on my pc)
the path is easy to find in main.pyc in the private.tar when opened in a text editor

[SomberNight]

hmm, can you try patching it on your side to see if it strips the unnecessary paths ?
the path is easy to find in main.pyc in the private.tar when opened in a text editor

I think I fixed the pyc issue with 2c6e768.

EDIT: as this patch is quite small, I'll try to upstream it in kivy/python-for-android#3066

---

Not sure about the .so files or other differences, I did not have time to look at those. The above change only affects the pyc-generation.

[thecockatiel]

if it fixes pyc, it should probably also fix so files I guess
do you want to make a release ? that way I can verify it against your build

[thecockatiel]

some stuff I tried and the build is still reproducible (comparing to Dockerfile):

• removed USE_SDK_WRAPPER and GRADLE_OPTS envs
• removed manual ANT installation
• removed manual SDK installation (sdk is present on f-droid server)
• skipped user altogether (user name is vagrant on f-droid build server image with id 1000), but still using the wspace path for build (as path proved to be the most important part of the build so far)
• skipped installing locales package and language envs (even in the docker build, it defaults to C.UTF-8 or C or sth like that if I remember right)
• combined all apt installs except for the one before replacing apt sources

[thecockatiel]

It seems like NDK path is also affecting the builds, specifically its affecting the .so files. I have no idea whats going on there.

[thecockatiel]

how does the build decide to include a file in private.tar or not ?
do we need to make some change to ignore the fastlane folder ?
I'm asking because I was ending up with my build log inside the tar file on my tests (when I was placing it in the root of project)

[SomberNight]

Is the fastlane folder getting included in the apk?

See the buildozer .spec file, e.g.:

electrum/contrib/android/buildozer_qml.spec

Lines 21 to 22 in 6d593e3

	# (list) List of directory to exclude (let empty to not exclude anything)
	source.exclude_dirs = bin, build, dist, contrib, env,

and contrib/android/blacklist.txt

[thecockatiel]

I think yes, but I haven't tested. I asked because of that exactly
(it should not be included, so let's add them to exclusions and blacklist?)