Remove some non-determinism

[licaon-kter]

Your app is not built reproducible in F-Droid, but we continuously test older versions on https://verification.f-droid.org and would like more and more apps to become repro.

Looking at this report: https://verification.f-droid.org/io.github.divverent.aaaaxy_106300004.apk.diffoscope.html#lib-arm---v-a-libgojni.so---readelf---wide---notes---

based on https://f-droid.org/docs/Reproducible_Builds/#golang maybe you can something to https://github.com/divVerent/aaaaxy/blob/v1.6.0/Makefile or https://github.com/divVerent/aaaaxy/blob/v1.6.0/AndroidStudioProjects/AAAAXY/app/build.gradle#L48 or https://github.com/divVerent/aaaaxy/blob/v1.6.0/AndroidStudioProjects/AAAAXY/app/build.gradle#L78 or etc

[divVerent]

First thing I see is that the two versions are built with different versions of Go:

@333333*@go1.23.6	70899	@333333*@go1.23.7

I doubt I can make that go away.

Trying to follow the recommendations though - how can I get an updated report after my changes?

[licaon-kter]

We can run multiple times locally and compare.

Can we find the exact go version in code so we can match yours?

[divVerent]

[rpolzer@brlogenshfegle ~]$ go version
go version go1.23.6 linux/amd64

Which just means I should go and update, so doing that now.

[rpolzer@brlogenshfegle ~]$ go version
go version go1.23.7 linux/amd64

I generally stick with the same minor version of Go until something requires me to go beyond, as they usually lose some platform support.

As for SDKs:

export ANDROID_HOME=$HOME/Android/Sdk

This is the following packages:

  Path                   | Version      | Description                             | Location              
  -------                | -------      | -------                                 | -------               
  build-tools;30.0.2     | 30.0.2       | Android SDK Build-Tools 30.0.2          | build-tools/30.0.2    
  build-tools;30.0.3     | 30.0.3       | Android SDK Build-Tools 30.0.3          | build-tools/30.0.3    
  build-tools;32.0.0     | 32.0.0       | Android SDK Build-Tools 32              | build-tools/32.0.0    
  build-tools;32.1.0-rc1 | 32.1.0 rc1   | Android SDK Build-Tools 32.1-rc1        | build-tools/32.1.0-rc1
  cmdline-tools;latest   | 6.0          | Android SDK Command-line Tools (latest) | cmdline-tools/latest  
  emulator               | 31.2.10      | Android Emulator                        | emulator              
  ndk;21.4.7075529       | 21.4.7075529 | NDK (Side by side) 21.4.7075529         | ndk/21.4.7075529      
  ndk;25.0.8775105       | 25.0.8775105 | NDK (Side by side) 25.0.8775105         | ndk/25.0.8775105      
  patcher;v4             | 1            | SDK Patch Applier v4                    | patcher/v4            
  platform-tools         | 33.0.1       | Android SDK Platform-Tools              | platform-tools        
  platforms;android-21   | 2            | Android SDK Platform 21                 | platforms/android-21  
  platforms;android-31   | 1            | Android SDK Platform 31                 | platforms/android-31  
  platforms;android-32   | 1            | Android SDK Platform 32                 | platforms/android-32  
  platforms;android-33   | 2            | Android SDK Platform 33                 | platforms/android-33  
  platforms;android-34   | 3            | Android SDK Platform 34                 | platforms/android-34  
  sources;android-21     | 1            | Sources for Android 21                  | sources/android-21    
  sources;android-33     | 1            | Sources for Android 33                  | sources/android-33    

I have compileSdk 34 in the gradle file, so I assume that's what it's using; no idea which of the NDK version it's using.

I am open to changing those versions - which does F-Droid prefer?

Other than that, I don't see quite the point of reproducible builds - as I am too new and was forced into Play App Signing, I can't match the signing key that the Play Store distributes anyway, so there definitely can't be any cross-updating between the two systems (otherwise I'd actually love to distribute the Google Play APK on F-Droid, so each can freely update the other).

But even without that, I guess having some verification is nice to have, yes, and generating deterministic binaries also reduces update size.

[divVerent]

For now testing that at least the Linux build has no randomness left:

$ for i in $(seq 1 5); do git clean -xdf && mak BUILDTYPE=release && cp aaaaxy /tmp/aaaaxy.$i; done

They have the same size but differ.

-00ef7800: 0800 8da2 735a b4b4 e286 e10e 0000 5e2c  ....sZ........^,
+00ef7800: 0800 0ba3 735a b4b4 e286 e10e 0000 5e2c  ....sZ........^,

Lots of similar differences where 8da2 becomes 0ba3.

[divVerent]

The other thing is - this seems fragile. Don't verifiable builds break all the time when I update a build tool version but F-Droid does not, or vice versa?

At this point I wonder if it wouldn't be better to use F-Droid's own build system for my releases on github as well. Is there an easy way from a shell script to run a build under F-Droid's environment, even without building from fdroiddata (as then I first have to actually create the version stanza etc. first)?

[divVerent]

Hm... I suspect this might be the embedded zip file with the assets.

This does store file dates, which git sets to current time when extracting.

Right now I am creating the zip file with Info-ZIP, which appears to have no way to not store file dates and times; is there a better tool for this? I can't find options to not do that in 7za's manpage either.

[divVerent]

I found https://salsa.debian.org/reproducible-builds/strip-nondeterminism - which might fix the zip issue.

Can this tool be used in the F-Droid build environment, and if so, how?

Edit: with this tool, I got identical Linux binaries. I guess that's a good start.

[divVerent]

I couldn't apply -buildvcs=false as ebitenmobile doesn't support it, but I do now have two APKs that are identical for all except the signing block (and if I extract the zip files, the contents are identical). That's a win, I guess. So I guess on next release we can try again?

[divVerent]

The .aab files (the ones I upload to Google Play) also are 100% identical now on rebuilds.

[licaon-kter]

Thanks, sounds good

I generally stick with the same minor version of Go until something requires me to go beyond, as they usually lose some platform support.

we don't have access to your local machine 😄 I meant grep it from
somewhere, eg. https://github.com/divVerent/aaaaxy/blob/main/.github/workflows/go.yml#L21 (which one of the files has the right version?)

no idea which of the NDK version it's using.

is this kept up to date https://github.com/divVerent/aaaaxy/blob/main/.github/workflows/go-buildonly.yml#L51 ?

But even without that, I guess having some verification is nice to have

this, yes

[divVerent]

Let's first see if the changes are sufficient.

As for the Go version - IMHO the proper place to look is the go.mod file, which lists the Go version in the line toolchain.

Sadly if a newer Go is installed, it'll use that instead, and that's what happened in this week's release - I am not going to make another release just for that, but in the future I can try keeping that line updated so you can use it.

Please do not use the github workflows files - they are rather the range of Go versions I want to test at.