Move uad_lists to a separate repository

[Artem13327]

Describe the feature you want

This list is used not only by this application and I think it would be good if it was developed in a separate repository

Acknowledgements

• This issue is not a duplicate of an existing feature request.
• I have chosen an appropriate title.
• All requested information has been provided properly.

[Rudxain]

I totally agree! This would also make PRs and Issues more organized.

The file is embedded into the UAD-ng binary at compile-time:

universal-android-debloater-next-generation/src/core/uad_lists.rs

Line 219 in 821587d

const DATA: &str = include_str!("../../resources/assets/uad_lists.json");

So we have 2 (or more?) options:

1. Always try to download an updated list at runtime, as UAD-ng won't include it.
   • This relies on the network, so the user should open UAD-ng for the 1st time, as soon as the program is downloaded (to ensure internet is available)
2. Use a macro that downloads the file (from the other repo) at compile-time, and embed it.
   • This seems complicated, but it's technically possible (Rust macros are "black magic" ✨️)

See also #608

[Rudxain]

Let's forget the 2nd option, it's too complicated, and increases memory usage (2 similar copies of the JSON in RAM).

Anyways, if we move the list to another repo, we should take the opportunity to split it (#608)

[Artem13327]

1. Always try to download an updated list.

I think this would be a good option, because if you have the Internet to download the program, you won't have the Internet to download something else? Of course, it's possible that you could have been given the program on some kind of media and you don't have the Internet, but in that case the caching function is enough

[AnonymousWP]

We can do this with 2.0.0, as it's a breaking change. Let's talk about the details once we get close to that (hopefully ever).

[Artem13327]

Let's talk about the details once we get close to that (hopefully ever).

Well, first of all, I would like to invite @samolego (Creator of https://github.com/samolego/Canta, the application that uses our list) to participate in the discussion.

[samolego]

Hi 👋, thanks!

Canta currently uses the 1st approach and I'd say it works just fine. If user launches app without internet, the local downloaded version is used (so yeah, for the first launch you need internet access).

Regarding the split into separate repository; I think it would be beneficial for this project and the community as a whole.

[samolego]

Also, you could probably use git subdirs to not hassle with macros for the embeding at compile time.

[Artem13327]

I had the idea how to smooth out this process after separation, you can leave this file for a short time and here so that everyone can update
For example, it will be useful to Canta users from F-Droid where the update will not appear immediately

[samolego]

Ah yeah the submodules. Basically you embed another repo in this one and you'd be able to copy it via path, maybe even reuse current code with just dir names changed? I'm not sure how it's embedded in the program currently.

[Artem13327]

submodules

Well, the peculiarity of submodules is that they are tied to a commit, and is there anyone willing to constantly monitor that there is a tie to the newest commit?

[crash5]

I don't know the rust ecosystem but in the CI pipeline you can simply download the latest uad json from the uad repo and overwrite the one came with the source code. Nothing really has to change anywhere else. (I would avoid any git submodul magic for a simple thing like this).

For the separation, at first I think it will be enough if users can load the uad json file from the filesystem and use it. Not at as a mandatory step, just an option instead using the "embedded" one. Afer the program can use the uad json from "anywhere" for the next step can be an update button which simply check some version file from the new repo to see if there is a new version available and show an option to download it. (maybe can be an option to where to look for the uad json file in the local filesytem, and use it if it is readable instead of the embedded one). So it will slowly transfer to the external uad json file usage.

[crash5]

on second thought, maybe better to store the md5/sha2 hash of the file, i guess the version doesn't really matter and in this case only need a simple == string comparison :)

[Rudxain]

I'll be honest, I understood ~5% of what you said:

download the latest uad json from the uad repo and overwrite the one came with the source code.

Do you mean having an empty "dummy" file in this repo, and the real file with content in the other repo?

The "dummy" file ensures the compiler doesn't complain.

For the separation

Of what? The repos, or the "list split" I mentioned on #608 ?

if users can load the uad json file from the filesystem

That can already be done (cache directory), unless you mean allowing arbitrary paths.

use the uad json from "anywhere"
[...]
where to look for the uad json file in the local filesytem

I guess you do mean arbitrary file paths 🤔

show an option to download it

Why not auto-update? I guess it's because updating arbitrary paths requires a config field (to remember it), while the cache directory is "always" (technically not) at the same path.

So it will slowly transfer to the external uad json file usage.

Did you mean "migration"? I guess you're anticipating the breakage of old UADNG versions, as those won't be coded to download from the new URL. We could have a transition period, where we keep an old copy of the list here, while we only maintain the one at the new.

The POC seems good enough! But I'm unsure if we should do it that way

[Rudxain]

Hash comparison seems like a good idea too, as every update will have a higher timestamp, so the timestamp is irrelevant. However, I'm concerned about hash collisions.

But, IIRC, the GH API provides a "last modified" timestamp, so we don't need to generate it

[crash5]

I don't know how the app works so I had a lot of a guesses. I thought the app embeds the json file into itself on compile time and needed a new release every time you wanted to update the json file.

On "split" I meant to extract the json into a separate repository. And if the app needs a json file to compile, it can be a dummy json. But the latest json file can be downloaded in the CI step from the json's repository before compiling the app. So the app will contain a proper json file in the compiled "release" version.

So, I thought only this is what needed to separate the app and the json repositories.

All the other things I wrote is about getting a new json file from the internet without the need to release a new app for it. But I guess it is working like this already, it just needs a json to compile?

If the old apps downloading the json from the current repo then they will break unfortunately. But I think it is unavoidable if you can't send a message to the users to update the app or it won't work after X days. In worst case scenario they download the new version because the old one stopped working. Of course the transition period is needed, but if someone hasn't updated the app in years it won't really help :)

hash collisions: the chances are pretty low for it, and in my opinion it doesn't really matter in this case. In the worst case scenario what happens? They skip one update for the json file? Getting the same has for the next X version is almost impossible. (and we will check the local version's hash to the latest remote version's hash, if they are different then it is a new version -> download)

[Rudxain]

So the app will contain a proper json file in the compiled "release" version.

Good idea!

getting a new json file from the internet without the need to release a new app for it. But I guess it is working like this already, it just needs a json to compile?

Yes, exactly. But it's possible to rewrite UAD so that it doesn't need it.

If we do rewrite, then (as I said earlier) it would "rely more" on the internet. But that isn't a serious problem, as users can simply download the List immediately after (or before, manually) they download UAD-ng.

send a message to the users to update the app

The app shows an "Update" button if the version is outdated! But users can ignore it, and it won't come with a "breakage warning" :(

But if the program doesn't panic, then this isn't a problem, as users can continue using old versions, but they would have to manually update the cache (assuming we don't change the format).

they download the new version because the old one stopped working.

Good point

someone hasn't updated the app in years

lol

In the worst case scenario what happens? They skip one update for the json file? Getting the same hash for the next X version is almost impossible

True! 2 consecutive hashes being identical is absurdly unlikely, much more improbable than any 2 hashes being identical

[AnonymousWP]

I would've preferred this thread to be a Discussion rather than an Issue. But I guess it's fine as-is

I see where you're coming from. I think a Discussion would be better too, as you can't just PR files by moving them to another repo. This only involves 1 repo (a delete). I'm converting it to a Discussion.

[Rudxain]

What if we create the repo now? Even if we won't use it for months (years?), I think it's good to have a "base structure" ready, to ease the migration. Having the repo now would also allow us to continue discussing there instead of here

[Rudxain]

I'll copy-paste my response from the guild, for non-discord-users:

well, yes, but we're still not sure about the directory structure and the file format.

Personally (not speaking for the team) I'm afraid of not getting it right the first time, because, while splitting the lists is "just an implementation detail", it's actually a major change that requires changing both the UAD-NG app and notifying samolego (main Canta dev), and possibly a "grace period" for other consumers of the list to adapt to the new format. After that happens, another major change would be undesirable

[Muriilow]

I mean I can guarantee that my change is working, and putting the new list in another repo would be safe because the current UADng doesnt use it anyway.

We could also hold the old version of the json file to not create problems to the current users of uad, and after some months of the change we would warn the users here in github and delete it.

Like I said to @AnonymousWP its not a big change, in the rust source code I changed like 8 lines, and I didnt overwrite any description or object, just removed the unecessary fields.

[Rudxain]

In that case, we could move it soon!

My intentions were that after moving the file, we would split it into multiple files. Simply moving the file is a tiny step forwards, but I guess that's a better idea, because it'd allow separating app issues/PRs from data issues/PRs.

My progressive side would be happy, but my conservative side is still afraid this might be a mistake. Like, have we considered all the benefits, costs, risks, etc...? This is why I feel we need more info before moving forward. I'm not a fan of "bureaucracy", but this might be serious enough to deserve it.

OTOH, projects like uBO have separate repos for data. If it works for them, then it might work fine for us!

[samolego]

Yeah, I agree too. I too think it makes managing issues and PRs simpler.

Canta will adapt, I'll be really grateful if you @ ping me when doing the transition :).

[AnonymousWP]

I'd like to make all those rigorous changes with 2.0.0, because then we can also fix some of the other list-related issues: https://github.com/Universal-Debloater-Alliance/universal-android-debloater-next-generation/milestone/7

[shuvashish76]

The UAD lists is messy and unorganized. Maintenance of such lists is the biggest concern. I like the idea of having separate lists according to their category as a complete separate project. I'd like to propose having some strict standard/criteria for the lists, similar to @MuntashirAkon 's ADL project.

• This documentation is useful for users who are interested to investigate their device packages. We definitely need separate documentation for how to contribute to the debloat list.
• Contributors Guide :

This guide is meant for users who want to contribute to the codebase of Universal Android Debloater Next Generation, whether that is the application code or the JSON-file for adding packages.

Currently there is no strict criteria for contributing to the debloat list. Users have to figure out themself about the JSON Schema from the list itself.

@Rudxain @AnonymousWP What's your opinion on this?

[Rudxain]

I agree. That's one of the reasons I opened this

[AnonymousWP]

I'll look into this again once we arrive at the milestone that will take care of all of this.