Decrease distribution size

[stefan6419846]

At the moment, wheels are about 11 MB in size. Uncompressing/installing the data takes about 36 MB. This appears to be quite large and might be reduced:

• The serialized JSON data currently does not use the most compact representation where there is no whitespace after the comma and colon.
• The word lists have lots of duplicate entries (see Document code properly #21 as well).
• About half of the (uncompressed) resources in the wheel file is test data (18.5 MB), which is a lot and is not required by most of the users.

[goneall]

@stefan6419846 I noticed this repo hasn't had any updates in a while. Did you want to put in a pull request for your recent issues and do you have interested / bandwidth to help maintain?

[stefan6419846]

@goneall I stumbled upon this repository when skimming through the parsing/matching docs (as you might have seen in the specification repository as well) and looking for existing implementations. When evaluating the general code quality, I opened some of the immediate issues I got aware of.

For now, the overall impression is not the best, as there is not really any development besides the GSoC stuff. Resources are part of an external personal repository without any indication on how to generate them (https://github.com/m1kit/yalm-resources, #20), it uses a library without any self-tests (https://github.com/m1kit/terregex) and so on.

What is the purpose/aim of the SPDX team with this repository? Should it become the reference implementation for the matching algorithm in Python as the https://github.com/spdx/tools-python repository does for the SPDX documents itself?

That being said, I will see if I manage to get some PR done to start removing the temporary stuff from this repository which does not belong here.

[goneall]

Thanks @stefan6419846 for looking into this.

In terms of purpose, I only know of one implementation of the full SPDX license matching algorithm which is written in Java.

Having a Python implementation as a library would be very beneficial to the community. We're currently using a Python Java Bridge in the SPDX online tools which has proven to be rather problematic - I'd love to replace that with a native Python implementation.

One suggestion before we invest too much time in this library, we should test that it properly matches the text files in the SPDX license list as a test just to make sure there isn't any significant design issues. This is documented in issue #14