framework: refactor common.mk to split base definitions

[mreid-tt]

Description

Refactors spksrc.common.mk into a structured set of focused include files to improve maintainability, clarity, and include-order safety. Refactors the WORK_DIR/version_ge dependency solution from #6901 by enhancing @hgy59's suggested approach by further dividing the mk file into a sub-directory tree.

Problem:

spksrc.directories.mk uses version_ge (line 67) but version_ge was defined in spksrc.common.mk. When directories.mk was included before common.mk, the macro wasn't available, causing silent failures in version conditionals.

Solution:

This change decomposes spksrc.common.mk into clearly scoped components under mk/spksrc.common/, while preserving existing behavior:

mk/spksrc.common.mk
└── mk/spksrc.common/
    ├── archs.mk   : architecture and toolchain classification
    ├── logs.mk    : build log paths and logging helpers
    └── macros.mk  : generic GNU Make helper macros

spksrc.common.mk now acts as the single aggregation entry point and is responsible for loading these components in a consistent and well-defined order.

Version comparison macros (version_le, version_ge, version_lt, version_gt) are now clearly defined in macros.mk

• Generic helper macros (uniq, dedup, merge) are isolated in macros.mk
• NAS architecture and toolchain classification logic is contained in archs.mk
• Logging-related variables and helpers are contained in logs.mk
• The include structure is explicitly documented in file headers
• Behavior remains unchanged; this is a structural refactor

IMPORTANT: This PR addresses a long-standing issue where previously spksrc.archs.mk was being used pre-spksrc.cross-cc|cmake|meson|etc.mk and was being slowly migrated over to directly calling spksrc.common.mk. This PR migrate all cross & spk to using spksrc.common.mk only.

Testing:

• cross/fossil-scm (x64-7.1) ✓
• spk/minio (x64-7.1) ✓
• spk/borgbackup (x64-7.1) ✓
• toolchain/syno-x64-7.1 ✓
• spk/python312 (x64-7.1, aarch64-7.1)
• spk/python312-wheels (aarch64-7.1)
• spk/synocli-videodriver (x64-7.1)
• spk/ffmpeg (x64-7.1)
• spk/tvheadend (x64-7.1)

Closes #6901

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Bug fix
• Includes small framework changes

[th0ma7]

I wonder if we shouldn't make a larger change and migrate to spksrc.arch.mk (see possible caveat below):

# Filter to exclude TC versions greater than DEFAULT_TC (from local configuration)
TCVERSION_DUPES = $(addprefix %,$(filter-out $(DEFAULT_TC),$(AVAILABLE_TCVERSIONS)))

# remove unsupported (outdated) archs
ARCHS_DUPES_DEPRECATED += $(addsuffix %,$(DEPRECATED_ARCHS))

# Filter for all-supported
ARCHS_DUPES = $(ARCHS_WITH_GENERIC_SUPPORT) $(ARCHS_DUPES_DEPRECATED) $(TCVERSION_DUPES)

# supported: used for all-supported target
SUPPORTED_ARCHS = $(sort $(filter-out $(ARCHS_DUPES), $(AVAILABLE_TOOLCHAINS)))

# default: used for all-latest target
LATEST_ARCHS = $(foreach arch,$(sort $(basename $(subst -,.,$(basename $(subst .,,$(SUPPORTED_ARCHS)))))),$(arch)-$(notdir $(subst -,/,$(sort $(filter %$(lastword $(notdir $(subst -,/,$(sort $(filter $(arch)%, $(AVAILABLE_TOOLCHAINS)))))),$(sort $(filter $(arch)%, $(AVAILABLE_TOOLCHAINS))))))))

# legacy: used for all-legacy and when kernel support is used
#         all archs except generic archs
LEGACY_ARCHS = $(sort $(filter-out $(addsuffix %,$(GENERIC_ARCHS)), $(AVAILABLE_TOOLCHAINS)))

There may be a caveat though with:

# Global arch definitions
include $(BASEDIR)mk/spksrc.archs.mk

# Load local configuration
LOCAL_CONFIG_MK = $(BASEDIR)local.mk
ifneq ($(wildcard $(LOCAL_CONFIG_MK)),)
include $(LOCAL_CONFIG_MK)
endif

And migrate to spksrc.wheel-env.mk:

# fallback by default to native/python*
PIP ?= pip

# System default pip outside from build environment
PIP_SYSTEM = $(shell which pip)

# System default pip outside from build environment
PIP_NATIVE = $(WORK_DIR)/../../../native/$(or $(PYTHON_PACKAGE),$(SPK_NAME))/work-native/install/usr/local/bin/pip

# Why ask for the same thing twice? Always cache downloads
PIP_CACHE_OPT ?= --find-links $(PIP_DISTRIB_DIR) --cache-dir $(PIP_CACHE_DIR)
PIP_BASIC_OPT ?= --no-color --disable-pip-version-check
PIP_WHEEL_ARGS = wheel $(PIP_BASIC_OPT) --no-binary :all: $(PIP_CACHE_OPT) --no-deps --wheel-dir $(WHEELHOUSE)

# Adding --no-index only for crossenv
# to force using localy downloaded version
PIP_WHEEL_ARGS_CROSSENV = $(PIP_WHEEL_ARGS) --no-index

# BROKEN: https://github.com/pypa/pip/issues/1884
# Current implementation is a work-around for the
# lack of proper source download support from pip
PIP_DOWNLOAD_ARGS = download --no-index --find-links $(PIP_DISTRIB_DIR) --disable-pip-version-check --no-binary :all: --no-deps --dest $(PIP_DISTRIB_DIR) --no-build-isolation --exists-action w

[th0ma7]

for that would be worth adding python312-wheels and ffmpeg7 to enlarge the spectrum of the testing...

[mreid-tt]

hey @th0ma7, I was hoping to be able to merge this and am unclear on the scope expansion you have proposed. I did build locally the python312-wheels package for x64-7.1 and this completed successfully. The ffmpeg7 had different issues. The first of which was the nasm download. I can fix this by updating the source to https://www.nasm.us/pub/nasm/releasebuilds/2.16.03/nasm-2.16.03.tar.xz but even after that the container fails to complete the build with this reported:

cd /spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5 && env PATH=/spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5/build/bin:/spksrc/native/llvm-14.0/work-native/install/usr/local/bin:$PATH PKG_CONFIG=/usr/bin/pkg-config PKG_CONFIG_LIBDIR=/spksrc/spk/ffmpeg7/work-x64-7.1/install//var/packages/ffmpeg7/target/lib/pkgconfig WORK_DIR=/spksrc/spk/ffmpeg7/work-x64-7.1 INSTALL_PREFIX=/var/packages/ffmpeg7/target TC_WORK_DIR=/spksrc/toolchain/syno-x64-7.1/work TC=syno-x64-7.1  TC_GCC=$(eval $(echo /spksrc/spk/ffmpeg7/work-x64-7.1/../../../toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc -dumpversion) 2>/dev/null || true) TC_GLIBC=2.26 TC_KERNEL=4.4.180+ ninja -C /spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5/build
ninja: Entering directory `/spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5/build'
[0/54] Building opencl_clang_options.inc...
[0/54] Building CXX object bin/CMakeFil..._linker.dir/linux_resource_linker.cpp.o
[0/54] Copying '/spksrc/spk/ffmpeg7/wor...s/module.modulemap' to module.modulemap
[0/54] Copying '/spksrc/spk/ffmpeg7/wor...de//opencl-c-base.h' to opencl-c-base.h
[0/54] Copying '/spksrc/spk/ffmpeg7/wor....0.5/include//opencl-c.h' to opencl-c.h
[1/54] Copying '/spksrc/spk/ffmpeg7/wor....0.5/include//opencl-c.h' to opencl-c.h
[2/54] Building opencl_clang_options.inc...
FAILED: opencl_clang_options.inc /spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5/build/opencl_clang_options.inc 
cd /spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5/build && /spksrc/native/llvm-14.0/work-native/install/usr/local/bin/llvm-tblgen -gen-opt-parser-defs -I /spksrc/spk/ffmpeg7/work-x64-7.1/install/var/packages/ffmpeg7/target/include -I /spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5 /spksrc/spk/ffmpeg7/work-x64-7.1/opencl-clang-14.0.5/opencl_clang_options.td --write-if-changed -o opencl_clang_options.inc -d opencl_clang_options.inc.d
/spksrc/native/llvm-14.0/work-native/install/usr/local/bin/llvm-tblgen: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /spksrc/native/llvm-14.0/work-native/install/usr/local/bin/llvm-tblgen)
/spksrc/native/llvm-14.0/work-native/install/usr/local/bin/llvm-tblgen: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found (required by /spksrc/native/llvm-14.0/work-native/install/usr/local/bin/llvm-tblgen)
[3/54] Copying '/spksrc/spk/ffmpeg7/wor...s/module.modulemap' to module.modulemap
[4/54] Copying '/spksrc/spk/ffmpeg7/wor...de//opencl-c-base.h' to opencl-c-base.h
[5/54] Building CXX object bin/CMakeFil..._linker.dir/linux_resource_linker.cpp.o
ninja: build stopped: subcommand failed.

Thoughts on next steps?

[th0ma7]

We'd need to go back at compiling nasm like we did previously as a native dependency... At least temporarily until we remove it all together as already available in the container now since our last debian upgrade. @hgy59 already did some work on that but needs to be completed. I might take a stab at it over the coming weeks/months but in the meantime rebuilding a native nasm is the shortcut...

Unless we can fake it and just have a symlink from native nasm install to the container nasm...? Worth a try i guess...

[th0ma7]

@mreid-tt I took time to revisit this PR, while in theory simple it does have a few implications. I am unsure of the way to go, but with recent changes included in #6914 I think we should reconsider a few things...

1. I suggest creating the following (similar to what I did for toolchain):

mk/spksrc.common.mk
mk/spksrc.common
├── archs.mk
├── logs.mk
├── macros.mk
└── rules.mk

2. spksrc.common-rules.mk would be renamed as mk/spksrc.common/rules.mk
3. archs.mk would also contain all ARCHS* related entries from spksrc.common.mk
4. macros.mk would contain all macros: version_*, uniq, dedup, merge.
5. logs.mk would contain all log related entries, including terminal colors and possibly my new LOG_WRAPPED macro
6. spksrc.common.mk would include all those 4 new mk files
7. spksrc.directories.mk could then include mk/spksrc.common/macros.mk only

Caveat:

1. need to go over all cross/*/Makefile to ensure they always load spksrc.common.mk and not spksrc.archs.mk

[th0ma7]

@mreid-tt I had time today to take a stab at it - tested against python312 and python312-wheels which both built properly. Let me know what you think of this? ... noting that there are now conflicts with your recent merge to master but easy to fix.

EDIT: Noting that there are toooo many cross makefiles updated to use spksrc.common.mk instead of spksrc.archs.mk to allow github-action builds to complete. Also, I did migrated python wheel related environment variables to spksrc.wheel-env.mk which works fine as I had expected 🚀 Although I decided not to move spksrc.common-rules.mk as it really is a different thing altogether.

[th0ma7]

While not related I hapend to have noticed that the pre-built native/llvm-14.0 had a runpath error on llvm-spirv. I've simply recreated the package (using native/llvm-14.0-build) and released a new version on our spksrc github repo, then updated the download link for the native/llvm-14.0, now all looks good.

[th0ma7]

@mreid-tt I confirm that the following do build ok:

• python312 (aarch64 + x64)
• python312-wheels (aarch64)
• synocli-videodriver (x64)
• ffmpeg7 (x64)
• tvheadend (x64)

From my perspective this is good to go.

[mreid-tt]

@th0ma7, excellent, thanks much. @hgy59 will merge this tonight barring any objection from you.

[th0ma7]

@mreid-tt the description of the PR would need updating though... if you have any spare cycles to do so.

[th0ma7]

@mreid-tt I had a few more spare cycles and was able to update the PR description. Let me know if anything else needs adjusting.

[mreid-tt]

@mreid-tt I had a few more spare cycles and was able to update the PR description. Let me know if anything else needs adjusting.

@th0ma7, amazing work and thanks for the assist. Will merge in the next few hours.