[Feature request] native sse2avx option support

[Andarwinux]

Clang and GCC support use -msse2avx to convert legacy SSE instructions into VEX-prefixed, which avoids AVX-SSE transition penalties and allows unnecessary vzeroupper to be omitted. For projects that use x86inc.asm, -DFORCE_VEX_ENCODING=1 can be used, but native option support in NASM would be preferable.

[lab-bat]

The VideoLAN Gitlab Projects seems to use NASM/YASM syntax to generate assembly source code. The switch FORCE_VEX_ENCODING was used in macros such as INIT_XMM under the /common/x86 of x264 GIT repository. The switch avx_enabled was carried out to more macros everywhere.

The AVX instruction set expansion can in place SSE instruction set expansion. Lots of legacy code rewrite that needs efforts can be free I guess.

However, I would not be happy to have assembler program to cross the third rail. The developer of x264 utializes the NASM macro. Macros are used to make a sequence of computing instructions available to the programmer as a single program statement. The NASM macro is nicely used for niche assembly optimazation here, but I believe NASM macros can be used more extensively in general scenarios.

[Andarwinux]

Yes, FORCE_VEX_ENCODING is a convenience macro provided by x86inc.asm, but not all projects or code use x86inc.asm. I noticed this issue when investigating the performance of x86-64-v3 distributions like CachyOS.

I think it would be useful if NASM itself provided a --sse2avx option to transparently transcode legacy SSE to VEX-prefixed when emitting object files in backend, rather than relying on frontend macro magic. This would allow even code that doesn't use x86inc.asm to migrate to AVX-128 for free, and still allow source-level support for baseline x86-64 to be retained.