Hierarchical namespace support for JBang directives

[wfouche]

A small game program written using LWJGL requires the following dependencies to work on various OS/CPU architectures. See PR jbangdev/jbang-examples#15

//DEPS org.lwjgl:lwjgl:3.3.6
//DEPS org.lwjgl:lwjgl-glfw:3.3.6
//DEPS org.lwjgl:lwjgl-opengl:3.3.6

//DEPS org.lwjgl:lwjgl:3.3.6:natives-windows
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-windows
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-windows

//DEPS org.lwjgl:lwjgl:3.3.6:natives-windows-arm64
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-windows-arm64
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-windows-arm64

//DEPS org.lwjgl:lwjgl:3.3.6:natives-linux
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-linux
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-linux

//DEPS org.lwjgl:lwjgl:3.3.6:natives-linux-arm64
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-linux-arm64
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-linux-arm64

//DEPS org.lwjgl:lwjgl:3.3.6:natives-macos
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-macos
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-macos

//DEPS org.lwjgl:lwjgl:3.3.6:natives-macos-arm64
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-macos-arm64
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-macos-arm64

Specifically on macOS (x64 and arm64) a Mac-specific runtime option is also required. This option is not supported by Java VMs on non-Mac operating systems and causes them to display an error message and stop.

//RUNTIME_OPTIONS -XstartOnFirstThread

JBang does not allow platform specific runtime options or dependencies to be configured that only take affect when a pre-determined OS or OS/CPU combination is detected.

This problem could be solved by allowing a small subset of JBang directives to be tagged with a namespace (package name in Java terminology).

• //RUNTIME_OPTIONS[namespace] -XstartOnFirstThread
• //DEPS[namespace] org.lwjgl:lwjgl:3.3.6:natives-macos-arm64

How is this envisioned to work in practice?

//DEPS org.lwjgl:lwjgl:3.3.6 org.lwjgl:lwjgl-glfw:3.3.6 org.lwjgl:lwjgl-opengl:3.3.6

//DEPS[os.windows.cpu.amd64] org.lwjgl:lwjgl:3.3.6:natives-windows org.lwjgl:lwjgl-glfw:3.3.6:natives-windows org.lwjgl:lwjgl-opengl:3.3.6:natives-windows
//DEPS[os.windows.cpu.arm64] org.lwjgl:lwjgl:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-windows-arm64

//DEPS[os.linux.cpu.amd64] org.lwjgl:lwjgl:3.3.6:natives-linux org.lwjgl:lwjgl-glfw:3.3.6:natives-linux org.lwjgl:lwjgl-opengl:3.3.6:natives-linux
//DEPS[os.linux.cpu.arm64] org.lwjgl:lwjgl:3.3.6:natives-linux-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-linux-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-linux-arm64

//DEPS[os.macos.cpu.amd64] org.lwjgl:lwjgl:3.3.6:natives-macos org.lwjgl:lwjgl-glfw:3.3.6:natives-macos org.lwjgl:lwjgl-opengl:3.3.6:natives-macos
//DEPS[os.macos.cpu.arm64] org.lwjgl:lwjgl:3.3.6:natives-macos-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-macos-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-macos-arm64

//RUNTIME_OPTIONS[os.macos] -XstartOnFirstThread

When a script is executed JBang would filter the configured directives based on OS and CPU architecture detected.

When JBang exports to Maven or Gradle, the namespace values will be used to ensure dependencies and runtime options only apply to specific OS/CPU combinations.

Fatjar support will also have to be revisited.

[wfouche]

A non-hierarchical representation of the same information:

//DEPS org.lwjgl:lwjgl:3.3.6 org.lwjgl:lwjgl-glfw:3.3.6 org.lwjgl:lwjgl-opengl:3.3.6

//DEPS[os=windows,cpu=amd64] org.lwjgl:lwjgl:3.3.6:natives-windows org.lwjgl:lwjgl-glfw:3.3.6:natives-windows org.lwjgl:lwjgl-opengl:3.3.6:natives-windows
//DEPS[os=windows,cpu=arm64] org.lwjgl:lwjgl:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-windows-arm64

//DEPS[os=linux,cpu=amd64] org.lwjgl:lwjgl:3.3.6:natives-linux org.lwjgl:lwjgl-glfw:3.3.6:natives-linux org.lwjgl:lwjgl-opengl:3.3.6:natives-linux
//DEPS[os=linux,cpu=arm64] org.lwjgl:lwjgl:3.3.6:natives-linux-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-linux-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-linux-arm64

//DEPS[os=macos,cpu=amd64] org.lwjgl:lwjgl:3.3.6:natives-macos org.lwjgl:lwjgl-glfw:3.3.6:natives-macos org.lwjgl:lwjgl-opengl:3.3.6:natives-macos
//DEPS[os=macos,cpu=arm64] org.lwjgl:lwjgl:3.3.6:natives-macos-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-macos-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-macos-arm64

//RUNTIME_OPTIONS[os=macos] -XstartOnFirstThread

[wfouche]

How could namespaces be supported on the JBang command-line? Simply add a --namespace option.

$ jbang --namespace "[os.macos]" --runtime-option -XstartOnFirstThread -namespace "[]" --deps org.lwjgl:lwjgl:3.3.6 ....

But then one can argue that JBang needs a //NAMESPACE directive.

//NAMESPACE [os.macos]
//RUNTIME_OPTIONS -XstartOnFirstThread

//NAMESPACE []
//DEPS org.lwjgl:lwjgl:3.3.6
//DEPS org.lwjgl:lwjgl-glfw:3.3.6
//DEPS org.lwjgl:lwjgl-opengl:3.3.6

//NAMESPACE [os.windows.cpu.amd64]
//DEPS org.lwjgl:lwjgl:3.3.6:natives-windows
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-windows
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-windows

//NAMESPACE [os.windows.cpu.arm64]
//DEPS org.lwjgl:lwjgl:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-windows-arm64

//NAMESPACE [os.linux.cpu.amd64]
//DEPS org.lwjgl:lwjgl:3.3.6:natives-linux org.lwjgl:lwjgl-glfw:3.3.6:natives-linux org.lwjgl:lwjgl-opengl:3.3.6:natives-linux

//NAMESPACE [os.linux.cpu.arm64]
//DEPS org.lwjgl:lwjgl:3.3.6:natives-linux-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-linux-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-linux-arm64

//NAMESPACE [os.macos.cpu.amd64]
//DEPS org.lwjgl:lwjgl:3.3.6:natives-macos org.lwjgl:lwjgl-glfw:3.3.6:natives-macos org.lwjgl:lwjgl-opengl:3.3.6:natives-macos

//NAMESPACE [os.macos.cpu.arm64]
//DEPS org.lwjgl:lwjgl:3.3.6:natives-macos-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-macos-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-macos-arm64

[maxandersen]

in #2133 the original idea was to have attribute list on //DEPS directive similar to how asciidoc does attributees, i.e. //DEPS{"run"} g:a:v, g:a2:v woulde be same as //DEPS g:a:v{scope="run"}, g:a2:v{scope="run"}

I didn't implement that as we didn't have a good usecase(s) enough to break the format of //DEPS for IDE's but also so that the attributes are part of the dependency request rather than to the directive.

But definitely something we could consider if we find enough usecases for it.

I find the idea of being able to say this directive only apply when certain criterias makes sense - and having a name for that (you call it namespace, maven would call it profile) actually could make the syntax quite compact.

We'll need a way to define which "profile"/"namespace" to run with and there would need to be some defaults defined as can't expect users caling app to define what namespace/profile to run with.

One way this could look like is:

//profile with name only active/true if fullfilling all properties
//PROFILE osx-arm64{os=mac,arch=aarch64}
//PROFILE osx{os=mac}
//PROFILE windows{os=windows}
//PROFILE windows-aarm64{os=windows,arch=aarch64}
//PROFILE linux={os=linux}

//runtime_options only include if profile=osx active
//RUNTIME_OPTIONS{profile=osx} -XstartOnFirstThread

//always included
//DEPS org.lwjgl:lwjgl:3.3.6
//DEPS org.lwjgl:lwjgl-glfw:3.3.6
//DEPS org.lwjgl:lwjgl-opengl:3.3.6

//only included when os and arch mathces
//DEPS{profile=windows} org.lwjgl:lwjgl:3.3.6:natives-windows,org.lwjgl:lwjgl-glfw:3.3.6:natives-windows,org.lwjgl:lwjgl-opengl:3.3.6:natives-windows

//DEPS{profile=windows-aarm64} org.lwjgl:lwjgl:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-glfw:3.3.6:natives-windows-arm64 org.lwjgl:lwjgl-opengl:3.3.6:natives-windows-arm64

//DEPS{profile=linux-arm64} org.lwjgl:lwjgl:3.3.6:natives-linux org.lwjgl:lwjgl-glfw:3.3.6:natives-linux org.lwjgl:lwjgl-opengl:3.3.6:natives-linux

//DEPS{profile=osx} org.lwjgl:lwjgl:3.3.6:natives-macos org.lwjgl:lwjgl-glfw:3.3.6:natives-macos org.lwjgl:lwjgl-opengl:3.3.6:natives-macos

issue with above is that windows and windows-aarm64 could both be active at the same time....

[wfouche]

@maxandersen , I'm working on a reponse but got something planned for the evening. Will reply more fully over the weekend. In the meantime I'll leave this hierarchical representation of a JBang configuration here. I think the PROFILE concept and NAMESPACE concepts are different. NAMESPACE is similar to PACKAGE (in Java). PROFILE would be created at runtime by os.name and os.arch which would then at runtime activate certain parts of the configuration tree.

[maxandersen]

yes, but that isn't (really) different from {os=macos,cpu=x64} except namespace is locked to only hardware notions where generic key/value is open ended.

I'm simply suggesting being able to name some of these combinations.

[wfouche]

I'm simply suggesting being able to name some of these combinations.

OK.

The main reason why I've been advocating for support for a largely pre-defined hierarchical configuration structure is that it will allow JBang to externalize configuration data into a JSON, TOML, or YAML configuration file like config.yaml similar to how Quarkus supports application.yaml

jbang:
  deps:
    - org.lwjgl:lwjgl:3.3.6
    - org.lwjgl:lwjgl-glfw:3.3.6
    - org.lwjgl:lwjgl-opengl:3.3.6
  runtimeOptions: -Xmx1g
  platforms:
    macos:
      runtimeOptions: -XstartOnFirstThread
      arch: 
        - x86_64:
            deps:
              - org.lwjgl:lwjgl:3.3.6:natives-macos
              - org.lwjgl:lwjgl-glfw:3.3.6:natives-macos
              - org.lwjgl:lwjgl-opengl:3.3.6:natives-macos
        - aarch64:
            deps:
              - org.lwjgl:lwjgl:3.3.6:natives-macos-arm64
              - org.lwjgl:lwjgl-glfw:3.3.6:natives-macos-arm64
              - org.lwjgl:lwjgl-opengl:3.3.6:natives-macos-arm64

In the main JBang script file a new directive //JBANG_CONFIG_FILE could then be used to retrieve additional JBang configuration data from a configuration file.

//JAVA 21+
//SOURCES File1.java File2.java
//JBANG_CONFIG_FILE myAppConfig.yaml

public static void main(String[] args) {
}

And if we have three different scripts (3 games implemented using LWJGL) that use the same configuration we do not have to duplicate the configuration directives in three programs; the shared config can simply be referenced using //JBANG_CONFIG_FILE.

The //NAMESPACE directive that was proposed earlier is really just a path to the location in the configuration data hierarchy where the data should be stored so it could be called JBANG_CONFIG_PATH.

//JBANG_CONFIG_PATH  /platforms/macos/cpu/x86_64
//DEPS org.lwjgl:lwjgl:3.3.6:natives-macos
...

[maxandersen]

The argument for being able to create toml isn't winning me over (yet :) but the argument that if we do a predefined structure based on nested arch parts could be used to define consistent profiles in maven and Gradle configs.

I like that.

[wfouche]

The argument for being able to create toml isn't winning me over (yet :)

It could be yaml as well. :-)

define consistent profiles in maven and Gradle configs

Nice.

[wfouche]

I'm not trying to advance the idea of including toml data in comments in a Java or other JBang supported language file anymore. But I do think that allowing configuration data (directives) to be externalized to a file (YAML might be best for this) has merit. It could be a nice evolutionary direction for JBang while retaining full support for all //JBANG-DIRECTIVES.

[wfouche]

Just realized that it is possible to have a dedicated Java source file containing only JBang directives which is then referenced using the //SOURCES directive from the main program. This approach works reasonably well to share config options between scripts or to externalize config options. This is new to me, but might already be a well-established JBang trick that several programs depend on.

[wfouche]

Example config updated to use //CONFIG_PATH.

//DEPS org.lwjgl:lwjgl:3.3.6
//DEPS org.lwjgl:lwjgl-glfw:3.3.6
//DEPS org.lwjgl:lwjgl-opengl:3.3.6
//RUNTIME_OPTIONS -Xmx1g

//CONFIG_PATH jbang/platforms/macos
//RUNTIME_OPTIONS -XstartOnFirstThread

//CONFIG_PATH jbang/platforms/windows/arch/x86_64
//DEPS org.lwjgl:lwjgl:3.3.6:natives-windows
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-windows
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-windows

//CONFIG_PATH jbang/platforms/windows/arch/aarch64
//DEPS org.lwjgl:lwjgl:3.3.6:natives-windows-arm64 
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-windows-arm64 
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-windows-arm64

//CONFIG_PATH jbang/platforms/linux/arch/x86_64
//DEPS org.lwjgl:lwjgl:3.3.6:natives-linux 
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-linux 
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-linux

//CONFIG_PATH jbang/platforms/linux/arch/aarch64
//DEPS org.lwjgl:lwjgl:3.3.6:natives-linux-arm64 
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-linux-arm64 
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-linux-arm64

//CONFIG_PATH jbang/platforms/macos/arch/x86_64
//DEPS org.lwjgl:lwjgl:3.3.6:natives-macos 
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-macos 
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-macos

//CONFIG_PATH jbang/platforms/macos/arch/aarch64
//DEPS org.lwjgl:lwjgl:3.3.6:natives-macos-arm64 
//DEPS org.lwjgl:lwjgl-glfw:3.3.6:natives-macos-arm64 
//DEPS org.lwjgl:lwjgl-opengl:3.3.6:natives-macos-arm64

//CONFIG_PATH jbang

[maxandersen]

What is config_path?

[wfouche]

What is config_path?

Thought it would be a better name to use than namespace:

• //NAMESPACE [jbang.platforms.macos.arch.arm64]
• //CONFIG_PATH jbang/platforms/macos/arch/arm64