Are Graal config files like reflect-config.json in a Quarkus extension not used?

[programmatix]

I've been getting confused why our Graal config files (native-image.properties, reflect-config.json et al) in the WIP Quarkus Couchbase extension didn't seem to be respected in the final native compilation. (And I think they surely used to be, since I was able to get a similar app compiling when I was last doing Quarkus work many months ago.)

I've used quarkus create and then added some very basic code to use the Couchbase extension, then tried to compile natively with mvn clean integration-test -Dnative -Dquarkus.version=999-SNAPSHOT.

I found that the produced jar code-with-quarkus-1.0.0-SNAPSHOT-runner.jar did not contain the reflect-config.json (or such files from other dependencies, like Netty), and then found https://quarkus.io/guides/writing-native-applications-tips: "By default, when building a native executable, GraalVM will not include any of the resources that are on the classpath into the native executable it creates. Resources that are meant to be part of the native executable need to be configured explicitly". So presumably that's related?

That guide also says how to configure reflection et al for an extension, using code like ReflectiveClassBuildItem.

We want to eventually upstream the reflect-config.json etc. files upstream into the Couchbase Java SDK, so that non-Quarkus Graal users would benefit too. But it seems we will need the same Graal config both in reflect-config.json, and written out in code form in the Quarkus extension. Is this correct?

We also have very large reflect-config.json originally output by the Graal agent and manually added to since then, and the idea of converting all that to code is... not fun.

Hopefully I'm just missing something and there's an easier solution to all this?

[gastaldi]

One solution proposed by @evanchooly would be to have a tool/extension that took those config files and generated the appropriate sources for Quarkus.

It would save from having to learn all the ins and outs and it would deal with the native-image config files.

[Sanne]

This should be unnecessary, we don't disable any native-image.properties from the 3rd party jars (nor from the extensions).

[maxandersen]

Same guide states:

“ you prefer relying on the GraalVM infrastructure, you can also create a resources-config.json (the most common location is within src/main/resources) JSON file defining which resources should be included”

Does that not work for you ?

Also note just blindly adding big massive auto generated json might be easy but it could also end up adding too much - so just be cautious :)

[programmatix]

Thanks for the replies both.

@maxandersen that line appears to apply to a native application rather than an extension? I'll give it a go when I'm back on Quarkus though.

The somewhat-autogenerated reflect-config.json isn't too bad for us, about 1000 lines after some initial manual pruning. Big enough that I don't relish turning it manually into code form though :)

@gastaldi yes, such a tool seems a reasonable approach.

[Sanne]

We don't disable any native-image.properties found in other libraries, they should get picked up by native-image and get applied in addition to other features and metadata that we generate via BuildItems.

We do however offer some APIs (ExcludeConfigBuildItem) for extension authors to explicitly ignore a particular instance of such properties, as often we find that library maintainers have included suboptimal instructions - or even in some cases these are totally wrong, so we can improve - but this is an explicit opt-in that the extension maintainer needs to use explicitly.

So if you don't see these resources being picked up, we should figure out why that is the case.

As a general thought, the BuildItems ins Quarkus are much more powerful than the static resources as you can emit them conditionally, for example based on configuration properties: for example you might be able to only register for reflection certain subsystems, ignoring others which are not applicable with the current build-time configuration settings. The same applies for JNI, service loading, etc.. so having a set of dynamic BuildItems in Quarkus can produce more efficient and performant code when compared to using the resources in libraries.
So while I do appreciate that some maintainers will need to keep things simple for them, please consider including in native-image.properties only the things which are truly essential for the library to work, and then agument these rules with build items for all optional aspects.

[holly-cummins]

Hi @programmatix, did Sanne's suggestion help?

[programmatix]

Sorry, I'd missed the reply notification.

So, it sounds like what I'm trying to do should work, and that if it's not, we should find out why?

What are the next steps for that? I spent a couple of days on this previously, trying a number of things, and couldn't get native-image to acknowledge my configuration files no matter what I did. Next time I can get back to this I'll upload the WIP latest of both my simple test app and the Couchbase extension, and perhaps you could take a look at them and see if I'm doing anything wrong? And perhaps you could point me at a simple project that includes a Quarkus extension using configuration files, that is successfully natively compiling, something I can use as a point of comparison?

Thank you for the advice on ExcludeConfigBuildItem (which we're not using) and BuildItems. I appreciate there's a lot of things we can do to optimise the native compilation by using Quarkus' code-based approach. My focus currently though is to get the tried-and-tested existing Graal config files working with Quarkus so we have a working alpha-level extension, and I can iterate on it from there.

[Sanne]

Sorry, I'd missed the reply notification.

So, it sounds like what I'm trying to do should work, and that if it's not, we should find out why?

Right

What are the next steps for that? I spent a couple of days on this previously, trying a number of things, and couldn't get native-image to acknowledge my configuration files no matter what I did. Next time I can get back to this I'll upload the WIP latest of both my simple test app and the Couchbase extension, and perhaps you could take a look at them and see if I'm doing anything wrong?

Yes, if you could share some code that shows the problem we'd be happy to help - doesn't need to be polished at all, just something we can run and play with as well.

And perhaps you could point me at a simple project that includes a Quarkus extension using configuration files, that is successfully natively compiling, something I can use as a point of comparison?

The oracle - jdbc extension does this: we rely on the metadata provided by the Oracle engineering team, as it's not open source and this would otherwise be unmaintainble.
We also apply some minor tuning by using ExcludeConfigBuildItem on a particular subset of such resources, to fix a problem with it; hopefully we'll be able to avoid that when an improved version gets released.

So it's a bit of a complex example, but it's a fairly comprehensive example of such capabilities. On top of my head I don't remember of other similar extensions relying on such metadata.

Thank you for the advice on ExcludeConfigBuildItem (which we're not using) and BuildItems. I appreciate there's a lot of things we can do to optimise the native compilation by using Quarkus' code-based approach. My focus currently though is to get the tried-and-tested existing Graal config files working with Quarkus so we have a working alpha-level extension, and I can iterate on it from there.

Sure, that's my preferred approach as well :)