jni-bind compatible with Visual Studio 2019 - 16.11.40?

[davidrodsanchez]

I am trying to setup the environment on a Windows 10 machine, using Visual Studio 2019 with compilation flags:
"/std:c++20 /Zc:__cplusplus"

I am including jni_bind_release.h and get a bunch of compilation errors:
... jni-bind\implementation\no_class_specified.h(46,3): error C2131: expression did not evaluate to a constant
... class.h(182,16): error C3547: template parameter 'RhsParams' cannot be used because it follows a template parameter pack and cannot be deduced from the function parameters of 'jni::operator =='
... jni_type.h(198,46): error C2975: 'jni::JniT': invalid template argument for 'class_v_', expected compile-time constant expression

Any idea on what I am doing wrong?

[jwhpryor]

I'm afraid not, and, this might sound ridiculous but I actually really struggle to get a Windows environment up and running. I don't work with VSCode so it's an extra lift for me to get it running. You're not doing anything wrong, it's likely a compiler incompatability.

That said, these types of compatibility issues when bringing up a new compiler are common. If you could provide me a Godbolt sample I can look at, preferably with as little code as possible, I'd be happy to take a look.

If you/we can make it work, I'd love to fold this into JNI Bind and add an MSVC compilation test. That said, my time is somewhat more limited and I need some help if folks want this.

This issue was tracked, but I unfortunately gave up due to lack of time/resources:
#197

[davidrodsanchez]

I am happy to try and make this happen, though I am affraid my level of metaprogramming is far from what I am seeing.
Any suggestion on how to start?

I can try to reproduce the first error using small chunks of code with Godbolt.

[jwhpryor]

OK awesome. No worries about metaprogramming, the main issue is just getting the problem in a form I can look at it.

Are you familiar with Bazel? Building the project from root with --keep_going is a nice way to isolate where the problem starts. The project is heavily tested so understanding where the issue begins by pasting the output of a build root is quite helpful.

e.g. if something is breaking in metaprogramming the failure is lower than higher in the stack. This is how I debug these types of issues myself.

Good luck, I look forward to hearing back

[davidrodsanchez]

jni_bind_release_ERRORS.txt

I don't use Bazel normally, but I will set up the environment and trigger a build as you indicate.
In the mean time, I attach a .txt file with the output I get, in case it helps, by just doing: #include "jni_bind_release.h"

I will reach back once I get results from Bazel

[jwhpryor]

Ah, hmmn, actually the first part of the bug stands out:

[build] C:\Users\jni-bind-test\cots\jni-bind\jni_bind_release.h(2165): message : see declaration of 'jni::operator !=' [C:\Users\jni-bind-test\build\Example\Example.vcxproj]
[build] C:\Users\jni-bind-test\cots\jni-bind\jni_bind_release.h(7911,26): error C2664: 'void JNIEnv_::ReleaseIntArrayElements(jintArray,jint *,jint)': cannot convert argument 2 from 'int *' to 'jint *' [C:\Users\jni-bind-test\build\Example\Example.vcxproj]

I think this might be the same as #198.

Basically, jint is a different type on Windows I think. If so I think I can probably fix it but it would help to have a way to repro this, which I'm not 100% sure how to do.

[davidrodsanchez]

The problem seems to point to #198 as you mention.
In that issue, it is also indicated that the user managed to make jni-bind compile on Windows. May it be worth asking?

I don't know how you normally prefer to deal with this kind of relatively big migrations.
If you want to create a separate branch where we collect all changes you make and from where I can later pull and test in my environment...?

Probably not a very efficient solution but I am happy to provide the Windows end point.
If you have any other suggestion, let me know

[jwhpryor]

I think this is what he did:

"If I take a big hammer and make the typedef to a C int on Windows, all is well with jni-bind.

// 'long' is always 32 bit on windows so this matches what jdk expects
typedef int jint; // local change long -> int --KC"

Obviously that isn't suitable for JNI Bind. I think what actually needs to happen is just a simple change in the proxy system but I need a mechanism to exercise it which I could never duplicate.

If you're able to create a pull request I can exercise it if you just propose it. We can work on merging pull requests later if we manage to get there, it's actually never come up before as all submissions have originated from Google (and mostly from me).

I think if you try the author's "big hammer" you should be able to at least try JNI Bind (beyond the Godbolt sample). To truly fix it, I think you need to modify here.

jni-bind/implementation/proxy_definitions.h

Line 98 in 2201a3e

template <typename LongType>

I think there also needs to be an int proxy which conditionally exposes a proxy function with the correct size.

Your simplest bet is to build with Bazel (modifying the monolith header will be hard). I can give it a quick try if that doesn't work but it's hard given I can't validate the failure. I guess an alternative way you could help would be to make a PR to add a Windows test runner so I can see failure.

[jwhpryor]

Curious if you made any progress? Let me know if I can provide support.

[davidrodsanchez]

Hey, I am sorry for not reaching back in a few days. It is being very busy for me lately, but I don't forget about this.
Let me ping you when I have some time to settle and prepare everything. I am interested in getting this done.
Hope this is fine for you too

[jwhpryor]

You're fine, take your time! There's interest on it being completed based on the web traffic, but I am not on any particular deadline for anything. I'm just glad to have some help.

[davidrodsanchez]

Hi again,

I made some progress setting up bazel, but I am not sure I am building the project correctly.

I ran from root: bazel build --check_direct_dependencies=off //:jni_bind

And apparently got no errors?

Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: target //:jni_bind (1 packages loaded, 0 targets configured)
Analyzing: target //:jni_bind (1 packages loaded, 0 targets configured)

Analyzing: target //:jni_bind (42 packages loaded, 7 targets configured)

Analyzing: target //:jni_bind (43 packages loaded, 7 targets configured)

Analyzing: target //:jni_bind (43 packages loaded, 7 targets configured)

Analyzing: target //:jni_bind (43 packages loaded, 7 targets configured)

Analyzing: target //:jni_bind (54 packages loaded, 407 targets configured)

Analyzing: target //:jni_bind (55 packages loaded, 412 targets configured)

Analyzing: target //:jni_bind (64 packages loaded, 764 targets configured)

Analyzing: target //:jni_bind (64 packages loaded, 764 targets configured)

Analyzing: target //:jni_bind (64 packages loaded, 764 targets configured)

INFO: Analyzed target //:jni_bind (70 packages loaded, 928 targets configured).
INFO: Found 1 target...
Target //:jni_bind up-to-date (nothing to build)
INFO: Elapsed time: 47.595s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

Am I missing something?

[jwhpryor]

It looks like you built nothing. Can you try bazel test --cxxopt='-std=c++20' ...

[jwhpryor]

Im not actually sure why that wouldn’t build anything but maybe windows treats precompiled headers differently.

Also add —keep-going so we can see all the failures.

[davidrodsanchez]

Ok, I think I managed to produce some meaningful output.
Please find the log file attached
output.zip

[davidrodsanchez]

How do you feel about creating a separate branch where you make your changes and I checkout and build on my end?

[jwhpryor]

I'm seeing the following in your log:
"C:\users\t0244288_bazel_t0244288\exu2l7ph\execroot_main\metaprogramming/string_concatenate.h(27): error C2039: 'string_view': is not a member of 'std'"

To me that sounds like something is way off, like you don't have the standard library. I'm not sure how you could be missing that include unless you had a pre C++17 compiler (string_view was first in 17).

I also see Command line warning D9002 : ignoring unknown option '-std=c++20'. Are you sure you're not defaulting to a very old C++11 compiler?

If your flags weren't being communicated to Bazel properly I can imagine it might default to a really old C++ version.

Happy to communicate through a branch when I have something to share, that sounds like a good idea.

[davidrodsanchez]

I think I managed to ask Bazel to set the correct C++ standard (notice the slash for Windows):
bazel test --keep_going --cxxopt='/std=c++20' ...

I am however hitting against another wall with Bazel:

Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Analyzing: 336 targets (0 packages loaded, 0 targets configured)
Analyzing: 336 targets (0 packages loaded, 0 targets configured)

Analyzing: 336 targets (0 packages loaded, 0 targets configured)
[1 / 22] Compiling googletest/src/gtest.cc; 1s local ... (12 actions running)
ERROR: C:/sw/jni-bind/implementation/jni_helper/BUILD:19:8: Compiling implementation/jni_helper/arg_string_test.cc failed: (Exit 2): cl.exe failed: error executing CppCompile command (from target //implementation/jni_helper:arg_string_test) C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\cl.exe ... (remaining 1 argument skipped)
cl : Command line warning D9024 : unrecognized source file type 'C:/Program', object file assumed
cl : Command line warning D9027 : source file 'C:/Program' ignored
cl : Command line warning D9024 : unrecognized source file type 'Files/Git/std=c++20', object file assumed
cl : Command line warning D9027 : source file 'Files/Git/std=c++20' ignored

I feel like someone does not understand spaces... but they are implicit in the compiler path itself...

[jwhpryor]

Interesting, so, I submitted a CL yesterday to stifle an unrelated build warning and it also forces C++20 (barring any other vs. being set), just FYI (it shouldn't impact your work).

What you're describing does indeed sound correct, but I'm afraid it's beyond my ability to debug, and I would expect it's unrelated to JNI Bind. That said, it does have some of your change incorporated in it, and I wonder if the forward slash is sinking it.

Could you try syncing to head and try without your additional flag? Otherwise, I'm not sure what to try, although I'm excited to see your progress.

[davidrodsanchez]

I pulled your last commits. Did you intend to push an empty .bazelrc?
7d07d37

git pull --rebase origin main
Updating f4bb949..7d07d37
Fast-forward
.bazelrc | 3 +++
.gitignore | 2 ++
metaprogramming/next.h | 2 +-

Doing:
bazel test --keep_going ...

At some point in the compilation I see:

...
ERROR: C:/sw/jni-bind/metaprogramming/BUILD:121:8: Compiling metaprogramming/call_test.cc failed: (Exit 2): cl.exe failed: error executing CppCompile command (from target //metaprogramming:call_test) C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\cl.exe @bazel-out/x64_windows-fastbuild/bin/metaprogramming/_objs/call_test/call_test.obj.params
C:\users\t0244288_bazel_t0244288\djneswl7\execroot_main\metaprogramming/call.h(21): error C2429: language feature 'nested-namespace-definition' requires compiler flag '/std:c++17'
...

Let me know and I try again

[jwhpryor]

Sigh, that was a mistake on my part. I submitted what it actually should be. Regardless, I don't think that will take away your whitespace error. I wonder if that issue is actually unrelated to Bazel. Have you tried building something unrelated using Bazel in a similar file structure?

If that doesn't work I can look at taking more drastic measures like adding another build system or an automated test runner for MSVC. It would be much better if we had a stake in the ground where one build system worked on all platforms however, and then add more build systems.

[davidrodsanchez]

Ok, I pulled again and at some point:
bazel test --keep_going ...
...
cl : Command line warning D9002 : ignoring unknown option '-std=c++20'
...

So MSVC does not like that flag.
So far I did not manage to pass a compilation argument using Bazel.

I was thinking exactly about that idea... I am more familiar with CMake, but I will investigate further what is going on with Bazel.

[jwhpryor]

Bummer, if Visual Studio doesn't allow Bazel to build with C++20 then it won't do very much at all.

I put together a branch which is pure AI slop to build with CMake: #395

It has other unrelated parent commits you will need to tease out. I have no idea if it compiles, I was just experimenting with AI tools.

Apologies this is all so hard, build systems always seem that way :( If you can come up with a way to make GitHub actions trigger the failure that would also be good.

[davidrodsanchez]

No problem at all.

I will work on trying to make this compile using CMake on Windows (at least up to the point where we can make progresss on what really matters :D )

I will try to organise the cmake scripts such that it can be made cross-platform at some point, so you can include your linux/macos compilation conditions.

[jwhpryor]

That'd be awesome! Feel free to snag #240

[davidrodsanchez]

Can you support me in doing so? (not familiar with open-source contribution)

I have some code to be pushed. I created branch feature/240_cmake, which can't push:

$ git push origin feature/240_cmake
remote: Permission to google/jni-bind.git denied to davidrodsanchez.

[jwhpryor]

Sure! You're the first outside contributor so bear with me if it takes a bit. Shot you an invite for write access.

[davidrodsanchez]

Great, apparently I managed to create the first pull request (as draft)
https://github.com/google/jni-bind/pull/396/files#

I also generated some compilation output:
compilation.txt

[jwhpryor]

Awesome! I will need some time on my end to figure out how to merge this upstream in google3. I'll try to get back to you soon on the compilation failure, but it looks familiar which is a good sign.

Specifically, [build] C:\SW\jni-bind\implementation/jni_helper/jni_array_helper.h(302,26): error C2664: 'void JNIEnv_::ReleaseIntArrayElements(jintArray,jint *,jint)': cannot convert argument 2 from 'int *' to 'jint *' [C:\SW\jni-bind\cmake_build\array_test.vcxproj]

This is almost certainly the issue. I recall a differently sized jint on MSVC was the root cause of the issue.

#198

[jwhpryor]

Sorry, I haven't had time to look at this, but I imagine this can be replicated by forcing jint to a different size on Linux (so I can test).

I imagine this can be fixed by adding a separate definition in proxy_definitions.h 's primitive proxy defintion.

[jwhpryor]

Alright, finally had time to take a look. I posted #398 but don't have a way to test it. It's rebased on top of your cmake so you should be able to try it out of the box.

If you feel so inclined you're welcome to setup a github action for CMake + Windows and I can validate my own work.

Please let me know how it goes!

[davidrodsanchez]

Ok, I pulled your changes from #398 and triggered a build

I copied some logs to the attached file (stopped the build process at some point):
output.log.txt

I will look into creating a github action as you suggest (never did before)
Would be nice to also setup an action for CMake + currently validated OS with Bazel, so we can validate the CMake part.
Can also support on that. I think using the CMakePresets.json file I included should help

[davidrodsanchez]

Ok, I think I got it running. We can now test builds on the same environment I got:
https://github.com/google/jni-bind/actions/runs/14158251801/job/39660304925

If you want to reduce the number of parallel jobs, update the field "jobs" in file CMakePresets.json
It may help when navigating the logs. Once we get it to compile, we can increase it again.

Let me know how I could help you further

[jwhpryor]

This is phenomenal, it will help me a lot. I checked out your output failure and I believe it makes sense, it's complaining that jint and int aren't the same size which is the same failure. I will take a look at it and get back to you.

[jwhpryor]

I tried to adopt your github action except I folded it into the main set of workflows. It didn't work as intended, complaining about a missing visual studio.

Do you see anything I could have done wrong here: https://github.com/google/jni-bind/actions/runs/14160602391/job/39665494437?pr=400 (#400).

Edit: Never mind, I see now you had 2019 as the windows vs. not latest. I can do that, although, is there any good reason for it?

[davidrodsanchez]

Exactly, my latest commits point to windows-2019, which provide Visual Studio 19

I chose that version as it is the baseline I have for my needs, but we can create different toolchains, pointing to different OS/compiler releases.

I assume once we get it to work for 2019, it "should" be easier to make it compile for newer VS releases

[jwhpryor]

Okay, I've made some progress on https://github.com/google/jni-bind/actions/runs/14163177478/job/39671863750?pr=401 #401 .

It's failing on an error that makes no sense to me and I can't really dig into. It also looks like it's struggling with tests in jni_helper. I think this is possibly a cmake issue? It's the first test in the sub directory and it's looking for files it can't find.

Beyond that I think there are other real failures (that I don't understand). Could I ask you to take a pass and see if the errors could be a cmake issue?

[davidrodsanchez]

Regarding compilation error:
D:\a\jni-bind\jni-bind\implementation\jni_helper\trace_test.cc(21,10): fatal error C1083: Cannot open include file: 'testing/base/public/googletest.h': No such file or directory [D:\a\jni-bind\jni-bind\cmake_build\trace_test.vcxproj]

I could not find header file "googletest.h" anywhere in the repo, even after googletest checkout.
I also noticed "trace_test.cc" is not mentioned anywhere in your bazel BUILD files. Is this test being compiled?

If this is the case, I can remove it from CMake compilation process

Is this the error you meant not to make any sense? Do you want me to look into any of the other errors?

[jwhpryor]

I think this is a red herring, because gtest is used elsewhere, and it's pulled in through the dependencies. Actually, this is widely used across this project (and others), so I think that's not the issue.

trace_test.cc is under jni_helper. I think this is the first test getting included from that directory and for some reason it's different and not finding gtest (maybe?). gtest is used in metaprogramming and I see those tests running first (e.g. https://github.com/google/jni-bind/blob/main/metaprogramming/tuple_from_size_test.cc#L22).

So I think somehow it's always failing immediately in jni_helper on the first test it sees (I think it's doing a DFS search for tests which IIRC bazel does too). Something about this new directory is causing cmake not to find the header there.

If you can't see anything obvious, could you write me a cmake that isolates on a single failing test from jni_helper that fails so I can play with one in isolation?

---

As an aside, I think a real failure is this D:\a\jni-bind\jni-bind\implementation/no_class_specified.h(48,3): message : failure was caused by call of undefined function or one not declared 'constexpr' [D:\a\jni-bind\jni-bind\cmake_build\signature_inherited_method_test.vcxproj]

That, and the surrounding error message is probably for me to deal with, but I'm struggling to make progress because I need to resolve earlier failures to be certain this isn't for fake reasons. Sometimes if a failure happens early it "train wrecks" for subsequent failures and I want to be sure this isn't that.

[davidrodsanchez]

Notice that I am able to compile "arg_string_test" (from jni_helper subset), which makes use of GTest as it includes gtest.h header file:

...
#include <gtest/gtest.h>
...

Compilation output:
[build] arg_string_test.vcxproj -> C:\SW\jni-bind\cmake_build\Debug\arg_string_test.exe
[driver] Build completed: 00:00:01.903

But the header file reported in the compilation error, is nowhere to be found:

In file trace_test.cc:
...
#include "testing/base/public/googletest.h"
#include <gtest/gtest.h>
...

So I think the problem is with this particular header file path.
This is why I was wondering if it is really being compiled with Bazel (I could not find any reference of this test in any BUILD file)

Other tests under jni_helper, such as "dry_run_test.cc" fail to compile for different reasons (jni-bind related compatibilities):
[build] dry_run_test.cc
[build] C:\SW\jni-bind\implementation/no_class_specified.h(48,3): error C2131: expression did not evaluate to a constant [C:\SW\jni-bind\cmake_build\dry_run_test.vcxproj]
[build] C:\SW\jni-bind\implementation/no_class_specified.h(48,3): message : failure was caused by call of undefined function or one not declared 'constexpr' [C:\SW\jni-bind\cmake_build\dry_run_test.vcxproj]
[build] C:\SW\jni-bind\implementation/no_class_specified.h(48,3): message : see usage of 'jni::NoClass::NoClass' [C:\SW\jni-bind\cmake_build\dry_run_test.vcxproj]
[build] C:\SW\jni-bind\implementation/jni_type.h(198,46): error C2975: 'jni::JniT': invalid template argument for 'class_v', expected compile-time constant expression
..._

[jwhpryor]

🤦🏻‍♂️ I see now. I updated it but looks like I'm now having other unrelated cmake issues... I took off a trailing newline you added from the version file but I'm not sure if it's causing issues. Looks like it is. I'll have to take a look.

[davidrodsanchez]

Yes, this is a problem as it is expecting the content to be in a single line.
Is there a reason why you put it back? CMake is probably not being able to pass the configure step

[jwhpryor]

Yes, unfortunately GitHub was failing because I added Windows, but then I guess for an unrelated reason, was failing to queue other presubmits. This meant HEAD was failing and I couldn't figure out why, so I was progressively reverting as much as I could until it compiled. I'm going to add it back now, sorry, this has been a slog, it's very slow to test.

[jwhpryor]

So, I've isolated at least one failure that needs to be fixed: https://github.com/google/jni-bind/actions/runs/14225648537/job/39864436223?pr=401

D:\a\jni-bind\jni-bind\metaprogramming\next_test.cc(118,1): error C3201: the template parameter list for class template 'jni::metaprogramming::ConstRefCounter' does not match the template parameter list for template parameter 'Container' [D:\a\jni-bind\jni-bind\cmake_build\next_test.vcxproj]
D:\a\jni-bind\jni-bind\metaprogramming\next_test.cc(135,2): error C2913: explicit specialization; 'jni::metaprogramming::NextConstRefVal' is not a specialization of a class template [D:\a\jni-bind\jni-bind\cmake_build\next_test.vcxproj]
D:\a\jni-bind\jni-bind\metaprogramming\pack_discriminator.h(76,36): error C2794: 'type': is not a member of any direct or indirect base class of 'jni::metaprogramming::PackDiscriminatedForward<jni::metaprogramming::NextType,jni::metaprogramming::NextConstRefVal>::Helper<jni::metaprogramming::ConstRefCounter<jni::metaprogramming::kTheValue0,jni::metaprogramming::kMaxValSizeGlobal>>' [D:\a\jni-bind\jni-bind\cmake_build\next_test.vcxproj]
D:\a\jni-bind\jni-bind\metaprogramming/next.h(49): message : see reference to alias template instantiation 'jni::metaprogramming::PackDiscriminatedForward<jni::metaprogramming::NextType,jni::metaprogramming::NextConstRefVal>::type<jni::metaprogramming::ConstRefCounter<jni::metaprogramming::kTheValue0,jni::metaprogramming::kMaxValSizeGlobal>>' being compiled [D:\a\jni-bind\jni-bind\cmake_build\next_test.vcxproj]

It looks like the pack discriminated forward is being treated differently by MSVC than it is on clang, although I'm not sure why. This will require more digging. For now I I'm running next_test only but it shows the behaviour fairly clearly.

[jwhpryor]

Sorry, it's unbelievably slow to try to fix this via presubmit tests. I tried to get a Windows distro up and running so I could try Cmake but I'm bizarrely unable to get there because of corp reasons. I don't have a personal Windows license so it might be a while.

[davidrodsanchez]

Do you think it would help to just build one test at a time? Once you get that fixed, I activate the next one, and so on...?
I can set up something like this in my branch.

By the way, I will be AFK for at least 1 week from the 8th.

[jwhpryor]

That's what I was doing, but trying to go one by one is quite slow. I comissioned a Windows machine I could connect into on my personal machine so we'll see how that goes.

Enjoy your holiday!

[jwhpryor]

(Feel free to add me on LinkedIn btw, you've been very helpful!)

[davidrodsanchez]

Sure, I just sent you an invite.
I saw you also enjoy chess ;D

[jwhpryor]

For posterity, the installer for msvc 2019 build tools are here https://learn.microsoft.com/en-us/visualstudio/releases/2019/history#release-dates-and-build-numbers

[jwhpryor]

And JDK here: https://www.oracle.com/java/technologies/downloads/#jdk24-windows