libload / dlopen Enzyme/autodiff

[ZuseZ4]

A draft @davidlattimore and I developed at RustChina Unconf.

Closes #143295

r? @bjorn3 Does that generally look like what you had in mind?

I'll update the compiler/rustc_codegen_llvm/src/llvm/enzyme_ffi.rs to also go through libloading, and load the right path rather than hardcoding it. So far it passes some tests, I assume the other ones only fail due to enzyme_ffi not being updated yet.

[bjorn3]

You could do something like

rust/compiler/rustc_interface/src/util.rs

Lines 371 to 381 in 8a1b399

	let target = host_tuple();
	let sysroot = sysroot
	.all_paths()
	.map(|sysroot| {
	filesearch::make_target_lib_path(sysroot, target).with_file_name("codegen-backends")
	})
	.find(|f| {
	info!("codegen backend candidate: {}", f.display());
	f.exists()
	})

[ZuseZ4]

Do you know a good way of getting the sysroot in rustc_codegen_llvm, where I do all the loading?
I tried to use a cell / static in my last commit to pass it from the frontend to the backend, but (presumably due to using different threads?) the CELL with the path to Enzyme ends up uninitialized in the thread trying to load it.
Since most of these functions are generic over backends, I also don't want to pass the Session object or Enzyme path to the llvm backend as fnc argument if it can be avoided.

[ZuseZ4]

also @Kobzol since you might know.

[Kobzol]

I think that adding the sysroot e.g. to CodegenContext should be fine. Btw, note that statics don't work like you are using them, it seems like you expect a static with the same name to be "shared" across all usages of that name? https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=d187e01b5231eaa65e9eaf64732545c0 Or maybe it's just some leftovers from debugging.

[Kobzol]

Sounds exciting! With this, will we still have to compile LLVM in some "special" way, so that it has support for Enzyme? Or is having the libEnzyme.so file on disk enough?

[ZuseZ4]

That's not an issue anymore, I had removed all extra handling multiple months ago, in my previous push to get it on nightly. The only thing left would now be to add Enzyme as a downloadable rustup component. Would you be able to help with that and take over once I cleaned up and landed this PR @Kobzol? I don't know much about distributions and would like to avoid to have this laying around too many extra months.

[Kobzol]

Yeah, if having libEnzyme.so on disk is enough, then I can help getting that into rustup. There are still some questions to resolve, but we can leave that for a separate issue.

[bors]

☔ The latest upstream changes (presumably #146685) made this pull request unmergeable. Please resolve the merge conflicts.