Port to LLVM 11

[anatol]

I am trying to compile tinygo with LLVM 11 release and I get following compilation failure:

# tinygo.org/x/go-llvm
backports.cpp:10:28: error: redefinition of ‘llvm::PassManagerBuilder* llvm::unwrap(LLVMPassManagerBuilderRef)’
   10 | inline PassManagerBuilder *unwrap(LLVMPassManagerBuilderRef P) {
      |                            ^~~~~~
In file included from backports.cpp:4:
/usr/include/llvm/Transforms/IPO/PassManagerBuilder.h:255:28: note: ‘llvm::PassManagerBuilder* llvm::unwrap(LLVMPassManagerBuilderRef)’ previously defined here
  255 | inline PassManagerBuilder *unwrap(LLVMPassManagerBuilderRef P) {
      |                            ^~~~~~
backports.cpp:14:34: error: redefinition of ‘LLVMOpaquePassManagerBuilder* llvm::wrap(llvm::PassManagerBuilder*)’
   14 | inline LLVMPassManagerBuilderRef wrap(PassManagerBuilder *P) {
      |                                  ^~~~
In file included from backports.cpp:4:
/usr/include/llvm/Transforms/IPO/PassManagerBuilder.h:259:34: note: ‘LLVMOpaquePassManagerBuilder* llvm::wrap(llvm::PassManagerBuilder*)’ previously defined here
  259 | inline LLVMPassManagerBuilderRef wrap(PassManagerBuilder *P) {
      |                                  ^~~~
make: *** [Makefile:187: tinygo] Error 2
make: *** Waiting for unfinished jobs....

[aykevl]

As you've noticed, TinyGo does not yet support statically linking against LLVM 11. This will probably be supported in the near future, when LLVM 11 becomes more widely available.

Related to that, you can already use LLVM 11 when using prebuilt binaries with #1056 (which is not yet merged).

[anatol]

I am actually building tinygo package for Arch Linux that started moving towards LLVM 11. Is it the only patch that is needed for LLVM 11? cc3ec22 Or more work is needed?

[aykevl]

That depends on whether you want to link statically or dynamically. Most distributions prefer dynamic linking and that should work fine with cc3ec22.

[anatol]

Thank you for your help @aykevl.

I applied cc3ec22 on top of 0.15.0 release and it gives me following build problem:

==> Installing missing dependencies...
resolving dependencies...
looking for conflicting packages...

Packages (5) compiler-rt-11.0.0-1  libedit-20191231_3.1-2  clang-11.0.0-1  lld-11.0.0-1  llvm-libs-11.0.0-1

Total Installed Size:  305.71 MiB

:: Proceed with installation? [Y/n] 
(5/5) checking keys in keyring           


.......


lib/cmsis-svd/data/Atmel/ATSAMR21E18A.svd
lib/cmsis-svd/data/Atmel/ATSAMR21E19A.svd
lib/cmsis-svd/data/Atmel/ATSAMR21G16A.svd
lib/cmsis-svd/data/Atmel/ATSAMR21G17A.svd
lib/cmsis-svd/data/Atmel/ATSAMR21G18A.svd
lib/cmsis-svd/data/Atmel/ATSAMV71J19.svd
lib/cmsis-svd/data/Atmel/ATSAMV71J19B.svd
lib/cmsis-svd/data/Atmel/ATSAMV71J20.svd
lib/cmsis-svd/data/Atmel/ATSAMV71J20B.svd
lib/cmsis-svd/data/Atmel/ATSAMV71J21.svd
lib/cmsis-svd/data/Atmel/ATSAMV71J21B.svd
lib/cmsis-svd/data/Atmel/ATSAMV71N19.svd
lib/cmsis-svd/data/Atmel/ATSAMV71N19B.svd
# tinygo.org/x/go-llvm
/usr/bin/ld: cannot find -lLLVM-10
collect2: error: ld returned 1 exit status
make: *** [Makefile:187: tinygo] Error 2
make: *** Waiting for unfinished jobs....
lib/cmsis-svd/data/Atmel/ATSAMV71N20.svd
lib/cmsis-svd/data/Atmel/ATSAMV71N20B.svd
lib/cmsis-svd/data/Atmel/ATSAMV71N21.svd
lib/cmsis-svd/data/Atmel/ATSAMV71N21B.svd
lib/cmsis-svd/data/Atmel/ATSAMV71Q19.svd
lib/cmsis-svd/data/Atmel/ATSAMV71Q19B.svd
lib/cmsis-svd/data/Atmel/ATSAMV71Q20.svd
lib/cmsis-svd/data/Atmel/ATSAMV71Q20B.svd
lib/cmsis-svd/data/Atmel/ATSAMV71Q21.svd
lib/cmsis-svd/data/Atmel/ATSAMV71Q21B.svd
GO111MODULE=off go fmt ./src/device/sam
src/device/sam/at91sam9g10.go
src/device/sam/at91sam9cn12.go
src/device/sam/at91sam9g20.go
src/device/sam/at91sam9cn11.go
src/device/sam/at91sam9g15.go

I do not see LLVM-10 anywhere in the checked out code, so it must be coming from tinygo.org/x/go-llvm dependency. Should this go library be linked with LLVM 11 instead?

[deadprogram]

The current dev branch now has this support merged. Please give a try from that branch.

[aykevl]

You will also need the llvm11 build tag to make this work (-tags=llvm11). Otherwise it will by default try to link to LLVM 10.

[anatol]

Hi @aykevl I am a bit confused now. The comments above state that LLVM11 support has been added in commit cc3ec22. But it turns out that some extra manual changes are required to the build process. Would it make sense to add this proposed -tags=llvm11 to the build system instead?

[aykevl]

We will likely switch the default from LLVM 10 to LLVM 11 in the near future, avoiding this issue. However, LLVM 11 may not have propagated everywhere so I'd rather wait a month or so before doing that.

To use the build tag:

go install -tags=llvm11

It appears that you're building LLVM from source instead of linking it dynamically, in which case using LLVM 11 is not yet supported (but that isn't necessary anyway as there is no need to match the distro LLVM version in this case).

[anatol]

I am actually trying not to build LLVM from sources an use llvm-11 from my system instead. I had to patch Makefile to disable building LLVM from sources, here is the patch https://github.com/archlinux/svntogit-community/blob/e0369803ed783f8f3f664d4a05fec999cdfb3c7f/trunk/disable_static_llvm.patch

This patch worked with LLVM-10, but does not work with LLVM-11. It look like I need extra patching.

Though I would personally prefer avoid patching sources. Is there a chance to make dynamic build configured by some build envvar instead? (the documentation only mentions static build https://github.com/tinygo-org/tinygo/blob/release/BUILDING.md)

[anatol]

By patching Makefile and adding -tags llvm11 to tinygo: build target I was able to compile the project with system LLVM11 library. The build seems fine. We are going to continue verification of tinygo+LLVM11 in Arch Linux testing repository. So my original question is resolved.

But having a simpler way to enable dynamic LLVM build without patching Makefile is still something desirable for us.

[aykevl]

Though I would personally prefer avoid patching sources. Is there a chance to make dynamic build configured by some build envvar instead? (the documentation only mentions static build https://github.com/tinygo-org/tinygo/blob/release/BUILDING.md)

The way to do that is to not use make, but instead call the Go toolchain directly. For example:

go build -tags=llvm11 -o build/tinygo
make gen-device

And to run the tests:

go test
make smoketest XTENSA=0

I've added XTENSA=0 above as the Xtensa backend (for esp32/esp8266) is not yet supported in upstream LLVM.

[anatol]

Could you please update the official build instructions https://github.com/tinygo-org/tinygo/blob/release/BUILDING.md ?? Once it is done I'll modify Arch package build as well.

[aykevl]

It's already more or less included in BUILDING.md:

TinyGo depends on LLVM and libclang, which are both big C++ libraries. It can
also optionally use a built-in lld to ease cross compiling. There are two ways
these can be linked: dynamically and statically. An install with go install is
dynamic linking because it is fast and works almost out of the box on
Debian-based systems with the right packages installed.

I'm not sure what you expect there to be? If you want a build with a dynamically linked tinygo binary, you can simply replace the tinygo binary in the build directory with the one produced by go build (or go install).

[zonghaishang]

Though I would personally prefer avoid patching sources. Is there a chance to make dynamic build configured by some build envvar instead? (the documentation only mentions static build https://github.com/tinygo-org/tinygo/blob/release/BUILDING.md)

The way to do that is to not use make, but instead call the Go toolchain directly. For example:

go build -tags=llvm11 -o build/tinygo
make gen-device

And to run the tests:

go test
make smoketest XTENSA=0

I've added XTENSA=0 above as the Xtensa backend (for esp32/esp8266) is not yet supported in upstream LLVM.

➜  tinygo git:(dev) go build -tags=llvm11 -o build/tinygo
# tinygo.org/x/go-llvm
../../../../pkg/mod/tinygo.org/x/go-llvm@v0.0.0-20200503224449-70c558526021/analysis.go:16:10: fatal error: 'llvm-c/Analysis.h' file not found
#include "llvm-c/Analysis.h" // If you are getting an error here read bindings/go/README.txt
         ^~~~~~~~~~~~~~~~~~~
1 error generated.
➜  tinygo git:(dev) brew install llvm
Updating Homebrew...
Warning: llvm 11.0.0 is already installed and up-to-date
To reinstall 11.0.0, run `brew reinstall llvm`

This has been resolved by compiling LLVM10