Install and test embedding examples

[tkelman]

I'm starting to think we should install examples/embedding.c and examples/Makefile along with examples/*.jl and test on Travis that it keeps working in an installed version of Julia. For this to make sense, the makefile would need to be modified quite a bit (or a new one autogenerated?) to be standalone.

Related to #8428, cc @nalimilan

[waTeim]

Ok seems like I'm the guy to do this, as I'm likely the one that would benefit most from it. Is there any sort of introductory information, etc?

[tkelman]

Our current make install is pretty much roll-our-own in the top-level makefile. Step one is make sure the current embedding example and Makefile are working properly. If that's all fine, then it's a matter of trying to decouple examples/Makefile from Make.inc to make it usable in a non-source build of Julia. There are various flags and paths that we could probably just cat into the start of the installed copy of examples/Makefile.

I've seen this done really well in a few autotools projects, but I think for this isolated case we can do something simpler without needing to redo the rest of the build system. If you happen to have done Pkg.add("Ipopt") on a Linux machine I can point you to what Ipopt's example makefiles look like, which I think are a good model for this.

[waTeim]

The critical question on OS/X and Linux is where are the header files (for -I) and where is libjulia.so (for -L and -rpath). Is that all occurring assuming it's been deployed (like in /usr/local?), or is the directory known? On Windows, I don't know. Is this something that will be part of Travis and how does it get invoked? It would be pretty easy to pull together an autotools config that will look for Julia in a bunch of standard places, but problem with autotools is on Windows, doesn't that imply cygwin?

[tkelman]

So, the top-level makefile that does make install will know where everything's going, and can put that information into the installed version of the example's Makefile. We can add some lines to .travis.yml to cd into the installed copy of the examples folder, run make, and verify the output. I wasn't going for "looking for Julia in a bunch of standard places," since it'll only ever be in a standard place if it's packaged by a distribution. It would be nice to allow this to work for those distributions, so people who get Julia from a distro package can have an installed example embedding Makefile that will have the correct paths for their system. Our binary installs that aren't from distro packages, either the Linux tarball or the OSX/Windows installers, may end up at any location. So hardcoding absolute paths probably isn't a good idea, but if we can make everything relative-path based and self-contained I think it could be made to work.

[waTeim]

If autotools, there there would eventually be some variable INSTALLDIR which could be overridden on the command line to point to the exact path during compilation, otherwise it searches, autotools style. For reference, here's an approach to do that definitely not autotools style, but gives an idea about where to look. Maybe some simplified version of this in a bash-autotools way?

[tkelman]

We have a bunch of pseudo-autotools-ish variables like that in our makefiles already. For install-time logic it probably makes sense to do the file modifications as part of the install target in the top-level Julia makefile.

Instead of doing any searching through system paths, I think we can set a default (relative?) path based on the normal layout of an install. Ideally set up in such a way that if distributions customize the installation layout, that would get reflected in the default path we put in the embedding makefile.

[waTeim]

I think there are at least a couple distros like that. pg_config and pythonX.Y-config come to mind. How about in addition to the julia executable there's a program called julia-config? There are a few holes, though, what if Julia is installed multiple times in different locations (simultaneously 0.3 and 0.4) then what? Actually, looks like the python-guys thought of that. How to find julia-config? Maybe assume the user will install in such a way so that the Julia he wants to use will have the corresponding julia-config first in PATH.

[tkelman]

That might not be a bad idea, llvm and pcre have the same sort of thing. There's also the classic, but marginally difficult-to-use-on-Windows, option of shipping a julia.pc pkg-config file. Related: #8650

[tkelman]

I'm wondering out loud here if, aside from any modifications to the embedding example, the more complicated aspects of this can be done out in a central package, PkgConfig.jl or EmbeddedConfig.jl or whatever, to collect convenient scripts, code snippets, etc that would be useful across multiple targets for embedding Julia - you with your node-julia work, pyjulia, the various efforts for calling Julia from R and Matlab, etc. Right now it seems like you're all reinventing the same bits and pieces, some of which is inevitable to fit with the different build systems each of those environments use, but maybe cross-target collaboration could save everyone some work in the long run. If you're calling Julia from some other language, it's probably safe to say you have Julia installed, right?

[waTeim]

Yea right now we're all a bunch of kangaroo's jumping around. We'd all jump to a common thing if it fit our needs.

I'm wary of something like EmbeddingConfig/PkgConfig.jl unless it was guaranteed part of base or Pkg.update() is not required as It needs to respond quickly and never fail. Putting stuff in a Pkg allows for the flexibility of the meta information contained by the package to change when the Julia install does not change. Would that be useful?

[tkelman]

I guess the argument elsewhere was "if it needs to be kept up-to-date with changes to base, it should be in base" but I'm also not sure how often these things will change. The slowness of Pkg.update() is annoying but will hopefully get better once the libgit2 transition is done? An alternative would be to make it a script or set of scripts in contrib (that get installed with binaries) the way build_sysimg.jl was just done.

[ViralBShah]

Should we target this for 0.3.4, or is it too close and go for 0.3.5?