Adding support for the Julia language

[Nosferican]

Planning to take a look at it in case anyone else has the same idea, we can be aware of the efforts.

[StoneT2000]

I'm not aware of anyone doing this so feel free to work on it! You can look at the C++ simple kit for the basic structure we would like for a kit.

[Nosferican]

I might push a draft soon to get some feedback but I have mostly followed the Python kit. The object-oriented makes it slightly awkward so minor changes to use multiple dispatch but seems to be moving along. I have seen minor differences from the implementation and the public API definition but looking at the kit helped me figure things out.

[Nosferican]

I have opened as a draft PR, so it isn't ready for a complete review, but would regardless appreciate some comments on how to start testing a new bot/port. I guess I would need to get a custom version of the CLI?

[StoneT2000]

Since Julia can be compiled, as long as you can compile the bot code into a file, eg main.out, then the CLI tool will try to execute it by running ./main.out, so no customization necessary

So a test command would be

lux-ai-2021 main.out main.out --replay=out.json

[StoneT2000]

Our team also is not fluent with Julia so if you can document the differences with the python API that would be great, and then feel free to implement however you like that best suits julia programmers.

[Nosferican]

I think I understand the last component of the pipeline now. Probably the easiest solution would be to use PyJulia...
Something like

import julia
julia.include("file.jl")
# other logic
commands = julia.agent(observations)
# other logic

[StoneT2000]

Can you not directly execute compiled Julia code? Or does it not compile to machine code

[Nosferican]

One could potentially use PackageCompiler (see this section) to create an app that can be sent and run on other machines without Julia being installed on that machine. However, at least when looking at the Kaggle submission / environment and the Docker file it seems to be a lot easier to:

1. Install Julia calling a file like install_julia.sh (might need to chmod first)

#!/bin/bash

JULIA_VERSION="1.6.3"

JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
wget -nv $URL -O /tmp/julia.tar.gz
tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
rm /tmp/julia.tar.gz

pip3 install --quiet julia

2. Call Julia from Python with these lines

import julia
julia.install()

from julia.api import Julia
jl = Julia(compiled_modules=False)

from julia import Base
from julia import Main

3. You can then add your starter/code and agent implementation like

from julia import Pkg
Pkg.add(url = "https://github.com/owner/LuxAI.jl", rev = "main") # Repo with agent implementation

Main.eval("using LuxAI")
from julia import LuxAI

4. Interact with the Julia process

...
commands = LuxAI.commands(observation["updates"])
...

I guess once I figure out the last kinks in the starter code either way would work.

[StoneT2000]

Ah so compiled julia code still requires some kind of julia runner, in which case the pyjulia looks like a better option.

[Nosferican]

As compiled code, it wouldn't require Julia, but at least for development, it is likely that just calling Julia from Python for that one line in that file to be an easier experience. I am testing making the executable files and as an option and basically people could do the same set up at the end regardless. I was mostly thinking of making it easier for folks to dev / work with the code. That way, they don't need to create the executables but have the option.

[StoneT2000]

Sounds great! I only had concerns over potentially speed and I wasn't sure how pyjulia works (e.g. is it transpiling? is it 100% accurate?)

[Nosferican]

I have had some experience calling Python and R from Julia as well as Julia from R and Python... Not too much of that anymore these days. In general, it's quite useful and works well in most cases with rare exceptions. In this case, I imagine it would be best case scenario since the only thing being passed around is a String.

[Nosferican]

I think I am about done with the startkit... Will push soon. I am building the test suite using

replay_game = JSON3.read(read(joinpath("data", "PySimpleS0.json"), String))

replay_game being the output of

lux-ai-2021 --seed=0 --statefulReplay kits/python/simple/main.py kits/python/simple/main.py

Using that to get verify the state of the game at initiation and how it progresses.
I am also using

observations = readlines(joinpath("data", "inputs.jsonl"))

which isn't technically a JSONL file but is just a capture of the read_input() lines the Python agent got during that game.

Inspecting that game, the first observation that is received from the API has lines

 "r uranium 15 15 341"
 "u 0 0 u_1 2 8 1 60 0 0"
 "u 0 1 u_2 13 8 1 60 0 0"
 "c 0 c_1 0 23"

meaning each player has their initial worker. Based on
https://github.com/Lux-AI-Challenge/Lux-Design-2021/blob/512357eda1fe22600c6281f31f88b386044ce8b6/kits/python/simple/lux/game.py#L55-L65

Worker u_1 has a cooldown at 1 and thus can't take an action. When looking at the commands sent to the API,

{
   "command": "m u_1 e",
   "agentID": 0
}
 {
   "command": "m u_2 w",
   "agentID": 1
}

Worker u_1 did actually take an action to move east. I might have skipped something so wanted to make sure my understanding of how to use the API components thus far was correct. Thanks!

[ziong]

@Nosferican Thanks for your time and effort to create the julia kits.
btw, I have a question, how to deal with the Julia Packages issue?
does it has a list of preinstalled julia packages? but how about if extra packages / different versions are required?

I am not sure but is it possible to create a compile.sh which let us to install required packages?
but it has a prerequisite that the container has the internet access / an internal package repository for downloading the packages.

or can we create a custom DockerFile together with submission to customize the container image that running the bot (agent)?

and for using pyjulia approach, seems that command call via python (main.py) approach, e.g. in the java kits, is more intuitive to me. any benefit using pyhulia instead of command call?