CLI to run Gleam Hex packages outside of a local Gleam package

[lpil]

Similar to npx.

We likely need to know when to recompile any Erlang bytecode as when the VM version changes the bytecode may no longer be valid. manveru on discord suggested this way of getting the Erlang version.

❮ time erl -noinput -boot no_dot_erlang +A0 +sbtu -mode minimal -noshell -eval 'io:format(standard_io, "~s~n", [erlang:system_info(system_version)]),halt()'
Erlang/OTP 25 [erts-13.2.2.6] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:1] [jit:ns]

erl -noinput -boot no_dot_erlang +A0 +sbtu -mode minimal -noshell -eval   0,45s user 0,12s system 453% cpu 0,127 total

What should the CLI be?

[samifouad]

i like this idea. would it be a separate command?

eg. gpx ____
(short for gleam package execution)

or something part of the main binary? eg. gleam x _____

also, a command i've been using to get the erlang version info is:

erl -eval 'io:format("OTP: ~s, ERTS: ~s~n", [erlang:system_info(otp_release), erlang:system_info(version)]), halt().' -noshell

output:
OTP: 26, ERTS: 14.2.2

[lpil]

It would be in the same binary. Perhaps something like gleam run --package blah

[PgBiel]

Personally, I think the gleam run command is pretty good for this, since it has a good bit of customizability (target, runtime, arguments) which is sometimes needed when running things. Or, at least, a subcommand for this should inherit gleam run's syntax.

[han-tyumi]

I like the idea of simply using run with an additional option.
I was thinking these may be more clear to the intent.

gleam run --hex <package>
gleam run --remote <package>

Coming from the JS world - mainly Deno..
Perhaps using a specifier before the package name could be an elegant way to denote that we want to run a non-local package.

gleam run hex:package

[lpil]

gleam run hex:package would be passing hex:package as a command line argument to your program, so that's not an option here.

[enkerewpo]

So maybe we need some wrapper like this:

1. gleam run --package foobar, then we fetch the package from hexpm and store the source code somewhere(under ~/.gleam maybe)
2. maintain something like local package management db for cli package cache
3. each time we run the gleam run --package foobar, we first get the “tripe” of current running Erlang vm. If there were no old tripes(first compile) or the new one won’t match the old one, we recompile and run it with the new triple cached along side with the source; else we just run the old beam binaries

[enkerewpo]

related discussion: https://discord.com/channels/768594524158427167/1322141464578494496
#4105

[enkerewpo]

I gathered some materials on this problem:

https://www.erlang.org/docs/26/man/escript
https://github.com/lpil/gleescript/blob/main/src/gleescript.gleam
https://www.erlang.org/doc/apps/erts/escript_cmd.html#bundling-escripts
erlang/rebar3#1800 -> Could rebar3 bundle escript with an Erlang runtime system to make it self-sufficient ?
erlang/rebar3#2366 -> [Question] How to make a "escript" file to a binary executable?

and I also take a look at @lpil 's gleescript tool which will pack all built BEAM code of a gleam project into an escript:

gleam add gleescript
gleam build && gleam run -m gleescript # e.g. will generate an escript `gleewhois`

% in erlang shell
2> escript:extract("gleewhois",[]).
{ok,[{shebang,default},
     {comment,[]},
     {emu_args,"-escript main gleescript_main_shim -env GLEESCRIPT_MAIN gleewhois"},
     {archive,<<80,75,3,4,20,0,0,0,8,0,124,75,35,90,165,219,
                87,151,204,24,0,...>>}]}

Originally, escript(s) are for small erlang code that can directly run as source code instead of having to build the BEAM code to run. So the packed escript gleewhois will not likely to run on different erlang environments beacuse of the packed binary codes.

Actually according to https://www.erlang.org/doc/apps/erts/escript_cmd.html#bundling-escripts, it says bundle escript(s) with an Erlang runtime:

It is possible to bundle escript(s) with an Erlang runtime system to make it self-sufficient and relocatable. In such a standalone system, the escript(s) should be located in the top bin directory of the standalone system and given .escript as file extension. Further the (built-in) escript program should be copied to the same directory and given the script's original name (without the .escript extension). This will enable use of the bundled Erlang runtime system.

The (built-in) escript program first determines which Erlang runtime system to use and then starts it to execute your script. Usually the runtime system is located in the same Erlang installation as the escript program itself. But for standalone systems with one or more escripts it may be the case that the escript program in your path actually starts the runtime system bundled with the escript. This is intentional, and typically happens when the standalone system bin directory is not in the execution path (as it may cause its erl program to override the desired one) and the escript(s) are referred to via symbolic links from a bin directory in the path.

But I think this "bundling" does not fix the above problem, it's just some erlang runtime selection mechansim. If the BEAM code doesn't match, I think the escript cannot run anyway.

ideas

1. I wrote a prototype of gleam CLI apps package management: https://github.com/enkerewpo/gleam-pkg, which manages stuff under ~/.gleam_pkgs, and I store the source code of downloaded gleam package from hexpm inside it.
2. maybe for each download gleam source code, add the gleescript deps and build a single escript packing all BEAM codes. (gleescript itself is actually a CLI app, but we have to add deps in a project to run it for now)
3. since the package manager for global CLI apps should have some "database" to store all the infos, we can store the compiled BEAM codes' erlang version and env infos for each packed escript(since according to escript documentation, there's no erlang version info embeded in the escript itself)
4. then generate a wrapper shell script to match the erlang version first, if everything is fine then we run the escript
5. if the erlang installed on user's machine does not match, trigger gleam to generate a new one with updated erlang version infos

[enkerewpo]

or instead of managing the Erlang versions in a centralized db, just embed it into escripts’ section like the comments will probably be better

[enkerewpo]

commit: enkerewpo/gleam-pkg@f2c358b

I implemented something like this:

1. auto generate a wrapper shell script that embed erlang version when building the gleam pkg (COMPILED_ERLANG_VERSION). This string is obtained by gleam-pkg's code when building the pkg:

let output = erl_eval(
    &"io:format(standard_io, \"~s~n\", [erlang:system_info(system_version)]).".to_string(),
)?;
let erlang_version = output.trim();
println!("Erlang system version: {}", erlang_version);

2. the binary escript(generated by gleescript) is encoded in base64 and stored in this auto-generated wrapper shell script.
3. after the script tests that the compile-time erlang version matches currently running erlang version, it extract the binary escript and run it

[lpil]

Why base64 encode the escript? That seems like it would be an unnecessary overhead. The program itself could do the check before running the main function, or your Rust program could do it before executing the escript.

[enkerewpo]

yep the base64 introduces overhead for binary extraction. Moreover I find it really slow to run an one line erlang evaluation which cold boots the erl, that’s another overhead…

[enkerewpo]

So maybe we can let the rust pkg manager to boot every escript while spawning aux threads to check Erlang env, whether pkg should be upgraded to latest version, etc. thus reducing the overhead when everything is fine(checkings done in other threads

I think most of the time users won’t change their Erlang env after the first compilation so “everything is fine” would be probably more likely to happen

[lpil]

Moreover I find it really slow to run an one line erlang evaluation which cold boots the erl, that’s another overhead…

It would be the first line in the Erlang program, not a second Erlang program that would be booted beforehand.

I think most of the time users won’t change their Erlang env after the first compilation so “everything is fine” would be probably more likely to happen

This is definitely not true. Gleam developers will be changing their Erlang version very frequently.

[enkerewpo]

It would be the first line in the Erlang program, not a second Erlang program that would be booted beforehand.

agreed🥹