Update FAQ with clarifications on Elixir vs Gleam

[josevalim]

This is mostly for discussion, I am adding comments with more context.

[josevalim]

I removed this because I am not aware of any data to support this but perhaps you are aware of some? There is an argument that Gleam compiles faster, because the Gleam compiler emits Erlang source files, but I don't think that's an apples to apples comparison, because when you run a command such as gleam test, it would still require those files to be compiled into .beam files. A good test would be the time to compile and start an app that immediately shuts down.

[josevalim]

To give more context, an end to end compilation + test in Gleam is (I believe):

boot up rust -> parse gleam -> write .erl -> boot up erlc -> parse .erl -> write .beam -> boot up erl -> start tests

And end to end compilation in Elixir is:

boot up erlang -> parse elixir -> expand macros (may load a temporary .beam in memory) -> write .beam -> start tests

[lpil]

This is including the Erlang -> BEAM compilation too. Compiling Elixir is more challenging than compiling Gleam as Gleam is a simpler language, and Gleam's compiler performs faster than Elixir's due to using native code rather than the BEAM, so in like-for-like codebases Gleam consistently compiles faster.

Gleam also doesn't permit cycles between modules, so it has an easier job of avoiding slowdown in large codebases.

I removed this because I am not aware of any data to support this but perhaps you are aware of some?

I've measured! Historically I measured like-for-like, but in recent years I've only measured the diff between new and old versions of the same language to see how they have changed with each release.

If you have some newer data that shows things have changed then we can correct this to show the current state of play.

[josevalim]

Ok, that gives me a reference point! I asked Claude to create two projects with 100 single-function files, ran gleam test vs mix test in each and I got 0.5s for Gleam, 0.7s for Elixir, so I am more than happy to add it back!

I will try to add a reference point, if that's ok.

[josevalim]

I have done further work and benchmarks on this.

The reason why the Elixir compiler was slower in my benchmarks was because our macro expansion uses intermediate modules behind the scenes and we were deleting and purging those modules immediately after compilation. Elixir v1.19 had already postponed the purging of modules, because it goes through Erlang's code server, which is a single process and makes it a bottleneck. But deleting modules also goes through this same process and on my machine, with ten cores, that was showing up. This commit I just pushed to main also postpones it like we did with purging and now, for the scenario above, Elixir compiles as fast as Gleam.

Gleam also doesn't permit cycles between modules, so it has an easier job of avoiding slowdown in large codebases.

Elixir doesn't allow cycles between modules at compile time either. We do allow it at runtime, but that has no effect at compile time!

Gleam's compiler performs faster than Elixir's due to using native code rather than the BEAM, so in like-for-like codebases Gleam consistently compiles faster.

In the last several years optimizing the Elixir compiler, the compiler front-end was almost never the bottleneck, only the compiler backend (i.e. the Erlang compiler), which in this case is shared between Gleam and Elixir. I agree that native code will be faster but Gleam's architecture also requires invoking erlc and starting erl separately.

Therefore, instead of guessing, I decided to run further benchmarks. This time, instead of giving single-function modules, each module has 100 functions. This means more work for both compiler frontends, which would give more opportunity for Gleam's native frontend to get ahead of Elixir's, but the times were the same for both on my machine: ~0.8s.

I have published the code here: https://github.com/josevalim/langcompilebench. I have measured using Elixir main (soon to be v1.20-rc.2) and Gleam v1.14.0.

Edit: oh, it is also worth saying that Elixir v1.19 made some codebases compile 3x faster. So I don't doubt Gleam codebases were consistently faster in the past, but there has been a lot of progress on our side here. :)

[michallepicki]

If there's no benchmarks provided, maybe it's worth rewording this sentence to something less absolute? E.g.

We've found in our testing that Gleam compiles faster than Elixir.

I write Elixir day to day but I do love Gleam as well, both have their advantages and disadvantages, and I think both have a place in the BEAM ecosystem :) To expand on some of the already brought up points, and add a few more from my POV:

To me the feeling that Gleam compiles faster than Elixir is most apparent when modifying an existing codebase with an LSP in the editor. The Gleam LSP does not need to run the Erlang compiler for it to work, only part of the whole "pipeline" is needed to surface type errors etc. The Gleam LSP is super responsive and reliable, while with Elixir I am not using it (I'm either using Zed without LSP, or VSCode based editors with a hacky hard to set up tree-sitter based highlighter extension).

There is a tradeoff of the convenience of having metaprogramming features (as opposed to having to explicitly run some codegen tasks in Gleam) vs the overhead it adds on recompilations due to "wasted" repeated work even when the result is the same. And in a real world codebase where compile dependencies between modules aren't always perfectly minimal, this not only makes recompilation slightly slower without apparent reason in a unnoticed-for-months way, it sometimes causes serious problems, there's a reason why there's been so many blog posts and conference talks about reducing compile dependencies and using mix xref to debug such issues, I think most Elixir projects hit such issues eventually. In Gleam the compiler will just yell at you? :) (but it's also limiting code organization and may be considered less convenient, btw see also gfngfn/Sesterl#38 )

Similarly, Elixir has a convenient config system, but modifying the config will trigger whole project recompilation (understandably, hard to track what effect it could have without limiting the config functionality). Gleam? Most people probably use only runtime config with loading things like env variables or toml files at startup, no recompilation needed (but I do like the Elixir config a lot)

Maybe this one is silly but I think Gleam emits much less Erlang code than Elixir? Maybe I'm wrong but even a simple variable.dot_access in Elixir is a case with matching on the variable being a map, and try-rescue with an attempt to call a function in case the variable is a module. So it's likely that the time spent in the Erlang compiler part (sans the filesystem writing, BEAM startup, file reading and Erlang parsing overhead) is lower for Gleam. But also Gleam emits a lot of variable re-assignments, which sometimes hits slow paths in the Erlang compiler, I saw issues on the Erlang/OTP repo for some pathological cases (<3 to the OTP team for fixing them btw).

AFAIK there's still a lot of performance gains left on the table in the Gleam compiler. Most importantly I think the Gleam compiler doesn't make much use of parallel compilation yet? Some of the module graph recompilation optimizations @josevalim mentioned could also apply in Gleam. Not to mention any lower level things in Rust, like leveraging mutability more, organizing memory layouts for cache efficiency, reducing dynamic memory allocations with arenas... Probably more optimizations will come once there's need for them for working bigger real world codebases, but having a simpler compiler codebase is also worth preserving

The Elixir compiler is not slow overall, and it is true that a lot of good optimization work was put into it. Some upcoming typechecker features like exhaustiveness checking might probably slow it down a bit but we'll see by how much and how it turns out, I know there's no guarantees on some of the typechecker features being added at all, maybe it won't be feasible. Fingers crossed though, I enjoy the Gleam typechecker and would love more type based checks in Elixir as well

To summarize, I think Gleam compilation is probably often faster because it's a simpler language, and (also thanks to some limitations or not having some functionality) the compiler can be implemented in more efficient Rust. The Elixir compiler architecture is better for providing its metaprogramming features, but also for performance in some ways, and the Elixir compiler is doing some optimizations that Gleam doesn't do (yet?). The Gleam typechecker is based on a standard, proven to be fast enough HM type system that is making it clearly a statically typed language, while the Elixir typechecker is also an impessive gradual typing research project, bringing as much type safety as possible, but keeping the language ergonomics dynamic.

[hayleigh-dot-dev]

Let's undo this change in this PR so we can merge the other changes until there's new benchmarks.

Let's be respectful of this and move the discussion to another venue so we can get the other changes merged and deployed 🙏.

[josevalim]

To me the feeling that Gleam compiles faster than Elixir is most apparent when modifying an existing codebase with an LSP in the editor. The Gleam LSP does not need to run the Erlang compiler for it to work, only part of the whole "pipeline" is needed to surface type errors etc

Excellent! But that doesn't mean Gleam compiles faster than Elixir, it means Gleam can provide much (if not all) of its LSP features faster than Elixir because it requires only the front-end while Elixir has to either compile or interpret code. If the claim was that "Gleam LSP is faster than Elixir LSP", I would have a hard time disagreeing with that.

there's a reason why there's been so many blog posts and conference talks about reducing compile dependencies and using mix xref to debug such issues

I'd say this is partially missing the argument of the diagrams above. The reason why we can use mix xref in Elixir is because Elixir makes a distinction between runtime and compile-time dependencies. Gleam AFAIK doesn't, everything is compile-time, so Elixir has an opportunity to improve over Gleam's baseline.

Alternatively, you could imagine an Elixir version where every call between modules is a compile-time dependency. This means we wouldn't have cycles, xref's would be gone and there'd be no need to talk about it, but now recompilations are way more frequent.

To summarize, I think Gleam compilation is probably often faster because it's a simpler language, and (also thanks to some limitations or not having some functionality) the compiler can be implemented in more efficient Rust.

If that's the case, then we should be able to have benchmarks or some metric that shows it! And I did provide such benchmark initially, but it will be outdated by the next Elixir release.

My whole point is: let's move beyond what we think is happening, towards what we can actually measure. If Gleam can be parallelized, better utilize memory layouts, then let's have a shared place where we can see those improvements! It is clear we all have opinions on why we think one is faster than the other and I tried my best to bring some actual data into the discussion, otherwise it will only lead to endless speculation.

[michallepicki]

Let's undo this change in this PR so we can merge the other changes until there's new benchmarks.

Let's be respectful of this and move the discussion to another venue so we can get the other changes merged and deployed 🙏.

But also it may feel disrespectful towards Elixir to keep this statement line without providing benchmarks to back it up

[josevalim]

Sorry @hayleigh-dot-dev, I didn't see your reply before my comment above. As said before:

You are welcome to close this pull request or edit it as you seem fit. :)

I have enabled permission to edit the pull request and you folks are welcome to do as you prefer. I am unsubscribing from the issue, so you won't hear from me anymore, but I really hope there will be some initiative to bring visibility into this. Apologies for any misunderstanding.

[josevalim]

Suggested change

	and Erlang code. Gleam shells require writing Erlang code.
	and Erlang code. Gleam shells require writing Erlang (or JavaScript).

?

[lpil]

We don't have any tooling for shells in JavaScript today, just the Erlang one.

[josevalim]

Thank you @giacomocavalieri and @lpil! I have answered all of my own questions, I am happy with the above, but if you want me to change anything, just let me know!

(it is also very cool you support -moduledoc/-doc, that made my day!)

[giacomocavalieri]

Yeah I remember as soon as I saw the -doc/-moduledoc annotations at one Code BEAM talk I had to add support for it! Really cool stuff