What about adding async/await?

[pwtail]

I am sure I'm not the first to ask about it, but I searched existing discussions by both async and await - nothing!

Imho, this is a good choice for a typed language, because, by annotating a function with "async" keyword, we can state the function is of an async type.

Also, having colored functions is less of an issue for async-only runtimes like Erlang or Javascript. It is more painful in languages like Rust, which allows both blocking and async i/o, and people can't easily reuse the code between those two.

If we try to use the "use" keyword for that purpose, it's like we get all the drawbacks without the benefits, because now our functions are colored, and we still need to pass callbacks as arguments.

What are the drawbacks / challenges to this?

[lpil]

Hello! Thanks for the suggestion!

JavaScript is like Rust in that it has the function colouring problem. Erlang is like Go in that its concurrency doesn't result in function colouring, so you don't have to worry about whether a function of synchronous or not. Because of this and our commitment to interop Gleam has coloured functions when compiled to JavaScript, but not when compiled to Erlang. This is an annoyance as certain kinds of code can't be shared between the two targets, but it is a good trade off because we get colourless functions when possible, without sacrificing interop or performance on the JavaScript target.

Having async/await in the language would unfortunately not remove the function colouring problem, instead it would likely make it worse as the future or promise type would need to be included into the core of the language, making it possible for function colouring to infect the Erlang target too. As well as all the usual problems of function colouring it would also make interop with Erlang/OTP impossible, and we would lose its actor model.

async/await is a convenience to write promise.then(continuation) in a procedural looking fashion. It's a nice UX, but we already have this! use is a superset of async/await syntax and has all of its advantages, but also additional advantages of being applicable to other types and functions beyond the promise combinators.

[lpil]

We have this today in the task module! https://hexdocs.pm/gleam_otp/gleam/otp/task.html

[mindplay-dk]

@lpil neat! :-)

this isn't listed in the standard library docs though? and nothing about asynchrony in the language tour either.

seems like this would be a common enough question/concern for any language, so maybe think about surfacing that? :-)

[lpil]

It's not part of the standard library, and concurrency is not part of the core language so it's not documented in the core language documentation. We'll continue to add more docs, but this won't be in either of those.

[inoas]

in stdlib's readme there could be a link to a page on gleam.run where one finds standard tools for:

• concurrency
• stdout/stderr io
• file io
• network io
• audio io
• video io
• mouse io
• keyboard io

[lpil]

We have! It's the packages site.

[pwtail]

Thank you, @lpil

What I meant is not introducing the promise type. await can be seen as not awaiting a value, but as calling an async function. Let's suppose you cannot instantiate a promise without immediately awaiting it, that would result in a compiler error. Since there is not a special type for a regular function call, there probably shouldn't be Promise type.

It's like you call a function with special color that can call both "special-color" functions and regular functions. For Erlang those two kinds of functions could be pretty much the same.

[lpil]

Oh I see! This is a very interesting idea!

I experimented with this in the past (and hinted at it in one of the JS release posts) but didn't make much progress. There are some problems we would need to overcome.

1. If async functions are auto-awaited how do we perform work concurrently?

We would need a way to disable auto-awaiting to do this:

let x = async_function()
let y = async_function()
await Promise.all([x, y])

2. How do deal with the asynchronicity of functions depending on their arguments?

For example the apply function (fn(f, a) { f(a) }) is async if f is async. The code we generate needs to be able to handle both sync and async functions.

We could make everything async, but this would have a large performance cost and make FFI very challenging.

We could perform a runtime check for a promise and then switch between async and sync versions of the generated code, but this would also have a performance cost and result in a big increase in code side. There may also various constructs which may not be easy to implement both ways, though I've not thought about this too much.

[pwtail]

We can't do concurrency with what I propose, that's right. Kind of aligns with the actor model, doesn't it? Actors do things sequentially.

To do concurrency we would need some help from the runtime - some notion of the logical process we are currently in. In the case of Javascript that could be the parent coroutine, in the case of Erlang - the process.

Something like this can make things execute concurrently:

async fn bring_happiness() {
  use <- await processes(2)  // this is provided by runtime
  await happiness()
}
// made 2 people happy

What I propose is to annotate i/o related calls with async and await in the gleam code everywhere, so we won't need to guess which function is async. We can do what we want with this information, including just ignoring it.

Or maybe I just didn't get what you meant in the second part.

[lpil]

We can't do concurrency with what I propose, that's right. Kind of aligns with the actor model, doesn't it? Actors do things sequentially.

In async/await the default is to not block the current task, instead giving you a value you can block on as desired. With actors the default is to block the current task, but you also get a way to instead not block, and likely also some wrapper to get the value back similar to async/await. In gleam_otp this last bit is the gleam/otp/task module. We would need some way to do this if we added an auto-awaiting runtime on JavaScript.

What I propose is to annotate i/o related calls with async and await in the gleam code everywhere, so we won't need to guess which function is async.

The challenge is not so much identifying async code (we can do it with an effects tracking system), the issue is that higher order function may be async or not depending on their arguments.

pub fn loop_until(f: fn() -> Bool) -> Nil {
  case f() {
    True -> Nil
    False -> f()
  }
}

This function is sync if f is a sync function, but async if f is an async function. What would the generated JavaScript for this function look like? We can't always make this an async function because it would mean it's no longer possible to construct a sync function in Gleam any more unless it doesn't use any higher order functions.

[pwtail]

@lpil I actually don't see auto-awaiting as a solution and don't have good suggestions for the example with a higher-order function.

I understand why you want to go with the auto-awaiting: async-await is not needed by the Erlang target. And I think I have a good idea on this: it can be made optional!

Async-await can be used for code that is common between the targets: the standard library, for example. The code targeting BEAM will be able to omit async-await (and even advised to do so). The async-await annotations can be just like types - optional.

So here is the picture I can imagine:

• the standard library uses async-await
• the code targeting BEAM can omit async-await
• the code targeting js uses async-await

What are the downsides to this?

Btw, speaking of the semantics of async-await:

In async/await the default is to not block the current task, instead giving you a value you can block on as desired

Yes, usually you can provide a custom Promise or Awaitable to wait on. I don't agree it is the default. The default is still the sequential execution: because it is the simplest and problem-free (concurrent execution brings a bag of problems even for a one-threaded runtime). By not providing the promises we just make things simpler and the rules stricter. Which can only be a good thing.

[lpil]

Unfortunately that would mean that many common functions such as list.map would not be implementable. If we have auto-awaiting we need to be able to solve this problem of higher order functions being sometimes async.

[pwtail]

cannot we check the type of a function with a case expression?

[lpil]

Could you give an example of what you are envisioning there?

[pwtail]

case f {
    Async(f) -> handle_async(f)
    f -> handle_regular(f)
}

On the Erlang target the first clause should never match.

[lpil]

Would the user be writing that Gleam code? If so it means that they would have to write it for every single function call as there's no way for the user to tell if a function may sometimes be async or not, unless it is in the type of the function. What would happen if they forget to do so?

What are the definitions of the two functions being called there?

[pwtail]

Well, not always does user need to write universal code like list.map. When user needs to do so he may write

MaybeAsync(f) = f

or specify that in the function definition

fn map(f: MaybeAsync(fn(x: Int) -> Int)) -> Int {}

If user is targeting Erlang he doesn't have to bother, if he is targeting js, he may specify the function has to be of async type:

fn some_func(f: async fn(x: Int) -> Int)) -> Int {}

[lpil]

That's just regular coloured functions, you can do this today in Gleam on JavaScript. 😁