headless anonymous function (->) syntax

[rapus95]

Since #24990 stalls on the question of what the right amount of tight capturing is

Idea

I want to propose a headless -> variant which has the same scoping mechanics as (args...)-> but automatically collects all not-yet-captured underscores into an argument list. EDIT: Nesting will follow the same rules as variable shadowing, that is, the underscore binds to the tightest headless -> it can find.

Before	After
lfold((x,y)->x+2y, A)	lfold(->_+2_,A)
lfold((x,y)->sin(x)-cos(y), A)	lfold(->sin(_)-cos(_), A)
map(x->5x+2, A)	map(->5_+2,A)
map(x->f(x.a), A)	map(->f(_.a),A)

Advantage(s)

In small anonymous functions underscores as variables can increase the readability since they stand out a lot more than ordinary letters. For multiple argument cases like anonymous functions for reduce/lfold it can even save a decent amount of characters.
Overall it reads very intuitively as start here and whatever arguments you get, just drop them into the slots from left to right

      -> ---| -----|
            V      V
lfold(->sin(_)-cos(_), A)

Sure, some more complex options like reordering ((x,y)->(y,x)), ellipsing ((x...)->x) and probably some other cases won't be possible but if everything would be possible in the headless variant we wouldn't have introduced the head in the first place.

Feasibility

1. Both, a leading -> and an _ as the right hand side (value side) error on 1.5 so that shouldn't be breaking.
2. Since it uses the well-defined scoping of the ordinary anonymous functions it should be easy to
   2a) switch between both variants mentally
   2b) reuse most of the current parser code and just extend it to collect/replace underscores

Compatibility with #24990

It shouldn't clash with the result of #24990 because that focuses more on tight single argument very tight argument cases. And even if you are in a situation where the headless -> consumes an underscore from #24990 unintentionally, it's enough to just put 2 more characters (->) in the right place to make that underscore once again standalone.

Further Explorations

This proposal can optionally be combined with #53946.
Additionally, the following links to comments further down explore different ideas to stretch into, all adding their own value to different parts of the ecosystem.
Alternative explorations: #38713 (comment) #38713 (comment)

[StefanKarpinski]

After that long, inconclusive debate, I think I've also come to the conclusion that having an explicit marker is better. Headless -> is the "natural" marker for this, so there you have it — it's simple and unambiguous. It even leaves room for letting Array{_, 3} be a shorthand for Array{<:Any, 3}. We would have to decide what to do in cases like -> (_, Array{_, 3}), but I would argue that it would probably be best to just make that an error and not allow _ in type parameter position inside of a headless lambda, which would need to be decided when implementing this, since otherwise making it an error after this feature is implemented would be a breaking change.

[StefanKarpinski]

To clarify, the question is which of these -> (_, Array{_, 3}) would mean:

• x -> (x, Array{<:Any, 3})
• (x, y) -> (x, Array{y, 3})

Both could potentially make sense. My suggestion is to make it an error and force the user to either use a normal lambda or not use _ as a type parameter. Alternatively, we could say that _ always "binds" to the tightest thing it could. That's also a consideration in the presence of nested headless lambdas. For example, it seems only sensible to interpret -> (_, -> _ + _) as meaning x -> (x, (y, z) -> y + z), so you could make the same case for -> (_, Array{_, 3}) that the _ as an argument to Array should mean Array{<:Any} since it's innermost.

[yurivish]

Sure, some more complex options like reordering ((x,y)->(y,x))

One solution is to say that inside of a headless anonymous function _n refers to the nth argument. So (x, y) -> (y, x) would be written as

-> (_2, _1)

There is precedent for this in Clojure:

The function literal supports multiple arguments via %, %n, and %&.

#(println %1 %2 %3)

and Mathematica:

#n represents the n ^(th) argument.

In[1] := f[#2, #1] &[x, y]
Out[1] = f[y, x]

[rapus95]

Alternatively, we could say that _ always "binds" to the tightest thing it could.

that's exactly what I meant when saying

automatically collects all not-yet-captured underscores into an argument list.

so yes, I'd clearly be in favor of making them bind the tightest.
Regarding the case of parametric underscore in headless lambda, I'd propose to already special case that in the parser (or wherever that belongs to 🙈) but make it error for now. That way we're free to add a proper rule once we've found a good solution without being breaking.
Though, right now I'm thinking about the following idea:

sugar	expanded
->(_, Array{_, 1})	x->(x, Array{<:Any, 1})
->(_, Array{<:_, 1})	(x,y)->(x, Array{<:y, 1})

for types the 2nd approach should work in the most cases since <:LeafType == LeafType for leaf types while one rarely needs actual abstract types as fixed parameters (and even then we have AbstractType <: AbstractType)
The drawback is that this approach wouldn't work for bitstypes like integers since <:1 is undefined iirc.
BUT! once we have something like Array{String,::Int} to denote the 2nd parameter to be an integer, we might be able to allow ->Array{String, _::T} as (x::T)->Array{String, x}

[rapus95]

regarding _1,_2 to denote the argument order still makes me have mixed feelings. On the one hand it reduces the readability since it's no more simply "put in from left to right" and on the other hand it doesn't save much anymore since now you need 2 characters to denote a single variable. a normal lambda would need 3 characters (definition, comma in argument list and usage)

[mcabbott]

The other advantage of numbering is that it lets you re-use the same argument:

sort(xs, by= -> _.re + _.im/100)  # x -> x.re + x.im/100
sort(xs, lt= -> _1.re < _2.re)    # (x,y) -> x.re < y.re

Edit -- You could argue that the first line only saves one character, and the second 3 not 5. But what it does still save is the need to invent a name for the variable.

Re-using the same symbol with different meanings in an expression seems confusing to me. Is it ever actually ambiguous? (The order of symbols in +(a,b) and a+b differ, with the same Expr, but _ will never be infix. So perhaps that cannot happen?)

Edit' -- Not the most convincing example, but notice that 1,2,3 occur in order in the lowered version of this, as Iterators.filter puts the condition before the argument, but the comprehension puts it afterwards:

dump(:(  [f(x,1) for x in g(z,3) if h(x,2)]  ), maxdepth=10)

[yurivish]

regarding _1,_2 to denote the argument order still makes me have mixed feelings. On the one hand it reduces the readability since it's no more simply "put in from left to right" and on the other hand it doesn't save much anymore since now you need 2 characters to denote a single variable. a normal lambda would need 3 characters (definition, comma in argument list and usage)

You can continue to use _ for that; the numbered underscores are just a syntax for referring to arguments by their position.

In Mathematica's and Clojure _ always refers to the first argument, and numbered underscores are usually used for the second/third/... arguments.

So @mcabbott's first example works as-is and the second example can also be written as

sort(xs, lt= -> _.re < _2.re)    # (x,y) -> x.re < y.re

[rapus95]

to be honest I'm absolutely against making the single underscore refer to the same argument. Because we'd lose the entire convenience for the multi argument cases only to save 1(!!) character in rare situations...
->_.re + _.im/100 vs x->x.re + x.im/100 that's just not worth it. So IMO each underscore should denote its own argument, from left to right.
By that approach compared to @mcabbott 's suggestion we'd save 2 characters for the different argument case and only lose a single character for the same argument case. (which btw would currently be handled by the tight binding approach of #24990)
I. e. whenever it feels like multiple underscores should denote the same element, just use an ordinary lambda (it'll only cost you a single character extra)

[rapus95]

the numbering approach won't be possible until 2.0 anyway since _2 etc are currently valid identifiers and thus that would be breaking AFAIU

[yurivish]

the numbering approach won't be possible until 2.0 anyway since _2 etc are currently valid identifiers and thus that would be breaking AFAIU

I think it wouldn't be breaking if _2 only means the second argument inside a headless anonymous function.

Good points re: just using an ordinary lambda. I'm curious what fraction of anonymous functions would be made shorter by the "headless" type. It'd be hard to measure accurately, since anonymous functions are used a lot in interactive non-package code.

[rapus95]

Re-using the same symbol with different meanings in an expression seems confusing to me

while certainly true for most characters, the underscore already has the meaning of
"fill in whatever you get, I won't refer to it anywhere else"
See underscore as left hand side where it's used to tell that we don't need the result; and as a parametric argument to denote "I don't care about the actual type"
So I'd try to use that freedom to not to be bound to ordinary variable behavior. If I wanted ordinary behaviour for it, I could just use an ordinary variable 👀
if we were crazy we could just bind any variable which would otherwise result in an undefvarerror 😂 but I'm strongly against that.

[StefanKarpinski]

I'm in agreement with @rapus95 here: let's stick to the simple win with _ for positional arguments. Anything more complicated seems to me better expresses with named arguments.

[StefanKarpinski]

Oh, one more thing to consider: interaction with |>. With just the basic proposal here, one would often need to write something like this:

collection |> -> map(lowercase, _) |> -> filter(in(words), _)

We could either introduce a new pipe syntax as a shorthand for this, or say that |> also acts as a headless lambda delimiter in the absence of -> so if there are unbound _ in the expression following the |> then the -> is implicitly inserted before the |> for you. So you'd write the above like this instead:

collection |> map(lowercase, _) |> filter(in(words), _)

This does allow using another headless lambda for the filter/map operation, like this for example:

collection |> map(-> _ ^ 7, _) |> filter(-> _ % 3 == 0, _)

Might want to do something similar with <| for symmetry. It feels a little weird to single out these two, but I can't see anything more general that makes much sense to me.

[rapus95]

I love that idea tbh because it would give us some part of #24990 for "free". The only thing that holds me back is that, in that case applying the same rule to \circ probably make sense aswell and then it starts to feel like arbitrary special casing again...

Btw, since the headless approach is currently primarily developed around multiple argument cases, it'd be very nice if we had a syntactical solution for splatting. Otherwise cases like
(1,2) |> (x,y)->x+y wont work anyway since we'd have to wrap the anonymous function into Base.splat

EDIT: could we use |>> as head that automatically splats Tuple? It already has that extra > which somewhat hints the -> and if it's made specifically for that, we could even include splatting since it would be a special operator the evolves around chaining anonymous functions.