Skip to content

Conversation

@c42f
Copy link
Owner

@c42f c42f commented Aug 18, 2025

A few special forms have a kind of "deferred top level evaluation" semantics for some of their children:

  • @cfunction - the function name and types
  • ccall / foreigncall / @ccall - the type arguments and sometimes the function name
  • cglobal - the function name
  • @generated - the expression defining the generated function stub

For example, in @ccall f()::Int, the Int means "the symbol Int as looked up in global scope in the module", and should fail if Int refers to a local variable. Currently all three of these cases are handled through different mechanisms with varying levels of hygiene inconsistency and ability to warn about access to local variables.

To fix this problem, introduce the new K"deferred_toplevel_eval"K"static_eval" form which wraps an expression and preserves it as a piece of AST in the output (rather than producing IR), but still resolves scope and hygiene.

Use this new form to remove all the special case child-index-dependent handling of these disparate forms from the IR.

Also fixes bugs in Base.@cfunction hygiene where the function name might be resolved to a global symbol in the wrong module.

Also move demo @ccall implementation into JuliaLowering, clean up and fix a few things which were broken and implement the gc_safe option from Base.@ccall. Makes use of deferred_toplevel_eval kind for more precise diagnostics.

Needed for #33 which makes the issues with @cfunction hygiene more obvious

@mlechu
Copy link
Collaborator

mlechu commented Aug 18, 2025

Linking JuliaLang/julia#57931 as a relevant discussion (though mostly about the shared library argument to ccall) and in case you have opinions to add

@topolarity
Copy link
Contributor

Does this still support get_item(::Type{T}) where T = @ccall foo()::Ref{T} ?

IIRC the ccall lowering is an awkward "local 'static' TypeVars + top-level evaluation" where the local static parameters have to be preserved as TypeVars in the resulting expression

@c42f
Copy link
Owner Author

c42f commented Aug 18, 2025

Does this still support get_item(::Type{T}) where T = @ccall foo()::Ref{T} ?

No, good catch! I think I've made it slightly too restrictive right now. Do the exact same rules apply to cglobal and cfunction?

I can easily add :static_parameter as allowed use of bindings, alongside :global. I should rename the kind to ... something. K"static_eval" might be good enough for now (more precise names welcome)

@c42f c42f force-pushed the caf/deferred-toplevel-eval branch from e7c46ce to 2b83623 Compare August 19, 2025 05:55
A few special forms have a kind of "deferred static evaluation"
semantics for some of their children:
* `@cfunction` - the function name and types
* `ccall` / `foreigncall` / `@ccall` - the type arguments and sometimes
  the function name
* `cglobal` - the function name
* `@generated` - the expression defining the generated function stub

For example, in `@ccall f()::Int`, the `Int` means "the symbol `Int` as
looked up in global scope in the module, or as a static parameter of the
method", and should fail if `Int` refers to a local variable. Currently
all three of these cases are handled through different mechanisms with
varying levels of hygiene inconsistency and ability to warn about access
to local variables.

To fix this problem, introduce the new `K"static_eval"` form which wraps
an expression and preserves it as a piece of AST in the output (rather
than producing IR), but still resolves scope and hygiene.

Use this new form to remove all the special case child-index-dependent
handling of these disparate forms from the IR.

Also fixes bugs in `Base.@cfunction` hygiene where the function name
might be resolved to a global symbol in the wrong module.

Also move demo `@ccall` implementation into JuliaLowering, clean up and
fix a few things which were broken and implement the gc_safe option from
`Base.@ccall`. Makes use of static_eval kind for more precise
diagnostics.
@c42f c42f force-pushed the caf/deferred-toplevel-eval branch from 2b83623 to 617970e Compare August 19, 2025 06:10
@c42f
Copy link
Owner Author

c42f commented Aug 19, 2025

@topolarity wow, the cases where static parameters are allowed are quite limited in the existing runtime's interpretation of ccall parameters because they're checked quite early. I didn't expect this:

julia> get_item(::Type{T}) where T = @ccall foo()::Ref{T}
get_item (generic function with 1 method)

julia> get_item(::Type{T}) where T = @ccall foo()::T
ERROR: TypeError: in ccall method definition, expected Type, got a value of type TypeVar

I've updated things here to allow any use of static parameters because it doesn't seem like lowering can make the decision about which uses of static parameters are valid.

@c42f c42f changed the title Add K"deferred_toplevel_eval" for cfunction/ccall/cglobal Add K"static_eval" for cfunction/ccall/cglobal Aug 19, 2025
@topolarity
Copy link
Contributor

Do the exact same rules apply to cglobal and cfunction?

I believe so!

Here @cfunction works the same:

julia> plus(x::T) where T = @cfunction($((y)->x+y), Ref{T}, (Ref{T},))
julia> ccall(plus(1).ptr, Ref{Int}, (Ref{Int},), 2)
3
julia> ccall(plus(1 + 3im).ptr, Ref{Complex{Int}}, (Ref{Complex{Int}},), 2 - 2im)
3 + 1im

and it looks like cglobal does too:

julia> find_sym(::Val{symbol}) where symbol = cglobal(symbol)
julia> find_sym(Val{:malloc}())
Ptr{Nothing} @0x00007bcdf5aad650

they're checked quite early

Yeah, that's true. That error message is arguably a defect of the type-checking in method.c (https://github.com/JuliaLang/julia/blob/44fdede182dbc8e1e6f0e40c5af0470060e5063b/src/method.c#L194-L204), but in general we do require that the ccall etc. has a static C ABI at method definition time, even though the underlying Julia types might be dynamic. I think we do that so that we can still successfully codegen the de-specialized method (without libffi and other shenanigans to handle fully-dynamic C ABIs).

That error message should probably be more like this one:

julia> struct Foo{T}; val::T; end
julia> foo(x::T) where T = @cfunction($((y)->[x] .+ y), Foo{T}, (Foo{T},))
ERROR: cfunction method definition: return type doesn't correspond to a C type

@c42f
Copy link
Owner Author

c42f commented Aug 19, 2025

I think we do that so that we can still successfully codegen the de-specialized method

Makes sense, that was also my guess after I got over the surprise :-)

That error message should probably be more like this one
ERROR: cfunction method definition: return type doesn't correspond to a C type

From the user's high level point of view they may very well be trying to use ccall with a C type (eg, Cint), but the TypeVar hasn't been resolved by the compiler at that early point in compilation (and may never be known statically if we compile a nospecialize version of the function). So it probably should hedge a bit ... like "return type couldn't be statically resolved to a C type" or something.

@c42f c42f merged commit bc0ad7f into main Aug 19, 2025
1 check passed
@c42f c42f deleted the caf/deferred-toplevel-eval branch August 19, 2025 23:46
@c42f
Copy link
Owner Author

c42f commented Aug 19, 2025

Ok, I think this will do. Thanks for the comments!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants