Can't stop global hostcalls 😺

[PhilipFackler]

I keep getting this:

Warning: Global hostcalls detected!
- Source: MethodInstance for ...
- Hostcalls: [:malloc_hostcall]

Use `AMDGPU.synchronize(; stop_hostcalls=false)` to synchronize and stop them.
Otherwise, performance might degrade if they keep running in the background.

I've tried using AMDGPU.synchronize(; stop_hostcalls=false) (and with stop_hostcalls=true which would make more sense 😛). No matter where I call that I can't get rid of it.

There are no explicit calls to malloc in my code, and it's not obvious where this is coming from. Is there a tool you all know of to determine where this is being initiated?

Thanks!

[pxl-th]

Hi! Sorry for the late reply.

I've tried using AMDGPU.synchronize(; stop_hostcalls=false) (and with stop_hostcalls=true which would make more sense 😛). No matter where I call that I can't get rid of it.

This is probably a bug. I'll check it.

As for malloc in the kernels, it originates most often during conversion from one type to another.

E.g. code below will contain malloc, because if the conversion to Int fails, it will raise an exception, boxing the original 1.0 value. And for that it needs malloc.

julia> @code_llvm Int(1.0)
; Function Signature: (::Type{Int64})(Float64)
;  @ float.jl:990 within `Int64`
define i64 @julia_Int64_1288(double %"x::Float64") #0 {
top:
  %jlcallframe1 = alloca [3 x ptr], align 8
  %gcframe2 = alloca [4 x ptr], align 16
  call void @llvm.memset.p0.i64(ptr align 16 %gcframe2, i8 0, i64 32, i1 true)
  %thread_ptr = call ptr asm "movq %fs:0, $0", "=r"() #10
  %tls_ppgcstack = getelementptr i8, ptr %thread_ptr, i64 -8
  %tls_pgcstack = load ptr, ptr %tls_ppgcstack, align 8
  store i64 8, ptr %gcframe2, align 16
  %frame.prev = getelementptr inbounds ptr, ptr %gcframe2, i64 1
  %task.gcstack = load ptr, ptr %tls_pgcstack, align 8
  store ptr %task.gcstack, ptr %frame.prev, align 8
  store ptr %gcframe2, ptr %tls_pgcstack, align 8
;  @ float.jl:991 within `Int64`
; ┌ @ float.jl:619 within `<=`
   %0 = fcmp ult double %"x::Float64", 0xC3E0000000000000
; └
  %1 = fcmp uge double %"x::Float64", 0x43E0000000000000
  %narrow.not = or i1 %0, %1
  br i1 %narrow.not, label %L16, label %L8

L8:                                               ; preds = %top
; ┌ @ floatfuncs.jl:45 within `isinteger`
; │┌ @ rounding.jl:469 within `trunc`
; ││┌ @ rounding.jl:469 within `#trunc#288`
; │││┌ @ float.jl:462 within `round`
      %2 = call double @llvm.trunc.f64(double %"x::Float64")
; │└└└
; │┌ @ float.jl:492 within `-`
    %3 = fsub double %"x::Float64", %2
; │└
; │┌ @ float.jl:654 within `==` @ float.jl:616
    %4 = fcmp une double %3, 0.000000e+00
; └└
  br i1 %4, label %L16, label %L14

L14:                                              ; preds = %L8
;  @ float.jl:992 within `Int64`
; ┌ @ float.jl:416 within `unsafe_trunc`
   %5 = fptosi double %"x::Float64" to i64
   %6 = freeze i64 %5
   %frame.prev8 = load ptr, ptr %frame.prev, align 8
   store ptr %frame.prev8, ptr %tls_pgcstack, align 8
; └
  ret i64 %6

L16:                                              ; preds = %L8, %top
;  @ float.jl:994 within `Int64`
  %7 = load ptr, ptr getelementptr (i8, ptr @jl_small_typeof, i64 256), align 8
  %gc_slot_addr_1 = getelementptr inbounds ptr, ptr %gcframe2, i64 3
  store ptr %7, ptr %gc_slot_addr_1, align 8
  %ptls_field = getelementptr inbounds ptr, ptr %tls_pgcstack, i64 2
  %ptls_load = load ptr, ptr %ptls_field, align 8
  %"box::Float64" = call noalias nonnull align 8 dereferenceable(16) ptr @ijl_gc_pool_alloc_instrumented(ptr %ptls_load, i32 752, i32 16, i64 136598580446688) #8
  %"box::Float64.tag_addr" = getelementptr inbounds i64, ptr %"box::Float64", i64 -1
  store atomic i64 136598580446688, ptr %"box::Float64.tag_addr" unordered, align 8
  store double %"x::Float64", ptr %"box::Float64", align 8
  %gc_slot_addr_0 = getelementptr inbounds ptr, ptr %gcframe2, i64 2
  store ptr %"box::Float64", ptr %gc_slot_addr_0, align 16
  store ptr @"jl_sym#Int64#1300.jit", ptr %jlcallframe1, align 8
  %8 = getelementptr inbounds ptr, ptr %jlcallframe1, i64 1
  store ptr %7, ptr %8, align 8
  %9 = getelementptr inbounds ptr, ptr %jlcallframe1, i64 2
  store ptr %"box::Float64", ptr %9, align 8
  %10 = call nonnull ptr @j1_InexactError_1298(ptr nonnull @"+Core.InexactError#1299.jit", ptr nonnull %jlcallframe1, i32 3)
  call void @ijl_throw(ptr nonnull %10)
  unreachable
}

To avoid this, you can use GPU-friendly (but unsafe) conversion:

julia> @code_llvm unsafe_trunc(Int, 1.0)
; Function Signature: unsafe_trunc(Type{Int64}, Float64)
;  @ float.jl:416 within `unsafe_trunc`
define i64 @julia_unsafe_trunc_1316(double %"x::Float64") #0 {
top:
  %0 = fptosi double %"x::Float64" to i64
  %1 = freeze i64 %0
  ret i64 %1
}

[PhilipFackler]

Thanks for the help. I had a couple calls to trunc which I have swapped for unsafe_trunc and now I'm not seeing the warnings.