Weird issue with distributed

[wcwitt]

Posting so we don't lose track of this:

using Distributed
using ACE1pack

addprocs(2, exeflags="--project=$(Base.active_project())")

@everywhere using Distributed
@everywhere using ACE1pack

elements = [:Si]
basis = ACE1x.ace_basis(elements = elements,
                        order = 2,
                        totaldegree = 8,
                        rcut = 5.5)

# pass basis to workers
@everywhere basis = $basis

Causes errors like:

ERROR: LoadError: On worker 2:
UndefVarError: #7#8 not defined

From discussions with @cortner, seems likely the problem relates to serializing an anonymous function.

[cortner]

These two fail:

t = basis.BB[1].J[1].trans
@everywhere t1 = $t
f = t.f
@everywhere f1 = $(t.f)

But weirdly enough this works:

g = x -> x^2
@everywhere g1 = $g

maybe something to do with scope.

[wcwitt]

For broader context, oversimplifying just a little, here are two lines from ACEfit

f = i -> linear_fill!(A, Y, W, data[i], basis; row_start=row_start[i])
pmap(f, 1:length(data))

These lines have worked in the past, but don't work now, presumably because there is difficulty sending and receiving the new basis. This approach is suboptimal in some ways, and I can think of a few possible refactorings, but I do like the simplicity

[cortner]

I'm not sure I want to send the basis everytime. I think it should be transferred or created just once?

[cortner]

Here is another workaround btw:

using Distributed
using ACE1pack
using ParallelDataTransfer

addprocs(2)

@everywhere begin 
   using Pkg 
   Pkg.activate(@__DIR__())
   using Distributed
   using ACE1pack
end

##

elements = [:Si]
basis = ACE1x.ace_basis(elements = elements,
                        order = 2,
                        totaldegree = 8,
                        rcut = 5.5)
@show length(basis)                        

##

sendto([2,], elements=elements)
@spawnat 2 display(elements)

##

sendto([2,], basis=basis)
@spawnat 2 display(basis)
sleep(4)

[wcwitt]

Hm, I tried the ParallelDataTransfer earlier and it failed exactly the same as basis=$basis.

[wcwitt]

I'll try again. And more generally, if there is a way to send the basis once before the pmap, I'm fine with that. In fact, I'm pretty sure an earlier version of ACEfit did it that way and I removed it because I didn't love the global variables.

What I don't want is to construct the basis separately on individual processes with @Everywhere clauses. Because the basis construction will likely happen somewhere else far away.

[cortner]

I don't really understand what happens behind the scenes... But for sure on my system the $ thing fails while the sendto works ok:

t = basis.BB[1].J[1].trans
@everywhere t1 = $t    # fails
sendto([2,], t2 = t)       # ok 

[wcwitt]

Some progress, but still weird. I'll go with the sendto. Should we close this, or do you want to keep it open until we understand?

# fails
@everywhere basis = $basis

# fails, this is what I tried initially from ParallelDataTransfer
@passobj 1 workers() basis

# works, your solution
sendto(workers(), basis=basis)

[cortner]

My understanding was that the $ thing is not really a general tool to transfer data but more of a hack? Do you have evidence otherwise? If that is actually the intended way to transfer data to another process then we should explore how to fix it. But it I'm right then we should look into a workflow that uses sendto

[wcwitt]

Oh I’ve already started converting to ‘sendto’ and I’m perfectly happy with that.

I still think the others should work (@passobj doesn’t seem like a hack, at least), but the practical question is resolved. Thanks!

[cortner]

then maybe this can be closed?

[wcwitt]

We need to revisit this unfortunately. The sendto solution only "works" in the sense that it doesn't error.

using Distributed

addprocs(2, exeflags="--project=$(Base.active_project())")

@everywhere using Distributed
@everywhere using ACE1pack
@everywhere using ParallelDataTransfer

elements = [:Si]
basis = ACE1x.ace_basis(elements = elements,
                        order = 2,
                        totaldegree = 8,
                        rcut = 5.5)

# fails
#@everywhere basis = $basis

# fails, this is what I tried initially from ParallelDataTransfer
#@passobj 1 workers() basis

# appears to succeed ...
sendto(workers(), basis=basis)
# ... but this fails with "UndefVarError: basis not defined"
@everywhere println(myid(), "  ", length(basis))

[wcwitt]

Possibly related? ChrisRackauckas/ParallelDataTransfer.jl#17