Parallel Sequential Monte Carlo

[yebai]

I looked into the newly released Threads.@threading construct in Julia 0.5. It seems that adding parallelization feature to SMC is simple - could be done in a few lines of code (see the parallelsmc branch).

However, the threading feature in Julia is still quite fragile (see e.g. here). I've also filed a bug in the Julia repo.

Maybe we should wait until the Julia team fix these threading bugs and revisit to this feature in a few months time.

[emilemathieu]

@yebai: I've reproduced the code on the issue JuliaLang/julia#19450 you opened, and I do not get any segfault. Yet, it seems that there is still this issue JuliaLang/julia#10441.

I can't find the parallelsmcbranch, do you still have the code ?

[emilemathieu]

The solution posted here https://discourse.julialang.org/t/how-do-i-deal-with-random-number-generation-when-multithreading/5636/2 seems to be a good solution to deal with random numbers while using@threads.

[yebai]

I can't find the parallelsmcbranch, do you still have the code?

Thanks for investigating into this. Please find my code in the following branch:

https://github.com/yebai/Turing.jl/tree/hg/parallelsmc

Ps. it might be broken since it was written long before.

[trappmartin]

I just implemented a version which seems to work on branch multithreaded-PG and will have a further look into this issue.

See: https://github.com/TuringLang/Turing.jl/blob/multithreaded-PG/src/core/container.jl#L152-L180

[emilemathieu]

FYI I implemented a few months ago a parallel version of IPMCMC: https://github.com/emilemathieu/Turing.jl/blob/7c72a238b4d278720409d845880738d5d2c44ed3/src/samplers/ipmcmc.jl#L70

[trappmartin]

Great! I’ll have a look at it.

[trappmartin]

The code on branch https://github.com/TuringLang/Turing.jl/tree/multithreaded-PG now implements a multi-threaded version of the ParticleContainer and an adaptation of the code in https://github.com/emilemathieu/Turing.jl/blob/7c72a238b4d278720409d845880738d5d2c44ed3/src/samplers/ipmcmc.jl#L70 for Julia 0.6. A test for the distributed IPMCMC is currently failing, see below.

As far as I understand the compiler code, it seems that the compiler currently does not generate the inner callback functions, *_model, on all processes. I'll, therefore, have a deeper look at the compiler implementation.

@emilemathieu Did you encounter the same issue?
cc @yebai

[emilemathieu]

Great work !
I'm not sure to completely understand what you mean by inner callback functions, *_model.
I did not encounter such an issue. The only "tricky" part I had was that within @parallel, objects defined outside the scope of the loop couldn't be updated. Thus I had to explicitly return the objects needed to be updated within the loop.

[trappmartin]

If I'm running a simple example to test the parallel implementation, e.g.

addprocs(1)

@everywhere using Turing
srand(125)
x = [1.5, 2]

@everywhere @model gdemo(x) = begin
	s ~ InverseGamma(2, 3)
	m ~ Normal(0, sqrt(s))
	for n in 1:length(x)
		x[n] ~ Normal(m, sqrt(s))
	end
	s, m
end

inference = IPMCMC(30, 500, 4)
chain = sample(gdemo(x), inference)

, I get an error that ###gdemo_model#700 is not defined on the second worker. Am I'm doing something wrong here?

Thanks!

[yebai]

@trappmartin You need to use @everywhere in all places, i.e.

addprocs(1)

@everywhere using Turing
srand(125)
@everywhere x = [1.5, 2]

@everywhere @model gdemo(x) = begin
	s ~ InverseGamma(2, 3)
	m ~ Normal(0, sqrt(s))
	for n in 1:length(x)
		x[n] ~ Normal(m, sqrt(s))
	end
	s, m
end

@everywhere inference = IPMCMC(30, 500, 4)
@everywhere mf = gdemo(x)
chain = sample(mf, inference)

Supporting paralliesm using processes is a pain since it involves data transfer. It's probably better to stick with threads for now until we have a cleaner and more organised code base.

[trappmartin]

Right, thanks for the tip!

I wanted to have both supported so that one can use multi-threading locally and also distributed computation if necessary. Unfortuanitelly, I'm currently having a broken test for the distributed code and see a similar problem as in TuringLang/Turing.jl#463 even though the non-distributed code works fine. Working on it...

[mohamed82008]

For shared-memory parallelism, I suggest making use of KissThreading.jl https://github.com/bkamins/KissThreading.jl which is shared-memory parallelism free of the closure bug hassle. That package is not registered yet, but I can work on getting it registered if it turns out to be useful here. GPU support is probably also worth considering at some point, but that's a much larger commitment.