Skip to content

Commit 410b496

Browse files
committed
Fix pmapbatch_elementwise, add docstrings
1 parent b573b38 commit 410b496

File tree

6 files changed

+667
-156
lines changed

6 files changed

+667
-156
lines changed

README.md

Lines changed: 12 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,12 @@ julia> using ParallelUtilities
2424
* `pmapreduce_elementwise`
2525
* `pmapsum_elementwise`
2626
* Functions to evenly split a Tuple of ranges
27-
* `evenlyscatterproduct`
28-
* `nworkersactive`
29-
* `workersactive`
27+
* `ProductSplit`
28+
* `ntasks`
3029
* `whichproc`
3130
* `procrange_recast`
3231
* `localindex`
33-
* `procid_and_localindex`
32+
* `whichproc_localindex`
3433
* `extremadims`
3534
* `extrema_commonlastdim`
3635
* Utility functions to query the cluster
@@ -82,10 +81,10 @@ julia> xrange,yrange,zrange = 1:3,2:4,3:6 # ranges should be strictly increasing
8281

8382
There are a total of 36 possible `(x,y,z)` combinations possible given these ranges. Let's say that we would like to split the evaluation of the function over 10 processors. We describe the simple way to evaluate this and then explain how this is achieved.
8483

85-
The set of parameters may be split up using the function `evenlyscatterproduct`. In this example each of the 10 processors receive a chunk as listed below
84+
The set of parameters may be split up using the function `ProductSplit`. In this example each of the 10 processors receive a chunk as listed below
8685

8786
```julia
88-
julia> [collect(evenlyscatterproduct((xrange,yrange,zrange),10,i)) for i=1:10]
87+
julia> [collect(ProductSplit((xrange,yrange,zrange),10,i)) for i=1:10]
8988
10-element Array{Array{Tuple{Int64,Int64,Int64},1},1}:
9089
[(1, 2, 3), (2, 2, 3), (3, 2, 3), (1, 3, 3)]
9190
[(2, 3, 3), (3, 3, 3), (1, 4, 3), (2, 4, 3)]
@@ -258,8 +257,6 @@ Note that this does not track the progress of the individual maps, it merely tra
258257

259258
The two separate functions `pmapreduce` and `pmapreduce_commutative` exist for historical reasons. They use different binary tree structures for reduction. The commutative one might be removed in the future in favour of `pmapreduce`.
260259

261-
262-
263260
## ProductSplit
264261

265262
In the above examples we have talked about the tasks being distributed approximately equally among the workers without going into details about the distribution, which is what we describe here. The package provides an iterator `ProductSplit` that lists that ranges of parameters that would be passed on to each core. This may equivalently be achieved using an
@@ -295,13 +292,6 @@ julia> Tuple(length(ProductSplit((xrange,yrange,zrange),10,i)) for i=1:10)
295292
(4, 4, 4, 4, 4, 4, 3, 3, 3, 3)
296293
```
297294

298-
The object may be generated through the function `evenlyscatterproduct` using the same signature
299-
300-
```julia
301-
julia> evenlyscatterproduct((xrange,yrange,zrange),10,4)
302-
ProductSplit{Tuple{Int64,Int64,Int64},3,UnitRange{Int64}}((1:3, 2:4, 3:6), (0, 3, 9), 10, 4, 13, 16)
303-
```
304-
305295
### Indexing
306296

307297
The iterator supports fast indexing
@@ -325,12 +315,9 @@ julia> params_long = (xrange_long,yrange_long,zrange_long);
325315
julia> ps_long = ProductSplit(params_long,10,4)
326316
ProductSplit{Tuple{Int64,Int64,Int64},3,UnitRange{Int64}}((1:3000, 1:3000, 1:3000), (0, 3000, 9000000), 10, 4, 8100000001, 10800000000)
327317

328-
julia> length(ps_long)
329-
2700000000
330-
331-
julia> @btime length($ps_long) # this is fast
332-
0.034 ns (0 allocations: 0 bytes)
333-
2700000000
318+
# Evaluate length using random ranges to avoid compiler optimizations
319+
julia> @btime length(p) setup=(n=rand(3000:4000);p=ProductSplit((1:n,1:n,1:n),200,2));
320+
2.674 ns (0 allocations: 0 bytes)
334321

335322
julia> @btime $ps_long[1000000] # also fast, does not iterate
336323
32.530 ns (0 allocations: 0 bytes)
@@ -384,18 +371,18 @@ julia> @btime whichproc($params_long,$val,10)
384371
We can compute the ranges of each variable on any processor in `O(1)` time.
385372

386373
```julia
387-
julia> extrema(ps,2) # extrema of the second parameter on this processor
374+
julia> extrema(ps,dim=2) # extrema of the second parameter on this processor
388375
(3, 4)
389376

390-
julia> Tuple(extrema(ps,i) for i in 1:3)
377+
julia> Tuple(extrema(ps,dim=i) for i in 1:3)
391378
((1, 3), (3, 4), (4, 4))
392379

393380
# Minimum and maximum work similarly
394381

395-
julia> (minimum(ps,2),maximum(ps,2))
382+
julia> (minimum(ps,dim=2),maximum(ps,dim=2))
396383
(3, 4)
397384

398-
julia> @btime extrema($ps_long,2)
385+
julia> @btime extrema($ps_long,dim=2)
399386
52.813 ns (0 allocations: 0 bytes)
400387
(1, 3000)
401388
```

src/ParallelUtilities.jl

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,13 @@ using Reexport
55
@reexport using Distributed
66

77
export ProductSplit,
8-
evenlyscatterproduct,
98
ntasks,
109
whichproc,
1110
procrange_recast,
1211
localindex,
13-
procid_and_localindex,
12+
whichproc_localindex,
1413
extremadims,
1514
extrema_commonlastdim,
16-
workersactive,
17-
nworkersactive,
1815
nodenames,
1916
gethostnames,
2017
nprocs_node,

0 commit comments

Comments
 (0)