-
Notifications
You must be signed in to change notification settings - Fork 2
CI on ALPS #242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
luraess
wants to merge
34
commits into
main
Choose a base branch
from
lr/alps-ci
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+212
−16
Open
CI on ALPS #242
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
3f5099a
Add CI rules for ALPS
luraess 383f33b
Update
luraess e873240
Fixup
luraess 632f44f
Fixup
luraess 4fe15a4
Add more testing
luraess 124e4de
Trigger buildkite on 1.12
luraess fd8226c
Fixup
luraess a96a984
Add missing deps
luraess da33c65
Fixup ref test
luraess a541bee
Fixup
luraess 1d7b5bc
Fixup IGG call
luraess f75b0ba
More fixes
luraess 001315d
Fixup
luraess 6651039
fix init particles call
albert-de-montserrat 6a89663
Fixup
luraess 7374f02
allow scalar indexing
albert-de-montserrat a6a8db0
Allow scalar
luraess 8b38f67
try baremetal runner
luraess 77c2265
Merge branch 'lr/alps-ci' of github.com:JuliaGeodynamics/JustPIC.jl i…
luraess afa9e82
Fixup
luraess 819f058
Fixup
luraess 6a920c0
Fixup
luraess 2eda7ff
update 2D MPI advection
albert-de-montserrat b356294
New try
luraess f62e319
Merge branch 'lr/alps-ci' of github.com:JuliaGeodynamics/JustPIC.jl i…
luraess 1500f05
Up
luraess 6016e34
Cleanup
luraess 504f439
Cleanup
luraess 39db6f2
Cleanup
luraess 9698797
Format
luraess f6d1da6
AMDGPU fix trial
luraess 0e296c2
Fixup project
luraess bca90e6
Fixup format
luraess ca1fb3f
Add badge
luraess File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| include: | ||
| - remote: 'https://gitlab.com/cscs-ci/recipes/-/raw/master/templates/v2/.ci-ext.yml' | ||
|
|
||
| unit_test: | ||
| extends: .uenv-runner-daint-gh200 | ||
| image: julia/25.5:v1 | ||
| script: | ||
| - export MPICH_GPU_SUPPORT_ENABLED=1 | ||
| - export IGG_CUDAAWARE_MPI=1 # IGG | ||
| - export JULIA_CUDA_USE_COMPAT=false # IGG | ||
| - julia -e 'println("Instantiating project"); | ||
| using Pkg; | ||
| Pkg.activate(pwd())' | ||
| - julia -e 'println("Running tests"); | ||
| using Pkg; | ||
| Pkg.activate(pwd()); | ||
| Pkg.test("JustPIC"; test_args=["--backend=CUDA"])' | ||
| variables: | ||
| WITH_UENV_VIEW: 'juliaup' | ||
| SLURM_JOB_NUM_NODES: 1 | ||
| SLURM_NTASKS_PER_NODE: 1 | ||
| SLURM_GPUS_PER_TASK: 1 | ||
| SLURM_TIMELIMIT: "00:30:00" | ||
|
|
||
| ref_test: | ||
| extends: .baremetal-runner-daint-gh200 | ||
| script: | ||
| - export MPICH_GPU_SUPPORT_ENABLED=1 | ||
| - export IGG_CUDAAWARE_MPI=1 # IGG | ||
| - export JULIA_CUDA_USE_COMPAT=false # IGG | ||
| - echo "Preparing the test environment (single rank)" | ||
| - srun -n 1 --uenv julia/25.5:v1 --view=juliaup julia --project=. -e 'using Pkg; Pkg.instantiate()' | ||
| - srun -n 1 --uenv julia/25.5:v1 --view=juliaup julia --project=. -e 'using Pkg; Pkg.add("CUDA")' | ||
| - echo "Running the reference test (multiple ranks)" | ||
| - srun --uenv julia/25.5:v1 --view=juliaup julia --project=. scripts/temperature_advection_MPI_ci.jl | ||
| variables: | ||
| SLURM_JOB_NUM_NODES: 2 | ||
| SLURM_NTASKS_PER_NODE: 4 | ||
| SLURM_GPUS_PER_TASK: 1 | ||
| SLURM_TIMELIMIT: "00:30:00" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| using CUDA | ||
| # import Pkg | ||
| # Pkg.resolve(); Pkg.update() | ||
| using JustPIC, JustPIC._2D | ||
|
|
||
| # Threads is the default backend, | ||
| # to run on a CUDA GPU load CUDA.jl (i.e. "using CUDA"), | ||
| # and to run on an AMD GPU load AMDGPU.jl (i.e. "using AMDGPU") | ||
| # const backend = JustPIC.CPUBackend # Options: CPUBackend, CUDABackend, AMDGPUBackend | ||
| const backend = CUDABackend # Options: CPUBackend, CUDABackend, AMDGPUBackend | ||
|
|
||
| # using GLMakie | ||
| using ImplicitGlobalGrid | ||
| import MPI | ||
|
|
||
| # Analytical flow solution | ||
| vx_stream(x, y) = 250 * sin(π * x) * cos(π * y) | ||
| vy_stream(x, y) = -250 * cos(π * x) * sin(π * y) | ||
| g(x) = Point2f( | ||
| vx_stream(x[1], x[2]), | ||
| vy_stream(x[1], x[2]) | ||
| ) | ||
|
|
||
| function expand_range(x::AbstractRange) | ||
| dx = x[2] - x[1] | ||
| n = length(x) | ||
| x1, x2 = extrema(x) | ||
| xI = x1 - dx | ||
| xF = x2 + dx | ||
| return LinRange(xI, xF, n + 2) | ||
| end | ||
|
|
||
| function main() | ||
| # Initialize particles ------------------------------- | ||
| nxcell, max_xcell, min_xcell = 24, 40, 1 | ||
| n = 64 | ||
| nx = ny = n - 1 | ||
| me, dims, = init_global_grid( | ||
| n - 1, n - 1, 1; | ||
| init_MPI = MPI.Initialized() ? false : true, | ||
| select_device = false | ||
| ) | ||
| Lx = Ly = 1.0 | ||
| dxi = dx, dy = Lx / (nx_g() - 1), Ly / (ny_g() - 1) | ||
| # nodal vertices | ||
| xvi = xv, yv = let | ||
| dummy = zeros(n, n) | ||
| xv = [x_g(i, dx, dummy) for i in axes(dummy, 1)] | ||
| yv = [y_g(i, dy, dummy) for i in axes(dummy, 2)] | ||
| LinRange(first(xv), last(xv), n), LinRange(first(yv), last(yv), n) | ||
| end | ||
| # nodal centers | ||
| xci = xc, yc = let | ||
| dummy = zeros(nx, ny) | ||
| xc = [x_g(i, dx, dummy) for i in axes(dummy, 1)] | ||
| yc = [y_g(i, dy, dummy) for i in axes(dummy, 2)] | ||
| LinRange(first(xc), last(xc), n), LinRange(first(yc), last(yc), n) | ||
| end | ||
|
|
||
| # staggered grid for the velocity components | ||
| grid_vx = xv, expand_range(yc) | ||
| grid_vy = expand_range(xc), yv | ||
|
|
||
| particles = init_particles( | ||
| backend, nxcell, max_xcell, min_xcell, xvi... | ||
| ) | ||
|
|
||
| # Cell fields ------------------------------- | ||
| Vx = TA(backend)([vx_stream(x, y) for x in grid_vx[1], y in grid_vx[2]]) | ||
| Vy = TA(backend)([vy_stream(x, y) for x in grid_vy[1], y in grid_vy[2]]) | ||
| T = TA(backend)([y for x in xv, y in yv]) | ||
| T0 = deepcopy(T) | ||
| V = Vx, Vy | ||
|
|
||
| nx_v = (size(T, 1) - 2) * dims[1] | ||
| ny_v = (size(T, 2) - 2) * dims[2] | ||
| T_v = zeros(nx_v, ny_v) | ||
| T_nohalo = TA(backend)(zeros(size(T) .- 2)) | ||
|
|
||
| dt = mapreduce(x -> x[1] / MPI.Allreduce(maximum(abs.(x[2])), MPI.MAX, MPI.COMM_WORLD), min, zip(dxi, V)) / 2 | ||
|
|
||
| # Advection test | ||
| particle_args = pT, = init_cell_arrays(particles, Val(1)) | ||
| grid2particle!(pT, xvi, T, particles) | ||
|
|
||
| niter = 250 | ||
| for iter in 1:niter | ||
| me == 0 && @show iter | ||
|
|
||
| # advect particles | ||
| advection!(particles, RungeKutta2(), V, (grid_vx, grid_vy), dt) | ||
|
|
||
| # update halos | ||
| update_cell_halo!(particles.coords...) | ||
| update_cell_halo!(particle_args...) | ||
| update_cell_halo!(particles.index) | ||
| # shuffle particles | ||
| move_particles!(particles, xvi, particle_args) | ||
| # interpolate T from particle to grid | ||
| particle2grid!(T, pT, xvi, particles) | ||
|
|
||
| @views T_nohalo .= T[2:(end - 1), 2:(end - 1)] | ||
| gather!(Array(T_nohalo), T_v) | ||
|
|
||
| # if me == 0 && iter % 1 == 0 | ||
| # x_global = range(0, Lx, length = size(T_v, 1)) | ||
| # y_global = range(0, Ly, length = size(T_v, 2)) | ||
| # f, ax, = heatmap(x_global, y_global, T_v) | ||
| # w = 0.504 | ||
| # offset = 0.5 - (w - 0.5) | ||
| # lines!( | ||
| # ax, | ||
| # [0, w, w, 0, 0], | ||
| # [0, 0, w, w, 0], | ||
| # linewidth = 3 | ||
| # ) | ||
| # lines!( | ||
| # ax, | ||
| # [0, w, w, 0, 0] .+ offset, | ||
| # [0, 0, w, w, 0], | ||
| # linewidth = 3 | ||
| # ) | ||
| # lines!( | ||
| # ax, | ||
| # [0, w, w, 0, 0] .+ offset, | ||
| # [0, 0, w, w, 0] .+ offset, | ||
| # linewidth = 3 | ||
| # ) | ||
| # lines!( | ||
| # ax, | ||
| # [0, w, w, 0, 0], | ||
| # [0, 0, w, w, 0] .+ offset, | ||
| # linewidth = 3 | ||
| # ) | ||
|
|
||
| # save("figs/T_MPI_$iter.png", f) | ||
| # end | ||
|
|
||
| # px = particles.coords[1].data[:] | ||
| # py = particles.coords[2].data[:] | ||
| # idx = particles.index.data[:] | ||
| # f = scatter(px[idx], py[idx], color=:black) | ||
| # save("figs/particles_$(iter)_$(me).png", f) | ||
| end | ||
|
|
||
| # f, ax, = heatmap(xvi..., T, colormap=:batlow) | ||
| # streamplot!(ax, g, xvi...) | ||
| # f | ||
| return finalize_global_grid() | ||
|
|
||
| end | ||
|
|
||
| main() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldnt it make more sense to have 'pre' here for the pre-releases?