Developing Oceananigans and running tests

[glwagner]

I'm starting this thread to share productivity tips about developing Oceananigans and also discuss in detail how Oceananigans testing system works. If anyone has other tips to share beyond this first post, please do! And questions are welcome.

For new developers the basic process, as suggested by @iuryt on #2281, is

1. Fork Oceananigans to create copy that you own
2. Make changes to the forked repository, either on the fork's main branch, or another branch.
3. Open a pull request between the branch with changes and Oceananigans main branch.

@iuryt also asked:

How can I test the forked repo while running julia?

There's a short answer and a long answer to this question.

The short answer is to clone the fork to your local machine, navigate to the root directory of the repository, and run

$ julia --project -e 'using Pkg; Pkg.test()'

This basically executes test/runtests.jl in an isolated environment.

However, you rarely want to run all the tests with one command, because this will take hours. Instead you want to test individual functionality. One way to do this is to navigate to /test/, and then launch julia:

$ julia --project

Next, you can include individual test files at the REPL

julia> include("test_field.jl")

For most of the test files (test/test_*.jl), this should work, because most of them have been written to "stand alone". For the most part, if this doesn't work for a particular file, then we just need to add include("dependencies_for_runtests.jl") at the top of the file.

The nice thing about including the tests interactively is that we can make changes to the tests without having to recompile Oceananigans. In addition, if we are using Revise.jl, we can also make changes to Oceananigans source code and rerun the tests with minimal recompilation cost. This results in a relatively speedy and productive workflow compared to, say, always executing the entire test suite to exercise a particular feature.

Some other ways to test new source code are:

1. Add a script to validation (in the appropriate sub-directory), and to run that script. Stuff in validation should read like ordinary scripts (like examples, but without the need for extensive annotation). For example, if I implement a new turbulence closure, I might write a simple script that implements an example to see if the closure is behaving as I "expect".

2. Use the REPL and Revise.jl. For example, if I'm developing a show method, I might just build the object that I'm developing the show method:

julia> grid = RectilinearGrid(size=(1, 1, 1), extent=(1, 1, 1))
1×1×1 RectilinearGrid{Float64, Periodic, Periodic, Bounded} on CPU with 1×1×1 halo
├── Periodic x ∈ [0.0, 1.0)  regularly spaced with Δx=1.0
├── Periodic y ∈ [0.0, 1.0)  regularly spaced with Δy=1.0
└── Bounded  z ∈ [-1.0, 0.0] regularly spaced with Δz=1.0

julia> c = CenterField(grid)
Field{Center, Center, Center} on RectilinearGrid on CPU
├── grid: 1×1×1 RectilinearGrid{Float64, Periodic, Periodic, Bounded} on CPU with 1×1×1 halo
├── boundary conditions: FieldBoundaryConditions
│   └── west: Periodic, east: Periodic, south: Periodic, north: Periodic, bottom: ZeroFlux, top: ZeroFlux, immersed: ZeroFlux
└── data: 3×3×3 OffsetArray(::Array{Float64, 3}, 0:2, 0:2, 0:2) with eltype Float64 with indices 0:2×0:2×0:2


julia> c # developed a feature! (nice display of field size on the first line)
1×1×1 Field{Center, Center, Center} on RectilinearGrid on CPU
├── grid: 1×1×1 RectilinearGrid{Float64, Periodic, Periodic, Bounded} on CPU with 1×1×1 halo
├── boundary conditions: FieldBoundaryConditions
│   └── west: Periodic, east: Periodic, south: Periodic, north: Periodic, bottom: ZeroFlux, top: ZeroFlux, immersed: ZeroFlux
└── data: 3×3×3 OffsetArray(::Array{Float64, 3}, 0:2, 0:2, 0:2) with eltype Float64 with indices 0:2×0:2×0:2

julia> c # developed another feature (data summary at the end)
1×1×1 Field{Center, Center, Center} on RectilinearGrid on CPU
├── grid: 1×1×1 RectilinearGrid{Float64, Periodic, Periodic, Bounded} on CPU with 1×1×1 halo
├── boundary conditions: FieldBoundaryConditions
│   └── west: Periodic, east: Periodic, south: Periodic, north: Periodic, bottom: ZeroFlux, top: ZeroFlux, immersed: ZeroFlux
└── data: 3×3×3 OffsetArray(::Array{Float64, 3}, 0:2, 0:2, 0:2) with eltype Float64 with indices 0:2×0:2×0:2
    └── max=0.0, min=0.0, mean=0.0

[glwagner]

If you want to know how Oceananigans CI works, check out the buildkite .yml:

https://github.com/CliMA/Oceananigans.jl/blob/main/.buildkite/pipeline.yml

For CI purposes, the Oceananigans tests are organized into "TEST_GROUPs". The test groups can be read off from runtests.jl:

https://github.com/CliMA/Oceananigans.jl/blob/main/test/runtests.jl

For example, the "abstract operations" tests are run by setting some environment variables:

Oceananigans.jl/.buildkite/pipeline.yml

Lines 375 to 378 in c74dfc6

	- label: "👻 gpu abstract operations tests"
	env:
	JULIA_DEPOT_PATH: "$SVERDRUP_HOME/.julia-$BUILDKITE_BUILD_NUMBER"
	TEST_GROUP: "abstract_operations"

and then executing

Oceananigans.jl/.buildkite/pipeline.yml

Lines 379 to 380 in c74dfc6

	commands:
	- "$SVERDRUP_HOME/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.test()'"

Groups can be exploited on your local machine with the command

TEST_GROUP=abstract_operations julia --project -e 'using Pkg; Pkg.test()'

or by just starting julia with

TEST_GROUP=abstract_operations julia --project

and running tests from "package manager mode" by pressing ] and then typing

pkg> test

Finally, note that dependencies_for_runtests.jl will try to detect if you have a valid GPU:

Oceananigans.jl/test/utils_for_runtests.jl

Line 12 in c74dfc6

test_architectures() = CUDA.has_cuda() ? tuple(GPU()) : tuple(CPU())

Oceananigans.jl/test/dependencies_for_runtests.jl

Line 77 in c74dfc6

archs = test_architectures()

The way it's written now, we run only on GPU if a GPU is detected. So change archs if you want different behavior.