Add vertical advection by brian-oneill · Pull Request #328 · E3SM-Project/Omega

brian-oneill · 2026-01-11T17:41:49Z

Add the VertAdv module to Omega, providing data structures and methods for vertical advection. This module includes:

Arrays for storing vertical velocity and tracer fluxes at layer interfaces.
A method to compute the vertical velocity from the divergence of the horizontal velocity.
Methods for computing the tendencies of thickness, horizontal velocity, and tracers due to vertical advection. The tracer tendency methods include multiple options for order of accuracy of the flux calculation, as well as algorithmic options for standard advection or flux-corrected transport (FCT).

Additional changes in this PR:

Extend parallelForOuter with an optional argument to enable the use of scratch arrays in hierarchical parallelism.
Add the auxiliary variable ProvThickness, which represents thickness after horizontal thickness flux. Further work required to fully implement provisional thickness.
Remove BottomDepth from HorzMesh, this field was previously added to VertCoord.

This PR builds and passes the unit tests successfully on Chrysalis, pm-cpu, pm-gpu, and Frontier (CPU & GPU).

Checklist

Fixes #338

mwarusz · 2026-01-15T19:18:08Z

When running VERTADV_TEST with bounds checking enabled I get out-of-bounds errors. The reason is VertCoord::init(false) gets NVertLayers from OmegaMesh.nc and this value is used to create the member arrays of VertAdv in VertAdv::init(). The value of NVertLayers in VertAdv and MaxLayerCell get overwritten later in the convergence test, but the VertAdv member arrays (e.g. VertFlux) don't get resized.

components/omega/configs/Default.yml

cbegeman · 2026-01-15T22:29:52Z

components/omega/src/ocn/VertAdv.cpp

+
+                 for (int KVec = 0; KVec < KLen; ++KVec) {
+                    const I4 K      = KStart + KVec;
+                    const Real WAvg = 0.5_Real * (LocTotVertVelocity(Cell1, K) +


Do we want to use

Omega/components/omega/src/ocn/HorzOperators.h

Line 139 in f2e951a

InterpCellToEdge(const HorzMesh *Mesh);

so that we can easily switch from anisotropic to isotropic interpolation for all variables (here and elsewhere)?

We can’t use InterpCellToEdge here as is, since it takes an Array1DReal as input, while TotalVerticalVelocity is an Array2DReal. Supporting this would require either an overload or a templated version of the interpolator. It may make more sense to handle this outside this PR and identify any other places to implement this refactor.

components/omega/src/ocn/VertAdv.cpp

components/omega/src/ocn/VertAdv.h

components/omega/src/ocn/VertAdv.cpp

brian-oneill · 2026-02-14T01:32:55Z

Polaris `omega_pr` suite

Baseline workdir: /lcrc/group/e3sm/ac.boneill/polaris/intelTest/baseline_omega_pr/
Baseline build: /lcrc/group/e3sm/ac.boneill/dev/intelCPU
PR build: /lcrc/group/e3sm/ac.boneill/myFork/intelBuild
PR workdir: /lcrc/group/e3sm/ac.boneill/polaris/intelTest/new_omega_pr
Machine: chrysalis
Partition: compute
Compiler: intel
Build type: Release
Log: /lcrc/group/e3sm/ac.boneill/polaris/intelTest/new_omega_pr/polaris_omega_pr.o1152821
Result: All tests passed

mark-petersen

Thanks for your documentation throughout. I fixed a few spelling errors.

components/omega/src/ocn/VertAdv.cpp

components/omega/src/ocn/VertAdv.h

components/omega/test/ocn/VertAdvTest.cpp

mark-petersen · 2026-02-19T18:06:16Z

See compiled documentation here:

components/omega/src/ocn/AuxiliaryState.h

components/omega/doc/devGuide/VertAdv.md

hyungyukang · 2026-02-19T21:25:00Z

Using my time-stepping working branch (https://github.com/hyungyukang/Omega/tree/hkang/omega/merge-vert-vel
), I tested this PR with the merry-go-round test in Polaris. Since the test does not currently support Omega directly, I made several adjustments to run it manually. I will document the procedure for running the test manually in a separate comment later.

As expected, I obtained a convergence rate of 1.971 with VerticalTracerFluxOrder: 2, which matches the convergence rate obtained in MPAS-Ocean:

components/omega/src/ocn/CustomTendencyTerms.cpp

components/omega/test/ocn/HorzMeshTest.cpp

hyungyukang · 2026-02-20T02:59:02Z

components/omega/src/ocn/Tendencies.cpp

+   } else if (VAdv->VertAdvChoice == VertAdvOption::FCT) {
+      ThicknessForVAdv = AuxState->LayerThicknessAux.ProvThickness;
+   }
+   VAdv->computeTracerVAdvTend(TracerTend, TracerArray, ThicknessForVAdv,


Suggested change

VAdv->computeTracerVAdvTend(TracerTend, TracerArray, ThicknessForVAdv,

VAdv->computeTracerVAdvTend(LocTracerTend, TracerArray, ThicknessForVAdv,

hyungyukang

I’m approving this PR based on my testing and visual inspection. All the changes I requested have been addressed, and the code works as expected.

@brian-oneill, thank you very much for your hard work on this!

mark-petersen

Approving based on Hyun's convergence test above, and visual inspection. Successfully passed all cTests on perlmutter CPU and GPU with gnu compilers.

components/omega/test/ocn/TendenciesTest.cpp

cbegeman

I ran the 5m merry-go-round case with Omega (above) and MPAS-Ocean (below) and the error patterns look very similar. The horizontal velocity for Omega is not yet available, as it relies on reconstruction to the x-direction.

cbegeman · 2026-02-22T22:58:44Z

Great work, @brian-oneill! I just wanted to check; has anyone yet verified the 3rd and 4th order convergence options? If not, I'll do this with the polaris merry-go-round test.

hyungyukang · 2026-02-23T15:54:21Z

Great work, @brian-oneill! I just wanted to check; has anyone yet verified the 3rd and 4th order convergence options? If not, I'll do this with the polaris merry-go-round test.

@cbegeman , I ran it last time with the 3rd convergence option. However, I was not able to obtain the 3rd order convergence rate for both Omega (1.981) and MPAS-Ocean (1.981). Could you please run it as well just in case I missed something?

cbegeman · 2026-02-23T22:22:32Z

Using one processor, I was able to reproduce @hyungyukang's results:

Testing whether this is fixed by #316

cbegeman · 2026-02-24T00:57:56Z

4th order advection with limiter (MPAS top, Omega bottom):

cbegeman · 2026-02-24T01:34:09Z

4th order advection without limiter (MPAS top, Omega bottom):

This PR adds ProvThickness to LayerThicknessAux, which is needed by the tracer equations. PR E3SM-Project#335 removes SSH from LayerThicknessAux and removes computeVarsOnCells from computeMomAux.

Also add TimeStep to Tendencies

sbrus89 · 2026-03-09T15:09:14Z

Merged locally, passes CTests on pm-gpu and pm-cpu for debug builds. I'm seeing a fail for the barotropic gyre case:

Polaris `omega_pr` suite

Baseline workdir: /pscratch/sd/s/sbrus/polaris_vert_advection_pr_suite_baseline/
Baseline build: /global/homes/s/sbrus/community/omega_worktrees/develop/components/omega
PR build: /global/homes/s/sbrus/scratch/polaris_vert_advection_pr_suite/build
PR workdir: /global/homes/s/sbrus/scratch/polaris_vert_advection_pr_suite
Machine: pm-cpu
Compiler: gnu
Build type: Release
Log: not found
Result:
- Diffs (1 of 8):
  - ocean/planar/barotropic_gyre/munk/free-slip

Note, this case is not passing the property check (Nans are introduced somehow) so it is not just a baseline diff.

cbegeman reviewed Jan 15, 2026

View reviewed changes

components/omega/configs/Default.yml Outdated Show resolved Hide resolved

cbegeman reviewed Jan 15, 2026

View reviewed changes

components/omega/src/ocn/VertAdv.cpp Show resolved Hide resolved

hyungyukang reviewed Jan 25, 2026

View reviewed changes

components/omega/src/ocn/VertAdv.h Outdated Show resolved Hide resolved

hyungyukang reviewed Jan 26, 2026

View reviewed changes

components/omega/src/ocn/VertAdv.cpp Show resolved Hide resolved

brian-oneill marked this pull request as ready for review February 14, 2026 01:01

vanroekel requested a review from mark-petersen February 18, 2026 15:26

mark-petersen reviewed Feb 19, 2026

View reviewed changes

components/omega/src/ocn/VertAdv.cpp Outdated Show resolved Hide resolved

components/omega/src/ocn/VertAdv.h Outdated Show resolved Hide resolved

components/omega/test/ocn/VertAdvTest.cpp Outdated Show resolved Hide resolved

hyungyukang reviewed Feb 19, 2026

View reviewed changes

components/omega/src/ocn/AuxiliaryState.h Outdated Show resolved Hide resolved

components/omega/src/ocn/AuxiliaryState.h Outdated Show resolved Hide resolved

mark-petersen requested review from hyungyukang and mark-petersen February 19, 2026 20:23

hyungyukang reviewed Feb 19, 2026

View reviewed changes

components/omega/doc/devGuide/VertAdv.md Outdated Show resolved Hide resolved

cbegeman reviewed Feb 19, 2026

View reviewed changes

components/omega/src/ocn/CustomTendencyTerms.cpp Show resolved Hide resolved

cbegeman reviewed Feb 19, 2026

View reviewed changes

components/omega/test/ocn/HorzMeshTest.cpp Show resolved Hide resolved

cbegeman mentioned this pull request Feb 19, 2026

BottomDepth should not be read from the horizontal mesh #338

Open

hyungyukang reviewed Feb 20, 2026

View reviewed changes

hyungyukang approved these changes Feb 20, 2026

View reviewed changes

mark-petersen approved these changes Feb 20, 2026

View reviewed changes

mark-petersen reviewed Feb 20, 2026

View reviewed changes

components/omega/test/ocn/TendenciesTest.cpp Outdated Show resolved Hide resolved

cbegeman reviewed Feb 22, 2026

View reviewed changes

cbegeman mentioned this pull request Feb 23, 2026

Standard non-monotonic higher order horizontal advection copied from mpas-ocean to Omega. #316

Merged

13 tasks

brian-oneill and others added 26 commits March 6, 2026 08:47

Linting fix

0bf7b2f

Resize flux arrays in VertAdvTest

794a788

Rename config option from Enabled to Enable

e393477

Remove unnecessary header

3de17b5

Skip vertical tracer advection when not enabled

7d38145

Add TimeStep to AuxState for ProvThickness

613fd6c

Remove commented out ssh calculation

76fea32

Add LayerThicknessAux.computeVarsOnCells to AuxState::computeAll

5a64211

This PR adds ProvThickness to LayerThicknessAux, which is needed by the tracer equations. PR E3SM-Project#335 removes SSH from LayerThicknessAux and removes computeVarsOnCells from computeMomAux.

Add VertAdv to Tendencies

c978e49

Disable VertAdv tends in TimeStepperTest

05298ed

Add VertAdv to AuxState

5cd322c

Add computeVerticalVelocity to computeMomAux

0e30499

Add VertAdv init and clear to OceanDriver methods

3af933d

Add computeThicknessVAdvTend to computeThicknessTendenciesOnly

1ab1cc4

Add computeVelocityVAdvTend to computeVelocityTendenciesOnly

4b34b8b

Add computeTracerVAdvTend to computeTracerTendenciesOnly

8a262ef

Also add TimeStep to Tendencies

Make TimeStep nonoptional for computeTracerVAdvTend

89596b0

Add VertAdv devGuide and userGuide

1ec51e4

Add early return to VertAdv methods when NVertLayers == 1

011ea2c

Fix typos in comments and docs

a09a2b8

Update HorzMesh user doc

c7adf4d

Add sanity check for bottom depth

c11cf1f

Fix linting fail

765e6b6

Rename TracerTend to LocTracerTend in function call

d5573b2

Remove commented-out line

46d0ca9

Fix issues from rebase

fbf582f

sbrus89 force-pushed the omega/add-vertical-advection branch from 10eec48 to fbf582f Compare March 6, 2026 20:01

sbrus89 self-assigned this Mar 6, 2026

sbrus89 added the Omega label Mar 6, 2026

	VAdv->computeTracerVAdvTend(TracerTend, TracerArray, ThicknessForVAdv,
	VAdv->computeTracerVAdvTend(LocTracerTend, TracerArray, ThicknessForVAdv,

Conversation

brian-oneill commented Jan 11, 2026 • edited by xylar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwarusz commented Jan 15, 2026

Uh oh!

Uh oh!

cbegeman Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brian-oneill Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brian-oneill commented Feb 14, 2026

Polaris omega_pr suite

Uh oh!

mark-petersen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mark-petersen commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hyungyukang commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

hyungyukang Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

hyungyukang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mark-petersen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cbegeman left a comment

Choose a reason for hiding this comment

Uh oh!

cbegeman commented Feb 22, 2026

Uh oh!

hyungyukang commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman commented Feb 23, 2026

Uh oh!

cbegeman commented Feb 24, 2026

Uh oh!

cbegeman commented Feb 24, 2026

Uh oh!

sbrus89 commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polaris omega_pr suite

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

brian-oneill commented Jan 11, 2026 •

edited by xylar

Loading

cbegeman Jan 15, 2026 •

edited

Loading

Polaris `omega_pr` suite

hyungyukang left a comment •

edited

Loading

hyungyukang commented Feb 23, 2026 •

edited

Loading

sbrus89 commented Mar 9, 2026 •

edited

Loading

Polaris `omega_pr` suite