Realistic parameter estimation: high-fidelity LES data versus 1D turbulence parameterization

[glwagner]

This WIP PR starts to implement a realistic parameter estimation procedure in which the free parameters of a turbulence parameterization are calibrated in a one-dimensional context against the horizontal average of three-dimensional, high-fidelity large eddy simulation of an ocean surface boundary layer. Parameter estimation is achieved by minimizing a loss function that quantifies error between the turbulence parameterization "model" and high-fidelity simulation "data".

The PR is WIP because for some reason julia is quitting and I'm getting the mysterious error

FATAL ERROR: Symbol "__nv_fmin"not found
signal (6): Abort trap: 6
in expression starting at /Users/gregorywagner/Projects/EnzymaticOcean/boundary_layer_turbulence_calibration/oceananigans_loss_function.jl:158
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 129274008 (Pool: 129226728; Big: 47280); GC: 88
Abort trap: 6

I've seen this before and I think it's some odd package issue that we can solve by restricting / updating the Manifest (since we use Oceananigans like this all the time).

Note that I had to downgrade some packages because Oceananigans only supports KernelAbstractions 0.6.0. Possibly we will need to do some development with Oceananigans to integrate all of our code.

For more information about the data see https://github.com/CliMA/LESbrary.jl
Some information about the turbulence parameterization is available on a poster I presented at a conference some time ago (the current implementation is an improvement on what I presented there): https://glwagner.github.io/assets/figures/ocean_sciences_2020_poster.png

[glwagner]

Note: Oceananigans#glw/catke needs to be updated to pull in changes that resolve a method ambiguity in a low level function.

[vchuravy]

We need KernelAbstractions 0.7 :/ the changes in there are specifically for AD support.

Re the nv_fmin error what is the version of Cassette you are using?

[vchuravy]

Looks like Cassette 0.3.5, please try https://github.com/JuliaLabs/Cassette.jl/releases/tag/v0.3.6 which has the fix for Cassette using functions from the future.

[glwagner]

Nice, ok!

@ali-ramadhan can Oceananigans release compat for KernelAbstractions?

[ali-ramadhan]

Should we update the Oceananigans.jl [compat] entry of KA to allow for ^0.7?

I think I tried upgrading to KernelAbstractions.jl v0.7 (via ] add KernelAbstractions#master) as part of CliMA/Oceananigans.jl#1514 but I don't think it was ready for use as tests errored.