What's the state of the stretched/variable spacing coordinates feature?

[tomchor]

Hi, all, I've been thinking about the stretched coordinates and it's a shame that that feature is not available yet, since it's super useful. Especially because for some simulations that I'm planning on running, the GPU memory is the limiting factor in my case. In talks with @glwagner I remember that a big portion of the work was already done, which is nice!

I'm checking the code and it seems that indeed all the derivative operators assume that the Δ's are arrays, which makes me believe that it's just a matter of defining those arrays accordingly? But also it seems like there's only one set of Δx and Δy, while if the vertical coordinate for some reason there's ΔzF and ΔzC. This kinda makes it look like the work for stretched coordinates is done for the vertical, but not for the horizontal directions.

So what's the state of affairs in this respect?

[christophernhill]

@tomchor there is tridiagonal solver code for stretched vertical too. Vertical stretching could be a good place to start if that is enough for some interesting work. I believe both the solver piece and Δz bits are mostly waiting someone with motivation and tenancity to set up a problem, test a bit and report bugs/help troubleshoot etc...!

[ali-ramadhan]

Yeah I agree that I think most of the infrastructure was set in place and I recall we were setting up some tests when priorities shifted.

The original issue I opened to discuss vertically stretched development is #471 (yes it's been a while lol) with 10 items on the TODO list.

I think we finished up to items 7 and 8 but some of the tests quickly went out of date so they've been skipped or nuked. Reviving those tests might be the first step?

---

This kinda makes it look like the work for stretched coordinates is done for the vertical, but not for the horizontal directions.

Yes. @christophernhill showed that we could still have a fast pressure solver if one dimension is stretched (tridiagonal solve along the stretched dimension, FFTs along the other two) which is potentially even faster than FFTs along all three dimensions.

However, if two or three dimensions are stretched then we have to fall back on a much slower pressure solver, e.g. an iterative conjugate gradient solver. So we haven't really considered grids beyond the RegularCartesianGrid and VerticallyStretchedCartesianGrid since they would probably be slower.

That said, we may want stretching in all directions anyways, and for other models without pressure solvers e.g. the ShallowWaterModel they shouldn't suffer even if all dimensions are stretched. So now there's some reason to stretch all three dimensions (while a year ago there wasn't as much).

[johncmarshall54]

I feel that stretching should be developed as and when it is needed. If someone has a problem that cries out for stretching then they should roll their sleeves up and get it done, working with the development team. But applications should drive this, not 'it would be nice to have this, nice to have that".

…

On Sat, Jan 23, 2021, 7:49 AM Ali Ramadhan ***@***.***> wrote: Yeah I agree that I think most of the infrastructure was set in place and I recall we were setting up some tests when priorities shifted. The original issue I opened to discuss vertically stretched development is #471 <#471> (yes it's been a while lol) with 10 items on the TODO list. I think we finished up to items 7 and 8 but some of the tests quickly went out of date so they've been skipped or nuked. Reviving those tests might be the first step? ------------------------------ This kinda makes it look like the work for stretched coordinates is done for the vertical, but not for the horizontal directions. Yes. @christophernhill <https://github.com/christophernhill> showed that we could still have a fast pressure solver if one dimension is stretched (tridiagonal solve along the stretched dimension, FFTs along the other two) which is potentially even faster than FFTs along all three dimensions. However, if two or three dimensions are stretched then we have to fall back on a much slower pressure solver, e.g. an iterative conjugate gradient solver. So we haven't really considered grids beyond the RegularCartesianGrid and VerticallyStretchedCartesianGrid since they would probably be slower. That said, we may want stretching in all directions anyways, and for other models without pressure solvers e.g. the ShallowWaterModel they shouldn't suffer even if all dimensions are stretched. So now there's some reason to stretch all three dimensions (while a year ago there wasn't as much). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1312 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKXUEQWSL2RXHSTZYE6WP7TS3LAXLANCNFSM4WPIYK7A> .

[francispoulin]

Very good point John. One thing I plan to do soon, after I finish the BickleyJet example for ShallowWaterModel, is to do a wind-driven gyre example. That is easy enough to do with a uniform grid but ideally I would like to get a stretched grid on the zonal direction. I am happy to work on this and build on whatever is done for the vertical direction.

[christophernhill]

That makes sense. @tomchor do you have a killer application for vertical stretched grid? That could be a good place to start.

…

On Sat, Jan 23, 2021 at 9:02 AM John Marshall ***@***.***> wrote: I feel that stretching should be developed as and when it is needed. If someone has a problem that cries out for stretching then they should roll their sleeves up and get it done, working with the development team. But applications should drive this, not 'it would be nice to have this, nice to have that". On Sat, Jan 23, 2021, 7:49 AM Ali Ramadhan ***@***.***> wrote: > Yeah I agree that I think most of the infrastructure was set in place and > I recall we were setting up some tests when priorities shifted. > > The original issue I opened to discuss vertically stretched development is > #471 <#471> (yes it's been > a while lol) with 10 items on the TODO list. > > I think we finished up to items 7 and 8 but some of the tests quickly went > out of date so they've been skipped or nuked. Reviving those tests might be > the first step? > ------------------------------ > > This kinda makes it look like the work for stretched coordinates is done > for the vertical, but not for the horizontal directions. > > Yes. @christophernhill <https://github.com/christophernhill> showed that > we could still have a fast pressure solver if one dimension is stretched > (tridiagonal solve along the stretched dimension, FFTs along the other two) > which is potentially even faster than FFTs along all three dimensions. > > However, if two or three dimensions are stretched then we have to fall > back on a much slower pressure solver, e.g. an iterative conjugate gradient > solver. So we haven't really considered grids beyond the > RegularCartesianGrid and VerticallyStretchedCartesianGrid since they > would probably be slower. > > That said, we may want stretching in all directions anyways, and for other > models without pressure solvers e.g. the ShallowWaterModel they shouldn't > suffer even if all dimensions are stretched. So now there's some reason to > stretch all three dimensions (while a year ago there wasn't as much). > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#1312 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AKXUEQWSL2RXHSTZYE6WP7TS3LAXLANCNFSM4WPIYK7A> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[tomchor]

I don't have a killer application for it right now, but it certainly would be useful for my research for two reasons:

• I'm planning on running some big applications in the future. Since the model most likely won't be parallelized yet until then, the main limitation in that case would be the GPU memory, in which case it's good to trim down the amount of "unnecessary" grid points.
• Post-processing is also kind of slow depending on what I'm calculating, so it'd be good to save on computational costs in general.

So, like I said, I don't have a killer application, but it definitely would be useful in general. The reason I'm asking is because I'm considering accepting the challenge of implementing a vertically-stretched grid in a near future.