the zero RMSE problem.

[clarype]

One thing that may overshadow all of this -- the zero RMSE problem.

I decided to look at the .csvs from this run. I'm looking at the TCW one, and I filtered it down to just our favorite cluster 37. And lo and behold, there are a ton of runs that have RMSE zero because it was just a straight line. Worse, the reason those are zero is because the recovery threshold is set to 0.25 or 0.5. I think roughly half the fits for cluster 37 are like this!!!!

In the TCW spreadsheet across all 200 points and 144 runs, only 700 out of more than 31,000 runs have the Zero RMSE problem. BUT, that RMSE issue is going to show up in the exactly the situation where we are judging our outputs -- where the water comes in for a reservoir! Because in those situations, the apparently signal is one of very fast recovery -- steep increase TCW. And so everywhere that sees a steep increase will preferentially have a straight line when the recovery threshold is set low, and that will result in the unfortunate RMSE of zero.

Thus, these runs with 0.25 will be preferentially chosen because they have a really good RMSE.

IN a quick check of the TCG spreadsheet, the problem seems to be even worse. Adn again, MOST of the times where the RMSE is showing up, it's where the recovery_threshold is 0.25.

Which means -- we NEED to fix the RMSE problem, no matter what else we find. We can calculate it from the fitted and source values -- we just need to find where in the workflow would be easiest to do that.

I suspect the easiest place would be in ltop_lt_paramater_scoring_01.py in the dfprep function. WE would need to read in each line and check for the RMSE.

Originally posted by @ramblingrek in #6 (comment)

[clarype]

Oh wait -- no, it would be better to do it right when it's written: in ltop.py / runLTversionsHelper, right when we get the rmse -- check to see if it's zero, and if so, then do the calculation right there. Like, right afer line 410.

Originally posted by @ramblingrek in #6 (comment)

[clarype]

NOTE: None of this solves the other problem we have -- which is that if there are multiple tied runs, we always take the zeroth one.

I don't think we want to randomly select one. We want something that we can replicate.

So, if we don't do it randomly, and we don't want to always pick the zeroth one, we have two options.

One would be to try different points within the same cluster (rather than just one), and keep adding points until we get to the point where one particular recipe wins the most often.

The other would be to simply apply some rules: rules to handle if there are multiple recipes of, say, recovery threshold, how to choose the one we want. I could just decree these rules based on simple logic -- like, always take the parameter choice that should lead to more complicated fits.

I suspect the second way would be easier to implement.

Originally posted by @ramblingrek in #6 (comment)

[clarype]

Second way would for sure be easier to implement...

Originally posted by @brobertspierel in #6 (comment)

[clarype]

So, @brobertspierel has been spearheading this, and added some functionality to replace the zero RMSE values using in script RMSE calculations from the source and fitted data arrays. However, when testing at locations where we do have RMSE values from LandTrendr these RMSE values do not match but there are close. @ramblingrek says that is okay, and possibly makes sense from how GEE runs Landtrendr on the back end. I am awaiting the final results before closing this issue.