Following LT Parameters in LTOP Workflow

[clarype]

There seems to be a possible mis-asigning of LandTrendr parameters and/or parameter ID to the appropriate LandTrendr run. This was found for cluster_id 37 where in the Pareto Frontier GUI a chart of fitting showed a fit not possible (figure1 and 2) for the parameter defined. This was confirmed in the LTOP_QA GUI (figure3) as the same defined parameters for cluster id 37 produce a different fitting.

(figure1)

(figure2)
GUI two

(figure 3)

[clarype]

Interestingly, in the above figure 2 there a key "param_num" with a value of 72. What is the LandTrendr parameter at that position in the mass LandTrendr parameter dictionary?

[clarype]

counting from zero (location 71) it is
{timeSeries: ee.ImageCollection([]), maxSegments: 8 , spikeThreshold: 1.0, vertexCountOvershoot: 3, preventOneYearRecovery: true, recoveryThreshold: 1.0, pvalThreshold: 0.15, bestModelProportion: 0.75, minObservationsNeeded: 8 }
which products this fitting

(figure 4)

[clarype]

counting from 1 (location 72) it is
{timeSeries: ee.ImageCollection([]), maxSegments: 10 , spikeThreshold: 0.75, vertexCountOvershoot: 3, preventOneYearRecovery: true, recoveryThreshold: 0.25, pvalThreshold: 0.05, bestModelProportion: 0.75, minObservationsNeeded: 10}
Which is the same parameters in figure 2 from above. Which produces the same fit as figure 3.

(figure 5)

[clarype]

What to do? Well, since there is ambiguity in LT parameter fields I have been instructed to add a LT parameter as a key : value pair in the LandTrendr parameter dictionary. This value will follow the parameters all the way through the workflow and hopefully provide more insight to what is going on.

Should these new ids start at 0 or 1?

The script that does this is lt_params.py

Example:
{timeSeries: ee.ImageCollection([]), maxSegments: 10 , spikeThreshold: 0.75, vertexCountOvershoot: 3, preventOneYearRecovery: true, recoveryThreshold: 0.25, pvalThreshold: 0.05, bestModelProportion: 0.75, minObservationsNeeded: 10, lt_param_id: 72 }

Note the last key:value pair. lt_param_id:72

[clarype]

In digging into this, we found some more interesting things. Starting from the LandTrendr information from the abstract image, we found LandTrendr outputs that should not be associated with the give LandTrendr parameters. Using the examples from above, the output for figure 5 should only have 3 breakpoints in its list of breakpoints. This is not the case for cluster id 37. In the CSV file

LTOP_Cambodia_200_abstractImageSample_lt_144params_TCW_c2_selected.csv

we find that for the row in question the LT parameters

{timeSeries: ee.ImageCollection([]), maxSegments: 10 , spikeThreshold: 0.75, vertexCountOvershoot: 3, preventOneYearRecovery: true, recoveryThreshold: 0.25, pvalThreshold: 0.05, bestModelProportion: 0.75, minObservationsNeeded: 10}

has a list with 4 breakpoints.

[[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0]]

More interestingly, the above list of breakpoints match the breakpoints of figure 4.

[clarype]

Also, it does not seem like param_num and paramNum are getting lost in translation. Both values seem to be consistent as they enter and pass through the workflow and are not called to assist the parameter selection.

[clarype]

I am going to look from more examples of LT parameters that do not match the list of breakpoints.

[ramblingrek]

One thing that may overshadow all of this -- the zero RMSE problem.

I decided to look at the .csvs from this run. I'm looking at the TCW one, and I filtered it down to just our favorite cluster 37. And lo and behold, there are a ton of runs that have RMSE zero because it was just a straight line. Worse, the reason those are zero is because the recovery threshold is set to 0.25 or 0.5. I think roughly half the fits for cluster 37 are like this!!!!

In the TCW spreadsheet across all 200 points and 144 runs, only 700 out of more than 31,000 runs have the Zero RMSE problem. BUT, that RMSE issue is going to show up in the exactly the situation where we are judging our outputs -- where the water comes in for a reservoir! Because in those situations, the apparently signal is one of very fast recovery -- steep increase TCW. And so everywhere that sees a steep increase will preferentially have a straight line when the recovery threshold is set low, and that will result in the unfortunate RMSE of zero.

Thus, these runs with 0.25 will be preferentially chosen because they have a really good RMSE.

IN a quick check of the TCG spreadsheet, the problem seems to be even worse. Adn again, MOST of the times where the RMSE is showing up, it's where the recovery_threshold is 0.25.

Which means -- we NEED to fix the RMSE problem, no matter what else we find. We can calculate it from the fitted and source values -- we just need to find where in the workflow would be easiest to do that.

I suspect the easiest place would be in ltop_lt_paramater_scoring_01.py in the dfprep function. WE would need to read in each line and check for the RMSE.

[ramblingrek]

Oh wait -- no, it would be better to do it right when it's written: in ltop.py / runLTversionsHelper, right when we get the rmse -- check to see if it's zero, and if so, then do the calculation right there. Like, right afer line 410.

[ramblingrek]

NOTE: None of this solves the other problem we have -- which is that if there are multiple tied runs, we always take the zeroth one.

I don't think we want to randomly select one. We want something that we can replicate.

So, if we don't do it randomly, and we don't want to always pick the zeroth one, we have two options.

One would be to try different points within the same cluster (rather than just one), and keep adding points until we get to the point where one particular recipe wins the most often.

The other would be to simply apply some rules: rules to handle if there are multiple recipes of, say, recovery threshold, how to choose the one we want. I could just decree these rules based on simple logic -- like, always take the parameter choice that should lead to more complicated fits.

I suspect the second way would be easier to implement.

[brobertspierel]

Second way would for sure be easier to implement...