Add support for fixing scenarios

[henrikfoss]

I propose adding support to fix certain scenarios to always be selected. This feature is particularly useful for cases where it is important to retain specific scenarios, such as the tails of a distribution, which often contain the most significant risks to be modeled.

Example Usage
outcomes_reduced, probabilities_reduced = FFreducer.reduce(np.inf, num_bins, keep_idx=[0, len(outcomes_original) - 1])

Benefits

• Enhanced Functionality: This change introduces additional functionality while maintaining backward compatibility.

• Risk Management: By allowing specific scenarios to be fixed, users can better model and manage risks located in the tails of the distribution.

[DanieleGioia]

Hi, the idea is nice. I want to run some tests on it first :)

[henrikfoss]

Hi @DanieleGioia, do you want me write some tests for this? I initially wanted to do this, and have actually also tested this locally, and it works for my use cases (fixing the tails), but I tried to clone your repo but couldn't get the tests to run.

[DanieleGioia]

Well, yeah why not. Have you had problems with PyTest? Sometimes, it can be a nasty package.
During weekdays I am crazy full. I will have a deeper look into it this weekend, also thinking a bit about the theoretical background of the original paper of Heitsch Holger, and Römisch

[henrikfoss]

Hi again @DanieleGioia,

I have added some unit tests, but I can't seem to make them run on my computer. Can you give it a go?

[DanieleGioia]

Hi @henrikfoss, sorry for the late answer, but Christmas was in the middle, and I wanted to check out what you did more carefully.

Even if your proposal is kinda sound computationally (see second comment), it doesn't convince me theoretically.
I explain why: In the original paper (Heitsch, Römisch 2003), Theorem 2.5, grants the consistency of the fast-forward algorithm. In its proof, you can see that taking the min{c_{ku},c_{ku^{i-1}}} is necessary to retrieve the correct minimization over the distances between scenarios.
This is done at line 60 of the current implementation and 70 of your proposal, where

dist_mtrx = np.minimum(dist_mtrx, dist_mtrx[u, :])

Since you fixed some "u"s a-priori, I think this is broken and should be managed differently and fixed scenarios should be included somehow. For example, by adding an additional loop for computing the minimum distance matrix w.r.t. the fixed scenarios before starting the actual algorithm, but I should think about it.

Another problem I think is that you do

        u = np.nanargmin(zeta)
        indxR.append(u)

        if fixed_idx:
            for idx in fixed_idx:
                if idx not in indxR:
                    indxR.append(idx)

Thus, if the first u is already in the indxR, will be added twice. Same with the for loop afterwards.

for it in range(n_scenarios-len(indxR)):

It should depend on the first u, checking if it was in the mandatory set already.

Looking at the positives, once J is fixed, the redistribution of the weights is optimal according to Theorem 2.1. (Heitsch, Römisch 2003). Thus, redistribution should work fine :)

What do you think, do you agree?