Merge pull request freqtrade#12126 from stash86/main-stash

Improve lookahead analysis to use full dataframe comparison instead of just the last row

Author: xmatthias

docs/lookahead-analysis.md

@@ -38,7 +38,7 @@ Many strategies, without the programmer knowing, have fallen prey to lookahead b
 This typically makes the strategy backtest look profitable, sometimes to extremes,  but this is not realistic as the strategy is "cheating" by looking at data it would not have in dry or live modes.
 
 The reason why strategies can "cheat" is because the freqtrade backtesting process populates the full dataframe including all candle timestamps at the outset.
-If the programmer is not careful or oblivious how things work internally 
+If the programmer is not careful or oblivious how things work internally
 (which sometimes can be really hard to find out) then the strategy will look into the future.
 
 This command is made to try to verify the validity in the form of the aforementioned lookahead bias.
@@ -50,8 +50,7 @@ After this initial backtest runs, it will look if the `minimum-trade-amount` is
 If this happens, use a wider timerange to get more trades for the analysis, or use a timerange where more trades occur.
 
 After setting the baseline it will then do additional backtest runs for every entry and exit separately.  
-When these verification backtests complete, it will compare the indicators at the signal candles (both entry or exit)
-and report the bias.  
+When these verification backtests complete, it will compare both dataframes (baseline and sliced) for any difference in columns' value and report the bias.
 After all signals have been verified or falsified a result table will be generated for the user to see.
 
 ### How to find and remove bias? How can I salvage a biased strategy?
@@ -98,8 +97,8 @@ If the strategy has many different signals / signal types, it's up to you to sel
 This would lead to a false-negative, i.e. the strategy will be reported as non-biased.
 - `lookahead-analysis` has access to the same backtesting options and this can introduce problems.
 Please don't use any options like enabling position stacking as this will distort the number of checked signals.
-If you decide to do so, then make doubly sure that you won't ever run out of `max_open_trades` slots, 
+If you decide to do so, then make doubly sure that you won't ever run out of `max_open_trades` slots,
 and that you have enough capital in the backtest wallet configuration.
-- In the results table, the `biased_indicators` column 
+- In the results table, the `biased_indicators` column
 will falsely flag FreqAI target indicators defined in `set_freqai_targets()` as biased.  
 **These are not biased and can safely be ignored.**

freqtrade/optimize/analysis/lookahead.py

@@ -70,34 +70,29 @@ def analyze_indicators(self, full_vars: VarHolder, cut_vars: VarHolder, current_
         cut_df: DataFrame = cut_vars.indicators[current_pair]
         full_df: DataFrame = full_vars.indicators[current_pair]
 
-        # cut longer dataframe to length of the shorter
-        full_df_cut = full_df[(full_df.date == cut_vars.compared_dt)].reset_index(drop=True)
-        cut_df_cut = cut_df[(cut_df.date == cut_vars.compared_dt)].reset_index(drop=True)
-
-        # check if dataframes are not empty
-        if full_df_cut.shape[0] != 0 and cut_df_cut.shape[0] != 0:
-            # compare dataframes
-            compare_df = full_df_cut.compare(cut_df_cut)
-
-            if compare_df.shape[0] > 0:
-                for col_name, values in compare_df.items():
-                    col_idx = compare_df.columns.get_loc(col_name)
-                    compare_df_row = compare_df.iloc[0]
-                    # compare_df now comprises tuples with [1] having either 'self' or 'other'
-                    if "other" in col_name[1]:
-                        continue
-                    self_value = compare_df_row.iloc[col_idx]
-                    other_value = compare_df_row.iloc[col_idx + 1]
-
-                    # output differences
-                    if self_value != other_value:
-                        if not self.current_analysis.false_indicators.__contains__(col_name[0]):
-                            self.current_analysis.false_indicators.append(col_name[0])
-                            logger.info(
-                                f"=> found look ahead bias in indicator "
-                                f"{col_name[0]}. "
-                                f"{str(self_value)} != {str(other_value)}"
-                            )
+        # trim full_df to the same index and length as cut_df
+        cut_full_df = full_df.loc[cut_df.index]
+        compare_df = cut_full_df.compare(cut_df)
+
+        if compare_df.shape[0] > 0:
+            for col_name in compare_df:
+                col_idx = compare_df.columns.get_loc(col_name)
+                compare_df_row = compare_df.iloc[0]
+                # compare_df now comprises tuples with [1] having either 'self' or 'other'
+                if "other" in col_name[1]:
+                    continue
+                self_value = compare_df_row.iloc[col_idx]
+                other_value = compare_df_row.iloc[col_idx + 1]
+
+                # output differences
+                if self_value != other_value:
+                    if not self.current_analysis.false_indicators.__contains__(col_name[0]):
+                        self.current_analysis.false_indicators.append(col_name[0])
+                        logger.info(
+                            f"=> found look ahead bias in column "
+                            f"{col_name[0]}. "
+                            f"{str(self_value)} != {str(other_value)}"
+                        )
 
     def prepare_data(self, varholder: VarHolder, pairs_to_load: list[DataFrame]):
         if "freqai" in self.local_config and "identifier" in self.local_config["freqai"]:
@@ -132,7 +127,13 @@ def prepare_data(self, varholder: VarHolder, pairs_to_load: list[DataFrame]):
         varholder.data, varholder.timerange = backtesting.load_bt_data()
         varholder.timeframe = backtesting.timeframe
 
-        varholder.indicators = backtesting.strategy.advise_all_indicators(varholder.data)
+        temp_indicators = backtesting.strategy.advise_all_indicators(varholder.data)
+        filled_indicators = dict()
+        for pair, dataframe in temp_indicators.items():
+            filled_indicators[pair] = backtesting.strategy.ft_advise_signals(
+                dataframe, {"pair": pair}
+            )
+        varholder.indicators = filled_indicators
         varholder.result = self.get_result(backtesting, varholder.indicators)
 
     def fill_entry_and_exit_varHolders(self, result_row):