Duplicate entry in "pysystemtrade/data/futures/fx_prices_csv/EURUSD.csv" for the date 2019-10-24.

[dt-scientist]

Issue (Code explaining the issue is included at the bottom):
File: pysystemtrade/data/futures/fx_prices_csv/EURUSD.csv
There is a duplicate value for 2019-10-24. This is the only duplicate value in the file.
As a result, the PnL calculation does not appear to be correct for dates from 2019-10-25 onwards.
For example, the PnL in base currency for 2019-10-25 is calculated as:
181.431080837765 = 16.3387394 × 10 × 1.110435
• PnL in points: 163.387394
• Value per point: 10
• FX rate used: 1.110435
However, 1.110435 is the FX rate for 2019-10-24, not 2019-10-25.
The issue affects all dates after 2019-10-24, not just 2019-10-25. The PnL calculation is using a lagged FX rate (lagged by one day because of the duplicate entry).
It seems the code cannot handle duplicate values in the EURUSD.csv file.

Code:
from systems.provided.futures_chapter15.estimatedsystem import futures_system
system=futures_system()
group = system.accounts.pandl_for_instrument_rules_unweighted("EUROSTX")
curve_ewmac16_64 = group["ewmac16_64"]
calc = curve_ewmac16_64._pandl_calculator_with_costs
df = pd.Series(calc.fx)
df.index = df.index.normalize()
startdate = '2019-10-23'
enddate = '2019-10-25'
df[startdate:enddate]
"""
index
2019-10-23 1.113300
2019-10-24 1.110100
2019-10-24 1.110435
2019-10-25 1.108025
"""

df_pnl_in_points = pd.Series(calc.pandl_in_points())
df_pnl_in_points.index = df_pnl_in_points.index.normalize()
df_pnl_in_points[startdate:enddate]
"""
index
2019-10-23 72.605228
2019-10-24 94.379872
2019-10-25 16.338739
"""

df_pnl_in_base_currency = pd.Series(calc.pandl_in_base_currency())
df_pnl_in_base_currency.index = df_pnl_in_base_currency.index.normalize()
df_pnl_in_base_currency[startdate:enddate]
"""
index
2019-10-23 808.314009
2019-10-24 1047.710960
2019-10-25 181.431081
"""

[bug-or-feature]

I gave up trying to work out what you're trying to say the problem is, because the formatting is all wrong. Please fix that. Use the preview feature to check your text before submitting

But I can see from the CSV file that there are two entries for 2019-10-24, but they are not duplicates. One is midnight, the next 11pm that evening. Daylight savings difference I guess

2019-10-23 00:00:00,1.1133
2019-10-24 00:00:00,1.1101
2019-10-24 23:00:00,1.110435
2019-10-25 23:00:00,1.108025

(That's what code formatting looks like if you do it properly)

[dt-scientist]

The _base_pandl_given_currency_pandl function converts a P&L series expressed in a foreign currency into a base currency using FX rates. It aligns the FX time series to the P&L dates by forward-filling the most recent available FX rate for each P&L date. Finally, it multiplies the currency P&L by the aligned FX rate to produce the base-currency P&L series.

    def _base_pandl_given_currency_pandl(self, pandl_in_ccy) -> pd.Series:
        fx = self.fx
        fx_aligned = fx.reindex(pandl_in_ccy.index, method="ffill")
        return pandl_in_ccy * fx_aligned

Step 1: Inputs
fx Series (DatetimeIndex with times):

datetime            fx
23/10/2019 00:00	1.1133
24/10/2019 00:00	1.1101
24/10/2019 23:00	1.110435
25/10/2019 23:00	1.108025

pandl_in_ccy (Date index, no time → interpreted as midnight):

date	       pandl_in_ccy
23/10/2019	-3.393870785
24/10/2019	-3.393870785
25/10/2019	-3.393870785

Step 2: reindex(..., method="ffill")
fx_aligned = fx.reindex(pandl_in_ccy.index, method="ffill")
For each P&L date, pandas takes the most recent FX value at or before that timestamp:
• 23/10/2019 00:00 → exact match → 1.1133
• 24/10/2019 00:00 → exact match → 1.1101
• 25/10/2019 00:00 → last FX before this time is
24/10/2019 23:00 → 1.110435
So:

date	    fx_aligned
23/10/2019	1.1133
24/10/2019	1.1101
25/10/2019	1.110435

Up to and including 23/10/2019, the dates in the FX series have a time of 00:00. On 24/10/2019, there are two entries: one with a time of 00:00 and one with a time of 23:00. From 25/10/2019 onward, the time is 23:00.
Up to and including 24/10/2019, the pandl_in_ccy values are multiplied by the FX rate from the FX series on the same date. From 25/10/2019 onward, pandl_in_ccy is multiplied by the previous FX rate; that is, the P&L for 25/10/2019 is multiplied by the FX rate from 24/10/2019 rather than the FX rate from 25/10/2019.
This occurs because the FX rate timestamps shift to 23:00 starting on 24/10/2019, whereas for earlier dates they were set to 00:00. This may not necessarily be an error, but it does represent a change in how the P&L is calculated due to the time at which the FX rate is sourced.