Introduce second date field to PP model for booking / EX AND valuta date

[ZfT2]

Here is a first draft to introduce a second date field to the PP files for handling the different dates used inside the transaction, see issues:

https://forum.portfolio-performance.info/t/buchung-von-dividenden-am-ex-tag-oder-bei-gutschrift
#2173
#2074
https://forum.portfolio-performance.info/t/buchungsdatum-vs-valutadatum

I tried to keep it small and simple (don't worry about the changed files count, I will explain it below ;) ) :

So the PR wants, as a first step, only provide the possibility to store a second date for each transaction from the PDF transaction files.

It can be the basis, that one can extend PP later on to use the right date for the performance / calculation stats :)

Here the main steps I did:

• added a second field to the Transaction class: dateBooking
• renamed date to dateValue
• as in (all or most?) PDF importers currently the valuta date (value) is used, my thought is to us this as new default date everywhere
• added a util method (secondDiffers in TextUtil) that schould be used when dates are read. If the both dates from the files are identical, only one date is imported. I think, In most cases the dates are identical, so no need to store the same date twice.
• adjusted the protofuf writer so that (hopefully, I don't have experience with protobuf) the binary files are also updated.
• extended JUnit tests regarding file writing and reading.

Through the renaming from "date" to "dateValue" all the 200+ PDF importers are shown as changed. But its is just the method name change :)
Only, as a demonstration, I already adjusted the Nordax PDF importer to use both dates, dateValue AND dateBooking.

Open points from my side:

naming: For now, I named the date fields, as explained, dateValue and dateBooking, but maybe that is not the best?
Should I, as "value" has already a meaning in a programming language, better call it "dateValuta" or something else?
I guess "dateBooking" is fine for account statements, but for dividends, dateEx is better? Or a more generic name, maybe something like "dateTransaction"?

To avoid that every PDF importer looks changed with this PR, maybe keep a setDate() and getDate() method in Transaction and delegate it to get/set dateValue?

Comments are welcome :)

[Nirus2000]

Hello @ZfT2 ,
Thank you very much for the pull request. This feels like the umpteenth attempt to integrate Ex-Date, and given how this pull request started, I'm afraid it can be closed again right away.

Perhaps you could take a look at pull request #3324 and then reconsider...

I'm really sorry, but no one is going to look at +6000 source code changes if the implementation of problems such as model and is not implemented step by step.

Best regards,
Alex

[ZfT2]

Ahh ok, I didn't see that there is #3324 already.

My change here tries to cover both: ex date and booking date. First for dividend, second for account statements.

Regarding the size of changes: As I already explaind, it is only because of the method renaming. It is no problem to add a delegate method and the change count becomes much smaller.

[buchen]

@ZfT2 Thanks for putting this on the agenda. @Nirus2000 would say "again" and hope for "finally included" ;-)

In short: my opinion is that PP should suport the Ex-dividend date (ex-date) but not separate between booking and valuta date.

My understanding is this:

• Booking date: The date on which the transaction is recorded (posted) by the bank in the account ledger.
• Valuta date: The date from which the transaction amount is value-effective, i.e. from which interest or balance calculation applies.
• Ex-dividend date (ex-date): The first trading day on which the security is traded without entitlement to the declared dividend. Only shareholders who own the security before this date receive the dividend.

All transactions could carry a booking and valuta data, only dividend transactions (and maybe interest?) additionally have a third date, the ex-dividend date.

I get the argument to include the "ex-dividend date" for dividend payments. To be 1000% correct, one would record the dividend on the ex-dividend date as performance (that is when the instrument is trading without the dividends) even if it only shows up on the account on the booking/valuta date. In fact, some users do it this way: record the dividend on an extra account, and then transfer it to the reference account on the date of the valuta date.

We can discuss the implementation later, but briefly that would mean extending the AccountTransaction with an exDate (of course, subclassing AccountTransaction could be an option as well, but let's see how that would complicate the code base), then update the UI to allow editing the ex-date, update the PDF importer with a correct import, update the performance calculation (e.g. with an virtual account for the accrued dividends).

I do not get the argument about the booking vs. valuta date. PP would need to calculate always with the valuta date. Yes, some operations would be easier for the user to document (say when the bank is "fixing" an old transaction, it has a current booking date but an old valuta date). However, overall I have the feeling that it would unnecessarily complicate the user interface and user experience with - I feel - limited benefits. I would also argue that 3 out of the 4 links in the description actually point to the "ex-dividend date" and not the "booking/valuta date". If you see this different, please make the argument here!

[ZfT2]

I do not get the argument about the booking vs. valuta date. PP would need to calculate always with the valuta date. Yes, some operations would be easier for the user to document (say when the bank is "fixing" an old transaction, it has a current booking date but an old valuta date). However, overall I have the feeling that it would unnecessarily complicate the user interface and user experience with - I feel - limited benefits. I would also argue that 3 out of the 4 links in the description actually point to the "ex-dividend date" and not the "booking/valuta date". If you see this different, please make the argument here!

Thank you for your thoughts about this, @buchen :)
Some banks book the yearly interest ( e.g., for daily savings accounts) with booking date 31.12. but value date 01.01.
I have a current example here:

Buchungsdatum Valutadatum Beschreibung
31.12.2025 01.01.2026 Interest

But the interest here is paid for the financial investment in 2025.

So if saving both dates, we have the change to represent this in the stats later on...

[buchen]

I have a current example here:

Booking date	Valuta date	Description
31.12.2025	01.01.2026	Interest

@ZfT2 Ok, but what do you want PP to do differently now? What algorithm do you want PP to change?

I understand the preference to choose whether to assign it to 2025 or keep it with the valuta date. But that seems to be a change on the level of each transaction. And then how do you imagine the user telling PP which date to use?

Please do not get me wrong: I push back to get to the best argument. My fear is that it clutters the user experience (two rows of dates, more input fields in the dialogs, more explanation what the difference is and when it matters and when not).

I just looked at a dividend payment. Here are the dates I found:

date	description
ex-date	13 March (according to DivvyDiary)
payment date	26 March (according to DivvyDiary and inside the document)
document date	27 March
valuta date	28 March

Here I see why I support (now) adding an ex-date for dividends. And the ex-date has specific semantics: performance is recorded on ex-date (on 13 March), kept in a separate "accrued dividends" account, and then moved to the original account on 28 March.

But the separating of booking date and valuta date seems negligible.

BTW, my Deutsche Bank PDF document does not contain the ex-date itself. @Nirus2000 is this different for other bank documents? Because even if we extend the model with the ex-date, filling it properly requires the info in the document. Do you have a gut feeling how many banks report the ex-date?

[ZfT2]

@ZfT2 Ok, but what do you want PP to do differently now? What algorithm do you want PP to change?

I understand the preference to choose whether to assign it to 2025 or keep it with the valuta date. But that seems to be a change on the level of each transaction. And then how do you imagine the user telling PP which date to use?

Please do not get me wrong: I push back to get to the best argument. My fear is that it clutters the user experience (two rows of dates, more input fields in the dialogs, more explanation what the difference is and when it matters and when not).

No worries, to be honest, I still have not thought about it in detail :) As described, I wanted to provide a "first step change" to store it, purely informational, without any impact on calculations or performance.
My idea is: If we introduce an additional field in the data model for the ex-date for dividends, we can directly make it "generic" and use it to store a second date for regular account, especially interest transactions. (Maybe, it could be called "transaction Date" or "effective Date"...)
I totally see your argument that it makes the user interfaces more complex. Maybe it could (later) be a field that is by default hidden in the dialogs and the user can unfold it only if needed, to keep it clean and simple? Regarding the calculation logic, maybe there could be a setting if PP should fetch the transactions for calculation by the booking or valuta date?

If storing it, first, purely informational, my personal use case would be I use the PP PDF importers to import transactions from my daily savings and fixed accounts from different banks. Of course, to track the performance in PP, but for those banks, who doesn’t provide a FinTS / HBCI interface, I use the PP CSV export and then import the CSV file in my homebanking software, which has also two date fields.
But I see also it might not be the main goal of PP to play the perfect instantaneous water heater ;) Maybe a compromise could be: Use the second date really for dividends calculation and only informational for account transactions.

BTW, my Deutsche Bank PDF document does not contain the ex-date itself. @Nirus2000 is this different for other bank documents? Because even if we extend the model with the ex-date, filling it properly requires the info in the document. Do you have a gut feeling how many banks report the ex-date?

SBroker, Raiffeisen, Postbank, Onvista, Traderepublic, Baaderbank, ING Diba, DKB, Santander, Targobank, Scalable ... and some more seem to provide it. Try a Text search over the PDF debug files with "Ex-Tag" to get an overview :)

[buchen]

My idea is: If we introduce an additional field in the data model for the ex-date for dividends, we can directly make it "generic" and use it to store a second date for regular account, especially interest transactions. (Maybe, it could be called "transaction Date" or "effective Date"...)

I would prefer to separate ex-date from the booking date. My gut feeling is that mixing both concerns makes it harder. And a dividend transaction can also have a valuta date and a booking date.

@ZfT2 Would you be willing to - maybe as a separate pull request - to create a first draft? At an "exDate". I assume only the date, not with a timestamp. To the AccountTransaction. Used only for the Dividends. Add a check that ensure consistency.

Maybe it could (later) be a field that is by default hidden in the dialogs and the user can unfold it only if needed, to keep it clean and simple?

That is a good idea. Also for the ex-date because I do not think that the majority of the users would want to look up and maintain the ex-date. There also the UI should remain simple.

Regarding the calculation logic, maybe there could be a setting if PP should fetch the transactions for calculation by the booking or valuta date?

Yes, one could add such an option, but what does it mean to switch globally between valuta and booking date?

And out of curiosity, what is your home banking software doing with those two dates (in terms of calculation)?

If storing it, first, purely informational, my personal use case would be I use the PP PDF importers to import transactions from my daily savings and fixed accounts from different banks. [...] then import the CSV file in my homebanking software, which has also two date fields.

Ok, that is a "purely informational" und "pass-thru use case" that would not require any UI. But it is also not a primary use case for PP... But I understand your use case. Maybe it could be an option to keep this things hidden from normal users, but one can activate a "nerd mode".

[ZfT2]

I would prefer to separate ex-date from the booking date. My gut feeling is that mixing both concerns makes it harder. And a dividend transaction can also have a valuta date and a booking date.

Good point. In my files I have no dividend transactions with different valuta and booking date, but who knows...

@ZfT2 Would you be willing to - maybe as a separate pull request - to create a first draft? At an "exDate". I assume only the date, not with a timestamp. To the AccountTransaction. Used only for the Dividends. Add a check that ensure consistency.

What do you mean with consistency check?

Yes, I can do that. So I would:

• provide ex date in AccountTransaction
• store it in all file formats
• add a check to store the ex date only if it is different from the other date?
• add and adjust test cases
• adjust one PDF importer (as a demonstration example)?

Is it too much or anything else for the "MVP"?

Yes, one could add such an option, but what does it mean to switch globally between valuta and booking date?

If, I think I would make sense not to have the option globally, but (at least) per account. So one could set it for the accounts where the bank books the account settlement with different dates, overlapping the reporting period.

And out of curiosity, what is your home banking software doing with those two dates (in terms of calculation)?

Nothing :) But it enables me to verify the interest calculation ;)
(while writing this, maybe it becomes useful if PP wants do do also interest calculations or forecasts in future...? Just an idea.)

Ok, that is a "purely informational" und "pass-thru use case" that would not require any UI. But it is also not a primary use case for PP... But I understand your use case. Maybe it could be an option to keep this things hidden from normal users, but one can activate a "nerd mode".

Would be really nice to have it... Yes, no UI is required for that, just storing the complete booking like the bank is providing it. I think if with "nerd mode" or not, should not matter, but both is fine for me. And don't be afraid the files are blown up :) In my experience, most bookings have only on date, so we can also check here too to add the second date only if it is different.

[buchen]

add a check to store the ex date only if it is different from the other date?

No. I think we need to know if the user has entered an ex-date (even if it is the same as the valuta date). Therefore I would propose: it is null (uses almost no storage) or it is given (uses little storage if it is the same).

add and adjust test cases

I hope there are no existing test cases to change (unless we change one importer). The "ex-date" should add functionality, but should not change existing functionality (assuming that existing dividend transactions have no ex-date).

adjust one PDF importer (as a demonstration example)

You can do so. But keep it as a separate commit on the same pull request.

I also think we should keep the pull requests separately. Let's first focus on adding the ex-date.