[16.0][IMP] account_invoice_triple_discount: Improve migration performance#2279
[16.0][IMP] account_invoice_triple_discount: Improve migration performance#2279
Conversation
Use the upgrade util library lib from Odoo if available to migrate data to improve performances. It allows accounting line data to be updated in parallel (10 threads) in ranges of 10,000 rows. This significantly reduces the time required for migration. On a database with more than 12,000,000 account.move.line entries, before the change, the process was killed after 3 hours because it was taking too long. After the change, the update was completed in 12 minutes.
|
I convert it to draft the time to verify the data in my migrated database... |
hildickethan
left a comment
hildickethan
left a comment
There was a problem hiding this comment.
first time seeing odoo.upgrade, seems like it was created in 2023, does this replace openupgrade for migration scripts? the parallel features at least look powerful and it seems like it has most openupgrade features at a glance
| discount = 100 * ( | ||
| 1 - ( | ||
| (100 - COALESCE(discount1, 0.0)) / 100 | ||
| (100 - COALESCE(discount, 0.0)) / 100 |
There was a problem hiding this comment.
wouldnt this add discount2 and discount3 to discount again?
There was a problem hiding this comment.
when the request is executed, discount1 is not yet initialized and discount on the right side of the operator contains the original value into the row...
This is the library used internally at Odoo for their migration scripts. It contains a lot of methods equivalent to those offered by OpenUpgrade-Utils. I now tend to prefer using it to OpenUpgrade-Utils because this library offers more advanced and powerful features. However, we can never be grateful enough for what OpenUpgrade-Utils has offered the community for so many years. The advantage of OpenUpgrade-Utils is that the community still controls its life cycle, which is not the case with the Odoo library.... |
pedrobaeza
left a comment
pedrobaeza
left a comment
There was a problem hiding this comment.
Maybe we can replicate that parallel query in openupgradelib in our OCA way...
@pedrobaeza IMHO it's not a good idea. The odoo library is well maintained, heavily used, tested and improved by the Odoo team with performance and usability in mind. For example take a look to the commit message of one of the last change and you will see the complexity of the problems taken into account odoo/upgrade-util@cef1907. => We could maybe switch to the odoo upgrade lib for our migration script 🤔 However, I didn't want to open this debate or take sides for one lib or another in the context of this PR, just highlight an alternative while remaining compatible with the user's final choice. But it's definitely a topic that deserves to be discussed in the openupgrade group because the performance gains are huge and the tools offered cover all the cases encountered in migrations. |
|
Yeah, I see it's complex... but if we want to improve the performance in OU in general, we should think about it. Maybe including calls to these utils in some places. Something to think in general. Opened OCA/OpenUpgrade#5543. |
| @openupgrade.migrate() | ||
| def migrate(env, version): | ||
| def migrate(cr, version): | ||
| env = Environment(cr, SUPERUSER_ID, {}) |
There was a problem hiding this comment.
Why this change ?
I think we could simplify the script.
-> Assuming that openupgrade exists. it's a regular OCA tools, used in many migrations scripts.
-> just implement USE_ODOOUPGRADE.
-> if openupgrade and odoopgrade exists, the script should use odooupgrade.
4 participants
Use the upgrade util library lib from Odoo if available to migrate data to improve performances.
It allows accounting line data to be updated in parallel (10 threads) in ranges of 10,000 rows. This significantly reduces the time required for migration. On a database with more than 12,000,000 account.move.line entries, before the change, the process was killed after 3 hours because it was taking too long. After the change, the update was completed in 12 minutes.