DMP V2 to V3 Migration Roadmap

[wsy19961129]

This issue lists the steps to migrate the DMP from V2 to V3 in a proper way. The list of items here may be modified during this process.

Prerequisites

Index	Item	Status	Ref
1.1	Fix issues caused by any casting	Done	#2358
1.2	Split core functions out of itmat-interface	Done	#2373

Migration

Stage 1: Updates all backend interfaces, functions and APIs. We split the tasks according to the list of tRPC procedures.

Index	Item	Status	Ref	Note
2.1	Update user-related codes	Done	#2423
2.2	Update file-related codes	Done	#2432
2.3	Update drive-related codes	Done	#2432
2.4	Update permission-related codes	Donw	#2463	This item only modifies the interfaces and GraphQL permissions; the tRPC permissions will be updated in the below features.
2.5	Update study-related codes	Done	#2464	Project-related codes will be removed in further
2.6	Update role-related codes	Done	#2466
2.7	Update data-related codes	Done	#2465
2.8	Update doc-related codes	Paused
2.9	Update domain-related codes	Done	#2467	The backend may be updated when working with the frontend in further
2.10	Update config-related codes	Done	#2468
2.11	Update job-related codes	in progress		To be implemented with LXD
2.12	Update log-related codes	Done	https://github.com/ideafast/ideafast-portal/tree/feat/trpc-log
2.13	Review other core functions	Done
2.14	Update merkle tree	Not start		yfm
2.15	Add webdav support	Done	#2481
2.16	Validate the data schema migration process	Reviewing
2.17	Optimize the codes	Done	#2482	Merge tRPC and GraphQL core functions into one; remove unused GraphQL APIs.
2.18	Update lxd-related codes	in progress		wjf
2.19	Update lxd job-related codes	in progress		wjf
2.20	Update instance-related codes	in progress		wjf
2.21	Update webauthn-related codes	delayed		wjf
Note:

1. When working on one item, some associated interfaces, and core functions may need to be adjusted in parallel; but that is to make sure the item we are working on can pass the build and unit tests.
2. The order of the items may be adjusted.
3. Some core functions rely on other 'more basic' core functions (such file core functions); thus such 'more basic' functions may be updated continuously.

Stage 2: Updates the frontend (In progress; all in one)

Index	Item	Status	Ref	Note
3.1	Update main page	Reviewing		Login, register, reset password, etc
3.2	Update dataset pages	Reviewing		Dataset metadata, file uploading and listing, metadata, permission, etc
3.3	Update visualization pages	Not start		Dashboard, analysis, etc
3.4	Update documents pages	Paused
3.5	Update domain pages	Reviewing
3.6	Update drive pages	Reviewing
3.7	Update job pages	Not start
3.8	Update log pages	Reviewing
3.9	Update organisation pages	Not start
3.10	Update user profile pages	Reviewing
3.11	Update user management pages	Reviewing
3.12	Update instance pages	No start		wjf
3.13	Update instance management pages	Not start		wjf
3.14	Update webauthn pages	Not start		wjf

Note:

1. Some items could be implemented in parallel.
2. The order of the items may be adjusted.

Validation

Item	Status	Ref	Note
Integration with AE	In progress		Update the dmpy package; test the webdav
Check consistency with existing pipelines	In progress	Make use of pipelines that we have up to now; make sure the existing users do not need to update their scripts after migration
Test on staging server	In progress		Check potential issues