AiiDA team
Testing of v2.0.0a1 (archive migration) #5407
Unanswered
chrisjsewell
asked this question in
General
Replies: 6 comments 17 replies
-
|
Example:
Migration:
Archive statistics: metadata:
export_version: main_0001
aiida_version: 2.0.0a1
conversion_info:
- Converted from version 0.3 to 0.4 with AiiDA v1.4.1
- Converted from version 0.4 to 0.5 with AiiDA v1.4.1
- Converted from version 0.5 to 0.6 with AiiDA v1.4.1
- Converted from version 0.6 to 0.7 with AiiDA v1.4.1
- Converted from version 0.7 to 0.8 with AiiDA v1.4.1
- Converted from version 0.8 to 0.9 with AiiDA v1.4.1
- Converted from version 0.9 to 0.10 with AiiDA v2.0.0a1
- Converted from version 0.10 to 0.11 with AiiDA v2.0.0a1
- Converted from version 0.11 to 0.12 with AiiDA v2.0.0a1
- Converted from version 0.12 to main_0000 with AiiDA v2.0.0a1
- Converted from version main_0000 to main_0001 with AiiDA v2.0.0a1
mtime: '2022-03-04T20:24:51.899366'
compression: 6
creation_parameters: {}
key_format: sha256
entities:
Users:
count: 6
emails:
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
Computers:
count: 14
labels:
- bellatrix
- brisi
- 'daint (Imported #1)'
- daint-gpu
- daint-mc
- daint_aprun
- daint_mc
- 'daint_mc (Imported #0)'
- daint_old
- dora
- dora_aprun
- localhost
- theospc14-direct_
- theospc27slurm
Nodes:
count: 113078
node_types:
- data.core.array.ArrayData.
- data.core.array.bands.BandsData.
- data.core.array.kpoints.KpointsData.
- data.core.array.trajectory.TrajectoryData.
- data.core.cif.CifData.
- data.core.code.Code.
- data.core.dict.Dict.
- data.core.folder.FolderData.
- data.core.remote.RemoteData.
- data.core.singlefile.SinglefileData.
- data.core.structure.StructureData.
- data.core.upf.UpfData.
- data.forceconstants.ForceconstantsData.
- process.calculation.calcfunction.CalcFunctionNode.
- process.calculation.calcjob.CalcJobNode.
process_types:
- aiida.calculations:codtools.ciffilter
- aiida.calculations:quantumespresso.matdyn
- aiida.calculations:quantumespresso.ph
- aiida.calculations:quantumespresso.pw
- aiida.calculations:quantumespresso.q2r
Groups:
count: 16
type_strings:
- core
Comments:
count: 0
Logs:
count: 0
Links:
count: 164968
repository:
objects:
count: 205805Import:
Import statistics: entities:
Users:
count: 7
emails:
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
Computers:
count: 14
labels:
- bellatrix
- brisi
- 'daint (Imported #1)'
- daint-gpu
- daint-mc
- daint_aprun
- daint_mc
- 'daint_mc (Imported #0)'
- daint_old
- dora
- dora_aprun
- localhost
- theospc14-direct_
- theospc27slurm
Nodes:
count: 113078
node_types:
- data.core.array.ArrayData.
- data.core.array.bands.BandsData.
- data.core.array.kpoints.KpointsData.
- data.core.array.trajectory.TrajectoryData.
- data.core.cif.CifData.
- data.core.code.Code.
- data.core.dict.Dict.
- data.core.folder.FolderData.
- data.core.remote.RemoteData.
- data.core.singlefile.SinglefileData.
- data.core.structure.StructureData.
- data.core.upf.UpfData.
- data.forceconstants.ForceconstantsData.
- process.calculation.calcfunction.CalcFunctionNode.
- process.calculation.calcjob.CalcJobNode.
process_types:
- aiida.calculations:codtools.ciffilter
- aiida.calculations:quantumespresso.matdyn
- aiida.calculations:quantumespresso.ph
- aiida.calculations:quantumespresso.pw
- aiida.calculations:quantumespresso.q2r
Groups:
count: 17
type_strings:
- core
- core.import
Comments:
count: 0
Logs:
count: 0
Links:
count: 164968
repository:
SHA-hash algorithm: sha256
Compression algorithm: zlib+1
Packs: 0
Objects:
unpacked: 205805
packed: 0
Size (MB):
unpacked: 4002.8430767059326
packed: 0.0
other: 0.01171875 |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thanks to Valeria, here's a list of all .aiida files on the Materials Cloud Archive with links. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Found an issue while migrating: Output: Report: Legacy migrations required from zip format
Report: Extracting data.json ...
Report: Legacy migration pathway: 0.10 -> 0.11 -> 0.12
Report: legacy '0.12' -> 'main_0000' conversion required
Report: Initialising new archive...
Report: Unique repository files written: 263897
Report: Converting DB to SQLite
Report: Performing SQLite migrations:
Report: - main_0000 -> main_0000a
Report: - main_0000a -> main_0000b
Report: - main_0000b -> main_0001
Report: Finalising the migration ...
Critical: failed to migrate the archive file (use `--verbosity DEBUG` to see traceback): RuntimeError:File size unexpectedly exceeded ZIP64 limit |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
Worked like a charm! $ verdi archive migrate export_discovery_cof_22Feb21.aiida test2-export_discovery_cof_22Feb21.aiida 2>&1 | tee -a test2-migrate.log
Report: Legacy migrations required from zip format
Report: Extracting data.json ...
Report: Legacy migration pathway: 0.10 -> 0.11 -> 0.12
Report: legacy '0.12' -> 'main_0000' conversion required
Report: Initialising new archive...
Report: Unique repository files written: 263897
Report: Converting DB to SQLite
Report: Performing SQLite migrations:
Report: - main_0000 -> main_0000a
Report: - main_0000a -> main_0000b
Report: - main_0000b -> main_0001
Report: Finalising the migration ...
Success: migrated the archive to version 'main_0001' |
Beta Was this translation helpful? Give feedback.
-
|
Cheers! Yeh I guess it would be good to also now check that it can be read etc |
Beta Was this translation helpful? Give feedback.
-
|
Running an import now 👍 |
Beta Was this translation helpful? Give feedback.
-
$ verdi archive import test2-export_discovery_cof_22Feb21.aiida -G test2
Report: starting import: test2-export_discovery_cof_22Feb21.aiida
Report: Parameters
------------------------------- ----------------------------------------
Archive test2-export_discovery_cof_22Feb21.aiida
New Node Extras keep
Merge Node Extras (in database) (k)eep
Merge Node Extras (in archive) do (n)ot create
Merge Node Extras (in both) (l)eave existing
Merge Comments leave
Computer Authinfos exclude
Report: Adding 1 new user(s)
Report: Adding 11 new computer(s)
Report: Collecting Node(s) ...
Report: Adding 586312 new node(s)
Report: Adding 242376 new log(s)
Report: Gathering existing 'create' Link(s)
Report: Added 283978 new 'create' Link(s)
Report: Gathering existing 'return' Link(s)
Report: Added 175378 new 'return' Link(s)
Report: Gathering existing 'input_calc' Link(s)
Report: Added 496716 new 'input_calc' Link(s)
Report: Gathering existing 'input_work' Link(s)
Report: Added 451147 new 'input_work' Link(s)
Report: Gathering existing 'call_calc' Link(s)
Report: Added 123453 new 'call_calc' Link(s)
Report: Gathering existing 'call_work' Link(s)
Report: Added 50808 new 'call_work' Link(s)
Report: Adding 632 new group(s)
Report: Adding 12586 Node(s) to new Group(s)
Report: Using existing import Group: PK=64, label=test2
Report: Checking keys against repository ...
Report: Skipping 1 existing repository files
Report: Adding 263896 new repository files
Report: Committing transaction to database...
Success: imported archive test2-export_discovery_cof_22Feb21.aiida |
Beta Was this translation helpful? Give feedback.
-
|
Also tried querying, loading nodes etc. Seems ok! |
Beta Was this translation helpful? Give feedback.
-
|
More trouble in paradise: https://archive.materialscloud.org/record/file?record_id=1120&filename=gamma_Fe.aiida Here is the output: Report: Legacy migrations required from zip format
Report: Extracting data.json ...
Report: Legacy migration pathway: 0.10 -> 0.11 -> 0.12
Report: legacy '0.12' -> 'main_0000' conversion required
Report: Initialising new archive...
Critical: failed to migrate the archive file (use `--verbosity DEBUG` to see traceback): error:Error -3 while decompressing data: invalid code lengths set |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
Haha, fair. I'm just running all of them first, will rerun with the flag afterwards 😅 |
Beta Was this translation helpful? Give feedback.
-
|
👍 |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, now one of the migrations seems stuck. Let me check if it is still doing anything. Either way, I'll try the problematic migration above again with |
Beta Was this translation helpful? Give feedback.
-
|
Full output of running with any ideas? |
Beta Was this translation helpful? Give feedback.
-
|
@mbercx this archive appears to be corrupt. Perhaps you can also try? |
Beta Was this translation helpful? Give feedback.
-
|
Seems the migration I mentioned above is really stuck... Here's the URL to the archive: https://archive.materialscloud.org/record/file?record_id=1005&filename=export_Nb_KKR-BdG.aiida I'll try to stop it and rerun with |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Wait, nvmd 😅 . When I pressed any button in the screen it suddenly is moving again. Seems the progress bar just froze, so ignore this one. Big archives though, I've been running these migrations for a while now. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, for reference, I'm adding here a report of the SSSP archive import, which is a bit problematic. Archive size: 14 GB
PPLOT import stdout: and the corresponding image: (Ignore the straight line, my computer was hibernated during that time.) The archive file is located on castor, in the container |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks @eimrek that very helpful 👍 |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Similar to #5379, this is intended for collecting reports on archive migrations.
Steps:
developinto an virtual environmentpip install process-plot(orpipx)pplot exec -i 0.5 -c screen --title "Archive migrate" "/path/to/bin/python -m aiida archive migrate -f -v info /path/to/input.aiida test.aiida"data.json, into memory, but future migrations ondb.sqlite3will not require this, so the memory profile will be a lot lower)verdi archive info --statistics test.aiidaverdi quicksetup --profile test_importpplot exec -i 0.5 -c screen --title "Archive import" "/path/to/bin/python -m aiida -p test_import archive import -v info test.aiida"verdi -p test_import storage info --statisticsBeta Was this translation helpful? Give feedback.
All reactions