-
Notifications
You must be signed in to change notification settings - Fork 39
Vorna guide
The server is at turso.cs.helsinki.fi, which is visible from e.g. login.physics.helsinki.fi. On your local (linux) computer you can add the following in your ~/.ssh/config file
Host turso.cs.helsinki.fi
ProxyJump <username>@melkinkari.cs.helsinki.fi
Replace with your UH user name. With the 2 lines above you can then access turso directly from your computer.
This is for new users without a /proj folder – replace the /wrk-vakka/users paths with /proj if you like.
module load Python/3.7.2-GCCcore-8.2.0 # or the version you want/need
mkdir /wrk-vakka/users/$USER/.local
export PYTHONUSERBASE=/wrk-vakka/users/$USER/.local
export PYTHONPATH=$PYTHONPATH:/wrk-vakka/users/$USER/.local
pip3 install <your package> --cache-dir /wrk-vakka/users/$USER/.cache --user
Update 25.05.2023: Configuring the path to the allas tools manually is no longer required, instead you can do
module load CSC/allas-cli-utils/1.0
allas-conf -u <your CSC username>
and then enter your password, and select the correct project.
module load Python/3.7.2-GCCcore-8.2.0
mkdir /proj/$USER/.local
export PYTHONUSERBASE=/proj/$USER/.local
export PYTHONPATH=$PYTHONPATH:/proj/$USER/.local
pip3.7 install python-openstackclient --cache-dir /proj/$USER/.cache --user
pip3.7 install python-swiftclient --cache-dir /proj/$USER/.cache --user
If e.g. pip doesn't find The Internet, add these to the exports or your bashrc:
export https_proxy=http://www-cache.cs.helsinki.fi:3128
export http_proxy=http://www-cache.cs.helsinki.fi:3128
export HTTP_PROXY=http://www-cache.cs.helsinki.fi:3128
export HTTPS_PROXY=http://www-cache.cs.helsinki.fi:3128
As of Sep. 2021, if installation fails on some ssh error on turso03, try the same procedure through turso01. If you get a long stack of errors ending in a gcc failure, make sure to load both the GCC and GCCcore modules with the version matching the one of your Python module.
- Add the following lines to your
~/.bashrcfile on Vorna (replace your CSC username in the last one):
module load Python/3.7.2-GCCcore-8.2.0
export PATH=$PATH:/proj/group/spacephysics/appl/rclone
export PATH=$PATH:/proj/group/spacephysics/appl/allas-cli-utils
export PATH=$PATH:/proj/$USER/.local/bin
alias allas-conf="source allas_conf -u <CSC username here>"
- Reload your .bashrc with
source ~/.bashrc
- Run
allas-conf, enter your CSC password, and select project_2000506
You should now have a connection to the correct Allas project container.
To see what files are available on Allas, go to https://pouta.csc.fi/dashboard/project/, login with your CSC username and set it to project_2000506 from the dropdown menu in the top left. Alternatively, you can use rclone lsd allas: to see the top-level directory and then navigate with increasingly specific rclone lsd commands.
- Example: Moving bulk file 2261 of run BFD to corresponding folder on Vorna
rclone copyto allas:vlasiator-2d-bfd/bulk/bulk.0002261.vlsv /wrk/group/spacephysics/vlasiator/2D/BFD/bulk/bulk.0002261.vlsv -P
where -P stands for "progress". Omit it if you want the transfer to be quiet.
- Example: Moving all bulk files of BFD to Vorna
rclone copy allas:vlasiator-2d-bfd/bulk/ /wrk/group/spacephysics/vlasiator/2D/BFD/bulk/ -P --checkers 16 --transfers 16
where --checkers and --transfers define how many simultaneous processes to use for checking files and transferring them, respectively. 16 works well if you are using 8 cores with 2 threads per core, for instance.
Adding the parameter --dry-run lets you see what rclone would do without actually transferring or changing any files. Very useful when using filters or worrying about accidentally overwriting files.
Some Vlasiator bulk files are only available on Datacloud currently. Configuring a Datacloud connection is slightly different from Allas. Firstly, to see what the folder structure is like on Datacloud and to find the files you need, go to https://datacloud.helsinki.fi, log in with your UH credentials, and go to the shared folder named spacephysics. Then, to configure the connection (assuming you have rclone installed):
-
Run
rclone config. Assuming you already set up a connection to Allas, you should see "Allas" as one of the existing configurations. If you also have a configuration called "Datacloud", your configuration is already set up, and you can jump ahead to step 3 of this guide. -
If the Datacloud configuration is not already set up, please follow the steps in the "Setting up rclone" section of the following guide https://wiki.helsinki.fi/pages/viewpage.action?spaceKey=Tutkimusdata&title=Uploading+large+data+sets+to+Datacloud+via+WebDAV
-
Done. You can now start transferring files in the way described in the "Moving Files" section of the Allas guide. Just make sure to point
rclone copyorcopytoatdatacloudand its subdirectories instead ofallas.
The Allas tools should be available as a module on all of these services, so to get access simply run
module load allas
allas-conf project_2000506
and enter your CSC password. Then, to transfer a directory from the server you're using to Allas, for instance subdirectory AFC/visualizations from Taito/Datamangler to Allas, run the following:
rclone copy /proj/vlasov/2D/AFC/visualizations/ allas:vlasiator-2d-afc/visualizations/ -P --checkers 16 --transfers 16
For more detailed instructions and options, see the Moving files section of this guide.
Example: Moving bulk files 500 to 1078 of AFC from Taito to Allas.
- First you need to create a list of files to include. Run
for n in {0500..1078}
do
echo "bulk.000$n.vlsv" >> ~/include_file.txt
done
- Perform the transfer, with the additional flag
--include-from ~/include_file.txt:
rclone copy /proj/vlasov/2D/AFC/production_halfres/ allas:vlasiator-2d-afc/production_halfres/ -P --checkers 16 --transfers 16 --include-from ~/include_file.txt
- After the transfer is done, you can delete
include_file.txtfrom your home directory. Note: If you don't delete it, the next time you run step 1, the text will be appended to the old file.
Wildcards can be used for selecting transferred files by providing the directory and indicating the requested files with --include
rclone copy /proj/vlasov/2D/AFC/production_halfres/ --include "bulk.0000[7-9]??.vlsv" allas:vlasiator-2d-afc/production_halfres/ -P --checkers 16 --transfers 16 --include-from ~/include_file.txt
Allas has a maximum file size of 5GB, but because Allas has server-side auto-chunking - meaning that files that exceed this limit are automatically split into 5GB chunks when uploaded to Allas - you don't normally have to worry about this. However, if you're uploading a very large file, say a restart file whose size is measured in TB, transferring this file will take a very long time because parallel transfers can't be used.
Thus you may want to manually split the file into smaller chunks before uploading to Allas.
Example: Transferring the run AFC file restart.0000578.2019-04-19_02-03-29.vlsv from Taito to Allas in 4GB chunks.
- Make a new directory (call it whatever you want) to hold the chunks:
mkdir /proj/vlasov/2D/AFC/production_halfres/restart_afc578_chunks
- Split the file with the following command:
split --verbose -d -b 4G -a 4 /proj/vlasov/2D/AFC/production_halfres/restart.0000578.2019-04-19_02-03-29.vlsv /proj/vlasov/2D/AFC/production_halfres/restart_afc578_chunks/restart.0000578.2019-04-19_02-03-29.vlsv_
This will create a sequence of files of format restart.0000578.2019-04-19_02-03-29.vlsv_<4 digit number starting from 0000 here> to the directory you created earlier. The suffix consists of 4 (due to flag -a 4) decimal numbers (due to flag -d), and the flag -b 4G specifies a chunk size of 4GB. --verbose lets you follow the progress of the splitting process.
- When the splitting is done, you can move on to transferring the chunks to Allas:
rclone copy /proj/vlasov/2D/AFC/production_halfres/restart_afc578_chunks/ allas:vlasiator-2d-afc/production_halfres/restart_afc578_chunks/ -P --checkers 16 --transfers 16
To download the chunks somewhere else, for instance Vorna, follow the "Moving Files" section of this guide (don't forget to configure the Allas connection first) and download the restart_afc578_chunks/ directory. Then, navigate to restart_afc578_chunks/ and run the following command:
cat restart.0000578.2019-04-19_02-03-29.vlsv_{0000..9999} > restart.0000578.2019-04-19_02-03-29.vlsv
(Note: Ending the range at 9999 ensures that all chunks are concatenated to the output file. You can of course replace it with the suffix of the last file if you want. For instance, if you have 400 files, you can use {0000..0399} instead.)
The restart file should now work properly.
If you need a more specific subset of files transferred or need to use more advanced parameters, please consult the rclone documentation at https://rclone.org/docs/
See https://wiki.helsinki.fi/display/it4sci/HPC+Environment+User+Guide#HPCEnvironmentUserGuide-4.2InteractiveSessions as the interactive command use is deprecated.
Vorna latex installation has some remaining issues. Jonas has a functional texlive installation for this. Analysator/Matplotlib (probably!) works with latex with these environment settings:
export PATH=/wrk-vakka/group/spacephysics/proj/appl/tex-basic/texlive/2023/bin/x86_64-linux:$PATH
module load matplotlib
You might need to add module load Perl before module load matplotlib for some cases(?).
If you get errors such as ! LaTeX Error: File 'underscore.sty' not found., send the error to Jonas so the missing package can be installed.