Input data backup from betzy

[MichaelSchulzMETNO]

goal: Save betzy NorESM input files to nird /projects/NS9560K/www/inputdata

steps:

1. Identify "NorESM input files" on betzy:/cluster/shared/noresm/inputdata of group owner noresm
   which are NOT in NCAR copy : /datalake/NS12077K/CESM-input-data
   Exclude some files: svn, lock, etc

2. Rsync those "NOT-in-NCAR NorESM input files" to /projects/NS9560K/www/inputdata
   2b) identify problems with permissions - communicate with sigma2 to make all files readable for NS9560K
   2c) rsync will only copy new files on www/inputdata
   2d) Only the rsync script should add files on www/inputdata

3. Users on betzy can add files to this process by making files and directories owned by group "noresm"

4. Make a repository under NorESMhub with scripts for steps 1+2 @jgriesfeller

5. Run script regularly (tbd) (Jan)

5b) Manually correct files on /projects/NS9560K/www/inputdata on demand with Data management group

6. Update NCAR files now and then (Marit?)

7. Look at input files from other machines , on demand

8. Move eventually www/inputdata to a specific input data project

Please comment @maritsandstad @gold2718 @tylov @jgriesfeller @DirkOlivie @monsieuralok @matsbn @lisesg

[gold2718]

One way to identify missing files is to run tests with DIN_LOC_ROOT set to an empty directory. This can be done both with standard regression tests and with a NorESM3 production setup. Any file that does not clear running ./check_input_data after ./case.setup needs to be copied to /projects/NS9560K/www/inputdata

One current issue with this method on Betzy is NorESMhub/ccs_config_noresm#70.

[MichaelSchulzMETNO]

Our idea was that the betzy directory would be the primary master copy of what we need. The www/inputdata directory would then get any missing files slowly also from betzy by the rsync script. So any file that is missing should be copied to betzy to my understanding. Or is that not logic?

[gold2718]

I think your logic is spot on (or at least matches mine). The hole is that there are some NorESM-only files that are created on Betzy and need to be copied by hand to the NIRD location. The test I outline above is just a well-tested method for finding any missing files (i.e., that are not on any server including NIRD).

[MichaelSchulzMETNO]

NorESM-only files will be copied to nird if they are owned by the group "noresm", using the unix attribute. So if one identifies missing files with your method, they should be copied to betzy and then the group ownership eventually has to be changed to noresm by the one finding missing files (or another script?)

[maritsandstad]

I have been doing 6 semi-sporadically @MichaelSchulzMETNO . Do you want some more regular update schedule?

[MichaelSchulzMETNO]

I am not sure how much that NCAR directory changes. Bimonthly sounds good to me.

[jgriesfeller]

while looking around a bit I noticed that all data in /datalake/NS12077K/CESM-input-data is owned by the group ns9560k and not by the group ns12077k. Doesn't that mean that the data is accounted to the ns9560k project? Or am I missing something?

[tylov]

No, I think it only matters under which project they are stored under (for who they are accounted to), but yes the wrong group ownership can be a problem regarding access.

[jgriesfeller]

Potential show stopper:
We want to automise the copying of new input data from betzy to nird using cron and rsync. But isn't the two factor authentication preventing that?
According to https://www.sigma2.no/documentation/getting_help/two_factor_authentication.html only scp can be run circumventing 2FA. This excludes the usage of rsync.

[TomasTorsvik]

nird is mounted on betzy. I use rsync regularly from betzy to nird without 2FA, using the local mount point. However, the permissions often get messed up, so sometimes I need to fix this afterwards on nird.

[jgriesfeller]

That will actually work.
But it means that betzy's login nodes will do the copying and not nird's as we wanted. Do we have any experince running cron jobs on betzy?

[TomasTorsvik]

Yes, that is right, so it is not optimal. I have never tried to run cron jobs on betzy.

[rosiealice]

@TomasTorsvik that's interesting. When I try rsync from betzy to nird it always asks for 2FA which is VERY annoying! What trick do you use to circumvent it?

[jgriesfeller]

the nird file systems are below /nird on betzy (login nodes only)

[TomasTorsvik]

If you look in the root folder on betzy, you will find a directory /nird that mirrors the nird storage servers.
(see https://documentation.sigma2.no/files_storage/nird/mounts_lmd.html)

NOTE1: /nird/datapeak is mounted on /nird/projects

NOTE2: The /nird mount point is not accessible on betzy work nodes, only login nodes, because the work nodes only see directories that are under the /cluster folder. At least, that is how it is on betzy.

[tylov]

NOTE1: /nird/datapeak is mounted on /nird/projects

You can also use /nird/projects from betzy. For scripts, you can use /nird/projects, /nird/datalake from both betzy and nird, and it will work on both machines (unless you use modules that is only on betzy).

[MichaelSchulzMETNO]

"Automising" is not the only goal. First priority is to rsync and secure the specific NorESM input data. Updating such securisation procedure comes after that.

[jgriesfeller]

Just for documentation:
The following input data directories are not readable by me although I was searching for group ownership for the noresm group only (betzy):

find: ‘/cluster/shared/noresm/inputdata/BlueAction/NHIST_f19_tn14_Init01_rest1900’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/old_output’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMCNRMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMNORESMhistorical’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMCNRMhistorical’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/ICONNORESMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/ICONCNRMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMNORESMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/ICONNORESMhistorical’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/lnd/clm2/rawdata/.svn’: Permission denied

If someone feels responsible for those, please either adjust permissions (readable for the group noresm) or change group ownership to a group that is NOT noresm.
I will ask sigma2 to to adjust their permissions once my script has progressed more.

[MichaelSchulzMETNO]

@lisesg @JorgSchwinger @kjetilaas ??

[lisesg]

I think the owner of these files is Iris Mužić at CICERO. I can contact her.

[jgriesfeller]

the PolarRES data is readable now.

[jgriesfeller]

Some very preliminary info regarding the amount of input data files on betzy that don't have backup:

Info: 639 source files were found in the backup folder.
Info: 19025 source files were found in the NCAR archive folder.
Info: This leaves 203853 source files to be copied.

Is this high number expected?
I haven't checked this very intensively though, but the fact that some matching files are found sort of indicates that my algorithm is working...
At the moment files matching these patterns have been excluded already:

SOURCE_PATH_EXCLUDE_LIST = [".svn/*", "*/.svn/*", "*.lock"]

Please suggest more file patterns for removal...

[jgriesfeller]

applying that additional exclude criterion leaves this:

Info: 639 source files were found in the backup folder.
Info: 19025 source files were found in the NCAR archive folder.
Info: This leaves 132243 source files to be copied.

If noone comes up with further comments, I will run the copy script as soon as I have actually added the copying functionality... Still looking for the most efficient way to do that with Python...

[jgriesfeller]

just to be sure:
the directory /cluster/shared/noresm/inputdata/restart is supposed to be backed up as well?

[MichaelSchulzMETNO]

do we need that backupped?? maybe not ? @DirkOlivie @TomasTorsvik ??

[oyvindseland]

I do not think it needs backup since it is a copy of the original output in the first place.

[TomasTorsvik]

@oyvindseland - I agree, at least for the NOIIAJRA* data. I suppose for N20TRAERCN we only have one copy (in NS2345K)? I don't think we will be able to reproduce these datasets bit-for-bit in the future, if there ever is a need.

[oyvindseland]

I think it is copied to the research archive though.

[jgriesfeller]

Hopefully the last post before copying...

The following list is the files below the path /cluster/shared/noresm/inputdata that my script would copy to nird for the moment: files_to_copy.txt
It's all in all 33TB of data and 226390 files.

Please confirm that these are the files that are supposed to be copied to nird

dusage for ns9560k on nird (that's where the data would go) as additional information:

====================================================================================
Project      Resource  Type    Usage       Soft Limit   Hard Limit               
NS9560K      datalake  Disk    752.247 TiB  800 TiB      800 TiB                   
NS9560K      datalake  Files   6095194      6488064      8957952                   

This would leave around 12TB and roughly 166480 files (soft quota) of the ns9560k quota.

[TomasTorsvik]

It looks correct to me. I think we might have to clean up/remove some of the older data on the ns9560k project. Probably, most of the NorESM2.5_alphaXX datasets are outdated now, except for long reference runs?

[jgriesfeller]

It has been decided to backup the inputdata to from betzy to the path /nird/datalake/NS12077K/NorESMInputdata for now.
I will start the backup after some more tests shortly.

[jgriesfeller]

Next question:
While testing the backup I noticed that e.g. the file atm/cam/solar/polarForcing1995-2005avg_c160929.nc-3650289664-2613228.loc disappeared.
Can these .loc files be added to the ignore list?

[jgriesfeller]

How about PAMIP/N2000_PAMIP_6_2_v1/Buildconf or CaseDocs directories? Are these supposed to be in the inputdata directory?

[MichaelSchulzMETNO]

@lisesg can you comment possibly?

[lisesg]

@AleksiNummelin: we are backing up the Betzy input data, do you want PAMIP/N2000_PAMIP_6_2_v1/Buildconf and/or CaseDocs to be included?

[jgriesfeller]

Just for info:
The initial backup has been started. It should finish by Friday.

[jgriesfeller]

Lorand just made it clear that the inputdata backup should not be located within the ns12077k project.
But since I just got access to the S3 based NS16001K project, I will shift my efforts to that.

[jgriesfeller]

Copying the the data to the new location (/nird/datalake/NS16001B/cdl-ns16001b-NorESMInputdata) now.

[jgriesfeller]

The initial backup has finished. Please check and report errors.

Note that

• the data can only be altered by members of the group ns16001b
• all executable permissions on files have been removed
• all data is readable for the world as this is needed for future web access

I will now polish the code and the documentation and try to setup a cron job that runs the backup on a weekly basis.

[TomasTorsvik]

@jgriesfeller - I don't have read access to the NS16001B storage space on Nird. Is this supposed to be through SSH, or is there some other procedure for S3 access (I haven't used S3 before)? Or perhaps it is not open for read access?

[jgriesfeller]

Unfortunately /nird/datalake/NS16001B is not world readable as it should. Sorry for missing that.
@tylov : I guess it's you as project owner who has to ask sigma2 to open the project for world access. Can you maybe ask for it?
NS9560K is open btw, so there is an example.

[jgriesfeller]

Next question:
There's a lot of file with executable permissions below /cluster/shared/noresm/inputdata.
Should there be any executable file there? The name seems to indicate that thats only data and no scripts.
I tend to remove all executable bits at the backup path. Any subjections?

[TomasTorsvik]

I don't see a reason for executable permissions on data files, should be safe to remove the executable bits.

[jgriesfeller]

Just for info:
I have just created the initial working version of the input data management tool for betzy: https://github.com/NorESMhub/inputdatamanagement
Please read through the minimal documentation (the README.md file) and report potential errors.
I'm waiting for sigma2 to open the backup directory for world and for the web accessibility as we have on ns9560k.

[TomasTorsvik]

@jgriesfeller , @tylov
Now we have two repositories:

• https://github.com/NorESMhub/inputdatamanagement
• https://github.com/NorESMhub/inputdata_management
  It seems the second one of these is not needed?

[jgriesfeller]

Indeed. The second was the first try I guess...
I just deleted it

[tylov]

There are two folders in /datalake/NS16001B, where the second is referred to from the www folder:

cdl-ns16001b-noresminputdata
cdl-ns16001b-NorESMInputdata

[jgriesfeller]

What's the question? I'm still testing things and wasn't sure if the S3 stuff likes CamelCase...
I will remove everything not needed once the testing has finished.

[jgriesfeller]

The inputdata backup is now reachable via this web address:
https://ns16001b.web.sigma2.no/datalake/inputdata/

All data from https://ns9560k.web.sigma2.no/datapeak/inputdata/ should also be there at the new location (except for the empty folder testfolder). Please confirm that data you need has been correctly copied.
If that is done, the redirect from noresm.org should be updated to the new location and we can free up a little space for the ns9560k project.

[gold2718]

In trying to update and test inputdata, I found a few files which are not backed up:
On Betzy (in /cluster/shared/noresm/inputdata)

-rw-rw-r-- 1 mdeb noresm 185461425 Dec 10 18:24 lnd/clm2/cropdata/calendars/processed/gdd20_baseline.c251120.tweaked_latlons.nc
-rw-rw-r-- 1 mdeb noresm    206288 Dec 10 18:21 lnd/clm2/paramdata/ctsm60_params.c251124.nc
-rw-rw-r-x 1 mdeb noresm 483085256 Dec 11 09:48 lnd/clm2/surfdata_esmf/ctsm5.4.0/surfdata_ne16np4.pg3_hist_1850_78pfts_c251022.nc

These files all have group, noresm and are world readable. On NIRD, I see directories created but no files copied.

On NIRD (/nird/datalake/NS16001B/inputdata), these are the directory entries:

drwxrwsr-x 2 jang ns16001b  4096 Dec 10 18:24 lnd/clm2/cropdata/calendars/processed
drwxrwsr-x 3 jang ns16001b 16384 Dec 18 11:27 lnd/clm2/paramdata
drwxrwsr-x 2 jang ns16001b  4096 Dec 17 13:27 lnd/clm2/surfdata_esmf/ctsm5.4.0

The cropdata/calendars/processed and surfdata_esmf/ctsm5.4.0 folders are empty. The paramdata folder contains earlier entries but not the one listed above.

I believe the backup script has run since 10. December. Are these files not configured correctly or is there an issue with the backup script?
@jgriesfeller, @mvdebolskiy

[mvdebolskiy]

Is there a separate discussion for Olivia?
If not:
Since there is no /shared/noresm/ on Olivia anymore, It is Ok to set DIN_LOC_ROOT to /cluster/work/projects/nn9560k/inputdata. If the config_inputdata.xml has nird as a server. There is no need for data to be locally on olivia all the time if it is not used there. The download from nird should not take long, since the machines are in the same room basically. The only times when people will need to download lots of data (and not just ~few Gb) is when someone is running high resolution experiments for which they have to make their own forcing/inputdata (f.e. for land), and this high-res data has no place in any shared space (unless lots of people are using it) and should not be a responsibility for INES2 anyway. If the model configuration is not part of NorESMhub default/release branches -- input data for that configuration has no business just lying around in the storage spaces paid by INES2.

[matsbn]

Sigma2 wants us to use the CDL service for inputdata on Olivia. To my understanding, this service is still in a pilot phase, so it could take some time before it is up and running in a satisfactory manner. Looking at https://documentation.sigma2.no/files_storage/clusters.html#storage-areas, I agree /cluster/work/projects/nn9560k/inputdata could be a good interim location.

[jgriesfeller]

Small update:
I asked sigma2 about cron jobs on the HPCs. They suggest to use the undocumented scrontab command which basically schedules jobs via the queue as cron does. For the input data backup this doesn't help, because the nird filesystems below /nird are deliberately not mounted on the compute nodes (see here).

[MichaelSchulzMETNO]

Can you explore with sigma2 how to do a regular automatic update then?

[jgriesfeller]

This discussion is ongoing.

[jgriesfeller]

Conclusion according to sigma2: The automatic backup of the current input data from betzy is not supported. The job is too huge. Entire message (not entirely understandable without the entire thread, but you will get the essence):

Well, yes, this is not going to work. Betzy compute nodes do not see NIRD partitions. So the only solution would be to place the cronjob on a login node. Yes, I see that the data you want to rsync is huge and this may kill the login nodes. Nobody will authorise this. scrontab only works to run jobs on compute nodes.

The solution will be to clean the /cluster/shared/noresm/inputdata partition, make it shrink, then copy it part by part to NIRD and then start a cronjob on the login nodes which will only copy the new files on Betzy. All this provided the volume of the new files remains reasonable.

Sorry for this.

Best regards

Nikolay

I will just run the script by hand with a calendar reminder

[maritsandstad]

Doesn't this mean, that if it is just running a diff for new files if most are already transferred, it's ok? That's how I read that...

[oyvindseland]

Does rsync really compare the whole set of files at the same time / or add overhead if you go through a large number of files?
Unless the files are really large I do not see how it can impact the login node that much?
Or is it the cron job that is the problem? That I can actually see. I do not know much about the linux systems but I have experienced systems that had a hard time with cron jobs even when it only woke up once a day.

[jgriesfeller]

In the standard settings rsync just compares file times and sizes not the hash (meaning it doesn't read the files). But the job is quite intrusive on the file system because it looks at these data for all files (around 1.5 million in total in our case). That's what other users will feel. Let's see this as a design decision on sigma2's side and just accept it for now.

[jgriesfeller]

Just for info:
The quota of the inputdata backup project (NS16001B) has been exhausted.
The sub directory cicero_mods contains 116TB alone (out of a total 150TB quota). Was that intended?

Looking a bit further, the main thing is the directory cicero_mods/ERA5-Land with 115TB including some directories that look redundant at first glance.

Please advise what's really needed in the backup. The script can exclude directory structures.

2,7T	cicero_mods/ERA5-Land/TPQWL3Hrly_old
683G	cicero_mods/ERA5-Land/Prec3Hrly_old
12T	cicero_mods/ERA5-Land/calculated_vars
683G	cicero_mods/ERA5-Land/Solar3Hrly_old
13T	cicero_mods/ERA5-Land/original_GRIB_files
6,6M	cicero_mods/ERA5-Land/prep_for_plotting
1,3T	cicero_mods/ERA5-Land/regrid_ERA5Land_to_CRUJRA
77T	cicero_mods/ERA5-Land/single_vars
8,0T	cicero_mods/ERA5-Land/masked_ocean
115T	cicero_mods/ERA5-Land

[maritsandstad]

@jgriesfeller For now maybe exclude all of cicero_mods from backup? There is also way too much here which needs to be cleared away, we are working on that offline.

[jgriesfeller]

fine with me. Just let me know once you want to add things to the backup again.

[jgriesfeller]

JFYI: This has been implemented and the backup works again. Current quota usage is 34.405 TiB

[MichaelSchulzMETNO]

@kjetilaas @maritsandstad can you clarify?

[maritsandstad]

See comment above @MichaelSchulzMETNO