Skip to content

Commit 5987edb

Browse files
authored
Update 02_data_transfers.md
adding transfer information from Greene to Torch because the google sites page is harder to access.
1 parent 9335f5f commit 5987edb

File tree

1 file changed

+31
-0
lines changed

1 file changed

+31
-0
lines changed

docs/hpc/03_storage/02_data_transfers.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,34 @@ Please only use OOD for small data transfers! Please use Data-Transfer Nodes (D
6060

6161
### FDT
6262
[FDT](https://fast-data-transfer.github.io/) stands for "Fast Data Transfer". It is a command line application written in Java. With the plugin mechanism, FDT allows users to load user-defined classes for Pre- and Post-Processing of file transfers. Users can start their own server processes. If you have use cases for FDT, visit the [download page](https://github.com/fast-data-transfer/fdt/releases) to get `fdt.jar` to start. Please contact [[email protected]](mailto:[email protected]) for any questions.
63+
64+
## Copying Data from the Greene HPC Cluster to Torch HPC Cluster
65+
### Copying Data to Torch via the Command Line
66+
Torch has an available data transfer node (DTN) that can be used to upload data. You must authenticate similarly to the steps outlined on the Accessing Torch page.
67+
68+
```
69+
netid@localpc ~ $ rsync -avz -e ssh ~/testfile.txt [email protected]:/scratch/netid/testfile.txt
70+
([email protected]) Authenticate with PIN BUUM5YHJX at https://microsoft.com/devicelogin and press ENTER.
71+
testfile.txt                      100%    0     0.0KB/s   00:00
72+
netid@localpc ~ $
73+
```
74+
## Copying Data to Torch via Globus
75+
Globus has been installed for Torch - you can log in at https://globus.org/ and search for the Torch home, scratch, or archive collections under nyu#torch.​
76+
77+
Globus works much the same as it did on Greene, so see our previous page on how to use it. Once you install Globus Connect Personal on your personal device you can easily transfer files between your local machine and Torch.
78+
79+
## File Transfer (Torch ↔ Greene)
80+
To ease data transfer between clusters, Torch storage has been mounted on the Greene Data Transfer Nodes (e.g. dtn-1.hpc.nyu.edu and dtn-2.hpc.nyu.edu).
81+
82+
This allows users to access the data under /torch:
83+
84+
```
85+
[netid@dtn-1]$ ls /torch
86+
archive home scratch share
87+
```
88+
89+
You can copy files using the rsync command from the Greene DTN:
90+
91+
```
92+
[netid@dtn-1]$ rsync /scratch/netid/my_file.txt /torch/scratch/netid/my_file.txt
93+
```

0 commit comments

Comments
 (0)