Accessing and downloading data

When things get sequenced, the sequencing facility will store the data, dozens (or hundreds) of gigabytes in size, on a secure server and give you instructions on how to download your data. This process will almost never be clicking a link, and will instead require the use of the curl wget or sftp commands in a *NIX system (Mac/Linux).

curl and wget

These commands are largely straighforward, and if the sequencing facility uses this method, they will pretty much just give you a link. To download this file to a specific directory, open up the terminal, cd into wherever you want it to be downloaded (e.g. /home/username/data/) and the syntax will be:

curl

curl -O the.url.provided

• -O is "save the output of the file"

If you wanted to have it running in the background (in case someone accidentally closes the terminal session):

curl -s -O the.url.provided &

• -s is "run silently"
• & is "run in the background"

wget

wget -b the.url.provided

• -b is "run in background"

SFTP

Some facilities choose to use the SFTP method (Secure File Transfer Protocol), which means they give you access to some kind of public server of theirs, you log into it, navigate to what you need, and download it from there onto the machine you are using. Fancy! It's easier than it sounds, and there are just a few commands you'll need to make it work.

Logging in

sftp username@remote.host.name

• the sequencing facility will provide you with a username and a host name (the server you're accessing)
• if it's your first time accessing a particular server, it will ask you to establish an autogenerated SHA5 key, which is just an extra layer of security for these kinds of connections, so just say Yes. Easy peasy.
• you will then be prompted for a password, which the sequencing facility should have also provided
• your terminal line will look a little different than usual, in that it will now display <sftp> at the start of your line. That means you're in!

Once you're in

When you're remotely accessing something with sftp, a new set of commands are available to distinguish navigating the remote server, and your local machine. There are more commands than the ones provided below, but these will just be the likeliest ones you'll need.

• cd this now changes directories in the remote server you've accessed and not your local machine
  • cd example.directory
• lcd use this to change the directories on your local machine (going to the folder you want data to be downloaded into)
  • lcd /home/username/Data/
• get this copies a file from the current remote directory to the current local directory
  • get example.file
• get -r this copies a folder from the current remote directory to the current local directory
  • get -r example.folder
  • get -r * this would download all folders/files from your remote directory into your local one