Skip to content

Commit 60b038d

Browse files
liamjberrisfordliamjberrisford
authored
Merge pull request #293 from coding-for-reproducible-research/285_additional_intro_to_hpc
Integrate additional intro to HPC content
2 parents 1e7f40c + 32d3a06 commit 60b038d

File tree

4 files changed

+211
-48
lines changed

4 files changed

+211
-48
lines changed

‎individual_modules/introduction_to_hpc/connection.ipynb

Lines changed: 175 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -11,88 +11,215 @@
1111
"tags": []
1212
},
1313
"source": [
14-
"# How do I connect to a HPC Cluster?\n",
14+
"# Getting Started with ISCA\n",
15+
"\n",
1516
"## Learning Objectives\n",
16-
"- Connect to the ISCA HPC system using university credentials\n",
17-
"- Create an SSH `config` file for easier access via the command line\n",
17+
"\n",
18+
"- Connect to the ISCA HPC system using SSH\n",
1819
"- Verify the installation of the SSH application on various operating systems\n",
19-
"- Understand the steps involved in logging into an HPC cluster for the first time\n",
20-
"- Utilize graphical interfaces like Putty for connecting to HPC systems\n",
20+
"- Create an SSH config file for easier access via the command line\n",
21+
"- Create and use an identity key for password-less login\n",
22+
"- Copy files using secure copy (`scp`)\n",
23+
"- Use graphical interfaces like PuTTY for connecting to HPC systems\n",
24+
"\n",
25+
"---\n",
26+
"\n",
27+
"## ISCA Compute Resources\n",
28+
"\n",
29+
"- A cluster of standard CPU nodes (Linux) (256 GB) \n",
30+
"- Large memory (3 TB) nodes \n",
31+
"- Xeon Phi accelerator nodes and GPU (Tesla K80) compute nodes \n",
32+
"\n",
33+
"### Fair Usage\n",
34+
"\n",
35+
"Central HPC systems provide powerful resources, but they are shared among many users. Both **compute** and **storage** are shared:\n",
36+
"\n",
37+
"- **Compute** usage is managed by a scheduler.\n",
38+
"- **Storage** may be restricted via quotas.\n",
39+
"\n",
40+
"A fair usage policy ensures everyone can access compute resources within a reasonable timeframe. Unlike your personal computer, you may need to wait for resources and act considerately toward other users.\n",
41+
"\n",
42+
"### Citing Use of ISCA\n",
43+
"\n",
44+
"Please use the following acknowledgement in research outputs:\n",
45+
"\n",
46+
"> *The authors would like to acknowledge the use of the University of Exeter High-Performance Computing (HPC) facility in carrying out this work.*\n",
47+
"\n",
48+
"When submitting to Symplectic, please add **ARC - ISCA** as an Unclassified Label.\n",
49+
"\n",
50+
"---\n",
51+
"\n",
52+
"## Connecting to ISCA\n",
53+
"\n",
54+
"Connections must be made from the university network or VPN. ISCA cannot be accessed directly from public Wi-Fi or external institutions.\n",
55+
"\n",
56+
"---\n",
2157
"\n",
58+
"## SSH: Secure Shell\n",
2259
"\n",
23-
"## Via the Command Line\n",
60+
"SSH (Secure SHell) provides:\n",
2461
"\n",
25-
"### Checking \n",
26-
"The most common way to connect to a remove HPC server is via a command line shell program `ssh`. This application uses the Secure SHell protocol (or SSH) to open an encrypted network connection between two machines, allowing you to send & receive text and data without having to worry about prying eyes. It should be available in all terminal apps (Windows, Linux, & Mac), but to doublecheck type `ssh` into your terminal of choice and press enter. If it is installed properly, you should get output like that below.\n",
62+
"- **Authentication** (you are who you say you are)\n",
63+
"- **Encrypted connections** (no one can read your data)\n",
2764
"\n",
28-
"``` bash\n",
29-
"you@laptop:~$ ssh\n",
65+
"SSH uses a **client-server model**. Your laptop is the *client*; ISCA is the *server*.\n",
66+
"\n",
67+
"---\n",
68+
"\n",
69+
"## Verifying SSH Installation\n",
70+
"\n",
71+
"To check if SSH is available on your system, run:\n",
72+
"\n",
73+
"```bash\n",
74+
"ssh\n",
3075
"```\n",
3176
"\n",
32-
"``` bash\n",
33-
"usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface]\n",
34-
" [-b bind_address] [-c cipher_spec] [-D [bind_address:]port]\n",
35-
" [-E log_file] [-e escape_char] [-F configfile] [-I pkcs11]\n",
36-
" [-i identity_file] [-J [user@]host[:port]] [-L address]\n",
37-
" [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]\n",
38-
" [-Q query_option] [-R address] [-S ctl_path] [-W host:port]\n",
39-
" [-w local_tun[:remote_tun]] destination [command [argument ...]]\n",
77+
"You should see output like:\n",
78+
"\n",
79+
"```bash\n",
80+
"usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] ...\n",
4081
"```\n",
4182
"\n",
42-
"If you do *not* get this output, please let the instructor know. But don't worry, there are other means of connecting to the system and well will cover those in a moment.\n",
83+
"If you don’t see this, inform the instructor — other options exist.\n",
84+
"\n",
85+
"---\n",
4386
"\n",
44-
"### Logging In\n",
87+
"## Logging In\n",
4588
"\n",
46-
"Assuming you were given access to the system by an administrator, you should be able to login with your standard university credentials. The syntax for logging into a machine with `ssh` is \n",
47-
"``` bash\n",
48-
"you@laptop:~$ ssh username@hostname\n",
89+
"Once access has been granted, log in using:\n",
90+
"\n",
91+
"```bash\n",
92+
"ssh [email protected]\n",
4993
"```\n",
5094
"\n",
51-
"where `username` is your standard login and the `hostname` is address of the server you wish to log into. For this workshop the `hostname` will be `login02.isca.ex.ac.uk`. Don't forget the **@** symbol, like you would see in an email address. You will then be prompted for your university password.\n",
95+
"Replace `userid` with your university login (e.g., `xy123`).\n",
96+
"\n",
97+
"### First-Time Warning\n",
5298
"\n",
53-
"### First Time Warning\n",
54-
"An important security warning will occur whenever you attempt to log into another machine for the first time. \n",
55-
"``` bash\n",
56-
"The authenticity of host 'login02.isca.ex.ac.uk (144.173.114.23)' can not be established.\n",
99+
"On your first connection, you may see:\n",
100+
"\n",
101+
"```bash\n",
102+
"The authenticity of host 'login02.isca.ex.ac.uk (144.173.114.132)' can’t be established.\n",
57103
"ED25519 key fingerprint is SHA256:rPqlD8myIfJUQBINKPYCp47oIQm3sGVMUPSAWT26E/M.\n",
58-
"ECDSA key fingerprint is SHA256:ah29tkixxl7NW/zdJM6TSbASY3UyLsKZEKUMMZSD3X0.\n",
59-
"This key is not known by any other names\n",
60104
"Are you sure you want to continue connecting (yes/no/[fingerprint])?\n",
61105
"```\n",
62106
"\n",
63-
"Make sure that the IP address (i.e. `144.173.114.23`) and the appropriate key fingerprint that you get match the information found above. Any subsequent time you log into that same server, you should **not** see this message, as this server's fingerprint is saved in a list of `known_hosts`. If you do get this message while connecting to a server you have logged into previously, please contact a system administrator **immediately**.\n",
107+
"Confirm the fingerprint is correct and type `yes`.\n",
108+
"\n",
109+
"This warning should only appear **once**. If it appears again later, contact system support.\n",
64110
"\n",
65-
"### Config File\n",
111+
"---\n",
66112
"\n",
67-
"Remembering the long hostnames can be a pain. We can create a config file that allows us to use shorter and more memorable ones for machines we use regularly using an ssh `config` file. For those using a Unix terminal, the two commands below will create a file with access restricted to just the current user.\n",
113+
"## SSH Config File\n",
68114
"\n",
69-
"``` bash\n",
115+
"To simplify SSH access, create a config file:\n",
116+
"\n",
117+
"```bash\n",
70118
"touch ~/.ssh/config\n",
71119
"chmod 600 ~/.ssh/config\n",
72120
"```\n",
73121
"\n",
74-
"Open up that file in your preferred text editor and copy the following example into it. **Remember** to swap your specific `username` for the placeholder.\n",
75-
"``` bash\n",
76-
"# Configuration file for simplifying SSH logins\n",
77-
"Host login.isca\n",
78-
" User your_username\n",
79-
" Hostname login02.isca.ex.ac.uk\n",
122+
"Edit the file with:\n",
123+
"\n",
124+
"```bash\n",
125+
"# SSH configuration for ISCA\n",
126+
"Host isca\n",
127+
" HostName login02.isca.ex.ac.uk\n",
128+
" User your_username\n",
129+
" ForwardAgent yes\n",
130+
" IdentityFile ~/.ssh/id_ed25519\n",
131+
"```\n",
132+
"\n",
133+
"Now you can log in with:\n",
134+
"\n",
135+
"```bash\n",
136+
"ssh isca\n",
137+
"```\n",
138+
"\n",
139+
"---\n",
140+
"\n",
141+
"## Creating a Cryptographic Key\n",
142+
"\n",
143+
"Rather than entering your password every time, you can use an SSH key pair.\n",
144+
"\n",
145+
"Generate a new key:\n",
146+
"\n",
147+
"```bash\n",
148+
"ssh-keygen -f ~/.ssh/id_ed25519\n",
80149
"```\n",
81150
"\n",
82-
"Now you can log in with a far shorter and easier to remember command. But you still will be prompted for your password.\n",
83-
"``` bash\n",
84-
"you@laptop:~$ ssh login.isca\n",
151+
"Choose no passphrase for now (for simplicity). This creates:\n",
152+
"\n",
153+
"- `id_ed25519` – your private key (keep it secure)\n",
154+
"- `id_ed25519.pub` – your public key (to be uploaded to ISCA)\n",
155+
"\n",
156+
"---\n",
157+
"\n",
158+
"### Installing the Public Key on ISCA\n",
159+
"\n",
160+
"The `.pub` key must be added to the server's `~/.ssh/authorized_keys`. An administrator may do this, or you may copy it manually.\n",
161+
"\n",
162+
"After setup, you can log in without a password:\n",
163+
"\n",
164+
"```bash\n",
165+
"ssh -i ~/.ssh/id_ed25519 isca\n",
166+
"```\n",
167+
"\n",
168+
"Or just:\n",
169+
"\n",
170+
"```bash\n",
171+
"ssh isca\n",
85172
"```\n",
86173
"\n",
174+
"(if configured in `~/.ssh/config`)\n",
175+
"\n",
176+
"---\n",
177+
"\n",
178+
"## Copying Files with `scp`\n",
179+
"\n",
180+
"Use `scp` (secure copy) to transfer files between your machine and ISCA:\n",
181+
"\n",
182+
"```bash\n",
183+
"scp <source> <destination>\n",
184+
"```\n",
185+
"\n",
186+
"### Example\n",
187+
"\n",
188+
"Copy a file to ISCA:\n",
189+
"\n",
190+
"```bash\n",
191+
"scp hello.txt [email protected]:\n",
192+
"```\n",
193+
"\n",
194+
"Or if using a config alias:\n",
195+
"\n",
196+
"```bash\n",
197+
"scp hello.txt isca:\n",
198+
"```\n",
199+
"\n",
200+
"Use `-r` to copy directories recursively:\n",
201+
"\n",
202+
"```bash\n",
203+
"scp -r my_folder/ isca:~/workspace/\n",
204+
"```\n",
205+
"\n",
206+
"---\n",
207+
"\n",
208+
"## Graphical Interfaces: PuTTY\n",
209+
"\n",
210+
"If you prefer a graphical tool, [PuTTY](https://putty.org/) is a free SSH client for Windows. You can:\n",
211+
"\n",
212+
"- Save session profiles\n",
213+
"- Use SSH keys\n",
214+
"- Avoid retyping hostnames\n",
87215
"\n",
88-
"## Graphical Interfaces\n",
216+
"When connecting for the first time, PuTTY will also show a security warning similar to the command line.\n",
89217
"\n",
90-
"An alternative means of connecting to a remote host server is via an application like the [Putty](https://putty.org/) SSH Client. It provides nice app-based way to save host connection configs so one need not input them again each time. Like the command line option, if you haven't logged in before, it will raise a *warning* message.\n",
218+
"![PuTTY Screenshot](fig/windows_putty.png)\n",
91219
"\n",
92-
"![](fig/windows_putty.png)\n",
220+
"After opening the session, you’ll be prompted for your username and password.\n",
93221
"\n",
94-
"You will be prompted in a new terminal session for your username and password.\n",
95-
"\n"
222+
"---\n"
96223
]
97224
},
98225
{

‎individual_modules/introduction_to_hpc/example_jobs.ipynb

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,8 @@
6767
"\n",
6868
"## R example\n",
6969
"\n",
70+
"[Download calc-squares.R](files/calc-squares.R)\n",
71+
"\n",
7072
"To run a job using R we need two scripts: \n",
7173
"\n",
7274
"1) an R script that contains all the R commands we want to run. This needs to be complete and run from top to bottom with no errors or user input. It should include all data loading, package loading, and saving of any output.\n",
@@ -108,6 +110,8 @@
108110
"\n",
109111
"## Python example\n",
110112
"\n",
113+
"[Download calc-squares.py](files/calc-squares.py)\n",
114+
"\n",
111115
"Running a python script is the same as running an R script. Again we need two scripts \n",
112116
"\n",
113117
"1) a Python script that contains all the python commands we want to run. This needs to be complete and run from top to bottom with no errors or user input. It should include all data loading, module loading, and saving of any output.\n",

‎individual_modules/introduction_to_hpc/files/calc-squares.R

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Initialise a vector with numbers 1 to 1000
2+
numbers <- 1:1000
3+
4+
# Print the initial vector
5+
cat("Initial list of numbers:\n")
6+
print(numbers)
7+
8+
# Calculate the square of each number in the vector
9+
squares <- numeric(length(numbers))
10+
for (i in 1:length(numbers)) {
11+
squares[i] <- numbers[i]^2
12+
}
13+
14+
# Print the results in full
15+
cat("\nSquares of the numbers:\n")
16+
print(squares)

‎individual_modules/introduction_to_hpc/files/calc-squares.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Initialise a list with numbers 1 to 1000
2+
numbers = list(range(1, 1001))
3+
4+
# Print the initial list
5+
print("Initial list of numbers:")
6+
print(numbers)
7+
8+
# Calculate the square of each number in the list
9+
squares = []
10+
for num in numbers:
11+
square = num ** 2
12+
squares.append(square)
13+
14+
# Print the results in full
15+
print("\nSquares of the numbers:")
16+
print(squares)

0 commit comments

Comments
 (0)