Skip to content

Commit dcd318d

Browse files
timholykdw503
timholy
and
kdw503
authored
Enhance description of working with private packages (#237)
* Add info on unregistered private package Also clarifies the multiple uses of SSH keys. * fix * more fix * yet more fix * and more * try * more indent * sigh * temp * again * more * more fmt * Add info about the private key --------- Co-authored-by: Dae Woo Kim <[email protected]>
1 parent 980fc68 commit dcd318d

File tree

1 file changed

+126
-95
lines changed

1 file changed

+126
-95
lines changed

‎README.md‎

Lines changed: 126 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# HolyLabRegistry
22

3-
This registry allows you to use packages from HolyLab in Julia 0.7/1.x.
3+
This registry allows you to use packages from HolyLab in 1.x. (For very old Julia versions, see notes at the bottom.)
44

55
# Usage
66

@@ -12,8 +12,6 @@ If you're using at least Julia 1.1, then you can add this registry with
1212

1313
(The `]` enters Pkg mode when you type it at the REPL prompt, see https://docs.julialang.org/en/v1/stdlib/Pkg/.)
1414

15-
For earlier Julia versions, manually `git clone` this repository under `DEPOT_PATH/registries`. (Usually, `DEPOT_PATH = /home/username/.julia`)
16-
1715
Then, we can use lab private packages (or unregistered public ones) as if they are registered ones.
1816

1917
# To use git protocol in GitHub
@@ -23,98 +21,108 @@ For windows users, you can get some information at https://gist.github.com/bsara
2321

2422
0. Specific preparation for Windows
2523

26-
- Create a folder at the root of your user home folder (Example: C:/Users/uname/) called .ssh.
27-
- Create the following files if they do not already exist (paths begin from the root of your user home folder):
28-
29-
.ssh/config<br>
30-
.bash_profile<br>
31-
.bashrc<br>
24+
- Create a folder at the root of your user home folder (Example: C:/Users/uname/) called .ssh.
25+
- Create the following files if they do not already exist (paths begin from the root of your user home folder):
26+
27+
.ssh/config<br>
28+
.bash_profile<br>
29+
.bashrc<br>
3230

3331
1. Create a New SSH Key
3432

35-
1.1 Generating a new SSH key at a local machine.
36-
- Open git bash and paste text below, substituting in your GitHub email address.
37-
```
38-
$ ssh-keygen -t ecdsa -b 521 -C "[email protected]"
39-
```
40-
Note: Around September 1, 2021, GitHub has added new security requirements for newly added RSA keys. Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
41-
<!-- Note: It might have error when you install package from Holylab repository after you finished all steps, the error is "ERROR: failed to fetch from [email protected]". If you face this problem, it might be helpful to replace the command above by -->
42-
43-
<!-- ```
44-
$ ssh-keygen -m PEM rsa -b 4096 -C "[email protected]"
45-
``` -->
46-
47-
- When you're prompted to "Enter a file in which to save the key," press Enter. This accepts the default file location.
48-
```
49-
Enter a file in which to save the key (/home/you/.ssh/id_ecdsa): [Press enter]
50-
```
33+
Most interactions with GitHub require authentication, which is handled by SSH. There are at least two distinct cases:
5134

52-
- At the prompt, type a secure passphrase if you want.
53-
```
54-
Enter passphrase (empty for no passphrase): [Type a passphrase]
55-
Enter same passphrase again: [Type passphrase again]
56-
```
35+
- `github_push_key`: when you want to `push` code from a local machine up to GitHub: for this case, your private key remains only on your local machine and your public key needs to be registered with your GitHub account.
36+
- `github_CI_key`: when you want a package you're developing and hosting on GitHub to be able to access private repositories (e.g., during CI): for this case, the private key gets pasted into a repository secret, and the public key needs to be registered with your GitHub account.
5737

58-
1.2 Adding your SSH key to the ssh-agent
59-
- Start the ssh-agent in the background.
60-
```
61-
$ eval "$(ssh-agent -s)"
62-
Agent pid 59566
63-
```
38+
If both of these uses apply to you, you should generate two separate keys for the two cases.
6439

65-
- Add your SSH private key to the ssh-agent
66-
```
67-
$ ssh-add ~/.ssh/id_ecdsa
68-
```
69-
2. Setup SSH Authentication for Git Bash on Windows (Safe to skip for Linux)
40+
Here are the steps needed to generate a key:
7041

71-
2.1 Configure SSH for Git Hosting Server
72-
- Add the following text to .ssh/config (.ssh should be found in the root of your user home folder):
73-
```
74-
Host github.com<br>
75-
Hostname github.com<br>
76-
IdentityFile ~/.ssh/id_ecdsa
77-
```
78-
79-
2.2 Enable SSH Agent Startup Whenever Git Bash is Started
80-
- First, ensure that following lines are added to .bash_profile, which should be found in your root user home folder:
81-
```
82-
test -f ~/.profile && . ~/.profile
83-
test -f ~/.bashrc && . ~/.bashrc
84-
```
85-
- Now, add the following text to .bashrc, which should be found in your root user home folder:
86-
```
87-
# Start SSH Agent
88-
#----------------------------
42+
1.1 Generating a new SSH key at a local machine.
8943

90-
SSH_ENV="$HOME/.ssh/environment"
44+
- Open git bash and paste text below, substituting in your GitHub email address.
45+
```
46+
$ ssh-keygen -t ecdsa -b 521 -C "[email protected]"
47+
```
48+
Note: Around September 1, 2021, GitHub has added new security requirements for newly added RSA keys. Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
9149
92-
function run_ssh_env {
93-
. "${SSH_ENV}" > /dev/null
94-
}
50+
- When you're prompted to "Enter a file in which to save the key," press Enter. This accepts the default file location.
51+
```
52+
Enter a file in which to save the key (/home/you/.ssh/id_ecdsa): [Press enter]
53+
```
54+
It's recommended to name the key something informative, e.g., `github_push_key` or `github_CI_key` for the two cases described at the top of this section.
9555
96-
function start_ssh_agent {
97-
echo "Initializing new SSH agent..."
98-
ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
99-
echo "succeeded"
100-
chmod 600 "${SSH_ENV}"
56+
- At the prompt, type a secure passphrase if you want. (Do not use a passphrase for `github_CI_key` cases, just hit <Enter> at the prompts below.)
57+
```
58+
Enter passphrase (empty for no passphrase): [Type a passphrase]
59+
Enter same passphrase again: [Type passphrase again]
60+
```
10161
102-
run_ssh_env;
62+
1.2 Adding your SSH key to the ssh-agent (only for `github_push_key`)
10363
104-
ssh-add ~/.ssh/id_ecdsa;
105-
}
64+
- Start the ssh-agent in the background.
65+
```
66+
$ eval "$(ssh-agent -s)"
67+
Agent pid 59566
68+
```
10669
107-
if [ -f "${SSH_ENV}" ]; then
108-
run_ssh_env;
109-
ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
110-
start_ssh_agent;
70+
- Add your SSH private key to the ssh-agent
71+
```
72+
$ ssh-add ~/.ssh/id_ecdsa
73+
```
74+
75+
76+
77+
2. Setup SSH Authentication for Git Bash on Windows (Safe to skip for Linux) (Only for `github_push_key`)
78+
79+
2.1 Configure SSH for Git Hosting Server
80+
Add the following text to .ssh/config (.ssh should be found in the root of your user home folder):
81+
```
82+
Host github.com<br>
83+
Hostname github.com<br>
84+
IdentityFile ~/.ssh/id_ecdsa
85+
```
86+
87+
2.2 Enable SSH Agent Startup Whenever Git Bash is Started
88+
- First, ensure that following lines are added to .bash_profile, which should be found in your root user home folder:
89+
```
90+
test -f ~/.profile && . ~/.profile
91+
test -f ~/.bashrc && . ~/.bashrc
92+
```
93+
- Now, add the following text to .bashrc, which should be found in your root user home folder:
94+
```
95+
# Start SSH Agent
96+
#----------------------------
97+
98+
SSH_ENV="$HOME/.ssh/environment"
99+
100+
function run_ssh_env {
101+
. "${SSH_ENV}" > /dev/null
111102
}
112-
else
113-
start_ssh_agent;
114-
fi
115-
```
116-
117-
3. Adding a new SSH key to your GitHub account
103+
104+
function start_ssh_agent {
105+
echo "Initializing new SSH agent..."
106+
ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
107+
echo "succeeded"
108+
chmod 600 "${SSH_ENV}"
109+
110+
run_ssh_env;
111+
112+
ssh-add ~/.ssh/id_ecdsa;
113+
}
114+
115+
if [ -f "${SSH_ENV}" ]; then
116+
run_ssh_env;
117+
ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
118+
start_ssh_agent;
119+
}
120+
else
121+
start_ssh_agent;
122+
fi
123+
```
124+
125+
3. Adding a new SSH key to your GitHub account:
118126
- Copies the contents of the id_ecdsa.pub file in the local machine to your clipboard
119127
- Go to GitHub site
120128
- In the upper-right corner of any page, click your profile photo, then click Settings.
@@ -148,14 +156,14 @@ You have two options:
148156
149157
- Using [PkgTemplates.jl](https://github.com/invenia/PkgTemplates.jl) (recommended):
150158
151-
To create a new package and host it in your own GitHub account, use
159+
To create a new package and host it the HolyLab organization account, use
152160
153161
```julia
154162
julia> using PkgTemplates
155163
156-
julia> t = Template(ssh=true, plugins=[TravisCI()]) # creates a template for your personal account
164+
julia> tpl = Template(user="HolyLab"; plugins=[GitHubActions(), Codecov(), Documenter{GitHubActions}()])
157165
Template:
158-
→ User: timholy
166+
→ User: HolyLab
159167
→ Host: github.com
160168
→ License: MIT (Tim Holy 2019)
161169
→ Package directory: /tmp/pkgs/dev
@@ -164,15 +172,11 @@ You have two options:
164172
→ Commit Manifest.toml: No
165173
→ Plugins: None
166174
167-
julia> generate("MyPkg", t)
175+
julia> tpl("MyPkg")
168176
# lots of output
169177
```
170178

171-
If you plan to host your new package on `HolyLab`, instead use
172-
173-
```julia
174-
julia> t = Template(user="HolyLab", ...)
175-
```
179+
If you instead want to host your package within your personal GitHub account, omit `user="HolyLab"` from the above `Template` call (or specify your own user name).
176180

177181
### Adding dependent packages
178182

@@ -243,9 +247,12 @@ $ git branch -D teh/SomeNewPkg
243247
3bd9afcd-55df-531a-9b34-dc642dce7b95 = { name = "RFFT", path = "RFFT" }
244248
```
245249
246-
## Accessing the HolyLabRegistry in a CI workflow
250+
## Accessing HolyLabRegistry or private packages in CI tests
247251
248-
This is required only if your package uses other packages which are registered in this HolyLabRegistry registry.
252+
This is required only if your package uses other packages which are registered in this HolyLabRegistry registry,
253+
or your package depends on unregistered private packages.
254+
255+
- In the repository's `Settings`, go to `Secrets and Variables` and expand the caret, click on `Actions`. Add a `New repository secret` called `SSH_PRIVATE_KEY`, and copy the contents of the *private* key for your `github_CI_key` (i.e., not the file ending in `.pub`)
249256
250257
- Set a secret key in the repository settings if you want to use private dependent packages in our lab.
251258
The key should not include a passphrase when the key pair is generated. Add the public key as a 'new SSH key'
@@ -263,8 +270,8 @@ jobs:
263270
strategy:
264271
# setting ...
265272
steps:
266-
- uses: actions/checkout@v3
267-
- name: Setup SSH Keys and known_hosts # This section is required if the dependent packages include private packages in our lab.
273+
- uses: actions/checkout@v3
274+
- name: Setup SSH Keys and known_hosts # This section is required if the dependent packages include private packages in our Lab.
268275
env:
269276
SSH_AUTH_SOCK: /tmp/ssh_agent.sock
270277
run: |
@@ -287,7 +294,25 @@ jobs:
287294
- uses: julia-actions/julia-runtest@v1
288295
```
289296
290-
## Tagging a new release
297+
If you're instead wanting to manually add unregistered dependencies, then you should have something like this:
298+
299+
```
300+
- name: Build package manually # manual because of dependency on MyPrivatePackage
301+
run: |
302+
using Pkg
303+
Pkg.Registry.add() # add Julia's General registry
304+
Pkg.develop(url="[email protected]:HolyLab/MyPrivatePackage.jl.git")
305+
Pkg.build()
306+
shell: julia --color=yes --project="@." {0}
307+
env:
308+
GITHUB_TOKEN: ${{ github.token }}
309+
SSH_AUTH_SOCK: /tmp/ssh_agent.sock
310+
JULIA_PKG_PRECOMPILE_AUTO: "no"
311+
```
312+
313+
These lines replace `julia-actions/julia-buildpkg@v1`, so do not use this action if you're doing things this way.
314+
315+
## Registering a new release in HolyLabRegistry
291316
292317
### In the package directory
293318
@@ -325,3 +350,9 @@ Use the sha from the `git cat-file` command above.
325350
# See also
326351
327352
- Creating a registry : https://discourse.julialang.org/t/creating-a-registry/12094
353+
354+
# Tips for Julia 0.7 and 1.0
355+
356+
In case of very old code, here are some tips:
357+
358+
- `pkg> registry add` does not work. For earlier Julia versions, manually `git clone` this repository under `DEPOT_PATH/registries`. (Usually, `DEPOT_PATH = /home/username/.julia`)

0 commit comments

Comments
 (0)