Enhance description of working with private packages

README.md

@@ -1,6 +1,6 @@
 # HolyLabRegistry
 
-This registry allows you to use packages from HolyLab in Julia 0.7/1.x.
+This registry allows you to use packages from HolyLab in 1.x. (For very old Julia versions, see notes at the bottom.)
 
 # Usage
 
@@ -12,8 +12,6 @@ If you're using at least Julia 1.1, then you can add this registry with
 
 (The `]` enters Pkg mode when you type it at the REPL prompt, see https://docs.julialang.org/en/v1/stdlib/Pkg/.)
 
-For earlier Julia versions, manually `git clone` this repository under `DEPOT_PATH/registries`. (Usually, `DEPOT_PATH = /home/username/.julia`)
-
 Then, we can use lab private packages (or unregistered public ones) as if they are registered ones.
 
 # To use git protocol in GitHub
@@ -23,98 +21,108 @@ For windows users, you can get some information at https://gist.github.com/bsara
 
 0. Specific preparation for Windows
 
-    - Create a folder at the root of your user home folder (Example: C:/Users/uname/) called .ssh.
-    - Create the following files if they do not already exist (paths begin from the root of your user home folder):
-    
-        .ssh/config<br>
-        .bash_profile<br>
-        .bashrc<br>
+  - Create a folder at the root of your user home folder (Example: C:/Users/uname/) called .ssh.
+  - Create the following files if they do not already exist (paths begin from the root of your user home folder):
+
+      .ssh/config<br>
+      .bash_profile<br>
+      .bashrc<br>
 
 1. Create a New SSH Key
 
-    1.1 Generating a new SSH key at a local machine.
-    - Open git bash and paste text below, substituting in your GitHub email address.
-    ```
-    $ ssh-keygen -t ecdsa -b 521 -C "[email protected]"
-    ```
-    Note: Around September 1, 2021, GitHub has added new security requirements for newly added RSA keys. Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
-    <!-- Note: It might have error when you install package from Holylab repository after you finished all steps, the error is "ERROR: failed to fetch from [email protected]". If you face this problem, it might be helpful to replace the command above by -->
-
-    <!-- ```
-    $ ssh-keygen -m PEM rsa -b 4096 -C "[email protected]"
-    ```  -->
-
-    - When you're prompted to "Enter a file in which to save the key," press Enter. This accepts the default file location.
-    ```
-    Enter a file in which to save the key (/home/you/.ssh/id_ecdsa): [Press enter]
-    ```
+   Most interactions with GitHub require authentication, which is handled by SSH. There are at least two distinct cases:
 
-    - At the prompt, type a secure passphrase if you want.
-    ```
-    Enter passphrase (empty for no passphrase): [Type a passphrase]
-    Enter same passphrase again: [Type passphrase again]
-    ```
+   - `github_push_key`: when you want to `push` code from a local machine up to GitHub: for this case, your private key remains only on your local machine and your public key needs to be registered with your GitHub account.
+   - `github_CI_key`: when you want a package you're developing and hosting on GitHub to be able to access private repositories (e.g., during CI): for this case, the private key gets pasted into a repository secret, and the public key needs to be registered with your GitHub account.
 
-    1.2 Adding your SSH key to the ssh-agent
-    - Start the ssh-agent in the background.
-    ```
-    $ eval "$(ssh-agent -s)"
-    Agent pid 59566
-    ```
+   If both of these uses apply to you, you should generate two separate keys for the two cases.
 
-    - Add your SSH private key to the ssh-agent
-    ```
-    $ ssh-add ~/.ssh/id_ecdsa
-    ```
-2. Setup SSH Authentication for Git Bash on Windows (Safe to skip for Linux)
+   Here are the steps needed to generate a key:
 
-    2.1 Configure SSH for Git Hosting Server
-    - Add the following text to .ssh/config (.ssh should be found in the root of your user home folder):
-    ```
-    Host github.com<br>
-    Hostname github.com<br>
-    IdentityFile ~/.ssh/id_ecdsa
-    ```
-
-    2.2 Enable SSH Agent Startup Whenever Git Bash is Started
-    - First, ensure that following lines are added to .bash_profile, which should be found in your root user home folder:
-    ```
-    test -f ~/.profile && . ~/.profile
-    test -f ~/.bashrc && . ~/.bashrc
-    ```    
-    - Now, add the following text to .bashrc, which should be found in your root user home folder:
-    ```
-    # Start SSH Agent
-    #----------------------------
+   1.1 Generating a new SSH key at a local machine.
 
-    SSH_ENV="$HOME/.ssh/environment"
+      - Open git bash and paste text below, substituting in your GitHub email address.
+        ```
+        $ ssh-keygen -t ecdsa -b 521 -C "[email protected]"
+        ```
+        Note: Around September 1, 2021, GitHub has added new security requirements for newly added RSA keys. Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
 
-    function run_ssh_env {
-      . "${SSH_ENV}" > /dev/null
-    }
+      - When you're prompted to "Enter a file in which to save the key," press Enter. This accepts the default file location.
+        ```
+        Enter a file in which to save the key (/home/you/.ssh/id_ecdsa): [Press enter]
+        ```
+        It's recommended to name the key something informative, e.g., `github_push_key` or `github_CI_key` for the two cases described at the top of this section.
 
-    function start_ssh_agent {
-      echo "Initializing new SSH agent..."
-      ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
-      echo "succeeded"
-      chmod 600 "${SSH_ENV}"
+      - At the prompt, type a secure passphrase if you want. (Do not use a passphrase for `github_CI_key` cases, just hit <Enter> at the prompts below.)
+        ```
+        Enter passphrase (empty for no passphrase): [Type a passphrase]
+        Enter same passphrase again: [Type passphrase again]
+        ```
 
-      run_ssh_env;
+    1.2 Adding your SSH key to the ssh-agent (only for `github_push_key`)
 
-      ssh-add ~/.ssh/id_ecdsa;
-    }
+      - Start the ssh-agent in the background.
+        ```
+        $ eval "$(ssh-agent -s)"
+        Agent pid 59566
+        ```
 
-    if [ -f "${SSH_ENV}" ]; then
-      run_ssh_env;
-      ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
-        start_ssh_agent;
+      - Add your SSH private key to the ssh-agent
+        ```
+        $ ssh-add ~/.ssh/id_ecdsa
+        ```
+
+
+
+2. Setup SSH Authentication for Git Bash on Windows (Safe to skip for Linux) (Only for `github_push_key`)
+
+    2.1 Configure SSH for Git Hosting Server
+      Add the following text to .ssh/config (.ssh should be found in the root of your user home folder):
+      ```
+      Host github.com<br>
+      Hostname github.com<br>
+      IdentityFile ~/.ssh/id_ecdsa
+      ```
+
+    2.2 Enable SSH Agent Startup Whenever Git Bash is Started
+      - First, ensure that following lines are added to .bash_profile, which should be found in your root user home folder:
+      ```
+      test -f ~/.profile && . ~/.profile
+      test -f ~/.bashrc && . ~/.bashrc
+      ```
+      - Now, add the following text to .bashrc, which should be found in your root user home folder:
+      ```
+      # Start SSH Agent
+      #----------------------------
+
+      SSH_ENV="$HOME/.ssh/environment"
+
+      function run_ssh_env {
+        . "${SSH_ENV}" > /dev/null
       }
-    else
-      start_ssh_agent;
-    fi
-    ```
-
-3. Adding a new SSH key to your GitHub account
+
+      function start_ssh_agent {
+        echo "Initializing new SSH agent..."
+        ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
+        echo "succeeded"
+        chmod 600 "${SSH_ENV}"
+
+        run_ssh_env;
+
+        ssh-add ~/.ssh/id_ecdsa;
+      }
+
+      if [ -f "${SSH_ENV}" ]; then
+        run_ssh_env;
+        ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
+          start_ssh_agent;
+        }
+      else
+        start_ssh_agent;
+      fi
+      ```
+
+3. Adding a new SSH key to your GitHub account:
     - Copies the contents of the id_ecdsa.pub file in the local machine to your clipboard
     - Go to GitHub site
     - In the upper-right corner of any page, click your profile photo, then click Settings.
@@ -148,14 +156,14 @@ You have two options:
 
 - Using [PkgTemplates.jl](https://github.com/invenia/PkgTemplates.jl) (recommended):
 
-  To create a new package and host it in your own GitHub account, use
+  To create a new package and host it the HolyLab organization account, use
 
   ```julia
   julia> using PkgTemplates
 
-  julia> t = Template(ssh=true, plugins=[TravisCI()])  # creates a template for your personal account
+  julia> tpl = Template(user="HolyLab"; plugins=[GitHubActions(), Codecov(), Documenter{GitHubActions}()])
   Template:
-      → User: timholy
+      → User: HolyLab
       → Host: github.com
       → License: MIT (Tim Holy 2019)
       → Package directory: /tmp/pkgs/dev
@@ -164,15 +172,11 @@ You have two options:
       → Commit Manifest.toml: No
       → Plugins: None
 
-  julia> generate("MyPkg", t)
+  julia> tpl("MyPkg")
   # lots of output
   ```
 
-  If you plan to host your new package on `HolyLab`, instead use
-
-  ```julia
-  julia> t = Template(user="HolyLab", ...)
-  ```
+  If you instead want to host your package within your personal GitHub account, omit `user="HolyLab"` from the above `Template` call (or specify your own user name).
 
 ### Adding dependent packages
 
@@ -243,9 +247,12 @@ $ git branch -D teh/SomeNewPkg
   3bd9afcd-55df-531a-9b34-dc642dce7b95 = { name = "RFFT", path = "RFFT" }
   ```
 
-## Accessing the HolyLabRegistry in a CI workflow
+## Accessing HolyLabRegistry or private packages in CI tests
 
-This is required only if your package uses other packages which are registered in this HolyLabRegistry registry.
+This is required only if your package uses other packages which are registered in this HolyLabRegistry registry,
+or your package depends on unregistered private packages.
+
+- In the repository's `Settings`, go to `Secrets and Variables` and expand the caret, click on `Actions`. Add a `New repository secret` called `SSH_PRIVATE_KEY`, and copy the contents of the *private* key for your `github_CI_key` (i.e., not the file ending in `.pub`)
 
 - Set a secret key in the repository settings if you want to use private dependent packages in our lab. 
   The key should not include a passphrase when the key pair is generated. Add the public key as a 'new SSH key'
@@ -263,8 +270,8 @@ jobs:
     strategy:
       # setting ...
     steps:
-      - uses: actions/checkout@v3      
-      - name: Setup SSH Keys and known_hosts # This section is required if the dependent packages include private packages in our lab.
+      - uses: actions/checkout@v3
+      - name: Setup SSH Keys and known_hosts # This section is required if the dependent packages include private packages in our Lab.
         env:
             SSH_AUTH_SOCK: /tmp/ssh_agent.sock
         run: |
@@ -287,7 +294,25 @@ jobs:
       - uses: julia-actions/julia-runtest@v1
 ```
 
-## Tagging a new release
+If you're instead wanting to manually add unregistered dependencies, then you should have something like this:
+
+```
+      - name: Build package manually   # manual because of dependency on MyPrivatePackage
+        run: |
+          using Pkg
+          Pkg.Registry.add()           # add Julia's General registry
+          Pkg.develop(url="[email protected]:HolyLab/MyPrivatePackage.jl.git")
+          Pkg.build()
+        shell: julia --color=yes --project="@." {0}
+        env:
+            GITHUB_TOKEN: ${{ github.token }}
+            SSH_AUTH_SOCK: /tmp/ssh_agent.sock
+            JULIA_PKG_PRECOMPILE_AUTO: "no"
+```
+
+These lines replace `julia-actions/julia-buildpkg@v1`, so do not use this action if you're doing things this way.
+
+## Registering a new release in HolyLabRegistry
 
 ### In the package directory
 
@@ -325,3 +350,9 @@ Use the sha from the `git cat-file` command above.
 # See also
 
 - Creating a registry : https://discourse.julialang.org/t/creating-a-registry/12094
+
+# Tips for Julia 0.7 and 1.0
+
+In case of very old code, here are some tips:
+
+- `pkg> registry add` does not work. For earlier Julia versions, manually `git clone` this repository under `DEPOT_PATH/registries`. (Usually, `DEPOT_PATH = /home/username/.julia`)