Merge #15165: contrib: Allow use of github API authentication in github-merge

f1bd219 contrib: Allow use of github API authentication in github-merge (Wladimir J. van der Laan)
a4c5bbf contrib: Add support for http[s] URLs in github-merge (Wladimir J. van der Laan)
059a3cf contrib: Detailed reporting for http errors in github-merge (Wladimir J. van der Laan)

Pull request description:

  Three commits I had locally for `github-merge.py`:

  -  *Detailed reporting for http errors in github-merge*: Print detailed error, this makes it easier to diagnose github API issues.
  - *Add support for http[s] URLs in github-merge*: Sometimes it can be useful to use github-merge with read-only access (say, for reviewing and testing from untrusted VMs).
  - *Allow use of github API authentication in github-merge*: The API request limit for unauthenticated requests is quite low. I started running into rate limiting errors. The limit for authenticated requests is much higher. This patch adds an optional configuration setting `user.ghtoken` that, when set, is used to authenticate requests to the API.

Tree-SHA512: ca8ae1874a787263e49d915d7cf31c0c0f50aba229c9440265bf1fda69f7e00641d1492512b93d76c17ff1766859283d640d37770acb120898736ad97efbd5c2

Author: laanwj

contrib/devtools/README.md

@@ -119,7 +119,25 @@ Configuring the github-merge tool for the bitcoin repository is done in the foll
 
     git config githubmerge.repository bitcoin/bitcoin
     git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing)
-    git config --global user.signingkey mykeyid (if you want to GPG sign)
+    git config --global user.signingkey mykeyid
+
+Authentication (optional)
+--------------------------
+
+The API request limit for unauthenticated requests is quite low, but the
+limit for authenticated requests is much higher. If you start running
+into rate limiting errors it can be useful to set an authentication token
+so that the script can authenticate requests.
+
+- First, go to [Personal access tokens](https://github.com/settings/tokens).
+- Click 'Generate new token'.
+- Fill in an arbitrary token description. No further privileges are needed.
+- Click the `Generate token` button at the bottom of the form.
+- Copy the generated token (should be a hexadecimal string)
+
+Then do:
+
+    git config --global user.ghtoken "pasted token"
 
 Create and verify timestamps of merge commits
 ---------------------------------------------

contrib/devtools/github-merge.py

@@ -23,6 +23,7 @@
 import json
 import codecs
 from urllib.request import Request, urlopen
+from urllib.error import HTTPError
 
 # External tools (can be overridden using environment)
 GIT = os.getenv('GIT','git')
@@ -46,17 +47,24 @@ def git_config_get(option, default=None):
     except subprocess.CalledProcessError:
         return default
 
-def retrieve_pr_info(repo,pull):
+def retrieve_pr_info(repo,pull,ghtoken):
     '''
     Retrieve pull request information from github.
     Return None if no title can be found, or an error happens.
     '''
     try:
         req = Request("https://api.github.com/repos/"+repo+"/pulls/"+pull)
+        if ghtoken is not None:
+            req.add_header('Authorization', 'token ' + ghtoken)
         result = urlopen(req)
         reader = codecs.getreader('utf-8')
         obj = json.load(reader(result))
         return obj
+    except HTTPError as e:
+        error_message = e.read()
+        print('Warning: unable to retrieve pull information from github: %s' % e)
+        print('Detailed error: %s' % error_message)
+        return None
     except Exception as e:
         print('Warning: unable to retrieve pull information from github: %s' % e)
         return None
@@ -134,6 +142,7 @@ def parse_arguments():
         In addition, you can set the following git configuration variables:
         githubmerge.repository (mandatory),
         user.signingkey (mandatory),
+        user.ghtoken (default: none).
         githubmerge.host (default: [email protected]),
         githubmerge.branch (no default),
         githubmerge.testcmd (default: none).
@@ -152,6 +161,7 @@ def main():
     host = git_config_get('githubmerge.host','[email protected]')
     opt_branch = git_config_get('githubmerge.branch',None)
     testcmd = git_config_get('githubmerge.testcmd')
+    ghtoken = git_config_get('user.ghtoken')
     signingkey = git_config_get('user.signingkey')
     if repo is None:
         print("ERROR: No repository configured. Use this command to set:", file=stderr)
@@ -162,14 +172,17 @@ def main():
         print("git config --global user.signingkey <key>",file=stderr)
         sys.exit(1)
 
-    host_repo = host+":"+repo # shortcut for push/pull target
+    if host.startswith(('https:','http:')):
+        host_repo = host+"/"+repo+".git"
+    else:
+        host_repo = host+":"+repo
 
     # Extract settings from command line
     args = parse_arguments()
     pull = str(args.pull[0])
 
     # Receive pull information from github
-    info = retrieve_pr_info(repo,pull)
+    info = retrieve_pr_info(repo,pull,ghtoken)
     if info is None:
         sys.exit(1)
     title = info['title'].strip()