Merge pull request #417 from some-natalie/master

Add SQL queries for GitHub Enterprise Server

Author: zkoppert

README.md

@@ -15,3 +15,4 @@ Make a pull request and we'll consider it.
 * _hooks_: want to find out how to write a consumer for [our web hooks](https://developer.github.com/webhooks/)? The examples in this subdirectory show you how. We are open for more contributions via pull requests.
 * _pre-receive-hooks_: this one contains [pre-receive-hooks](https://help.github.com/enterprise/admin/guides/developer-workflow/about-pre-receive-hooks/) that can block commits on GitHub Enterprise that do not fit your requirements. Do you have more great examples? Create a pull request and we will check it out.
 * _scripts_: want to analyze or clean-up your Git repository? The scripts in this subdirectory show you how. We are open for more contributions via pull requests
+* _sql_: here are sql scripts for custom reporting for GitHub Enterprise Server. We are open for more contributions via pull requests.

sql/README.md

@@ -0,0 +1,46 @@
+# SQL Queries for GitHub Enterprise Server
+
+:warning: While these are all read-only queries and do not write to the database, run these directly against your GitHub Enterprise Server database at your own risk.  A safer method to run these is outlined [here](USAGE.md).
+
+Each query has a comment at the top of the file elaborating what it does, etc.
+
+## Audit queries
+
+The `audit` folder has queries that are all around auditing credentials, webhooks, apps, etc.
+
+- `admin-tokens.sql` - A report of all tokens with the `site_admin` scope and when they were last used.
+- `authorizations.sql` - A report of all personal access tokens and when they were last used.  Same as above, but without the `site_admin` scope limitation.  This is a big report.
+- `deploy-keys.sql` - A report of all deploy keys, when it was last used, who set it up and when, how long the key is, and what repository it's tied to.
+- `github-apps.sql` - A report of all GitHub apps, who owns them, the scope it's installed at, if it's public or not, and the URL it's sending data to.
+- `hooks-repos.sql` - A report of all repository webhooks used in the past week, who owns it, and where the webhook goes.  This is limited to a week based on the length of time these are kept in the `hookshot_delivery_logs` table.
+- `hooks-users.sql` - Same report as above, but for user-owned webhooks.
+- `oauth-apps.sql` - A report of all OAuth apps, who owns it, where it goes, and when it was last used.
+- `user-emails.sql` - A report of all emails that don't match a list of approved domains you define in the `WHERE` clause.  This query should be deprecated by [this issue](https://github.com/github/roadmap/issues/204).
+- `user-ssh-keys.sql` - A report of all user SSH keys, when it was last used, when it was set up, and how long the key is.
+
+## Metrics queries
+
+The `metrics` folder has queries that are all around usage of various features in GitHub Enterprise Server.
+
+- `actions-summary.sql` - A monthly summary of runtime hours, seconds waiting in queue before dispatch, and job count for GitHub Actions usage.
+- `commit-count.sql` - This pulls a "high score" report of all users, all commits, from all time.
+- `commit-summary.sql` - A month-by-month summary of commits pushed to GitHub Enterprise Server (using the commit date).
+- `count-tabs.sql` - A report of the custom tabs users put in their repositories.
+- `issue-report.sql` - A report of active issues within the past X days.
+- `linguist-report.sql` - This returns the "size" of each language in each repository and when the repo was last updated.  This can be a very large report.
+- `linguist-stats.sql` - This returns the count of repositories containing each language and a sum "size" of code in that language for all repos pushed to in the past year.  The time limit is adjustable.
+- `most-recent-active-repos.sql` - A list of repositories, when they were last updated, who owns them, and the disk space associated with each.
+- `pr-report.sql` - This pulls a report of pull requests including the repo name, user name, files included, times it was created/updated/merged, and comments.  It can filter by organization or return all PRs in GHES.
+- `prereceive-hooks.sql` - A list of pre-receive hooks that are enabled by each repository and who owns the repo.
+- `public-repo-owners.sql` - A list of all users or orgs who own repositories marked as "public", a count of public repos, and the user or org email address.
+- `reaction-stats.sql` - A count of the reactions used in GHES for trivia.
+- `staff-notes.sql` - Returns a list of organizations or users with `staff_notes`.
+- `user-report.sql` - Returns username, id, created/suspended date, issues created for all time and in the past 30 days, number of repos owned, and how many pull requests they've opened.
+
+## Security queries
+
+The `security` folder has queries that are all around dependency alerts and any other security features.
+
+- `active-repo-report.sql` - A list of all detected HIGH and CRITICAL vulnerabilities from repos pushed to in the past 90 days.  It also returns who owns it and further details on the exact vulnerability.  The threshold of time and severity to return is adjustable.
+- `vuln-critical-count.sql` - A count of repositories affected by each CRITICAL vulnerability.
+- `vuln-report.sql` - A report of all detected vulnerabilities in every single repo in GHES, who owns it, when it was last pushed to, the platform of the vulnerability, and the GHSA/MITRE/WhiteSource info on it.  This can be a very large report.

sql/USAGE.md

@@ -0,0 +1,49 @@
+# Using the SQL queries
+
+The safest way to run these queries is by using the backup created by [backup-utils](https://github.com/github/backup-utils) loaded into another database server.  This database can be quite large and GitHub Enterprise Server can be sensitive to I/O intensive operations that aren't part of anticipated load.
+
+:warning:  This database contains sensitive information.  Please treat it appropriately within your company / network!
+
+A simple way to do this would be to install a MySQL 5.7 server on the VM receiving the backups and load it automatically.  You can then connect to in using `root` with no password, or whatever you set up for authentication.  What this looks like in practice would be similar to this shell script:
+
+```shell
+# Stop MySQL
+sudo systemctl stop mysqld.service
+
+# Unzip the most current backup
+gunzip -c /data/current/mysql.sql.gz > /data/mysql.tar
+
+# Untar the current backup
+tar xf /data/mysql.tar --directory=/home/github/restore-job/
+
+# Remove the temporary tarball
+rm /data/mysql.tar
+
+# Clear the data directory before restoring
+sudo rm -rf /var/lib/mysql-data/*
+
+# Run the Percona backup restore
+cd /home/github/restore-job && sudo innobackupex --defaults-file=backup-my.cnf --copy-back --datadir=/var/lib/mysql-data .
+
+# Restore the innodb buffer pool
+sudo cp -n /var/lib/mysql/ib_buffer_pool /var/lib/mysql-data/
+
+# Restore the innodb data
+sudo cp -n /var/lib/mysql/ibdata1 /var/lib/mysql-data/
+
+# Restore the first and second logs
+sudo cp -n /var/lib/mysql/ib_logfile0 /var/lib/mysql-data/
+sudo cp -n /var/lib/mysql/ib_logfile1 /var/lib/mysql-data/
+
+# Reset ownership
+sudo chown -R mysql:mysql /var/lib/mysql-data
+
+# Restore SELinux contexts (if applicable)
+sudo restorecon -R /var/lib/mysql-data
+
+# Start MySQL
+sudo systemctl start mysqld.service
+
+# Clear the working directory to save some disk space
+rm -rf /home/github/restore-job/*
+```

sql/audit/admin-tokens.sql

@@ -0,0 +1,23 @@
+/*
+ * This pulls a list of all apps, tokens, and scopes associated with that token
+ * as well as when it was last used, created, and updated for anything with 
+ * the `site_admin` scope.
+ */
+SELECT
+	z.id,
+	u.login as owner_name,
+	u.type as owner_type,
+	a.name as app_name,
+	z.accessed_at,
+	z.created_at,
+	z.updated_at,
+	z.description,
+	z.scopes
+FROM
+	github_enterprise.oauth_authorizations z
+JOIN github_enterprise.users u ON
+	z.user_id = u.id
+LEFT JOIN github_enterprise.oauth_applications a ON
+	z.application_id = a.id
+WHERE
+	z.scopes LIKE "%site_admin%"

sql/audit/authorizations.sql

@@ -0,0 +1,20 @@
+/* 
+ * This pulls a list of all apps, tokens, and scopes associated with that token
+ * as well as when it was last used, created, and updated.
+ */
+SELECT
+	z.id,
+	u.login as owner_name,
+	u.type as owner_type,
+	a.name as app_name,
+	z.accessed_at,
+	z.created_at,
+	z.updated_at,
+	z.description,
+	z.scopes
+FROM
+	github_enterprise.oauth_authorizations z
+JOIN github_enterprise.users u ON
+	z.user_id = u.id
+LEFT JOIN github_enterprise.oauth_applications a ON
+	z.application_id = a.id;

sql/audit/deploy-keys.sql

@@ -0,0 +1,32 @@
+/* 
+ * This query returns SSH deploy keys and what repo they're tied to, when last
+ * used, etc.
+ */
+SELECT
+	d.title as key_name,
+	d.created_at,
+	d.updated_at,
+	d.verified_at,
+	d.accessed_at as last_used,
+	length(d.key) as key_length,
+	u.login as created_by_name,
+	d.created_by as created_by_type,
+	r.name as repo_name,
+	x.login as repo_owner_name
+FROM
+	github_enterprise.public_keys d
+LEFT JOIN github_enterprise.users u ON
+	d.creator_id = u.id
+LEFT JOIN github_enterprise.repositories r ON
+	d.repository_id = r.id
+LEFT JOIN (
+	SELECT
+		id,
+		login,
+		type
+	FROM
+		github_enterprise.users u2
+    ) x ON
+	x.id = r.owner_id
+WHERE
+	d.repository_id IS NOT NULL;

sql/audit/github-apps.sql

@@ -0,0 +1,20 @@
+/* 
+ * This pulls a list of all github apps, who owns them, and when they were 
+ * created or updated.
+ */
+SELECT
+	i.id,
+	i.bot_id,
+	i.name as integration_name,
+	u.login as owner,
+	u.type,
+	i.url,
+	i.created_at,
+	i.updated_at,
+	i.public,
+	i.slug as friendly_name,
+	i.public
+FROM
+	github_enterprise.integrations i
+JOIN github_enterprise.users u ON
+	i.owner_id = u.id;

sql/audit/hooks-repos.sql

@@ -0,0 +1,30 @@
+/* 
+ * This brings up the list of REPOSITORY webhooks that have been active in the
+ * past week, who owns them, and where the webhook goes.
+ */
+SELECT
+	DISTINCT h.id,
+	u.login as creator,
+	h.updated_at,
+	r.name as repo_name,
+	u.login as repo_owner,
+	u.type as owner_type,
+	c.value as url,
+	MAX(l.delivered_at) as latest_delivery
+FROM
+	github_enterprise.hooks h
+JOIN github_enterprise.hook_config_attributes c ON
+	h.id = c.hook_id
+JOIN github_enterprise.users u ON
+	h.creator_id = u.id
+JOIN github_enterprise.hookshot_delivery_logs l ON
+	h.id = l.hook_id
+JOIN github_enterprise.repositories r ON
+	h.installation_target_id = r.id
+WHERE
+	c.key = 'url'
+	AND h.installation_target_type = 'Repository'
+GROUP BY
+	h.id
+ORDER BY
+	MAX(l.delivered_at) DESC;

sql/audit/hooks-users.sql

@@ -0,0 +1,27 @@
+/*
+ * This brings up the list of USER webhooks that have been active in the past
+ * week, who owns them, and where the webhook goes.
+ */
+SELECT
+	DISTINCT h.id,
+	u.login as creator,
+	h.updated_at,
+	u.login as repo_owner,
+	u.type as owner_type,
+	c.value as url,
+	MAX(l.delivered_at) as latest_delivery
+FROM
+	github_enterprise.hooks h
+JOIN github_enterprise.hook_config_attributes c ON
+	h.id = c.hook_id
+JOIN github_enterprise.users u ON
+	h.creator_id = u.id
+JOIN github_enterprise.hookshot_delivery_logs l ON
+	h.id = l.hook_id
+WHERE
+	c.key = 'url'
+	AND h.installation_target_type = 'User'
+GROUP BY
+	h.id
+ORDER BY
+	MAX(l.delivered_at) DESC;

sql/audit/oauth-apps.sql

@@ -0,0 +1,16 @@
+/*
+ * This pulls up a list of all OAuth apps and where they go, as well as when
+ * they were last updated and what login they are associated with.
+ */
+SELECT
+	o.name,
+	o.url,
+	o.callback_url,
+	o.created_at,
+	o.updated_at,
+	u.login,
+	u.type
+FROM
+	github_enterprise.oauth_applications o
+JOIN github_enterprise.users u ON
+	o.user_id = u.id;