diff --git a/content/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading.md b/content/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading.md new file mode 100644 index 000000000000..f2ec5fb4a337 --- /dev/null +++ b/content/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading.md @@ -0,0 +1,276 @@ +--- +title: Check system capacity before upgrading +intro: 'Before upgrading {% data variables.product.prodname_ghe_server %}, you should perform these capacity checks and take the recommended steps.' +shortTitle: Check capacity before upgrade +versions: + ghes: '*' +contentType: how-tos +topics: + - Enterprise + - Upgrades +--- + +Upgrading to newer versions of {% data variables.product.prodname_ghe_server %} typically increases resource consumption. Each feature release adds new functionality, some enabled by default, others opt-in, which requires more processing power. Customer usage patterns also affect demand; for example, enterprises with tens of thousands of organizations may see higher resource usage. + +Resource increases most often appear as higher CPU utilization, more I/O operations per second (IOPS), greater memory use, or larger Aqueduct queue backlogs. To prepare for these changes, check your system’s available capacity and apply any remediation recommendations before upgrading. Run these checks during your busiest times of day and week to get the most accurate results. + +## Resource requirements + +Before upgrading your instance, it's crucial to verify that your system meets the necessary resource requirements: + +1. [CPU usage below 70%](#cpu-usage-below-70) +1. [Memory usage below 70%](#memory-usage-below-70) +1. [Disk not saturated](#disk-not-saturated) +1. [Unicorn queue under 200–300](#unicorn-queue-under-200300) +1. [Aqueduct backlog under 1–2 hours](#aqueduct-backlog-under-12-hours) + +### CPU usage below 70% + +1. **Check CPU utilization.** + In the {% data variables.enterprise.management_console %}, go to the monitor page (`https://HOSTNAME.com:8443/setup/monitor`) and view the `CPU` graph. + + * If utilization is **regularly below 70%**, continue to [Memory usage](#memory-usage-below-70). + * If utilization is **regularly above 70%**, the system does not meet the criteria to upgrade. + +1. **Compare utilization with CPU load average.** + The comparison helps identify possible disk saturation. + + {% ifversion ghes > 3.15 %} + * Go to **Operational Health view** and check the `Load` graph. + * In the matrix, find the value where the `shortterm` row intersects with the `avg` column. + * Calculate load average percentage: + + ```text + (short-term avg ÷ number of vCPUs) × 100 + ``` + + * In the same view, check the `CPU` graph. In the matrix, find the value where the `idle` row intersects with the `avg` column. Subtract this value from 100 to get utilization. + {% endif %} + + {% ifversion ghes < 3.16 %} + * On the `Load` graph, click **short-term** to show only the short-term line. Find the peak load value. + * On the `CPU` graph, click **idle** to show only the idle line. Note the idle value at the same timestamp. + * Calculate utilization: + + ```text + 100 – idle + ``` + + * Calculate load average percentage: + + ```text + (peak load value ÷ number of vCPUs) × 100 + ``` + + {% endif %} + +1. **Interpret the results.** + + If the CPU load average percentage is more than 50% higher than utilization, this likely indicates resource contention. Do not proceed with the upgrade until you have investigated possible disk saturation (see [Disk not saturated](#disk-not-saturated)). + +### Memory usage below 70% + +1. **Check memory usage.** + In the {% data variables.enterprise.management_console %}, go to the monitor page (`https://HOSTNAME.com:8443/setup/monitor`) and view the `Memory` graph. + +1. **Interpret the results.** + + * If memory usage is **regularly below 70%**, continue to [Disk not saturated](#disk-not-saturated). + * If memory usage is **regularly above 70%**, the system does not meet the criteria to upgrade. + +### Disk not saturated + +1. **Check provider specifications.** + If your cloud or hardware provider offers disk utilization metrics, use them to confirm whether the disk is saturated. + + * If metrics are not available, request the disk specifications from your provider, including maximum throughput and maximum IOPS. + * Compare these limits with your observed disk usage. If usage is approaching the maximum values, the disk is saturated. + +1. **Check disk graphs in the {% data variables.enterprise.management_console %}.** + Go to the monitor page (`https://HOSTNAME.com:8443/setup/monitor`). + + * View the `Disk Operations` and `Disk Traffic` graphs. + * Compare Y-axis values with your provider’s specifications (not the maximum scale shown on the graph). + * Review both data and root disks. + + {% ifversion ghes < 3.16 %} + These graphs are available in the default dashboards on the monitor page. + {% endif %} + {% ifversion ghes > 3.15 %} + These graphs are available in the "System & Application Insights" view. + {% endif %} + +1. **Interpret the results.** + If disk usage is approaching provider-defined maximums, the disk is saturated. In this case, the system does not meet the criteria to upgrade. + +### Unicorn queue under 200–300 + +1. **Check the queued requests graph.** + In the {% data variables.enterprise.management_console %}, go to the monitor page (`https://HOSTNAME.com:8443/setup/monitor`) and view the `Queued Requests` graph. + + {% ifversion ghes < 3.16 %} + This graph is available in the default dashboards on the monitor page. + {% endif %} + {% ifversion ghes > 3.15 %} + This graph is available in the "System & Application Insights" view. + {% endif %} + +1. **Interpret the results.** + + * If queued requests are **consistently below 200**, continue to [Aqueduct backlog under 1–2 hours](#aqueduct-backlog-under-12-hours). + * If queued requests are **regularly at or above 200–300**, the system does not meet the criteria to upgrade. + +1. **Optional: Check unicorn worker utilization.** + From the administrative shell, run: + + ```shell + ps -ef | grep unicorn | grep -v gitauth | grep -v ".rb" | grep -v init | grep git + ``` + + Look at the last column of the output. If all processes show `> 90% utilization`, more unicorn workers are required. + +### Aqueduct backlog under 1–2 hours + +1. **Check the Aqueduct queue depth.** + In the {% data variables.enterprise.management_console %}, go to the monitor page (`https://HOSTNAME.com:8443/setup/monitor`) and view the `Aqueduct queue depth` graph. + + {% ifversion ghes < 3.16 %} + This graph appears in the default dashboards on the monitor page. + {% endif %} + {% ifversion ghes > 3.15 %} + This graph is available in the "System & Application Insights" view. + {% endif %} + +1. **Interpret the results.** + + * If the backlog **lasts less than 1–2 hours**, you meet this requirement. + * If the backlog **regularly lasts longer than 1–2 hours**, the system does not meet the criteria to upgrade. + +1. **Monitor the `index_high` queue.** + Large deployments may experience significant increases in `index_high` queue depth, which can worsen backlogs. Pay special attention to this queue when monitoring. + +If **all criteria** (CPU, memory, disk, unicorn queue, Aqueduct backlog) are met, you can proceed with upgrading to your target feature version. After upgrading, expect resource consumption to increase further. + +If **any criteria are not met**, resolve the underlying issues before attempting to upgrade. + +## Upgrading hardware and fine-tune workers + +If your system did not meet one or more of the resource requirements, you will need to increase capacity before upgrading. The following sections describe how to add hardware resources and adjust worker configuration to resolve common bottlenecks. + +1. [CPU above 70%](#cpu-above-70) +1. [Memory above 70%](#memory-above-70) +1. [Disk saturated](#disk-saturated) +1. [Unicorn queue above 200–300](#unicorn-queue-above-200300) +1. [Aqueduct backlog above 1–2 hours](#aqueduct-backlog-above-12-hours) + +### CPU above 70% + +If CPU utilization is regularly above 70%: + +* **Increase CPU resources.** + Add at least 20% more vCPUs. +* **Account for new workers.** + Allocate 1 vCPU per worker. For example, if you add 5 unicorn workers and 10 Resque workers, increase vCPUs by at least 15. + +### Memory above 70% + +If memory usage is regularly above 70%: + +* **Increase memory.** + Add additional RAM to reduce average usage below 70%. +* **Account for new workers.** + Allocate 1 GB of memory per worker. For example, if you add 5 unicorn workers and 10 Resque workers, increase memory by at least 15 GB. + +### Disk saturated + +If the disk saturation check indicates saturation, upgrade to disks with higher throughput and maximum IOPS. + +### Unicorn queue above 200–300 + +If unicorn requests are consistently queued above 200–300, you may need to add more unicorn workers. Follow these steps to determine the total target number of workers and update your configuration. + +#### 1. Estimate additional workers + +Run the following command during peak hours to view utilization per worker: + +```shell +ps -ef | grep unicorn | grep -v gitauth | grep -v ".rb" | grep -v init | grep git +``` + +Example output: + +```shell +git 3048972 3045762 0 Aug01 ? 00:07:47 unicorn 3-16-nightly.ghe-test.com[6e6ad46] worker[00]: 20491 reqs, 10.8 req/s, 13ms avg, 85.2% util +git 3048979 3045762 0 Aug01 ? 00:07:53 unicorn 3-16-nightly.ghe-test.com[6e6ad46] worker[01]: 20951 reqs, 12.5 req/s, 13ms avg, 80.3% util +git 3048985 3045762 0 Aug01 ? 00:08:04 unicorn 3-16-nightly.ghe-test.com[6e6ad46] worker[02]: 21502 reqs, 10.5 req/s, 15ms avg, 76.5% util +git 3048992 3045762 0 Aug01 ? 00:07:45 unicorn 3-16-nightly.ghe-test.com[6e6ad46] worker[03]: 20249 reqs, 14.2 req/s, 15ms avg, 86.9% util +``` + +The average requests/second is 12 req/s. + +From this output, calculate the average requests per second (req/s). + +* In the example above: 12 req/s. +* Target is to reduce queued requests to ≤100. +* Formula: + + ```bash + (Queued requests – 100) ÷ avg req/s + ``` + +* Example: (280 – 100) ÷ 12 = 15 additional workers needed. + + >[!TIP] If you want to confirm your findings, you can reach out to us by visiting {% data variables.contact.contact_ent_support %}, uploading a bundle, and asking for the total target number of unicorn workers. + +#### 2. Check current configuration + +Make sure the total number of workers (unicorn + Resque) does not exceed vCPUs. Allocate at least 1 vCPU per worker. + +Check current numbers: + +* Unicorn workers + + ```shell + ps -ef | grep unicorn | grep -v gitauth | grep -v ".rb" | grep -v init | grep git | wc -l + ``` + + Add your calculated number of new workers to this value to get the total target. + +* Resque workers + + ```shell + ps -ef | grep aqueduct-1.1.0 | grep -v "grep aqueduct-1.1.0" | wc -l + ``` + +#### 3. Adjust configuration + +If the sum of unicorn + Resque workers exceeds vCPUs, add more vCPUs before continuing. + +Update the number of unicorn workers: + +```shell +ghe-config app.github.github-workers +ghe-config-apply +``` + +Replace with the total target number of unicorn workers. + +### Aqueduct backlog above 1–2 hours + +If Aqueduct jobs are regularly backlogged for more than 1–2 hours, add resqued-low workers to reduce the risk of queue backups. This issue often worsens after upgrading. + +#### 1. Add resqued-low workers + +* Increase the number of workers by **5–10**. + Be mindful of CPU capacity—each worker requires at least **1 vCPU**. + +```shell +ghe-config app.github.resqued-low-workers +ghe-config-apply +``` + +Replace with the new total number of resqued-low workers. + +#### 2. Validate total worker count + +Ensure the combined number of unicorn + Resque workers does not exceed the total number of vCPUs. See [Unicorn queue above 200–300](#unicorn-queue-above-200300) for instructions on checking current worker configuration. diff --git a/content/admin/upgrading-your-instance/preparing-to-upgrade/index.md b/content/admin/upgrading-your-instance/preparing-to-upgrade/index.md index 6c101ffc398b..61b9c8b157c8 100644 --- a/content/admin/upgrading-your-instance/preparing-to-upgrade/index.md +++ b/content/admin/upgrading-your-instance/preparing-to-upgrade/index.md @@ -11,5 +11,6 @@ children: - /upgrade-requirements - /enabling-automatic-update-checks - /taking-a-snapshot + - /check-system-capacity-before-upgrading shortTitle: Prepare to upgrade --- diff --git a/content/admin/upgrading-your-instance/preparing-to-upgrade/upgrade-requirements.md b/content/admin/upgrading-your-instance/preparing-to-upgrade/upgrade-requirements.md index f48ed4ab73e2..86c9dfa2c8c7 100644 --- a/content/admin/upgrading-your-instance/preparing-to-upgrade/upgrade-requirements.md +++ b/content/admin/upgrading-your-instance/preparing-to-upgrade/upgrade-requirements.md @@ -35,6 +35,7 @@ topics: ## Requirements +* You must perform a capacity check. For more information, see [AUTOTITLE](/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading). * You must upgrade from a feature release that's **at most** two releases behind. For example, to upgrade to {% data variables.product.prodname_enterprise %} {{ enterpriseServerReleases.latest }}, you must be on {% data variables.product.prodname_enterprise %} {{ enterpriseServerReleases.supported[1] }} or {{ enterpriseServerReleases.supported[2] }}. * When upgrading using an upgrade package, schedule a maintenance window for {% data variables.product.prodname_ghe_server %} end users. * {% data reusables.enterprise_installation.hotpatching-explanation %} diff --git a/content/admin/upgrading-your-instance/troubleshooting-upgrades/known-issues-with-upgrades-to-your-instance.md b/content/admin/upgrading-your-instance/troubleshooting-upgrades/known-issues-with-upgrades-to-your-instance.md index bf5cf3482a0a..63fde0c33a7b 100644 --- a/content/admin/upgrading-your-instance/troubleshooting-upgrades/known-issues-with-upgrades-to-your-instance.md +++ b/content/admin/upgrading-your-instance/troubleshooting-upgrades/known-issues-with-upgrades-to-your-instance.md @@ -21,9 +21,9 @@ redirect_from: {% data variables.product.company_short %} strongly recommends regular backups of your instance's configuration and data. Before you proceed with any upgrade, back up your instance, then validate the backup in a staging environment. For more information, see [AUTOTITLE](/admin/configuration/configuring-your-enterprise/configuring-backups-on-your-appliance) and [AUTOTITLE](/admin/installation/setting-up-a-github-enterprise-server-instance/setting-up-a-staging-instance). -## Hold off on upgrading to 3.15 and above +## Lifting the pause on upgrades to version 3.15 and above -We have received a few reports of performance issues with {% data variables.product.prodname_ghe_server %} versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. +We have lifted the pause on upgrades to versions 3.15, 3.16, and 3.17. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. As an additional step, it is recommended to check system capacity before upgrading. See [AUTOTITLE](/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading). We are extending the support window for versions 3.14, 3.15, 3.16, and 3.17. The support window for 3.13 remains unchanged. The closing down date for each of 3.14, 3.15, 3.16, and 3.17 has been updated to "Support temporarily extended until further notice". For more information, see [AUTOTITLE](/admin/all-releases#releases-of-github-enterprise-server). diff --git a/data/release-notes/enterprise-server/3-14/17.yml b/data/release-notes/enterprise-server/3-14/17.yml new file mode 100644 index 000000000000..0dc5bad3ff9e --- /dev/null +++ b/data/release-notes/enterprise-server/3-14/17.yml @@ -0,0 +1,50 @@ +date: '2025-08-19' +sections: + security_fixes: + - | + **HIGH:** An improper access control vulnerability was identified that allowed authenticated users to obtain code content from private repositories they did not have permission to access. If a user knew the names of a private repository and its branches, tags, or commit SHAs, they could use the compare/diff functionality to retrieve code from those repositories without authorization. Exploiting this vulnerability also required the attacker to have legitimate access to another repository within the same fork network. This vulnerability has been assigned [CVE-2025-8447](https://www.cve.org/cverecord?id=CVE-2025-8447) and was reported through the [GitHub Bug Bounty program](https://bounty.github.com/). + - | + Packages have been updated to the latest security versions. + - | + Elasticsearch packages have been updated to the 8.18.0 security version. + bugs: + - | + After enabling GitHub Actions or performing an upgrade with GitHub Actions enabled, administrators experienced a delay of approximately 10 minutes longer than they should have due to a faulty connection check. This is fixed for future enablement and upgrades. + - | + After upgrading to GHES 3.14.16, GHES 3.15.11, GHES 3.16.7, or GHES 3.17.4, administrators found that draft pull requests for private repositories were no longer available. + changes: + - | + When administrators run the `ghe-support-bundle` command on an unconfigured node, the output clearly states that metadata collection was skipped, instead of producing misleading `curl` errors. This improves the clarity of support bundle diagnostics. + known_issues: + - | + During the validation phase of a configuration run, a `No such object` error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start. + - | + If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see [AUTOTITLE](/admin/configuration/administering-your-instance-from-the-management-console/troubleshooting-access-to-the-management-console#unlocking-the-root-site-administrator-account). + - | + On an instance with the HTTP `X-Forwarded-For` header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. + - | + {% data reusables.release-notes.large-adoc-files-issue %} + - | + Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised. + - | + When following the steps for [Replacing the primary MySQL node](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-the-primary-mysql-node), step 14 (running `ghe-cluster-config-apply`) may fail with errors. If this occurs, re-running `ghe-cluster-config-apply` is expected to succeed. + - | + Running `ghe-cluster-config-apply` as part of the steps for [Replacing a node in an emergency](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-a-node-in-an-emergency) may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps. + - | + {% data reusables.release-notes.2024-06-possible-frontend-5-minute-outage-during-hotpatch-upgrade %} + - | + When restoring data originally backed up from a 3.13 or greater appliance version, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair`. + - | + An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning. + - | + In the header bar displayed to site administrators, some icons are not available. + - | + When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded. + - | + When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed. + - | + After a restore, existing outside collaborators cannot be added to repositories in a new organization. This issue can be resolved by running `/usr/local/share/enterprise/ghe-es-search-repair` on the appliance. + - | + After a geo-replica is promoted to be a primary by running `ghe-repl-promote`, the actions workflow of a repository does not have any suggested workflows. + - | + Unexpected elements may appear in the UI on the repository overview page for locked repositories. diff --git a/data/release-notes/enterprise-server/3-15/0.yml b/data/release-notes/enterprise-server/3-15/0.yml index aed46f444af6..84ba7240e87a 100644 --- a/data/release-notes/enterprise-server/3-15/0.yml +++ b/data/release-notes/enterprise-server/3-15/0.yml @@ -4,7 +4,7 @@ deprecated: false intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/1.yml b/data/release-notes/enterprise-server/3-15/1.yml index 8e6f7b352fa6..4ee08451446c 100644 --- a/data/release-notes/enterprise-server/3-15/1.yml +++ b/data/release-notes/enterprise-server/3-15/1.yml @@ -2,7 +2,7 @@ date: '2024-12-17' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/10.yml b/data/release-notes/enterprise-server/3-15/10.yml index 4597fc3558f9..f7d19f69b21b 100644 --- a/data/release-notes/enterprise-server/3-15/10.yml +++ b/data/release-notes/enterprise-server/3-15/10.yml @@ -2,7 +2,7 @@ date: '2025-07-15' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/11.yml b/data/release-notes/enterprise-server/3-15/11.yml index 7266aa06b576..f7f12392b8e6 100644 --- a/data/release-notes/enterprise-server/3-15/11.yml +++ b/data/release-notes/enterprise-server/3-15/11.yml @@ -2,7 +2,7 @@ date: '2025-07-29' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: @@ -63,4 +63,4 @@ sections: - | The autolink references feature is missing from the repository settings page. - | - When attempting to open a pull request as a draft in a private or internal repository, users are incorrectly prompted to upgrade their plan.[Updated: 2025-08-11] \ No newline at end of file + When attempting to open a pull request as a draft in a private or internal repository, users are incorrectly prompted to upgrade their plan.[Updated: 2025-08-11] diff --git a/data/release-notes/enterprise-server/3-15/12.yml b/data/release-notes/enterprise-server/3-15/12.yml new file mode 100644 index 000000000000..7aee3db2498f --- /dev/null +++ b/data/release-notes/enterprise-server/3-15/12.yml @@ -0,0 +1,70 @@ +date: '2025-08-19' +intro: | + {% warning %} + + **Warning**: We are lifting the pause on upgrade to 3.15. You can now upgrade to version 3.15.12, but not to earlier releases of 3.15. This release includes optimizations that address performance issues reported in recent versions of GitHub Enterprise Server. As an additional step, it is recommended to check system capacity before upgrading. See [check system capacity before upgrading](/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading). + + {% endwarning %} +sections: + security_fixes: + - | + **HIGH:** An improper access control vulnerability was identified that allowed authenticated users to obtain code content from private repositories they did not have permission to access. If a user knew the names of a private repository and its branches, tags, or commit SHAs, they could use the compare/diff functionality to retrieve code from those repositories without authorization. Exploiting this vulnerability also required the attacker to have legitimate access to another repository within the same fork network. This vulnerability has been assigned [CVE-2025-8447](https://www.cve.org/cverecord?id=CVE-2025-8447) and was reported through the [GitHub Bug Bounty program](https://bounty.github.com/). + - | + Packages have been updated to the latest security versions. + - | + Elasticsearch packages have been updated to the 8.18.0 security version. + - | + The maintenance page in the Management Console did not include cross-site request forgery (CSRF) protection. + bugs: + - | + For enterprises with a large number of organizations, some authorization queries were non-performant. This patch includes a set of fixes improving the performance of authorization checks that enforce PAT access policies for both fine-grained and classic {% data variables.product.pat_generic_title_case_plural %} (PATs). + - | + On instances in a cluster configuration, builds of GitHub Pages sites timed out in GitHub Actions workflows. + - | + After enabling GitHub Actions or performing an upgrade with GitHub Actions enabled, administrators experienced a delay of approximately 10 minutes longer than they should have due to a faulty connection check. This is fixed for future enablement and upgrades. + - | + Secret scanning backfills for pull requests and discussions did not run as expected during backfills of new secret types. Site administrators and security teams may have noticed incomplete secret scanning coverage or unworked queues after upgrading. + - | + Site administrators observed that uploading a license failed to restart GitHub services after upgrading GitHub Enterprise Server due to file permission issues in `/var/log/license-upgrade`. + - | + After upgrading to GHES 3.15.11, GHES 3.16.7, or GHES 3.17.4, administrators found that draft pull requests for private repositories were no longer available. + changes: + - | + When administrators run the `ghe-support-bundle` command on an unconfigured node, the output clearly states that metadata collection was skipped, instead of producing misleading `curl` errors. This improves the clarity of support bundle diagnostics. + - | + Configuration runs do not output transient Elasticsearch health check failures. This update reduces log verbosity to address confusion reported by users. + - | + For administrators monitoring search index repairs, logs for repair jobs now include batch-level details, such as the ranges of updated IDs. This improvement makes it easier to track and debug the status of index repairs. + known_issues: + - | + During the validation phase of a configuration run, a `No such object` error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start. + - | + If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see [AUTOTITLE](/admin/configuration/administering-your-instance-from-the-management-console/troubleshooting-access-to-the-management-console#unlocking-the-root-site-administrator-account). + - | + On an instance with the HTTP `X-Forwarded-For` header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. + - | + {% data reusables.release-notes.large-adoc-files-issue %} + - | + Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised. + - | + When following the steps for [Replacing the primary MySQL node](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-the-primary-mysql-node), step 14 (running `ghe-cluster-config-apply`) may fail with errors. If this occurs, re-running `ghe-cluster-config-apply` is expected to succeed. + - | + Running `ghe-cluster-config-apply` as part of the steps for [Replacing a node in an emergency](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-a-node-in-an-emergency) may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps. + - | + {% data reusables.release-notes.2024-06-possible-frontend-5-minute-outage-during-hotpatch-upgrade %} + - | + When restoring data originally backed up from a 3.13 or greater appliance version, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair`. + - | + An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning. + - | + In the header bar displayed to site administrators, some icons are not available. + - | + When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded. + - | + When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed. + - | + When initializing a new GHES cluster, nodes with the `consul-server` role should be added to the cluster before adding additional nodes. Adding all nodes simultaneously creates a race condition between nomad server registration and nomad client registration. + - | + Admins setting up cluster high availability (HA) may encounter a spokes error when running `ghe-cluster-repl-status` if a new organization and repositories are created before using the `ghe-cluster-repl-bootstrap` command. To avoid this issue, complete the cluster HA setup with `ghe-cluster-repl-bootstrap` before creating new organizations and repositories. + - | + After a restore, existing outside collaborators cannot be added to repositories in a new organization. This issue can be resolved by running `/usr/local/share/enterprise/ghe-es-search-repair` on the appliance. diff --git a/data/release-notes/enterprise-server/3-15/2.yml b/data/release-notes/enterprise-server/3-15/2.yml index b6458c00cfdc..d6bf3b332728 100644 --- a/data/release-notes/enterprise-server/3-15/2.yml +++ b/data/release-notes/enterprise-server/3-15/2.yml @@ -2,7 +2,7 @@ date: '2025-01-21' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/3.yml b/data/release-notes/enterprise-server/3-15/3.yml index 3e0e3331c709..fc4ddd749722 100644 --- a/data/release-notes/enterprise-server/3-15/3.yml +++ b/data/release-notes/enterprise-server/3-15/3.yml @@ -2,7 +2,7 @@ date: '2025-02-18' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} {% warning %} diff --git a/data/release-notes/enterprise-server/3-15/4.yml b/data/release-notes/enterprise-server/3-15/4.yml index e57e00619f6d..3bfb4260116c 100644 --- a/data/release-notes/enterprise-server/3-15/4.yml +++ b/data/release-notes/enterprise-server/3-15/4.yml @@ -2,7 +2,7 @@ date: '2025-03-04' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/5.yml b/data/release-notes/enterprise-server/3-15/5.yml index 4da9ac092715..edbc167dc29b 100644 --- a/data/release-notes/enterprise-server/3-15/5.yml +++ b/data/release-notes/enterprise-server/3-15/5.yml @@ -2,7 +2,7 @@ date: '2025-03-25' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/6.yml b/data/release-notes/enterprise-server/3-15/6.yml index 824b1950dc13..e707f67b92ff 100644 --- a/data/release-notes/enterprise-server/3-15/6.yml +++ b/data/release-notes/enterprise-server/3-15/6.yml @@ -2,7 +2,7 @@ date: '2025-04-17' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/7.yml b/data/release-notes/enterprise-server/3-15/7.yml index 2c2fda18843e..01a02dab8fef 100644 --- a/data/release-notes/enterprise-server/3-15/7.yml +++ b/data/release-notes/enterprise-server/3-15/7.yml @@ -2,7 +2,7 @@ date: '2025-05-27' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/8.yml b/data/release-notes/enterprise-server/3-15/8.yml index 89e351b453bd..9bd5e35cfd7c 100644 --- a/data/release-notes/enterprise-server/3-15/8.yml +++ b/data/release-notes/enterprise-server/3-15/8.yml @@ -2,7 +2,7 @@ date: '2025-06-18' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-15/9.yml b/data/release-notes/enterprise-server/3-15/9.yml index 810d79c8ac35..e4a6b882b0c6 100644 --- a/data/release-notes/enterprise-server/3-15/9.yml +++ b/data/release-notes/enterprise-server/3-15/9.yml @@ -2,7 +2,7 @@ date: '2025-07-01' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-16/0.yml b/data/release-notes/enterprise-server/3-16/0.yml index 2e42b4745548..7c10a01aacd7 100644 --- a/data/release-notes/enterprise-server/3-16/0.yml +++ b/data/release-notes/enterprise-server/3-16/0.yml @@ -4,7 +4,7 @@ deprecated: false intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} {% warning %} diff --git a/data/release-notes/enterprise-server/3-16/1.yml b/data/release-notes/enterprise-server/3-16/1.yml index 6da127512384..62b62efe3b8e 100644 --- a/data/release-notes/enterprise-server/3-16/1.yml +++ b/data/release-notes/enterprise-server/3-16/1.yml @@ -2,7 +2,7 @@ date: '2025-03-25' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-16/2.yml b/data/release-notes/enterprise-server/3-16/2.yml index 98370513c51e..b9b2915f2486 100644 --- a/data/release-notes/enterprise-server/3-16/2.yml +++ b/data/release-notes/enterprise-server/3-16/2.yml @@ -2,7 +2,7 @@ date: '2025-04-17' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-16/3.yml b/data/release-notes/enterprise-server/3-16/3.yml index 60124b6df32c..5e2fb0a9d76a 100644 --- a/data/release-notes/enterprise-server/3-16/3.yml +++ b/data/release-notes/enterprise-server/3-16/3.yml @@ -2,7 +2,7 @@ date: '2025-05-27' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-16/4.yml b/data/release-notes/enterprise-server/3-16/4.yml index b57a2760a9f7..304bdb1bb960 100644 --- a/data/release-notes/enterprise-server/3-16/4.yml +++ b/data/release-notes/enterprise-server/3-16/4.yml @@ -2,7 +2,7 @@ date: '2025-06-18' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-16/5.yml b/data/release-notes/enterprise-server/3-16/5.yml index 8e3933cb7ff9..c23ffb5ae54e 100644 --- a/data/release-notes/enterprise-server/3-16/5.yml +++ b/data/release-notes/enterprise-server/3-16/5.yml @@ -2,7 +2,7 @@ date: '2025-07-01' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-16/6.yml b/data/release-notes/enterprise-server/3-16/6.yml index 026f928c18dc..717bdc5f016e 100644 --- a/data/release-notes/enterprise-server/3-16/6.yml +++ b/data/release-notes/enterprise-server/3-16/6.yml @@ -2,7 +2,7 @@ date: '2025-07-15' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-16/7.yml b/data/release-notes/enterprise-server/3-16/7.yml index 7c8097f024bc..e1f21222baeb 100644 --- a/data/release-notes/enterprise-server/3-16/7.yml +++ b/data/release-notes/enterprise-server/3-16/7.yml @@ -2,7 +2,7 @@ date: '2025-07-29' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: @@ -77,4 +77,4 @@ sections: - | The autolink references feature is missing from the repository settings page. - | - When attempting to open a pull request as a draft in a private or internal repository, users are incorrectly prompted to upgrade their plan.[Updated: 2025-08-11] \ No newline at end of file + When attempting to open a pull request as a draft in a private or internal repository, users are incorrectly prompted to upgrade their plan.[Updated: 2025-08-11] diff --git a/data/release-notes/enterprise-server/3-16/8.yml b/data/release-notes/enterprise-server/3-16/8.yml new file mode 100644 index 000000000000..728ba2d5152a --- /dev/null +++ b/data/release-notes/enterprise-server/3-16/8.yml @@ -0,0 +1,82 @@ +date: '2025-08-19' +intro: | + {% warning %} + + **Warning**: We are lifting the pause on upgrade to 3.16. You can now upgrade to version 3.16.8, but not to earlier releases of 3.16. This release includes optimizations that address performance issues reported in recent versions of GitHub Enterprise Server. As an additional step, it is recommended to check system capacity before upgrading. See [check system capacity before upgrading](/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading). + + {% endwarning %} +sections: + security_fixes: + - | + **HIGH:** An improper access control vulnerability was identified that allowed authenticated users to obtain code content from private repositories they did not have permission to access. If a user knew the names of a private repository and its branches, tags, or commit SHAs, they could use the compare/diff functionality to retrieve code from those repositories without authorization. Exploiting this vulnerability also required the attacker to have legitimate access to another repository within the same fork network. This vulnerability has been assigned [CVE-2025-8447](https://www.cve.org/cverecord?id=CVE-2025-8447) and was reported through the [GitHub Bug Bounty program](https://bounty.github.com/). + - | + Packages have been updated to the latest security versions. + - | + Elasticsearch packages have been updated to the 8.18.0 security version. + bugs: + - | + For enterprises with a large number of organizations, some authorization queries were non-performant. This patch includes a set of fixes improving the performance of authorization checks that enforce PAT access policies for both fine-grained and classic {% data variables.product.pat_generic_title_case_plural %} (PATs). + - | + After enabling GitHub Actions or performing an upgrade with GitHub Actions enabled, administrators experienced a delay of approximately 10 minutes longer than they should have due to a faulty connection check. + - | + Secret scanning backfills for pull requests and discussions did not run as expected during backfills of new secret types. Site administrators and security teams may have noticed incomplete secret scanning coverage or unworked queues after upgrading. + - | + Site administrators observed that uploading a license failed to restart GitHub services after upgrading GitHub Enterprise Server due to file permission issues in `/var/log/license-upgrade`. + - | + Audit log entries for some Dependabot-related events were missing for administrators and security teams due to an outdated allowlist configuration. + - | + Administrators debugging Elasticsearch index repairs previously did not see a "starting" log entry before a repair began, making it harder to track repair initiation in logs. + - | + After upgrading to GHES 3.16.7, or GHES 3.17.4, administrators found that draft pull requests for private repositories were no longer available. + - | + Site administrators experienced crashes in MySQL when running data backfills, such as during database maintenance or upgrades. + changes: + - | + When administrators run the `ghe-support-bundle` command on an unconfigured node, the output clearly states that metadata collection was skipped, instead of producing misleading `curl` errors. This improves the clarity of support bundle diagnostics. + - | + Configuration runs do not output transient Elasticsearch health check failures. This update reduces log verbosity to address confusion reported by users. + - | + For administrators monitoring search index repairs, logs for repair jobs now include batch-level details, such as the ranges of updated IDs. This improvement makes it easier to track and debug the status of index repairs. + - | + Administrators monitoring Elasticsearch index repair jobs benefit from improved log clarity. Log messages provide more detailed and actionable information, making it easier to troubleshoot and track the progress of index repair operations. + known_issues: + - | + Custom firewall rules are removed during the upgrade process. + - | + During the validation phase of a configuration run, a `No such object` error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start. + - | + If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see [AUTOTITLE](/admin/configuration/administering-your-instance-from-the-management-console/troubleshooting-access-to-the-management-console#unlocking-the-root-site-administrator-account). + - | + On an instance with the HTTP `X-Forwarded-For` header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. + - | + {% data reusables.release-notes.large-adoc-files-issue %} + - | + Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised. + - | + When following the steps for [Replacing the primary MySQL node](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-the-primary-mysql-node), step 14 (running `ghe-cluster-config-apply`) may fail with errors. If this occurs, re-running `ghe-cluster-config-apply` is expected to succeed. + - | + Running `ghe-cluster-config-apply` as part of the steps for [Replacing a node in an emergency](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-a-node-in-an-emergency) may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps. + - | + {% data reusables.release-notes.2024-06-possible-frontend-5-minute-outage-during-hotpatch-upgrade %} + - | + When restoring data originally backed up from a 3.13 or greater appliance version, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair`. + - | + An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning. + - | + When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded. + - | + When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed. + - | + When initializing a new GHES cluster, nodes with the `consul-server` role should be added to the cluster before adding additional nodes. Adding all nodes simultaneously creates a race condition between nomad server registration and nomad client registration. + - | + Admins setting up cluster high availability (HA) may encounter a spokes error when running `ghe-cluster-repl-status` if a new organization and repositories are created before using the `ghe-cluster-repl-bootstrap` command. To avoid this issue, complete the cluster HA setup with `ghe-cluster-repl-bootstrap` before creating new organizations and repositories. + - | + In a cluster, the host running restore requires access the storage nodes via their private IPs. + - | + On an instance hosted on Azure, commenting on an issue via email meant the comment was not added to the issue. + - | + After a restore, existing outside collaborators cannot be added to repositories in a new organization. This issue can be resolved by running `/usr/local/share/enterprise/ghe-es-search-repair` on the appliance. + - | + After a geo-replica is promoted to be a primary by running `ghe-repl-promote`, the actions workflow of a repository does not have any suggested workflows. + - | + Customers operating at high scale or near capacity may experience unexpected performance degradation, such as slow response times, background job queue spikes, elevated CPU usage, and increased MySQL load. Consider upgrading to {% ifversion ghes = 3.16 %}3.16{% endif %}{% ifversion ghes = 3.17 %}3.17{% endif %}{% ifversion ghes = 3.18 %}3.18{% endif %} with caution. diff --git a/data/release-notes/enterprise-server/3-17/0.yml b/data/release-notes/enterprise-server/3-17/0.yml index 92630b629e7d..9f14747b8339 100644 --- a/data/release-notes/enterprise-server/3-17/0.yml +++ b/data/release-notes/enterprise-server/3-17/0.yml @@ -4,7 +4,7 @@ deprecated: false intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-17/1.yml b/data/release-notes/enterprise-server/3-17/1.yml index 8a0242ec96d9..c784a4eeb148 100644 --- a/data/release-notes/enterprise-server/3-17/1.yml +++ b/data/release-notes/enterprise-server/3-17/1.yml @@ -2,7 +2,7 @@ date: '2025-06-18' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-17/2.yml b/data/release-notes/enterprise-server/3-17/2.yml index 5f880dd937af..4d45876b0996 100644 --- a/data/release-notes/enterprise-server/3-17/2.yml +++ b/data/release-notes/enterprise-server/3-17/2.yml @@ -2,7 +2,7 @@ date: '2025-07-01' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-17/3.yml b/data/release-notes/enterprise-server/3-17/3.yml index aa653558e196..8f47a7d0c89b 100644 --- a/data/release-notes/enterprise-server/3-17/3.yml +++ b/data/release-notes/enterprise-server/3-17/3.yml @@ -2,7 +2,7 @@ date: '2025-07-15' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: diff --git a/data/release-notes/enterprise-server/3-17/4.yml b/data/release-notes/enterprise-server/3-17/4.yml index 9c42bfcd8f36..6d8f3041e665 100644 --- a/data/release-notes/enterprise-server/3-17/4.yml +++ b/data/release-notes/enterprise-server/3-17/4.yml @@ -2,7 +2,7 @@ date: '2025-07-29' intro: | {% warning %} - **Warning**: We have received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17. Out of an abundance of caution, we recommend holding off on upgrading to these versions until further notice. [Updated: 2025-08-01] + **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25] {% endwarning %} sections: @@ -73,4 +73,4 @@ sections: - | The autolink references feature is missing from the repository settings page. - | - When attempting to open a pull request as a draft in a private or internal repository, users are incorrectly prompted to upgrade their plan.[Updated: 2025-08-11] \ No newline at end of file + When attempting to open a pull request as a draft in a private or internal repository, users are incorrectly prompted to upgrade their plan.[Updated: 2025-08-11] diff --git a/data/release-notes/enterprise-server/3-17/5.yml b/data/release-notes/enterprise-server/3-17/5.yml new file mode 100644 index 000000000000..857227f57576 --- /dev/null +++ b/data/release-notes/enterprise-server/3-17/5.yml @@ -0,0 +1,84 @@ +date: '2025-08-19' +intro: | + {% warning %} + + **Warning**: We are lifting the pause on upgrade to 3.17. You can now upgrade to version 3.17.5, but not to earlier releases of 3.17. This release includes optimizations that address performance issues reported in recent versions of GitHub Enterprise Server. As an additional step, it is recommended to check system capacity before upgrading. See [check system capacity before upgrading](/admin/upgrading-your-instance/preparing-to-upgrade/check-system-capacity-before-upgrading). + + {% endwarning %} +sections: + security_fixes: + - | + **HIGH:** An improper access control vulnerability was identified that allowed authenticated users to obtain code content from private repositories they did not have permission to access. If a user knew the names of a private repository and its branches, tags, or commit SHAs, they could use the compare/diff functionality to retrieve code from those repositories without authorization. Exploiting this vulnerability also required the attacker to have legitimate access to another repository within the same fork network. This vulnerability has been assigned [CVE-2025-8447](https://www.cve.org/cverecord?id=CVE-2025-8447) and was reported through the [GitHub Bug Bounty program](https://bounty.github.com/). + - | + **LOW:** In repositories where delegated alert dismissal was enabled, requests to resolve a secret scanning alert using the REST API were accepted when the actor had insufficient permission. The endpoint checked to see if the actor had permission to resolve secret scanning alerts but failed to verify that the actor was also a valid reviewer. This could allow an actor to bypass the review process. The endpoint was updated to use the same logic as the UI. + - | + Elasticsearch packages have been updated to the 8.18.0 security version. + - | + Packages have been updated to the latest security versions. + bugs: + - | + For enterprises with a large number of organizations, some authorization queries were non-performant. This patch includes a set of fixes improving the performance of authorization checks that enforce PAT access policies for both fine-grained and classic {% data variables.product.pat_generic_title_case_plural %} (PATs). + - | + After enabling GitHub Actions or performing an upgrade with GitHub Actions enabled, administrators experienced a delay of approximately 10 minutes longer than they should have due to a faulty connection check. This is fixed for future enablement and upgrades. + - | + Site administrators observed that secondary database nodes, including those in replica clusters, were unnecessarily rebuilding indexes meant for the primary database during maintenance operations, leading to data redundancy in secondary databases. + - | + Secret scanning backfills for pull requests and discussions did not run as expected during backfills of new secret types. Site administrators and security teams may have noticed incomplete secret scanning coverage or unworked queues after upgrading. + - | + Site administrators observed that uploading a license failed to restart GitHub services after upgrading GitHub Enterprise Server due to file permission issues in `/var/log/license-upgrade`. + - | + Organization administrators and integrators reviewing bypass requests using the API could not filter requests by the `approved` status. + - | + On instances configured for SAML authentication only, site administrators could not create new users via the API. + - | + Administrators debugging Elasticsearch index repairs previously did not see a "starting" log entry before a repair began, making it harder to track repair initiation in logs. + - | + Audit log entries for some Dependabot-related events were missing for administrators and security teams due to an outdated allowlist configuration. + - | + After upgrading to GHES 3.17.4, administrators found that draft pull requests for private repositories were no longer available. + - | + Site administrators experienced crashes in MySQL when running data backfills, such as during database maintenance or upgrades. + changes: + - | + When administrators run the `ghe-support-bundle` command on an unconfigured node, the output clearly states that metadata collection was skipped, instead of producing misleading `curl` errors. This improves the clarity of support bundle diagnostics. + - | + Configuration runs dont output transient Elasticsearch health check failures. This update reduces log verbosity to address confusion reported by users. + - | + Organization API responses included in migration workflows did not return all member privileges required by import APIs. Additional member privilege fields are now included in the organization hash when retrieving organization data via the API. + - | + For administrators monitoring search index repairs, logs for repair jobs now include batch-level details, such as the ranges of updated IDs. This improvement makes it easier to track and debug the status of index repairs. + - | + Administrators monitoring Elasticsearch index repair jobs benefit from improved log clarity. Log messages provide more detailed and actionable information, making it easier to troubleshoot and track the progress of index repair operations. + known_issues: + - | + Custom firewall rules are removed during the upgrade process. + - | + During the validation phase of a configuration run, a `No such object` error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start. + - | + If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see [AUTOTITLE](/admin/configuration/administering-your-instance-from-the-management-console/troubleshooting-access-to-the-management-console#unlocking-the-root-site-administrator-account). + - | + {% data reusables.release-notes.large-adoc-files-issue %} + - | + Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised. + - | + When following the steps for [Replacing the primary MySQL node](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-the-primary-mysql-node), step 14 (running `ghe-cluster-config-apply`) may fail with errors. If this occurs, re-running `ghe-cluster-config-apply` is expected to succeed. + - | + Running `ghe-cluster-config-apply` as part of the steps for [Replacing a node in an emergency](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-a-node-in-an-emergency) may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps. + - | + When restoring data originally backed up from a 3.13 or greater appliance version, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair`. + - | + When initializing a new GHES cluster, nodes with the `consul-server` role should be added to the cluster before adding additional nodes. Adding all nodes simultaneously creates a race condition between nomad server registration and nomad client registration. + - | + Admins setting up cluster high availability (HA) may encounter a spokes error when running `ghe-cluster-repl-status` if a new organization and repositories are created before using the `ghe-cluster-repl-bootstrap` command. To avoid this issue, complete the cluster HA setup with `ghe-cluster-repl-bootstrap` before creating new organizations and repositories. + - | + In a cluster, the host running restore requires access the storage nodes via their private IPs. + - | + On an instance hosted on Azure, commenting on an issue via email meant the comment was not added to the issue. + - | + After a restore, existing outside collaborators cannot be added to repositories in a new organization. This issue can be resolved by running `/usr/local/share/enterprise/ghe-es-search-repair` on the appliance. + - | + After a geo-replica is promoted to be a primary by running `ghe-repl-promote`, the actions workflow of a repository does not have any suggested workflows. + - | + When publishing npm packages in a workflow after restoring from a backup to GitHub Enterprise Server 3.13.5.gm4 or 3.14.2.gm3, you may encounter a `401 Unauthorized` error from the GitHub Packages service. This can happen if the restore is from an N-1 or N-2 version and the workflow targets the npm endpoint on the backup instance. To avoid this issue, ensure the access token is valid and includes the correct scopes for publishing to GitHub Packages. + - | + Customers operating at high scale or near capacity may experience unexpected performance degradation, such as slow response times, background job queue spikes, elevated CPU usage, and increased MySQL load. Consider upgrading to {% ifversion ghes = 3.16 %}3.16{% endif %}{% ifversion ghes = 3.17 %}3.17{% endif %}{% ifversion ghes = 3.18 %}3.18{% endif %} with caution.