From 14d6e216f69b6b2a78db6bdc6fa0f7c93a3e6bed Mon Sep 17 00:00:00 2001 From: nginx-seanmoloney <133861979+nginx-seanmoloney@users.noreply.github.com> Date: Fri, 6 Jun 2025 10:57:03 +0100 Subject: [PATCH 001/117] Update instructions for upgrading NGINX Agent (#648) * Update instructions for upgrading NGINX Agent * PR feedback Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> * PR feedback Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> * PR feedback Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> --------- Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> --- content/agent/installation-upgrade/upgrade.md | 57 ++++++++++--------- 1 file changed, 30 insertions(+), 27 deletions(-) diff --git a/content/agent/installation-upgrade/upgrade.md b/content/agent/installation-upgrade/upgrade.md index 50477e94b..cd11db359 100644 --- a/content/agent/installation-upgrade/upgrade.md +++ b/content/agent/installation-upgrade/upgrade.md @@ -12,9 +12,7 @@ type: Learn how to upgrade NGINX Agent. -## Upgrade NGINX Agent from version v2.31.0 or greater - -{{< note >}} Starting from version v2.31.0, NGINX Agent will automatically restart itself during an upgrade. {{< /note >}} +## Upgrade NGINX Agent To upgrade NGINX Agent, follow these steps: @@ -41,43 +39,48 @@ To upgrade NGINX Agent, follow these steps: sudo apt-get install -y --only-upgrade nginx-agent -o Dpkg::Options::="--force-confold" ``` +## Upgrade NGINX Agent to a Specific Version +To upgrade NGINX Agent to a specific **v2.x version**, follow these steps: -## Upgrade NGINX Agent from a version less than v2.31.0 - -To upgrade NGINX Agent, take the following steps: +#### Steps to Upgrade: -1. Open an SSH connection to the server where you’ve installed NGINX Agent and log in. +1. Open an SSH connection to the server running NGINX Agent and log in. -1. Make a backup copy of the following locations to ensure that you can successfully recover if the upgrade has issues: +1. Back up the following files and directories to ensure you can restore the environment in case of issues during the upgrade: - `/etc/nginx-agent` - - `config_dirs` values for any configuration specified in `/etc/nginx-agent/nginx-agent.conf` - -1. Stop NGINX Agent: + - Any `config_dirs` directory specified in `/etc/nginx-agent/nginx-agent.conf`. - ```shell - sudo systemctl stop nginx-agent - ``` - -1. Install the updated version of NGINX Agent: - - - CentOS, RHEL, RPM-Based +1. Perform the version-controlled upgrade. + - Debian, Ubuntu, Deb-Based + ```shell - sudo yum -y makecache - sudo yum update -y nginx-agent + sudo apt-get update + sudo apt-get install -y nginx-agent= -o Dpkg::Options::="--force-confold" ``` - - - Debian, Ubuntu, Deb-Based - + + Example (to upgrade to version 2.41.1~noble): + ```shell - sudo apt-get update - sudo apt-get install -y --only-upgrade nginx-agent -o Dpkg::Options::="--force-confold" + sudo apt-get install -y nginx-agent=2.41.1~noble -o Dpkg::Options::="--force-confold" ``` -1. Start NGINX Agent: + - CentOS, RHEL, RPM-Based + + ```shell + sudo yum install -y nginx-agent- + ``` + + Example (to upgrade to version `2.41.1`): + + ```shell + sudo yum install -y nginx-agent-2.41.1 + ``` +1. Verify the installed version: + ```shell - sudo systemctl start nginx-agent + sudo nginx-agent --version ``` From 39aed75510db87ea491b4bf18e98d599b4de2288 Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Fri, 6 Jun 2025 11:19:41 -0700 Subject: [PATCH 002/117] Add custom landing page to N1 via index.md (#636) * feat: Add custom landing page to N1 via index.md * Removed list-main * Finish adding all the custom content for N1 landing page * Added changelog shortcode to Changelog card --- content/nginx-one/_index.md | 65 +++++++++++++++++ layouts/partials/list-main.html | 119 -------------------------------- 2 files changed, 65 insertions(+), 119 deletions(-) delete mode 100644 layouts/partials/list-main.html diff --git a/content/nginx-one/_index.md b/content/nginx-one/_index.md index 21284cbf0..27bf8dbfe 100644 --- a/content/nginx-one/_index.md +++ b/content/nginx-one/_index.md @@ -2,7 +2,72 @@ title: F5 NGINX One Console description: The F5 NGINX One Console makes it easy to manage NGINX instances across locations and environments. The console lets you monitor and control your NGINX fleet from one place—you can check configurations, track performance metrics, identify security vulnerabilities, manage SSL certificates, and more. url: /nginx-one/ +hasCustomContent: true cascade: logo: "NGINX-One-product-icon.svg" --- +{{< card-layout >}} + {{< card-section >}} + {{< card title="Manage your NGINX fleet" >}} + Simplify, scale, secure, and collaborate with your NGINX fleet + {{}} + {{< card title="Get started" >}} + See benefits from the NGINX One Console + {{}} + {{< card title="Connect your instances" >}} + Work with data plane keys, containers, and proxy servers + {{}} + {{< card title="Manage your NGINX instances" >}} + Monitor and maintain your deployments + {{}} + {{< card title="Draft new configurations" >}} + Work with Staged Configurations + {{}} + {{< card title="Set up metrics" >}} + Review your deployments in a dashboard + {{}} + {{< card title="Organize users with RBAC" >}} + Assign responsibilities with role-based access control + {{}} + {{< card title="Automate with the NGINX One API" >}} + Manage your NGINX fleet over REST + {{}} + {{< card title="Glossary" >}} + Learn terms unique to NGINX One Console + {{}} + {{< card title="Changelog" >}} + {{< changelog-dates >}} + {{}} + {{}} + # Other Products + {{< card-section title="Kubernetes Solutions">}} + {{< card title="NGINX Ingress Controller" titleUrl="/nginx-ingress-controller/" icon="NGINX-Ingress-Controller-product-icon">}} + Kubernetes traffic management with API gateway, identity, and observability features. + {{}} + {{< card title="NGINX Gateway Fabric" titleUrl="/nginx-gateway-fabric" icon="NGINX-product-icon">}} + Next generation Kubernetes connectivity using the Gateway API. + {{}} + {{}} + {{< card-section title="Local Console Option">}} + {{< card title="NGINX Instance Manager" titleUrl="/nginx-instance-manager" icon="NGINX-Instance-Manager-product-icon">}} + Track and control NGINX Open Source and NGINX Plus instances. + {{}} + {{}} + {{< card-section title="Modern App Delivery">}} + {{< card title="NGINX Plus" titleUrl="/nginx" icon="NGINX-Plus-product-icon-RGB">}} + The all-in-one load balancer, reverse proxy, web server, content cache, and API gateway. + {{}} + {{< card title="NGINX Open Source" titleUrl="nginx.org" icon="NGINX-product-icon">}} + The open source all-in-one load balancer, content cache, and web server + {{}} + {{}} + {{< card-section title="Security">}} + {{< card title="NGINX App Protect WAF" titleUrl="/nginx-app-protect-waf" icon="NGINX-App-Protect-WAF-product-icon">}} + Lightweight, high-performance, advanced protection against Layer 7 attacks on your apps and APIs. + {{}} + {{< card title="NGINX App Protect DoS" titleUrl="/nginx-app-protect-dos" icon="NGINX-App-Protect-DoS-product-icon">}} + Defend, adapt, and mitigate against Layer 7 denial-of-service attacks on your apps and APIs. + {{}} + {{}} +{{}} \ No newline at end of file diff --git a/layouts/partials/list-main.html b/layouts/partials/list-main.html deleted file mode 100644 index be455babf..000000000 --- a/layouts/partials/list-main.html +++ /dev/null @@ -1,119 +0,0 @@ -{{/* TODO: Delete this page, and use the one from nginx-hugo-them */}} -
- {{ $PageTitle := .Title }} -
- -
- - {{ if or (lt .WordCount 1) (eq $PageTitle "F5 NGINX One Console") (eq $PageTitle "F5 NGINX App Protect DoS") (eq $PageTitle "F5 NGINX Plus") }} -
-
-
- {{ range .Pages.GroupBy "Section" }} - {{ range .Pages.ByWeight }} -
-
-

- - {{ .Title }} -

- {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Manage your NGINX fleet")}} -

Simplify, scale, secure, and collaborate with your NGINX fleet

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Get started")}} -

See benefits from the NGINX One Console

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Draft new configurations")}} -

Work with Staged Configurations

- {{ end }} - - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Manage your NGINX instances")}} -

Monitor and maintain your deployments

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Organize users with RBAC")}} -

Assign responsibilities with role-based access control

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Automate with the NGINX One API")}} -

Manage your NGINX fleet over REST

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Glossary")}} -

Learn terms unique to NGINX One Console

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Connect your instances") }} -

Work with data plane keys, containers, and proxy servers

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Set up metrics") }} -

Review your deployments in a dashboard

- {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "API")}} -

These are API docs

- - {{ end }} - {{ if and (eq $PageTitle "F5 NGINX One Console") (eq .Title "Changelog") }} - {{ partial "changelog-date.html" . }} - {{ end }} -
-
- {{ end }} - {{ end }} -
- {{ if eq $PageTitle "F5 NGINX One Console" }} -

Other Products

- {{ $nginxProducts := slice - (dict "title" "NGINX Instance Manager" "url" "/nginx-instance-manager" "imgSrc" "NGINX-Instance-Manager-product-icon" "type" "local-console-option" "description" "Track and control NGINX Open Source and NGINX Plus instances.") - (dict "title" "NGINX Ingress Controller" "url" "/nginx-ingress-controller" "imgSrc" "NGINX-Ingress-Controller-product-icon" "type" "kubernetes-solutions" "description" "Kubernetes traffic management with API gateway, identity, and observability features.") - (dict "title" "NGINX Gateway Fabric" "url" "/nginx-gateway-fabric" "imgSrc" "NGINX-product-icon" "type" "kubernetes-solutions" "description" "Next generation Kubernetes connectivity using the Gateway API.") - (dict "title" "NGINX App Protect WAF" "url" "/nginx-app-protect-waf" "imgSrc" "NGINX-App-Protect-WAF-product-icon" "type" "security" "description" "Lightweight, high-performance, advanced protection against Layer 7 attacks on your apps and APIs.") - (dict "title" "NGINX App Protect DoS" "url" "/nginx-app-protect-dos" "imgSrc" "NGINX-App-Protect-DoS-product-icon" "type" "security" "description" "Defend, adapt, and mitigate against Layer 7 denial-of-service attacks on your apps and APIs.") - (dict "title" "NGINX Plus" "url" "/nginx" "imgSrc" "NGINX-Plus-product-icon-RGB" "type" "modern-app-delivery" "description" "The all-in-one load balancer, reverse proxy, web server, content cache, and API gateway.") - (dict "title" "NGINX Open Source" "url" "https://nginx.org/en/docs/" "imgSrc" "NGINX-product-icon" "type" "modern-app-delivery" "description" "The open source all-in-one load balancer, content cache, and web server") - }} - {{ $groupedProducts := dict - "local-console-option" (where $nginxProducts "type" "local-console-option") - "kubernetes-solutions" (where $nginxProducts "type" "kubernetes-solutions") - "security" (where $nginxProducts "type" "security") - "modern-app-delivery" (where $nginxProducts "type" "modern-app-delivery") - }} - {{ range $type, $products := $groupedProducts }} -
-

{{ $type | humanize | title }}

- {{ range $products }} -
-
-

- - {{ .title }} -

-

- {{ if .description }}{{ .description | markdownify }}{{ end }} -

-
-
- {{ end }} -
- {{ end }} - {{ end }} -
-
- {{end}} -
From 2f385fa478951a6658c98613737640340aea0352 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 9 Jun 2025 09:13:57 +0100 Subject: [PATCH 003/117] build(deps): bump github/codeql-action from 3.28.16 to 3.28.19 (#653) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.28.16 to 3.28.19. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/28deaeda66b76a05916b6923827895f2b14ab387...fca7ace96b7d713c7035871441bd52efbe39e27e) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 3.28.19 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ossf_scorecard.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ossf_scorecard.yml b/.github/workflows/ossf_scorecard.yml index 86cd9f9bc..a3837c42f 100644 --- a/.github/workflows/ossf_scorecard.yml +++ b/.github/workflows/ossf_scorecard.yml @@ -56,6 +56,6 @@ jobs: # Upload the results to GitHub's code scanning dashboard. - name: Upload SARIF results to code scanning - uses: github/codeql-action/upload-sarif@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16 + uses: github/codeql-action/upload-sarif@fca7ace96b7d713c7035871441bd52efbe39e27e # v3.28.19 with: sarif_file: results.sarif From dcf33483345bd629b6b2185e05593781cc640712 Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Mon, 9 Jun 2025 03:32:47 -0700 Subject: [PATCH 004/117] Resolved dot org link in N1 landing page (#654) --- content/nginx-one/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/nginx-one/_index.md b/content/nginx-one/_index.md index 27bf8dbfe..12ff811af 100644 --- a/content/nginx-one/_index.md +++ b/content/nginx-one/_index.md @@ -58,7 +58,7 @@ cascade: {{< card title="NGINX Plus" titleUrl="/nginx" icon="NGINX-Plus-product-icon-RGB">}} The all-in-one load balancer, reverse proxy, web server, content cache, and API gateway. {{}} - {{< card title="NGINX Open Source" titleUrl="nginx.org" icon="NGINX-product-icon">}} + {{< card title="NGINX Open Source" titleUrl="https://nginx.org" icon="NGINX-product-icon">}} The open source all-in-one load balancer, content cache, and web server {{}} {{}} From 90cf13c4df3a7a529b7fc225cebcb16694baa025 Mon Sep 17 00:00:00 2001 From: Jack Hickey <133868041+nginx-jack@users.noreply.github.com> Date: Mon, 9 Jun 2025 12:00:45 +0100 Subject: [PATCH 005/117] Bump go.mod to v0.42.37 (#656) --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index fd5114fe6..eab95ad43 100644 --- a/go.mod +++ b/go.mod @@ -2,4 +2,4 @@ module github.com/nginxinc/docs go 1.19 -require github.com/nginxinc/nginx-hugo-theme v0.42.36 // indirect +require github.com/nginxinc/nginx-hugo-theme v0.42.37 // indirect diff --git a/go.sum b/go.sum index d4f5dc928..2f4bc5017 100644 --- a/go.sum +++ b/go.sum @@ -1,2 +1,2 @@ -github.com/nginxinc/nginx-hugo-theme v0.42.36 h1:vFBavxB+tw2fs0rLTpA3kYPMdBK15LtZkfkX21kzrDo= -github.com/nginxinc/nginx-hugo-theme v0.42.36/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M= +github.com/nginxinc/nginx-hugo-theme v0.42.37 h1:KtESjG1VG6ioSQBWfwgIiDVEmq9mylpunV2NENM8XvI= +github.com/nginxinc/nginx-hugo-theme v0.42.37/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M= From a615caece8feeaf8ce41c6ee0507178162fb849b Mon Sep 17 00:00:00 2001 From: "Jacob (Jake) Newfield" <104527109+jnewfield@users.noreply.github.com> Date: Tue, 10 Jun 2025 03:06:50 -0700 Subject: [PATCH 006/117] Add http_auth_jwt module to cryptographic boundary definition within 'NGINX Plus FIPS Compliance' document (#658) Changes have been made --- content/nginx/fips-compliance-nginx-plus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/nginx/fips-compliance-nginx-plus.md b/content/nginx/fips-compliance-nginx-plus.md index ecd72095d..7be51a7be 100644 --- a/content/nginx/fips-compliance-nginx-plus.md +++ b/content/nginx/fips-compliance-nginx-plus.md @@ -31,7 +31,7 @@ This statement uses the following terms: - **Cryptographic module**: The OpenSSL software, comprised of libraries of FIPS‑validated algorithms that can be used by other applications. -- **Cryptographic boundary**: The operational functions that use FIPS‑validated algorithms. For NGINX Plus, the cryptographic boundary includes all functionality that is implemented by the [http_ssl](http://nginx.org/en/docs/http/ngx_http_ssl_module.html), [http_v2](http://nginx.org/en/docs/http/ngx_http_v2_module.html), [stream_ssl](http://nginx.org/en/docs/stream/ngx_stream_ssl_module.html), and [mail_ssl](http://nginx.org/en/docs/mail/ngx_mail_ssl_module.html) modules. These modules implement SSL and TLS operations for inbound and outbound connections which use HTTP, HTTP/2, TCP, and mail protocols. +- **Cryptographic boundary**: The operational functions that use FIPS‑validated algorithms. For NGINX Plus, the cryptographic boundary includes all functionality that is implemented by the [http_ssl](http://nginx.org/en/docs/http/ngx_http_ssl_module.html), [http_v2](http://nginx.org/en/docs/http/ngx_http_v2_module.html), [stream_ssl](http://nginx.org/en/docs/stream/ngx_stream_ssl_module.html), [mail_ssl](http://nginx.org/en/docs/mail/ngx_mail_ssl_module.html), and [http_auth_jwt](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html) modules. These modules implement SSL and TLS operations for inbound and outbound connections which use HTTP, HTTP/2, TCP, and mail protocols. - **NGINX Plus**: The NGINX Plus software application developed by NGINX, Inc. and delivered in binary format from NGINX servers. From 77420464da7dc2d1083e5abdfdf74818bba5c70a Mon Sep 17 00:00:00 2001 From: nginx-seanmoloney <133861979+nginx-seanmoloney@users.noreply.github.com> Date: Tue, 10 Jun 2025 15:24:17 +0100 Subject: [PATCH 007/117] Agent info banner (#661) * Add an info banner, apply banner to all sections. Add a compatibility matrix to tech specs * add v3 banner * PR feedback Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> * PR feedback Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> * Update _banners/agent-v3-release.md * Update Agent version column to contain versions. * Minor change to remove v from versions * Update _banners/agent-v3-release.md --------- Co-authored-by: Jon Cahill-Torre Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> Co-authored-by: Mike Jang <3287976+mjang@users.noreply.github.com> --- _banners/agent-v3-release.md | 9 +++++++++ content/agent/_index.md | 8 ++++++-- content/agent/technical-specifications.md | 11 +++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) create mode 100644 _banners/agent-v3-release.md diff --git a/_banners/agent-v3-release.md b/_banners/agent-v3-release.md new file mode 100644 index 000000000..c82d87a10 --- /dev/null +++ b/_banners/agent-v3-release.md @@ -0,0 +1,9 @@ +{{< banner "notice" "NGINX Agent 3.0 is now available" >}} + + +F5 NGINX One Console does not currently support Agent 3.x. If you are using NGINX One Console in your environment, upgrade to the latest Agent 2.x version by following the [Upgrade NGINX Agent](/nginx-agent/installation-upgrade/upgrade/) guide. + +Please see the [Technical specifications](/nginx-agent/technical-specifications/) for product compatibility. + + +{{< /banner >}} \ No newline at end of file diff --git a/content/agent/_index.md b/content/agent/_index.md index 330490eb8..fcb621c38 100644 --- a/content/agent/_index.md +++ b/content/agent/_index.md @@ -1,8 +1,12 @@ --- title: NGINX Agent -description: NGINX Agent is a companion daemon for your NGINX Open Source or NGINX - Plus instance. url: /nginx-agent/ cascade: logo: NGINX-product-icon.png + banner: + enabled: true + type: deprecation + start-date: 2025-05-29 + end-date: 2025-09-09 + md: /_banners/agent-v3-release.md --- diff --git a/content/agent/technical-specifications.md b/content/agent/technical-specifications.md index 7ecd18398..149b08df9 100644 --- a/content/agent/technical-specifications.md +++ b/content/agent/technical-specifications.md @@ -12,6 +12,17 @@ type: This document provides technical specifications for NGINX Agent. It includes information on supported distributions, deployment environments, NGINX versions, sizing recommendations, and logging. +## NGINX Agent v3.0 Compatibility +{{< bootstrap-table "table table-striped table-bordered" >}} +| NGINX Product | Agent Version | +|------------------------------|----------------| +| **NGINX One Console** | 2.x | +| **NGINX Gateway Fabric** | 3.x | +| **NGINX Plus** | 2.x, 3.x | +| **NGINX Ingress Controller** | 2.x | +| **NGINX Instance Manager** | 2.x | +{{< /bootstrap-table >}} + ## Supported Distributions NGINX Agent can run in most environments. We support the following distributions: From de52c056afac71025d8ab4b867041c066cf42fba Mon Sep 17 00:00:00 2001 From: Arpith Varghese <127259427+arpith-f5@users.noreply.github.com> Date: Tue, 10 Jun 2025 08:06:41 -0700 Subject: [PATCH 008/117] NGINXaaS Platform metric migration (#647) --- .../monitoring/enable-monitoring.md | 4 +-- .../monitoring/migrate-to-platform-metrics.md | 34 +++++++++++++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) create mode 100644 content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md diff --git a/content/nginxaas-azure/monitoring/enable-monitoring.md b/content/nginxaas-azure/monitoring/enable-monitoring.md index c4bc181d2..7185e3320 100644 --- a/content/nginxaas-azure/monitoring/enable-monitoring.md +++ b/content/nginxaas-azure/monitoring/enable-monitoring.md @@ -12,7 +12,7 @@ Monitoring your application's performance is crucial for maintaining its reliabi Refer to the [Azure monitor overview](https://docs.microsoft.com/en-us/azure/azure-monitor/overview) documentation from Microsoft to learn more about Azure Monitor. -### Prerequisites +## Prerequisites - A system assigned managed identity with `Monitoring Metrics Publisher` role. @@ -25,7 +25,7 @@ Azure Monitor will collects metrics from the NGINXaaS deployment automatically i ## Exporting You can export Azure Monitor metrics to other destinations like Log Analytics workspace, Azure Storage Account, Azure Event Hubs or Azure Monitor partner solutions using Diagnostic Setting. For more information, see the [Metrics diagnostic setting](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings). -To configure diagnostic settings for a service, see [Create diagnostic settings in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/create-diagnostic-settings). +To configure diagnostic settings for a service, see [Create diagnostic settings in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/create-diagnostic-settings). You can export metrics by selecting the `AllMetrics` category in diagnostic settings. {{}} Not all metrics are exportable via diagnostic settings, for a list of exportable metrics, see [NGINXaaS exportable metrics](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/nginx-nginxplus-nginxdeployments-metrics).{{}} diff --git a/content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md b/content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md new file mode 100644 index 000000000..44c14df37 --- /dev/null +++ b/content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md @@ -0,0 +1,34 @@ +--- +title: Migrate from Custom Metrics to Platform Metrics +weight: 1000 +toc: true +url: /nginxaas/azure/getting-started/migrate-to-platform-metrics/ +type: +- how-to +--- + +## Overview +NGINXaaS previously supported monitoring using [Custom Metrics](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/metrics-custom-overview). Custom metrics is a preview feature in Azure and support for it will be dropped in the future. We have added support for Platform metrics which is the Azure recommended way for monitoring resources, we strongly recommend you to migrate your deployments monitoring to use Platform metrics to take advantage of reduced latency and higher reliability. + +## Migration Steps +This section contains instructions on how to migrate your deployment monitoring from Custom Metrics to Platform Metrics. + +1. Verify that your NGINXaaS deployment meets the [pre-requisites]({{< relref "/nginxaas-azure/monitoring/enable-monitoring.md#prerequisites">}}) for Platform metrics to work. +2. If the per-requisites are met, Platform metrics are enabled by default on all NGINXaaS deployment. Verify that you are able to see the new metrics in Azure Monitor under the `Standard Metrics` namespace. +3. Turn off legacy monitoring. + + - **Using Portal** + 1. Go to the **NGINX monitoring** page of the NGINXaaS deployment in the Azure portal. + 2. Toggle Off the `Send metrics to Azure Monitor` switch. + 3. Click Save. + + - **Using Terraform** + 1. Set `diagnose_support_enabled` to false in the `azurerm_nginx_deployment` resource. + 2. Run `terraform plan` followed by `terraform apply` to upgrade the deployment. + + - **Using Azure CLI** + Run the command below: + ```bash + az nginx deployment update --name myDeployment --resource-group \ + myResourceGroup --enable-diagnostics="false" + ``` \ No newline at end of file From 0f04b44c3259067c99049b7a0c28889415e10cec Mon Sep 17 00:00:00 2001 From: Jon Torre <78599298+JTorreG@users.noreply.github.com> Date: Tue, 10 Jun 2025 16:58:23 +0100 Subject: [PATCH 009/117] style guide corrections to migrate metrics doc (#664) * style guide corrections to migrate metrics doc * Apply suggestions from code review Co-authored-by: Ryan Davis Co-authored-by: Travis Martin <33876974+travisamartin@users.noreply.github.com> * Update content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md Co-authored-by: Travis Martin <33876974+travisamartin@users.noreply.github.com> --------- Co-authored-by: Ryan Davis Co-authored-by: Travis Martin <33876974+travisamartin@users.noreply.github.com> --- .../monitoring/migrate-to-platform-metrics.md | 38 ++++++++++--------- 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md b/content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md index 44c14df37..c33b781c7 100644 --- a/content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md +++ b/content/nginxaas-azure/monitoring/migrate-to-platform-metrics.md @@ -1,5 +1,5 @@ --- -title: Migrate from Custom Metrics to Platform Metrics +title: Migrate from Custom metrics to Platform metrics weight: 1000 toc: true url: /nginxaas/azure/getting-started/migrate-to-platform-metrics/ @@ -8,27 +8,29 @@ type: --- ## Overview -NGINXaaS previously supported monitoring using [Custom Metrics](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/metrics-custom-overview). Custom metrics is a preview feature in Azure and support for it will be dropped in the future. We have added support for Platform metrics which is the Azure recommended way for monitoring resources, we strongly recommend you to migrate your deployments monitoring to use Platform metrics to take advantage of reduced latency and higher reliability. -## Migration Steps -This section contains instructions on how to migrate your deployment monitoring from Custom Metrics to Platform Metrics. +F5 NGINXaaS for Azure previously supported monitoring through [Custom Metrics](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/metrics-custom-overview), which is a preview feature in Azure. As a preview feature, support for Custom Metrics will be removed in the future. We've added support for Platform Metrics, which is the recommended way to monitor resources in Azure. We strongly recommend switching your deployment's monitoring to Platform Metrics to take advantage of lower latency and better reliability. -1. Verify that your NGINXaaS deployment meets the [pre-requisites]({{< relref "/nginxaas-azure/monitoring/enable-monitoring.md#prerequisites">}}) for Platform metrics to work. -2. If the per-requisites are met, Platform metrics are enabled by default on all NGINXaaS deployment. Verify that you are able to see the new metrics in Azure Monitor under the `Standard Metrics` namespace. -3. Turn off legacy monitoring. +## Migration steps - - **Using Portal** - 1. Go to the **NGINX monitoring** page of the NGINXaaS deployment in the Azure portal. - 2. Toggle Off the `Send metrics to Azure Monitor` switch. - 3. Click Save. +Follow the steps in this section to migrate your deployment monitoring from Custom metrics to Platform metrics. - - **Using Terraform** - 1. Set `diagnose_support_enabled` to false in the `azurerm_nginx_deployment` resource. - 2. Run `terraform plan` followed by `terraform apply` to upgrade the deployment. +1. Verify that your NGINXaaS deployment meets the [pre-requisites]({{< ref "/nginxaas-azure/monitoring/enable-monitoring.md#prerequisites">}}) for Platform metrics to work. +2. If the pre-requisites are met, Platform metrics are enabled by default on all NGINXaaS deployment. Verify that you are able to see the new metrics in Azure Monitor under the `Standard Metrics` namespace. +3. Turn off legacy monitoring: - - **Using Azure CLI** - Run the command below: - ```bash + - Using the Azure portal + 1. In the Azure portal, go to the **NGINX monitoring** page for your NGINXaaS deployment. + 2. Turn off the **Send metrics to Azure Monitor** setting. + 3. Select **Save**. + + - Using Terraform + 1. Set `diagnose_support_enabled` to false in the `azurerm_nginx_deployment` resource. + 2. Run `terraform plan` followed by `terraform apply` to upgrade the deployment. + + - Using the Azure CLI + Run the following command: + ```shell az nginx deployment update --name myDeployment --resource-group \ myResourceGroup --enable-diagnostics="false" - ``` \ No newline at end of file + ``` From d99cf086deae9024fba74eb80dd17e0c22143e28 Mon Sep 17 00:00:00 2001 From: Alex Russell <91080557+arussellf5@users.noreply.github.com> Date: Wed, 11 Jun 2025 02:50:19 -0600 Subject: [PATCH 010/117] NGINXaaS for Azure: update helm chart version used for load balancer for kubernetes (#665) This will ensure users get latest bug fixes and functionality. --- content/nginxaas-azure/loadbalancer-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/nginxaas-azure/loadbalancer-kubernetes.md b/content/nginxaas-azure/loadbalancer-kubernetes.md index 51fd5e740..bd845dac1 100644 --- a/content/nginxaas-azure/loadbalancer-kubernetes.md +++ b/content/nginxaas-azure/loadbalancer-kubernetes.md @@ -148,7 +148,7 @@ The NLK controller can be installed in your Kubernetes cluster using either Helm Install the NLK controller using `helm install`. Be sure your kubectl context is pointed at the desired cluster. ```bash -helm install nlk oci://registry-1.docker.io/nginxcharts/nginxaas-loadbalancer-kubernetes --version 1.0.0 \ +helm install nlk oci://registry-1.docker.io/nginxcharts/nginxaas-loadbalancer-kubernetes --version 1.1.1 \ --set "nlk.dataplaneApiKey=${keyValue}" \ --set "nlk.config.nginxHosts=${dataplaneAPIEndpoint}nplus" ``` From 2f2be347be8bde4f8c97e1d3feeb6cd0bc99f919 Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Wed, 11 Jun 2025 03:18:27 -0700 Subject: [PATCH 011/117] Renamed from products to components for N1 (#667) --- content/nginx-one/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/nginx-one/_index.md b/content/nginx-one/_index.md index 12ff811af..d5f57a9a6 100644 --- a/content/nginx-one/_index.md +++ b/content/nginx-one/_index.md @@ -40,7 +40,7 @@ cascade: {{}} {{}} - # Other Products + # Other Components {{< card-section title="Kubernetes Solutions">}} {{< card title="NGINX Ingress Controller" titleUrl="/nginx-ingress-controller/" icon="NGINX-Ingress-Controller-product-icon">}} Kubernetes traffic management with API gateway, identity, and observability features. From d7f351785767aa402eb5c5351c124f8c33cba9a7 Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Wed, 11 Jun 2025 09:22:39 -0700 Subject: [PATCH 012/117] Fix changelog partial (#670) fix changelog partial --- layouts/partials/changelog-date.html | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/layouts/partials/changelog-date.html b/layouts/partials/changelog-date.html index 2c1547392..efe487b51 100644 --- a/layouts/partials/changelog-date.html +++ b/layouts/partials/changelog-date.html @@ -1,11 +1,10 @@ {{ $changelogContent := readFile "content/nginx-one/changelog.md" }} {{ $maxNumLogs := 3 }} {{ $headings := first $maxNumLogs (findRE `(?m)^##\s(.+)$` $changelogContent) }} -
    - {{ range $headings }} - {{$title := replaceRE "^##\\s" "" .}} -
  • {{ $title }}
  • - {{ end }} -
  • Older...
  • + {{ range $headings }} + {{ $title := replaceRE "^##\\s" "" . }} +
  • {{ $title }}
  • + {{ end }} +
  • Older...
\ No newline at end of file From 1f9fdcc2424e725b04cdf824cb30dc240ddf7b58 Mon Sep 17 00:00:00 2001 From: yar Date: Thu, 12 Jun 2025 10:49:42 +0100 Subject: [PATCH 013/117] feat: Update the value of ssl_protocols. (#672) --- .../load-balancer/http-load-balancer.md | 2 +- .../admin-guide/mail-proxy/mail-proxy.md | 4 +- .../securing-http-traffic-upstream.md | 4 +- .../securing-tcp-traffic-upstream.md | 4 +- .../security-controls/terminating-ssl-http.md | 8 +- .../security-controls/terminating-ssl-tcp.md | 4 +- .../microsoft-exchange.md | 150 +++++++++--------- .../f5-big-ip-configuration.md | 4 +- content/nginx/fips-compliance-nginx-plus.md | 2 +- 9 files changed, 90 insertions(+), 92 deletions(-) diff --git a/content/nginx/admin-guide/load-balancer/http-load-balancer.md b/content/nginx/admin-guide/load-balancer/http-load-balancer.md index a01b69aed..cfce044ca 100644 --- a/content/nginx/admin-guide/load-balancer/http-load-balancer.md +++ b/content/nginx/admin-guide/load-balancer/http-load-balancer.md @@ -427,7 +427,7 @@ http { listen 443 ssl; ssl_certificate /etc/nginx/ssl/company.com.crt; ssl_certificate_key /etc/nginx/ssl/company.com.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; location / { proxy_pass https://exchange; diff --git a/content/nginx/admin-guide/mail-proxy/mail-proxy.md b/content/nginx/admin-guide/mail-proxy/mail-proxy.md index ab47d4813..a2f29c46d 100644 --- a/content/nginx/admin-guide/mail-proxy/mail-proxy.md +++ b/content/nginx/admin-guide/mail-proxy/mail-proxy.md @@ -166,7 +166,7 @@ To enable SSL/TLS for the mail proxy: ```nginx mail { #... - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; } ``` @@ -223,7 +223,7 @@ mail { ssl on; ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; diff --git a/content/nginx/admin-guide/security-controls/securing-http-traffic-upstream.md b/content/nginx/admin-guide/security-controls/securing-http-traffic-upstream.md index d2e353616..978bab7c5 100644 --- a/content/nginx/admin-guide/security-controls/securing-http-traffic-upstream.md +++ b/content/nginx/admin-guide/security-controls/securing-http-traffic-upstream.md @@ -77,7 +77,7 @@ Optionally, you can specify which SSL protocols and ciphers are used: ```nginx location /upstream { #... - proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; } ``` @@ -133,7 +133,7 @@ http { proxy_pass https://backend.example.com; proxy_ssl_certificate /etc/nginx/client.pem; proxy_ssl_certificate_key /etc/nginx/client.key; - proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; proxy_ssl_trusted_certificate /etc/nginx/trusted_ca_cert.crt; diff --git a/content/nginx/admin-guide/security-controls/securing-tcp-traffic-upstream.md b/content/nginx/admin-guide/security-controls/securing-tcp-traffic-upstream.md index 13ec0bc3f..d40ccac1b 100644 --- a/content/nginx/admin-guide/security-controls/securing-tcp-traffic-upstream.md +++ b/content/nginx/admin-guide/security-controls/securing-tcp-traffic-upstream.md @@ -58,7 +58,7 @@ Optionally, specify which SSL protocols and ciphers to use: ```nginx server { ... - proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; } ``` @@ -98,7 +98,7 @@ stream { proxy_ssl_certificate /etc/ssl/certs/backend.crt; proxy_ssl_certificate_key /etc/ssl/certs/backend.key; - proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; proxy_ssl_trusted_certificate /etc/ssl/certs/trusted_ca_cert.crt; diff --git a/content/nginx/admin-guide/security-controls/terminating-ssl-http.md b/content/nginx/admin-guide/security-controls/terminating-ssl-http.md index 792b7ce00..66656532d 100644 --- a/content/nginx/admin-guide/security-controls/terminating-ssl-http.md +++ b/content/nginx/admin-guide/security-controls/terminating-ssl-http.md @@ -22,7 +22,7 @@ server { server_name www.example.com; ssl_certificate www.example.com.crt; ssl_certificate_key www.example.com.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; #... } @@ -39,10 +39,10 @@ In this case it is important to restrict access to the file. Note that although The [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) and [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers ) directives can be used to require that clients use only the strong versions and ciphers of SSL/TLS when establishing connections. -Since version 1.9.1, NGINX uses these defaults: +Since version 1.23.4, NGINX uses these defaults: ```nginx -ssl_protocols TLSv1 TLSv1.1 TLSv1.2; +ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; ``` @@ -118,7 +118,7 @@ http { ssl_certificate www.example.com.crt; ssl_certificate_key www.example.com.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; #... } diff --git a/content/nginx/admin-guide/security-controls/terminating-ssl-tcp.md b/content/nginx/admin-guide/security-controls/terminating-ssl-tcp.md index bf7ab028f..af416b053 100644 --- a/content/nginx/admin-guide/security-controls/terminating-ssl-tcp.md +++ b/content/nginx/admin-guide/security-controls/terminating-ssl-tcp.md @@ -62,7 +62,7 @@ Additionally, the [ssl_protocols](https://nginx.org/en/docs/stream/ngx_stream_ss ```nginx server { #... - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; } ``` @@ -152,7 +152,7 @@ stream { ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key; - ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; ssl_session_cache shared:SSL:20m; ssl_session_timeout 4h; diff --git a/content/nginx/deployment-guides/load-balance-third-party/microsoft-exchange.md b/content/nginx/deployment-guides/load-balance-third-party/microsoft-exchange.md index bfaabc179..1c1a48769 100644 --- a/content/nginx/deployment-guides/load-balance-third-party/microsoft-exchange.md +++ b/content/nginx/deployment-guides/load-balance-third-party/microsoft-exchange.md @@ -475,7 +475,7 @@ The directives in the top‑level `stream` configuration block configure TCP loa server { listen 993; status_zone exchange-imaps; - proxy_pass exchange-imaps; + proxy_pass exchange-imaps; } ``` @@ -488,7 +488,7 @@ The directives in the top‑level `stream` configuration block configure TCP loa server { listen 25; # SMTP port can be changed here (to 587, for example) status_zone exchange-smtp; - proxy_pass exchange-smtp; + proxy_pass exchange-smtp; } ``` @@ -517,7 +517,7 @@ These directives in the top‑level `http` configuration block configure global ```nginx # In the 'http' block - keepalive_timeout 3h; + keepalive_timeout 3h; proxy_read_timeout 3h; ``` @@ -542,7 +542,7 @@ These directives define virtual servers for HTTP and HTTPS traffic in the top‑ ```nginx # In the 'http' block server { - listen 443 ssl; + listen 443 ssl; status_zone exchange-combined; } ``` @@ -564,9 +564,9 @@ These directives define virtual servers for HTTP and HTTPS traffic in the top‑ ```nginx # In the 'server' block for HTTPS traffic - ssl_certificate /etc/nginx/ssl/company.com.crt; + ssl_certificate /etc/nginx/ssl/company.com.crt; ssl_certificate_key /etc/nginx/ssl/company.com.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; ``` Directive documentation: [ssl_certificate](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate), [ssl_certificate_key](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate_key), [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) @@ -668,11 +668,11 @@ These directives complete the configuration for basic load balancing of Exchange ```nginx # In the 'server' block for HTTPS traffic location / { - proxy_pass https://exchange; - proxy_buffering off; - proxy_http_version 1.1; + proxy_pass https://exchange; + proxy_buffering off; + proxy_http_version 1.1; proxy_request_buffering off; - proxy_set_header Connection "Keep-Alive"; + proxy_set_header Connection "Keep-Alive"; } ``` @@ -730,10 +730,10 @@ http { listen 443 ssl; http2 on; client_max_body_size 2G; - ssl_certificate /etc/nginx/ssl/company.com.crt; - ssl_certificate_key /etc/nginx/ssl/company.com.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; - status_zone exchange-combined; + ssl_certificate /etc/nginx/ssl/company.com.crt; + ssl_certificate_key /etc/nginx/ssl/company.com.key; + ssl_protocols TLSv1.2 TLSv1.3; + status_zone exchange-combined; location = / { return 301 "/owa/"; @@ -745,11 +745,11 @@ http { } location / { - proxy_pass https://exchange; - proxy_buffering off; - proxy_http_version 1.1; + proxy_pass https://exchange; + proxy_buffering off; + proxy_http_version 1.1; proxy_request_buffering off; - proxy_set_header Connection "Keep-Alive"; + proxy_set_header Connection "Keep-Alive"; } } } @@ -780,13 +780,13 @@ stream { server { listen 993; status_zone exchange-imaps; - proxy_pass exchange-imaps; + proxy_pass exchange-imaps; } server { listen 25; # SMTP port can be changed here (to 587, for example) status_zone exchange-smtp; - proxy_pass exchange-smtp; + proxy_pass exchange-smtp; } } ``` @@ -820,9 +820,6 @@ Exchange CASs interact with various applications used by clients on different ty {{}} - - - ##### Configuring Granular URL Location Control @@ -892,9 +889,9 @@ Exchange CASs interact with various applications used by clients on different ty ```nginx # In the 'server' block for HTTPS traffic location / { - proxy_pass https://exchange; + proxy_pass https://exchange; proxy_http_version 1.1; - proxy_set_header Connection ""; + proxy_set_header Connection ""; } ``` @@ -907,9 +904,9 @@ Exchange CASs interact with various applications used by clients on different ty location /ecp { #allow 172.16.0.0/16; # Replace with your admin network #deny all; - proxy_pass https://exchange-ecp; + proxy_pass https://exchange-ecp; proxy_http_version 1.1; - proxy_set_header Connection ""; + proxy_set_header Connection ""; } ``` @@ -920,9 +917,9 @@ Exchange CASs interact with various applications used by clients on different ty ```nginx # In the 'server' block for HTTPS traffic location /mapi { - proxy_pass https://exchange-mapi; + proxy_pass https://exchange-mapi; proxy_http_version 1.1; - proxy_set_header Connection ""; + proxy_set_header Connection ""; } ``` @@ -931,9 +928,9 @@ Exchange CASs interact with various applications used by clients on different ty ```nginx # In the 'server' block for HTTPS traffic location /Microsoft-Server-ActiveSync { - proxy_pass https://exchange-activesync; + proxy_pass https://exchange-activesync; proxy_http_version 1.1; - proxy_set_header Connection ""; + proxy_set_header Connection ""; } ``` @@ -942,9 +939,9 @@ Exchange CASs interact with various applications used by clients on different ty ```nginx # In the 'server' block for HTTPS traffic location /owa { - proxy_pass https://exchange-owa; + proxy_pass https://exchange-owa; proxy_http_version 1.1; - proxy_set_header Connection ""; + proxy_set_header Connection ""; } ``` @@ -953,11 +950,11 @@ Exchange CASs interact with various applications used by clients on different ty ```nginx # In the 'server' block for HTTPS traffic location /rpc/rpcproxy.dll { - proxy_pass https://exchange-rpc; - proxy_buffering off; - proxy_http_version 1.1; + proxy_pass https://exchange-rpc; + proxy_buffering off; + proxy_http_version 1.1; proxy_request_buffering off; - proxy_set_header Connection "Keep-Alive"; + proxy_set_header Connection "Keep-Alive"; } ``` @@ -1018,27 +1015,27 @@ These directives configure NGINX Plus health checks. location /ecp { #allow 172.16.0.0/16; # Replace with your admin network #deny all; - proxy_pass https://exchange-ecp; + proxy_pass https://exchange-ecp; proxy_http_version 1.1; - proxy_set_header Connection ""; - health_check uri=/ecp/healthcheck.htm interval=3s - match=exchange-health; + proxy_set_header Connection ""; + health_check uri=/ecp/healthcheck.htm interval=3s + match=exchange-health; } location /mapi { - proxy_pass https://exchange-mapi; + proxy_pass https://exchange-mapi; proxy_http_version 1.1; - proxy_set_header Connection ""; - health_check uri=/mapi/healthcheck.htm interval=3s - match=exchange-health; + proxy_set_header Connection ""; + health_check uri=/mapi/healthcheck.htm interval=3s + match=exchange-health; } location /owa { - proxy_pass https://exchange-owa; + proxy_pass https://exchange-owa; proxy_http_version 1.1; - proxy_set_header Connection ""; - health_check uri=/owa/healthcheck.htm interval=3s - match=exchange-health; + proxy_set_header Connection ""; + health_check uri=/owa/healthcheck.htm interval=3s + match=exchange-health; } ``` @@ -1234,10 +1231,10 @@ http { http2 on; client_max_body_size 2G; - ssl_certificate /etc/nginx/ssl/company.com.crt; + ssl_certificate /etc/nginx/ssl/company.com.crt; ssl_certificate_key /etc/nginx/ssl/company.com.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; - status_zone exchange-combined; + ssl_protocols TLSv1.2 TLSv1.3; + status_zone exchange-combined; location = / { return 301 "/owa/"; @@ -1249,9 +1246,9 @@ http { } location / { - proxy_pass https://exchange; + proxy_pass https://exchange; proxy_http_version 1.1; - proxy_set_header Connection ""; + proxy_set_header Connection ""; } location /ecp { @@ -1263,43 +1260,43 @@ http { #allow 172.16.0.0/16; # Replace with your admin network #deny all; - proxy_pass https://exchange-ecp; + proxy_pass https://exchange-ecp; proxy_http_version 1.1; - proxy_set_header Connection ""; - health_check uri=/ecp/healthcheck.htm interval=3s - match=exchange-health; + proxy_set_header Connection ""; + health_check uri=/ecp/healthcheck.htm interval=3s + match=exchange-health; } location /mapi { - proxy_pass https://exchange-mapi; + proxy_pass https://exchange-mapi; proxy_http_version 1.1; - proxy_set_header Connection ""; - health_check uri=/mapi/healthcheck.htm interval=3s - match=exchange-health; + proxy_set_header Connection ""; + health_check uri=/mapi/healthcheck.htm interval=3s + match=exchange-health; } location /Microsoft-Server-ActiveSync { - proxy_pass https://exchange-active-sync; + proxy_pass https://exchange-active-sync; proxy_http_version 1.1; - proxy_set_header Connection ""; + proxy_set_header Connection ""; } location /owa { - proxy_pass https://exchange-owa; + proxy_pass https://exchange-owa; proxy_http_version 1.1; - proxy_set_header Connection ""; - health_check uri=/owa/healthcheck.htm interval=3s - match=exchange-health; + proxy_set_header Connection ""; + health_check uri=/owa/healthcheck.htm interval=3s + match=exchange-health; } location /rpc/rpcproxy.dll { - proxy_pass https://exchange-rpc; - proxy_buffering off; - proxy_http_version 1.1; - proxy_request_buffering off; + proxy_pass https://exchange-rpc; + proxy_buffering off; + proxy_http_version 1.1; + proxy_request_buffering off; proxy_set_header Connection "Keep-Alive"; - health_check uri=/rpc/rpcproxy.dll interval=3s - match=exchange-auth; + health_check uri=/rpc/rpcproxy.dll interval=3s + match=exchange-auth; } } } @@ -1330,13 +1327,13 @@ stream { server { listen 993; status_zone exchange-imaps; - proxy_pass exchange-imaps; + proxy_pass exchange-imaps; } server { listen 25; # SMTP port can be changed here (to 587, for example) status_zone exchange-smtp; - proxy_pass exchange-smtp; + proxy_pass exchange-smtp; } } ``` @@ -1344,6 +1341,7 @@ stream { ### Revision History +- Version 7 (June 2025) – Update for the `ssl_protocols` directive - Version 6 (May 2024) – Update about HTTP/2 support (the [http2](https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2) directive) - Version 5 (April 2018) – Update for NGINX Plus API (NGINX Plus R14) - Version 4 (May 2017) – Update about HTTP/2 support (NGINX Plus Release 11 and later) diff --git a/content/nginx/deployment-guides/migrate-hardware-adc/f5-big-ip-configuration.md b/content/nginx/deployment-guides/migrate-hardware-adc/f5-big-ip-configuration.md index 1cc9c9455..f5e2a4dec 100644 --- a/content/nginx/deployment-guides/migrate-hardware-adc/f5-big-ip-configuration.md +++ b/content/nginx/deployment-guides/migrate-hardware-adc/f5-big-ip-configuration.md @@ -1,4 +1,4 @@ ---- + --- description: Migrate load-balancing configuration from F5 BIG-IP LTM to NGINX Plus, using our syntax conversion examples. docs: DOCS-460 @@ -240,7 +240,7 @@ There are two methods for handling SSL/TLS traffic on a load balancer instance, proxy_pass https://ssl_test_pool; proxy_ssl_certificate /etc/nginx/ssl/client.pem; proxy_ssl_certificate_key /etc/nginx/ssl/client.key; - proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; proxy_ssl_trusted_certificate /etc/nginx/ssl/trusted_ca_cert.crt; proxy_ssl_verify on; diff --git a/content/nginx/fips-compliance-nginx-plus.md b/content/nginx/fips-compliance-nginx-plus.md index 7be51a7be..52a378a44 100644 --- a/content/nginx/fips-compliance-nginx-plus.md +++ b/content/nginx/fips-compliance-nginx-plus.md @@ -110,7 +110,7 @@ server { ssl_certificate /etc/nginx/ssl/test.crt; ssl_certificate_key /etc/nginx/ssl/test.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_protocols TLSv1.2 TLSv1.3; location / { root /usr/share/nginx/html; From d91f4e05259e2860fea509eac0145b9af618ec0f Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Thu, 12 Jun 2025 19:25:13 +0530 Subject: [PATCH 014/117] NGF: Update release version to v2.0.1 (#671) Update doc version for NGF release 2.0.1 --- layouts/shortcodes/version-ngf.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/shortcodes/version-ngf.html b/layouts/shortcodes/version-ngf.html index 359a5b952..10bf840ed 100644 --- a/layouts/shortcodes/version-ngf.html +++ b/layouts/shortcodes/version-ngf.html @@ -1 +1 @@ -2.0.0 \ No newline at end of file +2.0.1 \ No newline at end of file From c4e6d911e957489e7685f931cd361b710ef6a9fd Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Thu, 12 Jun 2025 09:34:43 -0600 Subject: [PATCH 015/117] NGF: replace static-mode with controller (#669) Problem: The static-mode command has been renamed. Solution: Update to `controller`. --- .../includes/ngf/installation/expose-nginx-gateway-fabric.md | 2 +- content/ngf/overview/gateway-api-compatibility.md | 4 ++-- content/ngf/traffic-management/basic-routing.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/includes/ngf/installation/expose-nginx-gateway-fabric.md b/content/includes/ngf/installation/expose-nginx-gateway-fabric.md index 85d2f4f4c..9ae213ed8 100644 --- a/content/includes/ngf/installation/expose-nginx-gateway-fabric.md +++ b/content/includes/ngf/installation/expose-nginx-gateway-fabric.md @@ -50,4 +50,4 @@ There are two options for accessing NGINX Gateway Fabric depending on the type o NGINX Gateway Fabric uses the created service to update the **Addresses** field in the **Gateway Status** resource. Using a **LoadBalancer** service sets this field to the IP address and/or hostname of that service. Without a service, the pod IP address is used. -This gateway is associated with the NGINX Gateway Fabric through the **gatewayClassName** field. The default installation of NGINX Gateway Fabric creates a **GatewayClass** with the name **nginx**. NGINX Gateway Fabric will only configure gateways with a **gatewayClassName** of **nginx** unless you change the name via the `--gatewayclass` [command-line flag]({{< ref "/ngf/reference/cli-help.md#static-mode">}}). +This gateway is associated with the NGINX Gateway Fabric through the **gatewayClassName** field. The default installation of NGINX Gateway Fabric creates a **GatewayClass** with the name **nginx**. NGINX Gateway Fabric will only configure gateways with a **gatewayClassName** of **nginx** unless you change the name via the `--gatewayclass` [command-line flag]({{< ref "/ngf/reference/cli-help.md#controller">}}). diff --git a/content/ngf/overview/gateway-api-compatibility.md b/content/ngf/overview/gateway-api-compatibility.md index fc04731cc..207d9f94c 100644 --- a/content/ngf/overview/gateway-api-compatibility.md +++ b/content/ngf/overview/gateway-api-compatibility.md @@ -57,7 +57,7 @@ For a description of each field, visit the [Gateway API documentation](https://g {{< /bootstrap-table >}} -NGINX Gateway Fabric supports a single GatewayClass resource configured with the `--gatewayclass` flag of the [static-mode]({{< ref "/ngf/reference/cli-help.md#static-mode">}}) command. +NGINX Gateway Fabric supports a single GatewayClass resource configured with the `--gatewayclass` flag of the [controller]({{< ref "/ngf/reference/cli-help.md#controller>}}) command. **Fields**: @@ -87,7 +87,7 @@ NGINX Gateway Fabric supports a single GatewayClass resource configured with the NGINX Gateway Fabric supports multiple Gateway resources. The Gateway resources must reference NGINX Gateway Fabric's corresponding GatewayClass. -See the [static-mode]({{< ref "/ngf/reference/cli-help.md#static-mode">}}) command for more information. +See the [controller]({{< ref "/ngf/reference/cli-help.md#controller">}}) command for more information. **Fields**: diff --git a/content/ngf/traffic-management/basic-routing.md b/content/ngf/traffic-management/basic-routing.md index 41a81cc81..78b02c338 100644 --- a/content/ngf/traffic-management/basic-routing.md +++ b/content/ngf/traffic-management/basic-routing.md @@ -148,7 +148,7 @@ In a production environment, you should have a DNS record for the external IP ad {{< /note >}} -This Gateway is associated with NGINX Gateway Fabric through the **gatewayClassName** field. The default installation of NGINX Gateway Fabric creates a GatewayClass with the name **nginx**. NGINX Gateway Fabric will only configure Gateways with a **gatewayClassName** of **nginx** unless you change the name via the `--gatewayclass` [command-line flag]({{< ref "/ngf/reference/cli-help.md#static-mode" >}}). +This Gateway is associated with NGINX Gateway Fabric through the **gatewayClassName** field. The default installation of NGINX Gateway Fabric creates a GatewayClass with the name **nginx**. NGINX Gateway Fabric will only configure Gateways with a **gatewayClassName** of **nginx** unless you change the name via the `--gatewayclass` [command-line flag]({{< ref "/ngf/reference/cli-help.md#controller" >}}). We specify a [listener](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.Listener) on the Gateway to open an entry point on the cluster. In this case, since the coffee application accepts HTTP requests, we create an HTTP listener, named **http**, that listens on port 80. From 40af4bd29cdc39f54634f43858be07351dfc32ea Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Fri, 13 Jun 2025 01:58:34 -0600 Subject: [PATCH 016/117] Fix missing quote (#676) A quotation was missing that was not caught by the original PR pipeline. --- content/ngf/overview/gateway-api-compatibility.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ngf/overview/gateway-api-compatibility.md b/content/ngf/overview/gateway-api-compatibility.md index 207d9f94c..5638ed596 100644 --- a/content/ngf/overview/gateway-api-compatibility.md +++ b/content/ngf/overview/gateway-api-compatibility.md @@ -57,7 +57,7 @@ For a description of each field, visit the [Gateway API documentation](https://g {{< /bootstrap-table >}} -NGINX Gateway Fabric supports a single GatewayClass resource configured with the `--gatewayclass` flag of the [controller]({{< ref "/ngf/reference/cli-help.md#controller>}}) command. +NGINX Gateway Fabric supports a single GatewayClass resource configured with the `--gatewayclass` flag of the [controller]({{< ref "/ngf/reference/cli-help.md#controller">}}) command. **Fields**: From 85237e07f9835ff510e2fe2ab573a7e4d4bce958 Mon Sep 17 00:00:00 2001 From: bjee19 <139261241+bjee19@users.noreply.github.com> Date: Fri, 13 Jun 2025 06:31:04 -0700 Subject: [PATCH 017/117] feat: Fix NGINX Plus subsection link (#677) Fix small link to subsection --- .../ngf/installation/nginx-plus/docker-registry-secret.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/includes/ngf/installation/nginx-plus/docker-registry-secret.md b/content/includes/ngf/installation/nginx-plus/docker-registry-secret.md index c6d666b6d..0bfceec95 100644 --- a/content/includes/ngf/installation/nginx-plus/docker-registry-secret.md +++ b/content/includes/ngf/installation/nginx-plus/docker-registry-secret.md @@ -2,7 +2,7 @@ docs: "DOCS-000" --- -{{< note >}} If you would rather pull the NGINX Plus image and push to a private registry, you can skip this specific step and instead follow [this step]({{< ref "/ngf/install/nginx-plus.md#pulling-an-image-for-local-use" >}}). {{< /note >}} +{{< note >}} If you would rather pull the NGINX Plus image and push to a private registry, you can skip this specific step and instead follow [this step]({{< ref "/ngf/install/nginx-plus.md#pull-an-image-for-local-use" >}}). {{< /note >}} If the `nginx-gateway` namespace does not yet exist, create it: From cbf35df3b705d13c0e509da19c9238723daf106d Mon Sep 17 00:00:00 2001 From: yar Date: Fri, 13 Jun 2025 15:42:13 +0100 Subject: [PATCH 018/117] Actualizing SSL Termination for NGINX Plus (#666) * Removed HTML. * Directive formatting added. * Updated values in ssl_protocols. * Updated Compatibility Notes. * Updated info about browser SNI support. --- .../security-controls/terminating-ssl-http.md | 81 ++++++++++--------- 1 file changed, 43 insertions(+), 38 deletions(-) diff --git a/content/nginx/admin-guide/security-controls/terminating-ssl-http.md b/content/nginx/admin-guide/security-controls/terminating-ssl-http.md index 66656532d..455e36ea6 100644 --- a/content/nginx/admin-guide/security-controls/terminating-ssl-http.md +++ b/content/nginx/admin-guide/security-controls/terminating-ssl-http.md @@ -11,10 +11,9 @@ type: This section describes how to configure an HTTPS server on NGINX and F5 NGINX Plus. - -## Setting up an HTTPS Server +## Setting up an HTTPS Server {#setup} -To set up an HTTPS server, in your **nginx.conf** file include the `ssl` parameter to the [listen](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive in the [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) block, then specify the locations of the server certificate and private key files: +To set up an HTTPS server, in your **nginx.conf** file include the `ssl` parameter to the [`listen`](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive in the [`server`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) block, then specify the locations of the server certificate and private key files: ```nginx server { @@ -28,7 +27,7 @@ server { } ``` -The server certificate is a public entity. It is sent to every client that connects to the NGINX or NGINX Plus server. The private key is a secure entity and should be stored in a file with restricted access. However, the NGINX master process must be able to read this file. Alternatively, the private key can be stored in the same file as the certificate: +The server certificate is a public entity. It is sent to every client that connects to the NGINX or NGINX Plus server. The private key is a secure entity and should be stored in a file with restricted access. However, the NGINX master process must be able to read this file. Alternatively, the private key can be stored in the same file as the certificate: ```nginx ssl_certificate www.example.com.cert; @@ -37,20 +36,19 @@ ssl_certificate_key www.example.com.cert; In this case it is important to restrict access to the file. Note that although the certificate and the key are stored in one file in this case, only the certificate is sent to clients. -The [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) and [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers ) directives can be used to require that clients use only the strong versions and ciphers of SSL/TLS when establishing connections. +The [`ssl_protocols`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) and [`ssl_ciphers`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers ) directives can be used to require that clients use only the strong versions and ciphers of SSL/TLS when establishing connections. Since version 1.23.4, NGINX uses these defaults: ```nginx ssl_protocols TLSv1.2 TLSv1.3; -ssl_ciphers HIGH:!aNULL:!MD5; +ssl_ciphers HIGH:!aNULL:!MD5; ``` Vulnerabilities are sometimes found in the design of older ciphers, and we recommend disabling them in a modern NGINX configuration (unfortunately, the default configuration cannot easily be changed because of backward compatibility for existing NGINX deployments). Please note that CBC-mode ciphers might be vulnerable to a number of attacks (the BEAST attack in particular as described in [CVE-2011-3389](https://nvd.nist.gov/vuln/detail/CVE-2011-3389)), and we recommend not using SSLv3 due to the [POODLE](https://nvd.nist.gov/vuln/detail/CVE-2014-3566) attack, unless you need to support legacy clients. - -### OCSP Validation of Client Certificates +### OCSP Validation of Client Certificates {#setup_ocsp} NGINX can be configured to use Online Certificate Status Protocol (OCSP) to check the validity of X.509 client certificates as they are presented. An OCSP request for the client certificate status is sent to an OCSP responder which checks the certificate validity and returns the response with the certificate status: @@ -58,7 +56,7 @@ NGINX can be configured to use Online Certificate Status Protocol (OCSP) to chec - `Revoked` - the certificate is revoked - `Unknown` - no information is available about the client certificate -To enable OCSP validation of SSL client certificates, specify the [ssl_ocsp](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ocsp) directive along with the [ssl_verify_client](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client) directive, which enables certificate verification: +To enable OCSP validation of SSL client certificates, specify the [`ssl_ocsp`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ocsp) directive along with the [`ssl_verify_client`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client) directive, which enables certificate verification: ```nginx server { @@ -75,7 +73,7 @@ server { } ``` -NGINX sends the OCSP request to the OCSP URI embedded in the client certificate unless a different URI is defined with the [ssl_ocsp_responder](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ocsp_responder) directive. `Only http://` OCSP responders are supported: +NGINX sends the OCSP request to the OCSP URI embedded in the client certificate unless a different URI is defined with the [`ssl_ocsp_responder`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ocsp_responder) directive. Only `http://` OCSP responders are supported: ```nginx #... @@ -83,7 +81,7 @@ ssl_ocsp_responder http://ocsp.example.com/; #... ``` -To cache OCSP responses in a single memory zone shared by all worker processes, specify the [ssl_ocsp_cache](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ocsp_cache) directive to define the name and size of the zone. Responses are cached for `1` hour unless the `nextUpdate`value in the OCSP response specifies a different value: +To cache OCSP responses in a single memory zone shared by all worker processes, specify the [`ssl_ocsp_cache`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ocsp_cache) directive to define the name and size of the zone. Responses are cached for `1` hour unless the `nextUpdate`value in the OCSP response specifies a different value: ```nginx #... @@ -94,15 +92,14 @@ ssl_ocsp_cache shared:one:10m; The result of the client certificate validation is available in the [`$ssl_client_verify`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_verify) variable, including the reason for OCSP failure. - -## HTTPS Server Optimization +## HTTPS Server Optimization {#optimize} SSL operations consume extra CPU resources. The most CPU-intensive operation is the SSL handshake. There are two ways to minimize the number of these operations per client: - Enabling keepalive connections to send several requests via one connection - Reusing SSL session parameters to avoid SSL handshakes for parallel and subsequent connections -Sessions are stored in the SSL session cache shared between worker processes and configured by the [ssl_session_cache](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_cache) directive. One megabyte of cache contains about 4000 sessions. The default cache timeout is 5 minutes. This timeout can be increased using the [ssl_session_timeout](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_timeout) directive. Below is a sample configuration optimized for a multi-core system with 10 megabyte shared session cache: +Sessions are stored in the SSL session cache shared between worker processes and configured by the [`ssl_session_cache`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_cache) directive. One megabyte of cache contains about 4000 sessions. The default cache timeout is `5` minutes. This timeout can be increased using the [`ssl_session_timeout`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_timeout) directive. Below is a sample configuration optimized for a multi-core system with 10 megabyte shared session cache: ```nginx worker_processes auto; @@ -125,8 +122,7 @@ http { } ``` - -## SSL Certificate Chains +## SSL Certificate Chains {#cert_chains} Some browsers may complain about a certificate signed by a well-known certificate authority, while other browsers may accept the certificate without issues. This occurs because the issuing authority has signed the server certificate using an intermediate certificate that is not present in the base of well-known trusted certificate authorities which is distributed in a particular browser. In this case the authority provides a bundle of chained certificates that should be concatenated to the signed server certificate. The server certificate must appear before the chained certificates in the combined file: @@ -134,7 +130,7 @@ Some browsers may complain about a certificate signed by a well-known certificat cat www.example.com.crt bundle.crt > www.example.com.chained.crt ``` -The resulting file should be used in the [ssl_certificate](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate) directive: +The resulting file should be used in the [`ssl_certificate`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate) directive: ```nginx server { @@ -184,12 +180,11 @@ Certificate chain ... ``` -In this example the subject (“`s`”) of the `www.GoDaddy.com` server certificate `#0` is signed by an issuer (`“i”`) which itself is the subject of certificate #1. Certificate #1 is signed by an issuer which itself is the subject of certificate #2. This certificate, however, is signed by the well‑known issuer `ValiCert, Inc.` whose certificate is stored in the browsers themselves. +In this example the subject (“`s`”) of the `www.GoDaddy.com` server certificate `#0` is signed by an issuer (`“i”`) which itself is the subject of certificate #1. Certificate #1 is signed by an issuer which itself is the subject of certificate #2. This certificate, however, is signed by the well‑known issuer `ValiCert, Inc.` whose certificate is stored in the browsers themselves. -If a certificate bundle has not been added, only the server certificate (#0) is shown. +If a certificate bundle has not been added, only the server certificate (#0) is shown. - -## A Single HTTP/HTTPS Server +## A Single HTTP/HTTPS Server {#single} It is possible to configure a single server that handles both HTTP and HTTPS requests by placing one `listen` directive with the `ssl` parameter and one without in the same virtual server: @@ -204,10 +199,10 @@ server { } ``` -In NGINX version 0.7.13 and earlier, SSL cannot be enabled selectively for individual listening sockets, as shown above. SSL can only be enabled for the entire server using the [ssl](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl) directive, making it impossible to set up a single HTTP/HTTPS server. The `ssl` parameter to the [listen](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive was added to solve this issue. The [ssl](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl) directive therefore is deprecated in version 0.7.14 and later. +In NGINX version 0.7.13 and earlier, SSL cannot be enabled selectively for individual listening sockets, as shown above. SSL can only be enabled for the entire server using the [ssl](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl) directive, making it impossible to set up a single HTTP/HTTPS server. The `ssl` parameter to the [listen](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive was added to solve this issue. The [ssl](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl) directive therefore is deprecated in version 0.7.14 and later. - -## Name-Based HTTPS Servers + +## Name-Based HTTPS Servers {#name} A common issue arises when two or more HTTPS servers are configured to listen on a single IP address: @@ -247,7 +242,7 @@ server { } ``` -Note that there are also some specific proxy settings for HTTPS upstreams ([proxy_ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_ciphers), [proxy_ssl_protocols](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_protocols), and [proxy_ssl_session_reuse](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_session_reuse)) which can be used for fine‑tuning SSL between NGINX and upstream servers. You can read more about these in the [HTTP proxy module documentation](https://nginx.org/en/docs/http/ngx_http_proxy_module.html). +Note that there are also some specific proxy settings for HTTPS upstreams ([`proxy_ssl_ciphers`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_ciphers), [`proxy_ssl_protocols`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_protocols), and [`proxy_ssl_session_reuse`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_session_reuse)) which can be used for fine‑tuning SSL between NGINX and upstream servers. You can read more about these in the [HTTP proxy module documentation](https://nginx.org/en/docs/http/ngx_http_proxy_module.html). ### An SSL Certificate With Several Names @@ -276,17 +271,17 @@ server { ### Server Name Indication -A more generic solution for running several HTTPS servers on a single IP address is the [TLS Server Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication) (SNI) extension ([RFC 6066](https://tools.ietf.org/html/rfc6066)), which allows a browser to pass a requested server name during the SSL handshake. With this solution, the server will know which certificate it should use for the connection. However, SNI has limited browser support. Currently it is supported starting with the following browser versions: +A more generic solution for running several HTTPS servers on a single IP address is the [TLS Server Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication) (SNI) extension ([RFC 6066](https://tools.ietf.org/html/rfc6066)), which allows a browser to pass a requested server name during the SSL handshake. With this solution, the server will know which certificate it should use for the connection. Although SNI is widely supported by modern browsers, some old browsers may still not include support for it. Major browsers support SNI starting with the following versions: -- Opera 8.0 -- MSIE 7.0 (but only on Windows Vista or higher) +- Chrome 105 (Windows version supports SNI on Vista or higher, too) - Firefox 2.0 and other browsers using Mozilla Platform rv:1.8.1 +- Microsoft Internet Explorer 7.0 (but only on Windows Vista or higher) +- Opera 8.0 - Safari 3.2.1 (Windows version supports SNI on Vista or higher) -- Chrome (Windows version supports SNI on Vista or higher, too) Only domain names can be passed in SNI. However, some browsers will pass the IP address of the server as its name if a request includes a literal IP address. It is best not to rely on this. -In order to use SNI in NGINX, it must be supported in both the OpenSSL library with which the NGINX binary has been built, as well as the library with which it is being dynamically linked at runtime. OpenSSL supports SNI since the version 0.9.8f if it was built with configuration `option --enable-tlsext`. Since OpenSSL version 0.9.8j, this option is enabled by default. If NGINX was built with SNI support, NGINX shows the following when run with the `-V` switch: +In order to use SNI in NGINX, it must be supported in both the OpenSSL library with which the NGINX binary has been built, as well as the library with which it is being dynamically linked at runtime. OpenSSL supports SNI since the version 0.9.8f if it was built with configuration `option --enable-tlsext`. Since OpenSSL version 0.9.8j, this option is enabled by default. If NGINX was built with SNI support, NGINX shows the following when run with the `-V` switch: ```shell nginx -V @@ -305,21 +300,31 @@ therefore SNI is not available ## Compatibility Notes -- The SNI support status has been shown by the `-V` switch since versions 0.8.21 and 0.7.62. +- The SNI support status has been shown by the `-V` switch since versions 0.8.21 and 0.7.62. + +- The `ssl` parameter of the [`listen`](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive has been supported since version 0.7.14. Prior to version 0.8.21 it could only be specified along with the `default` parameter. + +- SNI has been supported since version 0.5.23. -- The `ssl` parameter to the [listen](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive has been supported since version 0.7.14. Prior to version 0.8.21 it could only be specified along with the `default` parameter. +- The shared SSL session cache has been supported since version 0.5.6. -- SNI has been supported since version 0.5.23. -- The shared SSL session cache has been supported since version 0.5.6. +### SSL protocols + + - Version 1.27.3 and later: the default SSL protocols are `TLSv1.2` and `TLSv1.3` (if supported by the OpenSSL library). Otherwise, when OpenSSL 1.0.0 or older is used, the default SSL protocols are `TLSv1` and `TLSv1.1`. + +- Version 1.23.4 and later: the default SSL protocols are `TLSv1`, `TLSv1.1`, `TLSv1.2`, and `TLSv1.3` (if supported by the OpenSSL library). - Version 1.9.1 and later: the default SSL protocols are `TLSv1`, `TLSv1.1`, and `TLSv1.2` (if supported by the OpenSSL library). -- From versions 0.7.65 and 0.8.19 and later, the default SSL protocols are `SSLv3`, `TLSv1`, `TLSv1.1`, and `TLSv1.2` (if supported by the OpenSSL library). -- In versions 0.7.64 and 0.8.18 and earlier, the default SSL protocols are `SSLv2`, `SSLv3`, and `TLSv1`. +- From versions 0.7.65 and 0.8.19 and later, the default SSL protocols are `SSLv3`, `TLSv1`, `TLSv1.1`, and `TLSv1.2` (if supported by the OpenSSL library). + +- In versions 0.7.64 and 0.8.18 and earlier, the default SSL protocols are `SSLv2`, `SSLv3`, and `TLSv1`. + +### SSL ciphers -- In version 1.0.5 and later, the default SSL ciphers are `HIGH:!aNULL:!MD5`. +- In version 1.0.5 and later, the default SSL ciphers are `HIGH:!aNULL:!MD5`. -- In versions 0.7.65 and 0.8.20 and later, the default SSL ciphers are `HIGH:!ADH:!MD5`. +- In versions 0.7.65 and 0.8.20 and later, the default SSL ciphers are `HIGH:!ADH:!MD5`. - From version 0.8.19 the default SSL ciphers are `ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM`. From 24cfa28bd357c9072cb9aa4fb67d6d841bcd37a1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 Jun 2025 09:42:01 +0100 Subject: [PATCH 019/117] build(deps): bump github/codeql-action from 3.28.19 to 3.29.0 (#678) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.28.19 to 3.29.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/fca7ace96b7d713c7035871441bd52efbe39e27e...ce28f5bb42b7a9f2c824e633a3f6ee835bab6858) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 3.29.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ossf_scorecard.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ossf_scorecard.yml b/.github/workflows/ossf_scorecard.yml index a3837c42f..63944f691 100644 --- a/.github/workflows/ossf_scorecard.yml +++ b/.github/workflows/ossf_scorecard.yml @@ -56,6 +56,6 @@ jobs: # Upload the results to GitHub's code scanning dashboard. - name: Upload SARIF results to code scanning - uses: github/codeql-action/upload-sarif@fca7ace96b7d713c7035871441bd52efbe39e27e # v3.28.19 + uses: github/codeql-action/upload-sarif@ce28f5bb42b7a9f2c824e633a3f6ee835bab6858 # v3.29.0 with: sarif_file: results.sarif From d6f0e0adec91ff7fef0b6ad8e8bd0d52091a0efb Mon Sep 17 00:00:00 2001 From: Travis Martin <33876974+travisamartin@users.noreply.github.com> Date: Mon, 16 Jun 2025 08:51:22 -0700 Subject: [PATCH 020/117] NIM release 2.20.0 (#684) * docs: add support for lightweight mode and clarify ClickHouse configuration * build(deps): bump github/codeql-action from 3.28.15 to 3.28.16 (#459) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.28.15 to 3.28.16. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/45775bd8235c68ba998cffa5171334d58593da47...28deaeda66b76a05916b6923827895f2b14ab387) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 3.28.16 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alan Dooley Co-authored-by: Mike Jang <3287976+mjang@users.noreply.github.com> * fix: Address typo, update metadata for Add a Staged Configuration topic (#549) This commit fixes a typo in the main body text of the documentation, and also updates the Hugo frontmatter to new parameter names and values. * Updated example description for uwsgi and Django. (#552) Co-authored-by: Alan Dooley * Fixed URL to dynamic modules for Wildfly deployment guide. (#548) * Fixed URL to dynamic modules for Wildfly deployment guide. * Apply suggestions from code review Co-authored-by: Alan Dooley --------- Co-authored-by: Alan Dooley Co-authored-by: Alan Dooley * Removed random symbols introduced in #512 (#546) * Add a new trigger to build push workflow (#554) feat: Add a new trigger to build push workflow * added flag to use to skip installing clickhouse * updated install guides with latest script changes * edits to system config titles * added NGINX Agent version requirement for lightweight NIM * added step for adding NGINX signing key for manual install * updated clickhouse version variable * added variable for lightweight NIM NGINX Agent requirement * reworded lightweight option to lightweight mode * added steps for adding instances * added steps for setting up metrics reporting * edits * added lightweight mode content to docker compose doc * Apply suggestions from code review Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> * edits to helm deploy for lightweight mode * Update license_usage_offline.sh * feat: Updated nim install-script (#202) * Update install-nim-bundle.sh * fix: Update install-nim-bundle.sh * removed cruft from docker-compose-lightweight.yaml * edits to docker compose comments * Update license_usage_offline.sh * feat: Updated nim install-script (#202) * Update install-nim-bundle.sh * fix: Update install-nim-bundle.sh * feat: Update the value of ssl_protocols. (#672) * NGF: Update release version to v2.0.1 (#671) Update doc version for NGF release 2.0.1 * NGF: replace static-mode with controller (#669) Problem: The static-mode command has been renamed. Solution: Update to `controller`. * Fix missing quote (#676) A quotation was missing that was not caught by the original PR pipeline. * updates to helm config settings with new chart name * docs: add 2.20 RNs * Update install-nim-manual.md * Update install.md * docs: add 2.20 KIs * incremented helm chart version table * nim 2.20 ki update steps * Update install-nim-bundle.sh (#208) --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alan Dooley Co-authored-by: Mike Jang <3287976+mjang@users.noreply.github.com> Co-authored-by: yar Co-authored-by: Alan Dooley Co-authored-by: Lam <150060045+lamATnginx@users.noreply.github.com> Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> Co-authored-by: Chetan Jain <46200286+Chetan-99@users.noreply.github.com> Co-authored-by: salonichf5 <146118978+salonichf5@users.noreply.github.com> Co-authored-by: Saylor Berman Co-authored-by: Jon Cahill-Torre Co-authored-by: Micheal Kingston <44472403+mkingst@users.noreply.github.com> --- cleanup_log.txt | 929 +++++++++++++ content/includes/installation/add-nms-repo.md | 38 +- .../installation/clickhouse-defaults.md | 21 - .../nginx-plus/usage-tracking/overview.md | 2 +- .../license/connected-install-license-note.md | 2 +- .../nim/clickhouse/cli-skip-clickhouse.md | 10 + .../nim/clickhouse/clickhouse-defaults.md | 23 + .../nim/clickhouse/clickhouse-install.md | 70 + .../nim/docker/docker-compose-env-vars.md | 8 +- .../installation/install-script-flags/cert.md | 8 + .../clickhouse-version.md | 8 + .../install-script-flags/distribution.md | 14 + .../installation/install-script-flags/key.md | 8 + .../install-script-flags/skip-clickhouse.md | 10 + .../optional-steps/configure-clickhouse.md | 23 + .../optional-steps/configure-selinux.md | 12 + .../disable-metrics-collection.md | 11 + .../optional-steps/install-configure-vault.md | 16 + .../nms-chart-supported-module-versions.md | 1 + .../nim/tech-specs/supported-distros.md | 2 + .../tech-specs/supported-nginx-versions.md | 4 +- content/nim/admin-guide/_index.md | 2 +- .../admin-guide/{license => }/add-license.md | 6 +- content/nim/admin-guide/license/_index.md | 7 - content/nim/admin-guide/rbac/overview-rbac.md | 2 +- .../report-usage-connected-deployment.md | 10 +- content/nim/deploy/_index.md | 2 +- ...y-nginx-instance-manager-docker-compose.md | 271 +++- .../deploy/kubernetes/deploy-using-helm.md | 438 +++--- .../deploy/kubernetes/helm-config-settings.md | 157 +-- .../vm-bare-metal/install-nim-deprecated.md | 315 ----- .../vm-bare-metal/install-nim-manual.md | 233 ++++ content/nim/deploy/vm-bare-metal/install.md | 170 ++- content/nim/disconnected/_index.md | 2 +- .../add-license-disconnected-deployment.md | 4 +- ...ted.md => offline-install-guide-manual.md} | 49 +- .../nim/disconnected/offline-install-guide.md | 107 +- .../report-usage-disconnected-deployment.md | 2 +- .../set-up-app-protect-instances.md | 2 +- content/nim/nginx-instances/add-instance.md | 49 + content/nim/releases/known-issues.md | 164 ++- content/nim/releases/release-notes.md | 93 +- .../configure-clickhouse.md | 90 +- .../configure-forward-proxy.md | 2 +- .../configure-high-availability.md | 2 +- .../configure-telemetry.md | 4 +- .../system-configuration/configure-vault.md | 2 +- .../configure-with-config.md | 2 +- .../system-configuration/secure-traffic.md | 4 +- .../how-to/deploy-api-connectivity-manager.md | 2 +- content/nms/acm/how-to/install-acm.md | 2 +- content/nms/acm/tutorials/advanced-routing.md | 2 +- .../solutions/about-subscription-licenses.md | 4 +- layouts/shortcodes/clickhouse-version.html | 1 + .../lightweight-nim-nginx-agent-version.html | 1 + .../docker-compose-lightweight.yaml | 93 ++ static/scripts/install-nim-bundle.sh | 1220 +++++++---------- static/scripts/license_usage_offline.sh | 743 +++++----- 58 files changed, 3413 insertions(+), 2066 deletions(-) create mode 100644 cleanup_log.txt delete mode 100644 content/includes/installation/clickhouse-defaults.md create mode 100644 content/includes/nim/clickhouse/cli-skip-clickhouse.md create mode 100644 content/includes/nim/clickhouse/clickhouse-defaults.md create mode 100644 content/includes/nim/clickhouse/clickhouse-install.md create mode 100644 content/includes/nim/installation/install-script-flags/cert.md create mode 100644 content/includes/nim/installation/install-script-flags/clickhouse-version.md create mode 100644 content/includes/nim/installation/install-script-flags/distribution.md create mode 100644 content/includes/nim/installation/install-script-flags/key.md create mode 100644 content/includes/nim/installation/install-script-flags/skip-clickhouse.md create mode 100644 content/includes/nim/installation/optional-steps/configure-clickhouse.md create mode 100644 content/includes/nim/installation/optional-steps/configure-selinux.md create mode 100644 content/includes/nim/installation/optional-steps/disable-metrics-collection.md create mode 100644 content/includes/nim/installation/optional-steps/install-configure-vault.md rename content/nim/admin-guide/{license => }/add-license.md (88%) delete mode 100644 content/nim/admin-guide/license/_index.md rename content/nim/admin-guide/{license => }/report-usage-connected-deployment.md (85%) delete mode 100644 content/nim/deploy/vm-bare-metal/install-nim-deprecated.md create mode 100644 content/nim/deploy/vm-bare-metal/install-nim-manual.md rename content/nim/disconnected/{offline-install-guide-deprecated.md => offline-install-guide-manual.md} (81%) create mode 100644 content/nim/nginx-instances/add-instance.md create mode 100644 layouts/shortcodes/clickhouse-version.html create mode 100644 layouts/shortcodes/lightweight-nim-nginx-agent-version.html create mode 100644 static/scripts/docker-compose/docker-compose-lightweight.yaml diff --git a/cleanup_log.txt b/cleanup_log.txt new file mode 100644 index 000000000..e4a129c9c --- /dev/null +++ b/cleanup_log.txt @@ -0,0 +1,929 @@ +2025-05-21 06:21:52,139: File read successfully. +2025-05-21 06:21:52,140: Initial shape (rows x columns): (743, 39) +2025-05-21 06:21:52,140: Subheading row detected and removed. +2025-05-21 06:21:52,143: Dataframe shape after header adjustments: (741, 39) +2025-05-21 06:21:52,143: Columns found: ['StartDate', 'EndDate', 'Status', 'IPAddress', 'Progress', 'Duration (in seconds)', 'Finished', 'RecordedDate', 'ResponseId', 'RecipientLastName', 'RecipientFirstName', 'RecipientEmail', 'ExternalReference', 'LocationLatitude', 'LocationLongitude', 'DistributionChannel', 'UserLanguage', 'Q1', 'Q2', 'Q3', 'Q4', 'current_url', 'QID2_OriginalResponse', 'QID2_AICategory', 'QID2_FollowUpPrompt', 'Link URL', 'Q2 - Actionability', 'Q2 - Effort', 'Q2 - Effort Numeric', 'Q2 - Emotion Intensity', 'Q2 - Emotion', 'Q2 - Parent Topics', 'Q2 - Sentiment Polarity', 'Q2 - Sentiment Score', 'Q2 - Sentiment', 'Q2 - Topic Sentiment Label', 'Q2 - Topic Sentiment Score', 'Q2 - Topics', 'Q2 - Topic Hierarchy Level 1'] +2025-05-21 06:21:52,157: Row 0 deleted because Link URL is missing. +2025-05-21 06:21:52,157: Row 1 deleted because Link URL is missing. +2025-05-21 06:21:52,157: Row 2 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 3 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 4 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 5 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 6 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 7 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 8 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 9 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 10 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 11 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 12 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 13 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 14 deleted because Link URL is missing. +2025-05-21 06:21:52,158: Row 15 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 16 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 17 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 18 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 19 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 20 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 21 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 22 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 23 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 24 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 25 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 26 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 27 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 28 deleted because Link URL is missing. +2025-05-21 06:21:52,159: Row 29 deleted because Link URL is missing. +2025-05-21 06:21:52,160: Row 30 deleted because Link URL is missing. +2025-05-21 06:21:52,160: Row 31 deleted because Link URL is missing. +2025-05-21 06:21:52,160: Row 32 deleted because Link URL is missing. +2025-05-21 06:21:52,160: Row 33 deleted because Link URL is missing. +2025-05-21 06:21:52,160: Row 34 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 35 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 36 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 37 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 38 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 39 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 40 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 41 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 42 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 43 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 44 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 45 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 46 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 47 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 48 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 49 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 50 deleted because Link URL is missing. +2025-05-21 06:21:52,161: Row 213 deleted because Link URL is missing. +2025-05-21 06:21:52,162: Rows before removing missing Link URL: 741, after: 689 +2025-05-21 06:21:52,163: Row 51 deleted because Q2 contains 'testing'. +2025-05-21 06:21:52,163: Row 77 deleted because Q2 contains 'testing'. +2025-05-21 06:21:52,163: Row 78 deleted because Q2 contains 'testing'. +2025-05-21 06:21:52,163: Row 169 deleted because Q2 contains 'testing'. +2025-05-21 06:21:52,163: Rows before removing Q2='testing': 689, after: 685 +2025-05-21 06:21:52,165: Row 58: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration to https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/ +2025-05-21 06:21:52,165: Row 59: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/upgrade-to-v4/#update-custom-resource-apiversion to https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/upgrade-to-v4/ +2025-05-21 06:21:52,165: Row 71: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/custom-templates to https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/custom-templates/ +2025-05-21 06:21:52,165: Row 88: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/overview/design/#nginx-ingress-controller-process-components to https://docs.nginx.com/nginx-ingress-controller/overview/design/ +2025-05-21 06:21:52,165: Row 115: Link URL updated from https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#weights to https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/ +2025-05-21 06:21:52,165: Row 120: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration to https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/ +2025-05-21 06:21:52,165: Row 152: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/glossary/#ingress-controller-ingress-controller to https://docs.nginx.com/nginx-ingress-controller/glossary/ +2025-05-21 06:21:52,165: Row 297: Link URL updated from https://docs.nginx.com/ossc to https://docs.nginx.com/ossc/ +2025-05-21 06:21:52,165: Row 308: Link URL updated from https://docs.nginx.com/ossc to https://docs.nginx.com/ossc/ +2025-05-21 06:21:52,166: Row 324: Link URL updated from https://docs.nginx.com/nginxaas-azure/known-issues/ to https://docs.nginx.com/nginxaas/azure/known-issues/ (URL replacement) +2025-05-21 06:21:52,166: Row 324: Link URL updated from https://docs.nginx.com/nginxaas-azure/known-issues/ to https://docs.nginx.com/nginxaas/azure/known-issues/ +2025-05-21 06:21:52,166: Row 367: Link URL updated from https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#prebuilt to https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/ +2025-05-21 06:21:52,166: Row 395: Link URL updated from https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#locations to https://docs.nginx.com/nginx/admin-guide/web-server/web-server/ +2025-05-21 06:21:52,166: Row 396: Link URL updated from https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#method to https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/ +2025-05-21 06:21:52,166: Row 417: Link URL updated from https://docs.nginx.com/nginx/admin-guide/basic-functionality/managing-configuration-files/#virtual-servers to https://docs.nginx.com/nginx/admin-guide/basic-functionality/managing-configuration-files/ +2025-05-21 06:21:52,166: Row 418: Link URL updated from https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/#upgrading-nginx-plus to https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/ +2025-05-21 06:21:52,166: Row 477: Link URL updated from https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/#master-and-worker-processes to https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/ +2025-05-21 06:21:52,166: Row 501: Link URL updated from https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source to https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/ +2025-05-21 06:21:52,167: Row 587: Link URL updated from https://docs.nginx.com/ossc to https://docs.nginx.com/ossc/ +2025-05-21 06:21:52,167: Row 590: Link URL updated from https://docs.nginx.com/ossc to https://docs.nginx.com/ossc/ +2025-05-21 06:21:52,167: Row 592: Link URL updated from https://docs.nginx.com/nginx-one/getting-started/#enable-nginx-plus-api to https://docs.nginx.com/nginx-one/getting-started/ +2025-05-21 06:21:52,167: Row 617: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource to https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/ +2025-05-21 06:21:52,167: Row 636: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration to https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/ +2025-05-21 06:21:52,167: Row 692: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration to https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/ +2025-05-21 06:21:52,167: Row 695: Link URL updated from https://docs.nginx.com/nginx-management-suite/acm/how-to/infrastructure/manage-api-infrastructure/#add-environment to https://docs.nginx.com/nginx-management-suite/acm/how-to/infrastructure/manage-api-infrastructure/ +2025-05-21 06:21:52,167: Row 729: Link URL updated from https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/command-line-arguments to https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/command-line-arguments/ +2025-05-21 06:21:52,168: Total Link URLs updated: 26 +2025-05-21 06:21:52,170: Exclude file loaded. Excluding 111 ResponseIDs. +2025-05-21 06:21:52,172: Excluded bogus responses: removed 104 rows based on ResponseId. +2025-05-21 06:21:52,172: Removed ResponseIDs: ['R_2BFXRROEE9C3BRP', 'R_2D8NGQPK1IWGIHY', 'R_2FP9OE7JF5GLHOP', 'R_2J2ML9WMO3BCXZ0', 'R_2LGQFCKEOOAPFCT', 'R_2RPFQ7PIR4C48DT', 'R_3J7OQ6V3GPEPYSJ', 'R_3JG7M3IHPME7LRX', 'R_401DHNTF0HMOWBL', 'R_40YHV7FFGANJU6F', 'R_424K44PYVRK8Y1G', 'R_429OWKCURJNOCTK', 'R_42NYN092DSVZXN7', 'R_45R6NUBXVOIPVEN', 'R_47VDGIRXA5NNCGO', 'R_4AK2INIGD1FFTDO', 'R_4BKCWEQU5MOZHA9', 'R_4DLQ3LQ6NHSZHBJ', 'R_4EFZQCTIWFGSTS9', 'R_4EXDDJGUFFTHDCC', 'R_4FO80F1TI4XXMDF', 'R_4GOFQWU2CBJQ9I8', 'R_4HOXH1YLAOVUM8Q', 'R_4JPODUU6FT5EJTD', 'R_4KHCYH2G4RX8GU6', 'R_4LGNMF3ZOJTCHSX', 'R_4MQKUOCWNMU1SPL', 'R_4N9TKVM3T4LLYKG', 'R_4NASUNJJHBDUNYB', 'R_4O67VYRCF2LMJJH', 'R_4OAKKCL6GPFZ6HE', 'R_4P2A79ACQ2DAXVN', 'R_4P9VTDYBBAGHH50', 'R_4QAAZ7C7ZBWLAZP', 'R_4QEFBD2SFDQ0REO', 'R_4QSJZTOLA1EOPTV', 'R_4QVW5AOMQL8JKAH', 'R_4QZ4BOZX5ZAWT8O', 'R_4SBC6FAX8YM8UGW', 'R_4SFJ4F2DSWFAHQZ', 'R_4ST9JXVOCQPNZWV', 'R_4U6KCOFBA0VPNFR', 'R_4VOH9KW1KCSSUJF', 'R_4WMMZUSHCE79AJG', 'R_4YE1NR6MWNJKK3P', 'R_51KKOSYZR16XRWF', 'R_5JBAKUSVN2PQSYJ', 'R_5MMU8NUYQQORLYE', 'R_6F3YPGOQKQMDPMC', 'R_6JLNAULKUPECILR', 'R_6PPUIKFYISENOOK', 'R_6VCPCX5BLHAFBKC', 'R_84BS33CREBBE09U', 'R_85HYVNJRFKFIUZ7', 'R_8C1JLR1AQTTTNE4', 'R_8NMFUZTTQCCOCN1', 'R_8OYXTWQHPPI60A9', 'R_8RTA9M4IQEQQDTR', 'R_8SBR3CGJS6QKOUL', 'R_8TTJC5OTE89JFJJ', 'R_8V8S2WSR76BZVDZ', 'R_8VPTIBSYWQJNJ42', 'R_8WBVOCJ7VLNATZF', 'R_906TQMCOD2GZKZH', 'R_93CMAHY49QS6HFS', 'R_93LGPITFTUNJBD4', 'R_96FPCSSZZ6OQGA5', 'R_97G6U0YYUPWZKAJ', 'R_9889CFZHAMBBJKL', 'R_9949KYHTOMG0ZXU', 'R_99ANGFU2UYWLPTX', 'R_9AGAOUORA4CPHTJ', 'R_9BAI4SFWRIEMME6', 'R_9BSJVEIWI90STOT', 'R_9C5BLPTCDY9OWIP', 'R_9CIYQQ0HJBEUA0O', 'R_9CPNDHPV7ZKBRFP', 'R_9DIMXPUAFKMVKJW', 'R_9DPMP66UHSZNLMP', 'R_9DUZARWZWO1OTNU', 'R_9EIFBZMGJ2RIMFJ', 'R_9EY7BUWKXFBNJWK', 'R_9FWXWDHSKHI0KCC', 'R_9HRYIJ0WJQNXLBV', 'R_9HVUPXH88UF2ALP', 'R_9J6C8A0F54FGTOE', 'R_9J9IEQJV2QIJNOO', 'R_9K6G6NUBNDQSLZC', 'R_9KOOA1DO1YZQCNN', 'R_9LIA90ICK0Z49X1', 'R_9MGHZSS6CHGXIGY', 'R_9MWUMGQY6B6KBSX', 'R_9OHCDWPHUIEEKKU', 'R_9OLOUQJMICBQCSJ', 'R_9PC2GVYGYM5RML5', 'R_9POGVOMHFMYBFJ2', 'R_9PPYOF1BOHS61LV', 'R_9QKQ89HE4TJYTIY', 'R_9ROSQUGFBCRUCUU', 'R_9RPKMREBNL1WAIO', 'R_9SUDDZVEJT2JFAS', 'R_9UASQESCV6I1RJD', 'R_9VIEDXC63JY6EL7', 'R_9ZAMFS0DAF1Z2ZT'] +2025-05-21 06:21:52,175: Row 712: Found and removed 1 email(s) from Q2. +2025-05-21 06:21:52,175: Row 728: Found and removed 1 email(s) from Q2. +2025-05-21 06:21:52,739: Row 52: Reverse geocoded lat=-29.9228, lon=-51.1744 -> country=Brazil, city=Canoas, state=Rio Grande do Sul +2025-05-21 06:21:53,745: Row 53: Reverse geocoded lat=26.8756, lon=80.9115 -> country=India, city=Lucknow, state=Uttar Pradesh +2025-05-21 06:21:54,735: Row 54: Reverse geocoded lat=13.6671, lon=100.5775 -> country=Thailand, city=Pu Chao Saming Phrai Town Municipality, state=None +2025-05-21 06:21:55,697: Row 55: Reverse geocoded lat=36.6229, lon=114.4698 -> country=China, city=Handan, state=Hebei +2025-05-21 06:21:56,656: Row 56: Reverse geocoded lat=-16.561, lon=-49.2063 -> country=Brazil, city=Goiânia, state=Goiás +2025-05-21 06:21:57,664: Row 57: Reverse geocoded lat=-16.561, lon=-49.2063 -> country=Brazil, city=Goiânia, state=Goiás +2025-05-21 06:21:58,695: Row 58: Reverse geocoded lat=53.481, lon=-2.2367 -> country=United Kingdom, city=Manchester, state=England +2025-05-21 06:21:59,700: Row 60: Reverse geocoded lat=50.1187, lon=8.6842 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:22:00,919: Row 61: Reverse geocoded lat=17.9641, lon=102.5987 -> country=Laos, city=Vientiane Capital, state=Vientiane Prefecture +2025-05-21 06:22:01,696: Row 62: Reverse geocoded lat=25.5908, lon=85.1348 -> country=India, city=Patna, state=Bihar +2025-05-21 06:22:02,796: Row 63: Reverse geocoded lat=13.7159, lon=100.417 -> country=Thailand, city=Bangkok, state=None +2025-05-21 06:22:04,250: Row 64: Reverse geocoded lat=48.157, lon=17.0915 -> country=Slovakia, city=Bratislava, state=Region of Bratislava +2025-05-21 06:22:04,695: Row 65: Reverse geocoded lat=20.6381, lon=-103.348 -> country=Mexico, city=Guadalajara, state=Jalisco +2025-05-21 06:22:05,783: Row 66: Reverse geocoded lat=22.155, lon=-101.9465 -> country=Mexico, city=Villa García, state=Zacatecas +2025-05-21 06:22:07,105: Row 68: Reverse geocoded lat=49.1231, lon=8.6011 -> country=Germany, city=Bruchsal, state=Baden-Württemberg +2025-05-21 06:22:07,783: Row 69: Reverse geocoded lat=49.1231, lon=8.6011 -> country=Germany, city=Bruchsal, state=Baden-Württemberg +2025-05-21 06:22:08,729: Row 70: Reverse geocoded lat=23.0276, lon=72.5871 -> country=India, city=Ahmedabad, state=Gujarat +2025-05-21 06:22:09,796: Row 71: Reverse geocoded lat=39.8655, lon=-86.122 -> country=United States, city=Indianapolis, state=Indiana +2025-05-21 06:22:10,932: Row 72: Reverse geocoded lat=38.7584, lon=-9.2495 -> country=Portugal, city=Sintra, state=None +2025-05-21 06:22:11,896: Row 73: Reverse geocoded lat=52.9044, lon=-1.2326 -> country=United Kingdom, city=Nottingham, state=England +2025-05-21 06:22:12,850: Row 74: Reverse geocoded lat=50.1155, lon=14.4124 -> country=Czechia, city=Capital City of Prague, state=Prague +2025-05-21 06:22:14,282: Row 75: Reverse geocoded lat=51.3593, lon=7.4763 -> country=Germany, city=Hagen, state=North Rhine-Westphalia +2025-05-21 06:22:14,794: Row 76: Reverse geocoded lat=-1.2841, lon=36.8155 -> country=Kenya, city=Nairobi, state=Nairobi County +2025-05-21 06:22:15,810: Row 79: Reverse geocoded lat=46.0653, lon=23.5917 -> country=Romania, city=Alba Iulia, state=None +2025-05-21 06:22:16,899: Row 80: Reverse geocoded lat=46.7657, lon=23.5943 -> country=Romania, city=Cluj-Napoca, state=None +2025-05-21 06:22:17,742: Row 81: Reverse geocoded lat=51.4964, lon=-0.1224 -> country=United Kingdom, city=London, state=England +2025-05-21 06:22:19,063: Row 82: Reverse geocoded lat=34.6946, lon=135.5021 -> country=Japan, city=Osaka, state=None +2025-05-21 06:22:19,876: Row 83: Reverse geocoded lat=37.5079, lon=127.1177 -> country=South Korea, city=Seoul, state=None +2025-05-21 06:22:20,974: Row 84: Reverse geocoded lat=46.4752, lon=30.7338 -> country=Ukraine, city=Odesa, state=Odesa Oblast +2025-05-21 06:22:21,672: Row 85: Reverse geocoded lat=46.4752, lon=30.7338 -> country=Ukraine, city=Odesa, state=Odesa Oblast +2025-05-21 06:22:22,667: Row 86: Reverse geocoded lat=-23.1798, lon=-46.8816 -> country=Brazil, city=Jundiaí, state=São Paulo +2025-05-21 06:22:23,748: Row 87: Reverse geocoded lat=37.5537, lon=-77.4602 -> country=United States, city=Richmond, state=Virginia +2025-05-21 06:22:24,717: Row 88: Reverse geocoded lat=22.2578, lon=114.1657 -> country=China, city=Hong Kong Island, state=Hong Kong +2025-05-21 06:22:25,754: Row 89: Reverse geocoded lat=36.7637, lon=-119.7602 -> country=United States, city=Fresno, state=California +2025-05-21 06:22:26,727: Row 90: Reverse geocoded lat=28.6542, lon=77.2373 -> country=India, city=Delhi, state=Delhi +2025-05-21 06:22:27,804: Row 91: Reverse geocoded lat=50.458, lon=30.5303 -> country=Ukraine, city=Kyiv, state=None +2025-05-21 06:22:29,090: Row 92: Reverse geocoded lat=3.1685, lon=101.7034 -> country=Malaysia, city=Kuala Lumpur, state=Kuala Lumpur +2025-05-21 06:22:29,779: Row 93: Reverse geocoded lat=36.7943, lon=-119.8856 -> country=United States, city=Fresno, state=California +2025-05-21 06:22:30,928: Row 94: Reverse geocoded lat=47.1085, lon=15.7054 -> country=Austria, city=Gleisdorf, state=Styria +2025-05-21 06:22:31,774: Row 95: Reverse geocoded lat=14.5971, lon=120.9798 -> country=Philippines, city=Manila, state=None +2025-05-21 06:22:32,873: Row 96: Reverse geocoded lat=30.5851, lon=114.2662 -> country=China, city=Jianghan District, state=Hubei +2025-05-21 06:22:33,711: Row 97: Reverse geocoded lat=32.4643, lon=14.5726 -> country=Libya, city=Zliten, state=Bani Walid +2025-05-21 06:22:34,796: Row 98: Reverse geocoded lat=59.3591, lon=17.9951 -> country=Sweden, city=Solna, state=None +2025-05-21 06:22:35,676: Row 99: Reverse geocoded lat=38.8897, lon=-121.2873 -> country=United States, city=Lincoln, state=California +2025-05-21 06:22:36,708: Row 100: Reverse geocoded lat=-22.9616, lon=-49.8632 -> country=Brazil, city=Ourinhos, state=São Paulo +2025-05-21 06:22:37,835: Row 101: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:22:38,698: Row 102: Reverse geocoded lat=39.018, lon=-77.539 -> country=United States, city=None, state=Virginia +2025-05-21 06:22:39,654: Row 103: Reverse geocoded lat=40.1086, lon=-75.2835 -> country=United States, city=Plymouth Township, state=Pennsylvania +2025-05-21 06:22:40,744: Row 104: Reverse geocoded lat=46.0653, lon=23.5917 -> country=Romania, city=Alba Iulia, state=None +2025-05-21 06:22:41,718: Row 105: Reverse geocoded lat=-6.175, lon=106.8286 -> country=Indonesia, city=Special capital Region of Jakarta, state=None +2025-05-21 06:22:42,737: Row 106: Reverse geocoded lat=37.1554, lon=9.7861 -> country=Tunisia, city=None, state=Bizerte +2025-05-21 06:22:43,679: Row 107: Reverse geocoded lat=51.2437, lon=-2.2746 -> country=United Kingdom, city=None, state=England +2025-05-21 06:22:44,800: Row 108: Reverse geocoded lat=12.9753, lon=77.591 -> country=India, city=Bengaluru, state=Karnataka +2025-05-21 06:22:45,823: Row 109: Reverse geocoded lat=32.4633, lon=44.4179 -> country=Iraq, city=Al Hillah, state=Babil Governorate +2025-05-21 06:22:46,743: Row 110: Reverse geocoded lat=-33.8715, lon=151.2006 -> country=Australia, city=Sydney, state=New South Wales +2025-05-21 06:22:47,768: Row 111: Reverse geocoded lat=14.3269, lon=120.9336 -> country=Philippines, city=Dasmariñas, state=Cavite +2025-05-21 06:22:48,782: Row 112: Reverse geocoded lat=36.3188, lon=139.579 -> country=Japan, city=Sano, state=None +2025-05-21 06:22:49,815: Row 114: Reverse geocoded lat=9.0245, lon=38.7485 -> country=Ethiopia, city=None, state=Addis Ababa +2025-05-21 06:22:50,739: Row 115: Reverse geocoded lat=9.0245, lon=38.7485 -> country=Ethiopia, city=None, state=Addis Ababa +2025-05-21 06:22:51,687: Row 116: Reverse geocoded lat=39.018, lon=-77.539 -> country=United States, city=None, state=Virginia +2025-05-21 06:22:52,702: Row 117: Reverse geocoded lat=42.6826, lon=23.3223 -> country=Bulgaria, city=Sofia, state=Sofia-City +2025-05-21 06:22:53,690: Row 119: Reverse geocoded lat=38.7219, lon=-9.1398 -> country=Portugal, city=Lisbon, state=None +2025-05-21 06:22:54,806: Row 120: Reverse geocoded lat=35.5641, lon=139.5295 -> country=Japan, city=Yokohama, state=None +2025-05-21 06:22:55,755: Row 121: Reverse geocoded lat=19.0748, lon=72.8856 -> country=India, city=Mumbai, state=Maharashtra +2025-05-21 06:22:56,698: Row 122: Reverse geocoded lat=23.0276, lon=72.5871 -> country=India, city=Ahmedabad, state=Gujarat +2025-05-21 06:22:57,803: Row 123: Reverse geocoded lat=11.3085, lon=106.0948 -> country=Vietnam, city=Tây Ninh, state=Tây Ninh Province +2025-05-21 06:22:58,705: Row 124: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:22:59,737: Row 125: Reverse geocoded lat=14.3614, lon=121.4875 -> country=Philippines, city=Paete, state=Laguna +2025-05-21 06:23:00,876: Row 126: Reverse geocoded lat=29.3959, lon=-98.4761 -> country=United States, city=San Antonio, state=Texas +2025-05-21 06:23:01,698: Row 127: Reverse geocoded lat=26.1885, lon=91.7392 -> country=India, city=None, state=Assam +2025-05-21 06:23:02,688: Row 128: Reverse geocoded lat=-18.913, lon=47.5296 -> country=Madagascar, city=Antananarivo, state=Analamanga +2025-05-21 06:23:03,716: Row 129: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:23:05,028: Row 131: Reverse geocoded lat=52.1928, lon=5.4199 -> country=Netherlands, city=Amersfoort, state=Utrecht +2025-05-21 06:23:06,180: Row 132: Reverse geocoded lat=51.9277, lon=4.5844 -> country=Netherlands, city=Capelle aan den IJssel, state=South Holland +2025-05-21 06:23:06,676: Row 133: Reverse geocoded lat=40.5828, lon=-73.9532 -> country=United States, city=New York, state=New York +2025-05-21 06:23:07,719: Row 134: Reverse geocoded lat=36.5312, lon=3.0698 -> country=Algeria, city=Bougara, state=Blida +2025-05-21 06:23:09,028: Row 135: Reverse geocoded lat=49.2625, lon=8.6235 -> country=Germany, city=None, state=Baden-Württemberg +2025-05-21 06:23:09,811: Row 137: Reverse geocoded lat=36.061, lon=120.3814 -> country=China, city=Shinan District, state=Shandong +2025-05-21 06:23:11,054: Row 138: Reverse geocoded lat=20.0024, lon=73.7945 -> country=India, city=Nashik, state=Maharashtra +2025-05-21 06:23:11,728: Row 139: Reverse geocoded lat=-25.0907, lon=-50.1653 -> country=Brazil, city=Ponta Grossa, state=Paraná +2025-05-21 06:23:12,696: Row 140: Reverse geocoded lat=25.0734, lon=55.2979 -> country=United Arab Emirates, city=None, state=Dubai +2025-05-21 06:23:14,214: Row 141: Reverse geocoded lat=52.3759, lon=4.8975 -> country=Netherlands, city=Amsterdam, state=North Holland +2025-05-21 06:23:15,323: Row 142: Reverse geocoded lat=36.1898, lon=44.0157 -> country=Iraq, city=Erbil, state=Erbil Governorate +2025-05-21 06:23:16,104: Row 143: Reverse geocoded lat=53.3382, lon=-6.2591 -> country=Ireland, city=Dublin, state=None +2025-05-21 06:23:16,986: Row 144: Reverse geocoded lat=37.751, lon=-97.822 -> country=United States, city=None, state=Kansas +2025-05-21 06:23:18,038: Row 145: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:23:19,029: Row 146: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:23:20,112: Row 147: Reverse geocoded lat=40.7547, lon=-73.9614 -> country=United States, city=New York, state=New York +2025-05-21 06:23:21,068: Row 148: Reverse geocoded lat=31.33, lon=75.5844 -> country=India, city=Jalandhar, state=Punjab +2025-05-21 06:23:22,064: Row 149: Reverse geocoded lat=26.8373, lon=80.9165 -> country=India, city=Lucknow, state=Uttar Pradesh +2025-05-21 06:23:23,110: Row 150: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:23:24,010: Row 151: Reverse geocoded lat=25.5943, lon=85.1352 -> country=India, city=Patna, state=Bihar +2025-05-21 06:23:24,998: Row 152: Reverse geocoded lat=36.61, lon=114.4876 -> country=China, city=Handan, state=Hebei +2025-05-21 06:23:26,476: Row 154: Reverse geocoded lat=44.776, lon=17.1995 -> country=Bosnia and Herzegovina, city=Banja Luka, state=Republika Srpska +2025-05-21 06:23:26,997: Row 155: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:23:28,037: Row 156: Reverse geocoded lat=51.4964, lon=-0.1224 -> country=United Kingdom, city=London, state=England +2025-05-21 06:23:29,244: Row 157: Reverse geocoded lat=35.5641, lon=45.4326 -> country=Iraq, city=Sulaymaniyah, state=Sulaymaniyah Governorate +2025-05-21 06:23:29,989: Row 158: Reverse geocoded lat=35.5641, lon=45.4326 -> country=Iraq, city=Sulaymaniyah, state=Sulaymaniyah Governorate +2025-05-21 06:23:31,087: Row 159: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:23:32,044: Row 160: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:23:33,016: Row 161: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:23:34,029: Row 162: Reverse geocoded lat=-22.8951, lon=-47.0439 -> country=Brazil, city=Campinas, state=São Paulo +2025-05-21 06:23:35,059: Row 163: Reverse geocoded lat=4.0505, lon=9.7023 -> country=Cameroon, city=Douala I, state=Littoral +2025-05-21 06:23:36,094: Row 164: Reverse geocoded lat=19.4371, lon=-99.0111 -> country=Mexico, city=Texcoco de Mora, state=State of Mexico +2025-05-21 06:23:37,037: Row 165: Reverse geocoded lat=60.1719, lon=24.9347 -> country=Finland, city=Helsinki, state=Uusimaa +2025-05-21 06:23:38,004: Row 166: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:23:39,241: Row 167: Reverse geocoded lat=49.5546, lon=25.5842 -> country=Ukraine, city=Ternopil, state=Ternopil Oblast +2025-05-21 06:23:40,016: Row 168: Reverse geocoded lat=40.4327, lon=-3.621 -> country=Spain, city=Madrid, state=Community of Madrid +2025-05-21 06:23:41,041: Row 170: Reverse geocoded lat=-24.0589, lon=-46.8456 -> country=Brazil, city=Itanhaem, state=São Paulo +2025-05-21 06:23:42,079: Row 171: Reverse geocoded lat=33.3967, lon=-111.6545 -> country=United States, city=Mesa, state=Arizona +2025-05-21 06:23:43,013: Row 172: Reverse geocoded lat=33.3967, lon=-111.6545 -> country=United States, city=Mesa, state=Arizona +2025-05-21 06:23:44,056: Row 173: Reverse geocoded lat=21.9974, lon=79.0011 -> country=India, city=Malhanwara, state=Madhya Pradesh +2025-05-21 06:23:45,153: Row 174: Reverse geocoded lat=51.7655, lon=-0.4482 -> country=United Kingdom, city=Dacorum, state=England +2025-05-21 06:23:46,204: Row 175: Reverse geocoded lat=26.58, lon=106.7223 -> country=China, city=Yunyan, state=Guizhou +2025-05-21 06:23:47,032: Row 176: Reverse geocoded lat=19.0748, lon=72.8856 -> country=India, city=Mumbai, state=Maharashtra +2025-05-21 06:23:48,146: Row 177: Reverse geocoded lat=30.6667, lon=104.0667 -> country=China, city=Chengdu, state=Sichuan +2025-05-21 06:23:48,996: Row 178: Reverse geocoded lat=48, lon=68 -> country=Kazakhstan, city=Jezqazğan, state=Ulytau Region +2025-05-21 06:23:50,382: Row 179: Reverse geocoded lat=41.0145, lon=28.9533 -> country=Turkey, city=Istanbul, state=Istanbul +2025-05-21 06:23:51,037: Row 180: Reverse geocoded lat=60.1719, lon=24.9347 -> country=Finland, city=Helsinki, state=Uusimaa +2025-05-21 06:23:52,005: Row 181: Reverse geocoded lat=37.1038, lon=-113.5764 -> country=United States, city=Saint George, state=Utah +2025-05-21 06:23:53,205: Row 182: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:23:54,144: Row 183: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:23:55,048: Row 184: Reverse geocoded lat=22.6276, lon=110.1507 -> country=China, city=Yulin, state=Guangxi +2025-05-21 06:23:55,996: Row 185: Reverse geocoded lat=-26.9873, lon=-48.6276 -> country=Brazil, city=Balneário Camboriú, state=Santa Catarina +2025-05-21 06:23:57,100: Row 186: Reverse geocoded lat=-27.6157, lon=-48.6235 -> country=Brazil, city=São José, state=Santa Catarina +2025-05-21 06:23:58,085: Row 187: Reverse geocoded lat=23.1181, lon=113.2539 -> country=China, city=Yuexiu District, state=Guangdong Province +2025-05-21 06:23:59,137: Row 188: Reverse geocoded lat=58.0701, lon=11.8131 -> country=Sweden, city=Stenungsön, state=None +2025-05-21 06:24:00,071: Row 189: Reverse geocoded lat=50.6223, lon=26.2396 -> country=Ukraine, city=Rivne, state=Rivne Oblast +2025-05-21 06:24:01,008: Row 190: Reverse geocoded lat=37.751, lon=-97.822 -> country=United States, city=None, state=Kansas +2025-05-21 06:24:02,316: Row 191: Reverse geocoded lat=17.3843, lon=78.4583 -> country=India, city=Hyderabad, state=Telangana +2025-05-21 06:24:03,051: Row 192: Reverse geocoded lat=11.5583, lon=104.9121 -> country=Cambodia, city=Khan Prampi Makara, state=Phnom Penh +2025-05-21 06:24:04,186: Row 193: Reverse geocoded lat=29.3885, lon=76.9669 -> country=India, city=Panipat, state=Haryana +2025-05-21 06:24:05,218: Row 194: Reverse geocoded lat=44.447, lon=26.0185 -> country=Romania, city=Roșu, state=None +2025-05-21 06:24:06,146: Row 195: Reverse geocoded lat=-6.2114, lon=106.8446 -> country=Indonesia, city=Special capital Region of Jakarta, state=None +2025-05-21 06:24:07,006: Row 196: Reverse geocoded lat=25.5943, lon=85.1352 -> country=India, city=Patna, state=Bihar +2025-05-21 06:24:08,129: Row 197: Reverse geocoded lat=33.8651, lon=-118.0673 -> country=United States, city=Cerritos, state=California +2025-05-21 06:24:09,049: Row 198: Reverse geocoded lat=25.2925, lon=51.5321 -> country=Qatar, city=Doha, state=None +2025-05-21 06:24:10,356: Row 199: Reverse geocoded lat=-6.2876, lon=106.722 -> country=Indonesia, city=South Tangerang, state=Banten +2025-05-21 06:24:11,120: Row 200: Reverse geocoded lat=47.1449, lon=8.1551 -> country=Switzerland, city=Eich, state=Lucerne +2025-05-21 06:24:12,261: Row 201: Reverse geocoded lat=41.6959, lon=44.832 -> country=Georgia, city=Tbilisi, state=None +2025-05-21 06:24:13,784: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=25.0389&lon=102.7183&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:14,128: Row 202: Reverse geocoded lat=25.0389, lon=102.7183 -> country=China, city=Guandu, state=Yunnan +2025-05-21 06:24:15,125: Row 203: Reverse geocoded lat=-23.4553, lon=-46.3348 -> country=Brazil, city=Itaquaquecetuba, state=São Paulo +2025-05-21 06:24:16,063: Row 204: Reverse geocoded lat=19.3808, lon=-99.1982 -> country=Mexico, city=Mexico City, state=Mexico City +2025-05-21 06:24:17,072: Row 206: Reverse geocoded lat=-23.5246, lon=-46.1835 -> country=Brazil, city=Mogi das Cruzes, state=São Paulo +2025-05-21 06:24:18,143: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=59.7531&lon=10.1419&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:19,111: Row 207: Reverse geocoded lat=59.7531, lon=10.1419 -> country=Norway, city=Drammen, state=None +2025-05-21 06:24:20,120: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=46.2253&lon=6.1516&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:21,105: Row 208: Reverse geocoded lat=46.2253, lon=6.1516 -> country=Switzerland, city=Geneva, state=Geneva +2025-05-21 06:24:22,110: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.5042&lon=-0.2199000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:23,207: Row 209: Reverse geocoded lat=51.5042, lon=-0.2199 -> country=United Kingdom, city=London, state=England +2025-05-21 06:24:24,090: Row 210: Reverse geocoded lat=51.5042, lon=-0.2199 -> country=United Kingdom, city=London, state=England +2025-05-21 06:24:25,124: Row 211: Reverse geocoded lat=51.5042, lon=-0.2199 -> country=United Kingdom, city=London, state=England +2025-05-21 06:24:26,220: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.7018&lon=90.3742&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:27,085: Row 212: Reverse geocoded lat=23.7018, lon=90.3742 -> country=Bangladesh, city=Keraniganj, state=Dhaka Division +2025-05-21 06:24:28,068: Row 214: Reverse geocoded lat=34.7732, lon=113.722 -> country=China, city=Jinshui District, state=Henan +2025-05-21 06:24:29,093: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.3917&lon=-0.5099000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:30,169: Row 215: Reverse geocoded lat=51.3917, lon=-0.5099 -> country=United Kingdom, city=Borough of Runnymede, state=England +2025-05-21 06:24:31,174: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=53.3382&lon=-6.2591&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:32,217: Row 216: Reverse geocoded lat=53.3382, lon=-6.2591 -> country=Ireland, city=Dublin, state=None +2025-05-21 06:24:33,108: Row 217: Reverse geocoded lat=50.458, lon=30.5303 -> country=Ukraine, city=Kyiv, state=None +2025-05-21 06:24:34,051: Row 218: Reverse geocoded lat=50.458, lon=30.5303 -> country=Ukraine, city=Kyiv, state=None +2025-05-21 06:24:35,082: Row 219: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:24:36,211: Row 220: Reverse geocoded lat=51.5088, lon=-0.093 -> country=United Kingdom, city=London, state=England +2025-05-21 06:24:37,231: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-30.3299&lon=-54.3251&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:38,102: Row 221: Reverse geocoded lat=-30.3299, lon=-54.3251 -> country=Brazil, city=São Gabriel, state=Rio Grande do Sul +2025-05-21 06:24:39,107: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=36.7906&lon=118.0633&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:40,047: Row 222: Reverse geocoded lat=36.7906, lon=118.0633 -> country=China, city=Zhangdian District, state=Shandong +2025-05-21 06:24:41,051: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.0&lon=70.0&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:42,032: Row 223: Reverse geocoded lat=30, lon=70 -> country=Pakistan, city=Fort Monroe, state=Punjab +2025-05-21 06:24:43,041: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.3388&lon=-121.8916&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:44,063: Row 224: Reverse geocoded lat=37.3388, lon=-121.8916 -> country=United States, city=San Jose, state=California +2025-05-21 06:24:45,039: Row 225: Reverse geocoded lat=25.0734, lon=55.2979 -> country=United Arab Emirates, city=None, state=Dubai +2025-05-21 06:24:46,071: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=34.6808&lon=112.4531&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:47,050: Row 226: Reverse geocoded lat=34.6808, lon=112.4531 -> country=China, city=Luoyang, state=Henan +2025-05-21 06:24:48,057: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=31.9555&lon=35.9435&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:49,125: Row 228: Reverse geocoded lat=31.9555, lon=35.9435 -> country=Jordan, city=None, state=Amman +2025-05-21 06:24:50,043: Row 229: Reverse geocoded lat=31.9555, lon=35.9435 -> country=Jordan, city=None, state=Amman +2025-05-21 06:24:51,032: Row 230: Reverse geocoded lat=31.9555, lon=35.9435 -> country=Jordan, city=None, state=Amman +2025-05-21 06:24:52,058: Row 231: Reverse geocoded lat=31.9555, lon=35.9435 -> country=Jordan, city=None, state=Amman +2025-05-21 06:24:53,109: Row 232: Reverse geocoded lat=30.7488, lon=120.7486 -> country=China, city=Nanhu District, state=Zhejiang +2025-05-21 06:24:54,077: Row 233: Reverse geocoded lat=31.9555, lon=35.9435 -> country=Jordan, city=None, state=Amman +2025-05-21 06:24:55,148: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=13.0895&lon=80.2739&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:56,076: Row 234: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:24:57,080: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-34.9087&lon=-56.1539&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:24:58,126: Row 235: Reverse geocoded lat=-34.9087, lon=-56.1539 -> country=Uruguay, city=Montevideo, state=Montevideo +2025-05-21 06:24:59,082: Row 236: Reverse geocoded lat=38.3169, lon=116.8478 -> country=China, city=Yunhe District, state=Hebei +2025-05-21 06:25:00,075: Row 240: Reverse geocoded lat=22.5643, lon=88.3693 -> country=India, city=Kolkata, state=West Bengal +2025-05-21 06:25:01,094: Row 241: Reverse geocoded lat=22.5643, lon=88.3693 -> country=India, city=Kolkata, state=West Bengal +2025-05-21 06:25:02,097: Row 243: Reverse geocoded lat=39.911, lon=116.395 -> country=China, city=Dongcheng District, state=Beijing +2025-05-21 06:25:03,109: Row 244: Reverse geocoded lat=34.1794, lon=117.1612 -> country=China, city=Tongshan District, state=Jiangsu +2025-05-21 06:25:04,149: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=53.4146&lon=-2.2274&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:05,127: Row 245: Reverse geocoded lat=53.4146, lon=-2.2274 -> country=United Kingdom, city=Manchester, state=England +2025-05-21 06:25:06,129: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.9158&lon=-1.3601&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:07,140: Row 246: Reverse geocoded lat=50.9158, lon=-1.3601 -> country=United Kingdom, city=Southampton, state=England +2025-05-21 06:25:08,145: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.1169&lon=8.6837&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:09,175: Row 247: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:25:10,113: Row 248: Reverse geocoded lat=23.0276, lon=72.5871 -> country=India, city=Ahmedabad, state=Gujarat +2025-05-21 06:25:11,098: Row 250: Reverse geocoded lat=26.9136, lon=75.7858 -> country=India, city=Jaipur, state=Rajasthan +2025-05-21 06:25:12,068: Row 251: Reverse geocoded lat=-34.1801, lon=-58.9181 -> country=Argentina, city=None, state=Buenos Aires +2025-05-21 06:25:13,178: Row 252: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:25:14,161: Row 253: Reverse geocoded lat=31.8696, lon=117.293 -> country=China, city=Luyang District, state=Anhui +2025-05-21 06:25:15,093: Row 254: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:25:16,115: Row 255: Reverse geocoded lat=22.3008, lon=73.2043 -> country=India, city=Vadodara, state=Gujarat +2025-05-21 06:25:17,128: Row 257: Reverse geocoded lat=39.0469, lon=-77.4903 -> country=United States, city=Ashburn, state=Virginia +2025-05-21 06:25:18,210: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=24.8591&lon=66.9983&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:19,043: Row 260: Reverse geocoded lat=24.8591, lon=66.9983 -> country=Pakistan, city=Karachi Division, state=None +2025-05-21 06:25:20,047: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=24.8591&lon=66.9983&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:21,030: Row 261: Reverse geocoded lat=24.8591, lon=66.9983 -> country=Pakistan, city=Karachi Division, state=None +2025-05-21 06:25:22,032: Row 264: Reverse geocoded lat=9.0245, lon=38.7485 -> country=Ethiopia, city=None, state=Addis Ababa +2025-05-21 06:25:23,036: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.5549&lon=13.4645&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:24,105: Row 265: Reverse geocoded lat=52.5549, lon=13.4645 -> country=Germany, city=Berlin, state=None +2025-05-21 06:25:25,060: Row 266: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:25:26,113: Row 267: Reverse geocoded lat=51.5081, lon=-0.1278 -> country=United Kingdom, city=City of Westminster, state=England +2025-05-21 06:25:27,068: Row 268: Reverse geocoded lat=39.911, lon=116.395 -> country=China, city=Dongcheng District, state=Beijing +2025-05-21 06:25:28,127: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=21.1463&lon=79.0849&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:29,083: Row 269: Reverse geocoded lat=21.1463, lon=79.0849 -> country=India, city=Nagpur, state=Maharashtra +2025-05-21 06:25:30,079: Row 270: Reverse geocoded lat=43.0147, lon=-81.3049 -> country=Canada, city=London, state=Ontario +2025-05-21 06:25:31,088: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-33.914&lon=18.4129&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:32,098: Row 271: Reverse geocoded lat=-33.914, lon=18.4129 -> country=South Africa, city=Cape Town, state=Western Cape +2025-05-21 06:25:33,102: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=41.5053&lon=-81.5566&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:25:34,193: Row 272: Reverse geocoded lat=41.5053, lon=-81.5566 -> country=United States, city=Cleveland Heights, state=Ohio +2025-05-21 06:25:35,011: Row 273: Reverse geocoded lat=22.3008, lon=73.2043 -> country=India, city=Vadodara, state=Gujarat +2025-05-21 06:25:36,116: Row 274: Reverse geocoded lat=-27.6157, lon=-48.6235 -> country=Brazil, city=São José, state=Santa Catarina +2025-05-21 06:25:37,119: Row 275: Reverse geocoded lat=40.4422, lon=-79.9927 -> country=United States, city=Pittsburgh, state=Pennsylvania +2025-05-21 06:25:38,061: Row 276: Reverse geocoded lat=25.5943, lon=85.1352 -> country=India, city=Patna, state=Bihar +2025-05-21 06:25:39,210: Row 277: Reverse geocoded lat=-36.8322, lon=174.7898 -> country=New Zealand, city=Devonport-Takapuna, state=Auckland +2025-05-21 06:25:40,165: Row 278: Reverse geocoded lat=35.7204, lon=-0.6659 -> country=Algeria, city=Oran, state=Oran +2025-05-21 06:25:41,206: Row 279: Reverse geocoded lat=41.4765, lon=1.9278 -> country=Spain, city=Martorell, state=Catalonia +2025-05-21 06:25:42,175: Row 281: Reverse geocoded lat=21, lon=96 -> country=Myanmar, city=Wet Toe, state=Mandalay +2025-05-21 06:25:43,213: Row 282: Reverse geocoded lat=14.6475, lon=121.0494 -> country=Philippines, city=Quezon City, state=None +2025-05-21 06:25:44,084: Row 283: Reverse geocoded lat=14.6475, lon=121.0494 -> country=Philippines, city=Quezon City, state=None +2025-05-21 06:25:45,232: Row 284: Reverse geocoded lat=59.4786, lon=18.2931 -> country=Sweden, city=Åkersberga, state=None +2025-05-21 06:25:46,462: Row 285: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:25:47,405: Row 287: Reverse geocoded lat=35.4711, lon=44.3925 -> country=Iraq, city=Kirkuk, state=Kirkuk Governorate +2025-05-21 06:25:48,051: Row 288: Reverse geocoded lat=33.8333, lon=35.8333 -> country=Lebanon, city=Jdita, state=Beqaa Governorate +2025-05-21 06:25:49,219: Row 289: Reverse geocoded lat=48.0295, lon=7.573 -> country=France, city=Biesheim, state=Grand Est +2025-05-21 06:25:50,104: Row 290: Reverse geocoded lat=22.5643, lon=88.3693 -> country=India, city=Kolkata, state=West Bengal +2025-05-21 06:25:51,173: Row 291: Reverse geocoded lat=22.5643, lon=88.3693 -> country=India, city=Kolkata, state=West Bengal +2025-05-21 06:25:52,465: Row 292: Reverse geocoded lat=51.2192, lon=4.3917 -> country=Belgium, city=Antwerp, state=Antwerp +2025-05-21 06:25:53,220: Row 294: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:25:54,345: Row 295: Reverse geocoded lat=50.8397, lon=12.9358 -> country=Germany, city=Chemnitz, state=Saxony +2025-05-21 06:25:55,064: Row 296: Reverse geocoded lat=-26.2309, lon=28.0583 -> country=South Africa, city=Johannesburg, state=Gauteng +2025-05-21 06:25:56,189: Row 297: Reverse geocoded lat=47.4121, lon=8.0864 -> country=Switzerland, city=Biberstein, state=Aargau +2025-05-21 06:25:57,113: Row 299: Reverse geocoded lat=50.458, lon=30.5303 -> country=Ukraine, city=Kyiv, state=None +2025-05-21 06:25:58,134: Row 300: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:25:59,281: Row 301: Reverse geocoded lat=30.7096, lon=31.2554 -> country=Egypt, city=Mit Ghamr City, state=Ad Dakahliya +2025-05-21 06:26:00,079: Row 302: Reverse geocoded lat=26.8373, lon=80.9165 -> country=India, city=Lucknow, state=Uttar Pradesh +2025-05-21 06:26:01,121: Row 303: Reverse geocoded lat=-22.9035, lon=-47.0565 -> country=Brazil, city=Campinas, state=São Paulo +2025-05-21 06:26:02,232: Row 304: Reverse geocoded lat=44.1145, lon=24.3402 -> country=Romania, city=Caracal, state=None +2025-05-21 06:26:03,072: Row 305: Reverse geocoded lat=41.8835, lon=-87.6305 -> country=United States, city=Chicago, state=Illinois +2025-05-21 06:26:04,061: Row 306: Reverse geocoded lat=49.2635, lon=-122.9331 -> country=Canada, city=Burnaby, state=British Columbia +2025-05-21 06:26:05,101: Row 307: Reverse geocoded lat=36.5312, lon=3.0698 -> country=Algeria, city=Bougara, state=Blida +2025-05-21 06:26:06,600: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=49.6113&lon=6.1294&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:07,249: Row 309: Reverse geocoded lat=49.6113, lon=6.1294 -> country=Luxembourg, city=Luxembourg, state=None +2025-05-21 06:26:08,163: Row 310: Reverse geocoded lat=0.321, lon=32.5714 -> country=Uganda, city=Kampala, state=Central Region +2025-05-21 06:26:09,135: Row 311: Reverse geocoded lat=-29.9434, lon=-50.9934 -> country=Brazil, city=Gravataí, state=Rio Grande do Sul +2025-05-21 06:26:10,215: Row 312: Reverse geocoded lat=23.0276, lon=72.5871 -> country=India, city=Ahmedabad, state=Gujarat +2025-05-21 06:26:11,056: Row 313: Reverse geocoded lat=-7.2333, lon=-62.9 -> country=Brazil, city=Humaitá, state=Amazonas +2025-05-21 06:26:12,118: Row 314: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:26:13,119: Row 315: Reverse geocoded lat=-16.7267, lon=-49.1091 -> country=Brazil, city=Senador Canedo, state=Goiás +2025-05-21 06:26:14,275: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.4466&lon=5.4736&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:15,135: Row 316: Reverse geocoded lat=51.4466, lon=5.4736 -> country=Netherlands, city=Eindhoven, state=North Brabant +2025-05-21 06:26:16,140: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=3.8661&lon=11.5154&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:17,081: Row 317: Reverse geocoded lat=3.8661, lon=11.5154 -> country=Cameroon, city=Yaoundé, state=Centre +2025-05-21 06:26:18,086: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=3.8661&lon=11.5154&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:19,033: Row 318: Reverse geocoded lat=3.8661, lon=11.5154 -> country=Cameroon, city=Yaoundé, state=Centre +2025-05-21 06:26:20,038: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=31.7126&lon=76.9328&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:21,081: Row 319: Reverse geocoded lat=31.7126, lon=76.9328 -> country=India, city=Mandi, state=Himachal Pradesh +2025-05-21 06:26:22,085: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=47.5534&lon=7.9519&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:23,182: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=47.5534&lon=7.9519&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:24,156: Row 320: Reverse geocoded lat=47.5534, lon=7.9519 -> country=Germany, city=Säckingen, state=Baden-Württemberg +2025-05-21 06:26:25,165: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.0276&lon=72.5871&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:26,089: Row 321: Reverse geocoded lat=23.0276, lon=72.5871 -> country=India, city=Ahmedabad, state=Gujarat +2025-05-21 06:26:27,095: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.0848&lon=14.4112&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:28,135: Row 322: Reverse geocoded lat=50.0848, lon=14.4112 -> country=Czechia, city=Capital City of Prague, state=Prague +2025-05-21 06:26:29,141: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.1477&lon=-3.5655&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:30,020: Row 323: Reverse geocoded lat=37.1477, lon=-3.5655 -> country=Spain, city=Huétor Vega, state=Andalusia +2025-05-21 06:26:31,024: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=46.0025&lon=8.9533&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:32,099: Row 325: Reverse geocoded lat=46.0025, lon=8.9533 -> country=Switzerland, city=Lugano, state=Ticino +2025-05-21 06:26:33,104: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=12.3729&lon=-1.5264&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:34,075: Row 326: Reverse geocoded lat=12.3729, lon=-1.5264 -> country=Burkina Faso, city=Ouagadougou, state=None +2025-05-21 06:26:35,078: Row 327: Reverse geocoded lat=35.6893, lon=139.6899 -> country=Japan, city=Shinjuku, state=None +2025-05-21 06:26:36,079: Row 328: Reverse geocoded lat=37.9842, lon=23.7353 -> country=Greece, city=Athens, state=Attica +2025-05-21 06:26:37,067: Row 329: Reverse geocoded lat=54.8636, lon=-7.7557 -> country=Ireland, city=None, state=None +2025-05-21 06:26:38,083: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=34.8305&lon=-87.656&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:39,123: Row 330: Reverse geocoded lat=34.8305, lon=-87.656 -> country=United States, city=Florence, state=Alabama +2025-05-21 06:26:40,128: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.8978&lon=-77.2885&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:41,231: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.8978&lon=-77.2885&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:42,087: Row 331: Reverse geocoded lat=38.8978, lon=-77.2885 -> country=United States, city=Reston, state=Virginia +2025-05-21 06:26:43,091: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=47.5&lon=19.0412&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:44,067: Row 332: Reverse geocoded lat=47.5, lon=19.0412 -> country=Hungary, city=Budapest, state=None +2025-05-21 06:26:45,072: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=21.4668&lon=83.9764&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:46,159: Row 333: Reverse geocoded lat=21.4668, lon=83.9764 -> country=India, city=Sambalpur, state=Odisha +2025-05-21 06:26:47,170: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.808&lon=-2.1118&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:48,040: Row 334: Reverse geocoded lat=52.808, lon=-2.1118 -> country=United Kingdom, city=Stafford, state=England +2025-05-21 06:26:49,050: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-22.2635&lon=-49.1364&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:50,033: Row 335: Reverse geocoded lat=-22.2635, lon=-49.1364 -> country=Brazil, city=Bauru, state=São Paulo +2025-05-21 06:26:51,039: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=40.8449&lon=14.2716&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:52,041: Row 336: Reverse geocoded lat=40.8449, lon=14.2716 -> country=Italy, city=Naples, state=Campania +2025-05-21 06:26:53,045: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=13.6687&lon=100.579&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:54,087: Row 337: Reverse geocoded lat=13.6687, lon=100.579 -> country=Thailand, city=Pu Chao Saming Phrai Town Municipality, state=None +2025-05-21 06:26:55,092: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=18.5211&lon=73.8502&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:56,061: Row 338: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:26:57,070: Row 339: Reverse geocoded lat=23.7018, lon=90.3742 -> country=Bangladesh, city=Keraniganj, state=Dhaka Division +2025-05-21 06:26:58,077: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.5112&lon=0.0300000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:26:59,110: Row 340: Reverse geocoded lat=51.5112, lon=0.03 -> country=United Kingdom, city=London, state=England +2025-05-21 06:27:00,093: Row 341: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:27:01,039: Row 342: Reverse geocoded lat=-7.2333, lon=-62.9 -> country=Brazil, city=Humaitá, state=Amazonas +2025-05-21 06:27:02,085: Row 343: Reverse geocoded lat=40.7547, lon=-73.9614 -> country=United States, city=New York, state=New York +2025-05-21 06:27:03,124: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.458&lon=30.5303&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:04,124: Row 344: Reverse geocoded lat=50.458, lon=30.5303 -> country=Ukraine, city=Kyiv, state=None +2025-05-21 06:27:05,122: Row 347: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:27:06,130: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=11.0027&lon=124.6083&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:07,107: Row 348: Reverse geocoded lat=11.0027, lon=124.6083 -> country=Philippines, city=Ormoc, state=Leyte +2025-05-21 06:27:08,086: Row 349: Reverse geocoded lat=22.717, lon=75.8337 -> country=India, city=Indore, state=Madhya Pradesh +2025-05-21 06:27:09,050: Row 350: Reverse geocoded lat=22.717, lon=75.8337 -> country=India, city=Indore, state=Madhya Pradesh +2025-05-21 06:27:10,056: Row 351: Reverse geocoded lat=7.0712, lon=125.6089 -> country=Philippines, city=Davao City, state=None +2025-05-21 06:27:11,126: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-4.7776&lon=11.8659&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:12,027: Row 352: Reverse geocoded lat=-4.7776, lon=11.8659 -> country=Congo-Brazzaville, city=None, state=Pointe-Noire (département) +2025-05-21 06:27:13,038: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=19.0748&lon=72.8856&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:14,086: Row 353: Reverse geocoded lat=19.0748, lon=72.8856 -> country=India, city=Mumbai, state=Maharashtra +2025-05-21 06:27:15,092: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=34.6874&lon=33.0366&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:16,128: Row 354: Reverse geocoded lat=34.6874, lon=33.0366 -> country=Cyprus, city=Limassol, state=Cyprus +2025-05-21 06:27:17,066: Row 355: Reverse geocoded lat=34.6874, lon=33.0366 -> country=Cyprus, city=Limassol, state=Cyprus +2025-05-21 06:27:18,138: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=46.1953&lon=6.1385&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:19,053: Row 356: Reverse geocoded lat=46.1953, lon=6.1385 -> country=Switzerland, city=Geneva, state=Geneva +2025-05-21 06:27:20,057: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.8951&lon=-1.1585&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:21,045: Row 357: Reverse geocoded lat=51.8951, lon=-1.1585 -> country=United Kingdom, city=Cherwell District, state=England +2025-05-21 06:27:22,055: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.8951&lon=-1.1585&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:23,039: Row 358: Reverse geocoded lat=51.8951, lon=-1.1585 -> country=United Kingdom, city=Cherwell District, state=England +2025-05-21 06:27:24,046: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=29.3706&lon=47.9697&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:25,074: Row 359: Reverse geocoded lat=29.3706, lon=47.9697 -> country=Kuwait, city=Kuwait City, state=Capital Governorate +2025-05-21 06:27:26,078: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.3177&lon=-122.0438&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:27,058: Row 360: Reverse geocoded lat=37.3177, lon=-122.0438 -> country=United States, city=Cupertino, state=California +2025-05-21 06:27:28,064: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.9609&lon=-77.3429&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:29,066: Row 361: Reverse geocoded lat=38.9609, lon=-77.3429 -> country=United States, city=Reston, state=Virginia +2025-05-21 06:27:30,072: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.1613&lon=-113.9518&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:31,126: Row 363: Reverse geocoded lat=51.1613, lon=-113.9518 -> country=Canada, city=Calgary, state=Alberta +2025-05-21 06:27:32,130: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.036&lon=114.4654&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:33,052: Row 364: Reverse geocoded lat=38.036, lon=114.4654 -> country=China, city=Shijiazhuang, state=Hebei +2025-05-21 06:27:34,057: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=18.0&lon=105.0&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:35,079: Row 365: Reverse geocoded lat=18, lon=105 -> country=Laos, city=Phonkeo, state=Bolikhamsai +2025-05-21 06:27:36,084: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=36.096&lon=114.3828&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:37,120: Row 366: Reverse geocoded lat=36.096, lon=114.3828 -> country=China, city=开发区银杏大街街道, state=Henan +2025-05-21 06:27:38,126: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=39.6933&lon=-104.7865&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:39,074: Row 367: Reverse geocoded lat=39.6933, lon=-104.7865 -> country=United States, city=Aurora, state=Colorado +2025-05-21 06:27:40,083: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.7219&lon=-9.1398&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:41,149: Row 368: Reverse geocoded lat=38.7219, lon=-9.1398 -> country=Portugal, city=Lisbon, state=None +2025-05-21 06:27:42,154: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=17.3843&lon=78.4583&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:43,053: Row 369: Reverse geocoded lat=17.3843, lon=78.4583 -> country=India, city=Hyderabad, state=Telangana +2025-05-21 06:27:44,058: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=40.8273&lon=-83.2813&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:45,126: Row 370: Reverse geocoded lat=40.8273, lon=-83.2813 -> country=United States, city=Upper Sandusky, state=Ohio +2025-05-21 06:27:46,132: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=31.2222&lon=121.4581&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:47,091: Row 371: Reverse geocoded lat=31.2222, lon=121.4581 -> country=China, city=Shanghai, state=Shanghai +2025-05-21 06:27:48,104: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.6542&lon=77.2373&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:49,036: Row 372: Reverse geocoded lat=28.6542, lon=77.2373 -> country=India, city=Delhi, state=Delhi +2025-05-21 06:27:50,026: Row 373: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:27:51,048: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=13.0895&lon=80.2739&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:52,072: Row 374: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:27:53,080: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.5&lon=-10.0&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:54,053: Row 375: Reverse geocoded lat=28.5, lon=-10 -> country=Morocco, city=Aouint Lahna عوينة لهنا, state=None +2025-05-21 06:27:55,059: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=45.0729&lon=7.689&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:56,182: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=45.0729&lon=7.689&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:27:57,227: Row 377: Reverse geocoded lat=45.0729, lon=7.689 -> country=Italy, city=Turin, state=Piedmont +2025-05-21 06:27:58,146: Row 378: Reverse geocoded lat=21.4668, lon=83.9764 -> country=India, city=Sambalpur, state=Odisha +2025-05-21 06:27:59,096: Row 379: Reverse geocoded lat=33.1976, lon=-96.6178 -> country=United States, city=McKinney, state=Texas +2025-05-21 06:28:00,242: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=40.7126&lon=-74.0066&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:01,224: Row 380: Reverse geocoded lat=40.7126, lon=-74.0066 -> country=United States, city=New York, state=New York +2025-05-21 06:28:02,142: Row 381: Reverse geocoded lat=33.4306, lon=-111.9256 -> country=United States, city=Tempe, state=Arizona +2025-05-21 06:28:03,102: Row 382: Reverse geocoded lat=38.7883, lon=-121.2366 -> country=United States, city=Rocklin, state=California +2025-05-21 06:28:04,072: Row 383: Reverse geocoded lat=22.5643, lon=88.3693 -> country=India, city=Kolkata, state=West Bengal +2025-05-21 06:28:05,222: Row 384: Reverse geocoded lat=51.1876, lon=71.4491 -> country=Kazakhstan, city=Astana, state=Astana +2025-05-21 06:28:06,135: Row 385: Reverse geocoded lat=22.3008, lon=73.2043 -> country=India, city=Vadodara, state=Gujarat +2025-05-21 06:28:07,247: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=35.6893&lon=139.6899&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:08,080: Row 387: Reverse geocoded lat=35.6893, lon=139.6899 -> country=Japan, city=Shinjuku, state=None +2025-05-21 06:28:09,087: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=35.6893&lon=139.6899&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:10,055: Row 388: Reverse geocoded lat=35.6893, lon=139.6899 -> country=Japan, city=Shinjuku, state=None +2025-05-21 06:28:11,060: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=35.6893&lon=139.6899&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:12,078: Row 389: Reverse geocoded lat=35.6893, lon=139.6899 -> country=Japan, city=Shinjuku, state=None +2025-05-21 06:28:13,083: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=35.6893&lon=139.6899&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:14,083: Row 390: Reverse geocoded lat=35.6893, lon=139.6899 -> country=Japan, city=Shinjuku, state=None +2025-05-21 06:28:15,090: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=3.1023&lon=101.6265&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:16,171: Row 391: Reverse geocoded lat=3.1023, lon=101.6265 -> country=Malaysia, city=Petaling Jaya, state=Selangor +2025-05-21 06:28:17,178: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.3392&lon=-77.4175&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:18,063: Row 392: Reverse geocoded lat=37.3392, lon=-77.4175 -> country=United States, city=None, state=Virginia +2025-05-21 06:28:19,067: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=21.2339&lon=110.3875&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:20,071: Row 394: Reverse geocoded lat=21.2339, lon=110.3875 -> country=China, city=Chikan District, state=Guangdong Province +2025-05-21 06:28:21,080: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=43.3699&lon=-8.4096&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:22,211: Row 395: Reverse geocoded lat=43.3699, lon=-8.4096 -> country=Spain, city=A Coruña, state=Galicia +2025-05-21 06:28:23,221: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=43.3699&lon=-8.4096&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:24,158: Row 396: Reverse geocoded lat=43.3699, lon=-8.4096 -> country=Spain, city=A Coruña, state=Galicia +2025-05-21 06:28:25,078: Row 398: Reverse geocoded lat=42.2645, lon=-88.0034 -> country=United States, city=Mundelein, state=Illinois +2025-05-21 06:28:26,034: Row 399: Reverse geocoded lat=-1.4428, lon=-79.4627 -> country=Ecuador, city=Ventanas, state=Los Ríos +2025-05-21 06:28:27,126: Row 400: Reverse geocoded lat=-1.4428, lon=-79.4627 -> country=Ecuador, city=Ventanas, state=Los Ríos +2025-05-21 06:28:28,174: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.0507&lon=31.2489&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:29,174: Row 401: Reverse geocoded lat=30.0507, lon=31.2489 -> country=Egypt, city=Cairo, state=Cairo +2025-05-21 06:28:30,179: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-33.914&lon=18.4129&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:31,130: Row 403: Reverse geocoded lat=-33.914, lon=18.4129 -> country=South Africa, city=Cape Town, state=Western Cape +2025-05-21 06:28:32,139: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.4597&lon=77.0282&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:33,081: Row 406: Reverse geocoded lat=28.4597, lon=77.0282 -> country=India, city=Gurgaon, state=Haryana +2025-05-21 06:28:34,082: Row 407: Reverse geocoded lat=37.3388, lon=-121.8916 -> country=United States, city=San Jose, state=California +2025-05-21 06:28:35,092: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.1181&lon=113.2539&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:36,089: Row 408: Reverse geocoded lat=23.1181, lon=113.2539 -> country=China, city=Yuexiu District, state=Guangdong Province +2025-05-21 06:28:37,093: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=49.3518&lon=0.8088000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:38,131: Row 409: Reverse geocoded lat=49.3518, lon=0.8088 -> country=France, city=Bourg-Achard, state=Normandy +2025-05-21 06:28:39,141: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=25.315&lon=83.0058&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:40,072: Row 410: Reverse geocoded lat=25.315, lon=83.0058 -> country=India, city=Varanasi, state=Uttar Pradesh +2025-05-21 06:28:41,076: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.3275&lon=78.0325&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:42,079: Row 411: Reverse geocoded lat=30.3275, lon=78.0325 -> country=India, city=Dehradun, state=Uttarakhand +2025-05-21 06:28:43,089: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=11.9924&lon=8.5173&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:44,125: Row 412: Reverse geocoded lat=11.9924, lon=8.5173 -> country=Nigeria, city=Tudun Wazurchi, state=Kano State +2025-05-21 06:28:45,133: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=25.0734&lon=55.2979&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:46,101: Row 413: Reverse geocoded lat=25.0734, lon=55.2979 -> country=United Arab Emirates, city=None, state=Dubai +2025-05-21 06:28:47,057: Row 414: Reverse geocoded lat=31.9555, lon=35.9435 -> country=Jordan, city=None, state=Amman +2025-05-21 06:28:48,050: Row 415: Reverse geocoded lat=25.0734, lon=55.2979 -> country=United Arab Emirates, city=None, state=Dubai +2025-05-21 06:28:49,113: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.6542&lon=77.2373&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:50,092: Row 416: Reverse geocoded lat=28.6542, lon=77.2373 -> country=India, city=Delhi, state=Delhi +2025-05-21 06:28:51,096: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=41.8835&lon=-87.6305&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:52,049: Row 417: Reverse geocoded lat=41.8835, lon=-87.6305 -> country=United States, city=Chicago, state=Illinois +2025-05-21 06:28:53,056: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.8707&lon=120.0898&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:54,132: Row 418: Reverse geocoded lat=30.8707, lon=120.0898 -> country=China, city=Wuxing District, state=Zhejiang +2025-05-21 06:28:55,135: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.1302&lon=13.329&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:56,025: Row 419: Reverse geocoded lat=38.1302, lon=13.329 -> country=Italy, city=Palermo, state=Sicily +2025-05-21 06:28:57,030: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.5852&lon=-0.2360000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:28:58,078: Row 420: Reverse geocoded lat=52.5852, lon=-0.236 -> country=United Kingdom, city=Peterborough, state=England +2025-05-21 06:28:59,082: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.4073&lon=-121.939&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:00,103: Row 421: Reverse geocoded lat=37.4073, lon=-121.939 -> country=United States, city=San Jose, state=California +2025-05-21 06:29:01,100: Row 422: Reverse geocoded lat=37.4073, lon=-121.939 -> country=United States, city=San Jose, state=California +2025-05-21 06:29:02,114: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=55.6922&lon=12.5478&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:03,221: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=55.6922&lon=12.5478&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:04,095: Row 423: Reverse geocoded lat=55.6922, lon=12.5478 -> country=Denmark, city=Copenhagen, state=Capital Region of Denmark +2025-05-21 06:29:05,099: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=25.5943&lon=85.1352&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:06,049: Row 425: Reverse geocoded lat=25.5943, lon=85.1352 -> country=India, city=Patna, state=Bihar +2025-05-21 06:29:07,058: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.5064&lon=-0.0200000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:08,080: Row 427: Reverse geocoded lat=51.5064, lon=-0.02 -> country=United Kingdom, city=London, state=England +2025-05-21 06:29:09,085: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=56.1567&lon=10.2153&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:10,051: Row 428: Reverse geocoded lat=56.1567, lon=10.2153 -> country=Denmark, city=Aarhus, state=Central Denmark Region +2025-05-21 06:29:11,061: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=45.5547&lon=5.9801&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:12,079: Row 429: Reverse geocoded lat=45.5547, lon=5.9801 -> country=France, city=Challes-les-Eaux, state=Auvergne-Rhône-Alpes +2025-05-21 06:29:13,085: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-19.9218&lon=-43.9359&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:14,028: Row 432: Reverse geocoded lat=-19.9218, lon=-43.9359 -> country=Brazil, city=Belo Horizonte, state=Minas Gerais +2025-05-21 06:29:15,033: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=47.615&lon=-122.1619&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:16,027: Row 433: Reverse geocoded lat=47.615, lon=-122.1619 -> country=United States, city=Bellevue, state=Washington +2025-05-21 06:29:17,033: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.8707&lon=120.0898&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:18,075: Row 436: Reverse geocoded lat=30.8707, lon=120.0898 -> country=China, city=Wuxing District, state=Zhejiang +2025-05-21 06:29:19,080: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=59.4381&lon=24.7369&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:20,133: Row 437: Reverse geocoded lat=59.4381, lon=24.7369 -> country=Estonia, city=Tallinn, state=None +2025-05-21 06:29:21,144: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=26.8373&lon=80.9165&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:22,051: Row 438: Reverse geocoded lat=26.8373, lon=80.9165 -> country=India, city=Lucknow, state=Uttar Pradesh +2025-05-21 06:29:23,055: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=26.8373&lon=80.9165&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:24,064: Row 439: Reverse geocoded lat=26.8373, lon=80.9165 -> country=India, city=Lucknow, state=Uttar Pradesh +2025-05-21 06:29:25,068: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=18.5211&lon=73.8502&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:26,114: Row 440: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:29:27,120: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.9439&lon=5.8939&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:28,100: Row 441: Reverse geocoded lat=51.9439, lon=5.8939 -> country=Netherlands, city=Arnhem, state=Gelderland +2025-05-21 06:29:29,105: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=18.7857&lon=98.9736&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:30,104: Row 442: Reverse geocoded lat=18.7857, lon=98.9736 -> country=Thailand, city=Chiang Mai City Municipality, state=None +2025-05-21 06:29:31,108: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=20.4636&lon=85.8783&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:32,202: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=20.4636&lon=85.8783&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:33,025: Row 443: Reverse geocoded lat=20.4636, lon=85.8783 -> country=India, city=Cuttack, state=Odisha +2025-05-21 06:29:34,029: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=46.2253&lon=6.1516&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:35,072: Row 444: Reverse geocoded lat=46.2253, lon=6.1516 -> country=Switzerland, city=Geneva, state=Geneva +2025-05-21 06:29:36,076: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.6667&lon=104.0667&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:37,169: Row 446: Reverse geocoded lat=30.6667, lon=104.0667 -> country=China, city=Chengdu, state=Sichuan +2025-05-21 06:29:38,174: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.1169&lon=8.6837&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:39,115: Row 447: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:29:40,041: Row 448: Reverse geocoded lat=26.3304, lon=-81.5871 -> country=United States, city=None, state=Florida +2025-05-21 06:29:41,056: Row 449: Reverse geocoded lat=-3.717, lon=-38.5287 -> country=Brazil, city=Fortaleza, state=Ceará +2025-05-21 06:29:42,047: Row 450: Reverse geocoded lat=55.7481, lon=12.5347 -> country=Denmark, city=Jægersborg, state=Capital Region of Denmark +2025-05-21 06:29:43,132: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.5625&lon=-122.0004&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:44,023: Row 451: Reverse geocoded lat=37.5625, lon=-122.0004 -> country=United States, city=Fremont, state=California +2025-05-21 06:29:45,027: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-12.0432&lon=-77.0282&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:46,105: Row 452: Reverse geocoded lat=-12.0432, lon=-77.0282 -> country=Peru, city=Lima, state=Lima +2025-05-21 06:29:47,111: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=59.3287&lon=18.0717&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:48,064: Row 453: Reverse geocoded lat=59.3287, lon=18.0717 -> country=Sweden, city=Stockholm, state=None +2025-05-21 06:29:49,045: Row 454: Reverse geocoded lat=59.3287, lon=18.0717 -> country=Sweden, city=Stockholm, state=None +2025-05-21 06:29:50,074: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.1922&lon=20.8846&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:51,089: Row 456: Reverse geocoded lat=52.1922, lon=20.8846 -> country=Poland, city=Warsaw, state=Masovian Voivodeship +2025-05-21 06:29:52,059: Row 457: Reverse geocoded lat=40.4153, lon=-3.694 -> country=Spain, city=Madrid, state=Community of Madrid +2025-05-21 06:29:53,098: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=40.4153&lon=-3.694&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:54,167: Row 458: Reverse geocoded lat=40.4153, lon=-3.694 -> country=Spain, city=Madrid, state=Community of Madrid +2025-05-21 06:29:55,171: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=59.3247&lon=18.056&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:56,113: Row 459: Reverse geocoded lat=59.3247, lon=18.056 -> country=Sweden, city=Stockholm, state=None +2025-05-21 06:29:57,067: Row 460: Reverse geocoded lat=59.3247, lon=18.056 -> country=Sweden, city=Stockholm, state=None +2025-05-21 06:29:58,118: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.0596&lon=19.9338&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:29:59,092: Row 461: Reverse geocoded lat=50.0596, lon=19.9338 -> country=Poland, city=Krakow, state=Lesser Poland Voivodeship +2025-05-21 06:30:00,095: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.3924&lon=16.9019&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:01,130: Row 462: Reverse geocoded lat=52.3924, lon=16.9019 -> country=Poland, city=Poznan, state=Greater Poland Voivodeship +2025-05-21 06:30:02,136: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=59.3287&lon=18.0717&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:03,111: Row 463: Reverse geocoded lat=59.3287, lon=18.0717 -> country=Sweden, city=Stockholm, state=None +2025-05-21 06:30:04,114: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.7599&lon=6.0953&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:05,112: Row 465: Reverse geocoded lat=50.7599, lon=6.0953 -> country=Germany, city=Aachen, state=North Rhine-Westphalia +2025-05-21 06:30:06,118: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=60.3951&lon=5.3237&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:07,090: Row 466: Reverse geocoded lat=60.3951, lon=5.3237 -> country=Norway, city=Bergen, state=None +2025-05-21 06:30:08,096: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=60.3951&lon=5.3237&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:09,200: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=60.3951&lon=5.3237&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:10,061: Row 467: Reverse geocoded lat=60.3951, lon=5.3237 -> country=Norway, city=Bergen, state=None +2025-05-21 06:30:11,055: Row 468: Reverse geocoded lat=60.3951, lon=5.3237 -> country=Norway, city=Bergen, state=None +2025-05-21 06:30:12,068: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.3759&lon=4.8975&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:13,176: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.3759&lon=4.8975&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:14,156: Row 469: Reverse geocoded lat=52.3759, lon=4.8975 -> country=Netherlands, city=Amsterdam, state=North Holland +2025-05-21 06:30:15,095: Row 470: Reverse geocoded lat=-25.5283, lon=-49.2161 -> country=Brazil, city=São José dos Pinhais, state=Paraná +2025-05-21 06:30:16,166: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=47.0042&lon=28.8574&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:17,106: Row 471: Reverse geocoded lat=47.0042, lon=28.8574 -> country=Moldova, city=Chișinău, state=None +2025-05-21 06:30:18,110: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-3.1032&lon=-60.0288&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:19,094: Row 473: Reverse geocoded lat=-3.1032, lon=-60.0288 -> country=Brazil, city=Manaus, state=Amazonas +2025-05-21 06:30:20,098: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=12.9753&lon=77.591&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:21,083: Row 474: Reverse geocoded lat=12.9753, lon=77.591 -> country=India, city=Bengaluru, state=Karnataka +2025-05-21 06:30:22,086: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=54.5455&lon=18.4494&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:23,100: Row 476: Reverse geocoded lat=54.5455, lon=18.4494 -> country=Poland, city=Gdynia, state=Pomeranian Voivodeship +2025-05-21 06:30:24,104: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.7219&lon=-9.1398&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:25,105: Row 477: Reverse geocoded lat=38.7219, lon=-9.1398 -> country=Portugal, city=Lisbon, state=None +2025-05-21 06:30:26,109: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=41.3946&lon=2.1406&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:27,155: Row 478: Reverse geocoded lat=41.3946, lon=2.1406 -> country=Spain, city=Barcelona, state=Catalonia +2025-05-21 06:30:28,105: Row 479: Reverse geocoded lat=-27.4683, lon=153.0322 -> country=Australia, city=None, state=Queensland +2025-05-21 06:30:29,165: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=45.7537&lon=21.2257&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:30,276: Row 480: Reverse geocoded lat=45.7537, lon=21.2257 -> country=Romania, city=Timișoara, state=None +2025-05-21 06:30:31,158: Row 481: Reverse geocoded lat=53.7606, lon=-0.3734 -> country=United Kingdom, city=Hull, state=England +2025-05-21 06:30:32,164: Row 482: Reverse geocoded lat=52.3759, lon=4.8975 -> country=Netherlands, city=Amsterdam, state=North Holland +2025-05-21 06:30:33,286: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.3759&lon=4.8975&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:34,191: Row 483: Reverse geocoded lat=52.3759, lon=4.8975 -> country=Netherlands, city=Amsterdam, state=North Holland +2025-05-21 06:30:35,102: Row 484: Reverse geocoded lat=31.3709, lon=73.0336 -> country=Pakistan, city=Faisalabad City Tehsil, state=Punjab +2025-05-21 06:30:36,097: Row 485: Reverse geocoded lat=-26.2309, lon=28.0583 -> country=South Africa, city=Johannesburg, state=Gauteng +2025-05-21 06:30:37,127: Row 486: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:30:38,115: Row 487: Reverse geocoded lat=12.9753, lon=77.591 -> country=India, city=Bengaluru, state=Karnataka +2025-05-21 06:30:39,209: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.1836&lon=11.5754&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:40,049: Row 488: Reverse geocoded lat=48.1836, lon=11.5754 -> country=Germany, city=Munich, state=Bavaria +2025-05-21 06:30:41,054: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=54.1085&lon=12.0645&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:42,050: Row 489: Reverse geocoded lat=54.1085, lon=12.0645 -> country=Germany, city=Rostock, state=Mecklenburg-Vorpommern +2025-05-21 06:30:43,056: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.8327&lon=-9.166&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:44,122: Row 490: Reverse geocoded lat=38.8327, lon=-9.166 -> country=Portugal, city=Loures, state=None +2025-05-21 06:30:45,127: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=56.9473&lon=24.0979&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:46,055: Row 491: Reverse geocoded lat=56.9473, lon=24.0979 -> country=Latvia, city=Riga, state=None +2025-05-21 06:30:47,057: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=29.9192&lon=73.8741&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:48,167: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=29.9192&lon=73.8741&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:49,080: Row 494: Reverse geocoded lat=29.9192, lon=73.8741 -> country=India, city=Ganganagar, state=Rajasthan +2025-05-21 06:30:50,085: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.1169&lon=8.6837&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:51,090: Row 495: Reverse geocoded lat=50.1169, lon=8.6837 -> country=Germany, city=Frankfurt, state=Hesse +2025-05-21 06:30:52,043: Row 496: Reverse geocoded lat=-29.8553, lon=31.0428 -> country=South Africa, city=Durban, state=KwaZulu-Natal +2025-05-21 06:30:53,095: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=29.5689&lon=106.5577&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:54,073: Row 498: Reverse geocoded lat=29.5689, lon=106.5577 -> country=China, city=Yuzhong District, state=Chongqing +2025-05-21 06:30:55,048: Row 500: Reverse geocoded lat=22.5643, lon=88.3693 -> country=India, city=Kolkata, state=West Bengal +2025-05-21 06:30:56,082: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=53.0582&lon=8.7868&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:57,050: Row 501: Reverse geocoded lat=53.0582, lon=8.7868 -> country=Germany, city=Bremen, state=Bremen +2025-05-21 06:30:58,053: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=19.0748&lon=72.8856&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:30:59,082: Row 502: Reverse geocoded lat=19.0748, lon=72.8856 -> country=India, city=Mumbai, state=Maharashtra +2025-05-21 06:31:00,085: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=35.0&lon=105.0&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:01,139: Row 503: Reverse geocoded lat=35, lon=105 -> country=China, city=Bangluo, state=Gansu +2025-05-21 06:31:02,145: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.4537&lon=-0.2320000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:03,054: Row 505: Reverse geocoded lat=51.4537, lon=-0.232 -> country=United Kingdom, city=London, state=England +2025-05-21 06:31:04,061: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=33.8333&lon=35.8333&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:05,056: Row 507: Reverse geocoded lat=33.8333, lon=35.8333 -> country=Lebanon, city=Jdita, state=Beqaa Governorate +2025-05-21 06:31:06,061: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=14.5971&lon=120.9798&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:07,077: Row 509: Reverse geocoded lat=14.5971, lon=120.9798 -> country=Philippines, city=Manila, state=None +2025-05-21 06:31:08,081: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.0729&lon=4.5475&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:09,043: Row 510: Reverse geocoded lat=52.0729, lon=4.5475 -> country=Netherlands, city=Benthuizen, state=South Holland +2025-05-21 06:31:10,049: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.0729&lon=4.5475&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:11,067: Row 511: Reverse geocoded lat=52.0729, lon=4.5475 -> country=Netherlands, city=Benthuizen, state=South Holland +2025-05-21 06:31:12,073: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.0276&lon=72.5871&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:13,109: Row 512: Reverse geocoded lat=23.0276, lon=72.5871 -> country=India, city=Ahmedabad, state=Gujarat +2025-05-21 06:31:14,113: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=38.9757&lon=-77.6414&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:15,077: Row 513: Reverse geocoded lat=38.9757, lon=-77.6414 -> country=United States, city=Aldie, state=Virginia +2025-05-21 06:31:16,081: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=10.4873&lon=-66.8738&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:17,046: Row 514: Reverse geocoded lat=10.4873, lon=-66.8738 -> country=Venezuela, city=Caracas, state=Miranda State +2025-05-21 06:31:18,053: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-23.6301&lon=-46.6378&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:19,058: Row 515: Reverse geocoded lat=-23.6301, lon=-46.6378 -> country=Brazil, city=São Paulo, state=São Paulo +2025-05-21 06:31:20,061: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=14.6475&lon=121.0494&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:21,038: Row 516: Reverse geocoded lat=14.6475, lon=121.0494 -> country=Philippines, city=Quezon City, state=None +2025-05-21 06:31:22,043: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=26.9136&lon=75.7858&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:23,079: Row 519: Reverse geocoded lat=26.9136, lon=75.7858 -> country=India, city=Jaipur, state=Rajasthan +2025-05-21 06:31:24,087: Row 520: Reverse geocoded lat=40.7592, lon=-111.8875 -> country=United States, city=Salt Lake City, state=Utah +2025-05-21 06:31:25,091: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=20.2706&lon=85.8334&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:26,124: Row 523: Reverse geocoded lat=20.2706, lon=85.8334 -> country=India, city=Bhubaneswar Municipal Corporation, state=Odisha +2025-05-21 06:31:27,131: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=17.3843&lon=78.4583&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:28,041: Row 525: Reverse geocoded lat=17.3843, lon=78.4583 -> country=India, city=Hyderabad, state=Telangana +2025-05-21 06:31:29,047: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.6327&lon=77.2198&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:30,139: Row 528: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:31:31,141: Row 529: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:31:32,145: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=25.5943&lon=85.1352&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:33,015: Row 530: Reverse geocoded lat=25.5943, lon=85.1352 -> country=India, city=Patna, state=Bihar +2025-05-21 06:31:34,021: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=24.6869&lon=46.7224&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:35,025: Row 531: Reverse geocoded lat=24.6869, lon=46.7224 -> country=Saudi Arabia, city=Riyadh, state=Riyadh Region +2025-05-21 06:31:36,032: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.7556&lon=21.2526&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:37,089: Row 532: Reverse geocoded lat=48.7556, lon=21.2526 -> country=Slovakia, city=Košice, state=Region of Košice +2025-05-21 06:31:38,094: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.3028&lon=-121.9982&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:39,114: Row 533: Reverse geocoded lat=37.3028, lon=-121.9982 -> country=United States, city=San Jose, state=California +2025-05-21 06:31:40,108: Row 536: Reverse geocoded lat=35.7449, lon=0.5663 -> country=Algeria, city=Relizane, state=Relizane +2025-05-21 06:31:41,081: Row 538: Reverse geocoded lat=23.1181, lon=113.2539 -> country=China, city=Yuexiu District, state=Guangdong Province +2025-05-21 06:31:42,124: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=39.9707&lon=32.7316&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:43,051: Row 540: Reverse geocoded lat=39.9707, lon=32.7316 -> country=Turkey, city=Yenimahalle, state=Ankara +2025-05-21 06:31:44,058: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.6533&lon=90.4923&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:45,101: Row 541: Reverse geocoded lat=23.6533, lon=90.4923 -> country=Bangladesh, city=Narayanganj, state=Dhaka Division +2025-05-21 06:31:46,060: Row 542: Reverse geocoded lat=22.3008, lon=73.2043 -> country=India, city=Vadodara, state=Gujarat +2025-05-21 06:31:47,109: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.0822&lon=-81.5498&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:48,085: Row 545: Reverse geocoded lat=30.0822, lon=-81.5498 -> country=United States, city=None, state=Florida +2025-05-21 06:31:49,090: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=3.5847&lon=98.6629&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:50,083: Row 546: Reverse geocoded lat=3.5847, lon=98.6629 -> country=Indonesia, city=City of Medan, state=North Sumatra +2025-05-21 06:31:51,089: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.8964&lon=76.5909&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:52,053: Row 547: Reverse geocoded lat=28.8964, lon=76.5909 -> country=India, city=Rohtak, state=Haryana +2025-05-21 06:31:53,057: Row 548: Reverse geocoded lat=30.5394, lon=-91.0522 -> country=United States, city=Central, state=Louisiana +2025-05-21 06:31:54,062: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.4562&lon=6.0573&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:55,152: Row 550: Reverse geocoded lat=51.4562, lon=6.0573 -> country=Netherlands, city=Horst, state=Limburg +2025-05-21 06:31:56,160: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.4073&lon=-121.939&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:57,154: Row 551: Reverse geocoded lat=37.4073, lon=-121.939 -> country=United States, city=San Jose, state=California +2025-05-21 06:31:58,160: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.8558&lon=2.3494&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:31:59,167: Row 552: Reverse geocoded lat=48.8558, lon=2.3494 -> country=France, city=Paris, state=Ile-de-France +2025-05-21 06:32:00,017: Row 553: Reverse geocoded lat=25.5943, lon=85.1352 -> country=India, city=Patna, state=Bihar +2025-05-21 06:32:01,177: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-37.867&lon=144.9973&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:02,067: Row 554: Reverse geocoded lat=-37.867, lon=144.9973 -> country=Australia, city=Melbourne, state=Victoria +2025-05-21 06:32:03,073: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=53.5956&lon=10.0044&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:04,071: Row 555: Reverse geocoded lat=53.5956, lon=10.0044 -> country=Germany, city=Hamburg, state=None +2025-05-21 06:32:05,076: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.144&lon=17.1332&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:06,024: Row 556: Reverse geocoded lat=51.144, lon=17.1332 -> country=Poland, city=Wrocław, state=Lower Silesian Voivodeship +2025-05-21 06:32:07,029: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.8558&lon=2.3494&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:08,151: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.8558&lon=2.3494&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:09,192: Row 557: Reverse geocoded lat=48.8558, lon=2.3494 -> country=France, city=Paris, state=Ile-de-France +2025-05-21 06:32:10,198: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=43.2951&lon=5.3861&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:11,103: Row 558: Reverse geocoded lat=43.2951, lon=5.3861 -> country=France, city=Marseille, state=Provence-Alpes-Côte d'Azur +2025-05-21 06:32:12,084: Row 559: Reverse geocoded lat=17.3843, lon=78.4583 -> country=India, city=Hyderabad, state=Telangana +2025-05-21 06:32:13,108: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.3275&lon=78.0325&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:14,090: Row 561: Reverse geocoded lat=30.3275, lon=78.0325 -> country=India, city=Dehradun, state=Uttarakhand +2025-05-21 06:32:15,094: Row 562: Reverse geocoded lat=21.4913, lon=39.1841 -> country=Saudi Arabia, city=Jeddah, state=Makkah Region +2025-05-21 06:32:16,099: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=1.3153&lon=103.9143&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:17,104: Row 563: Reverse geocoded lat=1.3153, lon=103.9143 -> country=Singapore, city=Singapore, state=None +2025-05-21 06:32:18,108: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.9414&lon=40.1933&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:19,057: Row 565: Reverse geocoded lat=37.9414, lon=40.1933 -> country=Turkey, city=Kayapınar, state=None +2025-05-21 06:32:20,062: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.767&lon=9.1827&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:21,042: Row 566: Reverse geocoded lat=48.767, lon=9.1827 -> country=Germany, city=Stuttgart, state=Baden-Württemberg +2025-05-21 06:32:22,033: Row 568: Reverse geocoded lat=17.3843, lon=78.4583 -> country=India, city=Hyderabad, state=Telangana +2025-05-21 06:32:23,047: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=47.3741&lon=9.5417&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:24,097: Row 569: Reverse geocoded lat=47.3741, lon=9.5417 -> country=Switzerland, city=Altstätten, state=St. Gallen +2025-05-21 06:32:25,101: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.1454&lon=77.3281&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:26,203: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.1454&lon=77.3281&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:27,049: Row 573: Reverse geocoded lat=28.1454, lon=77.3281 -> country=India, city=Palwal, state=Haryana +2025-05-21 06:32:28,056: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.4982&lon=13.4755&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:29,047: Row 575: Reverse geocoded lat=52.4982, lon=13.4755 -> country=Germany, city=Berlin, state=None +2025-05-21 06:32:30,052: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=52.4982&lon=13.4755&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:31,102: Row 576: Reverse geocoded lat=52.4982, lon=13.4755 -> country=Germany, city=Berlin, state=None +2025-05-21 06:32:32,107: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=14.6438&lon=120.9632&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:33,102: Row 578: Reverse geocoded lat=14.6438, lon=120.9632 -> country=Philippines, city=Caloocan, state=None +2025-05-21 06:32:34,105: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=41.0396&lon=-111.9361&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:35,184: Row 579: Reverse geocoded lat=41.0396, lon=-111.9361 -> country=United States, city=Kaysville, state=Utah +2025-05-21 06:32:36,189: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.8713&lon=4.3758&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:37,105: Row 580: Reverse geocoded lat=50.8713, lon=4.3758 -> country=Belgium, city=Schaerbeek - Schaarbeek, state=None +2025-05-21 06:32:38,108: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=55.867&lon=-4.2621&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:39,115: Row 581: Reverse geocoded lat=55.867, lon=-4.2621 -> country=United Kingdom, city=Glasgow, state=Scotland +2025-05-21 06:32:40,120: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=18.0&lon=105.0&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:41,107: Row 583: Reverse geocoded lat=18, lon=105 -> country=Laos, city=Phonkeo, state=Bolikhamsai +2025-05-21 06:32:42,028: Row 584: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:32:43,031: Row 586: Reverse geocoded lat=26.8373, lon=80.9165 -> country=India, city=Lucknow, state=Uttar Pradesh +2025-05-21 06:32:44,118: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=39.911&lon=116.395&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:45,043: Row 587: Reverse geocoded lat=39.911, lon=116.395 -> country=China, city=Dongcheng District, state=Beijing +2025-05-21 06:32:46,050: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.5081&lon=-0.1278000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:47,144: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.5081&lon=-0.1278000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:32:48,220: Row 591: Reverse geocoded lat=51.5081, lon=-0.1278 -> country=United Kingdom, city=City of Westminster, state=England +2025-05-21 06:32:49,171: Row 592: Reverse geocoded lat=32.3039, lon=-110.8368 -> country=United States, city=None, state=Arizona +2025-05-21 06:32:50,093: Row 594: Reverse geocoded lat=37.5453, lon=-77.9187 -> country=United States, city=Powhatan, state=Virginia +2025-05-21 06:32:51,168: Row 595: Reverse geocoded lat=30.6798, lon=-88.2914 -> country=United States, city=None, state=Alabama +2025-05-21 06:32:52,188: Row 596: Reverse geocoded lat=36.8178, lon=10.1656 -> country=Tunisia, city=Tunis, state=Tunis +2025-05-21 06:32:53,174: Row 597: Reverse geocoded lat=34.2635, lon=108.9246 -> country=China, city=Lianhu District, state=Shaanxi +2025-05-21 06:32:54,166: Row 598: Reverse geocoded lat=12.9753, lon=77.591 -> country=India, city=Bengaluru, state=Karnataka +2025-05-21 06:32:55,079: Row 599: Reverse geocoded lat=12.9753, lon=77.591 -> country=India, city=Bengaluru, state=Karnataka +2025-05-21 06:32:56,073: Row 600: Reverse geocoded lat=21.9974, lon=79.0011 -> country=India, city=Malhanwara, state=Madhya Pradesh +2025-05-21 06:32:57,009: Row 602: Reverse geocoded lat=-29.8553, lon=31.0428 -> country=South Africa, city=Durban, state=KwaZulu-Natal +2025-05-21 06:32:58,142: Row 603: Reverse geocoded lat=1.2872, lon=103.8507 -> country=Singapore, city=Singapore, state=None +2025-05-21 06:32:59,091: Row 604: Reverse geocoded lat=23.7207, lon=90.5635 -> country=Bangladesh, city=None, state=Dhaka Division +2025-05-21 06:33:00,155: Row 606: Reverse geocoded lat=51.5164, lon=-0.093 -> country=United Kingdom, city=City of London, state=England +2025-05-21 06:33:01,195: Row 607: Reverse geocoded lat=29.572, lon=-95.54 -> country=United States, city=Missouri City, state=Texas +2025-05-21 06:33:02,031: Row 609: Reverse geocoded lat=37.751, lon=-97.822 -> country=United States, city=None, state=Kansas +2025-05-21 06:33:03,096: Row 610: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:33:04,228: Row 611: Reverse geocoded lat=18.5211, lon=73.8502 -> country=India, city=Pune, state=Maharashtra +2025-05-21 06:33:05,147: Row 612: Reverse geocoded lat=40.2584, lon=-111.6591 -> country=United States, city=Provo, state=Utah +2025-05-21 06:33:06,139: Row 613: Reverse geocoded lat=37.4073, lon=-121.939 -> country=United States, city=San Jose, state=California +2025-05-21 06:33:07,134: Row 614: Reverse geocoded lat=22.2904, lon=70.7915 -> country=India, city=Rajkot, state=Gujarat +2025-05-21 06:33:08,126: Row 615: Reverse geocoded lat=37.9448, lon=-122.0279 -> country=United States, city=Concord, state=California +2025-05-21 06:33:09,246: Row 617: Reverse geocoded lat=34.2619, lon=-6.5821 -> country=Morocco, city=Kenitra, state=None +2025-05-21 06:33:10,158: Row 619: Reverse geocoded lat=45.3696, lon=-73.5654 -> country=Canada, city=Saint-Constant, state=Quebec +2025-05-21 06:33:11,127: Row 620: Reverse geocoded lat=19.0748, lon=72.8856 -> country=India, city=Mumbai, state=Maharashtra +2025-05-21 06:33:12,055: Row 621: Reverse geocoded lat=-23.5475, lon=-46.6361 -> country=Brazil, city=São Paulo, state=São Paulo +2025-05-21 06:33:13,134: Row 622: Reverse geocoded lat=55.6523, lon=12.0805 -> country=Denmark, city=Roskilde, state=Region Zealand +2025-05-21 06:33:14,059: Row 623: Reverse geocoded lat=-33.914, lon=18.4129 -> country=South Africa, city=Cape Town, state=Western Cape +2025-05-21 06:33:15,314: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=49.6113&lon=6.1294&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:16,173: Row 625: Reverse geocoded lat=49.6113, lon=6.1294 -> country=Luxembourg, city=Luxembourg, state=None +2025-05-21 06:33:17,144: Row 626: Reverse geocoded lat=-26.2309, lon=28.0583 -> country=South Africa, city=Johannesburg, state=Gauteng +2025-05-21 06:33:18,159: Row 627: Reverse geocoded lat=30.7339, lon=76.7889 -> country=India, city=Chandigarh, state=Chandigarh +2025-05-21 06:33:19,101: Row 628: Reverse geocoded lat=40.7547, lon=-73.9614 -> country=United States, city=New York, state=New York +2025-05-21 06:33:20,165: Row 629: Reverse geocoded lat=51.5081, lon=-0.1278 -> country=United Kingdom, city=City of Westminster, state=England +2025-05-21 06:33:21,085: Row 630: Reverse geocoded lat=31.2222, lon=121.4581 -> country=China, city=Shanghai, state=Shanghai +2025-05-21 06:33:22,096: Row 631: Reverse geocoded lat=31.2222, lon=121.4581 -> country=China, city=Shanghai, state=Shanghai +2025-05-21 06:33:23,172: Row 633: Reverse geocoded lat=9.0245, lon=38.7485 -> country=Ethiopia, city=None, state=Addis Ababa +2025-05-21 06:33:24,049: Row 634: Reverse geocoded lat=1, lon=38 -> country=Kenya, city=Waso ward, state=Samburu +2025-05-21 06:33:25,067: Row 635: Reverse geocoded lat=-22.1231, lon=-51.3881 -> country=Brazil, city=Presidente Prudente, state=São Paulo +2025-05-21 06:33:26,108: Row 636: Reverse geocoded lat=40.6036, lon=-3.7041 -> country=Spain, city=Tres Cantos, state=Community of Madrid +2025-05-21 06:33:27,099: Row 637: Reverse geocoded lat=33.1371, lon=-96.7488 -> country=United States, city=Frisco, state=Texas +2025-05-21 06:33:28,178: Row 638: Reverse geocoded lat=39.7593, lon=-104.9651 -> country=United States, city=Denver, state=Colorado +2025-05-21 06:33:29,096: Row 639: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:33:30,217: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=36.1898&lon=44.0157&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:31,034: Row 640: Reverse geocoded lat=36.1898, lon=44.0157 -> country=Iraq, city=Erbil, state=Erbil Governorate +2025-05-21 06:33:32,040: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=37.2845&lon=-107.8646&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:33,127: Row 641: Reverse geocoded lat=37.2845, lon=-107.8646 -> country=United States, city=Durango, state=Colorado +2025-05-21 06:33:34,126: Row 642: Reverse geocoded lat=35.9825, lon=-78.5376 -> country=United States, city=Wake Forest, state=North Carolina +2025-05-21 06:33:35,098: Row 643: Reverse geocoded lat=-14.785, lon=-39.2785 -> country=Brazil, city=Itabuna, state=Bahia +2025-05-21 06:33:36,135: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-14.785&lon=-39.2785&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:37,095: Row 644: Reverse geocoded lat=-14.785, lon=-39.2785 -> country=Brazil, city=Itabuna, state=Bahia +2025-05-21 06:33:38,100: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.3904&lon=-0.0992000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:39,138: Row 648: Reverse geocoded lat=51.3904, lon=-0.0992 -> country=United Kingdom, city=London, state=England +2025-05-21 06:33:40,142: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=11.0102&lon=76.9701&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:41,193: Row 649: Reverse geocoded lat=11.0102, lon=76.9701 -> country=India, city=Coimbatore, state=Tamil Nadu +2025-05-21 06:33:42,021: Row 650: Reverse geocoded lat=-22.4851, lon=-47.1733 -> country=Brazil, city=Engenheiro Coelho, state=São Paulo +2025-05-21 06:33:43,203: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.9171&lon=24.7158&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:44,162: Row 652: Reverse geocoded lat=48.9171, lon=24.7158 -> country=Ukraine, city=Ivano-Frankivsk, state=Ivano-Frankivsk Oblast +2025-05-21 06:33:45,168: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.8558&lon=2.3494&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:46,178: Row 653: Reverse geocoded lat=48.8558, lon=2.3494 -> country=France, city=Paris, state=Ile-de-France +2025-05-21 06:33:47,182: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-6.3088&lon=107.316&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:48,153: Row 654: Reverse geocoded lat=-6.3088, lon=107.316 -> country=Indonesia, city=Karawang, state=West Java +2025-05-21 06:33:49,151: Row 655: Reverse geocoded lat=43.6547, lon=-79.3623 -> country=Canada, city=Toronto, state=Ontario +2025-05-21 06:33:50,154: Row 657: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:33:51,149: Row 658: Reverse geocoded lat=39.338, lon=-82.986 -> country=United States, city=Chillicothe, state=Ohio +2025-05-21 06:33:52,164: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=22.2842&lon=114.1759&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:53,119: Row 659: Reverse geocoded lat=22.2842, lon=114.1759 -> country=China, city=Hong Kong Island, state=Hong Kong +2025-05-21 06:33:54,091: Row 660: Reverse geocoded lat=17.3843, lon=78.4583 -> country=India, city=Hyderabad, state=Telangana +2025-05-21 06:33:55,060: Row 661: Reverse geocoded lat=25.0734, lon=55.2979 -> country=United Arab Emirates, city=None, state=Dubai +2025-05-21 06:33:56,075: Row 662: Reverse geocoded lat=7.3782, lon=3.9062 -> country=Nigeria, city=None, state=Oyo State +2025-05-21 06:33:57,072: Row 663: Reverse geocoded lat=-31.8544, lon=115.8433 -> country=Australia, city=None, state=Western Australia +2025-05-21 06:33:58,136: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=55.6078&lon=12.9982&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:33:59,022: Row 664: Reverse geocoded lat=55.6078, lon=12.9982 -> country=Sweden, city=Malmö, state=None +2025-05-21 06:34:00,027: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=55.6078&lon=12.9982&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:01,017: Row 665: Reverse geocoded lat=55.6078, lon=12.9982 -> country=Sweden, city=Malmö, state=None +2025-05-21 06:34:02,020: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.2049&lon=16.3662&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:03,130: Row 667: Reverse geocoded lat=48.2049, lon=16.3662 -> country=Austria, city=Vienna, state=None +2025-05-21 06:34:04,134: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=53.5466&lon=10.021&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:05,028: Row 668: Reverse geocoded lat=53.5466, lon=10.021 -> country=Germany, city=Hamburg, state=None +2025-05-21 06:34:06,033: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=53.5247&lon=-7.3385&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:07,036: Row 669: Reverse geocoded lat=53.5247, lon=-7.3385 -> country=Ireland, city=Mullingar, state=None +2025-05-21 06:34:08,040: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=34.5329&lon=69.1682&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:09,149: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=34.5329&lon=69.1682&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:10,180: Row 670: Reverse geocoded lat=34.5329, lon=69.1682 -> country=Afghanistan, city=Kabul, state=Kabul Province +2025-05-21 06:34:11,115: Row 671: Reverse geocoded lat=59.3326, lon=18.0651 -> country=Sweden, city=Stockholm, state=None +2025-05-21 06:34:12,117: Row 672: Reverse geocoded lat=-31.8544, lon=115.8433 -> country=Australia, city=None, state=Western Australia +2025-05-21 06:34:13,143: Row 673: Reverse geocoded lat=-26.2309, lon=28.0583 -> country=South Africa, city=Johannesburg, state=Gauteng +2025-05-21 06:34:14,190: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.7169&lon=-1.9716&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:15,189: Row 674: Reverse geocoded lat=51.7169, lon=-1.9716 -> country=United Kingdom, city=Cotswold District, state=England +2025-05-21 06:34:16,111: Row 675: Reverse geocoded lat=19.0748, lon=72.8856 -> country=India, city=Mumbai, state=Maharashtra +2025-05-21 06:34:17,199: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-6.2056&lon=106.8376&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:18,107: Row 676: Reverse geocoded lat=-6.2056, lon=106.8376 -> country=Indonesia, city=Special capital Region of Jakarta, state=None +2025-05-21 06:34:19,111: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=47.409&lon=9.3361&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:20,173: Row 677: Reverse geocoded lat=47.409, lon=9.3361 -> country=Switzerland, city=St. Gallen, state=St. Gallen +2025-05-21 06:34:21,081: Row 678: Reverse geocoded lat=38.3802, lon=-75.6347 -> country=United States, city=None, state=Maryland +2025-05-21 06:34:22,046: Row 679: Reverse geocoded lat=19.0748, lon=72.8856 -> country=India, city=Mumbai, state=Maharashtra +2025-05-21 06:34:23,126: Row 681: Reverse geocoded lat=34.197, lon=-118.8199 -> country=United States, city=Thousand Oaks, state=California +2025-05-21 06:34:24,189: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=26.9136&lon=75.7858&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:25,044: Row 682: Reverse geocoded lat=26.9136, lon=75.7858 -> country=India, city=Jaipur, state=Rajasthan +2025-05-21 06:34:26,050: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-6.9172&lon=107.6296&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:27,069: Row 683: Reverse geocoded lat=-6.9172, lon=107.6296 -> country=Indonesia, city=Bandung City, state=West Java +2025-05-21 06:34:28,072: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=35.7681&lon=139.6258&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:29,116: Row 685: Reverse geocoded lat=35.7681, lon=139.6258 -> country=Japan, city=Nerima, state=None +2025-05-21 06:34:30,120: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=22.9505&lon=91.4608&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:31,051: Row 686: Reverse geocoded lat=22.9505, lon=91.4608 -> country=Bangladesh, city=Feni, state=Chittagong Division +2025-05-21 06:34:32,055: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=34.1211&lon=-117.4362&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:33,140: Row 687: Reverse geocoded lat=34.1211, lon=-117.4362 -> country=United States, city=Fontana, state=California +2025-05-21 06:34:34,135: Row 689: Reverse geocoded lat=-22.3135, lon=-42.8741 -> country=Brazil, city=Teresópolis, state=Rio de Janeiro +2025-05-21 06:34:35,144: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=32.8908&lon=13.1796&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:36,120: Row 690: Reverse geocoded lat=32.8908, lon=13.1796 -> country=Libya, city=Tripoli, state=Tripoli +2025-05-21 06:34:37,026: Row 691: Reverse geocoded lat=13.0895, lon=80.2739 -> country=India, city=Chennai, state=Tamil Nadu +2025-05-21 06:34:38,118: Row 692: Reverse geocoded lat=59.3287, lon=18.0717 -> country=Sweden, city=Stockholm, state=None +2025-05-21 06:34:39,129: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=16.2689&lon=-61.5073&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:40,073: Row 693: Reverse geocoded lat=16.2689, lon=-61.5073 -> country=France, city=Les Abymes, state=Guadeloupe +2025-05-21 06:34:41,076: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=14.6343&lon=-90.5155&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:42,143: Row 694: Reverse geocoded lat=14.6343, lon=-90.5155 -> country=Guatemala, city=Guatemala City, state=Guatemala Department +2025-05-21 06:34:43,097: Row 695: Reverse geocoded lat=-25.757, lon=28.1443 -> country=South Africa, city=Pretoria, state=Gauteng +2025-05-21 06:34:44,152: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=50.8713&lon=4.3758&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:45,112: Row 696: Reverse geocoded lat=50.8713, lon=4.3758 -> country=Belgium, city=Schaerbeek - Schaarbeek, state=None +2025-05-21 06:34:46,102: Row 698: Reverse geocoded lat=6.4474, lon=3.3903 -> country=Nigeria, city=Lagos, state=Lagos State +2025-05-21 06:34:47,119: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=33.6099&lon=73.0301&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:48,060: Row 699: Reverse geocoded lat=33.6099, lon=73.0301 -> country=Pakistan, city=Rawalpindi Cantonment, state=Punjab +2025-05-21 06:34:49,064: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=51.4964&lon=-0.1224000&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:50,061: Row 700: Reverse geocoded lat=51.4964, lon=-0.1224 -> country=United Kingdom, city=London, state=England +2025-05-21 06:34:51,046: Row 701: Reverse geocoded lat=33.6128, lon=-117.6622 -> country=United States, city=Mission Viejo, state=California +2025-05-21 06:34:52,037: Row 702: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:34:53,071: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-1.944&lon=30.062&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:54,102: Row 703: Reverse geocoded lat=-1.944, lon=30.062 -> country=Rwanda, city=Kigali, state=Kigali City +2025-05-21 06:34:55,071: Row 704: Reverse geocoded lat=31.9555, lon=35.9435 -> country=Jordan, city=None, state=Amman +2025-05-21 06:34:56,111: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=41.2615&lon=69.2177&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:57,076: Row 705: Reverse geocoded lat=41.2615, lon=69.2177 -> country=Uzbekistan, city=Tashkent, state=None +2025-05-21 06:34:58,087: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=-22.9072&lon=-43.1883&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:34:59,053: Row 706: Reverse geocoded lat=-22.9072, lon=-43.1883 -> country=Brazil, city=Rio de Janeiro, state=Rio de Janeiro +2025-05-21 06:35:00,057: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=35.6893&lon=139.6899&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:01,043: Row 707: Reverse geocoded lat=35.6893, lon=139.6899 -> country=Japan, city=Shinjuku, state=None +2025-05-21 06:35:02,050: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=30.9977&lon=29.7432&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:03,118: Row 708: Reverse geocoded lat=30.9977, lon=29.7432 -> country=Egypt, city=Al Wadi, state=Alexandria +2025-05-21 06:35:04,124: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=20.0024&lon=73.7945&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:05,042: Row 709: Reverse geocoded lat=20.0024, lon=73.7945 -> country=India, city=Nashik, state=Maharashtra +2025-05-21 06:35:06,046: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=28.6327&lon=77.2198&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:07,076: Row 712: Reverse geocoded lat=28.6327, lon=77.2198 -> country=India, city=New Delhi, state=Delhi +2025-05-21 06:35:08,080: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=18.1667&lon=-77.25&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:09,053: Row 714: Reverse geocoded lat=18.1667, lon=-77.25 -> country=Jamaica, city=None, state=None +2025-05-21 06:35:10,045: Row 715: Reverse geocoded lat=18.1667, lon=-77.25 -> country=Jamaica, city=None, state=None +2025-05-21 06:35:11,058: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=34.2635&lon=108.9246&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:12,107: Row 716: Reverse geocoded lat=34.2635, lon=108.9246 -> country=China, city=Lianhu District, state=Shaanxi +2025-05-21 06:35:13,112: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=9.0245&lon=38.7485&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:14,090: Row 717: Reverse geocoded lat=9.0245, lon=38.7485 -> country=Ethiopia, city=None, state=Addis Ababa +2025-05-21 06:35:15,094: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.8582&lon=2.3387&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:16,219: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.8582&lon=2.3387&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:17,242: Row 718: Reverse geocoded lat=48.8582, lon=2.3387 -> country=France, city=Paris, state=None +2025-05-21 06:35:18,209: Row 719: Reverse geocoded lat=48.8582, lon=2.3387 -> country=France, city=Paris, state=None +2025-05-21 06:35:19,250: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=48.8582&lon=2.3387&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:20,228: Row 720: Reverse geocoded lat=48.8582, lon=2.3387 -> country=France, city=Paris, state=None +2025-05-21 06:35:21,081: Row 721: Reverse geocoded lat=37.751, lon=-97.822 -> country=United States, city=None, state=Kansas +2025-05-21 06:35:22,084: Row 722: Reverse geocoded lat=51.5164, lon=-0.093 -> country=United Kingdom, city=City of London, state=England +2025-05-21 06:35:23,117: Row 723: Reverse geocoded lat=51.5164, lon=-0.093 -> country=United Kingdom, city=City of London, state=England +2025-05-21 06:35:24,076: Row 724: Reverse geocoded lat=24.1844, lon=88.2673 -> country=India, city=Murshidabad, state=West Bengal +2025-05-21 06:35:25,245: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=49.0719&lon=17.4638&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:26,063: Row 725: Reverse geocoded lat=49.0719, lon=17.4638 -> country=Czechia, city=Uherské Hradiště, state=Central Moravia +2025-05-21 06:35:27,066: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=39.1994&lon=-84.526&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:28,048: Row 726: Reverse geocoded lat=39.1994, lon=-84.526 -> country=United States, city=Cincinnati, state=Ohio +2025-05-21 06:35:29,058: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.0276&lon=72.5871&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:30,181: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.0276&lon=72.5871&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:31,148: Row 727: Reverse geocoded lat=23.0276, lon=72.5871 -> country=India, city=Ahmedabad, state=Gujarat +2025-05-21 06:35:32,083: Row 728: Reverse geocoded lat=12.9753, lon=77.591 -> country=India, city=Bengaluru, state=Karnataka +2025-05-21 06:35:33,115: Row 729: Reverse geocoded lat=12.9753, lon=77.591 -> country=India, city=Bengaluru, state=Karnataka +2025-05-21 06:35:34,141: Row 731: Reverse geocoded lat=34.7599, lon=113.6459 -> country=China, city=Zhengzhou, state=Henan +2025-05-21 06:35:35,069: Row 732: Reverse geocoded lat=19.4324, lon=-99.1229 -> country=Mexico, city=Mexico City, state=Mexico City +2025-05-21 06:35:36,162: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=45.4722&lon=9.1922&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:37,171: Row 733: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:35:38,131: Row 734: Reverse geocoded lat=29.1383, lon=-80.9956 -> country=United States, city=Port Orange, state=Florida +2025-05-21 06:35:39,127: Row 735: Reverse geocoded lat=45.4722, lon=9.1922 -> country=Italy, city=Milan, state=Lombardy +2025-05-21 06:35:40,054: Row 736: Reverse geocoded lat=35.6893, lon=139.6899 -> country=Japan, city=Shinjuku, state=None +2025-05-21 06:35:41,169: Row 737: Reverse geocoded lat=38.8574, lon=-77.1 -> country=United States, city=Arlington, state=Virginia +2025-05-21 06:35:42,189: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Read timed out. (read timeout=1)")': /reverse?lat=23.7891&lon=90.4126&format=json&accept-language=en-US&addressdetails=1 +2025-05-21 06:35:43,127: Row 739: Reverse geocoded lat=23.7891, lon=90.4126 -> country=Bangladesh, city=Dhaka, state=Dhaka Division +2025-05-21 06:35:43,159: Created columns 'Country', 'City', and 'State' with reverse geocoded data. +2025-05-21 06:35:43,429: Cleaned data written to /Users/T.Martin/Projects/reporting_doc_feedback/data_exports/cleaned_data.xlsx +2025-05-21 06:35:43,429: Script completed successfully. diff --git a/content/includes/installation/add-nms-repo.md b/content/includes/installation/add-nms-repo.md index fec0ae06f..eac86faa7 100644 --- a/content/includes/installation/add-nms-repo.md +++ b/content/includes/installation/add-nms-repo.md @@ -36,23 +36,39 @@ Add the NGINX Instance Manager repository: {{%tab name="Debian, Ubuntu, Deb-Based"%}} -Add the NGINX Instance Manager repository: - -- **Debian** +1. Add the NGINX signing key: ```shell - printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/nms/debian `lsb_release -cs` nginx-plus\n" | sudo tee /etc/apt/sources.list.d/nms.list - sudo wget -q -O /etc/apt/apt.conf.d/90pkgs-nginx https://cs.nginx.com/static/files/90pkgs-nginx + wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key \ + | gpg --dearmor \ + | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null ``` -
+2. Add the NGINX Instance Manager repository: -- **Ubuntu** + - **Debian** - ```shell - printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/nms/ubuntu `lsb_release -cs` nginx-plus\n" | sudo tee /etc/apt/sources.list.d/nms.list - sudo wget -q -O /etc/apt/apt.conf.d/90pkgs-nginx https://cs.nginx.com/static/files/90pkgs-nginx - ``` + ```shell + printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ + https://pkgs.nginx.com/nms/debian $(lsb_release -cs) nginx-plus\n" | \ + sudo tee /etc/apt/sources.list.d/nms.list + + sudo wget -q -O /etc/apt/apt.conf.d/90pkgs-nginx \ + https://cs.nginx.com/static/files/90pkgs-nginx + ``` + +
+ + - **Ubuntu** + + ```shell + printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ + https://pkgs.nginx.com/nms/ubuntu $(lsb_release -cs) nginx-plus\n" | \ + sudo tee /etc/apt/sources.list.d/nms.list + + sudo wget -q -O /etc/apt/apt.conf.d/90pkgs-nginx \ + https://cs.nginx.com/static/files/90pkgs-nginx + ``` {{%/tab%}} {{}} diff --git a/content/includes/installation/clickhouse-defaults.md b/content/includes/installation/clickhouse-defaults.md deleted file mode 100644 index 0921b6ebb..000000000 --- a/content/includes/installation/clickhouse-defaults.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -docs: DOCS-1238 ---- - -{{}} - -| Configuration | Default | Notes | -|--------------------------|------------------------------------|-------| -| clickhouse.address | tcp://localhost:9000 | | -| clickhouse.username | | | -| clickhouse.password | | | -| clickhouse.tls_mode | false | | -| clickhouse.tls.address | tcp://localhost:9440 | | -| clickhouse.tls.skip_verify | false |`clickhouse.tls.skip_verify` should be used only for self-signed certificates and is never recommended for production use. When set to `true`, certificates are not verified, which exposes the connection to man-in-the-middle attacks. | -| clickhouse.tls.key_path | | | -| clickhouse.tls.cert_path | | | -| clickhouse.tls.ca_path | /etc/ssl/certs/ca-certificates.crt |The default value for `clickhouse.tls.ca_path` works out-of-the-box for Ubuntu and Debian. You'll need to [configure a different Certificate Authority](#tls) for other distributions. Refer to your distribution's documentation for additional information.| - -{{}} - -
diff --git a/content/includes/nginx-plus/usage-tracking/overview.md b/content/includes/nginx-plus/usage-tracking/overview.md index 3a048abf9..85af6fb49 100644 --- a/content/includes/nginx-plus/usage-tracking/overview.md +++ b/content/includes/nginx-plus/usage-tracking/overview.md @@ -6,5 +6,5 @@ Start by installing NGINX Instance Manager on a dedicated host. Then, configure {{}} -To set up automatic reporting, [add your JWT-based license to NGINX Instance Manager]({{< ref "/nim/admin-guide/license/add-license.md#apply-jwt-license" >}}). This license can be downloaded from [MyF5](https://account.f5.com/myf5) if needed. +To set up automatic reporting, [add your JWT-based license to NGINX Instance Manager]({{< ref "/nim/admin-guide/add-license.md#apply-jwt-license" >}}). This license can be downloaded from [MyF5](https://account.f5.com/myf5) if needed. {{}} diff --git a/content/includes/nim/admin-guide/license/connected-install-license-note.md b/content/includes/nim/admin-guide/license/connected-install-license-note.md index 251043df4..9490e3fe3 100644 --- a/content/includes/nim/admin-guide/license/connected-install-license-note.md +++ b/content/includes/nim/admin-guide/license/connected-install-license-note.md @@ -4,4 +4,4 @@ docs: A valid license is required to make full use of all the features in NGINX Instance Manager. -Refer to the [Add a License]({{< ref "nim/admin-guide/license/add-license.md" >}}) topic for instructions on how to download and apply a trial license, subscription license, or Flexible Consumption Program license. \ No newline at end of file +Refer to the [Add a License]({{< ref "nim/admin-guide/add-license.md" >}}) topic for instructions on how to download and apply a trial license, subscription license, or Flexible Consumption Program license. \ No newline at end of file diff --git a/content/includes/nim/clickhouse/cli-skip-clickhouse.md b/content/includes/nim/clickhouse/cli-skip-clickhouse.md new file mode 100644 index 000000000..f4228b125 --- /dev/null +++ b/content/includes/nim/clickhouse/cli-skip-clickhouse.md @@ -0,0 +1,10 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide.md +--- + +If you skip installing ClickHouse, you need NGINX Agent {{< lightweight-nim-nginx-agent-version >}}. + +After installation, make sure to [disable metrics collection]({{< ref "nim/system-configuration/configure-clickhouse.md#disable-metrics-collection" >}}) in the `/etc/nms/nms.conf` file. diff --git a/content/includes/nim/clickhouse/clickhouse-defaults.md b/content/includes/nim/clickhouse/clickhouse-defaults.md new file mode 100644 index 000000000..1848b75f7 --- /dev/null +++ b/content/includes/nim/clickhouse/clickhouse-defaults.md @@ -0,0 +1,23 @@ +--- +docs: DOCS-1238 +files: +- content/nim/system-configuration/configure-clickhouse.md +- content/nim/deploy/vm-bare-metal/install-nim-manual.md +--- + +{{}} + +| Configuration | Default | Notes | +|------------------------------|------------------------------------|-------| +| clickhouse.enable | true | Set to `false` to disable metrics collection and run NGINX Instance Manager in lightweight mode. Requires a service restart. | +| clickhouse.address | tcp://localhost:9000 | The address of the ClickHouse database. | +| clickhouse.username | | The username NGINX Instance Manager uses to connect to ClickHouse, if authentication is enabled. | +| clickhouse.password | | The password for the specified ClickHouse user. | +| clickhouse.tls_mode | false | Set to `true` to enable TLS for the ClickHouse connection. This setting will be deprecated in a future release. Use the `clickhouse.tls` section instead. | +| clickhouse.tls.address | tcp://localhost:9440 | The address NGINX Instance Manager uses to connect to ClickHouse over TLS. Format: `:`. | +| clickhouse.tls.skip_verify | false | Set to `true` to skip TLS certificate verification. Use only for self-signed certificates in non-production environments. | +| clickhouse.tls.key_path | | Path to the client TLS key file in PEM format. Required for client authentication. | +| clickhouse.tls.cert_path | | Path to the client TLS certificate file in PEM format. Required for client authentication. | +| clickhouse.tls.ca_path | /etc/ssl/certs/ca-certificates.crt | Path to the system Certificate Authority used to verify the server certificate. The default path works for Ubuntu and Debian. Use a CA bundle appropriate to your system. See [TLS configuration](#tls) for details. | + +{{}} diff --git a/content/includes/nim/clickhouse/clickhouse-install.md b/content/includes/nim/clickhouse/clickhouse-install.md new file mode 100644 index 000000000..961aed96b --- /dev/null +++ b/content/includes/nim/clickhouse/clickhouse-install.md @@ -0,0 +1,70 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install-nim-manual.md +--- + +NGINX Instance Manager uses [ClickHouse](https://clickhouse.com) to store metrics, events, alerts, and configuration settings. + +If you install ClickHouse and choose to set a password (the default is an empty string), you must add it to the `clickhouse.password` setting in the `/etc/nms/nms.conf` file. If the password is missing or incorrect, NGINX Instance Manager **will not start**. + +For instructions and additional configuration options, including TLS settings, see [Configure ClickHouse]({{< ref "nim/system-configuration/configure-clickhouse.md" >}}). + +NGINX Instance Manager requires ClickHouse version {{< clickhouse-version >}} or later. + + +Follow these steps to install and enable ClickHouse on supported Linux distributions. + +1. First, set up the repository. + + - For RPM-based systems (CentOS, RHEL): + + ```shell + sudo yum install -y yum-utils + sudo yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo + ``` + + - For Debian-based systems (Debian, Ubuntu): + + ```shell + sudo apt-get install -y apt-transport-https ca-certificates dirmngr + GNUPGHOME=$(mktemp -d) + sudo GNUPGHOME="$GNUPGHOME" gpg --no-default-keyring --keyring /usr/share/keyrings/clickhouse-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8919F6BD2B48D754 + sudo rm -r "$GNUPGHOME" + sudo chmod +r /usr/share/keyrings/clickhouse-keyring.gpg + + echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb lts main" | sudo tee /etc/apt/sources.list.d/clickhouse.list + sudo apt-get update + ``` + +2. Next, install the ClickHouse server and client: + + - For RPM-based systems (CentOS, RHEL): + + ```shell + sudo yum install -y clickhouse-server clickhouse-client + ``` + + - For Debian-based systems (Debian, Ubuntu): + + ```shell + sudo apt-get install -y clickhouse-server clickhouse-client + ``` + +3. Then, enable the ClickHouse service so it starts automatically on reboot: + + ```shell + sudo systemctl enable clickhouse-server + ``` + +4. Start the ClickHouse service: + + ```shell + sudo systemctl start clickhouse-server + ``` + +5. Finally, confirm the service is running: + + ```shell + sudo systemctl status clickhouse-server + ``` diff --git a/content/includes/nim/docker/docker-compose-env-vars.md b/content/includes/nim/docker/docker-compose-env-vars.md index c34e6e767..5fb837435 100644 --- a/content/includes/nim/docker/docker-compose-env-vars.md +++ b/content/includes/nim/docker/docker-compose-env-vars.md @@ -4,14 +4,15 @@ docs: {{}} | Variable | Category | Description | -|----------|----------|------------| +|----------|----------|-------------| +| **ENABLE_METRICS** | General | Enables or disables metrics collection and storage. Set to `false` to run in lightweight mode without ClickHouse. Defaults to `true`. | +| **NIM_MAINTENANCE** | General | Enables maintenance mode for backup, restore, and troubleshooting (`true` or `false`). | +| **NIM_LICENSE_MODE_OF_OPERATION** | General | Sets the license mode to either `connected` (default) or `disconnected`. | | **NIM_LOG_LEVEL** | General | Sets the logging level for NGINX Instance Manager. | | **NIM_METRICS_TTL** | General | Specifies the number of days to retain metrics. | | **NIM_EVENTS_TTL** | General | Specifies the number of days to retain event logs. | | **NIM_SECURITY_TTL** | General | Specifies the number of days to retain security violation logs. | -| **NIM_MAINTENANCE** | General | Enables maintenance mode for backup, restore, and troubleshooting (`true` or `false`). | | **NIM_WATCHDOG_TIMEOUT** | General | Sets the timeout (in seconds) for the Data Plane Monitoring (DPM) watchdog. | -| **NIM_LICENSE_MODE_OF_OPERATION** | General | Sets the license mode to either `connected` (default) or `disconnected`. | | **PROXY_ENABLE** | Forward Proxy | Enables or disables the use of a forward proxy (`true` or `false`). | | **PROXY_HOST** | Forward Proxy | The IP address or hostname of the proxy server. | | **PROXY_PORT** | Forward Proxy | The port number of the proxy server. | @@ -20,7 +21,6 @@ docs: | **PROXY_USERNAME** | Forward Proxy | (Required if `PROXY_AUTH_REQUIRED=true`) The username for proxy authentication. | | **PROXY_PASSWORD** | Forward Proxy | (Required if `PROXY_AUTH_REQUIRED=true`) The password for proxy authentication. | | **PROXY_SSL_VERIFY** | Forward Proxy | Enables or disables SSL verification when `PROXY_PROTOCOL=https`. Default is `true`, meaning the proxy must have a valid certificate issued by a trusted Certificate Authority (CA). Set to `false` to allow self-signed or untrusted certificates (not recommended). | - {{}} diff --git a/content/includes/nim/installation/install-script-flags/cert.md b/content/includes/nim/installation/install-script-flags/cert.md new file mode 100644 index 000000000..578f9ff2c --- /dev/null +++ b/content/includes/nim/installation/install-script-flags/cert.md @@ -0,0 +1,8 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide.md +--- + +`-c `: Path to the SSL certificate file. (default: `/etc/ssl/nginx`) \ No newline at end of file diff --git a/content/includes/nim/installation/install-script-flags/clickhouse-version.md b/content/includes/nim/installation/install-script-flags/clickhouse-version.md new file mode 100644 index 000000000..aaf5e1849 --- /dev/null +++ b/content/includes/nim/installation/install-script-flags/clickhouse-version.md @@ -0,0 +1,8 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide.md +--- + +`-v `: Install a specific version of ClickHouse. If not set, version `{{< clickhouse-version >}}` is installed by default. diff --git a/content/includes/nim/installation/install-script-flags/distribution.md b/content/includes/nim/installation/install-script-flags/distribution.md new file mode 100644 index 000000000..2d6c65cd3 --- /dev/null +++ b/content/includes/nim/installation/install-script-flags/distribution.md @@ -0,0 +1,14 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide.md +--- + +`-d `: Target Linux distribution (for example, `ubuntu22.04`, `rhel8`) + +To see the list of supported distributions, run: + +```shell +grep '\-d distribution' install-nim-bundle.sh +``` \ No newline at end of file diff --git a/content/includes/nim/installation/install-script-flags/key.md b/content/includes/nim/installation/install-script-flags/key.md new file mode 100644 index 000000000..366786c45 --- /dev/null +++ b/content/includes/nim/installation/install-script-flags/key.md @@ -0,0 +1,8 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide.md +--- + +`-k `: Path to the private key file. (default: `/etc/ssl/nginx`) diff --git a/content/includes/nim/installation/install-script-flags/skip-clickhouse.md b/content/includes/nim/installation/install-script-flags/skip-clickhouse.md new file mode 100644 index 000000000..93a61aeb7 --- /dev/null +++ b/content/includes/nim/installation/install-script-flags/skip-clickhouse.md @@ -0,0 +1,10 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide.md +--- + +`-s`: Skip installing ClickHouse. This lightweight mode reduces system requirements and simplifies installation for users who do not need metrics. + +{{< include "nim/clickhouse/cli-skip-clickhouse.md" >}} \ No newline at end of file diff --git a/content/includes/nim/installation/optional-steps/configure-clickhouse.md b/content/includes/nim/installation/optional-steps/configure-clickhouse.md new file mode 100644 index 000000000..c2a086cd5 --- /dev/null +++ b/content/includes/nim/installation/optional-steps/configure-clickhouse.md @@ -0,0 +1,23 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/deploy/vm-bare-metal/install-nim-manual.md +- content/nim/disconnected/offline-install-guide.md +- content/nim/disconnected/offline-install-guide-manual.md +--- + +If you installed ClickHouse and set a password (the default is an empty string), you must add it to the `clickhouse.password` setting in the `/etc/nms/nms.conf` file after installing NGINX Instance Manager. If the password is missing or incorrect, NGINX Instance Manager will not start. + +You can also configure additional ClickHouse settings in the same section: + +- `clickhouse.username` – the username used to connect to ClickHouse +- `clickhouse.address` – the address of the ClickHouse server (default is `tcp://localhost:9000`) +- `clickhouse.tls_mode` – set to `true` to enable TLS +- TLS certificate settings, such as: + - `clickhouse.tls.cert_path` + - `clickhouse.tls.key_path` + - `clickhouse.tls.ca_path` + - `clickhouse.tls.skip_verify` + +For more details, see [Configure ClickHouse]({{< ref "nim/system-configuration/configure-clickhouse.md" >}}). \ No newline at end of file diff --git a/content/includes/nim/installation/optional-steps/configure-selinux.md b/content/includes/nim/installation/optional-steps/configure-selinux.md new file mode 100644 index 000000000..730c97811 --- /dev/null +++ b/content/includes/nim/installation/optional-steps/configure-selinux.md @@ -0,0 +1,12 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install-nim-manual.md +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide-manual.md +- content/nim/disconnected/offline-install-guide.md +--- + +SELinux helps secure your deployment by enforcing mandatory access control policies. + +If you use SELinux, follow the steps in the [Configure SELinux]({{< ref "/nim/system-configuration/configure-selinux.md" >}}) guide to restore SELinux contexts (`restorecon`) for the files and directories related to NGINX Instance Manager. diff --git a/content/includes/nim/installation/optional-steps/disable-metrics-collection.md b/content/includes/nim/installation/optional-steps/disable-metrics-collection.md new file mode 100644 index 000000000..9ccd93654 --- /dev/null +++ b/content/includes/nim/installation/optional-steps/disable-metrics-collection.md @@ -0,0 +1,11 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide-manual.md +- content/nim/disconnected/offline-install-guide.md +--- + +If you’re not collecting metrics—because you didn’t install ClickHouse or don’t plan to use it—you must disable metrics collection in the `/etc/nms/nms.conf` file. This setup requires NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}. + +For instructions, see [Disable metrics collection]({{< ref "nim/system-configuration/configure-clickhouse.md#disable-metrics-collection" >}}). \ No newline at end of file diff --git a/content/includes/nim/installation/optional-steps/install-configure-vault.md b/content/includes/nim/installation/optional-steps/install-configure-vault.md new file mode 100644 index 000000000..6d39a002a --- /dev/null +++ b/content/includes/nim/installation/optional-steps/install-configure-vault.md @@ -0,0 +1,16 @@ +--- +docs: +files: +- content/nim/deploy/vm-bare-metal/install-nim-manual.md +- content/nim/deploy/vm-bare-metal/install.md +- content/nim/disconnected/offline-install-guide-manual.md +- content/nim/disconnected/offline-install-guide.md +--- + +NGINX Instance Manager can use [Vault](https://www.vaultproject.io/) as a datastore for secrets. + +To install and enable Vault, follow these steps: + +- Follow Vault's instructions to [install Vault 1.8.8 or later](https://www.vaultproject.io/docs/install) for your distribution. +- Ensure you're running Vault in a [production-hardened environment](https://learn.hashicorp.com/tutorials/vault/production-hardening). +- After installing NGINX Instance Manager, follow the steps to [configure Vault for storing secrets]({{< ref "/nim/system-configuration/configure-vault.md" >}}). diff --git a/content/includes/nim/kubernetes/nms-chart-supported-module-versions.md b/content/includes/nim/kubernetes/nms-chart-supported-module-versions.md index f55e661b5..21811f7d4 100644 --- a/content/includes/nim/kubernetes/nms-chart-supported-module-versions.md +++ b/content/includes/nim/kubernetes/nms-chart-supported-module-versions.md @@ -6,6 +6,7 @@ docs: DOCS-1324 | NGINX Instance Manager chart | Instance Manager | |------------------------------|------------------| +| 2.0.0 | 2.20.0 | | 1.15.0 | 2.18.0 | | 1.14.4 | 2.17.4 | | 1.14.0 | 2.17.0 | diff --git a/content/includes/nim/tech-specs/supported-distros.md b/content/includes/nim/tech-specs/supported-distros.md index 35e62cdc2..632059861 100644 --- a/content/includes/nim/tech-specs/supported-distros.md +++ b/content/includes/nim/tech-specs/supported-distros.md @@ -1,5 +1,7 @@ --- docs: DOCS-1071 +files: +- content/nim/fundamentals/tech-specs.md --- The following table lists the Linux distributions supported by NGINX Instance Manager and NGINX App Protect: diff --git a/content/includes/nim/tech-specs/supported-nginx-versions.md b/content/includes/nim/tech-specs/supported-nginx-versions.md index cc6d9d4f9..ddabfd9b2 100644 --- a/content/includes/nim/tech-specs/supported-nginx-versions.md +++ b/content/includes/nim/tech-specs/supported-nginx-versions.md @@ -1,5 +1,7 @@ --- docs: DOCS-1075 +files: +- content/nim/fundamentals/tech-specs.md --- NGINX Instance Manager supports the following NGINX Open Source and NGINX Plus versions: @@ -8,7 +10,7 @@ NGINX Instance Manager supports the following NGINX Open Source and NGINX Plus v | NGINX Instance Manager | NGINX OSS | NGINX Plus | | ---------------------- | ----------- | ---------- | -| 2.18.0 and later | 1.18–1.25.1 | R31–R33 | +| 2.18.0 and later | 1.18–1.28.0 | R31–R34 | | 2.16.0–2.17.x | 1.18–1.25.1 | R31–R32 | | 2.7.0–2.15.x | 1.18–1.25.1 | R21–R30 | | 2.0.0–2.6.0 | 1.18–1.21.6 | R21–R27 | diff --git a/content/nim/admin-guide/_index.md b/content/nim/admin-guide/_index.md index 5dafe4707..431e4278e 100644 --- a/content/nim/admin-guide/_index.md +++ b/content/nim/admin-guide/_index.md @@ -1,5 +1,5 @@ --- -title: Platform administration +title: Administer your platform weight: 40 url: /nginx-instance-manager/admin-guide/ --- \ No newline at end of file diff --git a/content/nim/admin-guide/license/add-license.md b/content/nim/admin-guide/add-license.md similarity index 88% rename from content/nim/admin-guide/license/add-license.md rename to content/nim/admin-guide/add-license.md index 09373655f..7d9aa0f2a 100644 --- a/content/nim/admin-guide/license/add-license.md +++ b/content/nim/admin-guide/add-license.md @@ -1,12 +1,14 @@ --- docs: DOCS-789 -title: Add license +title: Add a license (connected) toc: true -weight: 1 +weight: 10 type: - how-to --- +{{< note >}}For disconnected environments, see [Add a license (disconnected)]({{< ref "nim/disconnected/add-license-disconnected-deployment.md" >}}).{{< /note >}} + ## Overview To unlock all of the features in NGINX Instance Manager, you’ll need to add a JSON Web Token (JWT) license from MyF5. This guide shows you how to set up your network for reporting, download the license file, and apply it to NGINX Instance Manager. If needed, you can also cancel the license at any time. diff --git a/content/nim/admin-guide/license/_index.md b/content/nim/admin-guide/license/_index.md deleted file mode 100644 index bdaf856b0..000000000 --- a/content/nim/admin-guide/license/_index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: License and usage reporting -weight: 1 -url: /nginx-instance-manager/admin-guide/license/ -cascade: - type: "nim-r33" ---- \ No newline at end of file diff --git a/content/nim/admin-guide/rbac/overview-rbac.md b/content/nim/admin-guide/rbac/overview-rbac.md index c49e14463..be72b3013 100644 --- a/content/nim/admin-guide/rbac/overview-rbac.md +++ b/content/nim/admin-guide/rbac/overview-rbac.md @@ -24,7 +24,7 @@ The following are essential concepts related to RBAC: ## Features {#features} -NGINX Instance Manager provides a range of capabilities called features, which system administrators can manage using role-based access control (RBAC). The availability of some features depends on your license. For more information on licensing, see the [Add a license]({{< ref "/nim/admin-guide/license/add-license.md" >}}) topic. +NGINX Instance Manager provides a range of capabilities called features, which system administrators can manage using role-based access control (RBAC). The availability of some features depends on your license. For more information on licensing, see the [Add a license]({{< ref "/nim/admin-guide/add-license.md" >}}) topic. ### NGINX Instance Manager features diff --git a/content/nim/admin-guide/license/report-usage-connected-deployment.md b/content/nim/admin-guide/report-usage-connected-deployment.md similarity index 85% rename from content/nim/admin-guide/license/report-usage-connected-deployment.md rename to content/nim/admin-guide/report-usage-connected-deployment.md index d57939516..6ce813807 100644 --- a/content/nim/admin-guide/license/report-usage-connected-deployment.md +++ b/content/nim/admin-guide/report-usage-connected-deployment.md @@ -1,8 +1,8 @@ --- -title: Report usage data to F5 +title: Report usage data to F5 (connected) draft: false description: '' -weight: 10 +weight: 20 toc: true docs: DOCS-1650 personas: @@ -14,6 +14,8 @@ type: - how-to --- +{{< note >}}For disconnected environments, see [Report usage data to F5 (disconnected)]({{< ref "nim/disconnected/report-usage-disconnected-deployment.md" >}}).{{< /note >}} + ## Overview In environments where NGINX Instance Manager has internet access but NGINX Plus doesn’t, NGINX Plus sends usage data to NGINX Instance Manager. NGINX Instance Manager will automatically send the usage reports to F5 for verification, or you can choose to send them manually. @@ -22,8 +24,6 @@ In environments where NGINX Instance Manager has internet access but NGINX Plus See the steps below to configure NGINX Plus to report usage data to NGINX Instance Manager and how to submit the report to F5 for verification. -{{}}If your deployment of NGINX Instance Manager doesn’t allow internet access, follow the steps in [Report usage data in network-restricted environments]({{< ref "nim/disconnected/report-usage-disconnected-deployment.md" >}}) to report usage data to F5.{{}} - --- ## Before you begin @@ -49,7 +49,7 @@ To configure NGINX Plus (R33 and later) to report usage data to NGINX Instance M ### Automatic reporting -When you [add your JSON Web Token (JWT)]({{< ref "nim/admin-guide/license/add-license.md" >}}) to NGINX Instance Manager, usage reporting is enabled by default. +When you [add your JSON Web Token (JWT)]({{< ref "nim/admin-guide/add-license.md" >}}) to NGINX Instance Manager, usage reporting is enabled by default. NGINX Instance Manager will automatically report subscription entitlement and usage data to F5 if internet access is available. diff --git a/content/nim/deploy/_index.md b/content/nim/deploy/_index.md index 0cd726566..37404e495 100644 --- a/content/nim/deploy/_index.md +++ b/content/nim/deploy/_index.md @@ -1,5 +1,5 @@ --- -title: Deploy +title: Deploy in a connected environment description: weight: 20 url: /nginx-instance-manager/deploy/ diff --git a/content/nim/deploy/docker/deploy-nginx-instance-manager-docker-compose.md b/content/nim/deploy/docker/deploy-nginx-instance-manager-docker-compose.md index aba19ad72..8e8a3fe99 100644 --- a/content/nim/deploy/docker/deploy-nginx-instance-manager-docker-compose.md +++ b/content/nim/deploy/docker/deploy-nginx-instance-manager-docker-compose.md @@ -10,48 +10,67 @@ type: ## Overview -This guide will show you how to deploy and use F5 NGINX Instance Manager in Docker using [Docker Compose](https://docs.docker.com/compose/). +This guide shows you how to run NGINX Instance Manager using [Docker Compose](https://docs.docker.com/compose/). -This NGINX Instance Manager docker compose deployment is a single Docker image containing NGINX Instance Manager, Security Monitoring, and the latest App Protect compilers, which is orchestrated using a Docker Compose docker-compose.yaml definition. +You can deploy it in two ways: -The ClickHouse database is deployed in a separate container to improve resilience and make this a fault tolerant solution. You can also configure persistent storage +- **Standard mode** includes full metrics and dashboards. This setup runs ClickHouse in a separate container. +- **Lightweight mode** (new in 2.20.0) skips ClickHouse entirely. It’s ideal if you don’t need monitoring data or want a simpler setup. This reduces system requirements and avoids the work of managing a metrics database. You can add ClickHouse later if your needs change. + +Both modes use a pre-built Docker image that includes NGINX Instance Manager, Security Monitoring, and the latest NGINX App Protect compilers. + +If you use the standard setup, ClickHouse runs in its own container. This helps with fault tolerance and keeps data separate. You can also use persistent storage. --- ## What you need -- A working version of [Docker](https://docs.docker.com/get-docker/) -- Your NGINX Instance Manager subscription's JSON Web Token from [MyF5](https://my.f5.com/manage/s/subscriptions) You can use the same JSON Web Token as NGINX Plus in your MyF5 portal. -- This pre-configured `docker-compose.yaml` file: - - {{}} {{}}. +Before you begin, make sure you have the following: + +- [Docker](https://docs.docker.com/get-docker/) installed on your system. +- A JSON Web Token (JWT) from your [MyF5 subscriptions page](https://my.f5.com/manage/s/subscriptions). This is the same token used for NGINX Plus. +- The right `docker-compose.yaml` file for your setup: + - For **standard mode** (with metrics and dashboards): + {{}} {{}} + - For **lightweight mode** (no ClickHouse, no metrics): + {{}} {{}} + +{{< note >}} If you're not sure which one to use, start with lightweight mode. You can always switch later by changing the Compose file and setting `ENABLE_METRICS: "true"`.{{< /note >}} --- ## Minimum requirements -Deploying NGINX Instance Manager with docker requires a minimum of 4 CPU cores and 4 GB of memory for basic use cases. However, every environment is unique, primarily due to variations in the NGINX configurations being managed. For instance, managing NGINX instances with hundreds of configuration files or those with WAF (NGINX App Protect) enabled can significantly increase resource demands. +Your system needs enough resources to run NGINX Instance Manager based on the mode you choose: + +| Deployment mode | CPU cores | Memory | ClickHouse required | +|-----------------|-----------|--------|---------------------| +| Standard | 4 | 4 GB | Yes | +| Lightweight | (Lower) | (Lower)| No | -If your use case is limited to usage tracking without active management or agent communication, the minimum requirements should suffice. For more complex deployments, we recommend reviewing the technical specifications guide to ensure the resources allocated are sufficient to handle an increased workload, particularly for the ClickHouse database, which may need to manage a higher volume of reads and writes. +Standard mode requires a minimum of 4 CPU cores and 4 GB of memory. This setup includes ClickHouse, which handles metrics and dashboards. Depending on your NGINX footprint, you may need more resources, especially for environments with many configuration files or NGINX App Protect enabled. + +Lightweight mode removes ClickHouse, which lowers memory and CPU usage. While there’s no official minimum, users with basic instance management needs may see success with fewer resources. Test in your environment before committing to a smaller footprint. + +{{< note >}} If you're not sure which mode to use, start with lightweight mode. It's easier to set up, and you can switch to standard mode later by reintroducing ClickHouse. {{< /note >}} ## Before you start {{< include "/nim/decoupling/note-legacy-nms-references.md" >}} -### Set up Docker for NGINX container registry +### Log in to the NGINX container registry -To set up Docker to communicate with the NGINX container registry located at `private-registry.nginx.com`, follow these steps: +Both standard and lightweight deployments use a private image hosted at `private-registry.nginx.com`. You need to log in before running Docker Compose. -{{< include "/nim/docker/docker-registry-login.md" >}} +To set up Docker to communicate with the NGINX container registry: -### Compose deployment - -{{}} +{{< include "/nim/docker/docker-registry-login.md" >}} -If you are configuring a **forward proxy**, follow the steps in the [Forward Proxy Configuration Guide]({{< ref "nim/system-configuration/configure-forward-proxy.md" >}}) to modify `docker-compose.yaml` with the correct proxy settings **before** deploying NGINX Instance Manager. +### Deploy NGINX Instance Manager -{{}} +If you're using a forward proxy, update the Compose file **before** deploying. Follow the steps in the [Forward Proxy Configuration Guide]({{< ref "nim/system-configuration/configure-forward-proxy.md" >}}). -Go to the directory where you downloaded `docker-compose.yaml`. Use the following commands to log in to `private-registry.nginx.com` and deploy NGINX Instance Manager. +Go to the directory where you downloaded your `docker-compose.yaml` file, and run the following commands: ```shell docker login private-registry.nginx.com --username= --password=none @@ -59,52 +78,84 @@ echo "admin" > admin_password.txt docker compose up -d ``` -If the deployment succeeds, you’ll see output similar to this: +The deployment output will vary depending on the mode: + +#### Standard mode (with metrics) ```text [+] Running 6/6 - ✔ Network nim_clickhouse Created 0.1s - ✔ Network nim_external_network Created 0.2s - ✔ Network nim_default Created 0.2s - ✔ Container nim-precheck-1 Started 0.8s - ✔ Container nim-clickhouse-1 Healthy 6.7s - ✔ Container nim-nim-1 Started 7.4s + ✔ Network nim_clickhouse Created + ✔ Network nim_external_network Created + ✔ Network nim_default Created + ✔ Container nim-precheck-1 Started + ✔ Container nim-clickhouse-1 Healthy + ✔ Container nim-nim-1 Started ``` +#### Lightweight mode (no ClickHouse) + +```text +[+] Running 3/3 + ✔ Network nim_default Created + ✔ Network nim_external_network Created + ✔ Container nim-nim-1 Started +``` + +In lightweight mode, only the NGINX Instance Manager service runs. ClickHouse and related containers are removed by design. + ### Supported environment variables +You can control Instance Manager behavior by setting environment variables in the `docker-compose.yaml` file. Here’s a summary of commonly used variables: + {{< include "nim/docker/docker-compose-env-vars.md" >}} -
+### Stop or remove services -{{}} -For details on configuring a forward proxy, see the [Forward Proxy Configuration Guide]({{< ref "nim/system-configuration/configure-forward-proxy.md" >}}). -{{}} +To stop NGINX Instance Manager, go to the directory where you downloaded your `docker-compose.yaml` file. -### Compose stop or tear down +If you used `docker compose up -d` to start services: -Navigate to the directory where you downloaded `docker-compose.yaml`. If you started NIM with `docker compose up -d`, stop NIM services once you've finished with them by running `docker compose stop`. You can bring everything down, removing the containers entirely, with the `docker compose down` command. +- Run `docker compose stop` to pause the containers +- Run `docker compose down` to remove them completely ```shell docker compose down ``` -``` + +The shutdown output will vary depending on the mode: + +#### Standard mode (with metrics) + +```text [+] Running 6/6 - ✔ Container nim-nim-1 Removed 30.6s - ✔ Container nim-clickhouse-1 Removed 1.4s - ✔ Container nim-precheck-1 Removed 0.0s - ✔ Network nim_default Removed 0.9s - ✔ Network nim_external_network Removed 0.4s - ✔ Network nim_clickhouse Removed 0.6s + ✔ Container nim-nim-1 Removed + ✔ Container nim-clickhouse-1 Removed + ✔ Container nim-precheck-1 Removed + ✔ Network nim_default Removed + ✔ Network nim_external_network Removed + ✔ Network nim_clickhouse Removed +``` + +#### Lightweight mode (no ClickHouse) + +```text +[+] Running 3/3 + ✔ Container nim-nim-1 Removed + ✔ Network nim_default Removed + ✔ Network nim_external_network Removed ``` +In lightweight mode, only the core service and networks are removed. ClickHouse and precheck containers aren’t present. + --- ## Secrets -In the same `docker-compose.yaml` file, you can modify the following credentials: +You can define secrets in the `docker-compose.yaml` file to set the admin password, optional access credentials, and TLS certificates. + +### Required -Set the admin password (required) +Set the admin password: ```yaml secrets: @@ -112,14 +163,17 @@ secrets: file: admin_password.txt ``` -Pass a custom `.htpasswd` file (Optional) +### Optional + +Use a custom `.htpasswd` file to control access to the user interface: ```yaml +secrets: nim_credential_file: file: nim_creds.txt ``` -Optionally, you can also set the external SSL certificate, key, and CA files, in PEM format for the NGINX Instance Manager Ingress proxy. +Use your own TLS certificate, key, and CA file for the ingress proxy: ```yaml secrets: @@ -135,64 +189,122 @@ secrets: ## Backup -Once you've set up your Docker containers, use the following command to back them up: +You can create a backup of NGINX Instance Manager at any time using the built-in `nim-backup` command. This works for both standard and lightweight deployments. + +Run the following command from your Docker host: ```shell -~$ docker exec nim-nim-1 nim-backup -... -Backup has been successfully created: /data/backup/nim-backup-.tgz +docker exec nim-nim-1 nim-backup ``` -If your system uses named volumes, inspect the `Mountpoint`. Alternatively, if you're using a shared NFS volume, then collect the data directly from the mount point. +If successful, you’ll see a message like: -```shell -~/compose$ docker inspect volume nim_nim-data | jq '.[0].Mountpoint' -"/var/lib/docker/volumes/nim_nim-data/_data" -ubuntu@ip-
:~/compose$ sudo ls -l /var/lib/docker/volumes/nim_nim-data/_data/backup --rw-r--r-- 1 root root 5786953 Sep 27 02:03 nim-backup-.tgz +```text +Backup has been successfully created: /data/backup/nim-backup-.tgz ``` +To locate the backup file: + +- **If using named volumes** + + Inspect the volume to find its mount point: + + ```shell + docker inspect volume nim_nim-data | jq '.[0].Mountpoint' + ``` + + Then list the backup directory: + + ```shell + sudo ls -l /var/lib/docker/volumes/nim_nim-data/_data/backup + ``` + + Example output: + + ```text + -rw-r--r-- 1 root root 5786953 Sep 27 02:03 nim-backup-2024-06-11.tgz + ``` + +- **If using NFS** + + Check the mount path directly, such as: + + ```shell + ls -l /mnt/nfs_share/data/backup + ``` + + --- ## Restore -Before you can restore a backup, set your containers to maintenance mode in the same `docker-compose.yaml` file: +To restore a backup, follow these steps: -```yaml +1. Enable maintenance mode: + + In your `docker-compose.yaml` file, set the following: + + ```yaml environment: NIM_MAINTENANCE: "true" -``` + ``` -```shell -~$ docker exec nim-nim-1 nim-restore /data/backup/nim-backup-.tgz -... -NGINX Instance Manager has been restored. -``` +2. Run the restore command: + + ```shell + docker exec nim-nim-1 nim-restore /data/backup/nim-backup-.tgz + ``` + +3. After the restore process finishes, disable maintenance mode. + + In `docker-compose.yaml`, change the value back: + + ```yaml + environment: + NIM_MAINTENANCE: "false" + ``` + +4. Restart the container: -Once the process is complete set `NIM_MAINTENANCE` to `false` and then run `docker-compose up -d`. + ```shell + docker compose up -d + ``` --- ## Storage -By default, the storage uses named volumes. Alternatively, you can use optional `driver_opts` settings to support other storage formats such as NFS. -For all storage volumes, make sure to mount them, before running `docker compose up -d`. For a mounted NFS volume, you might use the following commands: +NGINX Instance Manager uses named Docker volumes by default to persist data. You can also configure NFS or other storage backends using `driver_opts`. + +### Default volumes + +The standard `docker-compose.yaml` file defines these named volumes: + +- `nim-data`: stores configuration and state for Instance Manager +- `clickhouse-data`: used only in standard mode to persist metrics + +In lightweight mode, `clickhouse-data` is not needed and should be removed from the Compose file. + +### Example: use NFS for storage + +Before you run `docker compose up -d`, make sure your NFS volumes are mounted: ```shell -~$ sudo mount -t nfs <>:/mnt/nfs_share/clickhouse /mnt/nfs_share/clickhouse -~$ sudo mount -t nfs <>:/mnt/nfs_share/data /mnt/nfs_share/data +sudo mount -t nfs <>:/mnt/nfs_share/data /mnt/nfs_share/data +sudo mount -t nfs <>:/mnt/nfs_share/clickhouse /mnt/nfs_share/clickhouse ``` +Update the volumes section in your Compose file: + ```yaml volumes: - # By default docker compose will create a named volume - # Refer to https://docs.docker.com/reference/compose-file/volumes/ for additional storage options such as NFS nim-data: driver: local driver_opts: type: "nfs" o: "addr=<>,rw" device: ":/mnt/nfs_share/data" + clickhouse-data: driver: local driver_opts: @@ -201,11 +313,28 @@ volumes: device: ":/mnt/nfs_share/clickhouse" ``` +If you’re using lightweight mode, omit the `clickhouse-data` section entirely. + --- -## Support Data +## Gather support data + +If you [contact support]({{< ref "nim/support/contact-support.md" >}}), it's helpful to include logs and a current backup. These commands capture useful diagnostic data. -In case of problems, it's a good practice to: +### Get recent logs + +Save logs from the past 24 hours: + +```shell +docker compose logs --since 24h > my-logs-$(date +%Y-%m-%d).txt +``` + +### Create a fresh backup + +Run the backup command to capture the current state: + +```shell +docker exec nim-nim-1 nim-backup +``` -- Collect logs `docker-compose logs --since 24h > my-logs-$(date +%Y-%m-%d).txt` -- Collect backup information `docker exec nim-nim-1 nim-backup` +This creates a `.tgz` file inside the container under `/data/backup/`, which you can extract as described in the [Backup](#backup) section. \ No newline at end of file diff --git a/content/nim/deploy/kubernetes/deploy-using-helm.md b/content/nim/deploy/kubernetes/deploy-using-helm.md index 8d39db8eb..3b36c5a0e 100644 --- a/content/nim/deploy/kubernetes/deploy-using-helm.md +++ b/content/nim/deploy/kubernetes/deploy-using-helm.md @@ -9,230 +9,236 @@ type: ## Overview -{{< include "/nim/decoupling/note-legacy-nms-references.md" >}} +You can deploy F5 NGINX Instance Manager on Kubernetes using Helm. This method is quick, scalable, and supports both standard and lightweight modes. -This guide explains how to deploy F5 NGINX Instance Manager on a Kubernetes or OpenShift cluster using Helm. You’ll learn how to download and use Docker images and customize your deployment. +### New in 2.20.0 -{{< note >}} Starting in NGINX Instance Manager 2.19, you can deploy NGINX Instance Manager on an OpenShift cluster using Helm. {{< /note >}} +Starting with version 2.20.0, NGINX Instance Manager supports **lightweight mode**, which skips ClickHouse and disables metrics collection, ideal for simpler setups or resource-limited environments. -### About Helm +- Lightweight mode requires NGINX Agent v2.41.1 or later. -Helm charts are pre-configured packages of Kubernetes resources deployed with a single command. They let you define, install, and upgrade Kubernetes applications easily. +{{< call-out "note" "Chart renamed in NIM 2.20.0" >}} +The Helm chart has been renamed from `nginx-stable/nms-hybrid` to `nginx-stable/nim`. +Make sure to update your chart references if you’re using version 2.20.0 or later. +{{< /call-out >}} -Helm charts consist of files that describe a group of related Kubernetes resources, like deployments, services, and ingress. They also allow you to manage dependencies between applications, making it easier to deploy multi-tier or complex applications. --- -## Before you begin +## Requirements To deploy NGINX Instance Manager using a Helm chart, you need: -{{< bootstrap-table "table table-striped table-bordered" >}} -| Requirements | Notes | -| ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Docker 20.10 or later (linux/amd64) | [Docker documentation](https://docs.docker.com/get-docker) | -| Kubernetes 1.21.3 or later (linux/amd64) | Ensure your client can [access the Kubernetes API server](https://kubernetes.io/docs/concepts/security/controlling-access/). The Helm chart will enable persistent storage using the default storage class in your Kubernetes cluster. More info is available in [Dynamic Volume Provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/). | -| kubectl 1.21.3 or later | [kubectl documentation](https://kubernetes.io/docs/tasks/tools/#kubectl) | -| Helm 3.10.0 or later | [Helm installation guide](https://helm.sh/docs/intro/install/) | -| OpenSSL 1.1.1 or later | [OpenSSL source](https://www.openssl.org/source/) | -| `tar` 1.20 or later | The `tar` tool is usually installed by default. Check your version with `tar --version`. If `tar` is missing or outdated, install it from your distribution’s package manager (like YUM for CentOS/RHEL or APT for Debian/Ubuntu). | - -{{< /bootstrap-table >}} +## Requirements ---- +To deploy NGINX Instance Manager using a Helm chart, you need: -## Get the NGINX Instance Manager images +{{< bootstrap-table "table table-striped table-bordered" >}} +| Requirements | Notes | +|----------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Docker 20.10 or later (linux/amd64) | [Docker documentation](https://docs.docker.com/get-docker/) | +| Kubernetes 1.21.3 or later (linux/amd64) | Ensure your client can [access the Kubernetes API server](https://kubernetes.io/docs/concepts/overview/components/#kube-apiserver). Helm uses the default storage class for persistent volume provisioning. | +| `kubectl` 1.21.3 or later | [kubectl documentation](https://kubernetes.io/docs/reference/kubectl/) | +| Helm 3.10.0 or later | [Helm installation guide](https://helm.sh/docs/intro/install/) | +| OpenSSL 1.1.1 or later | [OpenSSL source](https://www.openssl.org/source/) | +| `tar` 1.20 or later | The `tar` tool is usually installed by default. Check with `tar --version`. | +| `values.yaml` file with `nmsClickhouse.mode` | Optional. Defaults to `internal`. Set to `external` or `disabled` to use an external ClickHouse instance or enable lightweight mode. In `external` mode, set `nim.externalClickhouse.address` to your ClickHouse host. | +| NGINX subscription JWT | Required to authenticate with `private-registry.nginx.com` to pull the image. Download your JWT from [MyF5](https://my.f5.com/manage/s/) under **My Products & Plans > Subscriptions**. | +{{}} -### Using Docker +--- -### Using Helm with a JWT token +## Set up image registry access for NGINX Instance Manager You can use your NGINX JWT as a Docker configuration secret with Helm charts. Create a Docker registry secret on the cluster, using the JWT token as the username and `none` as the password. The Docker server is `private-registry.nginx.com`. -{{< note >}} Make sure there are no extra characters or spaces when copying the JWT token. They can invalidate the token and cause 401 errors during authentication. {{< /note >}} +{{< call-out "note" "Note" >}} +Make sure there are no extra characters or spaces when copying the JWT token. They can invalidate the token and cause 401 errors during authentication. +{{< /call-out >}} -- **Kubernetes**: +### Kubernetes - ```shell - kubectl create namespace nms - ``` - - ```shell - kubectl create secret docker-registry regcred \ +```shell +kubectl create namespace nms +kubectl create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ - --docker-username= \ + --docker-username= \ --docker-password=none \ -n nms - ``` - -- **OpenShift**: +``` - ```shell - oc new-project nms - ``` +### OpenShift - ```shell - oc create secret docker-registry regcred \ +```shell +oc new-project nms +oc create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ - --docker-username= \ + --docker-username= \ --docker-password=none \ -n nms - ``` - -{{< warning >}} - -You might see a warning about `--password` being insecure. - -This can be ignored (since no password is used), but if others have access to this system, delete the JWT token and clear your shell history after deployment. +``` -{{< /warning >}} +{{< call-out "note" "Note" >}} +You might see a warning that `--password` is insecure. In this case, it’s safe to ignore—none is used as a placeholder. -To confirm the secret is created: +As a best practice, you can delete the JWT token and clear your shell history after deployment if others have access to the system. +{{< /call-out >}} -- **Kubernetes**: +### Confirm the secret - ```shell - kubectl get secret regcred --output=yaml -n nms - ``` +- Kubernetes -- **OpenShift**: + ```shell + kubectl get secret regcred --output=yaml -n nms + ``` - ```shell - oc get secret regcred --output=yaml -n nms - ``` +- OpenShift + ```shell + oc get secret regcred --output=yaml -n nms + ``` -You can now use this secret for Helm deployments and point the charts to the public registry. +You can now use this secret for Helm deployments and point the chart to the private registry. --- -## Add the Helm repository - -{{< note >}} You need Helm 3.10.0 or later for these steps. {{< /note >}} +## Add the repository {#add-repository} -Run these commands to install the NGINX Instance Manager chart from the Helm repository: +Run the following commands to add the official NGINX Helm repository and update your local chart list. ```shell helm repo add nginx-stable https://helm.nginx.com/stable helm repo update ``` -The first command adds the `nginx-stable` repository to your local Helm repo list. The second updates the list to ensure you have the latest versions of the charts. - --- -## Create a Helm deployment values.yaml file +## Create a values.yaml file {#configure-values-yaml} -The `values.yaml` file customizes the Helm chart installation without modifying the chart itself. You can use it to specify image repositories, environment variables, resource requests, and other settings. +Create a file named `values.yaml` using the following example. This file customizes your NGINX Instance Manager deployment with Helm. -1. Create a `values.yaml` file similar to this example: +The values file lets you: - - In the `imagePullSecrets` section, add the credentials for your private Docker registry. - - Change the version tag to the version of NGINX Instance Manager you would like to install. See "Install the chart" below for versions. - - If deploying on OpenShift, add the `openshift.enabled: true` setting. - - {{< see-also >}} For details on creating a secret, see Kubernetes [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/). {{}} - - ```yaml - imagePullSecrets: - - name: regcred - apigw: - image: - repository: private-registry.nginx.com/nms/apigw - tag: - core: - image: - repository: private-registry.nginx.com/nms/core - tag: - dpm: - image: - repository: private-registry.nginx.com/nms/dpm - tag: - ingestion: - image: - repository: private-registry.nginx.com/nms/ingestion - tag: - integrations: - image: - repository: private-registry.nginx.com/nms/integrations - tag: - secmon: - image: - repository: private-registry.nginx.com/nms/secmon - tag: - utility: - image: - repository: private-registry.nginx.com/nms/utility - tag: - ``` - - {{< note >}} Starting in NGINX Instance Manager 2.19, the `secmon` pod is included in the NGINX Instance Manager deployment. {{< /note >}} - -2. Save and close the `values.yaml` file. +- Set the deployment mode +- Provide registry credentials +- Specify image sources for each NIM service ---- +Set `nmsClickhouse.mode` to control ClickHouse deployment: -## Enabling OpenShift +| Mode | Description | +|------------|--------------------------------------------------------------------------------------------| +| `internal` | Deploys ClickHouse in the cluster (default). | +| `external` | Connects to an external ClickHouse instance and requires `nim.externalClickhouse.address`. | +| `disabled` | Disables ClickHouse and enables lightweight mode (no metrics). | -If deploying on OpenShift, include this setting in the `values.yaml` file: +{{< call-out "note" "See also" >}} +See the [Helm chart configuration settings]( +https://docs.nginx.com/nginx-instance-manager/deploy/kubernetes/helm-config-settings/ +) guide for a complete list of chart parameters. +{{< /call-out >}} ```yaml -openshift: - enabled: true +nmsClickhouse: + mode: internal # options: internal, external, disabled + +# when mode is external, uncomment and set this: +# externalClickhouse: +# address: : + +imagePullSecrets: + - name: regcred + +apigw: + image: + repository: private-registry.nginx.com/nms/apigw + tag: 2.20.0 + +core: + image: + repository: private-registry.nginx.com/nms/core + tag: 2.20.0 + +dpm: + image: + repository: private-registry.nginx.com/nms/dpm + tag: 2.20.0 + +ingestion: + image: + repository: private-registry.nginx.com/nms/ingestion + tag: 2.20.0 + +integrations: + image: + repository: private-registry.nginx.com/nms/integrations + tag: 2.20.0 + +secmon: + image: + repository: private-registry.nginx.com/nms/secmon + tag: 2.20.0 + +utility: + image: + repository: private-registry.nginx.com/nms/utility + tag: 2.20.0 ``` -{{< note >}} The NIM deployment on OpenShift has been tested with OpenShift v4.13.0 Server. {{< /note >}} +These values are required when pulling images from the NGINX private registry. The chart does not auto-resolve image tags. Update the tag: fields to match the NGINX Instance Manager version you want to install. -### How OpenShift handles security constraints +Use the file with the `-f values.yaml` flag when installing the chart. -When `openshift.enabled: true` is set in the `values.yaml` file, the NGINX Instance Manager deployment automatically creates a **custom [Security Context Constraints](https://docs.redhat.com/en/documentation/openshift_container_platform/4.13/html/authentication_and_authorization/managing-pod-security-policies) (SCCs)** and links it to the Service Account used by all pods. +{{< call-out "note" "OpenShift support" >}} +OpenShift support was added in NGINX Instance Manager 2.19. To enable it, add the setting `openshift.enabled: true` to your `values.yaml` file. +For more details, see [Appendix: OpenShift security constraints](#appendix-openshift-security-constraints). +{{< /call-out >}} -By default, OpenShift enforces strict security policies that require containers to run as **non-root** users. The NGINX Instance Manager deployment needs specific user IDs (UIDs) for certain services, such as **1000** for `nms` and **101** for `nginx` and `clickhouse`. Since the default SCCs do not allow these UIDs, a **custom SCC** is created. This ensures that the deployment can run with the necessary permissions while maintaining OpenShift’s security standards. The custom SCC allows these UIDs by setting the `runAsUser` field, which controls which users can run containers. +--- -{{< note >}} If you’re encountering errors with the custom SCC, you may not have permissions to access the Security Context Constraints resource. Please contact a Cluster Administrator to request access, either through a cluster role binding or by adjusting your user role. {{< /note >}} +## Install the chart -To verify that the custom SCC has been created, after installing the helm chart, run: +Install NGINX Instance Manager using Helm. The `adminPasswordHash` sets the default admin password. ```shell -oc get scc nms-restricted-v2-scc --output=yaml +helm install nim nginx-stable/nim \ + -n nim \ + --create-namespace \ + --set adminPasswordHash=$(openssl passwd -6 '') \ + -f \ + --version \ + --wait ``` ---- - -## Install the chart +- Replace `` with your preferred admin password. +- Replace `` with the path to your customized values.yaml file. +- Replace `` with the version you want to install (for example, `2.20.0`). -Run the `helm install` command to deploy NGINX Instance Manager: +**Note:** You can set the ClickHouse mode at install time instead of editing `values.yaml`: -1. Replace `` with the path to your `values.yaml` file. -1. Replace `YourPassword123#` with a secure password (containing a mix of uppercase, lowercase letters, numbers, and special characters). +For lightweight mode (no ClickHouse): - {{< important >}} Remember to save the password for future use. Only the encrypted password is stored, and there's no way to recover or reset it if lost. {{< /important >}} +```shell +--set nmsClickhouse.mode=disabled +``` -(Optional) Replace `` with the desired chart version. If omitted, the latest version will be installed. Currently only version 2.19.0 is supported. +For external ClickHouse: ```shell -helm install -n nms \ ---set adminPasswordHash=$(openssl passwd -6 'YourPassword123#') \ -nms nginx-stable/nms-hybrid \ ---create-namespace \ --f \ -[--version ] \ ---wait +--set nmsClickhouse.mode=external \ +--set nim.externalClickhouse.address=: ``` ---- - -## Validate the deployment +**Validate the deployment** -Check the status of the deployment: +After installation, run the following command to confirm the deployment was successful: ```shell -helm -n nms status nms +helm status nim -n nim ``` -The status should show `STATUS: deployed` if successful. +You should see `STATUS: deployed` in the output. --- @@ -246,88 +252,100 @@ The status should show `STATUS: deployed` if successful. A valid license is required to use all NGINX Instance Manager features. -For instructions on downloading and applying a license, see [Add a License]({{< ref "/nim/admin-guide/license/add-license.md" >}}). +For instructions on downloading and applying a license, see [Add a License]({{< ref "/nim/admin-guide/add-license.md" >}}). --- -## Upgrade NGINX Instance Manager +### Upgrade NGINX Instance Manager -To upgrade: +To upgrade your deployment: -1. [Update the Helm repository list](#add-helm-repository). -1. [Adjust your `values.yaml` file](#create-a-helm-deployment-values.yaml-file) if needed. -1. To upgrade the NGINX Instance Manager deployment, run the following command. This command updates the `nms` deployment with a new version from the `nginx-stable/nms-hybrid` repository. It also hashes the provided password and uses the `values.yaml` file at the path you specify. +1. [Update the Helm repository list](#add-repository). +2. [Review and adjust your `values.yaml` file](#configure-values-yaml) as needed. +3. Run the following command to upgrade the deployment. This command uses the current chart version from the `nginx-stable/nim` repository and applies the configuration from your `values.yaml` file. -(Optional) Replace `` with the desired chart version. If omitted, the latest version will be installed. Currently only version 2.19.0 is supported. +```shell +helm upgrade nim nginx-stable/nim \ + -n nim \ + --set adminPasswordHash=$(openssl passwd -6 '') \ + -f \ + --version \ + --wait +``` - ```bash - helm upgrade -n nms \ - --set adminPasswordHash=$(openssl passwd -6 'YourPassword123#') \ - nms nginx-stable/nms-hybrid \ - -f \ - [--version ] \ - --wait - ``` +- Replace `` with a secure password that includes uppercase and lowercase letters, numbers, and special characters. +- Replace `` with the full path to your configuration file. +- Replace `` with the version of NGINX Instance Manager you want to upgrade to. - - Replace `` with the path to the `values.yaml` file you created]({{< ref "/nim/deploy/kubernetes/deploy-using-helm.md#configure-chart" >}}). - - Replace `YourPassword123#` with a secure password that includes uppercase and lowercase letters, numbers, and special characters. - {{}} Save this password for future use. Only the encrypted password is stored in Kubernetes, and you can’t recover or reset it later. {{}} - - (Optional) Replace with the desired version number. If you don’t specify a version, the latest version will be installed. +{{< call-out "important" "Save the password!" >}} +Only the encrypted version of the admin password is stored in Kubernetes. If you lose it, it can’t be recovered or reset. +Make sure to save the password in a secure place. +{{< /call-out >}} +{{< call-out "note" "Upgrading from earlier versions" >}} +If you’re upgrading from a deployment that used the legacy `nms` chart or release name, you’ll need to update the chart reference and adjust the release name as needed. +The latest chart is now called `nginx-stable/nim`, and `nim` is the recommended release name. +{{< /call-out >}} --- ## Uninstall NGINX Instance Manager {#helm-uninstall-nim} -To uninstall: +To uninstall NGINX Instance Manager, run: -```bash -helm uninstall --namespace nms nms +```shell +helm uninstall -n ``` -This deletes the `nms` application and all associated Kubernetes resources. +This command removes the deployment and all Kubernetes resources managed by the Helm chart. + +For example, if you used the default release and namespace names: + +```shell +helm uninstall nim -n nim +``` --- ## Manage network policies -To apply network policies for NGINX Instance Manager, ensure Kubernetes has a [network plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) installed before the Helm chart installation. +If you plan to use network policies, make sure your Kubernetes cluster has a supported [network plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) installed **before** you install the Helm chart. -By default, the following network policies will be created in the release namespace: +By default, the Helm chart creates a set of network policies for NGINX Instance Manager in the deployment namespace. + +To view them: - **Kubernetes**: - ```shell - kubectl get netpol -n nms - ``` + ```shell + kubectl get netpol -n + ``` - **OpenShift**: - ```shell - oc get netpol -n nms - ``` - **Output**: - - ```text - NAME POD-SELECTOR AGE - apigw app.kubernetes.io/name=apigw 4m47s - clickhouse app.kubernetes.io/name=clickhouse 4m47s - core app.kubernetes.io/name=core 4m47s - dpm app.kubernetes.io/name=dpm 4m47s - ingestion app.kubernetes.io/name=ingestion 4m47s - integrations app.kubernetes.io/name=integrations 4m47s - secmon app.kubernetes.io/name=secmon 4m47s - utility app.kubernetes.io/name=integrations 4m47s - ``` - -To disable network policies, update the `values.yaml` file: + ```shell + oc get netpol -n + ``` + +The number and names of network policies vary depending on the deployment mode (standard vs. lightweight). For example, in standard mode, you might see output like this: + +```text +NAME POD-SELECTOR AGE +apigw app.kubernetes.io/name=apigw 2m +core app.kubernetes.io/name=core 2m +dpm app.kubernetes.io/name=dpm 2m +ingestion app.kubernetes.io/name=ingestion 2m +secmon app.kubernetes.io/name=secmon 2m +``` + +If you’re using lightweight mode, your output may include fewer entries. + +To disable network policies, add the following to your `values.yaml` file: ```yaml -nms-hybrid: - networkPolicies: - # Set this to true to enable network policies for NGINX Instance Manager. - enabled: false +networkPolicies: + enabled: false ``` --- @@ -388,7 +406,7 @@ Run the `helm install` command to deploy NGINX Instance Manager: {{< important >}} Remember to save the password for future use. Only the encrypted password is stored, and there's no way to recover or reset it if lost. {{< /important >}} -3. Replace `` with the desired chart version 1.15.0 or lower. If omitted, it will lead to an unsuccessful deployment as it will try to install the latest vesrion 1.16.0 or later. +3. (Optional) Replace `` with the desired chart version. If omitted, the latest version will be installed. ```shell helm install -n nms \ @@ -410,12 +428,12 @@ To help you choose the right NGINX Instance Manager chart version, see the table To upgrade: -1. [Update the Helm repository list](#add-helm-repository). +1. [Update the Helm repository list](#add-repository). 2. [Adjust your `values.yaml` file](#create-a-helm-deployment-values.yaml-file) if needed. 3. To upgrade the NGINX Instance Manager deployment, run the following command. This command updates the `nms` deployment with a new version from the `nginx-stable/nms` repository. It also hashes the provided password and uses the `values.yaml` file at the path you specify. 4. Replace `` with the desired chart version 1.15.0 or lower. If omitted, it will lead to an unsuccessful deployment as it will try to upgrade to the latest vesrion 1.16.0 or later. - ```bash + ```shell helm upgrade -n nms \ --set nms-hybrid.adminPasswordHash=$(openssl passwd -6 'YourPassword123#') \ nms nginx-stable/nms \ @@ -433,5 +451,33 @@ To upgrade: ## Troubleshooting -For instructions on creating a support package to share with NGINX Customer Support, see [Create a Support Package from a Helm Installation]({{< ref "/nms/support/k8s-support-package.md" >}}). +For instructions on creating a support package to share with NGINX Customer Support, see [Create a Support Package from a Helm Installation]({{< ref "nim/support/k8s-support-package.md" >}}). + +--- + +## Appendix: OpenShift security constraints {#appendix-openshift-security-constraints} + +OpenShift restricts containers from running as root by default. To support NGINX Instance Manager, the Helm chart creates a custom Security Context Constraint (SCC) when you set: + +```yaml +openshift: + enabled: true +``` + +This ensures pods can run with the user IDs required by NGINX Instance Manager services. + +{{< call-out "note" "Note" >}} +If you see permission errors during deployment, your user account might not have access to manage SCCs. Contact a cluster administrator to request access. +{{< /call-out >}} + +To verify that the SCC was created after installing the Helm chart, run: + +```shell +oc get scc nms-restricted-v2-scc --output=yaml +``` + +--- + +## Next steps +- [Add NGINX Open Source and NGINX Plus instances to NGINX Instance Manager]({{< ref "nim/nginx-instances/add-instance.md" >}}) diff --git a/content/nim/deploy/kubernetes/helm-config-settings.md b/content/nim/deploy/kubernetes/helm-config-settings.md index 83ce7682c..dd6a66d7e 100644 --- a/content/nim/deploy/kubernetes/helm-config-settings.md +++ b/content/nim/deploy/kubernetes/helm-config-settings.md @@ -1,95 +1,96 @@ --- docs: DOCS-1112 -title: Configurable Helm settings +title: Configurable helm settings toc: true weight: 300 -type: +nd-content-type: - reference +nd-product: NIM --- -{{< include "/nim/decoupling/note-legacy-nms-references.md" >}} - This reference guide lists the configurable Helm chart parameters and default settings for NGINX Instance Manager. ## NGINX Instance Manager Helm chart settings {#helm-settings} -The following table lists the configurable parameters and default values for NGINX Instance Manager when installing from a Helm chart. - -To modify a configuration for an existing release, run the `helm upgrade` command and use `-f `, where `` is the path to a values file with your desired configuration. - -{{}} - -| Parameter | Description | Default | -|:-----------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------| -| `nms-hybrid.adminPasswordHash` | The hashed value of the password for the admin user.
To generate the hash using `openssl`, run a command like: `openssl passwd -1 "YourPassword123#"` | N/A | -| `nms-hybrid.nmsClickhouse.enabled` | Enable this if external ClickHouse is not used. | `true` | -| `nms-hybrid.nmsClickhouse.fullnameOverride` | Modify the name of ClickHouse resources. | `clickhouse` | -| `nms-hybrid.nmsClickhouse.image.repository` | Repository name and path for the public ClickHouse image. | `clickhouse/clickhouse-server` | -| `nms-hybrid.nmsClickhouse.image.tag` | Tag used for pulling images from the registry. | `21.3.20.1-alpine` | -| `nms-hybrid.nmsClickhouse.image.pullPolicy` | Image pull policy. | `IfNotPresent` | -| `nms-hybrid.nmsClickhouse.user` | Username to connect to the ClickHouse server. | N/A | -| `nms-hybrid.nmsClickhouse.password` | Password for the ClickHouse server. | N/A | -| `nms-hybrid.nmsClickhouse.service.name` | ClickHouse service name. | `clickhouse` | -| `nms-hybrid.nmsClickhouse.service.rpcPort` | ClickHouse service port. | `9000` | -| `nms-hybrid.nmsClickhouse.resources.requests.cpu` | Minimum required CPU to run the ClickHouse server. | `500m` | -| `nms-hybrid.nmsClickhouse.resources.requests.memory` | Minimum required memory to run the ClickHouse server. | `1Gi` | -| `nms-hybrid.nmsClickhouse.persistence.enabled` | Use PVCs to persist ClickHouse data. | `true` | -| `nms-hybrid.nmsClickhouse.persistence.existingClaim` | Name of an existing Persistent Volume Claim (PVC) to use for ClickHouse persistence. | N/A | -| `nms-hybrid.nmsClickhouse.persistence.storageClass` | Storage class to use for creating a ClickHouse PVC. | | -| `nms-hybrid.nmsClickhouse.persistence.volumeName` | Name to use for a ClickHouse PVC volume. | | -| `nms-hybrid.nmsClickhouse.persistence.accessMode` | PVC access mode for ClickHouse. | `ReadWriteOnce` | -| `nms-hybrid.nmsClickhouse.persistence.size` | PVC size for ClickHouse. | `1G` | -| `nms-hybrid.nmsClickhouse.tolerations` | List of Kubernetes tolerations, if any. | See [Kubernetes taints and tolerations](#kubernetes-taints-and-tolerations) | -| `nms-hybrid.externalClickhouse.address` | Address of the external ClickHouse service. | | -| `nms-hybrid.externalClickhouse.user` | User of the external ClickHouse service. | | -| `nms-hybrid.externalClickhouse.password` | Password of the external ClickHouse service. | | -| `nms-hybrid.serviceAccount.annotations` | Set custom annotations for the service account used by NGINX Instance Manager. | `{}` | -| `nms-hybrid.apigw.name` | Name used for API Gateway resources. | `apigw` | -| `nms-hybrid.apigw.tlsSecret` | By default, this Helm chart creates its own Certificate Authority (CA) to self-sign HTTPS server cert key pairs. These are not managed by NGINX Instance Manager. You can bring your own NGINX API Gateway certificates for hosting the HTTPS server by setting `tlsSecret` to an existing Kubernetes secret name in the targeted namespace. The secret should include `tls.crt`, `tls.key`, and `ca.pem` in the data object. Using a self-provisioned "tlsSecret" is recommended for production.

Refer to the "Use your own certificates" section in [Frequently used Helm configurations]({{< ref "/nim/deploy/kubernetes/frequently-used-helm-configs.md#use-your-own-certificates" >}}) for an example. | | -| `nms-hybrid.apigw.image.repository` | Repository name and path for the `apigw` image. | `apigw` | -| `nms-hybrid.apigw.image.tag` | Tag used for pulling images from the registry. | `latest` | -| `nms-hybrid.apigw.image.pullPolicy` | Image pull policy. | `IfNotPresent` | -| `nms-hybrid.apigw.container.port.https` | Container HTTPS port. | `443` | -| `nms-hybrid.apigw.service.name` | Service name. | `apigw` | -| `nms-hybrid.apigw.service.type` | Service type (options: `ClusterIp`, `LoadBalancer`, `NodePort`). | `ClusterIp` | -| `nms-hybrid.apigw.service.httpsPort` | Service HTTPS port. | `443` | -| `nms-hybrid.apigw.resources.requests.cpu` | Minimum required CPU to run `apigw`. | `250m` | -| `nms-hybrid.apigw.resources.requests.memory` | Minimum required memory to run `apigw`. | `256Mi` | -| `nms-hybrid.apigw.tolerations` | List of Kubernetes tolerations, if any. | See [Kubernetes taints and tolerations](#kubernetes-taints-and-tolerations) | -| `nms-hybrid.core.name` | Name used for core resources. | `core` | -| `nms-hybrid.core.image.repository` | Repository name and path for the `core` image. | `core` | -| `nms-hybrid.core.image.tag` | Tag used for pulling images from the registry. | `latest` | -| `nms-hybrid.core.image.pullPolicy` | Image pull policy. | `IfNotPresent` | -| `nms-hybrid.core.container.port.http` | Container HTTP port. | `8033` | -| `nms-hybrid.core.container.port.db` | Container database port. | `7891` | -| `nms-hybrid.core.container.port.grpc` | Container gRPC port. | `8038` | -| `nms-hybrid.core.service.httpPort` | Service HTTP port. | `8033` | -| `nms-hybrid.core.service.grpcPort` | Service gRPC port. | `8038` | -| `nms-hybrid.core.resources.requests.cpu` | Minimum required CPU to run `core`. | `500m` | -| `nms-hybrid.core.resources.requests.memory` | Minimum required memory to run `core`. | `512Mi` | -| `nms-hybrid.core.persistence.enabled` | Enable persistence for `core` service. | `true` | -| `nms-hybrid.core.persistence.claims` | An array of persistent volume claims for Dqlite and secrets. Can be modified to use an existing PVC. | See [Dqlite](#nim-dqlite-storage-configuration) and [Secrets](#nim-secrets-storage-configuration) | -| `nms-hybrid.core.persistence.storageClass` | Storage class to use for creating a `core` PVC. | | -| `nms-hybrid.core.persistence.volumeName` | Name to use for a `core` PVC volume. | | -| `nms-hybrid.core.tolerations` | List of Kubernetes tolerations, if any. | See [Kubernetes taints and tolerations](#kubernetes-taints-and-tolerations) | -| `nms-hybrid.dpm.name` | Name used for `dpm`. | `dpm` | -| `nms-hybrid.dpm.image.repository` | Repository name and path for the `dpm` image. | `dpm` | -| `nms-hybrid.dpm.image.tag` | Tag used for pulling images from the registry. | `latest` | -| `nms-hybrid.dpm.image.pullPolicy` | Image pull policy. | `IfNotPresent` | -| `nms-hybrid.dpm.container.port.http` | Container HTTP port. | `8034` | -| `nms-hybrid.dpm.container.port.nats` | Container NATS port. | `9100` | -| `nms-hybrid.dpm.container.port.db` | Container database port. | `7890` | -| `nms-hybrid.dpm.container.port.grpc` | Container gRPC port. | `8036` | - -{{
}} +{{< call-out "important" "legacy chart name" >}} +In version 2.20.0, we renamed the Helm chart from `nms-hybrid` to `nim` when we moved it to its own repository. For versions 2.19.0 and earlier, use `nms-hybrid` instead of `nim` in each parameter name. +{{< /call-out >}} + +To update an existing release, run `helm upgrade` with the `-f ` flag, where `` is the path to your values file. + +{{< bootstrap-table "table table-bordered table-striped table-responsive table-sm" >}} + +| Parameter | Description | Default | +|:--------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------| +| `nim.adminPasswordHash` | The hashed value of the password for the admin user.
To generate a hash using OpenSSL, run `openssl passwd -1 "YourPassword123#"` | N/A | +| `nim.nmsClickhouse.mode` | Controls ClickHouse deployment:
- `internal` (default, in-cluster)
- `external` (requires `nim.externalClickhouse.address`)
- `disabled` (lightweight mode). Available in the `nim` chart 2.20.0 and later. | `internal` | +| `nms-hybrid.nmsClickhouse.enabled` | (legacy, available in `nms-hybrid` chart 2.19.0 and earlier) Enable ClickHouse when using the legacy chart. | `true` | +| `nim.nmsClickhouse.fullnameOverride` | Modify the name of ClickHouse resources. | `clickhouse` | +| `nim.nmsClickhouse.image.repository` | Repository path for the public ClickHouse image. | `clickhouse/clickhouse-server` | +| `nim.nmsClickhouse.image.tag` | Tag used for pulling ClickHouse images from the registry. | `21.3.20.1-alpine` | +| `nim.nmsClickhouse.image.pullPolicy` | Image pull policy. | `IfNotPresent` | +| `nim.nmsClickhouse.user` | Username for the ClickHouse server. | N/A | +| `nim.nmsClickhouse.password` | Password for the ClickHouse server. | N/A | +| `nim.nmsClickhouse.service.name` | ClickHouse service name. | `clickhouse` | +| `nim.nmsClickhouse.service.rpcPort` | ClickHouse service port. | `9000` | +| `nim.nmsClickhouse.resources.requests.cpu` | Minimum required CPU to run the ClickHouse server. | `500m` | +| `nim.nmsClickhouse.resources.requests.memory` | Minimum required memory to run the ClickHouse server. | `1Gi` | +| `nim.nmsClickhouse.persistence.enabled` | Use a PVC to persist ClickHouse data. | `true` | +| `nim.nmsClickhouse.persistence.existingClaim` | Name of an existing PVC to use for ClickHouse persistence. | N/A | +| `nim.nmsClickhouse.persistence.storageClass` | Storage class for creating a ClickHouse PVC. | | +| `nim.nmsClickhouse.persistence.volumeName` | Name to use for a ClickHouse PVC volume. | | +| `nim.nmsClickhouse.persistence.accessMode` | PVC access mode for ClickHouse. | `ReadWriteOnce` | +| `nim.nmsClickhouse.persistence.size` | PVC size for ClickHouse. | `1G` | +| `nim.nmsClickhouse.tolerations` | List of Kubernetes tolerations if any. | See [Kubernetes taints and tolerations](#kubernetes-taints-and-tolerations) | +| `nim.externalClickhouse.address` | Address of the external ClickHouse service. | | +| `nim.externalClickhouse.user` | User for the external ClickHouse service. | | +| `nim.externalClickhouse.password` | Password for the external ClickHouse service. | | +| `nim.serviceAccount.annotations` | Set custom annotations for the service account used by NGINX Instance Manager. | `{}` | +| `nim.apigw.name` | Name for API Gateway resources. | `apigw` | +| `nim.apigw.tlsSecret` | By default, the chart creates its own CA to self-sign HTTPS server certs. To bring your own certificates, set `tlsSecret` to an existing Kubernetes secret in the target namespace. The secret must include `tls.crt`, `tls.key`, and `ca.pem`. See [Use your own certificates]({{< ref "/nim/deploy/kubernetes/frequently-used-helm-configs.md#use-your-own-certificates" >}}). | | +| `nim.apigw.image.repository` | Repository path for the `apigw` image. | `apigw` | +| `nim.apigw.image.tag` | Tag used for pulling `apigw` images. | `latest` | +| `nim.apigw.image.pullPolicy` | Image pull policy. | `IfNotPresent` | +| `nim.apigw.container.port.https` | Container HTTPS port. | `443` | +| `nim.apigw.service.name` | Service name. | `apigw` | +| `nim.apigw.service.type` | Service type (`ClusterIp`, `LoadBalancer`, `NodePort`). | `ClusterIp` | +| `nim.apigw.service.httpsPort` | Service HTTPS port. | `443` | +| `nim.apigw.resources.requests.cpu` | Minimum required CPU to run `apigw`. | `250m` | +| `nim.apigw.resources.requests.memory` | Minimum required memory to run `apigw`. | `256Mi` | +| `nim.apigw.tolerations` | List of Kubernetes tolerations if any. | See [Kubernetes taints and tolerations](#kubernetes-taints-and-tolerations) | +| `nim.core.name` | Name for core resources. | `core` | +| `nim.core.image.repository` | Repository path for the `core` image. | `core` | +| `nim.core.image.tag` | Tag used for pulling `core` images. | `latest` | +| `nim.core.image.pullPolicy` | Image pull policy. | `IfNotPresent` | +| `nim.core.container.port.http` | Container HTTP port. | `8033` | +| `nim.core.container.port.db` | Container database port. | `7891` | +| `nim.core.container.port.grpc` | Container gRPC port. | `8038` | +| `nim.core.service.httpPort` | Service HTTP port. | `8033` | +| `nim.core.service.grpcPort` | Service gRPC port. | `8038` | +| `nim.core.resources.requests.cpu` | Minimum required CPU to run `core`. | `500m` | +| `nim.core.resources.requests.memory` | Minimum required memory to run `core`. | `512Mi` | +| `nim.core.persistence.enabled` | Enable persistence for `core`. | `true` | +| `nim.core.persistence.claims` | Array of PVCs for Dqlite and secrets. Modify to use an existing PVC. | See [Dqlite storage](#nim-dqlite-storage-configuration) and [Secrets storage](#nim-secrets-storage-configuration) | +| `nim.core.persistence.storageClass` | Storage class for creating a `core` PVC. | | +| `nim.core.persistence.volumeName` | Name for a `core` PVC volume. | | +| `nim.core.tolerations` | List of Kubernetes tolerations if any. | See [Kubernetes taints and tolerations](#kubernetes-taints-and-tolerations) | +| `nim.dpm.name` | Name for `dpm` resources. | `dpm` | +| `nim.dpm.image.repository` | Repository path for the `dpm` image. | `dpm` | +| `nim.dpm.image.tag` | Tag used for pulling `dpm` images. | `latest` | +| `nim.dpm.image.pullPolicy` | Image pull policy. | `IfNotPresent`| +| `nim.dpm.container.port.http` | Container HTTP port. | `8034` | +| `nim.dpm.container.port.nats` | Container NATS port. | `9100` | +| `nim.dpm.container.port.db` | Container database port. | `7890` | +| `nim.dpm.container.port.grpc` | Container gRPC port. | `8036` | +{{}} ## NGINX Instance Manager dqlite storage configuration ```yaml - - name: dqlite - existingClaim: - size: 500Mi - accessMode: ReadWriteOnce +- name: dqlite + existingClaim: + size: 500Mi + accessMode: ReadWriteOnce ``` ## NGINX Instance Manager secrets storage configuration @@ -112,7 +113,7 @@ To modify a configuration for an existing release, run the `helm upgrade` comman ## Kubernetes taints and tolerations -The following example snippet shows a toleration for an NGINX Instance Manager API Gateway deployment. In this example, Kubernetes will tolerate the "NoExecute" effect for 60 seconds before evicting the pod from the tainted node. +This example shows how to set the API Gateway pod to wait 60 seconds when Kubernetes applies the NoExecute taint (which marks a node as unschedulable) before it removes the pod. ```yaml tolerations: diff --git a/content/nim/deploy/vm-bare-metal/install-nim-deprecated.md b/content/nim/deploy/vm-bare-metal/install-nim-deprecated.md deleted file mode 100644 index d5af56dd0..000000000 --- a/content/nim/deploy/vm-bare-metal/install-nim-deprecated.md +++ /dev/null @@ -1,315 +0,0 @@ ---- -description: '' -docs: DOCS-1211 -title: Manually install on a virtual machine or bare metal (deprecated) -toc: true -weight: 10 -noindex: true -type: -- tutorial ---- - -## Overview - -Follow the steps in this guide to install or upgrade NGINX Instance Manager. - -{{}} -This document outlines manual steps that have been replaced by a simplified script-based process. For most users, we recommend using the updated process documented [here]({{< ref "nim/deploy/vm-bare-metal/install.md" >}}).{{}} - -## Before You Begin - -### Security Considerations - -{{< include "installation/secure-installation.md" >}} - -### Requirements {#requirements} - -To install NGINX Instance Manager, you need the following: - -- A trial or paid subscription for NGINX Instance Manager. [Sign up for NGINX Instance Manager at MyF5](https://account.f5.com/myf5). -- A Linux instance to host the NGINX Instance Manager platform and modules -- [NGINX Plus or NGINX OSS](#install-nginx) installed on the instance hosting NGINX Instance Manager - -Allow external systems access by opening network firewalls. NGINX Instance Manager uses port `443` for both gRPC and API/web interfaces. - ---- - -## Download Certificate and Key {#download-cert-key} - -Follow these steps to download the certificate and private key for NGINX Instance Manager. You'll need these files when adding the official repository for installing NGINX Instance Manager. You can also use the certificate and key when installing NGINX Plus. - -1. On the host where you're installing NGINX Instance Manager, create the `/etc/ssl/nginx/` directory: - - ``` bash - sudo mkdir -p /etc/ssl/nginx - ``` - -2. Download the NGINX Instance Manager `.crt` and `.key` files from [MyF5](https://account.f5.com/myf5) or follow the download link in your trial activation email. - -3. Move and rename the `.crt` and `.key` files: - - ```bash - sudo mv /etc/ssl/nginx/nginx-repo.crt - sudo mv /etc/ssl/nginx/nginx-repo.key - ``` - - {{}}The downloaded filenames may vary depending on your subscription type. Modify the commands above accordingly to match the actual filenames.{{}} - ---- - -## Install NGINX {#install-nginx} - -Install NGINX Open Source or NGINX Plus on the host where you'll install NGINX Instance Manager. NGINX Instance Manager uses NGINX as a front-end proxy and for managing user access. - -- [Installing NGINX and NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/) - - {{}}If you're installing NGINX Plus, you can use the `nginx-repo.key` and `nginx-repo.crt` that you added in the [previous section](#download-cert-key).{{}} - -
- Supported NGINX versions - -{{< include "nim/tech-specs/supported-nginx-versions.md" >}} - -
- -
- Supported Linux distributions - -{{< include "nim/tech-specs/supported-distros.md" >}} - -
- -{{}}Make sure to review the [Technical Specifications]({{< ref "/nim/fundamentals/tech-specs" >}}) guide for sizing requirements and other recommended specs.{{}} - ---- - -## Install ClickHouse {#install-clickhouse} - -{{}}NGINX Instance Manager requires ClickHouse 22.3.15.33 or later.{{}} - -NGINX Instance Manager uses [ClickHouse](https://clickhouse.com) to store metrics, events, and alerts, as well as configuration settings. - -Select the tab for your Linux distribution, then follow the instructions to install ClickHouse. - -{{}} - -{{%tab name="CentOS, RHEL, RPM-Based"%}} - -To install and enable ClickHouse CentOS, RHEL, and RPM-Based distributions, take the following steps: - -1. Set up the repository: - - ``` bash - sudo yum install -y yum-utils - sudo yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo - ``` - -1. Install the ClickHouse server and client: - - ```bash - sudo yum install -y clickhouse-server clickhouse-client - ``` - - > **IMPORTANT!** When installing ClickHouse, you have the option to specify a password or leave the password blank (the default is an empty string). If you choose to specify a password for ClickHouse, you must also edit the `/etc/nms/nms.conf` file after installing NGINX Instance Manager and enter your ClickHouse password; otherwise, NGINX Instance Manager won't start. - > - > For more information on customizing ClickHouse settings, refer to the [Configure ClickHouse]({{< ref "nim/system-configuration/configure-clickhouse.md" >}}) topic. - -1. Enable ClickHouse so that it starts automatically if the server is restarted: - - ```bash - sudo systemctl enable clickhouse-server - ``` - -1. Start the ClickHouse server: - - ```bash - sudo systemctl start clickhouse-server - ``` - -1. Verify ClickHouse is running: - - ```bash - sudo systemctl status clickhouse-server - ``` - -{{%/tab%}} - -{{%tab name="Debian, Ubuntu, Deb-Based"%}} - -To install and enable ClickHouse on Debian, Ubuntu, and Deb-Based distributions, take the following steps: - -1. Set up the repository: - - ```bash - sudo apt-get install -y apt-transport-https ca-certificates dirmngr - GNUPGHOME=$(mktemp -d) - sudo GNUPGHOME="$GNUPGHOME" gpg --no-default-keyring --keyring /usr/share/keyrings/clickhouse-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8919F6BD2B48D754 - sudo rm -r "$GNUPGHOME" - sudo chmod +r /usr/share/keyrings/clickhouse-keyring.gpg - - echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb lts main" | sudo tee /etc/apt/sources.list.d/clickhouse.list - sudo apt-get update - ``` - -1. Install the ClickHouse server and client: - - ``` bash - sudo apt-get install -y clickhouse-server clickhouse-client - ``` - - > **IMPORTANT!** When installing ClickHouse, you have the option to specify a password or leave the password blank (the default is an empty string). If you choose to specify a password for ClickHouse, you must also edit the `/etc/nms/nms.conf` file after installing NGINX Instance Manager and enter your ClickHouse password; otherwise, NGINX Instance Manager won't start. - > - > For more information on customizing ClickHouse settings, refer to the [Configure ClickHouse]({{< ref "nim/system-configuration/configure-clickhouse.md" >}}) topic. - -1. Enable ClickHouse so that it starts automatically if the server is restarted: - - ```bash - sudo systemctl enable clickhouse-server - ``` - -1. Start the ClickHouse server: - - ``` bash - sudo systemctl start clickhouse-server - ``` - -1. Verify ClickHouse is running: - - ```bash - sudo systemctl status clickhouse-server - ``` - -{{%/tab%}} - -{{}} - -### ClickHouse Default Settings - -NGINX Instance Manager uses the following default values for ClickHouse: - -{{}}You can customize these settings. However, if you use custom settings, make sure to follow the [Configure ClickHouse]({{< ref "nim/system-configuration/configure-clickhouse.md" >}}) instructions to update the `nms.conf` file after you've installed NGINX Instance Manager; otherwise NGINX Instance Manager won't be able to connect to ClickHouse.{{}} - -{{< include "installation/clickhouse-defaults.md" >}} - ---- - -## (Optional) Install and Configure Vault {#install-vault} - -NGINX Instance Manager can use [Vault](https://www.vaultproject.io/) as a datastore for secrets. - -To install and enable Vault, take the following steps: - -- Follow Vault's instructions to [install Vault 1.8.8 or later](https://www.vaultproject.io/docs/install) for your distribution. -- Ensure you are running Vault in a [Production Hardened Environment](https://learn.hashicorp.com/tutorials/vault/production-hardening). -- After installing NGINX Instance Manager, follow the steps to [Configure Vault for Storing Secrets]({{< ref "nim/system-configuration/configure-vault.md" >}}). - ---- - -## Add NGINX Instance Manager Repository {#add-nms-repo} - -To install NGINX Instance Manager, you need to add the official repository to pull the pre-compiled `deb` and `rpm` packages from. - -{{< include "installation/add-nms-repo.md" >}} - ---- - -## Install Instance Manager - -{{}} - -{{%tab name="CentOS, RHEL, RPM-Based"%}} - -1. To install the latest version of Instance Manager, run the following command: - - ```bash - sudo yum install -y nms-instance-manager - ``` - - > **IMPORTANT!** The Instance Manager's administrator username (default is `admin`) and generated password are displayed in the terminal during installation. You should make a note of the password and store it securely. - -{{%/tab%}} - -{{%tab name="Debian, Ubuntu, Deb-Based"%}} - -1. To install the latest version of Instance Manager, run the following commands: - - ```bash - sudo apt-get update - sudo apt-get install -y nms-instance-manager - ``` - - > **IMPORTANT!** The Instance Manager's administrator username (default is `admin`) and generated password are displayed in the terminal during installation. You should make a note of the password and store it securely. - -{{%/tab%}} - -{{}} - -2. Enable and start the NGINX Instance Manager platform services: - - ```bash - sudo systemctl enable nms nms-core nms-dpm nms-ingestion nms-integrations --now - ``` - - NGINX Instance Manager components started this way run by default as the non-root `nms` user inside the `nms` group, both of which are created during installation. - -3. Restart the NGINX web server: - - ```bash - sudo systemctl restart nginx - ``` - -### Post-Installation Steps - -{{< include "installation/optional-installation-steps.md" >}} - -### Accessing the Web Interface - -{{< include "installation/access-web-ui.md" >}} - - -### Add License - -{{< include "nim/admin-guide/license/connected-install-license-note.md" >}} - ---- - -## Upgrade Instance Manager {#upgrade-nim} - -{{}} -{{%tab name="CentOS, RHEL, RPM-Based"%}} - -1. To upgrade to the latest version of the Instance Manager, run the following command: - - ```bash - sudo yum update -y nms-instance-manager - ``` - -{{%/tab%}} - -{{%tab name="Debian, Ubuntu, Deb-Based"%}} - -1. To upgrade to the latest version of the Instance Manager, run the following command: - - ```bash - sudo apt-get update - sudo apt-get install -y --only-upgrade nms-instance-manager - ``` - -{{%/tab%}} -{{}} - -2. Restart the NGINX Instance Manager platform services: - - ```bash - sudo systemctl restart nms - ``` - - NGINX Instance Manager components started this way run by default as the non-root `nms` user inside the `nms` group, both of which are created during installation. - -3. Restart the NGINX web server: - - ```bash - sudo systemctl restart nginx - ``` - -4. (Optional) If you use SELinux, follow the steps in the [Configure SELinux]({{< ref "nim/system-configuration/configure-selinux.md" >}}) guide to restore the default SELinux labels (`restorecon`) for the files and directories related to NGINX Management suite. diff --git a/content/nim/deploy/vm-bare-metal/install-nim-manual.md b/content/nim/deploy/vm-bare-metal/install-nim-manual.md new file mode 100644 index 000000000..fa8f6df42 --- /dev/null +++ b/content/nim/deploy/vm-bare-metal/install-nim-manual.md @@ -0,0 +1,233 @@ +--- +description: '' +docs: DOCS-1211 +title: Manually install any version of NGINX Instance Manager +toc: true +weight: 10 +noindex: true +type: +- tutorial +--- + +## Overview + +Follow the steps in this guide to install or upgrade a specific version of NGINX Instance Manager. + +## Before You Begin + +### Security Considerations + +{{< include "installation/secure-installation.md" >}} + +### Requirements {#requirements} + +To install NGINX Instance Manager, you need the following: + +- A trial or paid subscription for NGINX Instance Manager. [Sign up for NGINX Instance Manager at MyF5](https://account.f5.com/myf5). +- A Linux instance to host the NGINX Instance Manager platform and modules +- [NGINX Plus or NGINX OSS](#install-nginx) installed on the instance hosting NGINX Instance Manager + +Allow external systems access by opening network firewalls. NGINX Instance Manager uses port `443` for both gRPC and API/web interfaces. + +--- + +## Download Certificate and Key {#download-cert-key} + +Follow these steps to download the certificate and private key for NGINX Instance Manager. You'll need these files when adding the official repository for installing NGINX Instance Manager. You can also use the certificate and key when installing NGINX Plus. + +1. On the host where you're installing NGINX Instance Manager, create the `/etc/ssl/nginx/` directory: + + ``` bash + sudo mkdir -p /etc/ssl/nginx + ``` + +2. Download the NGINX Instance Manager `.crt` and `.key` files from [MyF5](https://account.f5.com/myf5) or follow the download link in your trial activation email. + +3. Move and rename the `.crt` and `.key` files: + + ```bash + sudo mv /etc/ssl/nginx/nginx-repo.crt + sudo mv /etc/ssl/nginx/nginx-repo.key + ``` + + The downloaded filenames may vary depending on your subscription type. Modify the commands above accordingly to match the actual filenames. + +--- + +## Install NGINX {#install-nginx} + +Install NGINX Open Source or NGINX Plus on the host where you'll install NGINX Instance Manager. NGINX Instance Manager uses NGINX as a front-end proxy and for managing user access. + +- [Installing NGINX and NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/) + +
+ + If you're installing NGINX Plus, you can use the `nginx-repo.key` and `nginx-repo.crt` that you added in the [previous section](#download-cert-key). + +
+ Supported NGINX versions + +{{< include "nim/tech-specs/supported-nginx-versions.md" >}} + +
+ +
+ Supported Linux distributions + +{{< include "nim/tech-specs/supported-distros.md" >}} + +
+ +Make sure to review the [Technical Specifications]({{< ref "/nim/fundamentals/tech-specs" >}}) guide for sizing requirements and other recommended specs. + +--- + +## Configure metrics collection + +### Disable metrics collection + +NGINX Instance Manager uses ClickHouse to store metrics, events, alerts, and configuration data. + +In 2.20.0, we introduced Lightweight mode, which can skip the ClickHouse installation entirely. It’s ideal if you don’t need monitoring data or want a simpler setup. This reduces system requirements and avoids the work of managing a metrics database. You can add ClickHouse later if your needs change. + +If you don’t need to store metrics, you can skip installing ClickHouse. But you must use NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}, and you must disable metrics collection in the `/etc/nms/nms.conf` file. + +For instructions, see [Disable metrics collection]({{< ref "nim/system-configuration/configure-clickhouse.md#disable-metrics-collection" >}}). + +### Install ClickHouse to enable metrics + +{{< include "nim/clickhouse/clickhouse-install.md" >}} + +#### ClickHouse Default Settings + +NGINX Instance Manager uses the following default values for ClickHouse. To change these values, see the [Configure ClickHouse](nim/system-configuration/configure-clickhouse.md) guide. + +{{< include "nim/clickhouse/clickhouse-defaults.md" >}} + +--- + +## Add NGINX Instance Manager Repository {#add-nms-repo} + +To install NGINX Instance Manager, you need to add the official repository to pull the pre-compiled `deb` and `rpm` packages from. + +{{< include "installation/add-nms-repo.md" >}} + +--- + +## Install Instance Manager + +{{}} + +{{%tab name="CentOS, RHEL, RPM-Based"%}} + +1. To install the latest version of Instance Manager, run the following command: + + ```bash + sudo yum install -y nms-instance-manager + ``` + + > **IMPORTANT!** The Instance Manager's administrator username (default is `admin`) and generated password are displayed in the terminal during installation. You should make a note of the password and store it securely. + +{{%/tab%}} + +{{%tab name="Debian, Ubuntu, Deb-Based"%}} + +1. To install the latest version of Instance Manager, run the following commands: + + ```bash + sudo apt-get update + sudo apt-get install -y nms-instance-manager + ``` + + > **IMPORTANT!** The Instance Manager's administrator username (default is `admin`) and generated password are displayed in the terminal during installation. You should make a note of the password and store it securely. + +{{%/tab%}} + +{{}} + +2. Enable and start the NGINX Instance Manager platform services: + + ```bash + sudo systemctl enable nms nms-core nms-dpm nms-ingestion nms-integrations --now + ``` + + NGINX Instance Manager components started this way run by default as the non-root `nms` user inside the `nms` group, both of which are created during installation. + +3. Restart the NGINX web server: + + ```bash + sudo systemctl restart nginx + ``` + +## Optional post-installation steps + +### Configure ClickHouse + +{{< include "nim/installation/optional-steps/configure-clickhouse.md" >}} + +### Install and configure Vault {#install-vault} + +{{< include "nim/installation/optional-steps/install-configure-vault.md" >}} + + +### Configure SELinux + +{{< include "nim/installation/optional-steps/configure-selinux.md" >}} + +## Accessing the Web Interface + +{{< include "installation/access-web-ui.md" >}} + + +## Add License + +{{< include "nim/admin-guide/license/connected-install-license-note.md" >}} + +--- + +## Upgrade Instance Manager {#upgrade-nim} + +{{}} +{{%tab name="CentOS, RHEL, RPM-Based"%}} + +1. To upgrade to the latest version of the Instance Manager, run the following command: + + ```bash + sudo yum update -y nms-instance-manager + ``` + +{{%/tab%}} + +{{%tab name="Debian, Ubuntu, Deb-Based"%}} + +1. To upgrade to the latest version of the Instance Manager, run the following command: + + ```bash + sudo apt-get update + sudo apt-get install -y --only-upgrade nms-instance-manager + ``` + +{{%/tab%}} +{{}} + +2. Restart the NGINX Instance Manager platform services: + + ```bash + sudo systemctl restart nms + ``` + + NGINX Instance Manager components started this way run by default as the non-root `nms` user inside the `nms` group, both of which are created during installation. + +3. Restart the NGINX web server: + + ```bash + sudo systemctl restart nginx + ``` + +4. (Optional) If you use SELinux, follow the steps in the [Configure SELinux]({{< ref "nim/system-configuration/configure-selinux.md" >}}) guide to restore the default SELinux labels (`restorecon`) for the files and directories related to NGINX Management suite. + +--- + +## Next steps + +- [Add NGINX Open Source and NGINX Plus instances to NGINX Instance Manager]({{< ref "nim/nginx-instances/add-instance.md" >}}) diff --git a/content/nim/deploy/vm-bare-metal/install.md b/content/nim/deploy/vm-bare-metal/install.md index 8e4a1ae44..5e7dea31b 100644 --- a/content/nim/deploy/vm-bare-metal/install.md +++ b/content/nim/deploy/vm-bare-metal/install.md @@ -1,7 +1,7 @@ --- description: '' docs: DOCS-1211 -title: Install on a virtual machine or bare metal using a script +title: Install the latest NGINX Instance Manager with a script toc: true weight: 10 type: @@ -12,9 +12,16 @@ type: ## Overview -This guide explains how to install F5 NGINX Instance Manager on a virtual machine or bare metal system using the `install-nim-bundle.sh` script. +This guide shows you how to install and upgrade F5 NGINX Instance Manager on a virtual machine or bare metal system using the installation script in online mode. -The script simplifies the installation by automating tasks such as verifying system requirements, configuring services, and managing environment-specific options. For more control or an alternative approach, you can refer to the [manual installation guide]({{< ref "nim/deploy/vm-bare-metal/install-nim-deprecated.md" >}}), which provides detailed, step-by-step instructions. +The script installs: + +- The latest version of NGINX Open Source +- The latest version of NGINX Instance Manager +- ClickHouse by default, unless you choose to skip it +- Optionally, NGINX Plus (requires a license and additional flags) + +The script also installs all required operating system packages automatically. If you need to install earlier versions of NGINX or NGINX Instance Manager, follow the [manual installation process](({{< ref "nim/deploy/vm-bare-metal/install-nim-manual.md" >}})) instead. --- @@ -22,6 +29,10 @@ The script simplifies the installation by automating tasks such as verifying sys Follow these steps to prepare for installing NGINX Instance Manager: +- **Download the installation script**: + + {{}} {{}} + - **Download the certificate and private key** (see the steps [below](#download-cert-key)): Use the certificate and private key for NGINX Instance Manager (the same files used for NGINX Plus). - Ensure the files have `.crt` and `.key` extensions. @@ -31,43 +42,28 @@ Follow these steps to prepare for installing NGINX Instance Manager: - **Check for previous deployments**: Ensure that NGINX Instance Manager and its components are not already installed. - - If NGINX Instance Manager or its components (such as ClickHouse or NGINX) are detected, either follow the [upgrade instructions](#upgrade-nim) to update them or [manually remove the components](#uninstall-nim) before proceeding with the installation. - -- **Record the version details**: - Note the current version of NGINX Instance Manager and confirm the supported version of NGINX OSS or NGINX Plus you intend to use. - - By default, the script installs the latest version. + + If NGINX Instance Manager or its components (such as ClickHouse or NGINX) are detected, either follow the [upgrade instructions](#upgrade-nim) to update them or [manually remove the components](#uninstall-nim) before proceeding with the installation. - **(Optional) Install and configure Vault**: If you plan to use Vault, set it up before proceeding. -### Security considerations - -To ensure that your NGINX Instance Manager deployment remains secure, follow these recommendations: - -- Install NGINX Instance Manager on a dedicated machine (bare metal, container, cloud, or VM). -- Ensure that no other services are running on the same machine. - ---- - -## Requirements - ### Supported NGINX versions and Linux distributions -
- Supported NGINX versions - -{{< include "nim/tech-specs/supported-nginx-versions.md" >}} +The installation script installs the latest version of [NGINX Open Source](https://nginx.org/news.html) or [NGINX Plus](https://docs.nginx.com/nginx/releases/). -
+To see the list of supported distributions, run: -
- Supported Linux distributions - -{{< include "nim/tech-specs/supported-distros.md" >}} +```shell +grep '\-d distribution' install-nim-bundle.sh +``` -
+### Security considerations +To ensure that your NGINX Instance Manager deployment remains secure, follow these recommendations: +- Install NGINX Instance Manager on a dedicated machine (bare metal, container, cloud, or VM). +- Ensure that no other services are running on the same machine. --- @@ -77,7 +73,7 @@ Download the certificate and private key required for NGINX Instance Manager. Th 1. On the host where you're installing NGINX Instance Manager, create the **/etc/ssl/nginx/** directory: - ```bash + ```shell sudo mkdir -p /etc/ssl/nginx ``` @@ -85,16 +81,16 @@ Download the certificate and private key required for NGINX Instance Manager. Th 3. Move and rename the cert and key files to the correct directory: - ```bash + ```shell sudo mv nginx-.crt /etc/ssl/nginx/nginx-repo.crt sudo mv nginx-.key /etc/ssl/nginx/nginx-repo.key ``` --- -## Download and run the installation script {#download-install} +## Prepare the system and run the installation script {#download-install} -Download the `install-nim-bundle.sh` script: +If you haven’t already downloaded the script, you can download it here: {{}} {{}} @@ -116,66 +112,75 @@ The script supports only new installations. If NGINX Instance Manager is already Ensure that the required `.crt` and `.key` files are available, preferably in the default **/etc/ssl/nginx** directory. Missing certificates or keys will prevent the script from completing the installation. +#### Verify If Metrics Are Required (Exclude ClickHouse) + +In 2.20.0, we introduced Lightweight mode, which means you can skip the ClickHouse installation entirely. It’s ideal if you don’t need monitoring data or want a simpler setup. This reduces system requirements and avoids the work of managing a metrics database. You can add ClickHouse later if your needs change. + #### Use the manual installation steps if needed -If the script fails or if you prefer more control over the process, consider using the [manual installation steps]({{< ref "nim/deploy/vm-bare-metal/install-nim-deprecated.md" >}}). These steps provide a reliable alternative for troubleshooting or handling complex setups. +If the script fails or if you prefer more control over the process, consider using the [manual installation steps]({{< ref "nim/deploy/vm-bare-metal/install-nim-manual.md" >}}). These steps provide a reliable alternative for troubleshooting or handling complex setups. ### Run the installation script -The `install-nim-bundle.sh` script automates the installation of NGINX Instance Manager. By default, the script: - -- Assumes no prior installation of NGINX Instance Manager or its dependencies and performs a fresh installation. -- Reads SSL files from the `/etc/ssl/nginx` directory. -- Installs the latest version of NGINX Open Source (OSS). -- Installs the ClickHouse database. -- Installs NGINX Instance Manager. -- Requires an active internet connection. +The `install-nim-bundle.sh` script automates installing NGINX Instance Manager. -{{< warning >}} +By default, the script: -As noted in [About subscription licenses]({{< ref "solutions/about-subscription-licenses.md#apply-the-jwt" >}}), **custom paths won't work until you upgrade to NGINX Plus R33**. +- Assumes no prior installation of NGINX Instance Manager or its dependencies +- Reads SSL files from the `/etc/ssl/nginx` directory +- Installs the latest version of NGINX Open Source (OSS) +- Installs the ClickHouse database +- Installs the latest version of NGINX Instance Manager +- Requires an active internet connection -{{< /warning >}} +#### Installation script options -When you run the script, it downloads and installs NGINX Instance Manager. +You can customize the installation using the following options: -If you want to use the script with non-default options, use these switches: -- To point to a repository key stored in a directory other than **/etc/ssl/nginx**: `-k /path/to/your/` file -- To point to a repository certificate stored in a directory other than **/etc/ssl/nginx**: `-c /path/to/your/` file -- To install NGINX Plus (instead of NGINX OSS): `-p -j /path/to/license.jwt` +| Category | Option or Flag | +|----------|----------------| +| **Installation platform** | {{< include "nim/installation/install-script-flags/distribution.md" >}} | +| **SSL certificate and key** | {{< include "nim/installation/install-script-flags/cert.md" >}}
{{< include "nim/installation/install-script-flags/key.md" >}} | +| **NGINX installation** | `-n` Install the latest version of NGINX Open Source. *(Default if `-n` or `-p` not specified)*

`-p` Install NGINX Plus as the API gateway. Must be used with `-j` to provide a JWT license.

`-j ` Path to the `license.jwt` file. Required when using `-p`. | +| **ClickHouse installation** | {{< include "nim/installation/install-script-flags/skip-clickhouse.md" >}}
{{< include "nim/installation/install-script-flags/clickhouse-version.md" >}} | -{{< note >}} Starting from [NGINX Plus Release 33]({{< ref "nginx/releases.md#r33" >}}), a JWT file is required for each NGINX Plus instance. For more information, see [About Subscription Licenses]({{< ref "/solutions/about-subscription-licenses.md">}}). {{< /note >}} +**Example: install with default key and certificate paths** -You also need to specify the current operating system. To get the latest list supported by the script, run the following command: +To use the script to install NGINX Instance Manager on Ubuntu 24.04, with repository keys in the default `/etc/ssl/nginx` directory, with the latest version of NGINX Open Source, run the following command: ```bash -grep '\-d distribution' install-nim-bundle.sh +sudo bash install-nim-bundle.sh -d ubuntu24.04 ``` -For example, to use the script to install NGINX Instance Manager on Ubuntu 24.04, with repository keys in the default `/etc/ssl/nginx` directory, with the latest version of NGINX OSS, run the following command: +
-```bash -sudo bash install-nim-bundle.sh -n latest -d ubuntu24.04 -j /path/to/license.jwt -``` +**Example: install with custom repo key and certificate** To install NGINX Instance Manager on Ubuntu 24.04 with the latest version of NGINX Plus by pointing to the location of your NGINX cert and key, run the following command: ```bash -sudo bash install-nim-bundle.sh -c /path/to/nginx-repo.crt -k /path/to/nginx-repo.key -p latest -d ubuntu24.04 -j /path/to/license.jwt +sudo bash install-nim-bundle.sh \ + -c \ + -k \ + -p \ + -d ubuntu24.04 \ + -j ``` -In most cases, the script completes the installation of NGINX Instance Manager and associated packages. After installation is complete, the script takes a few minutes to generate a password. At the end of the process, you'll see an autogenerated password: +
+ +After installing NGINX Instance Manager and related packages, the script generates an admin password. This may take a few minutes to appear: ```bash Regenerated Admin password: ``` -Save that password. You'll need it when you sign in to NGINX Instance Manager. +Save this password. You'll need it to log in to NGINX Instance Manager. ### Problems and additional script parameters -There are multiple parameters to configure in the Installation script. If you see fatal errors when running the script, first run the following command, which includes command options that can help you bypass problems: +There are multiple parameters to configure in the installation script. If you see fatal errors when running the script, first run the following command, which includes command options that can help you bypass problems: ```bash bash install-nim-bundle.sh -h @@ -201,43 +206,30 @@ In some cases, the script may need to be re-run due to parameters not being set ```bash bash install-nim-bundle.sh -r ``` ---- - -## Post-installation steps - -### Configure ClickHouse {#configure-clickhouse} - - -{{}}NGINX Instance Manager relies on [ClickHouse](https://clickhouse.com) **24.9.2.42** or later to store essential data, including metrics, events, alerts, and configuration settings.{{}} - -{{}}The NGINX Instance Manager installation script also installs ClickHouse with a blank password. Update the **/etc/nms/nms.conf** file with it after installing NGINX Instance Manager. Otherwise, NGINX Instance Manager won't start. For more information on customizing ClickHouse settings, refer to the [Configure ClickHouse]({{< ref "/nim/system-configuration/configure-clickhouse.md" >}}) topic. {{}} +--- -#### ClickHouse default settings +## Optional post-installation steps -NGINX Instance Manager uses the following default values for ClickHouse: +### Configure ClickHouse -{{}}You can customize these settings. However, if you use custom settings, make sure to follow the [Configure ClickHouse]({{< ref "/nim/system-configuration/configure-clickhouse.md" >}}) instructions to update the **nms.conf** file after you've installed NGINX Instance Manager. Otherwise, NGINX Instance Manager won't be able to connect to ClickHouse.{{}} +{{< include "nim/installation/optional-steps/configure-clickhouse.md" >}} -{{< include "installation/clickhouse-defaults.md" >}} +### Disable metrics collection -### (Optional) Install and configure Vault {#install-vault} +{{< include "nim/installation/optional-steps/disable-metrics-collection.md" >}} -NGINX Instance Manager can use [Vault](https://www.vaultproject.io/) as a datastore for secrets. -To install and enable Vault, follow these steps: +### Install and configure Vault {#install-vault} -- Follow Vault's instructions to [install Vault 1.8.8 or later](https://www.vaultproject.io/docs/install) for your distribution. -- Ensure you're running Vault in a [production-hardened environment](https://learn.hashicorp.com/tutorials/vault/production-hardening). -- After installing NGINX Instance Manager, follow the steps to [configure Vault for storing secrets]({{< ref "/nim/system-configuration/configure-vault.md" >}}). +{{< include "nim/installation/optional-steps/install-configure-vault.md" >}} -### (Optional) Configure SELinux -SELinux helps secure your deployment by enforcing mandatory access control policies. +### Configure SELinux -If you use SELinux, follow the steps in the [Configure SELinux]({{< ref "/nim/system-configuration/configure-selinux.md" >}}) guide to restore SELinux contexts (`restorecon`) for the files and directories related to NGINX Instance Manager. +{{< include "nim/installation/optional-steps/configure-selinux.md" >}} -### License NGINX Instance Manager +## License NGINX Instance Manager {{< include "nim/admin-guide/license/connected-install-license-note.md" >}} @@ -271,7 +263,7 @@ If you use SELinux, follow the steps in the [Configure SELinux]({{< ref "/nim/sy sudo apt-get install -y --only-upgrade nms-instance-manager ``` -1. To upgrade to the latest version of Clickhouse, run the following commands: +1. To upgrade to the latest version of ClickHouse, run the following commands: ```bash sudo apt-get update @@ -308,3 +300,9 @@ If you use SELinux, follow the steps in the [Configure SELinux]({{< ref "/nim/sy ## Uninstall NGINX Instance Manager {#uninstall-nim} {{< include "nim/uninstall/uninstall-nim.md" >}} + +--- + +## Next steps + +- [Add NGINX Open Source and NGINX Plus instances to NGINX Instance Manager]({{< ref "nim/nginx-instances/add-instance.md" >}}) diff --git a/content/nim/disconnected/_index.md b/content/nim/disconnected/_index.md index c0f6f954b..3df3312cb 100644 --- a/content/nim/disconnected/_index.md +++ b/content/nim/disconnected/_index.md @@ -1,5 +1,5 @@ --- -title: Disconnected environments +title: Deploy in a disconnected environment weight: 30 url: /nginx-instance-manager/disconnected/ cascade: diff --git a/content/nim/disconnected/add-license-disconnected-deployment.md b/content/nim/disconnected/add-license-disconnected-deployment.md index 0a38ddec3..e4a272a6a 100644 --- a/content/nim/disconnected/add-license-disconnected-deployment.md +++ b/content/nim/disconnected/add-license-disconnected-deployment.md @@ -1,5 +1,5 @@ --- -title: Add a license in a disconnected environment +title: Add a license (disconnected) draft: false description: '' weight: 200 @@ -25,7 +25,7 @@ This guide shows you how to add a license to NGINX Instance Manager in a disconn ### Set the operation mode to disconnected -To configure NGINX Instance Manager for a network-restricted environment, you need to set the `mode_of_operation` to `disconnected` in the configuration file. +To configure NGINX Instance Manager for a disconnected environment, you need to set the `mode_of_operation` to `disconnected` in the configuration file. {{< include "nim/disconnected/set-mode-of-operation-disconnected.md" >}} diff --git a/content/nim/disconnected/offline-install-guide-deprecated.md b/content/nim/disconnected/offline-install-guide-manual.md similarity index 81% rename from content/nim/disconnected/offline-install-guide-deprecated.md rename to content/nim/disconnected/offline-install-guide-manual.md index 7f07ebff4..13456b043 100644 --- a/content/nim/disconnected/offline-install-guide-deprecated.md +++ b/content/nim/disconnected/offline-install-guide-manual.md @@ -1,5 +1,5 @@ --- -title: Manually install in a disconnected environment (deprecated) +title: Manually install any version of NGINX Instance Manager (disconnected) weight: 100 toc: true noindex: true @@ -8,18 +8,15 @@ product: NIM docs: DOCS-000 --- -{{}} -This document outlines manual steps that have been replaced by a simplified script-based process. For most users, we recommend using the updated process documented [here]({{< ref "nim/disconnected/offline-install-guide.md" >}}).{{}} - ## Overview This guide explains how to install and upgrade NGINX Instance Manager in environments without Internet access. It covers key steps, including downloading packages, managing dependencies, and configuring the system for offline use. You’ll also learn how to set up NGINX Instance Manager in disconnected mode and manually update the CVE list to keep your system secure. ## Before you begin -{{}} +{{< call-out "important" "Complete the required prerequisites" "fas fa-exclamation-triangle" >}} You must complete the following prerequisite steps **before** installing NGINX Instance Manager. **Skipping these steps could cause installation issues**. -{{}} +{{}} ### Security considerations @@ -96,17 +93,14 @@ To download external dependencies: sudo dpkg -i ./*.deb ``` - {{< call-out "important" "Setting a custom ClickHouse password" "fas fa-exclamation-triangle" >}} - - When installing ClickHouse, you can set a password or leave it blank (default is an empty string). If you set a password, make sure to update the **/etc/nms/nms.conf** file with it after installing NGINX Instance Manager. Otherwise, NGINX Instance Manager won't start. For more information on customizing ClickHouse settings, refer to the [Configure ClickHouse]({{< ref "/nim/system-configuration/configure-clickhouse.md" >}}) topic. - - {{}} - - --- ## Install NGINX Instance Manager {#install-nim-offline} +{{< call-out "important" "Save the password!" "" >}} +The administrator username (default: **admin**) and the generated password are displayed in the terminal during installation. Be sure to record the password and store it securely. +{{}} + 1. Log in to the [MyF5 Customer Portal](https://account.f5.com/myf5) and download the NGINX Instance Manager package files. 2. Install the NGINX Instance Manager package: @@ -123,10 +117,6 @@ To download external dependencies: sudo apt-get -y install -f /home//nms-instance-manager__amd64.deb ``` - {{< call-out "important" "Save the password!" "fas fa-exclamation-triangle" >}} - The administrator username (default: **admin**) and the generated password are displayed in the terminal during installation. Be sure to record the password and store it securely. - {{}} - 3. Enable and start NGINX Instance Manager services: ```bash @@ -149,9 +139,25 @@ To download external dependencies: --- -## Post-installation steps (optional) +## Optional post-installation steps + +### Configure ClickHouse + +{{< include "nim/installation/optional-steps/configure-clickhouse.md" >}} -{{< include "installation/optional-installation-steps.md" >}} +### Disable metrics collection + +{{< include "nim/installation/optional-steps/disable-metrics-collection.md" >}} + + +### Install and configure Vault {#install-vault} + +{{< include "nim/installation/optional-steps/install-configure-vault.md" >}} + + +### Configure SELinux + +{{< include "nim/installation/optional-steps/configure-selinux.md" >}} ## Upgrade NGINX Instance Manager {#upgrade-nim-offline} @@ -192,3 +198,8 @@ sudo chmod 644 /usr/share/nms/cve.xml && \ sudo systemctl restart nms-dpm ``` +--- + +## Next steps + +- [Add NGINX Open Source and NGINX Plus instances to NGINX Instance Manager]({{< ref "nim/nginx-instances/add-instance.md" >}}) \ No newline at end of file diff --git a/content/nim/disconnected/offline-install-guide.md b/content/nim/disconnected/offline-install-guide.md index 039854089..ae422d800 100644 --- a/content/nim/disconnected/offline-install-guide.md +++ b/content/nim/disconnected/offline-install-guide.md @@ -1,5 +1,5 @@ --- -title: Install in a disconnected environment using a script +title: Install the latest NGINX Instance Manager with a script (disconnected) toc: true weight: 100 type: how-to @@ -11,9 +11,17 @@ docs: DOCS-803 ## Overview -This guide shows you how to install and upgrade NGINX Instance Manager in environments without internet access. It covers key steps, including downloading packages, managing dependencies, and configuring the system for offline use. You’ll also learn how to set up NGINX Instance Manager in disconnected mode and update the CVE list manually to keep your system secure. +This guide shows you how to install and upgrade F5 NGINX Instance Manager in disconnected environments. -{{}}If you prefer to follow the original manual steps, you can access the [deprecated guide]({{< ref "nim/disconnected/offline-install-guide-deprecated.md" >}}). Please note that this guide is no longer actively maintained and may not reflect the latest updates or best practices.{{}} +The script installs: + +- The latest version of NGINX Open Source +- The latest version of NGINX Instance Manager +- ClickHouse by default, unless you choose to skip it + +NGINX Plus is not supported in disconnected mode. + +If you need to install earlier versions of NGINX or NGINX Instance Manager, follow the [manual installation process](({{< ref "nim/deploy/vm-bare-metal/install-nim-manual.md" >}})) instead. --- @@ -41,7 +49,7 @@ Ensure that the required `.crt` and `.key` files are available, preferably in th #### Use the manual installation steps if needed -If the script fails or if you prefer more control over the process, consider using the [manual installation steps]({{< ref "nim/disconnected/offline-install-guide-deprecated.md" >}}). These steps provide a reliable alternative for troubleshooting or handling complex setups. +If the script fails or if you prefer more control over the process, consider using the [manual installation steps]({{< ref "nim/disconnected/offline-install-guide-manual.md" >}}). These steps provide a reliable alternative for troubleshooting or handling complex setups. ### Download the SSL Certificate and Private Key from MyF5 @@ -56,47 +64,45 @@ Download the SSL certificate and private key required for NGINX Instance Manager {{}} {{}} -### Use the script to Download the necessary packages to Install NGINX Instance Manager in a Disconnected environment - -To run the script, enter the following command, replacing `` and `` with the full paths and filenames of your SSL certificate and private key files: +## Package NGINX Instance Manager and dependencies for offline installation -```shell -sudo bash install-nim-bundle.sh \ - -c \ - -k \ - -m offline \ - -d \ - -v -``` +Run the installation script in `offline` mode to download NGINX Instance Manager, NGINX Open Source, ClickHouse (unless skipped), and all required dependencies into a tarball for use in disconnected environments. -
+### Installation script options -By default, this command installs the latest version of NGINX Open Source to run NGINX Instance Manager. NGINX Plus is currently not supported when using the script in Disconnected mode. Please see this guide to install NGINX Plus offline if you would like to use this in front of NGINX Instance Manager +| Category | Option or Flag | +|----------|----------------| +| **Installation mode and platform** | `-m offline`: Required to package the installation files into a tarball for disconnected environments.
{{< include "nim/installation/install-script-flags/distribution.md" >}} | +| **SSL certificate and key** | {{< include "nim/installation/install-script-flags/cert.md" >}}
{{< include "nim/installation/install-script-flags/key.md" >}} | +| **NGINX installation** | `-n`: Include the latest version of NGINX Open Source in the tarball.

This option is optional in `offline` mode—if not specified, the script installs the latest version of NGINX Open Source by default.

NGINX Plus is **not supported** when using the script in offline mode.

To install NGINX Plus offline, see the [manual installation guide]({{< ref "nginx/admin-guide/installing-nginx/installing-nginx-plus.md#offline_install" >}}). | +| **ClickHouse installation** | {{< include "nim/installation/install-script-flags/skip-clickhouse.md" >}}
{{< include "nim/installation/install-script-flags/clickhouse-version.md" >}} | -
+### Example: packaging command -**Explanation of options:** - -- **`-c`**: Uses the specified SSL certificate file. Copies the file to the /etc/ssl/nginx directory. -- **`-k`**: Uses the specified private key file. Copies the file to the /etc/ssl/nginx directory. -- **`-m`**: Sets the installation mode (use `offline` for disconnected environments). -- **`-d`**: Defines the target distribution (replace `` with one of the supported options below). -- **`-n`**: Installs a specific version of NGINX Open Source. Use `latest` to install the most recent version or specify a version like `1.27.1`. The script defaults to installing the latest version of NGINX Open Source. -- **`-v`**: Installs the specified version of NGINX Instance Manager. Use `latest` for the newest version or a specific release like `2.18.0`. If you skip this option, the script assumes you want to install `latest`. + ```shell + sudo bash install-nim-bundle.sh \ + -c \ + -k \ + -m offline \ + -d \ + -v + ``` -**Supported distributions:** +--- -To get the latest list supported by the script, run the following command: +## Install NGINX Instance Manager -```bash -grep '\-d distribution' install-nim-bundle.sh -``` +After you’ve packaged the installation files on a connected system, copy the tarball, script, and SSL files to your disconnected system. Then, run the script again to install NGINX Instance Manager using the tarball. -The script downloads the required packages and adds them to a tarball file. You’ll need to copy this tarball to the target machine in the disconnected environment. +### Required flags for installing in offline mode ---- +- `-m offline`: Required to run the script in offline mode. When used with `-i`, the script installs NGINX Instance Manager and its dependencies from the specified tarball. +- `-i `: Path to the tarball created during the packaging step. +- {{< include "nim/installation/install-script-flags/cert.md" >}} +- {{< include "nim/installation/install-script-flags/key.md" >}} +- `-d `: Target Linux distribution (must match what was used during packaging). -## Install NGINX Instance Manager +### Install from the tarball 1. Copy the following files to the target system: - `install-nim-bundle.sh` script @@ -108,11 +114,11 @@ The script downloads the required packages and adds them to a tarball file. You ```shell sudo bash install-nim-bundle.sh \ - -c - -k \ - -m offline \ - -d \ + -m offline -i + -c + -k \ + -d \ ``` 3. **Save the admin password**. In most cases, the script completes the installation of NGINX Instance Manager and associated packages. After installation is complete, the script takes a few minutes to generate a password. At the end of the process, you'll see an autogenerated password: @@ -133,9 +139,25 @@ The script downloads the required packages and adds them to a tarball file. You --- -## Post-installation steps (optional) +## Optional post-installation steps + +### Configure ClickHouse + +{{< include "nim/installation/optional-steps/configure-clickhouse.md" >}} + +### Disable metrics collection + +{{< include "nim/installation/optional-steps/disable-metrics-collection.md" >}} + -{{< include "installation/optional-installation-steps.md" >}} +### Install and configure Vault {#install-vault} + +{{< include "nim/installation/optional-steps/install-configure-vault.md" >}} + + +### Configure SELinux + +{{< include "nim/installation/optional-steps/configure-selinux.md" >}} --- @@ -184,3 +206,8 @@ sudo chmod 644 /usr/share/nms/cve.xml && \ sudo systemctl restart nms-dpm ``` +--- + +## Next steps + +- [Add NGINX Open Source and NGINX Plus instances to NGINX Instance Manager]({{< ref "nim/nginx-instances/add-instance.md" >}}) \ No newline at end of file diff --git a/content/nim/disconnected/report-usage-disconnected-deployment.md b/content/nim/disconnected/report-usage-disconnected-deployment.md index a63763654..afe228b1a 100644 --- a/content/nim/disconnected/report-usage-disconnected-deployment.md +++ b/content/nim/disconnected/report-usage-disconnected-deployment.md @@ -1,5 +1,5 @@ --- -title: Report usage to F5 in a disconnected environment +title: Report usage data to F5 (disconnected) draft: false weight: 300 toc: true diff --git a/content/nim/nginx-app-protect/security-monitoring/set-up-app-protect-instances.md b/content/nim/nginx-app-protect/security-monitoring/set-up-app-protect-instances.md index 45745bd24..2a70c574d 100644 --- a/content/nim/nginx-app-protect/security-monitoring/set-up-app-protect-instances.md +++ b/content/nim/nginx-app-protect/security-monitoring/set-up-app-protect-instances.md @@ -30,7 +30,7 @@ Complete the following prerequisites before proceeding with the steps in this gu {{< include "nim/tech-specs/security-data-plane-dependencies.md" >}} 1. Determine your use case: **Security Monitoring only** or **Security Monitoring and Configuration Management**. -1. [Upload your license]({{< ref "/nim/admin-guide/license/add-license.md" >}}). +1. [Upload your license]({{< ref "/nim/admin-guide/add-license.md" >}}). --- diff --git a/content/nim/nginx-instances/add-instance.md b/content/nim/nginx-instances/add-instance.md new file mode 100644 index 000000000..760052759 --- /dev/null +++ b/content/nim/nginx-instances/add-instance.md @@ -0,0 +1,49 @@ +--- +title: "Add an NGINX instance" +weight: 100 +toc: false +nd-content-type: how-to +nd-product: NIM +--- + +This guide shows you how to add NGINX Open Source and NGINX Plus instances to F5 NGINX Instance Manager so you can manage them from a central dashboard. + +## Before you begin + +Make sure you have: + +- One or more instances running [NGINX Open Source]({{< ref "nginx/admin-guide/installing-nginx/installing-nginx-open-source.md" >}}) or [NGINX Plus]({{< ref "nginx/admin-guide/installing-nginx/installing-nginx-plus.md" >}}). +- Admin access to NGINX Instance Manager. + +## Add instances + +1. Open the NGINX Instance Manager web interface and log in. +2. In the **Manage** section on the left, select **Instances**. +3. Select **Add**. +4. Copy the `curl` command. +5. On the host where your NGINX instance is running, run the `curl` command to install NGINX Agent: + + ```shell + curl https:///install/nginx-agent | sudo sh + ``` + +6. On the same host, run the following command to start NGINX Agent: + + ```shell + sudo systemctl start nginx-agent + ``` + +## Set up metrics reporting + +### Enable NGINX Plus API + +{{< include "/use-cases/monitoring/enable-nginx-plus-api.md" >}} + +### Enable NGINX Open Source Stub Status API + +{{< include "/use-cases/monitoring/enable-nginx-oss-stub-status.md" >}} + +## Next steps + +- [Add instances to instance groups]({{< ref "nim/nginx-instances/manage-instance-groups.md" >}}) +- [Add managed certificates]({{< ref "nim/nginx-instances/manage-certificates.md" >}}) \ No newline at end of file diff --git a/content/nim/releases/known-issues.md b/content/nim/releases/known-issues.md index e880979e4..b45a829c8 100644 --- a/content/nim/releases/known-issues.md +++ b/content/nim/releases/known-issues.md @@ -13,6 +13,83 @@ This document lists and describes the known issues and possible workarounds in F --- +## 2.20.0 + +June 16, 2025 + +### {{% icon-bug %}} Failing to fetch CVE data when using forward proxy in K8s environments {#46177} + +{{}} +| Issue ID | Status | +|----------------|--------| +| 46177 | Open | +{{}} + +#### Description + +Fetching latest CVE data from internet might fail if you enable "ssl_verify" in Kubernetes environments. + + +#### Workaround + +1. Switch to the offline CVE database. To switch add the property "offline_nginx_cve: true" under the DPM section in the "nms.conf" file. + + ```cfg + dpm: + offline_nginx_cve: true + ``` + +2. Download the latest security advisories file from the [nginx.org repository]( https://raw.githubusercontent.com/nginx/nginx.org/main/xml/en/security_advisories.xml) and save them with "cve.xml" as filename in "/usr/share/nms/cve.xml” +3. Restart the `nms` service. + + ```shell + sudo systemctl restart nms + ``` + +After the restart you will see the line “loading CVE data from file” in the "nms.log" file. + +--- + +### {{% icon-bug %}} New warning message when no usage data or report is available {#46022} + +{{}} +| Issue ID | Status | +|----------------|--------| +| 46022 | Won't be resolved | +{{}} + +#### Description + +Users now see a warning message when they click the **Send Usage To F5** button if no new usage data or report is available. The message reads: + +> "Usage data is not available at the moment. Please try submitting usage details again later." + +--- + + +## 2.19.1 + +March 27, 2025 + +### {{% icon-resolved %}} The certificate stats are not displayed correctly in the Certificates and Keys page as well as the Dashboard page. {#45991} + +{{}} +| Issue ID | Status | +|----------------|--------| +| 45991 | Fixed in Instance Manager 2.20.0 | +{{}} + +#### Description + +When the dashboard page and certificates page are loaded, the count displayed for total, valid, expired, expires soon, managed and unmanaged are incorrect. + +#### Workaround + +The changes required have been made and the UI displays the values correctly now. Pagination also works well along with the certificate stats. + +--- + + ## 2.19.0 February 06, 2025 @@ -134,71 +211,19 @@ There is an issue that causes previous error messages to persist in the web inte --- -## 2.17.3 - -September 13, 2024 - -### {{% icon-resolved %}} The web interface can't display more than 100 certificates {#45565} - -{{}} -| Issue ID | Status | -|----------------|--------| -| 45565 | Fixed in Instance Manager 2.19.0 | -{{}} - -#### Description - -The Certificate Management screen can only show up to 100 certificates. - ---- - - ## 2.17.0 July 10, 2024 -### {{% icon-resolved %}} Mismatch in date formats in custom date selection on NGINX usage graph {#45512} - -{{}} -| Issue ID | Status | -|----------------|--------| -| 45512 | Fixed in Instance Manager 2.18.0 | -{{}} - -#### Description - -The months in the custom date range were not displayed correctly because NGINX Instance Manager assumed the data format was in the US timezone. - ---- - -### {{% icon-resolved %}} NGINX Agent 2.36.0 fails to validate certain NGINX configurations in NGINX Instance Manager 2.17.0 {#45153} - -{{}} -| Issue ID | Status | -|----------------|--------| -| 45153 | Fixed in nginxagent-2.36.0 | -{{}} - -#### Description - -In NGINX Instance Manager 2.17.0, an "invalid number of arguments" error appears in the web interface when using specific configuration parameters in NGINX Agent 2.36.0. - -#### Workaround - -Install NGINX Agent **2.35.1** if you're using NGINX Instance Manager 2.17.0. This version is included with NGINX Instance Manager 2.17.0 by default. - -If you're installing NGINX Agent from package files, follow the steps in the [Installing NGINX Agent](https://github.com/nginx/agent?tab=readme-ov-file#installing-nginx-agent-from-package-files) guide. - ---- - ### {{% icon-bug %}} Web Analytics are not enabled after upgrading Instance Manager when keeping existing nms-http.conf {#45131} {{}} + | Issue ID | Status | |----------------|--------| | 45131 | Open | -{{}} +{{}} #### Description When using NGINX Instance Manager, you configure OIDC by manually editing the /etc/nginx/conf.d/nms-http.conf and /etc/nms/nms.conf files. @@ -218,11 +243,12 @@ add_header Content-Security-Policy "default-src 'none'; block-all-mixed-content; ### {{% icon-bug %}} Failure to retrieve instance configuration when NAP-enabled instance doesn't register properly {#45113} {{}} + | Issue ID | Status | |----------------|--------| | 45113 | Open | -{{}} +{{}} #### Description If NGINX Agent is configured to monitor NGINX App Protect before App Protect is installed, NGINX Agent will send an empty App Protect metadata structure to NGINX Instance Manager. This causes Instance Manager to fail to register the NGINX instance properly. @@ -233,34 +259,6 @@ Edit the "/etc/nginx-agent/nginx-agent.conf" file and configure "precompiled_pub --- -### {{% icon-resolved %}} Failure to notify user when template configuration publish fails {#44975} - -{{}} -| Issue ID | Status | -|----------------|--------| -| 44975 | Fixed in Instance Manager 2.18.0 | -{{}} - -#### Description - -When publishing a configuration template fails, the system only displays "Accepted" without providing the final result, such as "Success" or "Failure." - ---- - -### {{% icon-resolved %}} Editing template submissions now allows for using most recent template version {#44971} - -{{}} -| Issue ID | Status | -|----------------|--------| -| 44971 | Fixed in Instance Manager 2.17.0 | -{{}} - -#### Description - -When editing a template submission, you can now choose between using a snapshot of the template from when it was first deployed or the latest version of the template. **Important:** Note that if you use the latest version, changes to the templates might make an augment template incompatible with a base template, causing the publication to the data plane to fail. - ---- - ## 2.15.0 diff --git a/content/nim/releases/release-notes.md b/content/nim/releases/release-notes.md index 9ab95c68a..0b0cbd200 100644 --- a/content/nim/releases/release-notes.md +++ b/content/nim/releases/release-notes.md @@ -17,11 +17,86 @@ The release notes for F5 NGINX Instance Manager highlight the latest features, i +--- + +## 2.20.0 + +June 16, 2025 + +### Upgrade Paths {#2-20-0-upgrade-paths} + +NGINX Instance Manager 2.20.0 supports upgrades from these previous versions: + +- 2.17.0 - 2.19.2 + +If your NGINX Instance Manager version is older, you may need to upgrade to an intermediate version before upgrading to the target version. + +### What's New{#2-20-0-whats-new} +This release includes the following updates: + +- {{% icon-feature %}} **Added support to report on multiple NGINX One subscriptions using a single instance of NGINX Instance Manager** + + Customers can have multiple NGINX One subscriptions or licenses with different attributes (prod, nonprod, etc.). Now a single NGINX Instance Manager can receive usage from these instances and send a bulk usage report to F5. + + We also simplified the licensing process for disconnected use cases. When running NGINX Instance Manager in disconnected mode, all features are enabled for 90 days when a JWT is uploaded. Customers have these 90 days to send the first usage report to activate the NGINX Instance Manager license. + +- {{% icon-feature %}} **Introducing lightweight mode** + + NGINX Instance Manager can now run in lightweight mode without the ClickHouse database dependency. This mode requires fewer resources and is perfect for customers who don’t need metrics-related functionality. It supports basic use cases such as: + + - Fleet management + - WAF configuration + - Usage tracking + - Certificate management + - Template automation + +- {{% icon-feature %}} **Improved web analytics sent to F5** + + NGINX Instance Manager now sends improved web analytics to F5. This helps F5 understand common use cases of NGINX Instance Manager and improve functionality. + + +### Changes in Default Behavior{#2-20-0-changes-in-behavior} +This release has the following changes in default behavior: + +- {{% icon-feature %}} **Breadcrumbs added for Overview, Manage, and Config templates** + + We have added breadcrumb navigation links for the "Overview", "Manage", and "Config" templates in the NGINX Instance Manager user interface. + +- {{% icon-feature %}} **Export instance groups to Excel** + + You can now export instance groups to a Microsoft Excel spreadsheet from the Instance Group section. + +- {{% icon-feature %}} **Export all instances functionality added** + + NGINX Instance Manager’s export function only included currently visible instances, not the full list. Customers had to run separate exports and merge data manually. + + An `Export All` button is now available under the Instances section. Clicking it downloads all instance details in a single Excel file. + +- {{% icon-feature %}} **Automatic feature enablement in disconnected instances on license upload** + + + NGINX Instance Manager now enables all features by default when you upload a license to a disconnected instance. This update ensures customers can begin using the full capabilities of the system without additional configuration. + + Customers have a 90-day window to complete the full license process. If they don’t complete it within this period, the instance is automatically deactivated. + + + +### Resolved Issues{#2-20-0-resolved-issues} + +This release fixes the following issues. Check the [Known Issues]({{< ref "/nim/releases/known-issues.md" >}}) topic for more information on the latest resolved issues. Use your browser's search function to find the issue ID in the page. + +- {{% icon-resolved %}} The certificate stats are not displayed correctly in the Certificates and Keys page as well as the Dashboard page. (45991) + + +### Known Issues{#2-20-0-known-issues} + +You can find information about known issues in the [Known Issues]({{< ref "/nim/releases/known-issues.md" >}}) topic. + --- ## 2.19.2 -May 08, 2025 +May 06, 2025 ### Upgrade Paths {#2-19-2-upgrade-paths} @@ -188,11 +263,11 @@ This release includes the following updates: This [Docker Compose option]({{< ref "nim/deploy/docker/deploy-nginx-instance-manager-docker-compose.md" >}}) unlocks another easy, production-ready installation method for customers using Docker. It will also make upgrades easier when new Docker images are released by F5 NGINX. This option includes health checking, NGINX App Protect compilation support, and security monitoring. -- {{% icon-feature %}} **Entitlement and visibility for NGINX Plus R33 – Telemetry reporting for network-restricted environments** +- {{% icon-feature %}} **Entitlement and visibility for NGINX Plus R33 – Telemetry reporting for disconnected environments** - If NGINX Instance Manager has internet access, customers can [automatically or manually send the usage data to F5]({{< ref "nim/admin-guide/license/report-usage-connected-deployment.md" >}}) as part of the new NGINX Plus R33 changes. + If NGINX Instance Manager has internet access, customers can [automatically or manually send the usage data to F5]({{< ref "nim/admin-guide/report-usage-connected-deployment.md" >}}) as part of the new NGINX Plus R33 changes. - For customers who have NGINX Instance Manager deployed in [network-restricted environments]({{< ref "nim/disconnected" >}}), this release also includes support for manual usage reporting. Customers can now manually license NGINX Instance Manager and export usage telemetry for fully disconnected environments. For usage reporting, customers can: + For customers who have NGINX Instance Manager deployed in [disconnected environments]({{< ref "nim/disconnected" >}}), this release also includes support for manual usage reporting. Customers can now manually license NGINX Instance Manager and export usage telemetry for fully disconnected environments. For usage reporting, customers can: - **Export the usage report**: Manually export the usage report from NGINX Instance Manager. - **Send the report to F5**: Submit the report to F5 for verification from a location with internet access. @@ -692,7 +767,7 @@ This release includes the following updates: - {{% icon-feature %}} **Use NGINX Plus Health Checks to easily track NGINX Plus Usage with NGINX Instance Manager** - The NGINX Plus Health Check feature now allows you to monitor the count of both NGINX Plus and NGINX App Protect instances that you've deployed. You can view this information in the "NGINX Plus" area of the "Instance Manager" web interface, or through the `/inventory` API. For guidance on how to set this up, refer to the following documentation: [View Count of NGINX Plus Instances]({{< ref "/nim/admin-guide/license/report-usage-connected-deployment.md" >}}). + The NGINX Plus Health Check feature now allows you to monitor the count of both NGINX Plus and NGINX App Protect instances that you've deployed. You can view this information in the "NGINX Plus" area of the "Instance Manager" web interface, or through the `/inventory` API. For guidance on how to set this up, refer to the following documentation: [View Count of NGINX Plus Instances]({{< ref "/nim/admin-guide/report-usage-connected-deployment.md" >}}). - {{% icon-feature %}} **Improved log output for better JSON parsing** @@ -731,7 +806,7 @@ This release includes the following updates: - {{% icon-feature %}} **New support for license tokens for automatic entitlement updates, renewals, and Flexible Consumption Reporting** - NGINX Management Suite now supports license tokens formatted as a JSON Web Token (JWT). With JWT licensing, you can automatically update entitlements during subscription renewals or amendments, and you can automate reporting for the Flexible Consumption Program (FCP). For more information, see the [Add a License]({{< ref "/nim/admin-guide/license/add-license.md" >}}) topic. + NGINX Management Suite now supports license tokens formatted as a JSON Web Token (JWT). With JWT licensing, you can automatically update entitlements during subscription renewals or amendments, and you can automate reporting for the Flexible Consumption Program (FCP). For more information, see the [Add a License]({{< ref "/nim/admin-guide/add-license.md" >}}) topic. ### Resolved Issues{#2-12-0-resolved-issues} @@ -1357,7 +1432,7 @@ This release includes the following updates: - {{% icon-feature %}} **Track NGINX Plus usage over time** - When viewing your NGINX Plus instances in the Instnace Manager web interface, you can set a date and time filter to review the [NGINX Plus instance count]({{< ref "/nim/admin-guide/license/report-usage-connected-deployment.md" >}}) for a specific period. Also, you can use the Instance Manager REST API to view the lowest, highest, and average number of NGINX Plus instances over time. + When viewing your NGINX Plus instances in the Instance Manager web interface, you can set a date and time filter to review the [NGINX Plus instance count]({{< ref "/nim/admin-guide/report-usage-connected-deployment.md" >}}) for a specific period. Also, you can use the Instance Manager REST API to view the lowest, highest, and average number of NGINX Plus instances over time. - {{% icon-feature %}} **New helm charts for each release of Instance Manager** @@ -1400,7 +1475,7 @@ This release includes the following updates: - {{% icon-feature %}} **See which of your NGINX Plus instances have NGINX App Protect installed** - Now, when you [view your NGINX Plus inventory]({{< ref "/nim/admin-guide/license/report-usage-connected-deployment.md" >}}), you can see which instances have [NGINX App Protect](https://www.nginx.com/products/nginx-app-protect/) installed. NGINX App Protect is a modern app‑security solution that works seamlessly in DevOps environments as a robust WAF or app‑level DoS defense, helping you deliver secure apps from code to customer. + Now, when you [view your NGINX Plus inventory]({{< ref "/nim/admin-guide/report-usage-connected-deployment.md" >}}), you can see which instances have [NGINX App Protect](https://www.nginx.com/products/nginx-app-protect/) installed. NGINX App Protect is a modern app‑security solution that works seamlessly in DevOps environments as a robust WAF or app‑level DoS defense, helping you deliver secure apps from code to customer. ### Changes in Default Behavior{#2-4-0-changes-in-behavior} @@ -1494,7 +1569,7 @@ This release includes the following updates: - {{% icon-feature %}} **Track the details for your NGINX Plus instances** - Easily track your NGINX Plus instances from the new NGINX Plus inventory list page. [View the current count for all your NGINX Plus instances]({{< ref "/nim/admin-guide/license/report-usage-connected-deployment.md" >}}), as well as each instance's hostname, UID, version, and the last time each instance was reported to Instance Manager. Select the `Export` button to export the list of NGINX Plus instances to a `.csv` file. + Easily track your NGINX Plus instances from the new NGINX Plus inventory list page. [View the current count for all your NGINX Plus instances]({{< ref "/nim/admin-guide/report-usage-connected-deployment.md" >}}), as well as each instance's hostname, UID, version, and the last time each instance was reported to Instance Manager. Select the `Export` button to export the list of NGINX Plus instances to a `.csv` file. - {{% icon-feature %}} **Explore events in NGINX Instance Manager with the Events Catalogs API** diff --git a/content/nim/system-configuration/configure-clickhouse.md b/content/nim/system-configuration/configure-clickhouse.md index 32e50bdb4..3f4a7f1be 100644 --- a/content/nim/system-configuration/configure-clickhouse.md +++ b/content/nim/system-configuration/configure-clickhouse.md @@ -12,84 +12,52 @@ type: ## Overview -Follow this guide to update the `/etc/nms/nms.conf` file if you used a custom address, username, or password, or enabled TLS during ClickHouse installation. NGINX Instance Manager requires this information to connect to ClickHouse. +NGINX Instance Manager uses ClickHouse to store metrics, events, alerts, and configuration data. +If your setup differs from the default configuration—for example, if you use a custom address, enable TLS, set a password, or turn off metrics—you need to update the `/etc/nms/nms.conf` file. -## Default ClickHouse values {#default-values} +This guide explains how to update those settings so that NGINX Instance Manager can connect to ClickHouse correctly. -Unless otherwise specified in the `/etc/nms/nms.conf` file, NGINX Instance Manager uses the following default values for ClickHouse: - -{{< include "installation/clickhouse-defaults.md" >}} - ---- - -## Use custom address, username, or password - -If your ClickHouse installation uses a custom address, username, or password, update the NGINX Instance Manager configuration to match. +## Change default settings {#change-settings} -To set custom values: +To change a ClickHouse setting: -1. Open the `/etc/nms/nms.conf` file on the NGINX Instance Manager server. -2. Update the following settings to match your ClickHouse configuration: +1. Open the configuration file at `/etc/nms/nms.conf`. - ```yaml - clickhouse: - address: tcp://localhost:9000 - username: - password: - ``` +2. In the `[clickhouse]` section, update the setting or settings you want to change. -3. Save and close the file. +3. Restart the NGINX Instance Manager service: ---- - -## Configure TLS - -If you enabled TLS for ClickHouse, update the `nms.conf` file settings to enable secure connections. + ```shell + sudo systemctl restart nms + ``` -To configure TLS: - -1. Open the `/etc/nms/nms.conf` file on the NGINX Instance Manager server. -2. Update the `clickhouse` TLS settings as needed: - - ```yaml - clickhouse: +Unless otherwise specified in the `/etc/nms/nms.conf` file, NGINX Instance Manager uses the following default values for ClickHouse: - # Sets the log level for ClickHouse processes within NMS. - log_level: debug +{{< include "nim/clickhouse/clickhouse-defaults.md" >}} - # Sets the address that will be used to connect to ClickHouse. - address: 127.0.0.1:9001 - ## Note: Username and password should only be set, if you have custom defined username and password for ClickHouse. - ## Ensure that any configured username or password is wrapped in single quotes. - # Sets the username that will be used to connect to ClickHouse. - username: 'test-1' +## Disable metrics collection - # Sets the password that will be used to connect to ClickHouse. - password: 'test-2' +As of version 2.20, NGINX Instance Manager can run without ClickHouse. This lightweight mode reduces system requirements and simplifies installation for users who do not need metrics. To use this setup, you must run NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}. - # Activates or deactivates TLS for connecting to ClickHouse. - # Note: `tls_mode` will be deprecated in the future, use the `tls` key to enable TLS connection for ClickHouse. - tls_mode: true +To disable metrics collection after installing NGINX Instance Manager: - tls: - # Sets the address (form )used to connect to ClickHouse with a TLS connection. - address: 127.0.0.1:9441 +1. Open the configuration file at `/etc/nms/nms.conf`. - # Activates or deactivates TLS verification of ClickHouse connection. - skip_verify: false +2. In the `[clickhouse]` section, set the following value: - # Sets the path of the certificate used for TLS connections in PEM encoded format. - cert_path: /etc/certs + ```yaml + enable = false + ``` - # Sets the path of the client key used for TLS connections in PEM encoded format. - key_path: /etc/key +3. Restart the NGINX Instance Manager service: - # Sets the path of the Certificate Authority installed on the system for verifying certificates. - cert_ca: /etc/ca + ```shell + sudo systemctl restart nms + ``` - # Sets directory containing ClickHouse migration files. - migrations_path: /test/migrations - ``` +When metrics are turned off: -3. Save and close the file. +- The web interface no longer shows metrics dashboards. Instead, it displays a message explaining that metrics are turned off. +- Metrics-related API endpoints return a 403 error. +- All other NGINX Instance Manager features continue to work as expected. diff --git a/content/nim/system-configuration/configure-forward-proxy.md b/content/nim/system-configuration/configure-forward-proxy.md index a5f961404..f0d1bda26 100644 --- a/content/nim/system-configuration/configure-forward-proxy.md +++ b/content/nim/system-configuration/configure-forward-proxy.md @@ -1,5 +1,5 @@ --- -Title: Configure NGINX Instance Manager to use a forward proxy +Title: Use a third-party forward proxy weight: 2 toc: true type: how-to diff --git a/content/nim/system-configuration/configure-high-availability.md b/content/nim/system-configuration/configure-high-availability.md index 183288d10..5e679df7e 100644 --- a/content/nim/system-configuration/configure-high-availability.md +++ b/content/nim/system-configuration/configure-high-availability.md @@ -1,5 +1,5 @@ --- -Title: Configure high availability (HA) for NGINX Instance Manager +Title: High availability with keepalived weight: 3 toc: true type: how-to diff --git a/content/nim/system-configuration/configure-telemetry.md b/content/nim/system-configuration/configure-telemetry.md index 56bde9408..afdf834b8 100644 --- a/content/nim/system-configuration/configure-telemetry.md +++ b/content/nim/system-configuration/configure-telemetry.md @@ -37,13 +37,13 @@ The table below shows the captured telemetry data points, the trigger conditions | Login | When a user logs in to NGINX Instance Manager. No data about the user is sent, only the fact that a user successfully authenticated and the timestamp of the login event. | To understand how often users or systems access NGINX Instance Manager. | | Start/Stop processes | When any NGINX Instance Manager processes are started or stopped. | To gauge how often users upgrade NGINX Instance Manager or troubleshoot issues. This information helps F5 Support diagnose issues. | | Adding Data Plane(s) | When NGINX Agent registers with NGINX Instance Manager for the first time. No data about the data plane is sent, just that an NGINX Agent registered with the platform. | To understand the frequency and quantity of data planes being added to NGINX Instance Manager. This information helps inform our scale and performance targets and helps F5 Support diagnose issues. | -| Product Usage | Data is sent daily or when Send Usage is selected from the Licenses page in the web interface or initiated using the API. (Requires a [JWT license]({{< ref "/nim/admin-guide/license/add-license.md#jwt-license" >}}).) | To track and report commercial usage in accordance with entitlement and Flexible Consumption Program (FCP) requirements. | +| Product Usage | Data is sent daily or when Send Usage is selected from the Licenses page in the web interface or initiated using the API. (Requires a [JWT license]({{< ref "/nim/admin-guide/add-license.md#jwt-license" >}}).) | To track and report commercial usage in accordance with entitlement and Flexible Consumption Program (FCP) requirements. | {{}} ### Enable or disable telemetry -Once you [apply a valid license]({{< ref "/nim/admin-guide/license/add-license.md" >}}), telemetry data starts transmitting. If the license is applied immediately after installation, the *Installation* data point is also sent. +Once you [apply a valid license]({{< ref "/nim/admin-guide/add-license.md" >}}), telemetry data starts transmitting. If the license is applied immediately after installation, the *Installation* data point is also sent. #### Disable telemetry diff --git a/content/nim/system-configuration/configure-vault.md b/content/nim/system-configuration/configure-vault.md index 1400edeac..5f61cd308 100644 --- a/content/nim/system-configuration/configure-vault.md +++ b/content/nim/system-configuration/configure-vault.md @@ -1,6 +1,6 @@ --- docs: DOCS-999 -title: Configure Vault for storing secrets +title: Use Vault for storing secrets toc: true weight: 200 type: diff --git a/content/nim/system-configuration/configure-with-config.md b/content/nim/system-configuration/configure-with-config.md index 87f54ba33..18db798ca 100644 --- a/content/nim/system-configuration/configure-with-config.md +++ b/content/nim/system-configuration/configure-with-config.md @@ -1,7 +1,7 @@ --- description: '' docs: DOCS-1100 -title: Configure NGINX Instance Manager with nms.conf +title: Configure settings with nms.conf toc: true weight: 1 type: diff --git a/content/nim/system-configuration/secure-traffic.md b/content/nim/system-configuration/secure-traffic.md index aacba1d61..a4d03c4bf 100644 --- a/content/nim/system-configuration/secure-traffic.md +++ b/content/nim/system-configuration/secure-traffic.md @@ -1,6 +1,6 @@ --- docs: DOCS-794 -title: Secure Client Access and Network Traffic +title: Secure client access and network traffic toc: true weight: 600 type: @@ -468,7 +468,7 @@ To generate the necessary certificates, follow these steps. You can modify these ## Configure SSL verification for usage reporting with self-signed certificates {#configure-ssl-verify} {{}} -Usage reporting for NGINX Plus R33 or later in network-restricted environments requires **NGINX Instance Manager version 2.18 or later**. +Usage reporting for NGINX Plus R33 or later in disconnected environments requires **NGINX Instance Manager version 2.18 or later**. {{}} Starting with NGINX Plus R33, NGINX Plus must report usage data to a reporting endpoint, such as NGINX Instance Manager. For more information, see [About subscription licenses]({{< ref "solutions/about-subscription-licenses.md" >}}). diff --git a/content/nms/acm/how-to/deploy-api-connectivity-manager.md b/content/nms/acm/how-to/deploy-api-connectivity-manager.md index b8a4a8352..70e2dd054 100644 --- a/content/nms/acm/how-to/deploy-api-connectivity-manager.md +++ b/content/nms/acm/how-to/deploy-api-connectivity-manager.md @@ -198,7 +198,7 @@ If you've already deployed API Connectivity Manager and would like to upgrade to A valid license is required to make full use of all the features in API Connectivity Manager. -Refer to the [Add a License]({{< ref "/nim/admin-guide/license/add-license.md" >}}) topic for instructions on how to download and apply a trial license, subscription license, or Flexible Consumption Program license. +Refer to the [Add a License]({{< ref "/nim/admin-guide/add-license.md" >}}) topic for instructions on how to download and apply a trial license, subscription license, or Flexible Consumption Program license. --- diff --git a/content/nms/acm/how-to/install-acm.md b/content/nms/acm/how-to/install-acm.md index 3aa0e760f..264143d9a 100644 --- a/content/nms/acm/how-to/install-acm.md +++ b/content/nms/acm/how-to/install-acm.md @@ -81,7 +81,7 @@ type: A valid license is required to make full use of all the features in API Connectivity Manager. -Refer to the [Add a License]({{< ref "/nim/admin-guide/license/add-license.md" >}}) topic for instructions on how to download and apply a trial license, subscription license, or Flexible Consumption Program license. +Refer to the [Add a License]({{< ref "/nim/admin-guide/add-license.md" >}}) topic for instructions on how to download and apply a trial license, subscription license, or Flexible Consumption Program license. --- diff --git a/content/nms/acm/tutorials/advanced-routing.md b/content/nms/acm/tutorials/advanced-routing.md index 0531a0eb1..8977fbe40 100644 --- a/content/nms/acm/tutorials/advanced-routing.md +++ b/content/nms/acm/tutorials/advanced-routing.md @@ -43,7 +43,7 @@ In the steps that follow, we will: To complete the instructions in this guide, you need the following: -- [API Connectivity Manager is installed]({{< ref "/nms/acm/how-to/install-acm.md" >}}), [licensed]({{< ref "/nim/admin-guide/license/add-license.md" >}}), and running +- [API Connectivity Manager is installed]({{< ref "/nms/acm/how-to/install-acm.md" >}}), [licensed]({{< ref "/nim/admin-guide/add-license.md" >}}), and running - One or more [Service workspaces]({{< ref "/nms/acm/how-to/services/publish-api.md#create-a-service-workspace" >}}) - One or more [Proxies]({{< ref "/nms/acm/how-to/services/publish-api.md" >}}) diff --git a/content/solutions/about-subscription-licenses.md b/content/solutions/about-subscription-licenses.md index 1e942a09c..6fb2e484f 100644 --- a/content/solutions/about-subscription-licenses.md +++ b/content/solutions/about-subscription-licenses.md @@ -140,12 +140,12 @@ The instructions below use the terms "internet-connected" and "network-restricte #### License NGINX Instance Manager -- **Internet-connected**: Follow the steps in [Add license]({{< ref "nim/admin-guide/license/add-license.md" >}}). +- **Internet-connected**: Follow the steps in [Add license]({{< ref "nim/admin-guide/add-license.md" >}}). - **Network-restricted**: Follow the steps in [Add a license in a disconnected environment]({{< ref "nim/disconnected/add-license-disconnected-deployment.md" >}}). #### Submit usage reports to F5 from NGINX Instance Manager {#submit-usage-reports-from-nim} -- **Internet-connected**: Follow the steps in [Report usage to F5]({{< ref "nim/admin-guide/license/report-usage-connected-deployment.md" >}}). +- **Internet-connected**: Follow the steps in [Report usage to F5]({{< ref "nim/admin-guide/report-usage-connected-deployment.md" >}}). - **Network-restricted**: Follow the steps in [Report usage to F5 in a disconnected environment]({{< ref "nim/disconnected/report-usage-disconnected-deployment.md" >}}). ### NGINX App Protect WAF diff --git a/layouts/shortcodes/clickhouse-version.html b/layouts/shortcodes/clickhouse-version.html new file mode 100644 index 000000000..bd2a1af86 --- /dev/null +++ b/layouts/shortcodes/clickhouse-version.html @@ -0,0 +1 @@ +24.8.12.28 \ No newline at end of file diff --git a/layouts/shortcodes/lightweight-nim-nginx-agent-version.html b/layouts/shortcodes/lightweight-nim-nginx-agent-version.html new file mode 100644 index 000000000..fe35b6ea2 --- /dev/null +++ b/layouts/shortcodes/lightweight-nim-nginx-agent-version.html @@ -0,0 +1 @@ +2.41.1 or later \ No newline at end of file diff --git a/static/scripts/docker-compose/docker-compose-lightweight.yaml b/static/scripts/docker-compose/docker-compose-lightweight.yaml new file mode 100644 index 000000000..a88759453 --- /dev/null +++ b/static/scripts/docker-compose/docker-compose-lightweight.yaml @@ -0,0 +1,93 @@ +name: nim +secrets: + nim_admin_password: + file: admin_password.txt + # Optional: set admin and user credentials from a .htpasswd file. + #nim_credential_file: + # file: nim_creds.txt + # Optional: set external SSL certificate, key, and CA in PEM format for the NIM ingress proxy. + #nim_proxy_cert_file: + # file: ./certs/nim_cert.pem + #nim_proxy_cert_key: + # file: ./certs/nim_key.pem + #nim_proxy_ca_cert: + # file: ./certs/nim_ca.pem +services: + nim: + image: private-registry.nginx.com/nms/nim-standalone-compose:2.20.0 + hostname: nim + ports: + - 443:443 + networks: + - external_network + volumes: + - nim-data:/data + - proxy-certs:/usr/local/share/ca-certificates + # Uncomment to set custom service limits. + #deploy: + # resources: + # limits: + # cpus: '0.5' + # memory: '512M' + restart: no + healthcheck: + test: ["CMD", "sh", "/etc/nms/scripts/health.sh"] + interval: 30s + timeout: 10s + retries: 3 + stop_grace_period: 30s + secrets: + - nim_admin_password + # Uncomment this line if you use a .htpasswd credential file. The credential file takes priority over the nim_admin_password file. + #- nim_credential_file + # Uncomment these lines to use user-provided certificate, key, and CA for the NIM ingress proxy. + #- nim_proxy_cert_file + #- nim_proxy_cert_key + #- nim_proxy_ca_cert + environment: + # Set the NIM logging level. + NIM_LOG_LEVEL: "INFO" + # Set a custom time-to-live value (in days) for metrics retention. + NIM_METRICS_TTL: "1" + # Set a custom time-to-live value (in days) for event retention. + NIM_EVENTS_TTL: "1" + # Set a custom time-to-live value (in days) for security violation retention. + NIM_SECURITY_TTL: "1" + # Enable maintenance mode to perform backup, restore, and troubleshooting. + #NIM_MAINTENANCE: "true" + # Set a custom watchdog timeout (in seconds). + NIM_WATCHDOG_TIMEOUT: "60" + # Set the NIM license mode to connected or disconnected. Default is connected. + NIM_LICENSE_MODE_OF_OPERATION: "connected" + # Enable basic proxy support. + PROXY_ENABLE: false + # Configure the proxy server host. + PROXY_HOST: "" + # Configure the proxy server protocol (http or https). Default is http. + PROXY_PROTOCOL: "http" + # Configure the proxy server port. + PROXY_PORT: 3128 + # Enable basic authentication for the proxy. + PROXY_AUTH_REQUIRED: false + # Configure proxy username for basic authentication (required when proxy_auth_required is true). + PROXY_AUTH_USERNAME: "" + # Configure proxy password for basic authentication (required when proxy_auth_required is true). + # We recommend using a .env file for the password. + # PROXY_PASSWORD: "" + # Configure proxy SSL verification to trust the proxy server. + PROXY_SSL_VERIFY: true + ENABLE_METRICS: false +networks: + external_network: + driver: bridge +volumes: + # By default, Docker Compose creates a named volume. + # For additional storage options such as NFS, see https://docs.docker.com/reference/compose-file/volumes/. + proxy-certs: + driver: local + nim-data: + driver: local + #driver_opts: + # type: "nfs" + # o: "addr=172.16.0.92,rw" + # device: ":/mnt/nfs_share/data" \ No newline at end of file diff --git a/static/scripts/install-nim-bundle.sh b/static/scripts/install-nim-bundle.sh index 105399f0f..7dc020eac 100755 --- a/static/scripts/install-nim-bundle.sh +++ b/static/scripts/install-nim-bundle.sh @@ -19,29 +19,116 @@ NGINX_CERT_PATH="/etc/ssl/nginx/nginx-repo.crt" NGINX_CERT_KEY_PATH="/etc/ssl/nginx/nginx-repo.key" LICENSE_JWT_PATH="" USE_NGINX_PLUS="false" -USE_SM_MODULE="false" UNINSTALL_NIM="false" MODE="online" INSTALL_PATH="" -NIM_VERSION="latest" -NGINX_VERSION="latest" -NGINX_PLUS_VERSION="latest" -NIM_SM_VERSION="latest" -CLICKHOUSE_VERSION="latest" -CLICKHOUSE_LATEST_VERSION="24.9.2.42" -NGINX_LATEST_VERSION=1.27.3-1 -NIM_LATEST_VERSION=2.19.1 +SKIP_CLICKHOUSE_INSTALL="false" CURRENT_TIME=$(date +%s) TEMP_DIR="/tmp/${CURRENT_TIME}" TARGET_DISTRIBUTION="" -PACKAGE_INSTALLER="" NMS_NGINX_MGMT_BLOCK="mgmt { \n usage_report endpoint=127.0.0.1 interval=30m; \n ssl_verify off; \n}"; NIM_FQDN="" -NIM_CERTS_DIR="/etc/nms/certs" -# Added to account for the renaming of the adc dimension from application to app. -if [ -f "/usr/share/nms/catalogs/dimensions/application.yml" ]; then - rm /usr/share/nms/catalogs/dimensions/application.yml -fi +OS_ARCH="amd64" +UBUNTU_2004="ubuntu20.04" +UBUNTU_2204="ubuntu22.04" +UBUNTU_2404="ubuntu24.04" +DEBIAN_11="debian11" +DEBIAN_12="debian12" +CENTOS_8="centos8" +REDHAT_8="rhel8" +REDHAT_9="rhel9" +ORACLE_8="oracle8" +ORACLE_9="oracle9" +CLICKHOUSE_VERSION="24.9.2.42" +PKG_EXTENSION="deb" +UBUNTU_OS=( + "${UBUNTU_2004}" + "${UBUNTU_2204}" + "${UBUNTU_2404}" +) +DEB_OS=( + "${DEBIAN_11}" + "${DEBIAN_12}" +) +CENT_OS=( + "${CENTOS_8}" +) +RPM_OS=( + "${REDHAT_8}" + "${REDHAT_9}" + "${ORACLE_8}" + "${ORACLE_9}" +) +SUPPORTED_OS=() +SUPPORTED_OS+=("${DEB_OS[@]}") +SUPPORTED_OS+=("${RPM_OS[@]}") +SUPPORTED_OS+=("${CENT_OS[@]}") +SUPPORTED_OS+=("${UBUNTU_OS[@]}") + +declare -A OS_DISTRO_MAP +OS_DISTRO_MAP['ubuntu20.04']="focal" +OS_DISTRO_MAP['ubuntu22.04']="jammy" +OS_DISTRO_MAP['ubuntu24.04']="noble" +OS_DISTRO_MAP['debian11']="bullseye" +OS_DISTRO_MAP['debian12']="bookworm" +OS_DISTRO_MAP['centos8']=".el8.ngx.x86_64" +OS_DISTRO_MAP['rhel8']=".el8.ngx.x86_64" +OS_DISTRO_MAP['rhel9']=".el9.ngx.x86_64" +OS_DISTRO_MAP['oracle8']=".el8.ngx.x86_64" +OS_DISTRO_MAP['oracle9']=".el9.ngx.x86_64" +OS_DISTRO_MAP['amzn2']=".amzn2.ngx.x86_64" + +declare -A NGINX_PLUS_REPO +NGINX_PLUS_REPO['ubuntu20.04']="https://pkgs.nginx.com/plus/ubuntu/pool/nginx-plus/n/nginx-plus" +NGINX_PLUS_REPO['ubuntu22.04']="https://pkgs.nginx.com/plus/ubuntu/pool/nginx-plus/n/nginx-plus" +NGINX_PLUS_REPO['ubuntu24.04']="https://pkgs.nginx.com/plus/ubuntu/pool/nginx-plus/n/nginx-plus" +NGINX_PLUS_REPO['debian11']="https://pkgs.nginx.com/plus/debian/pool/nginx-plus/n/nginx-plus" +NGINX_PLUS_REPO['debian12']="https://pkgs.nginx.com/plus/debian/pool/nginx-plus/n/nginx-plus" +NGINX_PLUS_REPO['centos8']="https://pkgs.nginx.com/plus/centos/8/x86_64/RPMS" +NGINX_PLUS_REPO['rhel8']="https://pkgs.nginx.com/plus/rhel/8/x86_64/RPMS" +NGINX_PLUS_REPO['rhel9']="https://pkgs.nginx.com/plus/rhel/9/x86_64/RPMS" +NGINX_PLUS_REPO['oracle8']="https://pkgs.nginx.com/plus/rhel/8/x86_64/RPMS" +NGINX_PLUS_REPO['oracle9']="https://pkgs.nginx.com/plus/rhel/9/x86_64/RPMS" +NGINX_PLUS_REPO['amzn2']="https://pkgs.nginx.com/plus/amzn2/2/x86_64/RPMS" + +declare -A NIM_REPO +NIM_REPO['ubuntu20.04']="https://pkgs.nginx.com/nms/ubuntu/pool/nginx-plus/n/nms-instance-manager" +NIM_REPO['ubuntu22.04']="https://pkgs.nginx.com/nms/ubuntu/pool/nginx-plus/n/nms-instance-manager" +NIM_REPO['ubuntu24.04']="https://pkgs.nginx.com/nms/ubuntu/pool/nginx-plus/n/nms-instance-manager" +NIM_REPO['debian11']="https://pkgs.nginx.com/nms/debian/pool/nginx-plus/n/nms-instance-manager" +NIM_REPO['debian12']="https://pkgs.nginx.com/nms/debian/pool/nginx-plus/n/nms-instance-manager" +NIM_REPO['centos8']="https://pkgs.nginx.com/nms/centos/8/x86_64/RPMS" +NIM_REPO['rhel8']="https://pkgs.nginx.com/nms/centos/8/x86_64/RPMS" +NIM_REPO['rhel9']="https://pkgs.nginx.com/nms/centos/9/x86_64/RPMS" +NIM_REPO['oracle8']="https://pkgs.nginx.com/nms/centos/8/x86_64/RPMS" +NIM_REPO['oracle9']="https://pkgs.nginx.com/nms/centos/9/x86_64/RPMS" +NIM_REPO['amzn2']="https://pkgs.nginx.com/nms/amzn2/2/x86_64/RPMS" + +declare -A NGINX_REPO +NGINX_REPO['ubuntu20.04']="https://nginx.org/packages/mainline/ubuntu/pool/nginx/n/nginx" +NGINX_REPO['ubuntu22.04']="https://nginx.org/packages/mainline/ubuntu/pool/nginx/n/nginx" +NGINX_REPO['ubuntu24.04']="https://nginx.org/packages/mainline/ubuntu/pool/nginx/n/nginx" +NGINX_REPO['debian11']="https://nginx.org/packages/mainline/debian/pool/nginx/n/nginx" +NGINX_REPO['debian12']="https://nginx.org/packages/mainline/debian/pool/nginx/n/nginx" +NGINX_REPO['centos8']="https://nginx.org/packages/mainline/centos/8/x86_64/RPMS" +NGINX_REPO['rhel8']="https://nginx.org/packages/mainline/rhel/8/x86_64/RPMS" +NGINX_REPO['rhel9']="https://nginx.org/packages/mainline/rhel/9/x86_64/RPMS" +NGINX_REPO['oracle8']="https://nginx.org/packages/mainline/rhel/8/x86_64/RPMS" +NGINX_REPO['oracle9']="https://nginx.org/packages/mainline/rhel/9/x86_64/RPMS" +NGINX_REPO['amzn2']="https://nginx.org/packages/mainline/amzn2/2/x86_64/RPMS" + +declare -A CLICKHOUSE_REPO +CLICKHOUSE_REPO['ubuntu20.04']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse" +CLICKHOUSE_REPO['ubuntu22.04']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse" +CLICKHOUSE_REPO['ubuntu24.04']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse" +CLICKHOUSE_REPO['debian11']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse" +CLICKHOUSE_REPO['debian12']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse" +CLICKHOUSE_REPO['centos8']="https://packages.clickhouse.com/rpm/stable" +CLICKHOUSE_REPO['rhel8']="https://packages.clickhouse.com/rpm/stable" +CLICKHOUSE_REPO['rhel9']="https://packages.clickhouse.com/rpm/stable" +CLICKHOUSE_REPO['oracle8']="https://packages.clickhouse.com/rpm/stable" +CLICKHOUSE_REPO['oracle9']="https://packages.clickhouse.com/rpm/stable" +CLICKHOUSE_REPO['amzn2']="https://packages.clickhouse.com/rpm/stable" set -o pipefail @@ -56,7 +143,16 @@ check_last_command_status(){ fi } -generate() { +url_file_download() { + url=$1 + dest=$2 + if ! http_code=$(curl -fs "${url}" --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} --output "${dest}" --write-out '%{http_code}'); then + echo "-- Failed to download $url with HTTP code $http_code. Exiting." + exit 1 + fi +} + +generate_admin_password() { character_pool='A-Za-z0-9' password_length=30 admin_password=$(LC_ALL=C tr -dc "$character_pool" /etc/nms/nginx/.htpasswd } -updateClickhouseConfig(){ - if [ ! -f "/etc/clickhouse-server/conf.d/nms-clickhouse.conf" ] ; then - if [ ! -d "/etc/clickhouse-server/conf.d/" ]; then - mkdir -p "/etc/clickhouse-server/conf.d" - fi -cat <> /etc/clickhouse-server/conf.d/nms-clickhouse.conf - - {{ 2 * number_of_cpu_cores }} - 5 - 100 - - 1073741824 - - 1000 - - 10 - - 100 - max_thread_pool_size - <-- number of seconds that clickHouse waits for incoming requests before closing the connection. Defaults to 10 seconds. --> - 5 - - 100 - - - - 4 - - 1024 - 1073741824 - 0 - - - -EOL -fi -} - -createNginxMgmtFile(){ +create_nginx_mgmt_file(){ # Check if the mgmt block exists in the file if grep -Eq '^[[:space:]]*mgmt' "/etc/nginx/nginx.conf"; then printf "Nginx 'mgmt' block found, skipping addition of nginx 'mgmt' block" @@ -126,43 +184,6 @@ createNginxMgmtFile(){ fi } -findVersionForPackage(){ - pkg_name=$1 - pkg_version=$2 - if [ "${PACKAGE_INSTALLER}" == "apt" ]; then - readarray -t versions < <(apt-cache madison "${pkg_name}" | grep "${pkg_version}" | cut -d '|' -f2 | tr -d ' ') - #readarray -t versions < <(echo "${available_versions[@]}"| grep "${pkg_version}") >&2 - else - if [[ "$pkg_name" == "nginx" ]]; then - readarray -t versions < <(yum list "${pkg_name}" --repo nginx-stable --showduplicates | grep "${pkg_version}" | cut -d' ' -f2) - else - readarray -t versions < <(yum list "${pkg_name}" --showduplicates | grep "${pkg_version}" | tr -s ' ' | cut -d' ' -f2) - fi - #readarray -t versions < <(echo "${available_versions[@]}"| grep "${pkg_version}" | cut -d' ' -f2) >&2 - fi - # Print the array contents - versions_count=${#versions[@]} - if [ "${versions_count}" -eq 0 ]; then - printf "Package %s with version %s not found. See available versions:" "${pkg_name}" "${pkg_version}" - if [ "${PACKAGE_INSTALLER}" == "apt" ]; then - apt-cache madison "${pkg_name}" - else - yum list "${pkg_name}" --showduplicates - fi - exit 110 - elif [ "${versions_count}" -gt 1 ]; then - printf "Multiple versions found for the package %s. Select your desired version:\n" "${pkg_name}" >&2 - for i in "${!versions[@]}"; do - printf "%s: %s\n" "$i" "${versions[$i]}" >&2 - done - read -rp "" index - echo "${versions[$index]}" - else - echo "${pkg_name} with version ${pkg_version} found">&2 - echo "${versions[0]}" - fi -} - debian_install_nginx(){ apt-get update \ && DEBIAN_FRONTEND=noninteractive \ @@ -210,40 +231,15 @@ debian_install_nginx(){ fi url_file_download "https://cs.nginx.com/static/files/90pkgs-nginx" "/etc/apt/apt.conf.d/90pkgs-nginx" check_last_command_status "curl https://cs.nginx.com/static/files/90pkgs-nginx" $? - apt-get update - if [ "${USE_NGINX_PLUS}" == "true" ]; then - printf "Installing NGINX Plus...\n" - if [ "${NGINX_PLUS_VERSION}" == "latest" ]; then - apt-get install -y nginx-plus - else - package_version=$(findVersionForPackage "nginx-plus" "${NGINX_PLUS_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package nginx-plus with version ${NGINX_PLUS_VERSION} not found" - exit $cmd_status - fi - apt-get install -y nginx-plus="${package_version}" - check_last_command_status "apt-get install -y nginx-plus=${package_version}" $? - fi - createNginxMgmtFile + printf "Installing NGINX Plus...\n" + DEBIAN_FRONTEND=noninteractive apt-get install -y nginx-plus + create_nginx_mgmt_file else - printf "Installing NGINX...\n" - if [ "${NGINX_VERSION}" == "latest" ]; then - apt install -y nginx - check_last_command_status "apt-get install -y nginx" $? - else - package_version=$(findVersionForPackage "nginx" "${NGINX_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package nginx with version ${NGINX_VERSION} not found" - exit $cmd_status - fi - echo "Installing nginx version ${package_version}" - apt-get install -y nginx="${package_version}" - check_last_command_status "apt-get install -y nginx='${package_version}'" $? - fi + printf "Installing NGINX...\n" + DEBIAN_FRONTEND=noninteractive apt install -y nginx + check_last_command_status "apt-get install -y nginx" $? fi } @@ -252,52 +248,36 @@ debian_install_clickhouse(){ | sudo tee /usr/share/keyrings/clickhouse-keyring.gpg >/dev/null check_last_command_status "curl https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key" $? - echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb stable main" | sudo tee \ + echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg arch=${OS_ARCH}] https://packages.clickhouse.com/deb stable main" | sudo tee \ /etc/apt/sources.list.d/clickhouse.list apt-get update - - printf "Installing clickhouse....\n" - if [ "${CLICKHOUSE_VERSION}" == "latest" ]; then - DEBIAN_FRONTEND=noninteractive apt-get install -y clickhouse-common-static clickhouse-server clickhouse-client - check_last_command_status "apt-get install -y clickhouse-server clickhouse-client" $? - else - ch_version=$(findVersionForPackage "clickhouse-server" "${CLICKHOUSE_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package clickhouse-server with version ${CLICKHOUSE_VERSION} not found" - exit $cmd_status - fi - echo "Installing clickhouse-server with version ${ch_version}" - DEBIAN_FRONTEND=noninteractive apt-get install -y clickhouse-common-static="${ch_version}" clickhouse-server="${ch_version}" - DEBIAN_FRONTEND=noninteractive apt-get install -y clickhouse-client - check_last_command_status "apt-get install -y clickhouse-server=${ch_version}" $? - fi + echo "Installing clickhouse-server with version ${CLICKHOUSE_VERSION}" + DEBIAN_FRONTEND=noninteractive apt-get install -y clickhouse-common-static="${CLICKHOUSE_VERSION}" clickhouse-server="${CLICKHOUSE_VERSION}" + DEBIAN_FRONTEND=noninteractive apt-get install -y clickhouse-client="${CLICKHOUSE_VERSION}" + check_last_command_status "apt-get install -y clickhouse-server=${CLICKHOUSE_VERSION}" $? } debian_install_nim(){ - echo "Installing NGINX Instance Manager..." - if [ "${NIM_VERSION}" == "latest" ]; then - apt-get install -y nms-instance-manager - check_last_command_status "installing NGINX Instance Manager" $? - else - package_version=$(findVersionForPackage "nms-instance-manager" "${NIM_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package nms-instance-manager with version ${NIM_VERSION} not found" - exit $cmd_status - fi - apt-get install -y nms-instance-manager="${package_version}" - check_last_command_status "apt-get install -y nms-instance-manager=${package_version}" $? + if [ "${SKIP_CLICKHOUSE_INSTALL}" == "true" ]; then + echo "SKIP_CLICKHOUSE_INSTALL = ${SKIP_CLICKHOUSE_INSTALL} | blocking clickhouse-server to be installed" + echo "apt-mark hold clickhouse-server" + apt-mark hold clickhouse-server fi - echo "Enabling clickhouse-server..." - systemctl enable clickhouse-server - check_last_command_status "systemctl enable clickhouse-server" $? + echo "Installing NGINX Instance Manager..." + DEBIAN_FRONTEND=noninteractive apt-get install -y nms-instance-manager + check_last_command_status "installing NGINX Instance Manager" $? - echo "Starting clickhouse-server..." - systemctl start clickhouse-server - check_last_command_status "systemctl start clickhouse-server" $? + if [ "${SKIP_CLICKHOUSE_INSTALL}" == "false" ]; then + echo "Enabling clickhouse-server..." + systemctl enable clickhouse-server + check_last_command_status "systemctl enable clickhouse-server" $? + + echo "Starting clickhouse-server..." + systemctl start clickhouse-server + check_last_command_status "systemctl start clickhouse-server" $? + fi echo "Starting nginx..." systemctl start nginx @@ -305,6 +285,8 @@ debian_install_nim(){ echo "Starting NGINX Instance Manager..." systemctl start nms + + sleep 5 check_last_command_status " systemctl start nms" $? echo "Installation is complete" @@ -329,40 +311,13 @@ installBundleForDebianDistro() { "${NIM_USER}" >/dev/null fi debian_install_nginx - debian_install_clickhouse - debian_install_nim - echo "security module installation opted : ${USE_SM_MODULE}" - if [ "${USE_SM_MODULE}" == "true" ]; then - nim_major_version=$(nms-core --version | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | awk -F. '{print $1}') - nim_minor_version=$(nms-core --version | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | awk -F. '{print $2}') - echo "Installed NIM major Version: ${nim_major_version}, minor Version:${nim_minor_version}" - if [[ $nim_major_version -ge 2 && $nim_minor_version -ge 19 ]]; then - echo "Note: NGINX Instance Manager version 2.19.0 or later comes with security monitoring installed. skipping installing security monitoring" - else - printf "Installing security module...\n" - if [ "${NIM_SM_VERSION}" == "latest" ]; then - apt-get install -y nms-sm - check_last_command_status "apt-get install -y nms-sm" $? - else - sm_pkg_version=$(findVersionForPackage "nms-sm" "${NIM_SM_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package nms-sm with version ${NIM_SM_VERSION} not found" - exit $cmd_status - fi - apt-get install -y nms-sm="${sm_pkg_version}" - check_last_command_status "apt-get install -y nms-sm=${NIM_SM_VERSION}" $? - fi - systemctl restart nms - sleep 5 - systemctl restart nginx - systemctl start nms-sm - fi - else - systemctl restart nms - sleep 5 - systemctl restart nginx + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "false" ]]; then + debian_install_clickhouse fi + debian_install_nim + systemctl restart nms + sleep 5 + systemctl restart nginx } installBundleForRPMDistro(){ @@ -407,52 +362,39 @@ installBundleForRPMDistro(){ if [ "${USE_NGINX_PLUS}" == "true" ]; then echo "Installing nginx plus..." - if [ "${NGINX_PLUS_VERSION}" == "latest" ]; then - yum install -y nginx-plus - check_last_command_status "yum install -y nginx-plus" $? - else - nginx_plus_pkg_version=$(findVersionForPackage "nginx-plus" "${NGINX_PLUS_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package nginx-plus with version ${NGINX_PLUS_VERSION} not found" - exit $cmd_status - fi - yum install -y nginx-plus-"${nginx_plus_pkg_version}" - check_last_command_status "yum install -y nginx-plus-${nginx_plus_pkg_version}" $? - fi + yum install -y nginx-plus + check_last_command_status "yum install -y nginx-plus" $? createNginxMgmtFile else echo "Installing nginx..." - if [ "${NGINX_VERSION}" == "latest" ]; then - yum install -y nginx --repo nginx-stable - check_last_command_status "yum install -y nginx" $? - else - nginx_pkg_version=$(findVersionForPackage "nginx" "${NGINX_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package nginx with version ${NGINX_VERSION} not found" - exit $cmd_status - fi - yum install -y nginx-"${nginx_pkg_version}" --repo nginx-stable - check_last_command_status "yum install -y nginx-${nginx_pkg_version}" $? - fi + yum install -y nginx --repo nginx-stable + check_last_command_status "yum install -y nginx" $? fi echo "Enabling nginx service" systemctl enable nginx.service check_last_command_status "systemctl enable nginx.service" $? - yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo - echo "Installing clickhouse-server and clickhouse-client" - yum install -y clickhouse-server clickhouse-client - check_last_command_status "yum install -y clickhouse-server clickhouse-client" $? + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "false" ]]; then + yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo + echo "Installing clickhouse-server and clickhouse-client" - echo "Enabling clickhouse-server" - systemctl enable clickhouse-server - check_last_command_status "systemctl enable clickhouse-server" $? + yum install -y "clickhouse-common-static-${CLICKHOUSE_VERSION}" + check_last_command_status "yum install -y clickhouse-common-static-${CLICKHOUSE_VERSION}" $? - echo "Starting clickhouse-server" - systemctl start clickhouse-server - check_last_command_status "systemctl start clickhouse-server" $? + yum install -y "clickhouse-server-${CLICKHOUSE_VERSION}" + check_last_command_status "yum install -y clickhouse-server-${CLICKHOUSE_VERSION}" $? + + yum install -y "clickhouse-client-${CLICKHOUSE_VERSION}" + check_last_command_status "yum install -y clickhouse-client-${CLICKHOUSE_VERSION}" $? + + echo "Enabling clickhouse-server" + systemctl enable clickhouse-server + check_last_command_status "systemctl enable clickhouse-server" $? + + echo "Starting clickhouse-server" + systemctl start clickhouse-server + check_last_command_status "systemctl start clickhouse-server" $? + fi curl -o /etc/yum.repos.d/nms.repo https://cs.nginx.com/static/files/nms.repo check_last_command_status "get -P /etc/yum.repos.d https://cs.nginx.com/static/files/nms.repo" $? @@ -462,128 +404,53 @@ installBundleForRPMDistro(){ fi echo "Installing NGINX Instance Manager" - if [ "${NIM_VERSION}" == "latest" ]; then - yum install -y nms-instance-manager - check_last_command_status "installing nginx-instance-manager(nim)" $? - else - nim_pkg_version=$(findVersionForPackage "nms-instance-manager" "${NIM_VERSION}") - yum install -y nms-instance-manager-"${nim_pkg_version}" - check_last_command_status "apt-get install -y nms-instance-manager=${nim_pkg_version}" $? - fi + yum install -y nms-instance-manager + check_last_command_status "installing nginx-instance-manager(nim)" $? + echo "Enabling nms nms-core nms-dpm nms-ingestion nms-integrations" systemctl enable nms nms-core nms-dpm nms-ingestion nms-integrations --now echo "Restarting NGINX Instance Manager" systemctl restart nms - if [ "${USE_SM_MODULE}" == "true" ]; then - nim_major_version=$(nms-core --version | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | awk -F. '{print $1}') - nim_minor_version=$(nms-core --version | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | awk -F. '{print $2}') - echo "Installed NIM major Version: ${nim_major_version}, minor Version:${nim_minor_version}" - if [[ $nim_major_version -ge 2 && $nim_minor_version -ge 19 ]]; then - echo "Note: NGINX Instance Manager version 2.19.0 or later comes with security monitoring installed. skipping installing security monitoring" - else - printf "Installing security module...\n" - if [ "${NIM_SM_VERSION}" == "latest" ]; then - yum install -y nms-sm - check_last_command_status "yum install -y nms-sm" $? - else - sm_pkg_version=$(findVersionForPackage "nms-sm" "${NIM_SM_VERSION}") - cmd_status=$? - if [ $cmd_status -ne 0 ]; then - echo "Package nms-sm with version ${NIM_SM_VERSION} not found" - exit $cmd_status - fi - yum install -y nms-sm="${sm_pkg_version}" - check_last_command_status "yum install -y nms-sm=${NIM_SM_VERSION}" $? - fi - systemctl restart nms - sleep 5 - systemctl restart nginx - systemctl start nms-sm - fi - fi sleep 5 echo "Restarting nginx API gateway" systemctl restart nginx } -url_file_download() { - url=$1 - dest=$2 - if ! http_code=$(curl -fs "${url}" --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} --output "${dest}" --write-out '%{http_code}'); then - echo "-- Failed to download $url with HTTP code $http_code. Exiting." - exit 1 - fi -} - install_nim_online(){ if cat /etc/*-release | grep -iq 'debian\|ubuntu'; then - PACKAGE_INSTALLER="apt" installBundleForDebianDistro - generate + generate_admin_password elif cat /etc/*-release | grep -iq 'centos\|fedora\|rhel\|Amazon Linux'; then - PACKAGE_INSTALLER="rpm" installBundleForRPMDistro - generate - + generate_admin_password else printf "Unsupported distribution" exit 1 fi if [[ -n ${NIM_FQDN} ]] ; then - if [ -d "${NIM_CERTS_DIR}" ]; then - echo "removing existing NIM certs" - rm -rf "/etc/nms/certs" - fi - bash -c "source /etc/nms/scripts/certs.sh 0 \"${NIM_FQDN}\" || /etc/nms/scripts/certs.sh 0 ${NIM_FQDN}" + echo "Using FQDN - ${NIM_FQDN}" + sudo rm -rf /etc/nms/certs/* + sudo bash /etc/nms/scripts/certs.sh 0 ${NIM_FQDN} fi + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "true" ]]; then + sed -i '/clickhouse:/a \ enabled: false' /etc/nms/nms.conf + fi + sudo systemctl restart nms curl -s -o /dev/null --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "https://pkgs.nginx.com/nms/?using_install_script=true&app=nim&mode=online" } -printUsageInfo(){ - echo "Usage: $0 [-c /path/to/nginx-repo.crt] [-k /path/to/nginx-repo.key] [-p nginx_plus_version] [-s security_module_version] -i [installable_tar_file_path] [-n nginx_oss_version] [-m mode(online/offline)] [-d distribution (ubuntu20.04,ubuntu22.04,ubuntu24.04,debian11,debian12,centos8,rhel8,rhel9,oracle7,oracle8,amzn2)] [-h print help]" - printf "\n\n -m online/offline. Controls whether to install from the internet or from a package created using this script. \n" - printf "\n -c /path/to/your/ file.\n" - printf "\n -k /path/to/your/ file.\n" - printf "\n -p . Include NGINX Plus version to install as an API gateway. Valid values are 'latest' and specific versions like R32. For a list, see https://docs.nginx.com/nginx/releases/. Supersedes -n.\n" - printf "\n -n . Provide NGINX OSS version to install as an API gateway. Valid values are 'latest' or a specific version like 1.27.1. Ignored if you use -p to specify an NGINX Plus version. For a list, see https://nginx.org/en/download.html .\n" - printf "\n -s . Installs a security module along with NGINX Instance Manager. You can specify latest or a version specified in https://docs.nginx.com/nginx-instance-manager/monitoring/security-monitoring/releases/release-notes/.\n" - printf "\n -i . Include the path with an archive file to support NGINX Instance Manager installation. Requires -m Offline." - printf "\n -d . Include the label of a distribution. Requires -m Offline. This creates a file with NGINX Instance Manager dependencies and NGINX Instance Manager install packages for the specified distribution.\n" - printf "\n -v . NGINX Instance Manager version to install/package.\n" - printf "\n -j . Path to the JWT token file used for license and usage consumption reporting.\n" - printf "\n -r To uninstall NGINX Instance Manager and its dependencies. \n" - printf "\n -l Print supported operating systems.\n" - printf "\n -h Print this help message.\n" - exit 0 -} - -printSupportedOS(){ - printf "This script can be run on the following operating systems" - printf "\n 1. ubuntu20.04(focal)" - printf "\n 2. ubuntu22.04(jammy)" - printf "\n 3. ubuntu24.04(noble)" - printf "\n 4. debian11(bullseye)" - printf "\n 5. debian12(bookworm)" - printf "\n 6. centos8(CentOS 8)" - printf "\n 7. rhel8(Redhat Enterprise Linux Version 8)" - printf "\n 8. rhel9( Redhat Enterprise Linux Version 9)" - printf "\n 9. oracle7(Oracle Linux Version 7)" - printf "\n 10. oracle8(Oracle Linux Version 8)" - printf "\n 11. amzn2(Amazon Linux 2)\n" - exit 0 -} - -check_NIM_status(){ - sleep 5 +check_nim_dashboard_status(){ + sleep 10 GREEN='\033[0;32m' NC='\033[0m' - if ! curl -k https://localhost/ui 2>/dev/null | grep -q "NGINX"; then - sleep 2 - if ! curl -k https://localhost/ui 2>/dev/null | grep -q "NGINX"; then + if ! curl -k https://localhost/ui/ 2>/dev/null | grep -q "NGINX"; then + sleep 60 + if ! curl -k -v https://localhost/ui/ 2>/dev/null| grep -q "NGINX"; then echo "NGINX Instance Manager failed to start" + cat /var/log/nms/nms.log exit 1 else echo -e "${GREEN}NGINX Instance Manager Successfully Started${NC}" @@ -597,41 +464,12 @@ check_NIM_status(){ fi } -check_cert_key_jwt_path(){ - if [[ ! -f "$NGINX_CERT_KEY_PATH" ]]; then - echo "Error: NGINX key not found. Please give cert path using -k" - exit 1 - fi - - if [[ ! -f "$NGINX_CERT_PATH" ]]; then - echo "Error: NGINX cert not found. Please give key path using -c" - exit 1 - fi - - if [[ "$USE_NGINX_PLUS" == true && "$NGINX_PLUS_VERSION" == "latest" ]]; then - if [[ ! -f "$LICENSE_JWT_PATH" ]]; then - echo "Error: JWT License not found. It is required with NGINX plus" - exit 1 - fi - echo "Copying jwt" - if [ ! -d "/etc/nginx" ]; then - mkdir /etc/nginx - check_last_command_status "mkdir /etc/nginx" $? - fi - cp "${LICENSE_JWT_PATH}" "/etc/nginx/license.jwt" - check_last_command_status "cp $LICENSE_JWT_PATH /etc/nginx/license.jwt" $? - fi -} - -check_if_nim_installed(){ - +validate_nim_installation(){ local all_services_present=0 - if nms-core --version > /dev/null 2>&1 && nms-dpm --version > /dev/null 2>&1 && nms-integrations --version > /dev/null 2>&1 \ - && nms-ingestion --version > /dev/null 2>&1 && nginx -version > /dev/null 2>&1; then + && nms-ingestion --version > /dev/null 2>&1; then all_services_present=1 fi - if [[ "$all_services_present" == 1 ]]; then if [ "$UNINSTALL_NIM" == "true" ]; then uninstall_nim @@ -647,6 +485,30 @@ check_if_nim_installed(){ fi } +validate_nginx_paths(){ + if [[ ! -f "$NGINX_CERT_KEY_PATH" ]]; then + echo "Error: NGINX key not found. Please give key path using -k" + exit 1 + fi + if [[ ! -f "$NGINX_CERT_PATH" ]]; then + echo "Error: NGINX cert not found. Please give cert path using -c" + exit 1 + fi + if [[ "$USE_NGINX_PLUS" == true ]]; then + if [[ ! -f "$LICENSE_JWT_PATH" ]]; then + echo "Error: JWT License $LICENSE_JWT_PATH not found. It is required with NGINX plus" + exit 1 + fi + echo "Copying jwt" + if [ ! -d "/etc/nginx" ]; then + mkdir /etc/nginx + check_last_command_status "mkdir /etc/nginx" $? + fi + cp "${LICENSE_JWT_PATH}" "/etc/nginx/license.jwt" + check_last_command_status "cp $LICENSE_JWT_PATH /etc/nginx/license.jwt" $? + fi +} + uninstall_nim(){ echo -e "\nAre you ready to remove all packages and files related to NGINX Instance Manager ? \n\ @@ -656,8 +518,11 @@ This action deletes all files in the following directories: /etc/nms , /etc/ngin if [[ "$response" =~ ^[Yy]$ ]]; then # Clickhouse server, Clickhouse client, clickhouse static, nms, nginx - systemctl stop clickhouse-server - check_last_command_status "systemctl stop clickhouse-server" $? + if systemctl status clickhouse-server &> /dev/null; then + systemctl stop clickhouse-server + check_last_command_status "systemctl stop clickhouse-server" $? + fi + systemctl stop nginx check_last_command_status "systemctl stop nginx" $? systemctl stop nms nms-core nms-dpm nms-ingestion nms-integrations @@ -687,6 +552,7 @@ This action deletes all files in the following directories: /etc/nms , /etc/ngin echo "NGINX Instance Manager Uninstalled successfully" exit 0 else + cat /etc/*release printf "Unsupported distribution" exit 1 fi @@ -697,44 +563,270 @@ This action deletes all files in the following directories: /etc/nms , /etc/ngin fi } -download_third_party_dependencies(){ - if cat /etc/*-release | grep -iq 'debian\|ubuntu'; then - if echo "${target_distribution}" | grep -iq 'debian\|ubuntu'; then - mkdir "${TEMP_DIR}/${target_distribution}/keepalived" - apt-get install --download-only -o Dir::Cache="${TEMP_DIR}/${target_distribution}/keepalived" keepalived - else - if command -v docker >/dev/null 2>&1; then - mkdir "${TEMP_DIR}/${target_distribution}/keepalived" - docker run --rm -it -v "${TEMP_DIR}/${target_distribution}/keepalived":/downloads fedora dnf download --resolve --destdir=/downloads keepalived - else - echo "Cross platform packing requires Docker. Please install Docker and try again." +getLatestPkgVersionFromRepo(){ + repoUrl=$1 + version=$2 + pkg_extension=$3 + if [[ "${pkg_extension}" == "rpm" ]]; then + response=$(curl --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} -sL "${repoUrl}" | awk -F '"' '/href=/ {print $2}' | grep -E "$version"| sort -t'-' -k4,4V | tac) + readarray -t versions < <(printf "%s" "${response}") + else + response=$(curl --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} -sL "${repoUrl}" | awk -F '"' '/href=/ {print $2}' | grep -E "$version"| sort -t'_' -k2,2V | tac) + readarray -t versions < <(printf "%s" "${response}") + fi + if [ "${#versions[@]}" -eq 0 ]; then + printf "Package %s not found. See available versions:" "${versions[@]}" + exit 1; + else + echo "${versions[0]}" + fi + } + +package_nim_offline(){ + if [[ -z ${TARGET_DISTRIBUTION} ]]; then + echo "Error: target distribution is required when mode set to offline." exit 1 fi - fi - elif cat /etc/*-release | grep -iq 'centos\|fedora\|rhel\|Amazon Linux'; then - if echo "${target_distribution}" | grep -iq 'centos\|fedora\|rhel\|Amazon Linux'; then - mkdir "${TEMP_DIR}/${target_distribution}/keepalived" - yumdownloader --destdir="${TEMP_DIR}/${target_distribution}/keepalived" --resolve keepalived - else - if command -v docker >/dev/null 2>&1; then - mkdir -p "${TEMP_DIR}/keepalived" - docker run --rm -it -v "${TEMP_DIR}/keepalived":/tmp/nim ubuntu bash -c "apt-get update && mkdir -p /tmp/nim && apt-get install -y --download-only -o Dir::Cache=\"/tmp/nim\" keepalived" - mkdir "${TEMP_DIR}/${target_distribution}/keepalived" - mv ${TEMP_DIR}/keepalived/archives/* ${TEMP_DIR}/${target_distribution}/keepalived - else - echo "Cross platform packing requires Docker. Please install Docker and try again." - exit 1 + if [[ ! ${SUPPORTED_OS[*]} =~ ${TARGET_DISTRIBUTION} ]]; then + echo "Error: The TARGET_DISTRIBUTION ${TARGET_DISTRIBUTION} is not supported in this script... please select one of the following options - ${SUPPORTED_OS[*]}" + exit 1 + fi + if [[ ${RPM_OS[*]} =~ ${TARGET_DISTRIBUTION} || ${CENT_OS[*]} =~ ${TARGET_DISTRIBUTION} ]]; then + PKG_EXTENSION="rpm" + fi + if [[ ! -d "${TEMP_DIR}/${TARGET_DISTRIBUTION}" ]]; then + echo "creating ${TEMP_DIR}/${TARGET_DISTRIBUTION}" + mkdir -p "${TEMP_DIR}/${TARGET_DISTRIBUTION}" + fi + CWD=$(pwd) + cd "${TEMP_DIR}/${TARGET_DISTRIBUTION}" || echo "directory ${TEMP_DIR} does not exits" + if [[ "${USE_NGINX_PLUS}" == "true" ]]; then + NGINX_PLUS_PACKAGE="^nginx-plus_[0-9]+-([0-9]+)~${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}_${OS_ARCH}\.${PKG_EXTENSION}$" + if [[ "${PKG_EXTENSION}" == "rpm" ]]; then + NGINX_PLUS_PACKAGE="^nginx-plus-[0-9]+-([0-9]+)${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}\.${PKG_EXTENSION}$" + fi + echo "regex for looking latest version : ${NGINX_PLUS_PACKAGE}" + NGINX_PLUS_VERSION=$(getLatestPkgVersionFromRepo "${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}" "${NGINX_PLUS_PACKAGE}" "${PKG_EXTENSION}") + echo "latest version for nginx_plus is ${NGINX_PLUS_VERSION}" + echo "Downloading ${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_PLUS_VERSION}...." + curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_PLUS_VERSION}" + check_last_command_status "curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} \"${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_PLUS_VERSION}\"" $? + else + NGINX_OSS_PACKAGE="^nginx_[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)~${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}_${OS_ARCH}\.${PKG_EXTENSION}$" + if [[ "${PKG_EXTENSION}" == "rpm" ]]; then + NGINX_OSS_PACKAGE="^nginx-[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}\.${PKG_EXTENSION}$" + fi + echo "fetching latest version using ${NGINX_OSS_PACKAGE}" + NGINX_OSS_VERSION=$(getLatestPkgVersionFromRepo "${NGINX_REPO[${TARGET_DISTRIBUTION}]}" "${NGINX_OSS_PACKAGE}" "${PKG_EXTENSION}") + echo "latest version for nginx is ${NGINX_OSS_VERSION}" + echo "Downloading ${NGINX_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_OSS_VERSION}...." + curl -sfLO "${NGINX_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_OSS_VERSION}" + check_last_command_status "curl -sfLO \"${NGINX_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_OSS_VERSION}\"" $? + fi + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "false" ]]; then + CLICKHOUSE_COMMON_PATH="${CLICKHOUSE_REPO[${TARGET_DISTRIBUTION}]}/clickhouse-common-static_${CLICKHOUSE_VERSION}_${OS_ARCH}.${PKG_EXTENSION}" + if [[ "${PKG_EXTENSION}" == "rpm" ]]; then + CLICKHOUSE_COMMON_PATH="${CLICKHOUSE_REPO[${TARGET_DISTRIBUTION}]}/clickhouse-common-static-${CLICKHOUSE_VERSION}.x86_64.${PKG_EXTENSION}" + fi + echo "Downloading ${CLICKHOUSE_COMMON_PATH}...." + curl -sfLO "${CLICKHOUSE_COMMON_PATH}" + check_last_command_status "curl -sfLO \"${CLICKHOUSE_COMMON_PATH}\"" $? + + CLICKHOUSE_SERVER_PATH="${CLICKHOUSE_REPO[${TARGET_DISTRIBUTION}]}/clickhouse-server_${CLICKHOUSE_VERSION}_${OS_ARCH}.${PKG_EXTENSION}" + if [[ "${PKG_EXTENSION}" == "rpm" ]]; then + CLICKHOUSE_SERVER_PATH="${CLICKHOUSE_REPO[${TARGET_DISTRIBUTION}]}/clickhouse-server-${CLICKHOUSE_VERSION}.x86_64.${PKG_EXTENSION}" + fi + echo "Downloading ${CLICKHOUSE_SERVER_PATH}...." + curl -sfLO "${CLICKHOUSE_SERVER_PATH}" + check_last_command_status "curl -sfLO \"${CLICKHOUSE_SERVER_PATH}\"" $? + + CLICKHOUSE_CLIENT_PATH="${CLICKHOUSE_REPO[${TARGET_DISTRIBUTION}]}/clickhouse-client_${CLICKHOUSE_VERSION}_${OS_ARCH}.${PKG_EXTENSION}" + if [[ "${PKG_EXTENSION}" == "rpm" ]]; then + CLICKHOUSE_CLIENT_PATH="${CLICKHOUSE_REPO[${TARGET_DISTRIBUTION}]}/clickhouse-client-${CLICKHOUSE_VERSION}.x86_64.${PKG_EXTENSION}" + fi + echo "Downloading ${CLICKHOUSE_CLIENT_PATH}...." + curl -sfLO "${CLICKHOUSE_CLIENT_PATH}" + check_last_command_status "curl -sfLO \"${CLICKHOUSE_CLIENT_PATH}\"" $? + fi + NIM_PACKAGE_PATH="^nms-instance-manager_[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)~${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}_${OS_ARCH}\.${PKG_EXTENSION}$" + if [[ "${PKG_EXTENSION}" == "rpm" ]]; then + NIM_PACKAGE_PATH="^nms-instance-manager-[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}\.${PKG_EXTENSION}$" + fi + NIM_PACKAGE_VERSION=$(getLatestPkgVersionFromRepo "${NIM_REPO[${TARGET_DISTRIBUTION}]}" "${NIM_PACKAGE_PATH}" "${PKG_EXTENSION}") + echo "Latest version for nginx instance manager is ${NIM_PACKAGE_VERSION}...." + curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "${NIM_REPO[${TARGET_DISTRIBUTION}]}/${NIM_PACKAGE_VERSION}" + check_last_command_status "curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} \"${NIM_REPO[${TARGET_DISTRIBUTION}]}/${NIM_PACKAGE_VERSION}\"" $? + + NIM_ARCHIVE_FILE_NAME="nim-oss-${TARGET_DISTRIBUTION}.tar.gz" + if [[ "${USE_NGINX_PLUS}" == "true" ]]; then + NIM_ARCHIVE_FILE_NAME="nim-plus-${TARGET_DISTRIBUTION}.tar.gz" + fi + echo -n "Creating NGINX instance manager install bundle ... ${NIM_ARCHIVE_FILE_NAME}" + cp ${NGINX_CERT_PATH} "${TEMP_DIR}/${TARGET_DISTRIBUTION}/nginx-repo.crt" + cp ${NGINX_CERT_KEY_PATH} "${TEMP_DIR}/${TARGET_DISTRIBUTION}/nginx-repo.key" + cd "${CWD}" || echo "failed to change directory to ${CWD}" + tar -zcf "/tmp/${NIM_ARCHIVE_FILE_NAME}" -C "${TEMP_DIR}/${TARGET_DISTRIBUTION}" . + cp "/tmp/${NIM_ARCHIVE_FILE_NAME}" "${CWD}" + echo -e "\nSuccessfully created the NGINX Instance Manager bundle - ${NIM_ARCHIVE_FILE_NAME}" + rm -rf "${TEMP_DIR}" || echo "failed to delete the temporary directory ${TEMP_DIR}" + curl -s -o /dev/null --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "https://pkgs.nginx.com/nms/?using_install_script=true&app=nim&mode=offline" +} + +install_nim_offline_from_file(){ + echo "Installing NGINX Instance Manager bundle from the path ${INSTALL_PATH}" + if [ -f "${INSTALL_PATH}" ]; then + if [ ! -f "${TEMP_DIR}" ]; then + mkdir -p "${TEMP_DIR}" + fi + tar xvf "${INSTALL_PATH}" -C "${TEMP_DIR}" + chmod -R 777 "${TEMP_DIR}" + chown -R "${USER}" "${TEMP_DIR}" + if cat /etc/*-release | grep -iq 'debian\|ubuntu'; then + for pkg_nginx in "${TEMP_DIR}"/nginx*.deb; do + echo "Installing nginx from ${pkg_nginx}" + DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_nginx" + check_last_command_status "dpkg -i \"$pkg_nginx\"" $? + done + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "false" ]]; then + for pkg_clickhouse in "${TEMP_DIR}"/clickhouse-common*.deb; do + echo "Installing clickhouse dependencies from ${pkg_clickhouse}" + DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_clickhouse" + check_last_command_status "dpkg -i \"$pkg_clickhouse\"" $? + done + for pkg_clickhouse_srv in "${TEMP_DIR}"/clickhouse-server*.deb; do + echo "Installing clickhouse dependencies from ${pkg_clickhouse_srv}" + DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_clickhouse_srv" + check_last_command_status "dpkg -i \"$pkg_clickhouse_srv\"" $? + done + fi + + for pkg_nim in "${TEMP_DIR}"/nms-instance-manager*.deb; do + echo "Installing NGINX Instance Manager from ${pkg_nim}" + DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_nim" + check_last_command_status "dpkg -i \"$pkg_nim\"" $? + done + generate_admin_password + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "false" ]]; then + echo "Starting clickhouse-server" + systemctl start clickhouse-server fi + echo "Enabling and starting NGINX Instance Manager" + systemctl enable nms nms-core nms-dpm nms-ingestion nms-integrations --now + systemctl start nms nms-core nms-dpm nms-ingestion nms-integrations || journalctl -xeu nms* + echo "Restart nginx configuration" + systemctl restart nginx || journalctl -xeu nginx + check_nim_dashboard_status + + elif cat /etc/*-release | grep -iq 'centos\|fedora\|rhel\|Amazon Linux'; then + yum update -y + for pkg_nginx in "${TEMP_DIR}"/nginx*.rpm; do + echo "Installing nginx from ${pkg_nginx}" + yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_nginx" + done + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "false" ]]; then + for pkg_clickhouse in "${TEMP_DIR}"/clickhouse-common*.rpm; do + echo "Installing clickhouse dependencies from ${pkg_clickhouse}" + yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_clickhouse" + done + for pkg_clickhouse_srv in "${TEMP_DIR}"/clickhouse-server*.rpm; do + echo "Installing clickhouse dependencies from ${pkg_clickhouse}" + yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_clickhouse_srv" + done + fi + for pkg_nim in "${TEMP_DIR}"/nms-instance-manager*.rpm; do + echo "Installing NGINX Instance Manager from ${pkg_nim}" + yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_nim" + done + + generate_admin_password + if [[ ${SKIP_CLICKHOUSE_INSTALL} == "false" ]]; then + echo "Starting clickhouse-server" + systemctl start clickhouse-server + fi + + echo "Enabling and starting NGINX Instance Manager" + systemctl enable nms nms-core nms-dpm nms-ingestion nms-integrations --now + systemctl start nms nms-core nms-dpm nms-ingestion nms-integrations || journalctl -xeu nms* + systemctl restart nginx || journalctl -xeu nginx + check_nim_dashboard_status + + else + echo "Unsupported distribution" + exit 1 + fi + + else + echo "Provided install path ${INSTALL_PATH} doesn't exists" + exit 1 fi - else - printf "Unsupported distribution" - exit 1 - fi + if [[ -n ${NIM_FQDN} ]] ; then + echo "Using FQDN - ${NIM_FQDN}" + sudo rm -rf /etc/nms/certs/* + sudo bash /etc/nms/scripts/certs.sh 0 ${NIM_FQDN} + fi + curl -s -o /dev/null --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "https://pkgs.nginx.com/nms/?using_install_script=true&app=nim&mode=offline" +} + +confirm_action() { + read -p "$1 (y/n)? " -n 1 -r + echo # (optional) move to a new line + if [[ ! $REPLY =~ ^[Yy]$ ]]; then + echo "Installation cancelled by user." + exit 1 + fi + echo "" } -OPTS_STRING="k:c:m:d:i:s:p:n:hv:t:j:rf:l" +printUsageInfo(){ + echo "Usage: $0 [OPTIONS]" + printf "\nThis script is used to install and setup Nginx Instance Manager\n" + printf "\n\n Options:\n" + printf "\n -a arch(amd64/arm64) for the offline packages to download. Valid only when mode is set to offline \n" + printf "\n -c /path/to/your/ file.\n" + printf "\n -d . Include the label of a distribution. Requires -m Offline. This creates a file with NGINX Instance Manager dependencies and NGINX Instance Manager install packages for the specified distribution" + printf "\n [ubuntu20.04,ubuntu22.04,ubuntu24.04,debian11,debian12,centos8,rhel8,rhel9,oracle8,oracle9]\n" + printf "\n -f use to specify the fully qualified domain name to use for generating nim certificates.\n" + printf "\n -h Print this help message.\n" + printf "\n -i . Include the path with an archive file to support NGINX Instance Manager installation. Requires -m Offline.\n" + printf "\n -j . Path to the JWT token file used for license and usage consumption reporting.\n" + printf "\n -k /path/to/your/ file.\n" + printf "\n -l Print supported operating systems.\n" + printf "\n -m online/offline. Controls whether to install from the internet or from a package created using this script. \n" + printf "\n -p Include NGINX Plus as an API gateway. \n" + printf "\n -r To uninstall NGINX Instance Manager and its dependencies. \n" + printf "\n -s Skip packaging/installing clickhouse packages. \n" + printf "\n -v clickhouse version to install/package. \n" + printf "\n -y Install Nginx Instance Manager if all the requirements are passed. \n" + exit 0 +} + +printSupportedOS(){ + printf "This script can be run on the following operating systems" + printf "\n 1. ubuntu20.04(focal)" + printf "\n 2. ubuntu22.04(jammy)" + printf "\n 3. ubuntu24.04(noble)" + printf "\n 4. debian11(bullseye)" + printf "\n 5. debian12(bookworm)" + printf "\n 6. centos8(CentOS 8)" + printf "\n 7. rhel8(Redhat Enterprise Linux Version 8)" + printf "\n 8. rhel9( Redhat Enterprise Linux Version 9)" + printf "\n 9. oracle8(Oracle Linux Version 8)\n" + printf "\n 10. oracle9(Oracle Linux Version 9)\n" + exit 0 +} + +OPTS_STRING="a:c:d:f:hi:j:k:lm:prsv:y" + while getopts ${OPTS_STRING} opt; do case ${opt} in + a) + if [[ "${OPTARG}" != "amd64" && "${OPTARG}" != "arm64" ]]; then + echo "invalid OS arch type ${OPTARG}" + echo "supported values are 'amd64' or 'arm64'" + exit 1 + fi + OS_ARCH=${OPTARG} + ;; c) if [ ! -d "/etc/ssl/nginx" ]; then mkdir /etc/ssl/nginx @@ -743,62 +835,56 @@ while getopts ${OPTS_STRING} opt; do cp "${OPTARG}" ${NGINX_CERT_PATH} check_last_command_status "cp ${OPTARG} ${NGINX_CERT_PATH}" $? ;; - k) - if [ ! -d "/etc/ssl/nginx" ]; then - mkdir /etc/ssl/nginx - check_last_command_status "mkdir /etc/ssl/nginx" $? - fi - cp "${OPTARG}" ${NGINX_CERT_KEY_PATH} - check_last_command_status "cp ${OPTARG} ${NGINX_CERT_KEY_PATH}" $? + d) + TARGET_DISTRIBUTION=${OPTARG} ;; - p) - USE_NGINX_PLUS="true" - NGINX_PLUS_VERSION=${OPTARG} + f) + NIM_FQDN=${OPTARG} ;; - s) - USE_SM_MODULE="true" - NIM_SM_VERSION="${OPTARG}" + h) + printUsageInfo + exit 0 ;; i) INSTALL_PATH=${OPTARG} ;; - n) - NGINX_VERSION=${OPTARG} + j) + LICENSE_JWT_PATH=${OPTARG} + ;; + k) + if [ ! -d "/etc/ssl/nginx" ]; then + mkdir /etc/ssl/nginx + check_last_command_status "mkdir /etc/ssl/nginx" $? + fi + cp "${OPTARG}" ${NGINX_CERT_KEY_PATH} + check_last_command_status "cp ${OPTARG} ${NGINX_CERT_KEY_PATH}" $? + ;; + l) + printSupportedOS + exit 0 ;; m) MODE="${OPTARG}" if [[ "${MODE}" != "online" && "${MODE}" != "offline" ]]; then - echo "invalid mode ${MODE}" - echo "supported values for mode are 'online' or 'offline'" - exit 1 + echo "invalid mode ${MODE}" + echo "supported values for mode are 'online' or 'offline'" + exit 1 fi ;; - d) - TARGET_DISTRIBUTION=${OPTARG} - ;; - v) - NIM_VERSION=${OPTARG} - ;; - j) - LICENSE_JWT_PATH=${OPTARG} - ;; - t) - CLICKHOUSE_VERSION=${OPTARG} + p) + USE_NGINX_PLUS="true" ;; r) UNINSTALL_NIM="true" ;; - f) - NIM_FQDN=${OPTARG} + s) + SKIP_CLICKHOUSE_INSTALL="true" ;; - h) - printUsageInfo - exit 0 + v) + CLICKHOUSE_VERSION=${OPTARG} + ;; + y) ;; - l) - printSupportedOS - exit 0 - ;; :) echo "Option -${OPTARG} requires an argument." exit 1 @@ -810,283 +896,27 @@ while getopts ${OPTS_STRING} opt; do esac done -check_if_nim_installed -check_cert_key_jwt_path +if [[ "$#" == "0" ]]; then + printUsageInfo + exit 0 +fi + +if [[ $EUID -ne 0 ]]; then + echo "This script is not being executed with sudo permissions." + echo "Please run it with sudo, e.g., sudo bash install-nim-bundle.sh" + exit 1 +fi + +validate_nim_installation +validate_nginx_paths if [ "${MODE}" == "online" ]; then install_nim_online - check_NIM_status - + check_nim_dashboard_status else - if [ "${NGINX_VERSION}" == "latest" ]; then - NGINX_VERSION=${NGINX_LATEST_VERSION} - fi - if [ "${NIM_VERSION}" == "latest" ]; then - NIM_VERSION=${NIM_LATEST_VERSION} - fi - if [ "${CLICKHOUSE_VERSION}" == "latest" ]; then - CLICKHOUSE_VERSION=${CLICKHOUSE_LATEST_VERSION} - fi - - declare -A CLICKHOUSE_REPO - CLICKHOUSE_REPO['ubuntu20.04']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse/" - CLICKHOUSE_REPO['ubuntu22.04']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse/" - CLICKHOUSE_REPO['ubuntu24.04']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse/" - CLICKHOUSE_REPO['debian11']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse/" - CLICKHOUSE_REPO['debian12']="https://packages.clickhouse.com/deb/pool/main/c/clickhouse/" - CLICKHOUSE_REPO['centos8']="https://packages.clickhouse.com/rpm/stable/" - CLICKHOUSE_REPO['rhel8']="https://packages.clickhouse.com/rpm/stable/" - CLICKHOUSE_REPO['rhel9']="https://packages.clickhouse.com/rpm/stable/" - CLICKHOUSE_REPO['oracle8']="https://packages.clickhouse.com/rpm/stable/" - CLICKHOUSE_REPO['oracle9']="https://packages.clickhouse.com/rpm/stable/" - CLICKHOUSE_REPO['amzn2']="https://packages.clickhouse.com/rpm/stable/" - - declare -A NGINX_REPO - NGINX_REPO['ubuntu20.04']="https://nginx.org/packages/mainline/ubuntu/pool/nginx/n/nginx/" - NGINX_REPO['ubuntu22.04']="https://nginx.org/packages/mainline/ubuntu/pool/nginx/n/nginx/" - NGINX_REPO['ubuntu24.04']="https://nginx.org/packages/mainline/ubuntu/pool/nginx/n/nginx/" - NGINX_REPO['debian11']="https://nginx.org/packages/mainline/debian/pool/nginx/n/nginx/" - NGINX_REPO['debian12']="https://nginx.org/packages/mainline/debian/pool/nginx/n/nginx/" - NGINX_REPO['centos8']="https://nginx.org/packages/mainline/centos/8/x86_64/RPMS/" - NGINX_REPO['rhel8']="https://nginx.org/packages/mainline/rhel/8/x86_64/RPMS/" - NGINX_REPO['rhel9']="https://nginx.org/packages/mainline/rhel/9/x86_64/RPMS/" - NGINX_REPO['oracle8']="https://nginx.org/packages/mainline/rhel/8/x86_64/RPMS/" - NGINX_REPO['oracle9']="https://nginx.org/packages/mainline/rhel/9/x86_64/RPMS/" - NGINX_REPO['amzn2']="https://nginx.org/packages/mainline/amzn2/2/x86_64/RPMS/" - CLICKHOUSE_KEY="https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key" - NGINX_KEY="https://nginx.org/keys/nginx_signing.key" - - declare -A NMS_REPO - NMS_REPO['ubuntu20.04']="https://pkgs.nginx.com/nms/ubuntu/pool/nginx-plus/n/nms-instance-manager/" - NMS_REPO['ubuntu22.04']="https://pkgs.nginx.com/nms/ubuntu/pool/nginx-plus/n/nms-instance-manager/" - NMS_REPO['ubuntu24.04']="https://pkgs.nginx.com/nms/ubuntu/pool/nginx-plus/n/nms-instance-manager/" - NMS_REPO['debian11']="https://pkgs.nginx.com/nms/debian/pool/nginx-plus/n/nms-instance-manager/" - NMS_REPO['debian12']="https://pkgs.nginx.com/nms/debian/pool/nginx-plus/n/nms-instance-manager/" - NMS_REPO['centos8']="https://pkgs.nginx.com/nms/centos/8/x86_64/RPMS/" - NMS_REPO['rhel8']="https://pkgs.nginx.com/nms/centos/8/x86_64/RPMS/" - NMS_REPO['rhel9']="https://pkgs.nginx.com/nms/centos/9/x86_64/RPMS/" - NMS_REPO['oracle8']="https://pkgs.nginx.com/nms/centos/8/x86_64/RPMS/" - NMS_REPO['oracle9']="https://pkgs.nginx.com/nms/centos/9/x86_64/RPMS/" - NMS_REPO['amzn2']="https://pkgs.nginx.com/nms/amzn2/2/x86_64/RPMS/" - - declare -A CLICKHOUSE_PACKAGES - # for Clickhouse package names are static between distributions - # we use ubuntu/centos entries as placeholders - CLICKHOUSE_PACKAGES['ubuntu']=$(printf "clickhouse-server_%s_amd64.deb\nclickhouse-common-static_%s_amd64.deb" ${CLICKHOUSE_VERSION} ${CLICKHOUSE_VERSION}) - CLICKHOUSE_PACKAGES['centos']=$(printf "clickhouse-server-%s.x86_64.rpm\nclickhouse-common-static-%s.x86_64.rpm" ${CLICKHOUSE_VERSION} ${CLICKHOUSE_VERSION}) - CLICKHOUSE_PACKAGES['ubuntu20.04']=${CLICKHOUSE_PACKAGES['ubuntu']} - CLICKHOUSE_PACKAGES['ubuntu22.04']=${CLICKHOUSE_PACKAGES['ubuntu']} - CLICKHOUSE_PACKAGES['ubuntu24.04']=${CLICKHOUSE_PACKAGES['ubuntu']} - CLICKHOUSE_PACKAGES['debian11']=${CLICKHOUSE_PACKAGES['ubuntu']} - CLICKHOUSE_PACKAGES['debian12']=${CLICKHOUSE_PACKAGES['ubuntu']} - CLICKHOUSE_PACKAGES['centos8']=${CLICKHOUSE_PACKAGES['centos']} - CLICKHOUSE_PACKAGES['rhel8']=${CLICKHOUSE_PACKAGES['centos']} - CLICKHOUSE_PACKAGES['rhel9']=${CLICKHOUSE_PACKAGES['centos']} - CLICKHOUSE_PACKAGES['oracle8']=${CLICKHOUSE_PACKAGES['centos']} - CLICKHOUSE_PACKAGES['oracle9']=${CLICKHOUSE_PACKAGES['centos']} - CLICKHOUSE_PACKAGES['amzn2']=${CLICKHOUSE_PACKAGES['centos']} - - declare -A NGINX_PACKAGES - NGINX_PACKAGES['ubuntu20.04']="nginx_${NGINX_VERSION}~focal_amd64.deb" - NGINX_PACKAGES['ubuntu22.04']="nginx_${NGINX_VERSION}~jammy_amd64.deb" - NGINX_PACKAGES['ubuntu24.04']="nginx_${NGINX_VERSION}~noble_amd64.deb" - NGINX_PACKAGES['debian11']="nginx_${NGINX_VERSION}~bullseye_amd64.deb" - NGINX_PACKAGES['debian12']="nginx_${NGINX_VERSION}~bookworm_amd64.deb" - NGINX_PACKAGES['centos8']="nginx-${NGINX_VERSION}.el8.ngx.x86_64.rpm" - NGINX_PACKAGES['rhel8']="nginx-${NGINX_VERSION}.el8.ngx.x86_64.rpm" - NGINX_PACKAGES['rhel9']="nginx-${NGINX_VERSION}.el9.ngx.x86_64.rpm" - NGINX_PACKAGES['oracle8']="nginx-${NGINX_VERSION}.el8.ngx.x86_64.rpm" - NGINX_PACKAGES['oracle9']="nginx-${NGINX_VERSION}.el9.ngx.x86_64.rpm" - NGINX_PACKAGES['amzn2']="nginx-${NGINX_VERSION}.amzn2.ngx.x86_64.rpm" - - declare -A NIM_PACKAGES - NIM_PACKAGES['ubuntu20.04']="nms-instance-manager_${NIM_VERSION}-\d*~focal_amd64\.deb" - NIM_PACKAGES['ubuntu22.04']="nms-instance-manager_${NIM_VERSION}-\d*~jammy_amd64\.deb" - NIM_PACKAGES['ubuntu24.04']="nms-instance-manager_${NIM_VERSION}-\d*~jammy_amd64\.deb" - NIM_PACKAGES['debian11']="nms-instance-manager_${NIM_VERSION}-\d*~bullseye_amd64\.deb" - NIM_PACKAGES['debian12']="nms-instance-manager_${NIM_VERSION}-\d*~bookworm_amd64\.deb" - NIM_PACKAGES['centos8']="nms-instance-manager-${NIM_VERSION}-\d*.el8.ngx.x86_64\.rpm" - NIM_PACKAGES['rhel8']="nms-instance-manager-${NIM_VERSION}-\d*.el8.ngx.x86_64\.rpm" - NIM_PACKAGES['rhel9']="nms-instance-manager-${NIM_VERSION}-\d*.el9.ngx.x86_64\.rpm" - NIM_PACKAGES['oracle8']="nms-instance-manager-${NIM_VERSION}-\d*.el8.ngx.x86_64\.rpm" - NIM_PACKAGES['oracle9']="nms-instance-manager-${NIM_VERSION}-\d*.el9.ngx.x86_64\.rpm" - NIM_PACKAGES['amzn2']="nms-instance-manager-${NIM_VERSION}-\d*.amzn2.ngx.x86_64\.rpm" - if [ -z "${INSTALL_PATH}" ]; then - target_distribution="$TARGET_DISTRIBUTION" - echo "Target distro - $target_distribution" - if [[ "${#CLICKHOUSE_REPO[$target_distribution]}" -eq 0 ]]; then - echo "Invalid target distribution. Supported target distributions: " "${!CLICKHOUSE_REPO[@]}" - exit 1 - fi - echo "Creating NGINX Instance Manager installation bundle for distribution ${CLICKHOUSE_PACKAGES[${target_distribution}]}..." - if [ -z "${target_distribution}" ]; then - echo "${target_distribution} - no target distribution specified" - exit 1 - fi - mkdir -p "${TEMP_DIR}/${target_distribution}" - echo "Downloading clickhouse signing keys ${CLICKHOUSE_KEY}... " - url_file_download ${CLICKHOUSE_KEY} "${TEMP_DIR}/${target_distribution}/clickhouse-key.gpg" - echo "Downloaded clickhouse signing keys" - - echo "downloading nginx signing keys ${NGINX_KEY}... " - url_file_download ${NGINX_KEY} "${TEMP_DIR}/${target_distribution}/nginx-key.gpg" - echo "Downloaded nginx signing keys" - - readarray -t clickhouse_files <<<"${CLICKHOUSE_PACKAGES[$target_distribution]}" - readarray -t nginx_files <<<"${NGINX_PACKAGES[$target_distribution]}" - readarray -t nim_files <<<"${NIM_PACKAGES[$target_distribution]}" - - for package_file in "${clickhouse_files[@]}"; do - if [ -z "$package_file" ]; then - continue - fi - file_to_download="${CLICKHOUSE_REPO[$target_distribution]}$package_file" - save_path="${TEMP_DIR}/${target_distribution}/$package_file" - echo -n "Downloading ${file_to_download} ... " - url_file_download "$file_to_download" "$save_path" - echo "Downloaded clickhouse package - $save_path" - done - for package_file in "${nginx_files[@]}"; do - if [ -z "$package_file" ]; then - continue - fi - file_to_download="${NGINX_REPO[$target_distribution]}$package_file" - save_path="${TEMP_DIR}/${target_distribution}/$package_file" - echo -n "Downloading ${package_file} ... " - url_file_download "$file_to_download" "$save_path" - echo "Downloaded nginx package - $save_path" - done - for package_file in "${nim_files[@]}"; do - if [ -z "$package_file" ]; then - continue - fi - nim_with_version=$(curl -fs --cert "${NGINX_CERT_PATH}" --key "${NGINX_CERT_KEY_PATH}" "${NMS_REPO[$target_distribution]}" \ - | grep -P "${package_file}" | sed -n 's/.*]*href="\([^"]*\)".*/\1/p') - if [[ "${#nim_with_version}" -eq 0 ]]; then - echo "Error: NGINX Instance Manager $NIM_VERSION ($target_distribution) version not found on ${NMS_REPO[$target_distribution]}" - exit 1 - fi - file_to_download="${NMS_REPO[$target_distribution]}${nim_with_version}" - save_path="${TEMP_DIR}/${target_distribution}/$nim_with_version" - echo -n "Downloading ${nim_with_version} ... " - url_file_download "$file_to_download" "$save_path" - echo "Downloaded NGINX Instance Manager package - $save_path" - done - download_third_party_dependencies - bundle_file="nim-${NIM_VERSION}-${target_distribution}.tar.gz" - echo -n "Creating NGINX Instance Manager install bundle ... ${bundle_file}" - cp ${NGINX_CERT_PATH} "${TEMP_DIR}/${target_distribution}/nginx-repo.crt" - cp ${NGINX_CERT_KEY_PATH} "${TEMP_DIR}/${target_distribution}/nginx-repo.key" - tar -zcf "$bundle_file" -C "${TEMP_DIR}/${target_distribution}/" . - echo -e "\nSuccessfully created the NGINX Instance Manager bundle - $bundle_file" - curl -s -o /dev/null --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "https://pkgs.nginx.com/nms/?using_install_script=true&app=nim&mode=offline" - + package_nim_offline else - echo "Installing NGINX Instance Manager bundle from the path ${INSTALL_PATH}" - if [ -f "${INSTALL_PATH}" ]; then - if [ ! -f "${TEMP_DIR}" ]; then - mkdir -p "${TEMP_DIR}" - fi - tar xvf "${INSTALL_PATH}" -C "${TEMP_DIR}" - chmod -R 777 "${TEMP_DIR}" - chown -R "${USER}" "${TEMP_DIR}" - if cat /etc/*-release | grep -iq 'debian\|ubuntu'; then - for pkg_nginx in "${TEMP_DIR}"/nginx*.deb; do - echo "Installing nginx from ${pkg_nginx}" - DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_nginx" - check_last_command_status "dpkg -i \"$pkg_nginx\"" $? - done - for pkg_clickhouse in "${TEMP_DIR}"/clickhouse-common*.deb; do - echo "Installing clickhouse dependencies from ${pkg_clickhouse}" - DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_clickhouse" - check_last_command_status "dpkg -i \"$pkg_clickhouse\"" $? - done - for pkg_clickhouse_srv in "${TEMP_DIR}"/clickhouse-server*.deb; do - echo "Installing clickhouse dependencies from ${pkg_clickhouse_srv}" - DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_clickhouse_srv" - check_last_command_status "dpkg -i \"$pkg_clickhouse_srv\"" $? - done - if [ -d "${TEMP_DIR}/keepalived" ]; then - echo "Installing keepalived from ${TEMP_DIR}/keepalived" - DEBIAN_FRONTEND=noninteractive dpkg -i ${TEMP_DIR}/keepalived/*.deb - check_last_command_status "dpkg -i ${TEMP_DIR}/keepalived/*.deb" $? - fi - for pkg_nim in "${TEMP_DIR}"/nms-instance-manager*.deb; do - echo "Installing NGINX Instance Manager from ${pkg_nim}" - DEBIAN_FRONTEND=noninteractive dpkg -i "$pkg_nim" - check_last_command_status "dpkg -i \"$pkg_nim\"" $? - done - - generate - echo "Starting clickhouse-server" - systemctl start clickhouse-server - - echo "Starting nginx" - systemctl start nginx - - echo "Reloading nginx configuration" - systemctl restart nginx - - echo "Enabling and starting NGINX Instance Manager" - systemctl enable nms nms-core nms-dpm nms-ingestion nms-integrations --now - - check_NIM_status - - elif cat /etc/*-release | grep -iq 'centos\|fedora\|rhel\|Amazon Linux'; then - for pkg_nginx in "${TEMP_DIR}"/nginx*.rpm; do - echo "Installing nginx from ${pkg_nginx}" - yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_nginx" - done - for pkg_clickhouse in "${TEMP_DIR}"/clickhouse-common*.rpm; do - echo "Installing clickhouse dependencies from ${pkg_clickhouse}" - yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_clickhouse" - done - for pkg_clickhouse_srv in "${TEMP_DIR}"/clickhouse-server*.rpm; do - echo "Installing clickhouse dependencies from ${pkg_clickhouse}" - yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_clickhouse_srv" - done - if [ -d "${TEMP_DIR}/keepalived" ]; then - echo "Installing keepalived from ${TEMP_DIR}/keepalived" - yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "${TEMP_DIR}/keepalived/*.rpm" - check_last_command_status "dpkg -i yum localinstall -y -v --disableplugin=subscription-manager --skip-broken ${TEMP_DIR}/keepalived/*.rpm" $? - fi - for pkg_nim in "${TEMP_DIR}"/nms-instance-manager*.rpm; do - echo "Installing NGINX Instance Manager from ${pkg_nim}" - yum localinstall -y -v --disableplugin=subscription-manager --skip-broken "$pkg_nim" - done - - generate - echo "Starting clickhouse-server" - systemctl start clickhouse-server - - echo "Starting nginx" - systemctl start nginx - - echo "Reloading nginx configuration" - systemctl restart nginx - - echo "Enabling and starting NGINX Instance Manager" - systemctl enable nms nms-core nms-dpm nms-ingestion nms-integrations --now - - check_NIM_status - - else - echo "Unsupported distribution" - exit 1 - fi - - else - echo "Provided install path ${INSTALL_PATH} doesn't exists" - exit 1 - fi - if [[ -n ${NIM_FQDN} ]] ; then - if [ -d "${NIM_CERTS_DIR}" ]; then - echo "removing existing nim certs" - rm -rf "/etc/nms/certs" - fi - bash -c "source /etc/nms/scripts/certs.sh 0 \"${NIM_FQDN}\" || /etc/nms/scripts/certs.sh 0 ${NIM_FQDN}" - fi - curl -s -o /dev/null --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "https://pkgs.nginx.com/nms/?using_install_script=true&app=nim&mode=offline" + install_nim_offline_from_file fi fi diff --git a/static/scripts/license_usage_offline.sh b/static/scripts/license_usage_offline.sh index 8a1a1eabd..be2cd3a65 100755 --- a/static/scripts/license_usage_offline.sh +++ b/static/scripts/license_usage_offline.sh @@ -1,282 +1,282 @@ #!/bin/bash -# Function to encode the username and password to Base64 -encode_base64() { - echo -n "$1:$2" | base64 -} +# Enable strict mode for better error handling +set -euo pipefail +IFS=$'\n\t' -# Print help text and exit. -if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then - echo - echo "Usage: $0 -j -i -u -p -o -s " - echo - echo "This script alows you to license NGINX Instance Manager and submit usage reports to F5 in a disconnected environment." - echo "It needs to be run on a system that can reach NGINX Instance Manager and the following endpoint on port 443: https://product.apis.f5.com/" +# Function to display usage +usage() { + echo "Usage: $0 -j -i -u -p -s " echo - echo "If NGINX Instance Manager is connected to the internet (Connected Mode), please use the UI." - echo "This script is only necessary if NGINX Instance Manager is set to Disconnected Mode." - echo - echo "For Licensing in Disconnected Mode:" - echo "$0 -j my-jwt.jwt -i -u admin -p -o report.zip -s initial" - echo - echo "For Usage Reporting in Disconnected Mode:" - echo "$0 -j my-jwt.jwt -i -u admin -p -o report.zip -s telemetry" - echo - echo "Note: Since NGINX Instance Manager comes with self-signed certificates by default, the --insecure flag is set in this script to run specific Curl commands." - echo "You can alter this script if you wish to change from self-signed to verified certificates in Instance Manager." - echo + echo "Options:" + echo " -j Path to the JWT (JSON Web Token) file used for authentication." + echo " -i IP address of the NIM (NGINX Instance Manager) to connect to." + echo " -u Username for login/authentication to NIM (NGINX Instance Manager)." + echo " -p Password corresponding to the username for NIM (NGINX Instance Manager) authentication." + echo " -s Script execution mode. One of the following:" + echo " initial - Perform Initial License Activation" + echo " telemetry - Perform telemetry submission: download usage report from NGINX Instance Manager and submit to F5." exit 1 -fi - -if ! command -v jq &> /dev/null; then - echo -e "\nPlease install jq (https://jqlang.github.io/jq/) as it is required to run the script\n" - exit 1 -fi +} # Parse command-line arguments -while getopts ":j:i:u:p:o:s:" opt; do +while getopts ":j:i:u:p:s:" opt; do case $opt in - j) jwt_file="$OPTARG" ;; - i) ip_address="$OPTARG" ;; - u) username="$OPTARG" ;; - p) password="$OPTARG" ;; - o) output_file="$OPTARG" ;; - s) step="$OPTARG" ;; - \?) echo "Invalid option -$OPTARG" >&2 ;; + j) JWT_FILE="$OPTARG" ;; + i) NIM_IP="$OPTARG" ;; + u) USERNAME="$OPTARG" ;; + p) PASSWORD="$OPTARG" ;; + s) USE_CASE="$OPTARG" ;; + *) usage ;; esac done +# Check if all required arguments are provided +if [ -z "${JWT_FILE:-}" ] || [ -z "${NIM_IP:-}" ] || [ -z "${USERNAME:-}" ] || [ -z "${PASSWORD:-}" ] || [ -z "${USE_CASE:-}" ]; then + usage +fi -# Check if JWT_FILE is not empty and if the file exists -if [ -z "$jwt_file" ]; then - echo "JWT file path is not defined. Please set the JWT_FILE variable." - exit 1 -elif [ ! -f "$jwt_file" ]; then - echo "JWT file not found: $jwt_file" - exit 1 -else - echo "JWT file found: $jwt_file" +echo "Running $USE_CASE report" + +# Ensure /tmp directory exists or else create it and proceed +if [ ! -d "/tmp" ]; then + echo "/tmp directory does not exist. Creating it now..." + mkdir -p /tmp || { echo "Failed to create /tmp directory. Exiting."; exit 1; } fi -# Read the Bearer token from the file -bearer_token=$(<"$jwt_file") +# Read JWT contents +if [ ! -f "$JWT_FILE" ]; then + echo -e "JWT file '$JWT_FILE' not found.$" >&2 + exit 1 +fi +JWT_CONTENT=$(<"$JWT_FILE") -# Encode the username and password to Base64 for Basic Authorization -basic_auth=$(encode_base64 "$username" "$password") +# Encode credentials +AUTH_HEADER=$(echo -n "$USERNAME:$PASSWORD" | base64) +# Check connectivity to NGINX Instance Manager IP and the F5 licensing server (product.apis.f5.com) +echo -e "Checking connectivity to NGINX Instance Manager and F5 licensing server..." -# Initialize report_save_path if it's empty -report_save_path=${output_file:-"report.zip"} +# Function to test ping +check_ping() { + local host=$1 + echo "Pinging $host... " + if ! ping -c 2 -W 2 "$host" > /dev/null 2>&1; then + echo -e "Cannot reach $host. Please check your network, DNS or check if any proxy is set.$" >&2 + exit 1 + fi + echo -e "$host is reachable." +} -############################################################################ -# Step 1: Upload JWT to NGINX Instance Manager and Download telemetry report -############################################################################ -echo "###################################################################" -echo "# (if not already licensed) Upload JWT and Download the Telemetry report from NGINX Instance Manager #" -echo "###################################################################" -if [ "$step" == "initial" ]; then +# Call the function for each host +is_ipv4() { + local ip=$1 + if [[ $ip =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]]; then + IFS='.' read -r -a octets <<< "$ip" + for octet in "${octets[@]}"; do + if ((octet < 0 || octet > 255)); then + return 1 + fi + done + return 0 + else + return 1 + fi +} - # Function to print banner - print_banner() { - echo "====================================" - echo "$1" - echo "====================================" +if is_ipv4 "$NIM_IP"; then + check_ping "$NIM_IP" +else + echo "Checking connectivity to NGINX Instance Manager using Curl ..." + if ! curl -sk --output /dev/null --silent --fail "https://$NIM_IP"; then + echo -e "The NGINX Instance Manager UI is not reachable on $NIM_IP" + exit 1 + fi +fi +check_ping "product.apis.f5.com" + +# NGINX Instance Manager Version check +VERSION_JSON=$(curl -sk -X GET "https://$NIM_IP/api/platform/v1/modules/versions" \ + --header "Content-Type: application/json" \ + --header "Authorization: Basic $AUTH_HEADER") +NIM_VER=$(echo "$VERSION_JSON" | sed -E 's/.*"nim"[ \t]*:[ \t]*"([0-9]+\.[0-9]+)(\.[0-9]+)?".*/\1/') +echo "Current version of NGINX Instance Manager is $NIM_VER" +JWT_CONTENT=$(<"$JWT_FILE") + +# Construct JSON payload +JSON_PAYLOAD=$(cat <&2 + exit 1 + fi + # Step 1: Apply JWT license (only if use case is 'initial' or 'intial_only') + echo "Applying JWT license" + RESPONSE=$(curl -sS -k --max-time 10 -w "\n%{http_code}" -X POST "https://$NIM_IP/api/platform/v1/license?telemetry=true" \ + -H "Origin: $ORIGIN" \ + -H "Referer: $REFERER" \ + -H "Content-Type: application/json" \ + -H "Authorization: Basic $AUTH_HEADER" \ + -d "$JSON_PAYLOAD") + echo "Uploaded License" + HTTP_BODY=$(echo "$RESPONSE" | sed '$d') + HTTP_STATUS=$(echo "$RESPONSE" | tail -n1) + if [ "$HTTP_STATUS" -ne 202 ]; then + echo -e "HTTP request failed with status code $HTTP_STATUS. + Response: $HTTP_BODY$" >&2 + if echo "$HTTP_BODY" | jq -r '.message' | grep -q "failed to register token. already registered"; then + echo -e "NGINX Instance Manager already registered and licensed. + If needed, terminate the current license manually in the NGINX Instance Manager UI and re-run the script with the correct license. + https://docs.nginx.com/nginx-instance-manager/disconnected/add-license-disconnected-deployment/" + fi + exit 1 + fi fi -if [ "$step" == "telemetry" ]; then - echo "Running telemetry stage: " - - # Run the saved command and store the response - response=$(eval $prepare_usage_command) - - # Print the response - echo "Response: $response" - sleep 2 - # Validate if the response contains "Report generation in progress" - if echo "$response" | grep -q '"telemetry":"Report generation in progress"'; then - echo "Success: Report generation is in progress." + +if [[ "$NIM_VER" < "2.18" ]]; then + echo "NGINX Instance Manager version $NIM_VER is not supported by this script. Please use NGINX Instance Manager 2.18 or later" + exit 1 + +elif [[ "$NIM_VER" == "2.18" ]] || [[ "$NIM_VER" == "2.19" ]]; then + echo "NGINX Instance Manager version $NIM_VER detected." + ORIGIN="https://$NIM_IP" + +# Send the PUT request and separate body and status code + PUT_RESPONSE_CODE=$(curl -k -s -w "%{http_code}" -o /tmp/put_response.json --location --request PUT "https://$NIM_IP/api/platform/v1/license?telemetry=true" \ + --header "Origin: $ORIGIN" \ + --header "Referer: https://$NIM_IP/ui/settings/license" \ + --header "Content-Type: application/json" \ + --header "Authorization: Basic $AUTH_HEADER" \ + --data '{ + "desiredState": { + "content": "'"$JWT_CONTENT"'", + "type": "JWT", + "features": [ + {"limit": 0, "name": "NGINX_NAP_DOS", "valueType": ""}, + {"limit": 0, "name": "IM_INSTANCES", "valueType": ""}, + {"limit": 0, "name": "TM_INSTANCES", "valueType": ""}, + {"limit": 0, "name": "DATA_PER_HOUR_GB", "valueType": ""}, + {"limit": 0, "name": "NGINX_INSTANCES", "valueType": ""}, + {"limit": 0, "name": "NGINX_NAP", "valueType": ""}, + {"limit": 0, "name": "SUCCESSFUL_API_CALLS_MILLIONS", "valueType": ""}, + {"limit": 0, "name": "IC_PODS", "valueType": ""}, + {"limit": 0, "name": "IC_K8S_NODES", "valueType": ""} + ] + }, + "metadata": { + "name": "license" + } + }') + + echo "Response status code: $PUT_RESPONSE_CODE" + + if [[ "$PUT_RESPONSE_CODE" == "200" ]]; then + echo -e "(legacy): License applied successfully in DISCONNECTED mode for version $NIM_VER." else - echo "Failure: Report generation not in progress or unexpected response." + echo -e "(legacy): License PUT request failed. Status code: $PUT_RESPONSE_CODE$" + echo "clear the license database and re-trigger the script again" exit 1 fi +fi + +if [[ "$USE_CASE" != "telemetry" ]]; then + + RESPONSE=$(curl -sS -k --max-time 10 -w "\n%{http_code}" -X POST "https://$NIM_IP/api/platform/v1/license?telemetry=true" \ + -H "Origin: $ORIGIN" \ + -H "Referer: $REFERER" \ + -H "Content-Type: application/json" \ + -H "Authorization: Basic $AUTH_HEADER" \ + -d "$JSON_PAYLOAD") + + HTTP_BODY=$(echo "$RESPONSE" | sed '$d') + HTTP_STATUS=$(echo "$RESPONSE" | tail -n1) + + echo -e "License applied successfully in DISCONNECTED mode." +fi + + + +# Continue with further steps for version >= 2.20... +echo "Executing telemetry tasks " +# Step 2: Download the usage report +echo "Downloading usage report from NGINX Instance Manager at $NIM_IP..." +if [[ "$NIM_VER" == "2.18" ]] || [[ "$NIM_VER" == "2.19" ]]; then + if [[ "$USE_CASE" == "initial" ]]; then + # Perform the request and capture the status code and output + HTTP_RESPONSE=$(curl -k -sS -w "\n%{http_code}" --location "https://$NIM_IP/api/platform/v1/report/download?format=zip&reportType=initial" \ + --header "accept: application/json" \ + --header "authorization: Basic $AUTH_HEADER" \ + --header "content-type: application/json" \ + --header "origin: https://$NIM_IP" \ + --output /tmp/response.zip) + else + prepare_usage_command="curl --insecure --location 'https://$NIM_IP/api/platform/v1/report/download?format=zip&reportType=telemetry&telemetryAction=prepare' \ + --header 'accept: application/json' \ + --header 'authorization: Basic $AUTH_HEADER' \ + --header 'referer: https://$NIM_IP/ui/settings/license'" + report_save_path="${output_file:-/tmp/response.zip}" + + download_usage_command="curl --insecure --location 'https://$NIM_IP/api/platform/v1/report/download?format=zip&reportType=telemetry&telemetryAction=download' \ + --header 'accept: */*' \ + --header 'authorization: Basic $AUTH_HEADER' \ + --output \"$report_save_path\"" + if [ "$USE_CASE" == "telemetry" ]; then + echo "Running telemetry stage: " + + # Run the saved command and store the response + response=$(eval $prepare_usage_command) + + sleep 2 + # Validate if the response contains "Report generation in progress" + if echo "$response" | grep -q '"telemetry":"Report generation in progress"'; then + echo -e "Success: Report generation is in progress." + else + echo -e "Failure: Report generation not in progress or unexpected response." + exit 1 + fi echo "Running command: $download_usage_command" eval $download_usage_command @@ -284,122 +284,187 @@ if [ "$step" == "telemetry" ]; then echo "Running command: $download_usage_command" eval $download_usage_command fi + fi +else + # Perform the request and capture the status code and output + HTTP_RESPONSE=$(curl -k -sS -w "\n%{http_code}" --location "https://$NIM_IP/api/platform/v1/report/download?format=zip" \ + --header "accept: application/json" \ + --header "authorization: Basic $AUTH_HEADER" \ + --header "content-type: application/json" \ + --header "origin: https://$NIM_IP" \ + --output /tmp/response.zip) + # Extract the HTTP status code (last line) + HTTP_STATUS=$(echo "$HTTP_RESPONSE" | tail -n1) + + # Check the status code + if [ "$HTTP_STATUS" -ne 200 ]; then + echo -e "Failed to download usage report from NGINX Instance Manager. HTTP Status Code: $HTTP_STATUS" >&2 + echo "Please verify that NGINX Instance Manager is reachable and the credentials are correct." >&2 + echo "(or) Verify that NGINX Instance Manager is licensed before using the 'telemetry' flag (run it with 'initial' first)." + # Optionally, remove the partial or corrupt file + rm -f /tmp/response.zip + exit 1 + fi +fi + +echo -e "Usage report downloaded successfully as '/tmp/response.zip'." + +# Step 3: Upload the usage report to F5 server +echo "Uploading the usage report to F5 Licensing server" + +TEEM_UPLOAD_URL="https://product.apis.f5.com/ee/v1/entitlements/telemetry/bulk" + +# Capture both response body and status code +UPLOAD_RESULT=$(curl -sS -w "\n%{http_code}" --location "$TEEM_UPLOAD_URL" \ + --header "Authorization: Bearer $JWT_CONTENT" \ + --form "file=@/tmp/response.zip") + +# Extract HTTP status code and body +UPLOAD_STATUS=$(echo "$UPLOAD_RESULT" | tail -n1) +UPLOAD_BODY=$(echo "$UPLOAD_RESULT" | sed '$d') + +# Validate HTTP status code +if [ "$UPLOAD_STATUS" -ne 202 ]; then + echo -e "Usage report upload failed. HTTP Status: $UPLOAD_STATUS$" >&2 + echo "Response Body: $UPLOAD_BODY" >&2 + exit 1 +fi + +# Check if response is valid JSON +if ! echo "$UPLOAD_BODY" | jq empty >/dev/null 2>&1; then + echo -e "Upload response is not valid JSON. Response: $UPLOAD_BODY$" >&2 + exit 1 +fi + +# Extract the statusLink +STATUS_LINK=$(echo "$UPLOAD_BODY" | jq -r '.statusLink // empty') -############################################################ -# Step 3: Upload the telemetry report to F5's licensing endpoint -############################################################ -echo "############################################################" -echo "# Upload the telemetry report to F5's licensing endpoint #" -echo "############################################################" - -echo "Uploading $report_save_path to telemetry endpoint..." -upload_command="curl --location 'https://product.apis.f5.com/ee/v1/entitlements/telemetry/bulk' \ ---header 'Authorization: Bearer $bearer_token' \ ---form 'file=@\"$report_save_path\"'" - -response=$(eval $upload_command) -echo "Response from upload: $response" - -############################################################ -# Step 4: Extract the status ID from the response -############################################################ -echo "############################################################" -echo "# Extract status ID from the response#" -echo "############################################################" - -status_link=$(jq -r '.statusLink | split("/") | last' <<< "$response") -if [ -z "$status_link" ]; then - echo "Failed to extract status link from response. Please try again." +if [ -z "$STATUS_LINK" ]; then + echo -e "Failed to extract statusLink from the upload response. Response: $UPLOAD_BODY$" >&2 exit 1 fi -echo "Extracted status link ID: $status_link" -############################################################## -# Step 5: Check the status to ensure the acknowledgement file is ready to download -############################################################ -echo "############################################################" -echo "# Check the status to ensure the acknowledgement file is ready to download #" -echo "############################################################" +echo "StatusLink extracted: $STATUS_LINK" -# Initialize variables for the loop -max_attempts=10 -attempt=0 +# Extract the unique status ID from the statusLink +STATUS_ID=$(echo "$STATUS_LINK" | sed 's|/ee/v1/entitlements/telemetry/bulk/status/||') -# Loop to check the status until percentageComplete and percentageSuccessful are 100 or the max attempts is reached -while [ $attempt -lt $max_attempts ]; do - echo "Checking status for the upload... (Attempt: $((attempt + 1)))" +# Step 4: Validate the submit report status +echo "Validating the report status" +echo "Validating report status using status ID: $STATUS_LINK" - # Run the curl command to get the status - status_command="curl --location 'https://product.apis.f5.com/ee/v1/entitlements/telemetry/bulk/status/$status_link' --header 'Authorization: Bearer $bearer_token'" +STATUS_URL="https://product.apis.f5.com/ee/v1/entitlements/telemetry/bulk/status/$STATUS_ID" - status_response=$(eval $status_command) +# Get the response from F5 status endpoint +sleep 5 +STATUS_RESPONSE=$(curl -k -sS -w "\n%{http_code}" --location "$STATUS_URL" \ + --header "Authorization: Bearer $JWT_CONTENT") - # Extract values from the response - percentage_complete=$(jq -r '.percentageComplete' <<< "$status_response") - percentage_successful=$(jq -r '.percentageSuccessful' <<< "$status_response") - ready_for_download=$(jq -r '.readyForDownload' <<< "$status_response") +# Separate body and HTTP status code +STATUS_BODY=$(echo "$STATUS_RESPONSE" | sed '$d') +STATUS_CODE=$(echo "$STATUS_RESPONSE" | tail -n1) - echo "Percentage Complete: $percentage_complete" - echo "Percentage Successful: $percentage_successful" - echo "Ready for Download: $ready_for_download" +# Validate HTTP status +if [ "$STATUS_CODE" -ne 200 ]; then + echo -e "Status check failed. HTTP Status: $STATUS_CODE$" >&2 + echo "Response Body: $STATUS_BODY" >&2 + exit 1 +fi + +# Validate that it's valid JSON +if ! echo "$STATUS_BODY" | jq empty >/dev/null 2>&1; then + echo -e "Invalid JSON in status body: $STATUS_BODY$" >&2 + exit 1 +fi + +# Now safely extract values +PERCENTAGE_COMPLETE=$(echo "$STATUS_BODY" | jq -r '.percentageComplete') +PERCENTAGE_SUCCESSFUL=$(echo "$STATUS_BODY" | jq -r '.percentageSuccessful') +READY_FOR_DOWNLOAD=$(echo "$STATUS_BODY" | jq -r '.readyForDownload') - # Check if the report is ready for download - if [ "$percentage_complete" == "100" ] && [ "$percentage_successful" == "100" ] && [ "$ready_for_download" == "true" ]; then - echo "File is ready for download." +# Set a time limit (in seconds) +TIME_LIMIT=30 # 30 sec for example +START_TIME=$(date +%s) + +# Function to calculate elapsed time +elapsed_time() { + echo $(($(date +%s) - $START_TIME)) +} + +# Keep checking until conditions are met or time limit is reached +while true; do + # Validate all required conditions + if [ "$PERCENTAGE_COMPLETE" -eq "100" ] && [ "$READY_FOR_DOWNLOAD" == "true" ]; then + echo -e "Validating Report." break - else - echo "File is not ready for download yet. Sleeping for 2 seconds..." - sleep 2 fi - # Increment the attempt counter - attempt=$((attempt + 1)) + # Check if we've exceeded the time limit + if [ $(elapsed_time) -ge "$TIME_LIMIT" ]; then + echo -e "Time limit exceeded. Report validation failed." + echo " percentageComplete: $PERCENTAGE_COMPLETE" + echo " percentageSuccessful: $PERCENTAGE_SUCCESSFUL" + echo " readyForDownload: $READY_FOR_DOWNLOAD" + echo " F5 upload issue failed even after 30 seconds" + echo " re-run the script" + exit 1 + fi + + # Print current validation status + echo -e "Report validation failed. Waiting for conditions to be met...$" + echo " percentageComplete: $PERCENTAGE_COMPLETE" + echo " percentageSuccessful: $PERCENTAGE_SUCCESSFUL" + echo " readyForDownload: $READY_FOR_DOWNLOAD" + + # Sleep for some time before checking again (e.g., 30 seconds) + sleep 5 done -# If after 10 attempts it's still not ready, show a message -if [ $attempt -eq $max_attempts ]; then - echo "Reached maximum attempts. The file is not ready for download." +echo -e "Report validated successfully. All conditions met." + +# Step 5: Download the report from F5 + +echo "Downloading report from F5 License server..." + +DOWNLOAD_URL="https://product.apis.f5.com/ee/v1/entitlements/telemetry/bulk/download/$STATUS_ID" +DOWNLOAD_RESPONSE=$(curl -sS -w "%{http_code}" --location "$DOWNLOAD_URL" \ + --header "Authorization: Bearer $JWT_CONTENT" \ + --output /tmp/response_teem.zip) + +# Check HTTP status code +HTTP_STATUS=$(echo "$DOWNLOAD_RESPONSE" | tail -n1) +if [ "$HTTP_STATUS" -ne 200 ]; then + echo -e "Failed to download the report from F5. HTTP Status Code: $HTTP_STATUS$" >&2 exit 1 fi -############################################################ -# Step 6: Download the acknowledgement file from F5's licensing endpoint -############################################################ -echo "############################################################" -echo "# Download the acknowledgement report from F5's licensing endpoint #" -echo "############################################################" - -# Extract downloadLink -download_link=$(jq -r '.downloadLink | split("/") | last' <<< "$status_response") -echo $download_link -# Ensure the report_save_path is not empty -if [ -z "$report_save_path" ]; then - # If the report_save_path is empty, set a default path and filename - report_save_path="report_download.zip" -fi +echo -e "Report downloaded successfully from F5 as '/tmp/response_teem.zip'." -echo "Downloading file from telemetry..." +# Step 6: Upload the acknowledgement report to NGINX Instance Manager +echo "Uploading the license acknowledgement to NGINX Instance Manager..." -download_command="curl --location 'https://product.apis.f5.com/ee/v1/entitlements/telemetry/bulk/download/$download_link' \ - --header 'Authorization: Bearer $bearer_token' \ - --output '$report_save_path'" +UPLOAD_URL="https://$NIM_IP/api/platform/v1/report/upload" +UPLOAD_RESPONSE=$(curl -k -sS --location "$UPLOAD_URL" \ + --header "Authorization: Basic $AUTH_HEADER" \ + --form "file=@/tmp/response_teem.zip" \ + -w "%{http_code}" -o /tmp/temp_response.json) -curl --location "https://product.apis.f5.com/ee/v1/entitlements/telemetry/bulk/download/$download_link" \ ---header "Authorization: Bearer $bearer_token" \ ---output "$report_save_path" +# Extract the HTTP status code +HTTP_STATUS=$(echo "$UPLOAD_RESPONSE" | tail -n1) -echo "Downloaded file saved as: $report_save_path" +# Extract the message from the response body (temp_response.json) +UPLOAD_MESSAGE=$(cat /tmp/temp_response.json | jq -r '.message') -############################################################ -# Step 7: Upload the acknowledgement report to NGINX Instance Manager -############################################################ -echo "############################################################" -echo "# Upload the acknowledgement report to NGINX Instance Manager #" -echo "############################################################" +# Ensure HTTP_STATUS is a valid integer +if ! [[ "$HTTP_STATUS" =~ ^[0-9]+$ ]]; then + echo -e "Invalid HTTP status code. Response: $UPLOAD_RESPONSE$" >&2 + exit 1 +fi -curl --insecure --location "https://$ip_address/api/platform/v1/report/upload" \ ---header "Authorization: Basic $basic_auth" \ ---form "file=@\"$report_save_path\"" \ ---silent --output /dev/null +# Check if the upload response contains the success message and validate the status code +if [ "$UPLOAD_MESSAGE" != "Report uploaded successfully." ] || [ "$HTTP_STATUS" -ne 200 ]; then + echo -e "Upload failed. Response: $UPLOAD_RESPONSE$" >&2 + exit 1 +fi +echo -e "Acknowledgement uploaded successfully to NGINX Instance Manager." -echo "Report acknowledgement successfully uploaded to NGINX Instance Manager $ip_address." From a4f9eb261c0c5a4d81929b85a41b7634de4e37ce Mon Sep 17 00:00:00 2001 From: Travis Martin <33876974+travisamartin@users.noreply.github.com> Date: Mon, 16 Jun 2025 11:28:50 -0700 Subject: [PATCH 021/117] edits to NIM 2.20 docs and install script (#686) * edits to NIM 2.20 docs and install script * updates to install script * edits * edits to tech spec guide * fix: Update install-nim-bundle.sh --------- Co-authored-by: Chetan Jain <46200286+Chetan-99@users.noreply.github.com> --- .../disable-metrics-collection.md | 2 +- .../deploy/kubernetes/deploy-using-helm.md | 2 +- .../vm-bare-metal/install-nim-manual.md | 8 +++---- .../add-license-disconnected-deployment.md | 8 +++---- content/nim/fundamentals/tech-specs.md | 23 ++++++++++++++----- .../configure-clickhouse.md | 20 +++++++++++----- static/scripts/install-nim-bundle.sh | 23 ++++++++++--------- 7 files changed, 53 insertions(+), 33 deletions(-) mode change 100755 => 100644 static/scripts/install-nim-bundle.sh diff --git a/content/includes/nim/installation/optional-steps/disable-metrics-collection.md b/content/includes/nim/installation/optional-steps/disable-metrics-collection.md index 9ccd93654..3204e6299 100644 --- a/content/includes/nim/installation/optional-steps/disable-metrics-collection.md +++ b/content/includes/nim/installation/optional-steps/disable-metrics-collection.md @@ -6,6 +6,6 @@ files: - content/nim/disconnected/offline-install-guide.md --- -If you’re not collecting metrics—because you didn’t install ClickHouse or don’t plan to use it—you must disable metrics collection in the `/etc/nms/nms.conf` file. This setup requires NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}. +If you’re not collecting metrics — because you didn’t install ClickHouse or don’t plan to use it — you must disable metrics collection in the `/etc/nms/nms.conf` and `/etc/nms/nms-sm-conf.yaml` files. This setup requires NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}. For instructions, see [Disable metrics collection]({{< ref "nim/system-configuration/configure-clickhouse.md#disable-metrics-collection" >}}). \ No newline at end of file diff --git a/content/nim/deploy/kubernetes/deploy-using-helm.md b/content/nim/deploy/kubernetes/deploy-using-helm.md index 3b36c5a0e..bed9203a9 100644 --- a/content/nim/deploy/kubernetes/deploy-using-helm.md +++ b/content/nim/deploy/kubernetes/deploy-using-helm.md @@ -72,7 +72,7 @@ kubectl create secret docker-registry regcred \ ### OpenShift ```shell -oc new-project nms +oc new-project nms && \ oc create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ --docker-username= \ diff --git a/content/nim/deploy/vm-bare-metal/install-nim-manual.md b/content/nim/deploy/vm-bare-metal/install-nim-manual.md index fa8f6df42..3c8e956c0 100644 --- a/content/nim/deploy/vm-bare-metal/install-nim-manual.md +++ b/content/nim/deploy/vm-bare-metal/install-nim-manual.md @@ -58,7 +58,7 @@ Follow these steps to download the certificate and private key for NGINX Instanc Install NGINX Open Source or NGINX Plus on the host where you'll install NGINX Instance Manager. NGINX Instance Manager uses NGINX as a front-end proxy and for managing user access. -- [Installing NGINX and NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/) +- [Installing NGINX and NGINX Plus]({{< ref "/nginx/admin-guide/installing-nginx/installing-nginx-plus.md" >}})
@@ -88,9 +88,9 @@ Make sure to review the [Technical Specifications]({{< ref "/nim/fundamentals/te NGINX Instance Manager uses ClickHouse to store metrics, events, alerts, and configuration data. -In 2.20.0, we introduced Lightweight mode, which can skip the ClickHouse installation entirely. It’s ideal if you don’t need monitoring data or want a simpler setup. This reduces system requirements and avoids the work of managing a metrics database. You can add ClickHouse later if your needs change. +Starting in version 2.20.0, you can run NGINX Instance Manager in Lightweight mode, which skips the ClickHouse installation entirely. This setup works well if you don’t need monitoring data or want to reduce system requirements. It also avoids the effort of managing a metrics database. You can add ClickHouse later if your needs change. -If you don’t need to store metrics, you can skip installing ClickHouse. But you must use NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}, and you must disable metrics collection in the `/etc/nms/nms.conf` file. +If you don’t need to store metrics, you can skip installing ClickHouse. But you must use NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}, and you must disable metrics collection in the `/etc/nms/nms.conf` and `/etc/nms-sm.conf.yaml` files. For instructions, see [Disable metrics collection]({{< ref "nim/system-configuration/configure-clickhouse.md#disable-metrics-collection" >}}). @@ -203,7 +203,7 @@ To install NGINX Instance Manager, you need to add the official repository to pu 1. To upgrade to the latest version of the Instance Manager, run the following command: ```bash - sudo apt-get update + sudo apt-get update && \ sudo apt-get install -y --only-upgrade nms-instance-manager ``` diff --git a/content/nim/disconnected/add-license-disconnected-deployment.md b/content/nim/disconnected/add-license-disconnected-deployment.md index e4a272a6a..1039852aa 100644 --- a/content/nim/disconnected/add-license-disconnected-deployment.md +++ b/content/nim/disconnected/add-license-disconnected-deployment.md @@ -18,7 +18,7 @@ type: This guide shows you how to add a license to NGINX Instance Manager in a disconnected (offline) environment. In this setup, systems don’t have internet access. You’ll download and apply your subscription’s JSON Web Token (JWT) license, then verify your entitlements with F5. -{{< call-out "tip" "Using the REST API" "" >}}{{< include "nim/how-to-access-nim-api.md" >}}{{}} +{{< call-out "tip" "Using the REST API" "" >}}{{< include "nim/how-to-access-nim-api.md" >}}{{}} ## Before you begin @@ -39,7 +39,7 @@ To configure NGINX Instance Manager for a disconnected environment, you need to ## Add license and submit initial usage report {#add-license-submit-initial-usage-report} -{{}} +{{< tabs name="submit-usage-report" >}} {{%tab name="Bash script (recommended)"%}} @@ -86,7 +86,7 @@ To license NGINX Instance Manager, complete each of the following steps in order Run these `curl` commands on a system that can access NGINX Instance Manager and connect to `https://product.apis.f5.com/` on port `443`. Replace each placeholder with your specific values. -{{}}The `-k` flag skips SSL certificate validation. Use this only if your NGINX Instance Manager is using a self-signed certificate or if the certificate is not trusted by your system.{{}} +{{< important >}}The `-k` flag skips SSL certificate validation. Use this only if your NGINX Instance Manager is using a self-signed certificate or if the certificate is not trusted by your system.{{}} 1. **Add the license to NGINX Instance Manager**: @@ -117,7 +117,7 @@ Run these `curl` commands on a system that can access NGINX Instance Manager and --header "referer: https:///ui/settings/license" ``` -1. **Update the license configuration on NGINX Instance Manager**: +1. **Update the license configuration on NGINX Instance Manager (not required in 2.20 or later)**: This step ensures that the license configuration is fully applied. diff --git a/content/nim/fundamentals/tech-specs.md b/content/nim/fundamentals/tech-specs.md index d5035bac0..685794aec 100644 --- a/content/nim/fundamentals/tech-specs.md +++ b/content/nim/fundamentals/tech-specs.md @@ -38,13 +38,13 @@ This section outlines the recommendations for NGINX Instance Manager deployments We recommend using SSDs to enhance storage performance. -{{}} +{{< bootstrap-table "table table-striped table-bordered" >}} | Number of Data Plane Instances | CPU | Memory | Network | Storage | |--------------------------------|--------|----------|-----------|---------| | 10 | 2 vCPU | 4 GB RAM | 1 GbE NIC | 100 GB | | 100 | 2 vCPU | 4 GB RAM | 1 GbE NIC | 1 TB | | 1000 | 4 vCPU | 8 GB RAM | 1 GbE NIC | 3 TB | -{{}} +{{}} These values represent the minimum resources needed for deployments that fall under standard configurations. @@ -52,22 +52,33 @@ These values represent the minimum resources needed for deployments that fall un For environments requiring more resources, **large configurations** are suitable. These configurations can support up to **300 upstream servers** and are designed for enterprise environments or applications handling high traffic and complex configurations, without NGINX App Protect. -{{}} +{{< bootstrap-table "table table-striped table-bordered" >}} | Number of Data Plane Instances | CPU | Memory | Network | Storage | |--------------------------------|--------|----------|-----------|---------| | 50 | 4 vCPU | 8 GB RAM | 1 GbE NIC | 1 TB | | 250 | 4 vCPU | 8 GB RAM | 1 GbE NIC | 2 TB | -{{}} +{{}} ### NGINX configuration deployments with NGINX App Protect {#system-sizing-app-protect} If using NGINX App Protect features in NGINX Instance Manager, this requires additional CPU and Memory for policy compilation and security monitoring features. At a minimum, 8gb Memory and 4 CPUs are required for a standard NGINX App Protect use case (under 20 NGINX Plus instances). The requirements are heavily dependent on the number of policies being managed, the frequency of updates and the number of events being that occur in the security monitoring feature. +### Lightweight mode {#lightweight-mode} + +(New in 2.20.0) You can run NGINX Instance Manager without installing ClickHouse. This setup is useful if you don’t need monitoring data or prefer a simpler deployment. It reduces system requirements and removes the need to manage a metrics database. You can add ClickHouse later if your needs change. For instructions, see [Disable metrics collection]({{< ref "nim/system-configuration/configure-clickhouse.md#disable-metrics-collection" >}}). + +In Lightweight mode, we tested NGINX Instance Manager with ten managed NGINX instances and configuration publishing. It ran with as little as 1 CPU core and 1 GB of memory (without App Protect). When App Protect was enabled, we needed 2 CPU cores and 4 GB of memory to compile policies. + +These figures are guidelines only. They reflect the minimum tested configuration and may cause performance issues depending on your setup. For better performance, consider allocating more system resources. + + ### License and usage reporting only {#reporting-sizing} -This section assumes you've configured NGINX Instance Manager to manage your NGINX instances for licensing and usage reporting only. NGINX commercial license and usage reporting is done in an “unmanaged” way, where NGINX sends a request periodically to NGINX Instance Manager solely for counting purposes. For more information, see how you would [Prepare your environment for reporting]({{< ref "/solutions/about-subscription-licenses.md#set-up-environment" >}}). +This section applies when you’ve set up NGINX Instance Manager to handle licensing and usage reporting only. In this setup, NGINX instances report license and usage data in an "unmanaged" way. Each instance sends periodic updates to NGINX Instance Manager for counting purposes only. + +For details on how to configure this setup, see [Prepare your environment for reporting]({{< ref "/solutions/about-subscription-licenses.md#set-up-environment" >}}). -Therefore, the requirements for NGINX Instance Manager when used solely for licensing and usage reporting are minimal. +When used only for licensing and usage reporting, NGINX Instance Manager has minimal system requirements. We recommend using [Lightweight mode](#lightweight-mode) in this case to avoid the ClickHouse dependency, especially if you don’t plan to use other features. {{}} | Number of Data Plane Instances | CPU | Memory | Network | Storage | diff --git a/content/nim/system-configuration/configure-clickhouse.md b/content/nim/system-configuration/configure-clickhouse.md index 3f4a7f1be..2e2990d21 100644 --- a/content/nim/system-configuration/configure-clickhouse.md +++ b/content/nim/system-configuration/configure-clickhouse.md @@ -13,7 +13,7 @@ type: ## Overview NGINX Instance Manager uses ClickHouse to store metrics, events, alerts, and configuration data. -If your setup differs from the default configuration—for example, if you use a custom address, enable TLS, set a password, or turn off metrics—you need to update the `/etc/nms/nms.conf` file. +If your setup differs from the default configuration — for example, if you use a custom address, enable TLS, set a password, or turn off metrics — you need to update the `/etc/nms/nms.conf` file. This guide explains how to update those settings so that NGINX Instance Manager can connect to ClickHouse correctly. @@ -38,19 +38,27 @@ Unless otherwise specified in the `/etc/nms/nms.conf` file, NGINX Instance Manag ## Disable metrics collection -As of version 2.20, NGINX Instance Manager can run without ClickHouse. This lightweight mode reduces system requirements and simplifies installation for users who do not need metrics. To use this setup, you must run NGINX Agent version {{< lightweight-nim-nginx-agent-version >}}. +Starting in version 2.20, NGINX Instance Manager can run without ClickHouse. This lightweight mode reduces system requirements and simplifies installation for users who don't need metrics. To use this setup, you must run NGINX Agent version `{{< lightweight-nim-nginx-agent-version >}}`. To disable metrics collection after installing NGINX Instance Manager: -1. Open the configuration file at `/etc/nms/nms.conf`. - +1. Open the config file at `/etc/nms/nms.conf`. + 2. In the `[clickhouse]` section, set the following value: ```yaml - enable = false + clickhouse: + enable = false ``` -3. Restart the NGINX Instance Manager service: +3. Open the `/etc/nms/nms-sm-conf.yaml` file and set: + + ```yaml + clickhouse: + enable = false + ``` + +4. Restart the NGINX Instance Manager service: ```shell sudo systemctl restart nms diff --git a/static/scripts/install-nim-bundle.sh b/static/scripts/install-nim-bundle.sh old mode 100755 new mode 100644 index 7dc020eac..fb60d615a --- a/static/scripts/install-nim-bundle.sh +++ b/static/scripts/install-nim-bundle.sh @@ -566,14 +566,9 @@ This action deletes all files in the following directories: /etc/nms , /etc/ngin getLatestPkgVersionFromRepo(){ repoUrl=$1 version=$2 - pkg_extension=$3 - if [[ "${pkg_extension}" == "rpm" ]]; then - response=$(curl --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} -sL "${repoUrl}" | awk -F '"' '/href=/ {print $2}' | grep -E "$version"| sort -t'-' -k4,4V | tac) - readarray -t versions < <(printf "%s" "${response}") - else - response=$(curl --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} -sL "${repoUrl}" | awk -F '"' '/href=/ {print $2}' | grep -E "$version"| sort -t'_' -k2,2V | tac) - readarray -t versions < <(printf "%s" "${response}") - fi + sort_fields=$3 + response=$(curl --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} -sL "${repoUrl}" | awk -F '"' '/href=/ {print $2}' | grep -E "$version"| eval sort "$sort_fields" | tac) + readarray -t versions < <(printf "%s" "${response}") if [ "${#versions[@]}" -eq 0 ]; then printf "Package %s not found. See available versions:" "${versions[@]}" exit 1; @@ -602,22 +597,26 @@ package_nim_offline(){ cd "${TEMP_DIR}/${TARGET_DISTRIBUTION}" || echo "directory ${TEMP_DIR} does not exits" if [[ "${USE_NGINX_PLUS}" == "true" ]]; then NGINX_PLUS_PACKAGE="^nginx-plus_[0-9]+-([0-9]+)~${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}_${OS_ARCH}\.${PKG_EXTENSION}$" + SORT_FIELDS="-t'_' -k2,2V" if [[ "${PKG_EXTENSION}" == "rpm" ]]; then NGINX_PLUS_PACKAGE="^nginx-plus-[0-9]+-([0-9]+)${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}\.${PKG_EXTENSION}$" + SORT_FIELDS="-t'-' -k3,3V" fi echo "regex for looking latest version : ${NGINX_PLUS_PACKAGE}" - NGINX_PLUS_VERSION=$(getLatestPkgVersionFromRepo "${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}" "${NGINX_PLUS_PACKAGE}" "${PKG_EXTENSION}") + NGINX_PLUS_VERSION=$(getLatestPkgVersionFromRepo "${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}" "${NGINX_PLUS_PACKAGE}" "${SORT_FIELDS}") echo "latest version for nginx_plus is ${NGINX_PLUS_VERSION}" echo "Downloading ${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_PLUS_VERSION}...." curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_PLUS_VERSION}" check_last_command_status "curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} \"${NGINX_PLUS_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_PLUS_VERSION}\"" $? else NGINX_OSS_PACKAGE="^nginx_[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)~${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}_${OS_ARCH}\.${PKG_EXTENSION}$" + SORT_FIELDS="-t'_' -k2,2V" if [[ "${PKG_EXTENSION}" == "rpm" ]]; then NGINX_OSS_PACKAGE="^nginx-[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}\.${PKG_EXTENSION}$" + SORT_FIELDS="-t'-' -k2,2V" fi echo "fetching latest version using ${NGINX_OSS_PACKAGE}" - NGINX_OSS_VERSION=$(getLatestPkgVersionFromRepo "${NGINX_REPO[${TARGET_DISTRIBUTION}]}" "${NGINX_OSS_PACKAGE}" "${PKG_EXTENSION}") + NGINX_OSS_VERSION=$(getLatestPkgVersionFromRepo "${NGINX_REPO[${TARGET_DISTRIBUTION}]}" "${NGINX_OSS_PACKAGE}" "${SORT_FIELDS}") echo "latest version for nginx is ${NGINX_OSS_VERSION}" echo "Downloading ${NGINX_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_OSS_VERSION}...." curl -sfLO "${NGINX_REPO[${TARGET_DISTRIBUTION}]}/${NGINX_OSS_VERSION}" @@ -649,10 +648,12 @@ package_nim_offline(){ check_last_command_status "curl -sfLO \"${CLICKHOUSE_CLIENT_PATH}\"" $? fi NIM_PACKAGE_PATH="^nms-instance-manager_[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)~${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}_${OS_ARCH}\.${PKG_EXTENSION}$" + SORT_FIELDS="-t'_' -k2,2V" if [[ "${PKG_EXTENSION}" == "rpm" ]]; then NIM_PACKAGE_PATH="^nms-instance-manager-[0-9]+\.[0-9]+\.[0-9]+-([0-9]+)${OS_DISTRO_MAP[${TARGET_DISTRIBUTION}]}\.${PKG_EXTENSION}$" + SORT_FIELDS="-t'-' -k4,4V" fi - NIM_PACKAGE_VERSION=$(getLatestPkgVersionFromRepo "${NIM_REPO[${TARGET_DISTRIBUTION}]}" "${NIM_PACKAGE_PATH}" "${PKG_EXTENSION}") + NIM_PACKAGE_VERSION=$(getLatestPkgVersionFromRepo "${NIM_REPO[${TARGET_DISTRIBUTION}]}" "${NIM_PACKAGE_PATH}" "${SORT_FIELDS}") echo "Latest version for nginx instance manager is ${NIM_PACKAGE_VERSION}...." curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} "${NIM_REPO[${TARGET_DISTRIBUTION}]}/${NIM_PACKAGE_VERSION}" check_last_command_status "curl -sfLO --cert ${NGINX_CERT_PATH} --key ${NGINX_CERT_KEY_PATH} \"${NIM_REPO[${TARGET_DISTRIBUTION}]}/${NIM_PACKAGE_VERSION}\"" $? From a1263f99fa4ad093918d1eee37790a74f31eff56 Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Mon, 16 Jun 2025 13:43:01 -0600 Subject: [PATCH 022/117] Update NGF example certs (#683) Problem: The BackendTLSPolicy guide was using expired certs in its example configuration. Solution: Update the sample certs to expire in 10 years. --- .../ngf/traffic-security/secure-backend.md | 53 +++++++++++-------- 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/content/ngf/traffic-security/secure-backend.md b/content/ngf/traffic-security/secure-backend.md index d532bdbc3..cde85cd2a 100644 --- a/content/ngf/traffic-security/secure-backend.md +++ b/content/ngf/traffic-security/secure-backend.md @@ -105,8 +105,8 @@ metadata: name: app-tls-secret type: Opaque data: - tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQwRENDQXJpZ0F3SUJBZ0lVVDQwYTFYd3doUHVBdDJNMkdZZUovYXluZlFBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1JqRWZNQjBHQTFVRUF3d1djMlZqZFhKbExXRndjQzVsZUdGdGNHeGxMbU52YlRFTE1Ba0dBMVVFQmhNQwpWVk14RmpBVUJnTlZCQWNNRFZOaGJpQkdjbUZ1YzJselkyOHdIaGNOTWpRd01URTRNVGd3TVRBeFdoY05NalV3Ck1URTNNVGd3TVRBeFdqQi9NUXN3Q1FZRFZRUUdFd0pWVXpFVE1CRUdBMVVFQ0F3S1EyRnNhV1p2Y201cFlURVcKTUJRR0ExVUVCd3dOVTJGdUlFWnlZVzV6YVhOamJ6RU9NQXdHQTFVRUNnd0ZUa2RKVGxneEVqQVFCZ05WQkFzTQpDVTVIU1U1WUlFUmxkakVmTUIwR0ExVUVBd3dXYzJWamRYSmxMV0Z3Y0M1bGVHRnRjR3hsTG1OdmJUQ0NBU0l3CkRRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFMeUx0eURNbTZ4M0ZEUFJsOGZ0azNweCtrRWQKYTVpTGZOQ3lDbUVjYktBQVBDNEhZckl5b1B5QXpSTlJCMWErekE0UTlrbzJZRG5vR0dkeFJaMEdydldKZUV2Mgo3MWlHNGxhbHRVTS9WOWNvSktQY0UyTEI0R3R6cFA3ckdIWXNvRDlOUXFpV3YwZ0lOdE42MjdrWGg4UW41V1hYCk92Y2FkS2h0bjJER3RvU0VzT3dpNzR5NEt3SmFkWnlwLzJaM0hPakRTNjVIVmxydmUxUXpBMVRzTEp6S3cva3gKbHBSR0lWK0lhUjZXbXZsaVFVdDJxWFg0L3hGeVVEM2Vic05TeXpHUk5mQ0NOTWxlWlV3MTR3ZUdhOEVnc2tDcQprOGdYSmpFZXQxMlR4OGxkY3BpVWlxYVpkOStYZjJmUS8yL2Y5c1IzM3Q4K0VVUWpoZ2ZIbHlsLzV1RUNBd0VBCkFhTjlNSHN3SHdZRFZSMGpCQmd3Rm9BVTRUT096c1d0Q3ZWdGJlWXFSU0FqN2tXajFkb3dDUVlEVlIwVEJBSXcKQURBTEJnTlZIUThFQkFNQ0JQQXdJUVlEVlIwUkJCb3dHSUlXYzJWamRYSmxMV0Z3Y0M1bGVHRnRjR3hsTG1OdgpiVEFkQmdOVkhRNEVGZ1FVZmtWREFFWmIwcjRTZ2swck10a0FvQ2c2RjRnd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBQWFiQit6RzVSODl6WitBT2RsRy9wWE9nYjF6VkJsQ0dMSkhyYTl1cTMvcXRPR1VacDlnd2dZSWJ4VnkKUkVLbWVRa05pV0haSDNCSlNTZ3czbE9abGNxcW5xbUJ2OFAxTUxDZ3JqbDJSN1d2NVhkb2RlQkJxc0lvZkNxVgp3ZG51THJUU3RTbmd2MGhDcldBNlBmTnlQeXMzSGJva1k3RExNREhuNmhBQWcwMUNDT0pWWGpNZjFqLzNIMFNCClBQSWxtek5aRUpEd0JMR2hyb1V3aUY3NkNUV1Fudi8yc1pvWHMwUlFiRTY3TmNraXc2Z0svaWRwVTVzMmlkOEQKVExjVjNxenVFaE1ZeUlua0ZWNEJLZlFkTWxDQnE1QWdyU1Jqb2FoaCszbFRwYVpUalJGUGFVd3VZYXVsQXRzNgpra1ROaGltWWQ3Ym1aVk5MK2I0MzhmN1RMaGc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0= - tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRQzhpN2Nnekp1c2R4UXoKMFpmSDdaTjZjZnBCSFd1WWkzelFzZ3BoSEd5Z0FEd3VCMkt5TXFEOGdNMFRVUWRXdnN3T0VQWktObUE1NkJobgpjVVdkQnE3MWlYaEw5dTlZaHVKV3BiVkRQMWZYS0NTajNCTml3ZUJyYzZUKzZ4aDJMS0EvVFVLb2xyOUlDRGJUCmV0dTVGNGZFSitWbDF6cjNHblNvYlo5Z3hyYUVoTERzSXUrTXVDc0NXbldjcWY5bWR4em93MHV1UjFaYTczdFUKTXdOVTdDeWN5c1A1TVphVVJpRmZpR2tlbHByNVlrRkxkcWwxK1A4UmNsQTkzbTdEVXNzeGtUWHdnalRKWG1WTQpOZU1IaG12QklMSkFxcFBJRnlZeEhyZGRrOGZKWFhLWWxJcW1tWGZmbDM5bjBQOXYzL2JFZDk3ZlBoRkVJNFlICng1Y3BmK2JoQWdNQkFBRUNnZ0VBUXJucGJleXJmVTVKTW91Ty9UenBrQkJ0UWdVZzhvUVBBS2E1d0tONEYrbnQKWWxiUHlZUGNjSEErNDRLdUo3ZHZiTno0NU11NG8xV3Q2Vkh2a29KdWdjd01iRW53YTdLVXdKaDFmVjZaL2pXaApQZkpoVS9hTUwwcm1qaWJ5YWNRaVZEVEtEZk1Ici96a05sVEpGUWlzVGpIV1lBUGJSTjh5Z1BjR3pBK1hRVzg5CmxsOFdoeWwrZndjRVAzNTM4TUdKYUpHVmV4Q2d5cVRyKzZwQ29yRUpFL1pSNytiMTNyejRsbmxpZXVYa2pCVkcKSnYvUVI0RVhTSDhuRit3K2FvWTFQaGd2QnFJTnFQZjJMT1V0MzNiN3FDTkFmSVBRdFZFejJsN3NqbmJlcElTTwpvTkhNUFY0N21XTzY2dzhFMzJVMjNjVEtUbytJcWovM0d1eGhweXlYaHdLQmdRRDVNRDhmY3ZyM0xVZHkvZ3I0Ci82MVBqUXNSaWRYdjN0Mk1MVEk4UkduNzJWcGxvVVExNDZHV2xGTGVVVDY1L1ZVMjNsZFUrUWt2eFNMK3U1bW4KRUJIdXUyVmtBUWVYcUJXWkpPTmFSZG9Ia1YzK2Fyc0U4Qld0SWVtZHJ0MWN6bHFjc1VzMWdGdG1COGI3RHB3UwpHKzRoZDlzZG0weDBNS2hoOFVGOUQrZytUd0tCZ1FEQnN4dUpoZ1hnck14S0dVMHJ5MW9WWTJtMGpDUEpJcTkzCmNZRUZGY3lYZ0U4OWlDVmlob3dVVE8yMXpTZ3o0SVVKNUxoc2M5N3VIVER1VXdwcVI5NFBsdjlyaXJvakowM1UKT3FyWHgwbWdNN2xibVM1L1RwS0czZG1QblZ0WEZMektISFgyWDVnUW56emYvdXVKK3NtbDVLQW5WN0VZc1oxcgpkVXJvRm8zcnp3S0JnUUNZdjM5aUdzeEdLaVpMRWZqTjY0UmthRFBwdTFFOTZhSnE0K1dRVmV1VnF3V2ptTGhFClJGWHdCTm5MVjRnWTRIYVUzTFF4N1RvNVl5RnhmclBRV2FSMGI4RFdEVitIRWt5ekJJNnM3bmFZL3YzY0Q3YTIKYnlrS2FPaFlkVEZTUzFmMkJ5UHdGczl2K3NKNWNOb3dxNWhNUWJrNks5RXd4QWJqaXN5M0NjSTJOd0tCZ1FDbwo2c2pZNVVlNjV2WkFxRS9rSVRJdDlNUDU3enhGNnptWnNDSVRqUzhkNzRjcTRjKzRYQjFNbHNtMkFYTk55ajQ2Cm9uc3lHTm9RVE9TZThVdmo0MGlEeitwdW5rdzAyOUhEZ21YNlJwQ3VaRzBBdEZVWU1DMFg3K0FLbmU5SndZdmgKdFhBcHFyT3h5eXdMS3dPOUVEZEp0RmIxK0VNNGhhd0NTZ2RJM21KbGdRS0JnRzIxeEJNRXRzMFBVN3lDYTZ0YwpadDc1NUV4aEdkR3F5MmtHYmtmdzBEaHBQQVVUZmdncVF3NVBYdGVIS1ZBSDlKaG5kVnBBZFFxNmZ1MER1MDNKCkl0cGpxNWluZXVoR0x0alpMR1Nhd0dwY0FUU3h4Z3dCM0l0Z29LKzBCRFhteWxId0lEcUc5Z2crRU5KK0VhL0MKeTFOMmV0ZG1sQ01hNjM4cVJlNFlTWk55Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0= + tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVpekNDQW5PZ0F3SUJBZ0lVQ0g2NWhwVDNkSlI0SVdPTXdaL2lCb3M5Q0Rrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNWVk14Q3pBSkJnTlZCQWdNQWxkQk1SQXdEZ1lEVlFRSERBZFRaV0YwZEd4bApNUk13RVFZRFZRUUtEQXBGZUdGdGNHeGxJRU5CTVJnd0ZnWURWUVFEREE5RmVHRnRjR3hsSUZKdmIzUWdRMEV3CkhoY05NalV3TmpFMk1UVXpOelUzV2hjTk16VXdOakUwTVRVek56VTNXakJnTVFzd0NRWURWUVFHRXdKVlV6RUwKTUFrR0ExVUVDQXdDVjBFeEVEQU9CZ05WQkFjTUIxTmxZWFIwYkdVeEVUQVBCZ05WQkFvTUNFWTFJRTVIU1U1WQpNUjh3SFFZRFZRUUREQlp6WldOMWNtVXRZWEJ3TG1WNFlXMXdiR1V1WTI5dE1JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTM3M0JyRG5kU1Z1UG0xSm5VNCtFVFZqSTFVSTJydVhIY0srUUJ5am4KNk5jMklEZ0NPNzdMOVdTajQzcmRicnlKNm1LMEFrVTBjekhlM1ZVUndaaG95TlM2QnEwaHJIOTFOWHlVcGNaMQpzaTJrbEdFbnVRYSs4dHgrVUwrUGYzck4yTlZ2TytTdnZSL2NxUWpYNnEzeURVMXJLWTZEQUlWaWxBNytDdHhVClE0KzI2MXluSlNTaWZ6YnB0R1ExTmZuV1Y5eHNoOWNyVklqbk9MNlhiek90eHNoYnBxU04xVENoQTR0KzVSb0kKOFo0aG0wWmpMNll3bVYycDB0R2ZFbVV0WGszelRjVVRYTitSODE4MVE0c0JPbEt5YjJpMG1seE9GYUpqc3hQVAprYjBsQmU2WS95TVBrallaT0o5c3YwR0k1YmszaHUwb0lGRDBOb0N5bFNmUmx3SURBUUFCbzBJd1FEQWRCZ05WCkhRNEVGZ1FVS29FbkhWU1dqSFRIeEVoK2xFazFQT2hUYWU4d0h3WURWUjBqQkJnd0ZvQVU0NS9URHQwUVE5dTQKTDN5OVJUOW50Z0VhQnE0d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dJQkFBN1ZNKzZ6bEtKZGlZVElWTjN4ZzJrOApkUG1zSm8vR1UwRW1WTFRzaFJtQWppdnI0bXBkYXRpL0p2UEtIcGtSTTNLcDNqQURmVFRqRnhuZ2ZJajNSN1J5CkpJamZVMGdrdHBKOG9UWVJlMlNsdXdVQ2VrYWlLa1BmWDRLdHcwUWVRSU4vSVN6YTdOS2krWklXOVJhRE1kbzIKNHZYUzIySDRIcjN6dmtFYVhvdHB6YXBlTzJBU1BDS3hoOWRkZWlyVzNydGZpZkRhSFF2UUpZSGpGYWpYcGZhaQpMazRHSWNrdHY2WGNqeFA5V1ArQ1RUWWFxZTVIMkg1dXlheDNyckRWRm43Y3Yzb29mcDJrTVhGUmRjTG9ERGdOCllUT3czTjhRLzl1bjNPRHBZR1l6b3V5RmJieXJ1MUlhcTZKeTRQNURnLzFSaEl3MnpuaWlCdzR0aXBzV2tqSTgKSEtRcWxKZE1KYlZUMHkraUlPMThxdWVsL1hDSC9hS1FEclpMd2Z0WVEvczVTWUtXUG5UZzhIbjJpeUlhUmZ0dQorbkgyMCszZkJuQnorK0hOd1duY2g0WmJDM0k3UklpcXFpdm1ML09YMWkvbng2UG5BVTlQMFlsOGhsMFFINndGCmtOd0NuTmE5TEp6eXBCandjY0IzU1ZaNUIxanhRTTJTcTYrSnB2cFZJSDQzL29paWJ0anhLMDhQenVVZ2luTUgKVC9aMXQ0NDBhRmk2V2VIZHRHM2RkVkM0SGxhS0JqQXVkZm44MHA1M3RPbXN6dkFRUUFidlhpd3pzZVZxYW5mdgpnS1BqWTd5aE9oZzZTdUJ2OFRTVTkvSVBDUklBTnRiMXkvOGxnVVlkVDJ0dGVmTVlnMnhFempndGIxQWEzMzBWCk1ONXV5ZFpCTU92aTRReHJuVzVICi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0= + tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRRGZ2Y0dzT2QxSlc0K2IKVW1kVGo0Uk5XTWpWUWphdTVjZHdyNUFIS09mbzF6WWdPQUk3dnN2MVpLUGpldDF1dklucVlyUUNSVFJ6TWQ3ZApWUkhCbUdqSTFMb0dyU0dzZjNVMWZKU2x4bld5TGFTVVlTZTVCcjd5M0g1UXY0OS9lczNZMVc4NzVLKzlIOXlwCkNOZnFyZklOVFdzcGpvTUFoV0tVRHY0SzNGUkRqN2JyWEtjbEpLSi9OdW0wWkRVMStkWlgzR3lIMXl0VWlPYzQKdnBkdk02M0d5RnVtcEkzVk1LRURpMzdsR2dqeG5pR2JSbU12cGpDWlhhblMwWjhTWlMxZVRmTk54Uk5jMzVIegpYelZEaXdFNlVySnZhTFNhWEU0Vm9tT3pFOU9SdlNVRjdwai9JdytTTmhrNG4yeS9RWWpsdVRlRzdTZ2dVUFEyCmdMS1ZKOUdYQWdNQkFBRUNnZ0VBYXhSQXpYRkFFNnlyVlBXaUY5NjJ2ZUhBOURkbFBsMGdEekVtcUJhT3J1UFkKdHFDM2lPcHVhSG9LNllMUzJQMklyOUVmUDNycGVEd2s0aDZsaWRhc1IzbHZzbVJIbW12QnA2Q0E3N25FZUVyWgoybDJKQ2tkTk9hUUhIQlFoMUN2c3VscWppckdPM2QrUzFwOHgzdEh5NXlUbkpaTmI1UEx4VTlTOUJtdWVORnA5ClBiSWwwV0Ewb1h6VWlPa0VESFpJaE9LeEQ1SExNQkpEUGIzYy9ZRE9HSlFtTGRtRldYS1VsL3NHNHJJdzFGNWYKZzhvYzRyekJpRlhVY0EvRlhoWllNcW9lMlp6MFhXWWVCYWhINFB2Sm4zdFVhd0xFQ0NTbW1nYkVzT1ltNHdvZQptUHZmT2k3TDR3VmhGNjNBNkxnL3hpR3VFUkd0cmJlMS8xcVFlTXFGU1FLQmdRRDRmT2ZPaUVpUkViODlJeDdKCjJra3ZuS1ZERXl1YjExc3NzZTNjU054R1haVFBkK0JjTGc5YVBTdXFrWC9NbmJsQ2JBOXZhSzZOL3U2YkxsSkEKQXNnVFM0Rkc0K2J2d2hBTzQybUVKQTJIRG0zNDVGRmRFQ2JYblhjV1VxVzJWblBqNkpDdDB2d1R1ZDNMVGdxbgpFM3RmZVZuM0J4bkpsVU4vOHdXUlB1M2NoUUtCZ1FEbWdWVDJHUDZvanBudC92SjRPOTBrYnEzVVJHYklGK0tOCjYrUnJmb0kxWkN2dElXaGV4OVhtZi8wWVlBaFQvaXJTSDd3RTk3QmpIbWJTMlZId051cDNmc1dDNFNYMDFYTkwKYnFjQmxOeW1JUzFiNGN0U01jMFFyeWM5YWlMUVM1d3Fvc1UwNGZubGdkM2o2cHRjdW8vd0Z6Q3JoV0xUZDk1bgpHeUc5NTZYdWF3S0JnUURySDJWSUsvUmVNR2pBTk1jaFFJY1hvaVZOL29tNUFHR3BQUU5RK1RCVTlKK21ZRXZQCmJWWGhrUmdNWVhpSDZJWXZyNGc3WnRZa1RpRUFmU2dlb1lNbm5yNUlrY1VuQUgycFdNMnkxMXBsZk9YYUtGQkUKdXMvR0haMWRaZjZmTmRhYXhLaUJrYTRzRENjdUJENVlNVHIvOEJlTWd3K0hpdEUvOUhoRUkwTjI4UUtCZ1FDagoxazJEVnFTN1BoQ2ZIMVZNckpBMHN3NlBEOGRXZGRPc09IejFBc2llRm9NNlcwS0tDOVEzcjhVL3JCSi9VT3N5Cnl5ZWpDRUt4VVF5WTFhcnQ2THFqRU5KbWdvMnVCb0dhbmgzS2UvcVJnb2R4Qlg2MC8zellYUWF4R2wyQVhCMjIKR0ZlL2pOZElrQlFkU2NZQUZRTDJEaVdqNUgwbi9jMXd6OUlkM3ljTDNRS0JnQ3NNV1NxU1lobUN3ajdON1U2awpyWndjTGxJdS94UEZsMWFTL2Q2MGV4UXpzWmxzd0xkMzJMcStKdkxVYXNPdWVQaGlCajZZNFNiNlk3bDBLMmNvCkZndXFFbmxBL3JNY2Qxa1h6L0dEbVZvYVVoSk8rZlFNSmk3ZEQzS0tidm80SGxhSG1kSkUvNGN5ek1jQmZVeUUKUUtXUElyVS9WZFU0ckZHV0xqRjJKc1ZPCi0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0= EOF ``` @@ -224,25 +224,36 @@ metadata: data: ca.crt: | -----BEGIN CERTIFICATE----- - MIIDbTCCAlWgAwIBAgIUPA3fFnkLl63GZ7noUjb5NoLhSYkwDQYJKoZIhvcNAQEL - BQAwRjEfMB0GA1UEAwwWc2VjdXJlLWFwcC5leGFtcGxlLmNvbTELMAkGA1UEBhMC - VVMxFjAUBgNVBAcMDVNhbiBGcmFuc2lzY28wHhcNMjQwMTE4MTgwMTAxWhcNMjUw - MTA4MTgwMTAxWjBGMR8wHQYDVQQDDBZzZWN1cmUtYXBwLmV4YW1wbGUuY29tMQsw - CQYDVQQGEwJVUzEWMBQGA1UEBwwNU2FuIEZyYW5zaXNjbzCCASIwDQYJKoZIhvcN - AQEBBQADggEPADCCAQoCggEBAJGgn81BrqzmI4aQmGrg7RgkO5oYwlThQ9X/xVHB - YVFptjRPAZz9g92g5birI/NZ43C6nEbZrJrSCqN3wgvV84jJmBAgpAvW+LhF4caa - nhAnecJCcTbwrd542vCDoDRsNV5ffbpESgC4FxPGkRVbSa0KHQz8qCLqS2+uaB7X - t76iw6y4pQ3klobVp1XtUpzZMGMBqZFnsAdl+PWMmSTvqjixkSlfcUY6Crnk9W6d - Sns5cpzKdUs+2ZkBe6VkBgSs8xbaz8Y2YC1GhRqGlxYLT3WBaIlSCKPuRrGjwE3r - AsW6gSL919H1O1a+MjQuLuQ4lnCbCpNzM9OV1JISMWfwifMCAwEAAaNTMFEwHQYD - VR0OBBYEFOEzjs7FrQr1bW3mKkUgI+5Fo9XaMB8GA1UdIwQYMBaAFOEzjs7FrQr1 - bW3mKkUgI+5Fo9XaMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB - AG/eX4pctINIrHvRyHOusdac5iXSJbQRZgWvP1F2p95qoIDESciAU1Sh1oJv+As5 - IlJOZPJNuZFpDLjc8kzSoEbc1Q5+QyTBlyNNsagWYYwK0CEJ6KJt80vytffmdOIg - z8/a+2Ax829vcn1w1SUi5V6ea/l8K74f2SL/zSSHgtEiz8V0TlvT7J6wurgmnk4t - yQRmsXlDGefuijMNCVf7jWwLx2BODfKoEA1pJkthnNvdizlikmz+9elxhV9bRf3Y - NnubytWPfO1oeHjVGvxVjCouIYine+VlskvwHmMi/dYod6yd7aFYu4CU3g/hjwKo - LY2WNv5j3JhDnEYK9Zj3z7A= + MIIFlzCCA3+gAwIBAgIULQcgBeB9ApX+Wf+FYLMLAVOLwYIwDQYJKoZIhvcNAQEL + BQAwWzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAldBMRAwDgYDVQQHDAdTZWF0dGxl + MRMwEQYDVQQKDApFeGFtcGxlIENBMRgwFgYDVQQDDA9FeGFtcGxlIFJvb3QgQ0Ew + HhcNMjUwNjE2MTUzNjQzWhcNMzUwNjE0MTUzNjQzWjBbMQswCQYDVQQGEwJVUzEL + MAkGA1UECAwCV0ExEDAOBgNVBAcMB1NlYXR0bGUxEzARBgNVBAoMCkV4YW1wbGUg + Q0ExGDAWBgNVBAMMD0V4YW1wbGUgUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQAD + ggIPADCCAgoCggIBAI4fKNzZU/9IEifa6k9L38Ei4lFwFt6wjCZt70Rb5IpJfOkr + PsZf+XaCgIQZh3ZTBIPZoa2uNxVF1BoKEhEqzGdWdH82lVDmbkSaLeW4YnYhM9fe + IuYHqyVrNFsp7M+xJD3XWwwCXDN//H+vx8JLQIOX4tQ9RM4rB7avcu0rmaXJuqJi + N0AU8AsEFpoIiC/vFBucqpG2KzC+FsVPe/Ri+PQTWJ6aLVUvGJ3hXRAQdOMzIGys + +WTugzqk+Sv9pKDB7/EMJg+5IagNP5QWrDmdBwROdRhd8wq+zYJMxVaaSqetNrnY + aqWPdGj+RSB5YuiL8kwDiJOPc5G4t2I/hbol5hpBleL84qijclzJQeFOKdEbRzMa + w2QfyZxZ24TCZGwDzs480x5bKmUoRufdk8X4DSjV4tnKMh9sfHX3w4cokp2IWBqt + B59HMN4nAKELftjfKjI9L0jnEbJR/Xae0qPLxjAuN0HARQNx1/EvvsWGhgu4Sr5S + Ua604wHU9p85a5zWOgOJ471f/sA3q2yEiiHPYqbZ2YXmVMrHvMROO5EVCN7HOjoL + QW4z85fX7QT3OwF6Xckk8jA45tcy0cIlOHsl19XbZUJ5Gc7jAiYbmy2IRYgNiMmb + iSBL0eoC2jan1Y6Of0UcOiAfsMvt5f21dehYkP7Mwe7sGmMk8Lkva1PRSUFfAgMB + AAGjUzBRMB0GA1UdDgQWBBTjn9MO3RBD27gvfL1FP2e2ARoGrjAfBgNVHSMEGDAW + gBTjn9MO3RBD27gvfL1FP2e2ARoGrjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3 + DQEBCwUAA4ICAQBD/b0Vu1vbRYQIeBbaPLCHS5IO8R2G18w+RyUBYpcr50FLI81X + YPTCnEXw5sVd7Y7PNZ6q3crr4u/tGWwDWr9vHL5YWApEnUJzdp9NGv9Z29pgRrQS + Dzr+ggv8/NRMy4xFZ/U5JKy1lHgNFhSp68PrZI6T4xN3zyTCj29qC9hxoxhYpbXS + 1m3PB84T1VqZu2dwNZll+kOIMv+Qon8MWZbuGLkFrcxuDqHB6RRAWyRB7WKKZZXu + r+Uw+9+0htNcASNuEx1yXUaE9Bf9V22fs4ARilP2QpFtMN8BCVQiKp8tj+em3ZX0 + 5KfUavNIZekrHKn/3pc9M+PhX+CEPLNncgywBVYZN89ZujyNVCKVwWD3LmH5NZ8w + JMJ9f3SC8ZqTu7xqX6FVLvmNAm5sS7lE8M0wGzp9cZHHEmstC1//jc+JzhRu8XyU + wGJyANVFlPn7+nnSq7dWbv4OTFsurbWrHQlzwjHaht84DGR4E0QgLUtA+ICGU0Ig + 3/Ma5N7Jugw2mbx8NupgOE3AL6aScbuSt2E1/4QZ2uNPh8soxbANu9WbEXOXzxd5 + LItmVd4ltG/4i2qwLu4NLSNA56iaCDIRD/cfr6WmNLfcMTGRMIJ+AlxnETAIy+pK + s6cpSfBLcDUhStgvr5oNDFeCbmsXRgyJZE9I4mKijG8v+LkS9RgKsS7tRw== -----END CERTIFICATE----- EOF ``` From 232e708ba5feef8b4c147a3b89393dcd8773605e Mon Sep 17 00:00:00 2001 From: Chetan Jain <46200286+Chetan-99@users.noreply.github.com> Date: Tue, 17 Jun 2025 17:03:14 +0530 Subject: [PATCH 023/117] fix: Update deploy-using-helm.md nms => nim (#687) --- content/nim/deploy/kubernetes/deploy-using-helm.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/nim/deploy/kubernetes/deploy-using-helm.md b/content/nim/deploy/kubernetes/deploy-using-helm.md index bed9203a9..cfb7e23e3 100644 --- a/content/nim/deploy/kubernetes/deploy-using-helm.md +++ b/content/nim/deploy/kubernetes/deploy-using-helm.md @@ -61,23 +61,23 @@ Make sure there are no extra characters or spaces when copying the JWT token. Th ### Kubernetes ```shell -kubectl create namespace nms +kubectl create namespace nim kubectl create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ --docker-username= \ --docker-password=none \ - -n nms + -n nim ``` ### OpenShift ```shell -oc new-project nms && \ +oc new-project nim && \ oc create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ --docker-username= \ --docker-password=none \ - -n nms + -n nim ``` {{< call-out "note" "Note" >}} @@ -91,13 +91,13 @@ As a best practice, you can delete the JWT token and clear your shell history af - Kubernetes ```shell - kubectl get secret regcred --output=yaml -n nms + kubectl get secret regcred --output=yaml -n nim ``` - OpenShift ```shell - oc get secret regcred --output=yaml -n nms + oc get secret regcred --output=yaml -n nim ``` You can now use this secret for Helm deployments and point the chart to the private registry. From b515241f508155218b3aae9c4c0acb9078fbee5a Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Tue, 17 Jun 2025 06:34:36 -0700 Subject: [PATCH 024/117] Changed variable name for Slack webhook in notification workflow (#689) chore: Changed variable name for Slack webhook in notification workflow --- .github/workflows/notification.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/notification.yml b/.github/workflows/notification.yml index 076db92ca..810345ea2 100644 --- a/.github/workflows/notification.yml +++ b/.github/workflows/notification.yml @@ -75,4 +75,4 @@ jobs: }] } env: - SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} \ No newline at end of file + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL_DOCS_INCIDENT }} \ No newline at end of file From 259a174b3692b74cad8f3b763e9271a82ed76596 Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Tue, 17 Jun 2025 07:10:12 -0700 Subject: [PATCH 025/117] Moved slack notification direct into job instead of dispatch for security (#690) --- .github/workflows/build-push.yml | 39 ++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/.github/workflows/build-push.yml b/.github/workflows/build-push.yml index e990e6afb..db52d7759 100644 --- a/.github/workflows/build-push.yml +++ b/.github/workflows/build-push.yml @@ -79,16 +79,37 @@ jobs: runs-on: ubuntu-latest permissions: read-all steps: - - name: Trigger 'Slack notification for new theme release' workflow in 'nginx-hugo-theme' repo. - run: | - curl -L \ - -X POST \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${{ secrets.THEME_SLACK_FLOW_PAT }}" \ - -H "X-GitHub-Api-Version: 2022-11-28" \ - "https://api.github.com/repos/${{ secrets.OWNER }}/${{ secrets.REPO }}/dispatches" \ - -d "{\"event_type\": \"trigger-slack-notification\", \"client_payload\": {\"previewURL\": \"${{ env.PREVIEW_URL }}\", \"author\": \"${{ github.event.client_payload.author}}\", \"tag_name\": \"${{ github.event.client_payload.tag_name }}\", \"release_name\": \"${{ github.event.client_payload.release_name }}\"}}" + - name: Send notification + uses: 8398a7/action-slack@1750b5085f3ec60384090fb7c52965ef822e869e # v3.18.0 + with: + status: custom + custom_payload: | + { + username: 'Github', + mention: 'channel', + attachments: [{ + title: `New theme release - ${{ github.event.client_payload.release_name }}`, + color: '#009223', + fields: [ + { + title: 'Tag', + value: ${{ github.event.client_payload.tag_name }}, + short: true + }, + { + title: 'Author', + value: ${{ github.event.client_payload.author }}, + short: true + }, + { + title: 'Preview URL', + value: ${{ env.PREVIEW_URL }}, + short: true + }] + }] + } env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL_FRIENDS_OF_DOCS }} PREVIEW_URL: ${{ needs.call-docs-build-push.outputs.PREVIEW_URL }} From a6d621400e4969fcab3d4e0748fef2f667d66676 Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Tue, 17 Jun 2025 07:19:52 -0700 Subject: [PATCH 026/117] Small fix to add quotes to build-push for slack (#691) --- .github/workflows/build-push.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build-push.yml b/.github/workflows/build-push.yml index db52d7759..1d2bdee5f 100644 --- a/.github/workflows/build-push.yml +++ b/.github/workflows/build-push.yml @@ -93,17 +93,17 @@ jobs: fields: [ { title: 'Tag', - value: ${{ github.event.client_payload.tag_name }}, + value: `${{ github.event.client_payload.tag_name }}`, short: true }, { title: 'Author', - value: ${{ github.event.client_payload.author }}, + value: `${{ github.event.client_payload.author }}`, short: true }, { title: 'Preview URL', - value: ${{ env.PREVIEW_URL }}, + value: `${{ env.PREVIEW_URL }}`, short: true }] }] From 17fb33129f225a879a9c0b944286ac0e6e1e4901 Mon Sep 17 00:00:00 2001 From: Lam <150060045+lamATnginx@users.noreply.github.com> Date: Tue, 17 Jun 2025 07:49:04 -0700 Subject: [PATCH 027/117] Bump docs-action to 1.0.8 from 1.0.7 (#692) chore: Bump docs-action to 1.0.8 from 1.0.7 --- .github/workflows/build-push.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build-push.yml b/.github/workflows/build-push.yml index 1d2bdee5f..da2534a36 100644 --- a/.github/workflows/build-push.yml +++ b/.github/workflows/build-push.yml @@ -58,7 +58,7 @@ jobs: call-docs-build-push: needs: prod-check-branch - uses: nginxinc/docs-actions/.github/workflows/docs-build-push.yml@9c59fab05a8131f4d691ba6ea2b6a119f3ef832a # v1.0.7 + uses: nginxinc/docs-actions/.github/workflows/docs-build-push.yml@bc2ae667727ba02b3d70dacee231ce04f7d94344 # v1.0.8 with: production_url_path: "" preview_url_path: "${{ vars.PREVIEW_URL_PATH }}" From 8a256bad89970b564c23384821c368411f85282a Mon Sep 17 00:00:00 2001 From: Travis Martin <33876974+travisamartin@users.noreply.github.com> Date: Tue, 17 Jun 2025 08:16:35 -0700 Subject: [PATCH 028/117] fixed broken links in NIM deployment guides (#696) * fixed broken links in NIM deployment guides * fixed link to correct doc --- content/nim/deploy/vm-bare-metal/install.md | 2 +- content/nim/disconnected/offline-install-guide.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/nim/deploy/vm-bare-metal/install.md b/content/nim/deploy/vm-bare-metal/install.md index 5e7dea31b..c6afac65d 100644 --- a/content/nim/deploy/vm-bare-metal/install.md +++ b/content/nim/deploy/vm-bare-metal/install.md @@ -21,7 +21,7 @@ The script installs: - ClickHouse by default, unless you choose to skip it - Optionally, NGINX Plus (requires a license and additional flags) -The script also installs all required operating system packages automatically. If you need to install earlier versions of NGINX or NGINX Instance Manager, follow the [manual installation process](({{< ref "nim/deploy/vm-bare-metal/install-nim-manual.md" >}})) instead. +The script also installs all required operating system packages automatically. If you need to install earlier versions of NGINX or NGINX Instance Manager, follow the [manual installation process]({{< ref "nim/deploy/vm-bare-metal/install-nim-manual.md" >}}) instead. --- diff --git a/content/nim/disconnected/offline-install-guide.md b/content/nim/disconnected/offline-install-guide.md index ae422d800..6dda33a5d 100644 --- a/content/nim/disconnected/offline-install-guide.md +++ b/content/nim/disconnected/offline-install-guide.md @@ -21,7 +21,7 @@ The script installs: NGINX Plus is not supported in disconnected mode. -If you need to install earlier versions of NGINX or NGINX Instance Manager, follow the [manual installation process](({{< ref "nim/deploy/vm-bare-metal/install-nim-manual.md" >}})) instead. +If you need to install earlier versions of NGINX or NGINX Instance Manager, follow the [manual installation process]({{< ref "nim/disconnected/offline-install-guide-manual.md" >}}) instead. --- From 90f5e085c526ef890ac88ef17ec753fa9326ac88 Mon Sep 17 00:00:00 2001 From: Alan Dooley Date: Tue, 17 Jun 2025 17:04:03 +0100 Subject: [PATCH 029/117] feat: Migrate NGINX Ingress Controller documentation into repository (#688) * feat: Add NIC content * feat: Move NIC files into appropriate folders * feat: Fix remaining references and configuration * feat: Add NIC to CODEOWNERS * Update .github/CODEOWNERS * Update .github/CODEOWNERS * feat: Codeowners formatting change * Update content/nic/installation/integrations/app-protect-waf-v5/compile-waf-policies.md --------- Co-authored-by: Mike Jang <3287976+mjang@users.noreply.github.com> Co-authored-by: Jon Torre <78599298+JTorreG@users.noreply.github.com> --- .github/CODEOWNERS | 4 + config/_default/config.toml | 5 +- .../nic/compatibility-tables/nic-nap.md | 10 + content/includes/nic/configuration/_index.md | 8 + .../nic/configuration/access-control.md | 120 + .../configuration/configuration-examples.md | 13 + .../global-configuration/_index.md | 8 + .../command-line-arguments.md | 685 +++++ .../configmap-resource.md | 250 ++ .../global-configuration/custom-templates.md | 11 + .../globalconfiguration-resource.md | 183 ++ .../mgmt-configmap-resource.md | 50 + .../reporting-resources-status.md | 195 ++ .../host-and-listener-collisions.md | 168 ++ .../configuration/ingress-resources/_index.md | 8 + ...advanced-configuration-with-annotations.md | 225 ++ .../advanced-configuration-with-snippets.md | 128 + .../ingress-resources/basic-configuration.md | 107 + .../cross-namespace-configuration.md | 14 + .../ingress-resources/custom-annotations.md | 146 + .../nic/configuration/policy-resource.md | 945 +++++++ .../includes/nic/configuration/security.md | 102 + .../configuration/transportserver-resource.md | 416 +++ ...server-and-virtualserverroute-resources.md | 1101 ++++++++ .../installation/create-common-resources.md | 27 + .../installation/create-custom-resources.md | 49 + .../nic/installation/deploy-controller.md | 10 + .../includes/nic/installation/download-jwt.md | 10 + .../nic/installation/jwt-password-note.md | 11 + .../nic/installation/manifests/daemonset.md | 21 + .../nic/installation/manifests/deployment.md | 21 + .../manifests/verify-pods-are-running.md | 10 + content/includes/nic/rbac/set-up-rbac.md | 33 + content/nic/_index.md | 5 + content/nic/community.md | 23 + content/nic/configuration/_index.md | 4 + content/nic/configuration/access-control.md | 120 + .../configuration/configuration-examples.md | 13 + .../global-configuration/_index.md | 8 + .../command-line-arguments.md | 685 +++++ .../configmap-resource.md | 250 ++ .../global-configuration/custom-templates.md | 11 + .../globalconfiguration-resource.md | 183 ++ .../mgmt-configmap-resource.md | 50 + .../reporting-resources-status.md | 195 ++ .../host-and-listener-collisions.md | 168 ++ .../configuration/ingress-resources/_index.md | 8 + ...advanced-configuration-with-annotations.md | 225 ++ .../advanced-configuration-with-snippets.md | 128 + .../ingress-resources/basic-configuration.md | 107 + .../cross-namespace-configuration.md | 14 + .../ingress-resources/custom-annotations.md | 146 + content/nic/configuration/policy-resource.md | 947 +++++++ content/nic/configuration/security.md | 102 + .../configuration/transportserver-resource.md | 416 +++ ...server-and-virtualserverroute-resources.md | 1101 ++++++++ content/nic/glossary.md | 30 + content/nic/installation/_index.md | 8 + .../build-nginx-ingress-controller.md | 203 ++ .../nic/installation/create-license-secret.md | 155 ++ content/nic/installation/ingress-nginx.md | 546 ++++ .../nic/installation/installing-nic/_index.md | 5 + .../deploy-with-nap-using-helm.md | 343 +++ .../installing-nic/installation-with-helm.md | 493 ++++ .../installation-with-manifests.md | 300 ++ .../installation-with-operator.md | 72 + .../installing-nic/upgrade-to-v4.md | 129 + .../nic/installation/integrations/_index.md | 5 + .../integrations/app-protect-dos/_index.md | 8 + .../app-protect-dos/configuration.md | 164 ++ .../app-protect-dos/dos-protected.md | 126 + .../app-protect-dos/installation.md | 230 ++ .../troubleshoot-app-protect-dos.md | 111 + .../integrations/app-protect-waf-v5/_index.md | 8 + .../compile-waf-policies.md | 371 +++ .../app-protect-waf-v5/configuration.md | 180 ++ .../app-protect-waf-v5/installation.md | 505 ++++ .../troubleshoot-app-protect-waf.md | 136 + .../integrations/app-protect-waf/_index.md | 9 + .../app-protect-waf/configuration.md | 563 ++++ .../app-protect-waf/installation.md | 221 ++ .../integrations/f5-ingresslink.md | 95 + .../integrations/nic-n1-console.md | 130 + .../installation/integrations/opentracing.md | 114 + content/nic/installation/nic-images/_index.md | 4 + .../nic-images/get-image-using-jwt.md | 209 ++ .../nic-images/get-registry-image.md | 193 ++ .../installation/nic-images/use-aws-image.md | 139 + .../installation/nic-images/use-gcp-image.md | 118 + .../run-multiple-ingress-controllers.md | 60 + content/nic/logging-and-monitoring/_index.md | 8 + content/nic/logging-and-monitoring/logging.md | 35 + .../nic/logging-and-monitoring/prometheus.md | 85 + .../logging-and-monitoring/service-insight.md | 56 + .../nic/logging-and-monitoring/status-page.md | 64 + content/nic/overview/_index.md | 8 + content/nic/overview/about.md | 17 + content/nic/overview/controller-comparison.md | 65 + content/nic/overview/design.md | 346 +++ content/nic/overview/nginx-plus.md | 35 + content/nic/overview/product-telemetry.md | 84 + content/nic/releases.md | 2407 +++++++++++++++++ content/nic/technical-specifications.md | 99 + content/nic/troubleshooting/_index.md | 8 + .../troubleshooting/troubleshoot-common.md | 201 ++ .../troubleshoot-configmap-policy.md | 48 + .../troubleshooting/troubleshoot-ingress.md | 29 + .../troubleshooting/troubleshoot-support.md | 38 + .../troubleshoot-transportserver.md | 9 + .../troubleshoot-virtualserver.md | 38 + content/nic/tutorials/_index.md | 8 + content/nic/tutorials/custom-listen-ports.md | 159 ++ .../ingress-path-regex-annotation.md | 456 ++++ content/nic/tutorials/nginx-dynamic-module.md | 117 + content/nic/tutorials/nginx-ingress-istio.md | 262 ++ .../nic/tutorials/nginx-ingress-linkerd.md | 181 ++ content/nic/tutorials/nginx-ingress-osm.md | 459 ++++ .../tutorials/oidc-custom-configuration.md | 214 ++ content/nic/tutorials/security-monitoring.md | 96 + ...rtual-server-with-custom-listener-ports.md | 185 ++ content/nic/usage-reporting.md | 294 ++ layouts/shortcodes/call-out.html | 3 + layouts/shortcodes/nic-helm-version.html | 1 + layouts/shortcodes/nic-operator-version.html | 1 + layouts/shortcodes/nic-version.html | 1 + static/nic/control-loop.png | Bin 0 -> 5633 bytes static/nic/controller-sync.png | Bin 0 -> 34320 bytes static/nic/ecr-pull-instructions.png | Bin 0 -> 148462 bytes static/nic/gke-create-cluster.png | Bin 0 -> 101052 bytes static/nic/gke-creating-cluster.png | Bin 0 -> 66820 bytes static/nic/gke-existing-cluster.png | Bin 0 -> 169834 bytes .../gke-ingress-controller-application.png | Bin 0 -> 150715 bytes static/nic/gke-install-to-new-cluster.png | Bin 0 -> 201085 bytes static/nic/ic-high-level.png | Bin 0 -> 99912 bytes static/nic/ic-pod.png | Bin 0 -> 54934 bytes static/nic/ic-process-components.png | Bin 0 -> 51942 bytes static/nic/ic-process.png | Bin 0 -> 24035 bytes static/nic/nginx-envoy.png | Bin 0 -> 152518 bytes static/nic/nginx_istio_small.png | Bin 0 -> 125996 bytes static/nic/nginx_plain.png | Bin 0 -> 119752 bytes 140 files changed, 22101 insertions(+), 2 deletions(-) create mode 100644 content/includes/nic/compatibility-tables/nic-nap.md create mode 100644 content/includes/nic/configuration/_index.md create mode 100644 content/includes/nic/configuration/access-control.md create mode 100644 content/includes/nic/configuration/configuration-examples.md create mode 100644 content/includes/nic/configuration/global-configuration/_index.md create mode 100644 content/includes/nic/configuration/global-configuration/command-line-arguments.md create mode 100644 content/includes/nic/configuration/global-configuration/configmap-resource.md create mode 100644 content/includes/nic/configuration/global-configuration/custom-templates.md create mode 100644 content/includes/nic/configuration/global-configuration/globalconfiguration-resource.md create mode 100644 content/includes/nic/configuration/global-configuration/mgmt-configmap-resource.md create mode 100644 content/includes/nic/configuration/global-configuration/reporting-resources-status.md create mode 100644 content/includes/nic/configuration/host-and-listener-collisions.md create mode 100644 content/includes/nic/configuration/ingress-resources/_index.md create mode 100644 content/includes/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md create mode 100644 content/includes/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md create mode 100644 content/includes/nic/configuration/ingress-resources/basic-configuration.md create mode 100644 content/includes/nic/configuration/ingress-resources/cross-namespace-configuration.md create mode 100644 content/includes/nic/configuration/ingress-resources/custom-annotations.md create mode 100644 content/includes/nic/configuration/policy-resource.md create mode 100644 content/includes/nic/configuration/security.md create mode 100644 content/includes/nic/configuration/transportserver-resource.md create mode 100644 content/includes/nic/configuration/virtualserver-and-virtualserverroute-resources.md create mode 100644 content/includes/nic/installation/create-common-resources.md create mode 100644 content/includes/nic/installation/create-custom-resources.md create mode 100644 content/includes/nic/installation/deploy-controller.md create mode 100644 content/includes/nic/installation/download-jwt.md create mode 100644 content/includes/nic/installation/jwt-password-note.md create mode 100644 content/includes/nic/installation/manifests/daemonset.md create mode 100644 content/includes/nic/installation/manifests/deployment.md create mode 100644 content/includes/nic/installation/manifests/verify-pods-are-running.md create mode 100644 content/includes/nic/rbac/set-up-rbac.md create mode 100644 content/nic/_index.md create mode 100644 content/nic/community.md create mode 100644 content/nic/configuration/_index.md create mode 100644 content/nic/configuration/access-control.md create mode 100644 content/nic/configuration/configuration-examples.md create mode 100644 content/nic/configuration/global-configuration/_index.md create mode 100644 content/nic/configuration/global-configuration/command-line-arguments.md create mode 100644 content/nic/configuration/global-configuration/configmap-resource.md create mode 100644 content/nic/configuration/global-configuration/custom-templates.md create mode 100644 content/nic/configuration/global-configuration/globalconfiguration-resource.md create mode 100644 content/nic/configuration/global-configuration/mgmt-configmap-resource.md create mode 100644 content/nic/configuration/global-configuration/reporting-resources-status.md create mode 100644 content/nic/configuration/host-and-listener-collisions.md create mode 100644 content/nic/configuration/ingress-resources/_index.md create mode 100644 content/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md create mode 100644 content/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md create mode 100644 content/nic/configuration/ingress-resources/basic-configuration.md create mode 100644 content/nic/configuration/ingress-resources/cross-namespace-configuration.md create mode 100644 content/nic/configuration/ingress-resources/custom-annotations.md create mode 100644 content/nic/configuration/policy-resource.md create mode 100644 content/nic/configuration/security.md create mode 100644 content/nic/configuration/transportserver-resource.md create mode 100644 content/nic/configuration/virtualserver-and-virtualserverroute-resources.md create mode 100644 content/nic/glossary.md create mode 100644 content/nic/installation/_index.md create mode 100644 content/nic/installation/build-nginx-ingress-controller.md create mode 100644 content/nic/installation/create-license-secret.md create mode 100644 content/nic/installation/ingress-nginx.md create mode 100644 content/nic/installation/installing-nic/_index.md create mode 100644 content/nic/installation/installing-nic/deploy-with-nap-using-helm.md create mode 100644 content/nic/installation/installing-nic/installation-with-helm.md create mode 100644 content/nic/installation/installing-nic/installation-with-manifests.md create mode 100644 content/nic/installation/installing-nic/installation-with-operator.md create mode 100644 content/nic/installation/installing-nic/upgrade-to-v4.md create mode 100644 content/nic/installation/integrations/_index.md create mode 100644 content/nic/installation/integrations/app-protect-dos/_index.md create mode 100644 content/nic/installation/integrations/app-protect-dos/configuration.md create mode 100644 content/nic/installation/integrations/app-protect-dos/dos-protected.md create mode 100644 content/nic/installation/integrations/app-protect-dos/installation.md create mode 100644 content/nic/installation/integrations/app-protect-dos/troubleshoot-app-protect-dos.md create mode 100644 content/nic/installation/integrations/app-protect-waf-v5/_index.md create mode 100644 content/nic/installation/integrations/app-protect-waf-v5/compile-waf-policies.md create mode 100644 content/nic/installation/integrations/app-protect-waf-v5/configuration.md create mode 100644 content/nic/installation/integrations/app-protect-waf-v5/installation.md create mode 100644 content/nic/installation/integrations/app-protect-waf-v5/troubleshoot-app-protect-waf.md create mode 100644 content/nic/installation/integrations/app-protect-waf/_index.md create mode 100644 content/nic/installation/integrations/app-protect-waf/configuration.md create mode 100644 content/nic/installation/integrations/app-protect-waf/installation.md create mode 100644 content/nic/installation/integrations/f5-ingresslink.md create mode 100644 content/nic/installation/integrations/nic-n1-console.md create mode 100644 content/nic/installation/integrations/opentracing.md create mode 100644 content/nic/installation/nic-images/_index.md create mode 100644 content/nic/installation/nic-images/get-image-using-jwt.md create mode 100644 content/nic/installation/nic-images/get-registry-image.md create mode 100644 content/nic/installation/nic-images/use-aws-image.md create mode 100644 content/nic/installation/nic-images/use-gcp-image.md create mode 100644 content/nic/installation/run-multiple-ingress-controllers.md create mode 100644 content/nic/logging-and-monitoring/_index.md create mode 100644 content/nic/logging-and-monitoring/logging.md create mode 100644 content/nic/logging-and-monitoring/prometheus.md create mode 100644 content/nic/logging-and-monitoring/service-insight.md create mode 100644 content/nic/logging-and-monitoring/status-page.md create mode 100644 content/nic/overview/_index.md create mode 100644 content/nic/overview/about.md create mode 100644 content/nic/overview/controller-comparison.md create mode 100644 content/nic/overview/design.md create mode 100644 content/nic/overview/nginx-plus.md create mode 100644 content/nic/overview/product-telemetry.md create mode 100644 content/nic/releases.md create mode 100644 content/nic/technical-specifications.md create mode 100644 content/nic/troubleshooting/_index.md create mode 100644 content/nic/troubleshooting/troubleshoot-common.md create mode 100644 content/nic/troubleshooting/troubleshoot-configmap-policy.md create mode 100644 content/nic/troubleshooting/troubleshoot-ingress.md create mode 100644 content/nic/troubleshooting/troubleshoot-support.md create mode 100644 content/nic/troubleshooting/troubleshoot-transportserver.md create mode 100644 content/nic/troubleshooting/troubleshoot-virtualserver.md create mode 100644 content/nic/tutorials/_index.md create mode 100644 content/nic/tutorials/custom-listen-ports.md create mode 100644 content/nic/tutorials/ingress-path-regex-annotation.md create mode 100644 content/nic/tutorials/nginx-dynamic-module.md create mode 100644 content/nic/tutorials/nginx-ingress-istio.md create mode 100644 content/nic/tutorials/nginx-ingress-linkerd.md create mode 100644 content/nic/tutorials/nginx-ingress-osm.md create mode 100644 content/nic/tutorials/oidc-custom-configuration.md create mode 100644 content/nic/tutorials/security-monitoring.md create mode 100644 content/nic/tutorials/virtual-server-with-custom-listener-ports.md create mode 100644 content/nic/usage-reporting.md create mode 100644 layouts/shortcodes/call-out.html create mode 100644 layouts/shortcodes/nic-helm-version.html create mode 100644 layouts/shortcodes/nic-operator-version.html create mode 100644 layouts/shortcodes/nic-version.html create mode 100644 static/nic/control-loop.png create mode 100644 static/nic/controller-sync.png create mode 100644 static/nic/ecr-pull-instructions.png create mode 100644 static/nic/gke-create-cluster.png create mode 100644 static/nic/gke-creating-cluster.png create mode 100644 static/nic/gke-existing-cluster.png create mode 100644 static/nic/gke-ingress-controller-application.png create mode 100644 static/nic/gke-install-to-new-cluster.png create mode 100644 static/nic/ic-high-level.png create mode 100644 static/nic/ic-pod.png create mode 100644 static/nic/ic-process-components.png create mode 100644 static/nic/ic-process.png create mode 100644 static/nic/nginx-envoy.png create mode 100644 static/nic/nginx_istio_small.png create mode 100644 static/nic/nginx_plain.png diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 9f28ec942..3d6f1daf8 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -37,6 +37,10 @@ content/includes/nginxaas-azure/* @nginx/n4a-docs-approvers content/ngf/* @nginx/nginx-gateway-fabric content/includes/ngf/* @nginx/nginx-gateway-fabric +# NGINX Ingress Controller +content/nic/* @nginx/kic +content/includes/nic/* @nginx/kic + # NGINX Instance Manager content/nms/nim/* @nginx/nim-docs-approvers content/nim/* @nginx/nim-docs-approvers diff --git a/config/_default/config.toml b/config/_default/config.toml index 10337a908..faf409f46 100644 --- a/config/_default/config.toml +++ b/config/_default/config.toml @@ -7,18 +7,19 @@ pygmentsUseClasses = true enableGitInfo = true [permalinks] + agent = '/nginx-agent/:sections[1:]/:filename' amplify = '/nginx-amplify/:sections[1:]/:filename' controller = '/nginx-controller/:sections[1:]/:filename' mesh = '/nginx-service-mesh/:sections[1:]/:filename' modsec-waf = '/nginx-waf/:sections[1:]/:filename' nap-dos = '/nginx-app-protect-dos/:sections[1:]/:filename' nap-waf = '/nginx-app-protect-waf/:sections[1:]/:filename' + nginxaas = '/nginxaas/azure/:sections[1:]/:filename' ngf = '/nginx-gateway-fabric/:sections[1:]/:filename' + nic = '/nginx-ingress-controller/:sections[1:]/:filename' nim = '/nginx-instance-manager/:sections[1:]/:filename' nms = '/nginx-management-suite/:sections[1:]/:filename' unit = '/nginx-unit/:sections[1:]/:filename' - agent = '/nginx-agent/:sections[1:]/:filename' - nginxaas = '/nginxaas/azure/:sections[1:]/:filename' [caches] [caches.modules] diff --git a/content/includes/nic/compatibility-tables/nic-nap.md b/content/includes/nic/compatibility-tables/nic-nap.md new file mode 100644 index 000000000..aaf914ca1 --- /dev/null +++ b/content/includes/nic/compatibility-tables/nic-nap.md @@ -0,0 +1,10 @@ +The following table shows compatibility between NGINX Ingress Controller (NIC) and NGINX App Protect WAF (NAP-WAF) versions: + +{{< bootstrap-table "table table-striped table-responsive" >}} +| NIC Version | NAP-WAF Version | Config Manager | Enforcer | +| ------------------- | --------------- | -------------- | -------- | +| {{< nic-version >}} | 34+5.332 | 5.6.0 | 5.6.0 | +| 4.0.1 | 33+5.264 | 5.5.0 | 5.5.0 | +| 3.7.2 | 32+5.1 | 5.3.0 | 5.3.0 | +| 3.6.2 | 32+5.48 | 5.2.0 | 5.2.0 | +{{% /bootstrap-table %}} diff --git a/content/includes/nic/configuration/_index.md b/content/includes/nic/configuration/_index.md new file mode 100644 index 000000000..7e1500973 --- /dev/null +++ b/content/includes/nic/configuration/_index.md @@ -0,0 +1,8 @@ +--- +title: Configuration +description: +weight: 1400 +menu: + docs: + parent: NGINX Ingress Controller +--- diff --git a/content/includes/nic/configuration/access-control.md b/content/includes/nic/configuration/access-control.md new file mode 100644 index 000000000..fade2941c --- /dev/null +++ b/content/includes/nic/configuration/access-control.md @@ -0,0 +1,120 @@ +--- +title: Deploy a Policy for access control +weight: 900 +toc: true +docs: DOCS-000 +--- + +This topic describes how to use F5 NGINX Ingress Controller to apply and update a Policy for access control. It demonstrates it using an example application and a [VirtualServer custom resource]({{< ref "/configuration/virtualserver-and-virtualserverroute-resources.md" >}}). + +--- + +## Before you begin + +You should have a [working NGINX Ingress Controller]({{< ref "/installation/installing-nic/installation-with-helm.md" >}}) instance. + +For ease of use in shell commands, set two shell variables: + +1. The public IP address for your NGINX Ingress Controller instance. + +```shell +IC_IP= +``` + +2. The HTTP port of the same instance. + +```shell +IC_HTTP_PORT= +``` + +--- + +## Deploy the example application + +Create the file _webapp.yaml_ with the following contents: + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/webapp.yaml" >}} + +Apply it using `kubectl`: + +```shell +kubectl apply -f webapp.yaml +``` + +--- + +## Deploy a Policy to create a deny rule + +Create a file named _access-control-policy-deny.yaml_. The highlighted _deny_ field will be used by the example application, and should be changed to the subnet of your machine. + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/access-control-policy-deny.yaml" "hl_lines=7-8" >}} + +Apply the policy: + +```shell +kubectl apply -f access-control-policy-deny.yaml +``` + +--- + +## Configure load balancing + +Create a file named _virtual-server.yaml_ for the VirtualServer resource. The _policies_ field references the access control Policy created in the previous section. + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/virtual-server.yaml" "hl_lines=7-8" >}} + +Apply the policy: + +```shell +kubectl apply -f virtual-server.yaml +``` + +--- + +## Test the example application + +Use `curl` to attempt to access the application: + +```shell +curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT +``` +```text + +403 Forbidden + +

403 Forbidden

+ + +``` + +The *403* response is expected, successfully blocking your machine. + +--- + +## Update the Policy to create an allow rule + +Update the Policy with the file _access-control-policy-allow.yaml_, setting the _allow_ field to the subnet of your machine. + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/access-control-policy-allow.yaml" "hl_lines=7-8" >}} + +Apply the Policy: + +```shell +kubectl apply -f access-control-policy-allow.yaml +``` + +---- + +## Verify the Policy update + +Attempt to access the application again: + +```shell +curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT +``` +```text +Server address: 10.64.0.13:8080 +Server name: webapp-5cbbc7bd78-wf85w +``` + +The successful response demonstrates that the policy has been updated. diff --git a/content/includes/nic/configuration/configuration-examples.md b/content/includes/nic/configuration/configuration-examples.md new file mode 100644 index 000000000..0a0e6e3bb --- /dev/null +++ b/content/includes/nic/configuration/configuration-examples.md @@ -0,0 +1,13 @@ +--- +docs: DOCS-584 +doctypes: +- '' +title: Configuration examples +toc: true +weight: 400 +--- + +Our [GitHub repo](https://github.com/nginx/kubernetes-ingress) includes a number of configuration examples: + +- [*Examples of Custom Resources*](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources) show how to advanced NGINX features by using VirtualServer, VirtualServerRoute, TransportServer and Policy Custom Resources. +- [*Examples of Ingress Resources*](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources) show how to use advanced NGINX features in Ingress resources with annotations. diff --git a/content/includes/nic/configuration/global-configuration/_index.md b/content/includes/nic/configuration/global-configuration/_index.md new file mode 100644 index 000000000..4f71e2f5f --- /dev/null +++ b/content/includes/nic/configuration/global-configuration/_index.md @@ -0,0 +1,8 @@ +--- +title: Global configuration +description: +weight: 100 +menu: + docs: + parent: NGINX Ingress Controller +--- diff --git a/content/includes/nic/configuration/global-configuration/command-line-arguments.md b/content/includes/nic/configuration/global-configuration/command-line-arguments.md new file mode 100644 index 000000000..230345052 --- /dev/null +++ b/content/includes/nic/configuration/global-configuration/command-line-arguments.md @@ -0,0 +1,685 @@ +--- +docs: DOCS-585 +doctypes: +- '' +title: Command-line arguments +toc: true +weight: 100 +--- + +F5 NGINX Ingress Controller supports several command-line arguments, which are set based on installation method: + +- If you're using *Kubernetes Manifests* to install NGINX Ingress Controller, modify the Manifests to set the command-line arguments. View the [Installation with Manifests]({{}}) topic for more information. +- If you're using *Helm* to install NGINX Ingress Controller, modify the parameters of the Helm chart to set the command-line arguments. View the [Installation with Helm]({{}}) topic for more information. + + + +### -enable-snippets + +Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources. + +Default `false`. + + + +--- + +### -default-server-tls-secret `` + +Secret with a TLS certificate and key for TLS termination of the default server. + +- If not set, certificate and key in the file `/etc/nginx/secrets/default` are used. +- If `/etc/nginx/secrets/default` doesn't exist, NGINX Ingress Controller will configure NGINX to reject TLS connections to the default server. +- If a secret is set, but NGINX Ingress Controller is not able to fetch it from Kubernetes API, or it is not set and NGINX Ingress Controller fails to read the file "/etc/nginx/secrets/default", NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -wildcard-tls-secret `` + +A Secret with a TLS certificate and key for TLS termination of every Ingress/VirtualServer host for which TLS termination is enabled but the Secret is not specified. + +- If the argument is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection +- If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -enable-custom-resources + +Enables custom resources. + +Default `true`. + + + + +--- + +### -enable-oidc + +Enables OIDC policies. + +Default `false`. + + + +--- + +### -enable-leader-election + +Enables Leader election to avoid multiple replicas of the controller reporting the status of Ingress, VirtualServer and VirtualServerRoute resources -- only one replica will report status. +Default `true`. + +See [-report-ingress-status](#cmdoption-report-ingress-status) flag. + + + +--- + +### -enable-tls-passthrough + +Enable TLS Passthrough on port 443. + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -tls-passthrough-port `` + +Set the port for TLS Passthrough. +Format: `[1024 - 65535]` (default `443`) + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -enable-cert-manager + +Enable x509 automated certificate management for VirtualServer resources using cert-manager (cert-manager.io). + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -enable-external-dns + +Enable integration with ExternalDNS for configuring public DNS entries for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + +--- + +### -external-service `` + +Specifies the name of the service with the type LoadBalancer through which the NGINX Ingress Controller pods are exposed externally. The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. + +For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). + + + +--- + +### -ingresslink `` + +Specifies the name of the IngressLink resource, which exposes the NGINX Ingress Controller pods via a BIG-IP system. The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. + +For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). + + + +--- + +### -global-configuration `` + +A GlobalConfiguration resource for global configuration of NGINX Ingress Controller. + +Format: `/` + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -health-status + +Adds a location "/nginx-health" to the default server. The location responds with the 200 status code for any request. + +Useful for external health-checking of NGINX Ingress Controller. + + + +--- + +### -health-status-uri `` + +Sets the URI of health status location in the default server. Requires [-health-status](#cmdoption-health-status). (default `/nginx-health`) + + + +--- + +### -ingress-class `` + +The `-ingress-class` argument refers to the name of the resource `kind: IngressClass`. An IngressClass resource with a name equal to the class must be deployed. Otherwise, NGINX Ingress Controller will fail to start. +NGINX Ingress Controller will only process Ingress resources that belong to its class (Whose `ingressClassName` value matches the value of `-ingress-class`), skipping the ones without it. It will also process all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the `ingressClassName` field. + +Default `nginx`. + + + +--- + +### -ingress-template-path `` + +Path to the ingress NGINX configuration template for an ingress resource. Default for NGINX is `nginx.ingress.tmpl`; default for NGINX Plus is `nginx-plus.ingress.tmpl`. + + + +--- + +### -leader-election-lock-name `` + +Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. + +Requires [-enable-leader-election](#cmdoption-enable-leader-election). + + + +--- + +### -log_backtrace_at `` + +When logging hits line `file:N`, emit a stack trace. + + + +--- + +### -main-template-path `` + +Path to the main NGINX configuration template. + +- Default for NGINX is `nginx.tmpl`. +- Default for NGINX Plus is `nginx-plus.tmpl`. + + + +--- + +### -nginx-configmaps `` + +A ConfigMap resource for customizing NGINX configuration. If a ConfigMap is set, but NGINX Ingress Controller is not able to fetch it from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -mgmt-configmap `` + +The Management ConfigMap resource is used for customizing the NGINX mgmt block. If using NGINX Plus, a Management ConfigMap must be set. If NGINX Ingress Controller is not able to fetch it from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -nginx-debug + +Enable debugging for NGINX. Uses the nginx-debug binary. Requires 'error-log-level: debug' in the ConfigMap. + + + +--- + +### -nginx-plus + +Enable support for NGINX Plus. + + + +--- + +### -nginx-reload-timeout `` + +Timeout in milliseconds which NGINX Ingress Controller will wait for a successful NGINX reload after a change or at the initial start. + +Default is 60000. + + + +--- + +### -nginx-status + +Enable the NGINX stub_status, or the NGINX Plus API. + +Default `true`. + + + +--- + +### -nginx-status-allow-cidrs `` + +Add IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. + +Separate multiple IP/CIDR by commas. (default `127.0.0.1,::1`) + + + +--- + +### -nginx-status-port `` + +Set the port where the NGINX stub_status or the NGINX Plus API is exposed. + +Format: `[1024 - 65535]` (default `8080`) + + + +--- + +### -proxy `` + +{{< warning >}} This argument is intended for testing purposes only. {{< /warning >}} + +Use a proxy server to connect to Kubernetes API started with `kubectl proxy`. + +NGINX Ingress Controller does not start NGINX and does not write any generated NGINX configuration files to disk. + + + +--- + +### -report-ingress-status + +Updates the address field in the status of Ingress resources. + +Requires the [-external-service](#cmdoption-external-service) or [-ingresslink](#cmdoption-ingresslink) flag, or the `external-status-address` key in the ConfigMap. + + + +--- + +### -transportserver-template-path `` + +Path to the TransportServer NGINX configuration template for a TransportServer resource. + +- Default for NGINX is `nginx.transportserver.tmpl`. +- Default for NGINX Plus is `nginx-plus.transportserver.tmpl`. + + + +--- + +### -log-level `` + +Log level for Ingress Controller logs. Allowed values: fatal, error, warn, info, debug, trace. + +- Default is `info`. + + + +--- + +### -log-format `` + +Log format for Ingress Controller logs. Allowed values: glog, json, text. + +- Default is `glog`. + + + +--- + +### -version + +Print the version, git-commit hash and build date and exit. + + + +--- + +### -virtualserver-template-path `` + +Path to the VirtualServer NGINX configuration template for a VirtualServer resource. + +- Default for NGINX is `nginx.virtualserver.tmpl`. +- Default for NGINX Plus is `nginx-plus.virtualserver.tmpl`. + + + + +--- + +### -vmodule `` + +A comma-separated list of pattern=N settings for file-filtered logging. + + + +--- + +### -watch-namespace `` + +Comma separated list of namespaces NGINX Ingress Controller should watch for resources. By default NGINX Ingress Controller watches all namespaces. Mutually exclusive with "watch-namespace-label". + + + +--- + +### -watch-namespace-label `` + +Configures NGINX Ingress Controller to watch only those namespaces with label foo=bar. By default NGINX Ingress Controller watches all namespaces. Mutually exclusive with "watch-namespace". + + + +--- + +### -watch-secret-namespace `` + +Comma separated list of namespaces NGINX Ingress Controller should watch for secrets. If this arg is not configured, NGINX Ingress Controller watches the same namespaces for all resources, see "watch-namespace" and "watch-namespace-label". All namespaces included with this argument must be part of either `-watch-namespace` or `-watch-namespace-label`. + + + +--- + +### -enable-prometheus-metrics + +Enables exposing NGINX or NGINX Plus metrics in the Prometheus format. + + + +--- + +### -prometheus-metrics-listen-port `` + +Sets the port where the Prometheus metrics are exposed. + +Format: `[1024 - 65535]` (default `9113`) + + + +--- + +### -prometheus-tls-secret `` + +A Secret with a TLS certificate and key for TLS termination of the Prometheus metrics endpoint. + +- If the argument is not set, the Prometheus endpoint will not use a TLS connection. +- If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. + + + +--- + +### -enable-service-insight + +Exposes the Service Insight endpoint for Ingress Controller. + + + +--- + +### -service-insight-listen-port `` + +Sets the port where the Service Insight is exposed. + +Format: `[1024 - 65535]` (default `9114`) + + + +--- + +### -service-insight-tls-secret `` + +A Secret with a TLS certificate and key for TLS termination of the Service Insight endpoint. + +- If the argument is not set, the Service Insight endpoint will not use a TLS connection. +- If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -spire-agent-address `` + +Specifies the address of a running Spire agent. **For use with NGINX Service Mesh only**. + +- If the argument is set, but NGINX Ingress Controller is unable to connect to the Spire Agent, NGINX Ingress Controller will fail to start. + + + + +--- + +### -enable-internal-routes + +Enable support for internal routes with NGINX Service Mesh. **For use with NGINX Service Mesh only**. + +Requires [-spire-agent-address](#cmdoption-spire-agent-address). + +- If the argument is set, but `spire-agent-address` is not provided, NGINX Ingress Controller will fail to start. + + + +--- + +### -enable-latency-metrics + +Enable collection of latency metrics for upstreams. +Requires [-enable-prometheus-metrics](#cmdoption-enable-prometheus-metrics). + + + +--- + +### -enable-app-protect + +Enables support for App Protect. + +Requires [-nginx-plus](#cmdoption-nginx-plus). + +- If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-log-level `` + +Sets log level for App Protect. Allowed values: fatal, error, warn, info, debug, trace. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect](#cmdoption-enable-app-protect). + +- If the argument is set, but `nginx-plus` and `enable-app-protect` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -enable-app-protect-dos + +Enables support for App Protect DoS. + +Requires [-nginx-plus](#cmdoption-nginx-plus). + +- If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-debug + +Enable debugging for App Protect DoS. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-max-daemons + +Max number of ADMD instances. + +Default `1`. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-max-workers + +Max number of nginx processes to support. + +Default `Number of CPU cores in the machine`. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-memory + +RAM memory size to consume in MB + +Default `50% of free RAM in the container or 80MB, the smaller`. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -ready-status + +Enables the readiness endpoint `/nginx-ready`. The endpoint returns a success code when NGINX has loaded all the config after the startup. + +Default `true`. + + + +--- + +### -ready-status-port + +The HTTP port for the readiness endpoint. + +Format: `[1024 - 65535]` (default `8081`) + +--- + +### -disable-ipv6 + +Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack. + +Default `false`. + + + +--- + +### -default-http-listener-port + +Sets the port for the HTTP `default_server` listener. + +Default `80`. + + + +--- + +### -default-https-listener-port + +Sets the port for the HTTPS `default_server` listener. + +Default `443`. + + + +--- + +### -ssl-dynamic-reload + +Used to activate or deactivate lazy loading for SSL Certificates. + +The default value is `true`. + + + +--- + +### -weight-changes-dynamic-reload + +Enables the ability to change the weight distribution of two-way split clients without reloading NGINX. + +Requires [-nginx-plus](#cmdoption-nginx-plus). + +Using this feature may require increasing `map_hash_bucket_size`, `map_hash_max_size`, `variable_hash_bucket_size`, and `variable_hash_max_size` in the ConfigMap based on the number of two-way splits. + +The default value is `false`. + +- If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will ignore the flag. + + + +--- + +### -enable-telemetry-reporting + +Enable gathering and reporting of software telemetry. + +The default value is `true`. + + + +--- + +### -agent + +Enable NGINX Agent which can used with `-enable-app-protect` to send events to Security Monitoring. + +The default value is `false`. + + + +--- + +### -agent-instance-group + +Specify the instance group name to use for the NGINX Ingress Controller deployment when using `-agent`. + + diff --git a/content/includes/nic/configuration/global-configuration/configmap-resource.md b/content/includes/nic/configuration/global-configuration/configmap-resource.md new file mode 100644 index 000000000..afa0bcf17 --- /dev/null +++ b/content/includes/nic/configuration/global-configuration/configmap-resource.md @@ -0,0 +1,250 @@ +--- +title: ConfigMap resources +weight: 300 +toc: true +type: how-to +product: NIC +docs: DOCS-586 +--- + +When using F5 NGINX Ingress Controller, you can customize or fine tune NGINX behavior using ConfigMap resources. Examples include setting the number of worker processes or customizing the access log format. + +## Using ConfigMap + +1. The [Installation with Manifests]({{< relref "installation/installing-nic/installation-with-manifests.md" >}}) documentation deploy an empty ConfigMap while the default installation manifests specify it in the command-line arguments of the Ingress Controller. However, if you customized the manifests, to use ConfigMap, make sure to specify the ConfigMap resource to use the [command-line arguments]({{< ref "/nic/configuration/global-configuration/command-line-arguments" >}}) of NGINX Ingress Controller. + +1. Create a ConfigMap file with the name *nginx-config.yaml* and set the values +that make sense for your setup: + + ```yaml + kind: ConfigMap + apiVersion: v1 + metadata: + name: nginx-config + namespace: nginx-ingress + data: + proxy-connect-timeout: "10s" + proxy-read-timeout: "10s" + client-max-body-size: "2m" + ``` + + See the section [Summary of ConfigMap Keys](#summary-of-configmap-keys) for the explanation of the available ConfigMap keys (such as `proxy-connect-timeout` in this example). + +1. Create a new (or update the existing) ConfigMap resource: + + ```shell + kubectl apply -f nginx-config.yaml + ``` + + The NGINX configuration will be updated. + +--- + +## ConfigMap and Ingress annotations + +ConfigMap applies globally, meaning that it affects every Ingress resource. In contrast, annotations always apply to their Ingress resource. Annotations can override some ConfigMap keys: an example is that the `nginx.org/proxy-connect-timeout` annotations overrides the `proxy-connect-timeout` ConfigMap key. + +For more information, view the [Advanced configuration with annotations]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-annotations" >}}) topic. + +--- + +## ConfigMap and VirtualServer/VirtualServerRoute resources + +The ConfigMap affects every VirtualServer and VirtualServerRoute resources. However, the fields of those resources allow overriding some ConfigMap keys. For example, the `connect-timeout` field of the `upstream` overrides the `proxy-connect-timeout` ConfigMap key. + +For more information, view the [VirtualServer and VirtualServerRoute resources]({{< ref "/nic/configuration/virtualserver-and-virtualserverroute-resources" >}}) topic. + +--- + +## ConfigMap keys + +### Ingress Controller (Unrelated to NGINX Configuration) + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*external-status-address* | Sets the address to be reported in the status of Ingress resources. Requires the *-report-status* command-line argument. Overrides the *-external-service* argument. | N/A | [Reporting resource status]({{< ref "/nic/configuration/global-configuration/reporting-resources-status" >}}) | +{{}} + +--- + +### General customization + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | +|*proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | +|*proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | +|*client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | +|*proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | +|*proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | +|*proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | +|*proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | +|*set-real-ip-from* | Sets the value of the [set_real_ip_from](https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from) directive. | N/A | | +|*real-ip-header* | Sets the value of the [real_ip_header](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header) directive. | *X-Real-IP* | | +|*real-ip-recursive* | Enables or disables the [real_ip_recursive](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_recursive) directive. | *False* | | +|*default-server-return* | Configures the [return](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return) directive in the default server, which handles a client request if none of the hosts of Ingress or VirtualServer resources match. The default value configures NGINX to return a 404 error page. You can configure a fixed response or a redirect. For example, *default-server-return: 302 https://nginx.org* will redirect a client to *https://nginx.org*. | *404* | | +|*server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. Additionally, with the NGINX Plus, you can specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | +|*worker-processes* | Sets the value of the [worker_processes](https://nginx.org/en/docs/ngx_core_module.html#worker_processes) directive. | *auto* | | +|*worker-rlimit-nofile* | Sets the value of the [worker_rlimit_nofile](https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile) directive. | N/A | | +|*worker-connections* | Sets the value of the [worker_connections](https://nginx.org/en/docs/ngx_core_module.html#worker_connections) directive. | *1024* | | +|*worker-cpu-affinity* | Sets the value of the [worker_cpu_affinity](https://nginx.org/en/docs/ngx_core_module.html#worker_cpu_affinity) directive. | N/A | | +|*worker-shutdown-timeout* | Sets the value of the [worker_shutdown_timeout](https://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout) directive. | N/A | | +|*server-names-hash-bucket-size* | Sets the value of the [server_names_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size) directive. | *256* | | +|*server-names-hash-max-size* | Sets the value of the [server_names_hash_max_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_max_size) directive. | *1024* | | +|*map-hash-bucket-size* | Sets the value of the [map_hash_bucket_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size) directive.| *256* | | +|*map-hash-max-size* | Sets the value of the [map_hash_max_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_max_size) directive. | *2048* | | +|*resolver-addresses* | Sets the value of the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) addresses. Note: If you use a DNS name (for example, *kube-dns.kube-system.svc.cluster.local* ) as a resolver address, NGINX Plus will resolve it using the system resolver during the start and on every configuration reload. If the name cannot be resolved or the DNS server doesn't respond, NGINX Plus will fail to start or reload. To avoid this, we recommend using IP addresses as resolver addresses instead of DNS names. Supported in NGINX Plus only. | N/A | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*resolver-ipv6* | Enables IPv6 resolution in the resolver. Supported in NGINX Plus only. | *True* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*resolver-valid* | Sets the time NGINX caches the resolved DNS records. Supported in NGINX Plus only. | TTL value of a DNS record | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*resolver-timeout* | Sets the [resolver_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver_timeout) for name resolution. Supported in NGINX Plus only. | *30s* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*keepalive-timeout* | Sets the value of the [keepalive_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) directive. | *75s* | | +|*keepalive-requests* | Sets the value of the [keepalive_requests](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests) directive. | *1000* | | +|*variables-hash-bucket-size* | Sets the value of the [variables_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_bucket_size) directive. | *256* | | +|*variables-hash-max-size* | Sets the value of the [variables-hash-max-size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_max_size) directive. | *1024* | | +{{}} + +--- + +### Logging + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*error-log-level* | Sets the global [error log level](https://nginx.org/en/docs/ngx_core_module.html#error_log) for NGINX. | *notice* | | +|*access-log* | Sets the directive [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). A syslog destination is the only valid value. The value will be set to its default in-case user tries to configure it with location other than a syslog. +| ``/dev/stdout main`` | ``syslog:server=localhost:514`` | +|*access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). | *False* | | +|*default-server-access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) for the default server. If access log is disabled globally (*access-log-off: "True"*), then the default server access log is always disabled. | *False* | | +|*log-format* | Sets the custom [log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for HTTP and HTTPS traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx.tmpl) for the access log. | [Custom Log Format](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/shared-examples/custom-log-format). | +|*log-format-escaping* | Sets the characters escaping for the variables of the log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | +|*stream-log-format* | Sets the custom [log format](https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format) for TCP, UDP, and TLS Passthrough traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx.tmpl). | | +|*stream-log-format-escaping* | Sets the characters escaping for the variables of the stream log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | +{{}} + +--- + +### Request URI/Header manipulation + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: *"nginx.org/proxy-hide-headers": "header-a,header-b"* | N/A | | +|*proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: *"nginx.org/proxy-pass-headers": "header-a,header-b"* | N/A | | +{{}} + +--- + +### Auth and SSL/TLS + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*redirect-to-https* | Sets the 301 redirect rule based on the value of the *http_x_forwarded_proto* header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of the Ingress Controller — see [115](https://github.com/nginx/kubernetes-ingress/issues/115) | *False* | | +|*ssl-redirect* | Sets an unconditional 301 redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS. | *True* | | +|*hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/) : the HSTS header is added to the responses from backends. The *preload* directive is included in the header. | *False* | | +|*hsts-max-age* | Sets the value of the *max-age* directive of the HSTS header. | *2592000* (1 month) | | +|*hsts-include-subdomains* | Adds the *includeSubDomains* directive to the HSTS header. | *False* | | +|*hsts-behind-proxy* | Enables HSTS based on the value of the *http_x_forwarded_proto* request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of the Ingress Controller. Note: to control redirection from HTTP to HTTPS configure the *nginx.org/redirect-to-https* annotation. | *False* | | +|*ssl-protocols* | Sets the value of the [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) directive. | *TLSv1 TLSv1.1 TLSv1.2* | | +|*ssl-prefer-server-ciphers* | Enables or disables the [ssl_prefer_server_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers) directive. | *False* | | +|*ssl-ciphers* | Sets the value of the [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) directive. | *HIGH:!aNULL:!MD5* | | +|*ssl-dhparam-file* | Sets the content of the dhparam file. The controller will create the file and set the value of the [ssl_dhparam](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam) directive with the path of the file. | N/A | | +{{}} + +--- + +### Listeners + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*http2* | Enables HTTP/2 in servers with SSL enabled. | *False* | | +|*proxy-protocol* | Enables PROXY Protocol for incoming connections. | *False* | [Proxy Protocol](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/shared-examples/proxy-protocol). | +{{}} + +--- + +### Backend services (Upstreams) + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*lb-method* | Sets the [load balancing method](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method). To use the round-robin method, specify *"round_robin"*. | *"random two least_conn"* | | +|*max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the *server* directive. | *1* | | +|*upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. For NGINX, the special value 0 disables the shared memory zones. For NGINX Plus, shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *256k* for NGINX, *512k* for NGINX Plus | | +|*fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the *server* directive. | *10s* | | +|*keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that *proxy_set_header Connection "";* is added to the generated configuration when the value > 0. | *0* | | +{{}} + +--- + +### Zone Sync + +Zone Sync enables the [ngx_stream_zone_sync_module](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) in NGINX Ingress Controller when NGINX Plus is used. Multiple replicas are required to effectively utililise this functionality. More information is available in the [How NGINX Plus Performs Zone Synchronization](https://docs.nginx.com/nginx/admin-guide/high-availability/zone_sync_details/) topic. + +Zone synchronization with TLS for NGINX Ingress Controller is not yet available with ConfigMap. If you would like to enable Zone Sync with TLS, please remove `zone-sync` from ConfigMap and add Zone Sync parameters via [`stream-snippets`]({{< ref "/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) similar to [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-config.yaml) and adding the [zone_sync_ssl directive](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl) along with any other TLS parameters to the `stream-snippets`. + +You will also need to manually add the headless service, such as in [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml). + +{{< caution >}} +If you previously installed OIDC or used the `zone_sync` directive with `stream-snippets` in [v4.0.1](https://github.com/nginx/kubernetes-ingress/tree/v4.0.1) or earlier, and you plan to enable the `zone-sync` ConfigMap key, the `zone_sync` directive should be removed from `stream-snippets`. + +If you encounter the error `error [emerg] 13#13: "zone_sync" directive is duplicate in /etc/nginx/nginx.conf:164` it is likely due to `zone_sync` being enabled in both `stream-snippets` and the ConfigMap. Once upgraded, remove the [old headless service](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml) deployed for OIDC. +{{< /caution >}} + + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*zone-sync* | Enables zone synchronization between NGINX Ingress Controller Pods. This autogenerates a [zone_sync_server](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_server) and a headless service using the `ReplicaSet` or `DaemonSet` name. Please note that this headless service will be automatically cleaned up when uninstalling via Helm or by removing the value from the ConfigMap. The headless service will need to be manually removed if the `controller.customConfigMap` value is set via Helm or the deployment is uninstalled via Manifests. Each Ingress Controller manages its own headless service. NGINX Plus Required. | *False* | | +|*zone-sync-port* | Specifies the optional port on which NGINX Ingress Controller listens for zone sync traffic. NGINX Plus & `zone-sync` Required. | *12345* | | +|*zone-sync-resolver-addresses* | Configures optional addresses used in the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync. This field takes a comma separated list of addresses. NGINX Plus & `zone-sync` Required | `kube-dns.kube-system.svc.cluster.local` | | +|*zone-sync-resolver-ipv6* | Configures whether the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will look up IPv6 addresses. NGINX Plus & `zone-sync` Required | `true` | | +|*zone-sync-resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will override the TTL value of responses from nameservers with. NGINX Plus & `zone-sync` Required | `5s` | | +{{}} + +--- + +### Snippets and custom templates + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*main-snippets* | Sets a custom snippet in main context. | N/A | | +|*http-snippets* | Sets a custom snippet in http context. | N/A | | +|*location-snippets* | Sets a custom snippet in location context. | N/A | | +|*server-snippets* | Sets a custom snippet in server context. | N/A | | +|*stream-snippets* | Sets a custom snippet in stream context. | N/A | [Support for TCP/UDP Load Balancing](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/tcp-udp). | +|*main-template* | Sets the main NGINX configuration template. | By default the template is read from the file in the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +|*ingress-template* | Sets the NGINX configuration template for an Ingress resource. | By default the template is read from the file on the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +|*virtualserver-template* | Sets the NGINX configuration template for an VirtualServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +|*transportserver-template* | Sets the NGINX configuration template for a TransportServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +{{}} + +--- + +### Modules + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*otel-exporter-endpoint* | OTLP/gRPC endpoint that will accept [OpenTelemetry](https://opentelemetry.io) data. Set `otel-trace-in-http` to *"true"* to enable OpenTelemetry at the global level. | N/A | *"https://otel-collector:4317"* | +|*otel-exporter-header-name* | The name of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-value` required. | N/A | *"X-custom-header"* | +|*otel-exporter-header-value* | The value of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-name` required. | N/A | *"custom-value"* | +|*otel-service-name* | Sets the `service.name` attribute of the OTel resource. `otel-exporter-endpoint` required. | N/A | *"nginx-ingress-controller:nginx"* | +| *otel-trace-in-http* | Enables [OpenTelemetry](https://opentelemetry.io) globally (for all Ingress, VirtualServer and VirtualServerRoute resources). Set this to *"false"* to enable OpenTelemetry for individual routes with snippets. `otel-exporter-endpoint` required. | *"false"* | *"true"* | +|*opentracing* | Removed in v5.0.0. Enables [OpenTracing](https://opentracing.io) globally (for all Ingress, VirtualServer and VirtualServerRoute resources). Note: requires the Ingress Controller image with OpenTracing module and a tracer. See the [docs]({{< relref "/installation/integrations/opentracing.md" >}}) for more information. | *False* | | +|*opentracing-tracer* | Removed in v5.0.0. Sets the path to the vendor tracer binary plugin. | N/A | | +|*opentracing-tracer-config* | Removed in v5.0.0. Sets the tracer configuration in JSON format. | N/A | | +|*app-protect-compressed-requests-action* | Sets the *app_protect_compressed_requests_action* [global directive](/nginx-app-protect/configuration/#global-directives). | *drop* | | +|*app-protect-cookie-seed* | Sets the *app_protect_cookie_seed* [global directive](/nginx-app-protect/configuration/#global-directives). | Random automatically generated string | | +|*app-protect-failure-mode-action* | Sets the *app_protect_failure_mode_action* [global directive](/nginx-app-protect/configuration/#global-directives). | *pass* | | +|*app-protect-cpu-thresholds* | Sets the *app_protect_cpu_thresholds* [global directive](/nginx-app-protect/configuration/#global-directives). | *high=100 low=100* | | +|*app-protect-physical-memory-util-thresholds* | Sets the *app_protect_physical_memory_util_thresholds* [global directive](/nginx-app-protect/configuration/#global-directives). | *high=100 low=100* | | +|`app-protect-reconnect-period-seconds` | Sets the `app_protect_reconnect_period_seconds` [global directive](/nginx-app-protect/configuration/#global-directives). | `5` | | +|*app-protect-dos-log-format* | Sets the custom [log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for Dos Access log traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | `, vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, policy_name=$app_protect_dos_policy_name, dos_version=$app_protect_dos_version, ip_tls=$remote_addr:$app_protect_dos_tls_fp,` | | +|*app-protect-dos-log-format-escaping* | Sets the characters escaping for the variables of the stream log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | +|*app-protect-dos-arb-fqdn* | Sets the *app-protect-dos-arb-fqdn* [directive](/nginx-app-protect-dos/directives-and-policy/learn-about-directives-and-policy/#arbitrator-fqdn-directive-app_protect_dos_arb_fqdn). | *svc-appprotect-dos-arb* | | +{{}} diff --git a/content/includes/nic/configuration/global-configuration/custom-templates.md b/content/includes/nic/configuration/global-configuration/custom-templates.md new file mode 100644 index 000000000..589ce5951 --- /dev/null +++ b/content/includes/nic/configuration/global-configuration/custom-templates.md @@ -0,0 +1,11 @@ +--- +docs: DOCS-587 +doctypes: +- '' +title: Custom templates +toc: true +weight: 500 +--- + + +F5 NGINX Ingress Controller uses templates to generate NGINX configuration for Ingress resources, VirtualServer resources and the main NGINX configuration file. You can customize the templates and apply them via the ConfigMap. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/shared-examples/custom-templates). diff --git a/content/includes/nic/configuration/global-configuration/globalconfiguration-resource.md b/content/includes/nic/configuration/global-configuration/globalconfiguration-resource.md new file mode 100644 index 000000000..605ab301f --- /dev/null +++ b/content/includes/nic/configuration/global-configuration/globalconfiguration-resource.md @@ -0,0 +1,183 @@ +--- +docs: DOCS-588 +doctypes: +- '' +title: GlobalConfiguration resource +toc: true +weight: 200 +--- + +This page explains how to use the GlobalConfiguration resource to define the global configuration parameters of F5 NGINX Ingress Controller. + +The resource supports configuring listeners for TCP and UDP load balancing, and is implemented as a [Custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +Listeners are required by [TransportServer resources]({{< relref "/configuration/transportserver-resource.md" >}}) and can be used to [configure custom listeners for VirtualServers]({{< relref "tutorials/virtual-server-with-custom-listener-ports.md" >}}). + +--- + +## Prerequisites + +When [installing NGINX Ingress Controller using Manifests]({{< relref "/installation/installing-nic/installation-with-manifests.md" >}}), you need to reference a GlobalConfiguration resource in the [`-global-configuration`](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments#cmdoption-global-configuration) command-line argument. NGINX Ingress Controller only needs one GlobalConfiguration resource. + +--- + +## GlobalConfiguration specification + +The GlobalConfiguration resource defines the global configuration parameters of the Ingress Controller. Below is an example: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: GlobalConfiguration +metadata: + name: nginx-configuration + namespace: nginx-ingress +spec: + listeners: + - name: dns-udp + port: 5353 + protocol: UDP + - name: dns-tcp + port: 5353 + protocol: TCP + - name: http-8083 + port: 8083 + protocol: HTTP + - name: https-8443 + port: 8443 + protocol: HTTP + ssl: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +| *listeners* | A list of listeners. | [listener](#listener) | No | +{{}} + +### Listener + +The `listeners:` key defines a listener (a combination of a protocol and a port) that NGINX will use to accept traffic for a [TransportServer]({{< relref "/configuration/transportserver-resource.md" >}}) and a [VirtualServer]({{< relref "/configuration/virtualserver-and-virtualserverroute-resources.md" >}}): + +```yaml +- name: dns-tcp + port: 5353 + protocol: TCP +- name: http-8083 + port: 8083 + protocol: HTTP +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +| *name* | The name of the listener. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``listener-123`` are valid. The name must be unique among all listeners. The name ``tls-passthrough`` is reserved for the built-in TLS Passthrough listener and cannot be used. | *string* | Yes | +| *port* | The port of the listener. The port must fall into the range ``1..65535`` with the following exceptions: ``80``, ``443``, the [status port](/nginx-ingress-controller/logging-and-monitoring/status-page), the [Prometheus metrics port](/nginx-ingress-controller/logging-and-monitoring/prometheus). Among all listeners, only a single combination of a port-protocol is allowed. | *int* | Yes | +| *protocol* | The protocol of the listener. Supported values: ``TCP``, ``UDP`` and ``HTTP``. | *string* | Yes | +| *ssl* | Configures the listener with SSL. This is currently only supported for ``HTTP`` listeners. Default value is ``false`` | *bool* | No | +| *ipv4* | Specifies the IPv4 address to listen on. | *string* | No | +| *ipv6* | Specifies the IPv6 address to listen on. | *string* | No | + +{{}} + +--- + +## Using GlobalConfiguration + +You can use the usual `kubectl` commands to work with a GlobalConfiguration resource. + +For example, the following command creates a GlobalConfiguration resource defined in `global-configuration.yaml` with the name `nginx-configuration`: + +```shell +kubectl apply -f global-configuration.yaml +``` +```shell +globalconfiguration.k8s.nginx.org/nginx-configuration created +``` + +Assuming the namespace of the resource is `nginx-ingress`, you can get the resource by running: + +```shell +kubectl get globalconfiguration nginx-configuration -n nginx-ingress +``` +```shell +NAME AGE +nginx-configuration 13s +``` + +With `kubectl get` and similar commands, you can use the short name `gc` instead of `globalconfiguration`. + +--- + +### Validation + +Two types of validation are available for the GlobalConfiguration resource: + +- *Structural validation* by `kubectl` and Kubernetes API server. +- *Comprehensive validation* by NGINX Ingress Controller. + + +#### Structural validation + +The custom resource definition for the GlobalConfiguration includes structural OpenAPI schema which describes the type of every field of the resource. + +If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of a listener), `kubectl` and Kubernetes API server will reject such a resource: + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f global-configuration.yaml + ``` + ```text + error: error validating "global-configuration.yaml": error validating data: ValidationError(GlobalConfiguration.spec.listeners[0].port): invalid type for org.nginx.k8s.v1.GlobalConfiguration.spec.listeners.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f global-configuration.yaml --validate=false + ``` + ```text + The GlobalConfiguration "nginx-configuration" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: + spec.listeners.port in body must be of type integer: "string" + ``` + +If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. + +#### Comprehensive validation + +NGINX Ingress Controller validates the fields of a GlobalConfiguration resource. If a GlobalConfiguration resource is partially invalid, NGINX Ingress Controller use the valid listeners and emit events about invalid listeners. + +You can check if the Ingress Controller successfully applied the configuration for a GlobalConfiguration. For our `nginx-configuration` GlobalConfiguration, we can run: + +```shell +kubectl describe gc nginx-configuration -n nginx-ingress +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Updated 11s nginx-ingress-controller GlobalConfiguration nginx-ingress/nginx-configuration was updated +``` + +The events section includes a Normal event with the Updated reason that informs us that the configuration was successfully applied. + +If you create a GlobalConfiguration `nginx-configuration` with two or more listeners that have the same protocol UDP and port 53, you will get: + +```shell +kubectl describe gc nginx-configuration -n nginx-ingress +``` +```text +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Updated 55s nginx-ingress-controller GlobalConfiguration nginx-ingress/nginx-configuration was updated + Warning AddedOrUpdatedWithError 6s nginx-ingress-controller GlobalConfiguration nginx-ingress/nginx-configuration is invalid and was rejected: spec.listeners: Duplicate value: "Duplicated port/protocol combination 53/UDP" +``` + +The events section includes a Warning event with the AddedOrUpdatedWithError reason. + + +## Using IPV4 and IPV6 Addresses with GlobalConfiguration + +You can customize the IPv4 and IPv6 Address listeners in the global configuration and apply them to your VirtualServer resources. See the corresponding example [here](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/custom-ip-listeners/virtualserver/) diff --git a/content/includes/nic/configuration/global-configuration/mgmt-configmap-resource.md b/content/includes/nic/configuration/global-configuration/mgmt-configmap-resource.md new file mode 100644 index 000000000..e1d9a120c --- /dev/null +++ b/content/includes/nic/configuration/global-configuration/mgmt-configmap-resource.md @@ -0,0 +1,50 @@ +--- +docs: DOCS-586 +doctypes: +- '' +title: Management ConfigMap resource +toc: true +weight: 300 +--- + +When using F5 NGINX Ingress Controller with NGINX Plus, it is required to pass a [command line argument]({{< ref "/nic/configuration/global-configuration/command-line-arguments" >}}) to NGINX Ingress Controller, `--mgmt-configmap=` which specifies the ConfigMap to use. The minimal required ConfigMap must have a `license-token-secret-name` key. Helm users will not need to create this map or pass the argument, it will be created with a Helm install. + +--- + +1. Create a ConfigMap file with the name *nginx-config-mgmt.yaml* and set the values +that make sense for your setup: + + ```yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: nginx-config-mgmt + namespace: nginx-ingress + data: + license-token-secret-name: "license-token" + ``` +1. Create a new (or update the existing) ConfigMap resource: + + ```shell + kubectl apply -f nginx-config-mgmt.yaml + ``` + + The [NGINX Management](https://nginx.org/en/docs/ngx_mgmt_module.html) block configuration will be updated. +--- +## Management ConfigMap keys + +{{}} +|ConfigMap Key | Description | Default | +| ---| ---| ---| +|*license-token-secret-name* | Configures the secret used in the [license_token](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.com/license` with the base64 encoded JWT in the `license.jwt` key. | N/A | +|*ssl-verify* | Configures the [ssl_verify](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_verify) directive, which enables or disables verification of the usage reporting endpoint certificate. | `true` | +|*enforce-initial-report* | Configures the [enforce_initial_report](https://nginx.org/en/docs/ngx_mgmt_module.html#enforce_initial_report) directive, which enables or disables the 180-day grace period for sending the initial usage report. | `false` | +|*usage-report-endpoint* | Configures the endpoint of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This is used to configure the endpoint NGINX uses to send usage reports to NIM. | `product.connect.nginx.com` | +|*usage-report-interval* | Configures the interval of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This specifies the frequency that usage reports are sent. This field takes an [NGINX time](https://nginx.org/en/docs/syntax.html). | `1h` | +|*usage-report-proxy-host* | Configures the host name of the [proxy](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy) directive with optional port. | N/A | +|*ssl-trusted-certificate-secret-name* | Configures the secret used to create the file(s) referenced the in [ssl_trusted_certifcate](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_trusted_certificate), and [ssl_crl](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.org/ca`, where the `ca.crt` key contains a base64 encoded trusted cert, and the optional `ca.crl` key can contain a base64 encoded CRL. If the optional `ca.crl` key is supplied, it will configure the NGINX `ssl_crl` directive. | N/A | +|*ssl-certificate-secret-name* | Configures the secret used to create the `ssl_certificate` and `ssl_certificate_key` directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `kubernetes.io/tls`| N/A | +|*resolver-addresses* | Configures addresses used in the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive. This field takes a comma separated list of addresses. | N/A | +|*resolver-ipv6* | Configures whether the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will look up IPv6 addresses. | `true` | +|*resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will override the TTL value of responses from nameservers with. | N/A | +{{}} diff --git a/content/includes/nic/configuration/global-configuration/reporting-resources-status.md b/content/includes/nic/configuration/global-configuration/reporting-resources-status.md new file mode 100644 index 000000000..8e36733eb --- /dev/null +++ b/content/includes/nic/configuration/global-configuration/reporting-resources-status.md @@ -0,0 +1,195 @@ +--- +docs: DOCS-589 +doctypes: +- '' +title: Reporting resource status +toc: true +weight: 600 +--- + +This page describes how to view the status of resources managed by F5 NGINX Ingress Controller. + +## Ingress resources + +An Ingress resource status includes the address (an IP address or a DNS name), through which the hosts of that Ingress resource are publicly accessible. + +You can see the address in the output of the `kubectl get ingress` command, in the ADDRESS column, as shown below: + +```shell +kubectl get ingresses +``` +```text +NAME HOSTS ADDRESS PORTS AGE +cafe-ingress cafe.example.com 12.13.23.123 80, 443 2m +``` + +NGINX Ingress Controller must be configured to report an Ingress status: + +1. Use the command-line flag `-report-ingress-status`. +1. Define a source for an external address. This can be either of: + 1. A user defined address, specified in the `external-status-address` ConfigMap key. + 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. + +View the [ConfigMap keys](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) and [Command-line arguments](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments) topics for more information. + +{{< note >}} NGINX Ingress Controller does not clear the status of Ingress resources when it is being shut down. {{< /note >}} + +## VirtualServer and VirtualServerRoute resources + +A VirtualServer or VirtualServerRoute resource includes the status field with information about the state of the resource and the IP address, through which the hosts of that resource are publicly accessible. + +You can see the status in the output of the `kubectl get virtualservers` or `kubectl get virtualserverroutes` commands as shown below: + +```shell +kubectl get virtualservers +``` +```text + NAME STATE HOST IP PORTS AGE + cafe Valid cafe.example.com 12.13.23.123 [80,443] 34s +``` + +To see an external hostname address associated with a VirtualServer resource, use the `-o wide` option: + +```shell +kubectl get virtualservers -o wide +``` +```text + NAME STATE HOST IP EXTERNALHOSTNAME PORTS AGE + cafe Valid cafe.example.com ae430f41a1a0042908655abcdefghijkl-12345678.eu-west-2.elb.amazonaws.com [80,443] 106s +``` + +{{< note >}} If there are multiple addresses, only the first one is shown. {{< /note >}} + +In order to see additional addresses or extra information about the `Status` of the resource, use the following command: + +```shell +kubectl describe virtualserver +``` +```text +... +Status: + External Endpoints: + Ip: 12.13.23.123 + Ports: [80,443] + Message: Configuration for cafe/cafe was added or updated + Reason: AddedOrUpdated + State: Valid +``` + +### Status specification + +The following fields are reported in both VirtualServer and VirtualServerRoute status: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +|*State* | Current state of the resource. Can be ``Valid``, ``Warning`` an ``Invalid``. For more information, refer to the ``message`` field. | *string* | +|*Reason* | The reason of the last update. | *string* | +|*Message* | Additional information about the state. | *string* | +|*ExternalEndpoints* | A list of external endpoints for which the hosts of the resource are publicly accessible. | *[externalEndpoint](#externalendpoint)* | +{{}} + +The *ReferencedBy* field is reported for the VirtualServerRoute status only: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +| *ReferencedBy* | The VirtualServer that references this VirtualServerRoute. Format as ``namespace/name`` | *string* | +{{}} + +### externalEndpoint + +{{}} +|Field | Description | Type | +| ---| ---| --- | +|``IP`` | The external IP address. | ``string`` | +|``Hostname`` | The external LoadBalancer Hostname address. | ``string`` | +|``Ports`` | A list of external ports. | ``string`` | +{{}} + +NGINX Ingress Controller must be configured to report a VirtualServer or VirtualServerRoute status: + +1. If you want NGINX Ingress Controller to report the `externalEndpoints`, define a source for an external address (The rest of the fields will be reported without the external address configured). This can be: + 1. A user defined address, specified in the `external-status-address` ConfigMap key. + 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. + +View the [ConfigMap keys](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) and [Command-line arguments](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments) topics for more information. + +{{< note >}} NGINX Ingress Controller does not clear the status of VirtualServer and VirtualServerRoute resources when it is being shut down. {{< /note >}} + +## Policy resources + +A Policy resource includes the status field with information about the state of the resource. + +You can see the status in the output of the `kubectl get policy` command as shown below: + +```shell +kubectl get policy +``` +```text + NAME STATE AGE + webapp-policy Valid 30s +``` + +In order to see additional addresses or extra information about the `Status` of the resource, use the following command: + +```shell +kubectl describe policy +``` +```text +... +Status: + Message: Configuration for default/webapp-policy was added or updated + Reason: AddedOrUpdated + State: Valid +``` + +### Status specification + +The following fields are reported in Policy status: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +|``State`` | Current state of the resource. Can be ``Valid`` or ``Invalid``. For more information, refer to the ``message`` field. | ``string`` | +|``Reason`` | The reason of the last update. | ``string`` | +|``Message`` | Additional information about the state. | ``string`` | +{{}} + +## TransportServer resources + +A TransportServer resource includes the status field with information about the state of the resource. + +You can see the status in the output of the `kubectl get transportserver` command as shown below: + +```shell +kubectl get transportserver +``` +```text + NAME STATE REASON AGE + dns-tcp Valid AddedOrUpdated 47m +``` + +To see additional addresses or extra information about the `Status` of the resource, use the following command: + +```shell +kubectl describe transportserver +``` +```text +Status: + Message: Configuration for default/dns-tcp was added or updated + Reason: AddedOrUpdated + State: Valid +``` + +### Status specification + +The following fields are reported in TransportServer status: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +| *State* | Current state of the resource. Can be ``Valid``, ``Warning`` or ``Invalid``. For more information, refer to the ``message`` field. | *string* | +| *Reason* | The reason of the last update. | *string* | +| *Message* | Additional information about the state. | *string* | +{{}} diff --git a/content/includes/nic/configuration/host-and-listener-collisions.md b/content/includes/nic/configuration/host-and-listener-collisions.md new file mode 100644 index 000000000..502ef830c --- /dev/null +++ b/content/includes/nic/configuration/host-and-listener-collisions.md @@ -0,0 +1,168 @@ +--- +title: Host and Listener collisions +toc: true +weight: 800 +docs: DOCS-590 +--- + +This document explains how F5 NGINX Ingress Controller handles host and listener collisions between resources. + +--- + +## Winner Selection Algorithm + +If multiple resources contend for the same host or listener, NGINX Ingress Controller will pick the winner based on the `creationTimestamp` of the resources: the oldest resource will win. In case there are more than one oldest resource (their `creationTimestamp` is the same), NGINX Ingress Controller will choose the resource with the lexicographically smallest `uid`. + +{{< note >}} The `creationTimestamp` and `uid` fields are part of the [ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/) resource. {{< /note >}} + +--- + +## Host collisions + +A host collision occurs when multiple Ingress, VirtualServer, and TransportServer (configured for TLS Passthrough) resources configure the same `host`. NGINX Ingress Controller has two strategies for handling host collisions: + +- Choosing a single "winner" resource to handle the host. +- Merging the configuration of the conflicting resources. + +--- + +### Choosing the winner + +Consider the following two resources: + +- `cafe-ingress` Ingress: + + ```yaml + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: cafe-ingress + spec: + ingressClassName: nginx + rules: + - host: cafe.example.com + . . . + ``` + +- `cafe-virtual-server` VirtualServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: VirtualServer + metadata: + name: cafe-virtual-server + spec: + host: cafe.example.com + . . . + ``` + +If a user creates both resources in the cluster, a host collision will occur. NGINX Ingress Controller will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). + +If `cafe-virtual-server` was created first, it will win the host `cafe.example.com` and NGINX Ingress Controller will reject `cafe-ingress`. This will be reflected in the events and in the resource's status field: + +```shell +kubectl describe vs cafe-virtual-server +``` +```text +... +Status: + ... + Message: Configuration for default/cafe-virtual-server was added or updated + Reason: AddedOrUpdated + State: Valid +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 9s nginx-ingress-controller Configuration for default/cafe-virtual-server was added or updated +``` + +```shell +kubectl describe ingress cafe-ingress +``` +```text +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 66s nginx-ingress-controller All hosts are taken by other resources +``` + +Similarly, if `cafe-ingress` was created first, it will win `cafe.example.com` and NGINX Ingress Controller will reject `cafe-virtual-server`. + +{{< note >}} You can configure multiple hosts for Ingress resources, and its possible that an Ingress resource can be the winner for some of its hosts and a loser for the others. + +For example, if `cafe-ingress` had an additional rule host rule for `pub.example.com`, NGINX Ingress Controller would not reject the Ingress. Instead, it would allow `cafe-ingress` to handle `pub.example.com`. {{< /note >}} + +--- + +### Merging configuration for the same host + +It is possible to merge configuration for multiple Ingress resources for the same host. One common use case for this approach is distributing resources across multiple namespaces. + +The [Cross-namespace configuration]({{< ref "/nic/configuration/ingress-resources/cross-namespace-configuration.md">}}) topic has more information. + +It is *not* possible to merge the configurations for multiple VirtualServer resources for the same host. However, you can split the VirtualServers into multiple VirtualServerRoute resources, which a single VirtualServer can then reference. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/cross-namespace-configuration) on GitHub. + +It is *not* possible to merge configuration for multiple TransportServer resources. + +--- + +## Listener/Host collisions + +Listener/Host collisions occur when multiple TransportServer resources (configured for TCP/UDP load balancing) specify the same combination of `spec.listener.name` and `spec.host`. + +The combination of `spec.listener.name` and `spec.host` must be unique among all TransportServer resources. If two TransportServer resources specify the same spec.listener.name and spec.host, one of them will be rejected to prevent conflicts. In the case where spec.host is not specified, it is considered an empty string. + +NGINX Ingress Controller will choose the winner, which will own that listener and host combination. + +--- + +### Choosing the winner + +Consider the following two resources: + +- `tcp-1` TransportServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: tcp-1 + spec: + host: dns.example.com + listener: + name: dns-tcp + protocol: TCP + . . . + ``` + +- `tcp-2` TransportServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: tcp-2 + spec: + host: dns.example.com + listener: + name: dns-tcp + protocol: TCP + . . . + ``` + +If a user creates both resources in the cluster, a listener collision will occur. As a result, NGINX Ingress Controller will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). + +In our example, if `tcp-1` was created first, it will win the listener `dns-tcp` and NGINX Ingress Controller will reject `tcp-2`. This will be reflected in the events and in the resource's status field: + +```shell +kubectl describe ts tcp-2 +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 10s nginx-ingress-controller Listener dns-tcp is taken by another resource +``` + +Similarly, if `tcp-2` was created first, it will win `dns-tcp` and NGINX Ingress Controller will reject `tcp-1`. diff --git a/content/includes/nic/configuration/ingress-resources/_index.md b/content/includes/nic/configuration/ingress-resources/_index.md new file mode 100644 index 000000000..8a2b5b282 --- /dev/null +++ b/content/includes/nic/configuration/ingress-resources/_index.md @@ -0,0 +1,8 @@ +--- +title: Ingress resources +description: +weight: 200 +menu: + docs: + parent: NGINX Ingress Controller +--- diff --git a/content/includes/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md b/content/includes/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md new file mode 100644 index 000000000..ecc12435f --- /dev/null +++ b/content/includes/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md @@ -0,0 +1,225 @@ +--- +docs: DOCS-591 +doctypes: +- '' +title: Advanced configuration with Annotations +toc: true +weight: 200 +--- + +This topic explains how to enable advanced features in F5 NGINX Ingress Controller with Annotations. + +The Ingress resource can use basic NGINX features such as host or path-based routing and TLS termination. Advanced features like rewriting the request URI or inserting additional response headers can be enabled with Annotations. + +Outside of advanced features, Annotations are necessary for customizing NGINX behavior such as setting the value of connection timeouts. + +Customization is also available through the [ConfigMap]({{< relref "/configuration/global-configuration/configmap-resource.md" >}}) resources: Annotations take priority. + +## Using Annotations + +This example uses Annotations to customize the configuration for an Ingress resource: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: cafe-ingress-with-annotations + annotations: + nginx.org/proxy-connect-timeout: "30s" + nginx.org/proxy-read-timeout: "20s" + nginx.org/client-max-body-size: "4m" + nginx.org/server-snippets: | + location / { + return 302 /coffee; + } +spec: + rules: + - host: cafe.example.com + http: + paths: + - path: /tea + pathType: Prefix + backend: + service: + name: tea-svc + port: + number: 80 + - path: /coffee + pathType: Prefix + backend: + service: + name: coffee-svc + port: + number: 80 +``` + +## Validation + +NGINX Ingress Controller validates the annotations of Ingress resources. If an Ingress is invalid, NGINX Ingress Controller will reject it: the Ingress will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can check if NGINX Ingress Controller successfully applied the configuration for an Ingress resource. For the example `cafe-ingress-with-annotations` Ingress, you can run: + +```shell +kubectl describe ing cafe-ingress-with-annotations +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/cafe-ingress-with-annotations was added or updated +``` + +The events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid Ingress, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create an Ingress `cafe-ingress-with-annotations`, with an annotation `nginx.org/redirect-to-https` set to `yes please` instead of `true`, you will get: + +```shell +kubectl describe ing cafe-ingress-with-annotations +``` +```text +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 13s nginx-ingress-controller annotations.nginx.org/redirect-to-https: Invalid value: "yes please": must be a boolean +``` + +Note how the events section includes a Warning event with the Rejected reason. + +{{< note >}} If you make an existing Ingress invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. {{< /note >}} + +The `nginx.com/jwt-token` Ingress annotation has limited validation. + +## Summary of Annotations + +The table below summarizes the available annotations. + +{{< note >}} Annotations that start with `nginx.com` are only supported with NGINX Plus. {{< /note >}} + +### General customization + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/proxy-connect-timeout* | *proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | +| *nginx.org/proxy-read-timeout* | *proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | +| *nginx.org/proxy-send-timeout* | *proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | +| *nginx.org/client-max-body-size* | *client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | +| *nginx.org/proxy-buffering* | *proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | +| *nginx.org/proxy-buffers* | *proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | +| *nginx.org/proxy-buffer-size* | *proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | +| *nginx.org/proxy-max-temp-file-size* | *proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | +| *nginx.org/server-tokens* | *server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. Additionally, with the NGINX Plus, you can specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | +| *nginx.org/path-regex* | N/A | Enables regular expression modifiers for Ingress path parameter. This translates to the NGINX [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive. You can specify one of these values: "case_sensitive", "case_insensitive", or "exact". The annotation is applied to the entire Ingress resource and its paths. While using Master and Minion Ingresses i.e. Mergeable Ingresses, this annotation can be specified on Minion types. The `path-regex` annotation specified on Master is ignored, and has no effect on paths defined on Minions. | N/A | [path-regex](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/path-regex) | +{{}} + +### Request URI/Header Manipulation + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/proxy-hide-headers* | *proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: ``"nginx.org/proxy-hide-headers": "header-a,header-b"* | N/A | | +| *nginx.org/proxy-pass-headers* | *proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: ``"nginx.org/proxy-pass-headers": "header-a,header-b"* | N/A | | +| *nginx.org/rewrites* | N/A | Configures URI rewriting using [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive. | N/A | [rewrites](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/rewrites) | +|*nginx.org/proxy-set-headers* | N/A | Enables customization of proxy headers and values using the [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. Example: ``"nginx.org/proxy-set-headers": "header-a: valueA,header-b: valueB,header-c: valueC"`` | N/A | [Proxy Set Headers](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/proxy-set-headers). | +{{}} + +### Auth and SSL/TLS + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/redirect-to-https* | *redirect-to-https* | Sets the 301 redirect rule based on the value of the ``http_x_forwarded_proto* header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of NGINX Ingress Controller — see [115](https://github.com/nginx/kubernetes-ingress/issues/115) | *False* | | +| *ingress.kubernetes.io/ssl-redirect* | *ssl-redirect* | Sets an unconditional 301 redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS. | *True* | | +| *nginx.org/hsts* | *hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/)\ : the HSTS header is added to the responses from backends. The ``preload* directive is included in the header. | *False* | | +| *nginx.org/hsts-max-age* | *hsts-max-age* | Sets the value of the ``max-age* directive of the HSTS header. | *2592000* (1 month) | | +| *nginx.org/hsts-include-subdomains* | *hsts-include-subdomains* | Adds the ``includeSubDomains* directive to the HSTS header. | *False* | | +| *nginx.org/hsts-behind-proxy* | *hsts-behind-proxy* | Enables HSTS based on the value of the ``http_x_forwarded_proto* request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of NGINX Ingress Controller. Note: to control redirection from HTTP to HTTPS configure the ``nginx.org/redirect-to-https* annotation. | *False* | | +| *nginx.org/basic-auth-secret* | N/A | Specifies a Secret resource with a user list for HTTP Basic authentication. | N/A | | +| *nginx.org/basic-auth-realm* | N/A | Specifies a realm. | N/A | | +| *nginx.com/jwt-key* | N/A | Specifies a Secret resource with keys for validating JSON Web Tokens (JWTs). | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +| *nginx.com/jwt-realm* | N/A | Specifies a realm. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +| *nginx.com/jwt-token* | N/A | Specifies a variable that contains a JSON Web Token. | By default, a JWT is expected in the ``Authorization* header as a Bearer Token. | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +| *nginx.com/jwt-login-url* | N/A | Specifies a URL to which a client is redirected in case of an invalid or missing JWT. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +{{}} + +### Listeners + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/listen-ports* | N/A | Configures HTTP ports that NGINX will listen on. | *[80]* | | +| *nginx.org/listen-ports-ssl* | N/A | Configures HTTPS ports that NGINX will listen on. | *[443]* | | +{{}} + +### Backend services (Upstreams) + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/lb-method* | *lb-method* | Sets the [load balancing method](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method). To use the round-robin method, specify ``"round_robin"``. | *"random two least_conn"* | | +| *nginx.org/ssl-services* | N/A | Enables HTTPS or gRPC over SSL when connecting to the endpoints of services. | N/A | [ssl-services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/ssl-services) | +| *nginx.org/grpc-services* | N/A | Enables gRPC for services. Note: requires HTTP/2 (see ``http2* ConfigMap key); only works for Ingresses with TLS termination enabled. | N/A | [grpc-services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/grpc-services) | +| *nginx.org/websocket-services* | N/A | Enables WebSocket for services. | N/A | [websocket](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/websocket) | +| *nginx.org/max-fails* | *max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the ``server* directive. | *1* | | +| *nginx.org/max-conns* | N\A | Sets the value of the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the ``server* directive. | *0* | | +| *nginx.org/upstream-zone-size* | *upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. For NGINX, the special value 0 disables the shared memory zones. For NGINX Plus, shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *256K* | | +| *nginx.org/fail-timeout* | *fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the ``server* directive. | *10s* | | +| *nginx.com/sticky-cookie-services* | N/A | Configures session persistence. | N/A | [session-persistence](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/session-persistence) | +| *nginx.org/keepalive* | *keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that ``proxy_set_header Connection "";* is added to the generated configuration when the value > 0. | *0* | | +| *nginx.com/health-checks* | N/A | Enables active health checks. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks) | +| *nginx.com/health-checks-mandatory* | N/A | Configures active health checks as mandatory. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks) | +| *nginx.com/health-checks-mandatory-queue* | N/A | When active health checks are mandatory, creates a queue where incoming requests are temporarily stored while NGINX Plus is checking the health of the endpoints after a configuration reload. | *0* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks) | +| *nginx.com/slow-start* | N/A | Sets the upstream server [slow-start period](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#server-slow-start). By default, slow-start is activated after a server becomes [available](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#passive-health-checks) or [healthy](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#active-health-checks). To enable slow-start for newly-added servers, configure [mandatory active health checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks). | *"0s"* | | +| *nginx.org/use-cluster-ip* | N/A | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method* and ``next-upstream``) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | *False* | | +{{}} + +### Rate limiting + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/limit-req-rate* | N/A | Enables request-rate-limiting for this ingress by creating a [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) and matching [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) for each location. All servers/locations of one ingress share the same zone. Must have unit r/s or r/m. | N/A | 200r/s | +| *nginx.org/limit-req-key* | N/A | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ${}. | ${binary_remote_addr} | ${binary_remote_addr} | +| *nginx.org/limit-req-zone-size* | N/A | Configures the size of the created [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone). | 10m | 20m | +| *nginx.org/limit-req-delay* | N/A | Configures the delay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | 0 | 100 | +| *nginx.org/limit-req-no-delay* | N/A | Configures the nodelay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | false | true | +| *nginx.org/limit-req-burst* | N/A | Configures the burst-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | N/A | 100 | +| *nginx.org/limit-req-dry-run* | N/A | Enables the dry run mode. In this mode, the rate limit is not actually applied, but the number of excessive requests is accounted as usual in the shared memory zone. | false | true | +| *nginx.org/limit-req-log-level* | N/A | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are info, notice, warn or error. | error | info | +| *nginx.org/limit-req-reject-code* | N/A | Sets the status code to return in response to rejected requests. Must fall into the range 400..599. | 429 | 503 | +| *nginx.org/limit-req-scale* | N/A | Enables a constant rate-limit by dividing the configured rate by the number of nginx-ingress pods currently serving traffic. This adjustment ensures that the rate-limit remains consistent, even as the number of nginx-pods fluctuates due to autoscaling. Note: This will not work properly if requests from a client are not evenly distributed accross all ingress pods (sticky sessions, long lived TCP-Connections with many requests etc.). In such cases using [zone-sync]({{< ref "/configuration/global-configuration/configmap-resource.md#zone-sync" >}}) instead would give better results. Enabling `zone-sync` will suppress this setting. | false | true | +{{}} + +### Snippets and custom templates + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/location-snippets* | *location-snippets* | Sets a custom snippet in location context. | N/A | | +| *nginx.org/server-snippets* | *server-snippets* | Sets a custom snippet in server context. | N/A | | +{{}} + +### App Protect WAF {#app-protect} + +{{< note >}} The App Protect annotations only work if the App Protect WAF module is [installed]({{< relref "installation/integrations/app-protect-waf/installation.md" >}}). {{< /note >}} + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *appprotect.f5.com/app-protect-policy* | N/A | The name of the App Protect Policy for the Ingress Resource. Format is ``namespace/name``. If no namespace is specified, the same namespace of the Ingress Resource is used. If not specified but ``appprotect.f5.com/app-protect-enable* is true, a default policy id applied. If the referenced policy resource does not exist, or policy is invalid, this annotation will be ignored, and the default policy will be applied. | N/A | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-enable* | N/A | Enable App Protect for the Ingress Resource. | *False* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-security-log-enable* | N/A | Enable the [security log](/nginx-app-protect/troubleshooting/#app-protect-logging-overview) for App Protect. | *False* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-security-log* | N/A | The App Protect log configuration for the Ingress Resource. Format is ``namespace/name``. If no namespace is specified, the same namespace as the Ingress Resource is used. If not specified the default is used which is: filter: ``illegal``, format: ``default``. Multiple configurations can be specified in a comma separated list. Both log configurations and destinations list (see below) must be of equal length. Configs and destinations are paired by the list indices. | N/A | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-security-log-destination* | N/A | The destination of the security log. For more information check the [DESTINATION argument](/nginx-app-protect/troubleshooting/#app-protect-logging-overview). Multiple destinations can be specified in a comma-separated list. Both log configurations and destinations list (see above) must be of equal length. Configs and destinations are paired by the list indices. | *syslog:server=localhost:514* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +{{}} + +### App Protect DoS + +{{< note >}} The App Protect DoS annotations only work if the App Protect DoS module is [installed]({{< relref "installation/integrations/app-protect-dos/installation.md" >}}). {{< /note >}} + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *appprotectdos.f5.com/app-protect-dos-resource* | N/A | Enable App Protect DoS for the Ingress Resource by specifying a [DosProtectedResource]({{< relref "installation/integrations/app-protect-dos/dos-protected.md" >}}). | N/A | [app-protect-dos](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-dos) | +{{}} diff --git a/content/includes/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md b/content/includes/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md new file mode 100644 index 000000000..642d041ae --- /dev/null +++ b/content/includes/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md @@ -0,0 +1,128 @@ +--- +docs: DOCS-592 +doctypes: +- '' +title: Advanced configuration with Snippets +toc: true +weight: 400 +--- + +Snippets allow you to insert raw NGINX config into different contexts of the NGINX configurations that F5 NGINX Ingress Controller generates. + +Snippets are intended for advanced NGINX users who need more control over the generated NGINX configuration, and can be used in cases where Annotations and ConfigMap entries would not apply. + + + +## Disadvantages of snippets + +Snippets are configured [using Annotations]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#snippets-and-custom-templates" >}}), but are disabled by default due to their complexity. They are also available through the [ConfigMap]({{< relref "/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates" >}}) resource. + +To use snippets, set the [`enable-snippets`]({{< ref "/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-snippets" >}}) command-line argument. + +Snippets have the following disadvantages: + +- *Complexity*. Snippets require you to: + - Understand NGINX configuration primitives and implement a correct NGINX configuration. + - Understand how NGINX Ingress Controller generates NGINX configuration so that a snippet doesn't interfere with the other features in the configuration. +- *Decreased robustness*. An incorrect snippet can invalidate NGINX configuration, causing reload failures. Until the snippet is fixed, it will prevent any new configuration updates, including updates for the other Ingress resources. +- *Security implications*. Snippets give access to NGINX configuration primitives, which are not validated by NGINX Ingress Controller. For example, a snippet can configure NGINX to serve the TLS certificates and keys used for TLS termination for Ingress resources. + +{{< note >}} If the NGINX configuration includes an invalid snippet, NGINX will continue to operate with the last valid configuration. {{< /note >}} + +## Using snippets + +The example below shows how to use snippets to customize the NGINX configuration template using annotations. + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: cafe-ingress-with-snippets + annotations: + nginx.org/server-snippets: | + location / { + return 302 /coffee; + } + nginx.org/location-snippets: | + add_header my-test-header test-value; +spec: + rules: + - host: cafe.example.com + http: + paths: + - path: /tea + pathType: Prefix + backend: + service: + name: tea-svc + port: + number: 80 + - path: /coffee + pathType: Prefix + backend: + service: + name: coffee-svc + port: + number: 80 +``` + +These snippets generate the following NGINX configuration: + +{{< note >}} The example is shortened for conciseness. {{< /note >}} + +```nginx +server { + listen 80; + + + location / { + return 302 /coffee; + } + + + location /coffee { + proxy_http_version 1.1; + + + add_header my-test-header test-value; + ... + proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-coffee-svc-80; + } + + location /tea { + proxy_http_version 1.1; + + add_header my-test-header test-value; + ... + proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-tea-svc-80; + } +} +``` + +## Troubleshooting + +If a snippet includes an invalid NGINX configuration, NGINX Ingress Controller will fail to reload NGINX. The error will be reported in NGINX Ingress Controller logs and an event with the error will be associated with the Ingress resource: + +An example of an error from the logs: + +```text +[emerg] 31#31: unknown directive "badd_header" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54 +Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cafe-ingress-with-snippets", UID:"f9656dc9-63a6-41dd-a499-525b0e0309bb", APIVersion:"extensions/v1beta1", ResourceVersion:"2322030", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithError' Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" +stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" +finished with error: exit status 1 +``` + +An example of an event with an error (you can view events associated with the Ingress by running `kubectl describe -n nginx-ingress ingress nginx-ingress`): + +```text +Events: +Type Reason Age From Message +---- ------ ---- ---- ------- +Normal AddedOrUpdated 52m (x3 over 61m) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated +finished with error: exit status 1 +Warning AddedOrUpdatedWithError 54s (x2 over 89s) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" +stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" +finished with error: exit status 1 +``` + +Additionally, to help troubleshoot snippets, a number of Prometheus metrics show the stats about failed reloads – `controller_nginx_last_reload_status` and `controller_nginx_reload_errors_total`. diff --git a/content/includes/nic/configuration/ingress-resources/basic-configuration.md b/content/includes/nic/configuration/ingress-resources/basic-configuration.md new file mode 100644 index 000000000..bd2c10d26 --- /dev/null +++ b/content/includes/nic/configuration/ingress-resources/basic-configuration.md @@ -0,0 +1,107 @@ +--- +title: Basic configuration +weight: 100 +toc: true +type: reference +product: NIC +docs: DOCS-593 +--- + +This document shows a basic Ingress resource definition for F5 NGINX Ingress Controller. It load balances requests for two services as part of a single application. + +{{< ghcode `https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/ingress-resources/complete-example/cafe-ingress.yaml`>}} + +Here is a breakdown of what this Ingress resource definition means: + +- The `metadata.name` field defines the name of the resource `cafe‑ingress`. +- The `spec.tls` field sets up SSL/TLS termination: + - The `hosts` field applies the certificate and key to the `cafe.example.com` host. + - The `secretName` references a secret resource by its name, `cafe‑secret`. The secret must belong to the same namespace as the Ingress, of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that hold the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls>). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the hosts to which the secret is applied. +- The `spec.rules` field defines a host with the domain name `cafe.example.com`. +- The `paths` field defines two path‑based rules: + - The rule with the path `/tea` instructs NGINX to distribute the requests with the `/tea` URI among the pods of the *tea* service, which is deployed with the name `tea‑svc` in the cluster. + - The rule with the path `/coffee` instructs NGINX to distribute the requests with the `/coffee` URI among the pods of the *coffee* service, which is deployed with the name `coffee‑svc` in the cluster. + - Both rules instruct NGINX to distribute the requests to `port 80` of the corresponding service (the `servicePort` field). + +To learn more about the Ingress resource, view [the official Kubernetes documentation for Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/). + +{{< note >}} For complete instructions on deploying Ingress and Secret resources in the cluster, see the [complete example](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/complete-example) in the GitHub repository. {{< /note >}} + +--- + +## New features available in Kubernetes 1.18 + +Starting from Kubernetes 1.18, you can use the following new features: + +- The host field supports wildcard domain names, such as `*.example.com`. +- The path supports different matching rules with the new field `pathType`, which takes the following values: `Prefix` for prefix-based matching, `Exact` for exact matching and `ImplementationSpecific`, which is the default type and is the same as `Prefix`. For example: + + ```yaml {hl_lines=[2, 7, 14]} + - path: /tea + pathType: Prefix + backend: + serviceName: tea-svc + servicePort: 80 + - path: /tea/green + pathType: Exact + backend: + service: + name: tea-svc + port: + number: 80 + - path: /coffee + pathType: ImplementationSpecific + backend: + service: + name: coffee-svc + port: + number: 80 + ``` + +- The `ingressClassName` field is now supported: + + ```yaml {hl_lines=[6]} + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: cafe-ingress + spec: + ingressClassName: nginx + tls: + - hosts: + - cafe.example.com + secretName: cafe-secret + rules: + - host: cafe.example.com + . . . + ``` + + When using this field you need to create the `IngressClass` resource with the corresponding `name`. View the [Create common resources]({{< relref "installation/installing-nic/installation-with-manifests.md#create-common-resources" >}}) section of the Installation with Manifests topic for more information. + +--- + +## Restrictions + +NGINX Ingress Controller imposes the following restrictions on Ingress resources: + +- When defining an Ingress resource, the `host` field is required. +- The `host` value needs to be unique among all Ingress and VirtualServer resources unless the Ingress resource is a [mergeable minion]({{< ref "/nic/configuration/ingress-resources/cross-namespace-configuration.md" >}}). View the [Host and Listener collisions]({{< ref "/nic/configuration/host-and-listener-collisions.md" >}}) topic for more information. +- The `path` field in `spec.rules[].http.paths[]` is required for `Exact` and `Prefix` `pathTypes`. +- The ImplementationSpecific `pathType` is treated as equivalent to `Prefix` `pathType`, with the exception that when this `pathType` is configured, the `path` field in `spec.rules[].http.paths[]` is not mandatory. `path` defaults to `/` if not set but the `pathType` is set to ImplementationSpecific. + +--- + +## Advanced configuration + +NGINX Ingress Controller generates NGINX configuration by executing a template file that contains the configuration options. + +These options are set with the Ingress resource and NGINX Ingress Controller's ConfigMap. + +The Ingress resource only allows you to use basic NGINX features: host and path-based routing and TLS termination. + +For advanced configuration, you have two options: + +- [Annotations]({{< ref "/configuration/ingress-resources/advanced-configuration-with-annotations.md" >}}) can be used to rewrite request URIs or inserting additional response headers. +- [Snippets]({{< ref "/configuration/ingress-resources/advanced-configuration-with-snippets" >}}) can be used to insert raw NGINX configuration, changing generated files. + +Additionally, it is possible to customize the template, described in the [Custom templates]({{< relref "/configuration/global-configuration/custom-templates.md" >}}) topic. diff --git a/content/includes/nic/configuration/ingress-resources/cross-namespace-configuration.md b/content/includes/nic/configuration/ingress-resources/cross-namespace-configuration.md new file mode 100644 index 000000000..3e92b6403 --- /dev/null +++ b/content/includes/nic/configuration/ingress-resources/cross-namespace-configuration.md @@ -0,0 +1,14 @@ +--- +docs: DOCS-594 +doctypes: +- '' +title: Cross-namespace configuration +toc: true +weight: 500 +--- + +This topic explains how to spread Ingress configuration across different namespaces in F5 NGINX Ingress Controller. + +You can spread the Ingress configuration for a common host across multiple Ingress resources using Mergeable Ingress resources. Such resources can belong to the *same* or *different* namespaces. This enables easier management when using a large number of paths. See the [Mergeable Ingress Resources](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/mergeable-ingress-types) example in our GitHub repo. + +As an alternative to Mergeable Ingress resources, you can use [VirtualServer and VirtualServerRoute resources](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/) for cross-namespace configuration. See the [Cross-Namespace Configuration](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/cross-namespace-configuration) example in our GitHub repo. diff --git a/content/includes/nic/configuration/ingress-resources/custom-annotations.md b/content/includes/nic/configuration/ingress-resources/custom-annotations.md new file mode 100644 index 000000000..8bbd136b0 --- /dev/null +++ b/content/includes/nic/configuration/ingress-resources/custom-annotations.md @@ -0,0 +1,146 @@ +--- +docs: DOCS-595 +doctypes: +- '' +title: Custom annotations +toc: true +weight: 300 +--- + +This topic explains how you can use custom annotations with F5 NGINX Ingress Controller. + +Custom annotations enable you to quickly extend the Ingress resource to support many advanced features of NGINX, such as rate limiting, caching, etc. + +## Overview + +NGINX Ingress Controller supports a number of annotations for the Ingress resource that fine tune NGINX configuration (for example, connection timeouts) or enable additional features (for example, JWT validation). The complete list of annotations is available [here](/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations). + +The annotations are provided only for the most common features and use cases, meaning that not every NGINX feature or a customization option is available through the annotations. Additionally, even if an annotation is available, it might not give you the satisfactory level of control of a particular NGINX feature. + +Custom annotations allow you to add an annotation for an NGINX feature that is not available as a regular annotation. In contrast with regular annotations, to add a custom annotation, you don't need to modify the Ingress Controller source code -- just modify the template. Additionally, with a custom annotation, you get full control of how the feature is implemented in NGINX configuration. + +## Usage + +The Ingress Controller generates NGINX configuration for Ingress resources by executing a configuration template. See [NGINX template](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx.ingress.tmpl) or [NGINX Plus template](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx-plus.ingress.tmpl). + +To support custom annotations, the template has access to the information about the Ingress resource - its *name*, *namespace* and *annotations*. It is possible to check if a particular annotation present in the Ingress resource and conditionally insert NGINX configuration directives at multiple NGINX contexts - `http`, `server`, `location` or `upstream`. Additionally, you can get the value that is set to the annotation. + +Consider the following excerpt from the template, which was extended to support two custom annotations: + +```jinja2 +# This is the configuration for {{$.Ingress.Name}}/{{$.Ingress.Namespace}} + +{{if index $.Ingress.Annotations "custom.nginx.org/feature-a"}} +# Insert config for feature A if the annotation is set +{{end}} + +{{with $value := index $.Ingress.Annotations "custom.nginx.org/feature-b"}} +# Insert config for feature B if the annotation is set +# Print the value assigned to the annotation: {{$value}} +{{end}} +``` + +Consider the following Ingress resource and note how we set two annotations: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: example-ingress + namespace: production + annotations: + custom.nginx.org/feature-a: "on" + custom.nginx.org/feature-b: "512" +spec: + rules: + - host: example.com + . . . +``` + +Assuming that the Ingress Controller is using that customized template, it will generate a config for the Ingress resource that will include the following part, generated by our template excerpt: + +```yaml +# This is the configuration for cafe-ingress/default + +# Insert config for feature A if the annotation is set + + + +# Insert config for feature B if the annotation is set +# Print the value assigned to the annotation: 512 +``` + +**Notes**: + +- You can customize the template to insert you custom annotations via [custom templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). +- The Ingress Controller uses go templates to generate NGINX config. You can read more information about go templates [here](https://golang.org/pkg/text/template/). + +See the examples in the next section that use custom annotations to configure NGINX features. + +### Custom Annotations with Mergeable Ingress Resources + +A Mergeable Ingress resource consists of multiple Ingress resources - one master and one or several minions. Read more about Mergeable Ingress resources [here](/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration). + +If you'd like to use custom annotations with Mergeable Ingress resources, please keep the following in mind: + +- Custom annotations can be used in the Master and in Minions. For Minions, you can access them in the template only when processing locations. + + If you access `$.Ingress` anywhere in the Ingress template, you will get the master Ingress resource. To access a Minion Ingress resource, use `$location.MinionIngress`. However, it is only available when processing locations: + + ```jinja2 + {{range $location := $server.Locations}} + location {{$location.Path}} { + {{with $location.MinionIngress}} + # location for minion {{$location.MinionIngress.Namespace}}/{{$location.MinionIngress.Name}} + {{end}} + } {{end}} + ``` + + **Note**: `$location.MinionIngress` is a pointer. When a regular Ingress resource is processed in the template, the value of the pointer is `nil`. Thus, it is important that you check that `$location.MinionIngress` is not `nil` as in the example above using the `with` action. + +- Minions do not inherent custom annotations of the master. + +### Helper Functions + +Helper functions can be used in the Ingress template to parse the values of custom annotations. + +{{% table %}} +| Function | Input Arguments | Return Arguments | Description | +| ---| ---| ---| --- | +| ``split`` | ``s, sep string`` | ``[]string`` | Splits the string ``s`` into a slice of strings separated by the ``sep``. | +| ``trim`` | ``s string`` | ``string`` | Trims the trailing and leading whitespace from the string ``s``. | +| ``contains`` | ``s, substr string`` | ``bool`` | Tests whether the string ``substr`` is a substring of the string ``s``. | +| ``hasPrefix`` | ``s, prefix string`` | ``bool`` | Tests whether the string ``prefix`` is a prefix of the string ``s``. | +| ``hasSuffix`` | ``s, suffix string`` | ``bool`` | Tests whether the string ``suffix`` is a suffix of the string ``s``. | +| ``toLower`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their lower case. | +| ``toUpper`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their upper case. | +| ``replaceAll`` | ``s, old, new string`` | ``string`` | Replaces all occurrences of ``old`` with ``new`` in the string ``s``. | +{{% /table %}} + +Consider the following custom annotation `custom.nginx.org/allowed-ips`, which expects a comma-separated list of IP addresses: + +```yaml +annotations: + custom.nginx.org/allowed-ips: "192.168.1.3, 10.0.0.13" +``` + + The helper functions can parse the value of the `custom.nginx.org/allowed-ips` annotation, so that in the template you can use each IP address separately. Consider the following template excerpt: + +```jinja2 +{{range $ip := split (index $.Ingress.Annotations "custom.nginx.org/allowed-ips") ","}} + allow {{trim $ip}}; +{{end}} +deny all; +``` + +The template excerpt will generate the following configuration: + +``` +allow 192.168.1.3; +allow 10.0.0.13; +deny all; +``` + +## Example + +See the [custom annotations example](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/examples/ingress-resources/custom-annotations). diff --git a/content/includes/nic/configuration/policy-resource.md b/content/includes/nic/configuration/policy-resource.md new file mode 100644 index 000000000..6ea2c8121 --- /dev/null +++ b/content/includes/nic/configuration/policy-resource.md @@ -0,0 +1,945 @@ +--- +title: Policy resources +weight: 500 +toc: true +type: how-to +product: NIC +docs: DOCS-596 +--- + +The Policy resource allows you to configure features like access control and rate-limiting, which you can add to your [VirtualServer and VirtualServerRoute resources](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/). + +The resource is implemented as a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +This document is the reference documentation for the Policy resource. An example of a Policy for access control is available in our [GitHub repository](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/examples/custom-resources/access-control). + +## Prerequisites + +Policies work together with [VirtualServer and VirtualServerRoute resources](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/), which you need to create separately. + +## Policy Specification + +Below is an example of a policy that allows access for clients from the subnet `10.0.0.0/8` and denies access for any other clients: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: Policy +metadata: + name: allow-localhost +spec: + accessControl: + allow: + - 10.0.0.0/8 +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``accessControl`` | The access control policy based on the client IP address. | [accessControl](#accesscontrol) | No | +|``ingressClassName`` | Specifies which instance of NGINX Ingress Controller must handle the Policy resource. | ``string`` | No | +|``rateLimit`` | The rate limit policy controls the rate of processing requests per a defined key. | [rateLimit](#ratelimit) | No | +|``apiKey`` | The API Key policy configures NGINX to authorize requests which provide a valid API Key in a specified header or query param. | [apiKey](#apikey) | No | +|``basicAuth`` | The basic auth policy configures NGINX to authenticate client requests using HTTP Basic authentication credentials. | [basicAuth](#basicauth) | No | +|``jwt`` | The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. | [jwt](#jwt) | No | +|``ingressMTLS`` | The IngressMTLS policy configures client certificate verification. | [ingressMTLS](#ingressmtls) | No | +|``egressMTLS`` | The EgressMTLS policy configures upstreams authentication and certificate verification. | [egressMTLS](#egressmtls) | No | +|``waf`` | The WAF policy configures WAF and log configuration policies for [NGINX AppProtect]({{< relref "installation/integrations/app-protect-waf/configuration.md" >}}) | [WAF](#waf) | No | +{{% /table %}} + +\* A policy must include exactly one policy. + +### AccessControl + +The access control policy configures NGINX to deny or allow requests from clients with the specified IP addresses/subnets. + +For example, the following policy allows access for clients from the subnet `10.0.0.0/8` and denies access for any other clients: + +```yaml +accessControl: + allow: + - 10.0.0.0/8 +``` + +In contrast, the policy below does the opposite: denies access for clients from `10.0.0.0/8` and allows access for any other clients: + +```yaml +accessControl: + deny: + - 10.0.0.0/8 +``` +{{< note >}} + +The feature is implemented using the NGINX [ngx_http_access_module](http://nginx.org/en/docs/http/ngx_http_access_module.html). NGINX Ingress Controller access control policy supports either allow or deny rules, but not both (as the module does). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``allow`` | Allows access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | +|``deny`` | Denies access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | \* an accessControl must include either `allow` or `deny`. | +{{% /table %}} + +#### AccessControl Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple access control policies. For example, here we reference two policies, each with configured allow lists: + +```yaml +policies: +- name: allow-policy-one +- name: allow-policy-two +``` + +When you reference more than one access control policy, NGINX Ingress Controller will merge the contents into a single allow list or a single deny list. + +Referencing both allow and deny policies, as shown in the example below, is not supported. If both allow and deny lists are referenced, NGINX Ingress Controller uses just the allow list policies. + +```yaml +policies: +- name: deny-policy +- name: allow-policy-one +- name: allow-policy-two +``` + +### RateLimit + +The rate limit policy configures NGINX to limit the processing rate of requests. + +For example, the following policy will limit all subsequent requests coming from a single IP address once a rate of 10 requests per second is exceeded: + +```yaml +rateLimit: + rate: 10r/s + zoneSize: 10M + key: ${binary_remote_addr} +``` +{{< note >}} + +The feature is implemented using the NGINX [ngx_http_limit_req_module](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html). + +{{< /note >}} + +{{< note >}} + +When the [Zone Sync feature]({{< ref "/configuration/global-configuration/configmap-resource.md#zone-sync" >}}) is enabled with NGINX Plus, the rate limiting zone will be synchronized across all replicas in the cluster. This means all replicas are aware of the requests that have been rate limited by other replicas in the cluster. + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``rate`` | The rate of requests permitted. The rate is specified in requests per second (r/s) or requests per minute (r/m). | ``string`` | Yes | +|``key`` | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ``${}``. For example: ``${binary_remote_addr}``. Accepted variables are ``$binary_remote_addr``, ``$request_uri``,``$request_method``, ``$url``, ``$http_``, ``$args``, ``$arg_``, ``$cookie_``, ``$jwt_claim_``. | ``string`` | Yes | +|``zoneSize`` | Size of the shared memory zone. Only positive values are allowed. Allowed suffixes are ``k`` or ``m``, if none are present ``k`` is assumed. | ``string`` | Yes | +|``delay`` | The delay parameter specifies a limit at which excessive requests become delayed. If not set all excessive requests are delayed. | ``int`` | No | +|``noDelay`` | Disables the delaying of excessive requests while requests are being limited. Overrides ``delay`` if both are set. | ``bool`` | No | +|``burst`` | Excessive requests are delayed until their number exceeds the ``burst`` size, in which case the request is terminated with an error. | ``int`` | No | +|``dryRun`` | Enables the dry run mode. In this mode, the rate limit is not actually applied, but the number of excessive requests is accounted as usual in the shared memory zone. | ``bool`` | No | +|``logLevel`` | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are ``info``, ``notice``, ``warn`` or ``error``. Default is ``error``. | ``string`` | No | +|``rejectCode`` | Sets the status code to return in response to rejected requests. Must fall into the range ``400..599``. Default is ``503``. | ``int`` | No | +|``scale`` | Enables a constant rate-limit by dividing the configured rate by the number of nginx-ingress pods currently serving traffic. This adjustment ensures that the rate-limit remains consistent, even as the number of nginx-pods fluctuates due to autoscaling. **This will not work properly if requests from a client are not evenly distributed across all ingress pods** (Such as with sticky sessions, long lived TCP Connections with many requests, and so forth). In such cases using [zone-sync]({{< ref "/configuration/global-configuration/configmap-resource.md#zone-sync" >}}) instead would give better results. Enabling `zone-sync` will suppress this setting. | ``bool`` | No | +|``condition`` | Add a condition to a rate-limit policy. | [ratelimit.condition](#ratelimitcondition) | No | +{{% /table %}} + +{{< note >}} + +For each policy referenced in a VirtualServer and/or its VirtualServerRoutes, NGINX Ingress Controller will generate a single rate limiting zone defined by the [`limit_req_zone`](http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive. If two VirtualServer resources reference the same policy, NGINX Ingress Controller will generate two different rate limiting zones, one zone per VirtualServer. + +{{< /note >}} + +#### RateLimit Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple rate limit policies. For example, here we reference two policies: + +```yaml +policies: +- name: rate-limit-policy-one +- name: rate-limit-policy-two +``` + +When you reference more than one rate limit policy, NGINX Ingress Controller will configure NGINX to use all referenced rate limits. When you define multiple policies, each additional policy inherits the `dryRun`, `logLevel`, and `rejectCode` parameters from the first policy referenced (`rate-limit-policy-one`, in the example above). + +### RateLimit.Condition + +RateLimit.Condition defines a condition for a rate limit policy. For example: + +```yaml +condition: + jwt: + claim: user_details.level + match: premium + default: true +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``jwt`` | defines a JWT condition to rate limit against. | [ratelimit.condition.jwt](#ratelimitconditionjwt) | No | +|``default`` | sets the rate limit in this policy to be the default if no conditions are met. In a group of policies with the same JWT condition, only one policy can be the default. | ``bool`` | No | +{{% /table %}} + +The rate limit policy with condition is designed to be used in combination with one or more rate limit policies. For example, multiple rate limit policies with [RateLimit.Condition.JWT](#ratelimitconditionjwt) can be used to apply different tiers of rate limit based on the value of a JWT claim. For a practical example of tiered rate limiting by the value of a JWT claim, see the example in our [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/rate-limit-tiered-jwt-claim/README.md). + +### RateLimit.Condition.JWT +{{< note >}} + +This feature is only available with NGINX Plus. + +{{< /note >}} + +RateLimit.Condition.JWT defines a condition for a rate limit by JWT claim. For example, here we define a condition for a rate limit policy that only applies to requests with a JWT claim `user_details.level` with a value `premium`: + +```yaml +jwt: + claim: user_details.level + match: premium +``` + +The rate limit policy will only apply to requests that contain a JWT with the specified claim and value. For example, the following JWT payload will match the JWT condition: + +```json +{ + "user_details": { + "level": "premium" + }, + "sub": "client1" +} +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``claim`` | Claim is the JWT claim to be rate limit by. Nested claims should be separated by ".". | ``string`` | Yes | +|``match`` | the value of the claim to match against. | ``string`` | Yes | +{{% /table %}} + +### APIKey + +The API Key auth policy configures NGINX to authorize client requests based on the presence of a valid API Key in a header or query param specified in the policy. + +{{< note >}} + +The feature is implemented using NGINX [ngx_http_auth_request_module](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html) and [NGINX JavaScript (NJS)](https://nginx.org/en/docs/njs/). + +{{< /note >}} + +The policies' API keys are securely stored using SHA-256 hashing. When a client sends an API Key, it is hashed by NJS and then compared to the hashed API Key in the NGINX config. + +If the hashed keys match, the NGINX JavaScript (NJS) subrequest issues a 204 No Content response to the `auth_request` directive, indicating successful authorization. Conversely, if no API Key is provided in the specified header or query parameter, a 401 Unauthorized response is returned. Similarly, if an invalid key is presented in the expected header or query parameter, a 403 Forbidden response is issued, denying access. + +It is possible to use the [errorPages](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#errorpage) property on a route, to change the default behaviour of 401 or 403 errors. + +At least one header or query param is required. + +The policy below configures NGINX Ingress Controller to require the API Key `password` in the header "my-header". + +```yaml +apiKey: + suppliedIn: + header: + - "my-header" + clientSecret: api-key-secret +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: api-key-secret +type: nginx.org/apikey +data: + client1: cGFzc3dvcmQ= # password +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``suppliedIn`` | `header` or `query`. | | Yes | +|``suppliedIn.header`` | An array of headers that the API Key may appear in. | ``string[]`` | No | +|``suppliedIn.query`` | An array of query params that the API Key may appear in. | ``string[]`` | No | +|``clientSecret`` | The name of the Kubernetes secret that stores the API Key(s). It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/apikey``, and the API Key(s) must be stored in a key: val format where each key is a unique clientID and each value is a unique base64 encoded API Key | ``string`` | Yes | +{{% /table %}} + +{{}}An APIKey Policy must include a minimum of one of the `suppliedIn.header` or `suppliedIn.query` parameters. Both can also be supplied.{{}} + +#### APIKey Merging Behavior + +A VirtualServer or VirtualServerRoute can be associated with only one API Key policy per route or subroute. However, it is possible to replace an API Key policy from a higher-level with a different policy defined on a more specific route. + +For example, a VirtualServer can implement different API Key policies at various levels. In the configuration below, the server-wide api-key-policy-server applies to /backend1 for authorization, as it lacks a more specific policy. Meanwhile, /backend2 uses the api-key-policy-route defined at the route level. + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: virtual-server +spec: + host: virtual-server.example.com + policies: + - name: api-key-policy-server + upstreams: + - name: backend2 + service: backend2-svc + port: 80 + - name: backend1 + service: backend1-svc + port: 80 + routes: + - path: /backend1 + action: + pass: backend1 + - path: /backend2 + action: + pass: backend2 + policies: + - name: api-key-policy-route +``` + +### BasicAuth + +The basic auth policy configures NGINX to authenticate client requests using the [HTTP Basic authentication scheme](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication). + +For example, the following policy will reject all requests that do not include a valid username/password combination in the HTTP header `Authentication` + +```yaml +basicAuth: + secret: htpasswd-secret + realm: "My API" +``` +{{< note >}} +The feature is implemented using the NGINX [ngx_http_auth_basic_module](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html). +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of the Kubernetes secret that stores the Htpasswd configuration. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/htpasswd``, and the config must be stored in the secret under the key ``htpasswd``, otherwise the secret will be rejected as invalid. | ``string`` | Yes | +|``realm`` | The realm for the basic authentication. | ``string`` | No | +{{% /table %}} + +#### BasicAuth Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple basic auth policies. However, only one can be applied. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: basic-auth-policy-one +- name: basic-auth-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `basic-auth-policy-one`, and ignores `basic-auth-policy-two`. + +### JWT Using Local Kubernetes Secret + +{{< note >}} + +This feature is only available with NGINX Plus. + +{{< /note >}} + +The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. + +The following example policy will reject all requests that do not include a valid JWT in the HTTP header `token`: + +```yaml +jwt: + secret: jwk-secret + realm: "My API" + token: $http_token +``` + +You can pass the JWT claims and JOSE headers to the upstream servers. For example: + +```yaml +action: + proxy: + upstream: webapp + requestHeaders: + set: + - name: user + value: ${jwt_claim_user} + - name: alg + value: ${jwt_header_alg} +``` + +We use the `requestHeaders` of the [Action.Proxy](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#actionproxy) to set the values of two headers that NGINX will pass to the upstream servers. + +The value of the `${jwt_claim_user}` variable is the `user` claim of a JWT. For other claims, use `${jwt_claim_name}`, where `name` is the name of the claim. Note that nested claims and claims that include a period (`.`) are not supported. Similarly, use `${jwt_header_name}` where `name` is the name of a header. In our example, we use the `alg` header. + +{{< note >}} + +This feature is implemented using the NGINX Plus [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of the Kubernetes secret that stores the JWK. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/jwk``, and the JWK must be stored in the secret under the key ``jwk``, otherwise the secret will be rejected as invalid. | ``string`` | Yes | +|``realm`` | The realm of the JWT. | ``string`` | Yes | +|``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | +{{% /table %}} + +#### JWT Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple JWT policies. However, only one can be applied: every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: jwt-policy-one +- name: jwt-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `jwt-policy-one`, and ignores `jwt-policy-two`. + +### JWT Using JWKS From Remote Location + +{{< note >}} + +This feature is only available with NGINX Plus. + +{{< /note >}} + +The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens, allowing import of the keys (JWKS) for JWT policy by means of a URL (for a remote server or an identity provider) as a result they don't have to be copied and updated to the IC pod. + +The following example policy will reject all requests that do not include a valid JWT in the HTTP header fetched from the identity provider: + +```yaml +jwt: + realm: MyProductAPI + token: $http_token + jwksURI: + keyCache: 1h +``` + +{{< note >}} + +This feature is implemented using the NGINX Plus directive [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) under [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``jwksURI`` | The remote URI where the request will be sent to retrieve JSON Web Key set| ``string`` | Yes | +|``keyCache`` | Enables in-memory caching of JWKS (JSON Web Key Sets) that are obtained from the ``jwksURI`` and sets a valid time for expiration. | ``string`` | Yes | +|``realm`` | The realm of the JWT. | ``string`` | Yes | +|``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | +{{% /table %}} + +{{< note >}} + +Content caching is enabled by default for each JWT policy with a default time of 12 hours. + +This is done to ensure to improve resiliency by allowing the JWKS (JSON Web Key Set) to be retrieved from the cache even when it has expired. + +{{< /note >}} + +#### JWT Merging Behavior + +This behavior is similar to using a local Kubernetes secret where a VirtualServer/VirtualServerRoute can reference multiple JWT policies. However, only one can be applied: every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: jwt-policy-one +- name: jwt-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `jwt-policy-one`, and ignores `jwt-policy-two`. + +### IngressMTLS + +The IngressMTLS policy configures client certificate verification. + +For example, the following policy will verify a client certificate using the CA certificate specified in the `ingress-mtls-secret`: + +```yaml +ingressMTLS: + clientCertSecret: ingress-mtls-secret + verifyClient: "on" + verifyDepth: 1 +``` + +Below is an example of the `ingress-mtls-secret` using the secret type `nginx.org/ca` + +```yaml +kind: Secret +metadata: + name: ingress-mtls-secret +apiVersion: v1 +type: nginx.org/ca +data: + ca.crt: +``` + +A VirtualServer that references an IngressMTLS policy must: + +- Enable [TLS termination](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualservertls). +- Reference the policy in the VirtualServer [`spec`](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserver-specification). It is not allowed to reference an IngressMTLS policy in a [`route`](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserverroute) or in a VirtualServerRoute [`subroute`](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserverroutesubroute). + +If the conditions above are not met, NGINX will send the `500` status code to clients. + +You can pass the client certificate details, including the certificate, to the upstream servers. For example: + +```yaml +action: + proxy: + upstream: webapp + requestHeaders: + set: + - name: client-cert-subj-dn + value: ${ssl_client_s_dn} # subject DN + - name: client-cert + value: ${ssl_client_escaped_cert} # client certificate in the PEM format (urlencoded) +``` + +We use the `requestHeaders` of the [Action.Proxy](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#actionproxy) to set the values of the two headers that NGINX will pass to the upstream servers. See the [list of embedded variables](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#variables) that are supported by the `ngx_http_ssl_module`, which you can use to pass the client certificate details. + +{{< note >}} + + The feature is implemented using the NGINX [ngx_http_ssl_module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html). + + {{< /note >}} + +#### Using a Certificate Revocation List + +The IngressMTLS policy supports configuring at CRL for your policy. +This can be done in one of two ways. + +{{< note >}} + + Only one of these configurations options can be used at a time. + +{{< /note >}} + +1. Adding the `ca.crl` field to the `nginx.org/ca` secret type, which accepts a base64 encoded certificate revocation list (crl). + Example YAML: + +```yaml +kind: Secret +metadata: + name: ingress-mtls-secret +apiVersion: v1 +type: nginx.org/ca +data: + ca.crt: + ca.crl: +``` + +2. Adding the `crlFileName` field to your IngressMTLS policy spec with the name of the CRL file. + +{{< note >}} + +This configuration option should only be used when using a CRL that is larger than 1MiB. + +Otherwise we recommend using the `nginx.org/ca` secret type for managing your CRL. + +{{< /note >}} + +Example YAML: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: Policy +metadata: + name: ingress-mtls-policy +spec: +ingressMTLS: + clientCertSecret: ingress-mtls-secret + crlFileName: webapp.crl + verifyClient: "on" + verifyDepth: 1 +``` + +**IMPORTANT NOTE** +When configuring a CRL with the `ingressMTLS.crlFileName` field, there is additional context to keep in mind: + +1. NGINX Ingress Controller will expect the CRL, in this case `webapp.crl`, will be in `/etc/nginx/secrets`. A volume mount will need to be added to NGINX Ingress Controller deployment add your CRL to `/etc/nginx/secrets` +2. When updating the content of your CRL (e.g a new certificate has been revoked), NGINX will need to be reloaded to pick up the latest changes. Depending on your environment this may require updating the name of your CRL and applying this update to your `ingress-mtls.yaml` policy to ensure NGINX picks up the latest CRL. + +Please refer to the Kubernetes documentation on [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) to find the best implementation for your environment. + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``clientCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``, otherwise the secret will be rejected as invalid. | ``string`` | Yes | +|``verifyClient`` | Verification for the client. Possible values are ``"on"``, ``"off"``, ``"optional"``, ``"optional_no_ca"``. The default is ``"on"``. | ``string`` | No | +|``verifyDepth`` | Sets the verification depth in the client certificates chain. The default is ``1``. | ``int`` | No | +|``crlFileName`` | The file name of the Certificate Revocation List. NGINX Ingress Controller will look for this file in `/etc/nginx/secrets` | ``string`` | No | +{{% /table %}} + +#### IngressMTLS Merging Behavior + +A VirtualServer can reference only a single IngressMTLS policy. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: ingress-mtls-policy-one +- name: ingress-mtls-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `ingress-mtls-policy-one`, and ignores `ingress-mtls-policy-two`. + +### EgressMTLS + +The EgressMTLS policy configures upstreams authentication and certificate verification. + +For example, the following policy will use `egress-mtls-secret` to authenticate with the upstream application and `egress-trusted-ca-secret` to verify the certificate of the application: + +```yaml +egressMTLS: + tlsSecret: egress-mtls-secret + trustedCertSecret: egress-trusted-ca-secret + verifyServer: on + verifyDepth: 2 +``` + +{{< note >}} + +The feature is implemented using the NGINX [ngx_http_proxy_module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``tlsSecret`` | The name of the Kubernetes secret that stores the TLS certificate and key. It must be in the same namespace as the Policy resource. The secret must be of the type ``kubernetes.io/tls``, the certificate must be stored in the secret under the key ``tls.crt``, and the key must be stored under the key ``tls.key``, otherwise the secret will be rejected as invalid. | ``string`` | No | +|``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``, otherwise the secret will be rejected as invalid. | ``string`` | No | +|``verifyServer`` | Enables verification of the upstream HTTPS server certificate. | ``bool`` | No | +|``verifyDepth`` | Sets the verification depth in the proxied HTTPS server certificates chain. The default is ``1``. | ``int`` | No | +|``sessionReuse`` | Enables reuse of SSL sessions to the upstreams. The default is ``true``. | ``bool`` | No | +|``serverName`` | Enables passing of the server name through ``Server Name Indication`` extension. | ``bool`` | No | +|``sslName`` | Allows overriding the server name used to verify the certificate of the upstream HTTPS server. | ``string`` | No | +|``ciphers`` | Specifies the enabled ciphers for requests to an upstream HTTPS server. The default is ``DEFAULT``. | ``string`` | No | +|``protocols`` | Specifies the protocols for requests to an upstream HTTPS server. The default is ``TLSv1 TLSv1.1 TLSv1.2``. | ``string`` | No | > Note: the value of ``ciphers`` and ``protocols`` is not validated by NGINX Ingress Controller. As a result, NGINX can fail to reload the configuration. To ensure that the configuration for a VirtualServer/VirtualServerRoute that references the policy was successfully applied, check its [status](/nginx-ingress-controller/configuration/global-configuration/reporting-resources-status/#virtualserver-and-virtualserverroute-resources). The validation will be added in the future releases. | +{{% /table %}} + +#### EgressMTLS Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple EgressMTLS policies. However, only one can be applied. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: egress-mtls-policy-one +- name: egress-mtls-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `egress-mtls-policy-one`, and ignores `egress-mtls-policy-two`. + +### OIDC + +{{< tip >}} + +This feature is disabled by default. To enable it, set the [enable-oidc]({{< ref "/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-oidc" >}}) command-line argument of NGINX Ingress Controller. + +{{< /tip >}} + +The OIDC policy configures NGINX Plus as a relying party for OpenID Connect authentication. + +For example, the following policy will use the client ID `nginx-plus` and the client secret `oidc-secret` to authenticate with the OpenID Connect provider `https://idp.example.com`: + +```yaml +spec: + oidc: + clientID: nginx-plus + clientSecret: oidc-secret + authEndpoint: https://idp.example.com/openid-connect/auth + tokenEndpoint: https://idp.example.com/openid-connect/token + jwksURI: https://idp.example.com/openid-connect/certs + endSessionEndpoint: https://idp.example.com/openid-connect/logout + postLogoutRedirectURI: / + accessTokenEnable: true +``` + +NGINX Plus will pass the ID of an authenticated user to the backend in the HTTP header `username`. + +{{< note >}} + +The feature is implemented using the [reference implementation](https://github.com/nginxinc/nginx-openid-connect/) of NGINX Plus as a relying party for OpenID Connect authentication. + +{{< /note >}} + +#### Prerequisites + +In order to use OIDC, you need to enable [zone synchronization](https://docs.nginx.com/nginx/admin-guide/high-availability/zone_sync/). If you don't set up zone synchronization, NGINX Plus will fail to reload. +You also need to configure a resolver, which NGINX Plus will use to resolve the IDP authorization endpoint. You can find an example configuration [in our GitHub repository](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/examples/custom-resources/oidc#step-7---configure-nginx-plus-zone-synchronization-and-resolver). + +{{< warning >}} + +The configuration in the example doesn't enable TLS and the synchronization between the replica happens in clear text. This could lead to the exposure of tokens. + +{{< /warning >}} + +#### Limitations + +The OIDC policy defines a few internal locations that can't be customized: `/_jwks_uri`, `/_token`, `/_refresh`, `/_id_token_validation`, `/logout`. In addition, as explained below, `/_codexch` is the default value for redirect URI, and `/_logout` is the default value for post logout redirect URI, both of which can be customized. Specifying one of these locations as a route in the VirtualServer or VirtualServerRoute will result in a collision and NGINX Plus will fail to reload. + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``clientID`` | The client ID provided by your OpenID Connect provider. | ``string`` | Yes | +|``clientSecret`` | The name of the Kubernetes secret that stores the client secret provided by your OpenID Connect provider. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/oidc``, and the secret under the key ``client-secret``, otherwise the secret will be rejected as invalid. | ``string`` | Yes | +|``authEndpoint`` | URL for the authorization endpoint provided by your OpenID Connect provider. | ``string`` | Yes | +|``authExtraArgs`` | A list of extra URL arguments to pass to the authorization endpoint provided by your OpenID Connect provider. Arguments must be URL encoded, multiple arguments may be included in the list, for example ``[ arg1=value1, arg2=value2 ]`` | ``string[]`` | No | +|``tokenEndpoint`` | URL for the token endpoint provided by your OpenID Connect provider. | ``string`` | Yes | +|``endSessionEndpoint`` | URL provided by your OpenID Connect provider to request the end user be logged out. | ``string`` | No | +|``jwksURI`` | URL for the JSON Web Key Set (JWK) document provided by your OpenID Connect provider. | ``string`` | Yes | +|``scope`` | List of OpenID Connect scopes. The scope ``openid`` always needs to be present and others can be added concatenating them with a ``+`` sign, for example ``openid+profile+email``, ``openid+email+userDefinedScope``. The default is ``openid``. | ``string`` | No | +|``redirectURI`` | Allows overriding the default redirect URI. The default is ``/_codexch``. | ``string`` | No | +|``postLogoutRedirectURI`` | URI to redirect to after the logout has been performed. Requires ``endSessionEndpoint``. The default is ``/_logout``. | ``string`` | No | +|``zoneSyncLeeway`` | Specifies the maximum timeout in milliseconds for synchronizing ID/access tokens and shared values between Ingress Controller pods. The default is ``200``. | ``int`` | No | +|``accessTokenEnable`` | Option of whether Bearer token is used to authorize NGINX to access protected backend. | ``boolean`` | No | +{{% /table %}} + +{{< note >}} + +Only one OIDC policy can be referenced in a VirtualServer and its VirtualServerRoutes. However, the same policy can still be applied to different routes in the VirtualServer and VirtualServerRoutes. + +{{< /note >}} + +#### OIDC Merging Behavior + +A VirtualServer/VirtualServerRoute can reference only a single OIDC policy. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: oidc-policy-one +- name: oidc-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `oidc-policy-one`, and ignores `oidc-policy-two`. + +## Using Policy + +You can use the usual `kubectl` commands to work with Policy resources, just as with built-in Kubernetes resources. + +For example, the following command creates a Policy resource defined in `access-control-policy-allow.yaml` with the name `webapp-policy`: + +```shell +kubectl apply -f access-control-policy-allow.yaml + +policy.k8s.nginx.org/webapp-policy configured +``` + +You can get the resource by running: + +```shell +kubectl get policy webapp-policy + +NAME AGE +webapp-policy 27m +``` + +For `kubectl get` and similar commands, you can also use the short name `pol` instead of `policy`. + +### WAF {#waf} + +{{< note >}} The feature is implemented using the NGINX Plus [NGINX App Protect WAF Module](https://docs.nginx.com/nginx-app-protect/configuration/). {{< /note >}} + +The WAF policy configures NGINX Plus to secure client requests using App Protect WAF policies. + +For example, the following policy will enable the referenced APPolicy. You can configure multiple APLogConfs with log destinations: + +```yaml +waf: + enable: true + apPolicy: "default/dataguard-alarm" + securityLogs: + - enable: true + apLogConf: "default/logconf" + logDest: "syslog:server=syslog-svc.default:514" + - enable: true + apLogConf: "default/logconf" + logDest: "syslog:server=syslog-svc-secondary.default:514" +``` + +{{< note >}} The field `waf.securityLog` is deprecated and will be removed in future releases.It will be ignored if `waf.securityLogs` is populated. {{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables NGINX App Protect WAF. | ``bool`` | Yes | +|``apPolicy`` | The [App Protect WAF policy]({{< relref "installation/integrations/app-protect-waf/configuration.md#waf-policies" >}}) of the WAF. Accepts an optional namespace. Mutually exclusive with ``apBundle``. | ``string`` | No | +|``apBundle`` | The [App Protect WAF policy bundle]({{< relref "installation/integrations/app-protect-waf/configuration.md#waf-bundles" >}}). Mutually exclusive with ``apPolicy``. | ``string`` | No | +|``securityLog.enable`` | Enables security log. | ``bool`` | No | +|``securityLog.apLogConf`` | The [App Protect WAF log conf]({{< relref "installation/integrations/app-protect-waf/configuration.md#waf-logs" >}}) resource. Accepts an optional namespace. Only works with ``apPolicy``. | ``string`` | No | +|``securityLog.apLogBundle`` | The [App Protect WAF log bundle]({{< relref "installation/integrations/app-protect-waf/configuration.md#waf-bundles" >}}) resource. Only works with ``apBundle``. | ``string`` | No | +|``securityLog.logDest`` | The log destination for the security log. Only accepted variables are ``syslog:server=:``, ``stderr``, ````. | ``string`` | No | +{{% /table %}} + +#### WAF Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple WAF policies. However, only one can be applied. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: waf-policy-one +- name: waf-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `waf-policy-one`, and ignores `waf-policy-two`. + +### Applying Policies + +You can apply policies to both VirtualServer and VirtualServerRoute resources. For example: + +- VirtualServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: VirtualServer + metadata: + name: cafe + namespace: cafe + spec: + host: cafe.example.com + tls: + secret: cafe-secret + policies: # spec policies + - name: policy1 + upstreams: + - name: coffee + service: coffee-svc + port: 80 + routes: + - path: /tea + policies: # route policies + - name: policy2 + namespace: cafe + route: tea/tea + - path: /coffee + policies: # route policies + - name: policy3 + namespace: cafe + action: + pass: coffee + ``` + + For VirtualServer, you can apply a policy: + * to all routes (spec policies) + * to a specific route (route policies) + + Route policies of the *same type* override spec policies. In the example above, if the type of the policies `policy-1` and `policy-3` is `accessControl`, then for requests to `cafe.example.com/coffee`, NGINX will apply `policy-3`. + + The overriding is enforced by NGINX: the spec policies are implemented in the `server` context of the config, and the route policies are implemented in the `location` context. As a result, the route policies of the same type win. + +- VirtualServerRoute, which is referenced by the VirtualServer above: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: VirtualServerRoute + metadata: + name: tea + namespace: tea + spec: + host: cafe.example.com + upstreams: + - name: tea + service: tea-svc + port: 80 + subroutes: # subroute policies + - path: /tea + policies: + - name: policy4 + namespace: tea + action: + pass: tea + ``` + + For VirtualServerRoute, you can apply a policy to a subroute (subroute policies). + + Subroute policies of the same type override spec policies. In the example above, if the type of the policies `policy-1` (in the VirtualServer) and `policy-4` is `accessControl`, then for requests to `cafe.example.com/tea`, NGINX will apply `policy-4`. As with the VirtualServer, the overriding is enforced by NGINX. + + Subroute policies always override route policies no matter the types. For example, the policy `policy-2` in the VirtualServer route will be ignored for the subroute `/tea`, because the subroute has its own policies (in our case, only one policy `policy4`). If the subroute didn't have any policies, then the `policy-2` would be applied. This overriding is enforced by NGINX Ingress Controller -- the `location` context for the subroute will either have route policies or subroute policies, but not both. + +### Invalid Policies + +NGINX will treat a policy as invalid if one of the following conditions is met: + +- The policy doesn't pass the [comprehensive validation](#comprehensive-validation). +- The policy isn't present in the cluster. +- The policy doesn't meet its type-specific requirements. For example, an `ingressMTLS` policy requires TLS termination enabled in the VirtualServer. + +For an invalid policy, NGINX returns the 500 status code for client requests with the following rules: + +- If a policy is referenced in a VirtualServer `route` or a VirtualServerRoute `subroute`, then NGINX will return the 500 status code for requests for the URIs of that route/subroute. +- If a policy is referenced in the VirtualServer `spec`, then NGINX will return the 500 status code for requests for all URIs of that VirtualServer. + +If a policy is invalid, the VirtualServer or VirtualServerRoute will have the [status](/nginx-ingress-controller/configuration/global-configuration/reporting-resources-status#virtualserver-and-virtualserverroute-resources) with the state `Warning` and the message explaining why the policy wasn't considered invalid. + +### Validation + +Two types of validation are available for the Policy resource: + +- *Structural validation*, done by `kubectl` and the Kubernetes API server. +- *Comprehensive validation*, done by NGINX Ingress Controller. + +#### Structural Validation + +The custom resource definition for the Policy includes a structural OpenAPI schema, which describes the type of every field of the resource. + +If you try to create (or update) a resource that violates the structural schema -- for example, the resource uses a string value instead of an array of strings in the `allow` field -- `kubectl` and the Kubernetes API server will reject the resource. + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f access-control-policy-allow.yaml + + error: error validating "access-control-policy-allow.yaml": error validating data: ValidationError(Policy.spec.accessControl.allow): invalid type for org.nginx.k8s.v1.Policy.spec.accessControl.allow: got "string", expected "array"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f access-control-policy-allow.yaml --validate=false + + The Policy "webapp-policy" is invalid: spec.accessControl.allow: Invalid value: "string": spec.accessControl.allow in body must be of type array: "string" + ``` + +If a resource passes structural validation, then NGINX Ingress Controller's comprehensive validation runs. + +#### Comprehensive Validation + +NGINX Ingress Controller validates the fields of a Policy resource. If a resource is invalid, NGINX Ingress Controller will reject it. The resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can use `kubectl` to check whether or not NGINX Ingress Controller successfully applied a Policy configuration. For our example `webapp-policy` Policy, we can run: + +```shell +kubectl describe pol webapp-policy + +. . . +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated +``` + +Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a Policy `webapp-policy` with an invalid IP `10.0.0.` in the `allow` field, you will get: + +```shell +kubectl describe policy webapp-policy + +. . . +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 7s nginx-ingress-controller Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP +``` + +Note how the events section includes a Warning event with the Rejected reason. + +Additionally, this information is also available in the `status` field of the Policy resource. Note the Status section of the Policy: + +```shell +kubectl describe pol webapp-policy + +. . . +Status: + Message: Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP + Reason: Rejected + State: Invalid +``` + +**Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it. diff --git a/content/includes/nic/configuration/security.md b/content/includes/nic/configuration/security.md new file mode 100644 index 000000000..ae2f9648f --- /dev/null +++ b/content/includes/nic/configuration/security.md @@ -0,0 +1,102 @@ +--- +docs: DOCS-597 +doctypes: +- '' +title: Security recommendations +toc: true +weight: 300 +--- + +F5 NGINX Ingress Controller follows Kubernetes best practices: this page outlines configuration specific to NGINX Ingress Controller you may require, including links to examples in the [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples). + +For general guidance, we recommend the official Kubernetes documentation for [Securing a Cluster](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/). + +## Kubernetes recommendations + +### RBAC and Service Accounts + +Kubernetes uses [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) to control the resources and operations available to different types of users. + +NGINX Ingress Controller requires RBAC to configure a [ServiceUser](https://kubernetes.io/docs/concepts/security/service-accounts/#default-service-accounts), and provides least privilege access in its standard deployment configurations: + +- [Helm](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/deployments/rbac/rbac.yaml) +- [Manifests](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/deployments/rbac/rbac.yaml) + +By default, the ServiceAccount has access to all Secret resources in the cluster. + +### Secrets + +[Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) are required by NGINX Ingress Controller for certificates and privacy keys, which Kubernetes stores unencrypted by default. We recommend following the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) to store these Secrets using at-rest encryption. + + +## NGINX Ingress Controller recommendations + +### Configure root filesystem as read-only + +{{< caution >}} + This feature is compatible with [NGINX App Protect WAFv5](https://docs.nginx.com/nginx-app-protect-waf/v5/). It is not compatible with [NGINX App Protect WAFv4](https://docs.nginx.com/nginx-app-protect-waf/v4/) or [NGINX App Protect DoS](https://docs.nginx.com/nginx-app-protect-dos/). +{{< /caution >}} + +NGINX Ingress Controller is designed to be resilient against attacks in various ways, such as running the service as non-root to avoid changes to files. We recommend setting filesystems on all containers to read-only, this includes `nginx-ingress-controller`, though also includes `waf-enforcer` and `waf-config-mgr` when NGINX App Protect WAFv5 is in use. This is so that the attack surface is further reduced by limiting changes to binaries and libraries. + +This is not enabled by default, but can be enabled with **Helm** using the [**readOnlyRootFilesystem**]({{< relref "installation/installing-nic/installation-with-helm.md#configuration" >}}) argument in security contexts on all containers: `nginx-ingress-controller`, `waf_enforcer` and `waf_config_mgr`. + +For **Manifests**, uncomment the following sections of the deployment and add sections for `waf-enforcer` and `waf-config-mgr` containers: + +- `readOnlyRootFilesystem: true` +- The entire **volumeMounts** section +- The entire **initContainers** section + +The block below shows the code you will look for: + +```yaml +# volumes: +# - name: nginx-etc +# emptyDir: {} +# - name: nginx-cache +# emptyDir: {} +# - name: nginx-lib +# emptyDir: {} +# - name: nginx-log +# emptyDir: {} +. +. +. +# readOnlyRootFilesystem: true +. +. +. +# volumeMounts: +# - mountPath: /etc/nginx +# name: nginx-etc +# - mountPath: /var/cache/nginx +# name: nginx-cache +# - mountPath: /var/lib/nginx +# name: nginx-lib +# - mountPath: /var/log/nginx +# name: nginx-log +``` + +- Add **waf-enforcer** and **waf-config-mgr** container sections +- Add `readOnlyFilesystem: true` in both containers security context sections + +### Prometheus + +If Prometheus metrics are [enabled]({{< relref "/logging-and-monitoring/prometheus.md" >}}), we recommend [using HTTPS]({{< ref "/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-prometheus-tls-secret" >}}). + +### Snippets + +Snippets allow raw NGINX configuration to be inserted into resources. They are intended for advanced NGINX users and could create vulnerabilities in a cluster if misused. + +Snippets are disabled by default. To use snippets, set the [**enable-snippets**]({{< relref"configuration/global-configuration/command-line-arguments.md#cmdoption-enable-snippets" >}}) command-line argument. + +{{< caution >}} + Snippets are **always** enabled for ConfigMap. +{{< /caution >}} + +For more information, read the following: + +- [Advanced configuration using Snippets]({{< relref "/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) +- [Using Snippets with VirtualServer/VirtualServerRoute]({{< ref "/nic/configuration/virtualserver-and-virtualserverroute-resources.md#using-snippets" >}}) +- [Using Snippets with TransportServer]({{< relref "/configuration/transportserver-resource.md#using-snippets" >}}) +- [ConfigMap snippets and custom templates]({{< ref "/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates" >}}) diff --git a/content/includes/nic/configuration/transportserver-resource.md b/content/includes/nic/configuration/transportserver-resource.md new file mode 100644 index 000000000..b725ea2e6 --- /dev/null +++ b/content/includes/nic/configuration/transportserver-resource.md @@ -0,0 +1,416 @@ +--- +title: TransportServer resources +toc: true +weight: 600 +docs: DOCS-598 +--- + +This document is reference material for the TransportServer resource used by F5 NGINX Ingress Controller. + +The TransportServer resource allows you to configure TCP, UDP, and TLS Passthrough load balancing. The resource is implemented as a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources) for specific use cases. + +## Prerequisites + +- For TCP and UDP, the TransportServer resource must be used in conjunction with the [GlobalConfiguration resource]({{< ref "/nic/configuration/global-configuration/globalconfiguration-resource.md" >}}), which must be created separately. +- For TLS Passthrough, make sure to enable the [`-enable-tls-passthrough`]({{< ref "/nic/configuration/global-configuration/command-line-arguments#cmdoption-enable-tls-passthrough.md" >}}) command-line argument of NGINX Ingress Controller. + +## TransportServer Specification + +The TransportServer resource defines load balancing configuration for TCP, UDP, or TLS Passthrough traffic. Below are a few examples: + +- TCP load balancing: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: dns-tcp + spec: + host: dns.example.com + listener: + name: dns-tcp + protocol: TCP + tls: + secret: cafe-secret + upstreams: + - name: dns-app + service: dns-service + port: 5353 + action: + pass: dns-app + ``` + +- UDP load balancing: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: dns-udp + spec: + listener: + name: dns-udp + protocol: UDP + upstreams: + - name: dns-app + service: dns-service + port: 5353 + upstreamParameters: + udpRequests: 1 + udpResponses: 1 + action: + pass: dns-app + ``` + +- TLS passthrough load balancing: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: secure-app + spec: + listener: + name: tls-passthrough + protocol: TLS_PASSTHROUGH + host: app.example.com + upstreams: + - name: secure-app + service: secure-app + port: 8443 + action: + pass: secure-app + ``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``listener`` | The listener on NGINX that will accept incoming connections/datagrams. | [listener](#listener) | Yes | +|``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. Wildcard domains like ``*.example.com`` are not allowed. When specified, NGINX will use this host for SNI-based routing. For TLS Passthrough, this field is required. For TCP with TLS termination, specifying the host enables SNI routing and requires specifying a TLS secret.| ``string`` | No | +|``tls`` | The TLS termination configuration. Not supported for TLS Passthrough load balancing. | [tls](#tls) | No | +|``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | Yes | +|``upstreamParameters`` | The upstream parameters. | [upstreamParameters](#upstreamparameters) | No | +|``action`` | The action to perform for a client connection/datagram. | [action](#action) | Yes | +|``ingressClassName`` | Specifies which Ingress Controller must handle the TransportServer resource. | ``string`` | No | +|``streamSnippets`` | Sets a custom snippet in the ``stream`` context. | ``string`` | No | +|``serverSnippets`` | Sets a custom snippet in the ``server`` context. | ``string`` | No | +{{}} + +\* -- Required for TLS Passthrough load balancing. + +### Listener + +The listener field references a listener that NGINX will use to accept incoming traffic for the TransportServer. For TCP and UDP, the listener must be defined in the [GlobalConfiguration resource]({{< ref "/nic/configuration/global-configuration/globalconfiguration-resource.md" >}}). When referencing a listener, both the name and the protocol must match. For TLS Passthrough, use the built-in listener with the name `tls-passthrough` and the protocol `TLS_PASSTHROUGH`. + +The combination of ``spec.listener.name`` and ``spec.host`` must be unique among all TransportServers. If two TransportServers specify the same combination of ``spec.listener.name`` and ``spec.host``, one of them will be rejected to prevent conflicts. In the case where no host is specified, it is considered an empty string. + +An example: + +```yaml +listener: + name: dns-udp + protocol: UDP +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the listener. | ``string`` | Yes | +|``protocol`` | The protocol of the listener. | ``string`` | Yes | +{{}} + +### TLS + +The tls field defines TLS configuration for a TransportServer. When using TLS termination (not TLS Passthrough), you can specify the host field to enable SNI-based routing, allowing multiple applications to share the same listener port and be distinguished by the TLS SNI hostname. Each application can use its own TLS certificate and key specified via the secret field. + +```yaml +secret: cafe-secret +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the TransportServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). | ``string`` | No | +{{}} + +### Upstream + +The upstream defines a destination for the TransportServer. For example: + +```yaml +name: secure-app +service: secure-app +port: 8443 +maxFails: 3 +maxConns: 100 +failTimeout: 30s +loadBalancingMethod: least_conn +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | +|``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). The service must belong to the same namespace as the resource. If the service doesn't exist, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. | ``string`` | Yes | +|``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. The port must fall into the range ``1..65535``. | ``int`` | Yes | +|``maxFails`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_fails) of unsuccessful attempts to communicate with the server that should happen in the duration set by the failTimeout parameter to consider the server unavailable. The default ``1``. | ``int`` | No | +|``maxConns`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_conns) of maximum connections to the proxied server. Default value is zero, meaning there is no limit. The default is ``0``. | ``int`` | No | +|``failTimeout`` | Sets the [time](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#fail_timeout) during which the specified number of unsuccessful attempts to communicate with the server should happen to consider the server unavailable and the period of time the server will be considered unavailable. The default is ``10s``. | ``string`` | No | +|``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive. Note: this feature is supported only in NGINX Plus. | [healthcheck](#upstreamhealthcheck) | No | +|``loadBalancingMethod`` | The method used to load balance the upstream servers. By default, connections are distributed between the servers using a weighted round-robin balancing method. See the [upstream](http://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) section for available methods and their details. | ``string`` | No | +|``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | +|``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | +{{}} + +### Upstream.Healthcheck + +The Healthcheck defines an [active health check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html?#health_check). In the example below we enable a health check for an upstream and configure all the available parameters: + +```yaml +name: secure-app +service: secure-app +port: 8443 +healthCheck: + enable: true + interval: 20s + timeout: 30s + jitter: 3s + fails: 5 + passes: 5 + port: 8080 +``` + +{{< note >}} This feature is only supported with NGINX Plus. {{< /note >}} + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | +|``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | +|``timeout`` | This overrides the timeout set by [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) which is set in `SessionParameters` for health checks. The default value is ``5s``. | ``string`` | No | +|``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | +|``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | +|``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | +|``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | +|``match`` | Controls the data to send and the response to expect for the healthcheck. | [match](#upstreamhealthcheckmatch) | No | +{{}} + +### Upstream.Healthcheck.Match + +The match controls the data to send and the response to expect for the healthcheck: + +```yaml +match: + send: 'GET / HTTP/1.0\r\nHost: localhost\r\n\r\n' + expect: "~200 OK" +``` + +Both `send` and `expect` fields can contain hexadecimal literals with the prefix `\x` followed by two hex digits, for example, `\x80`. + +See the [match](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) directive for details. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``send`` | A string to send to an upstream server. | ``string`` | No | +|``expect`` | A literal string or a regular expression that the data obtained from the server should match. The regular expression is specified with the preceding ``~*`` modifier (for case-insensitive matching), or the ``~`` modifier (for case-sensitive matching). NGINX Ingress Controller validates a regular expression using the RE2 syntax. | ``string`` | No | +{{}} + +### UpstreamParameters + +The upstream parameters define various parameters for the upstreams: + +```yaml +upstreamParameters: + udpRequests: 1 + udpResponses: 1 + connectTimeout: 60s + nextUpstream: true + nextUpstreamTimeout: 50s + nextUpstreamTries: 1 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``udpRequests`` | The number of datagrams, after receiving which, the next datagram from the same client starts a new session. See the [proxy_requests](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_requests) directive. The default is ``0``. | ``int`` | No | +|``udpResponses`` | The number of datagrams expected from the proxied server in response to a client datagram. See the [proxy_responses](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses) directive. By default, the number of datagrams is not limited. | ``int`` | No | +|``connectTimeout`` | The timeout for establishing a connection with a proxied server. See the [proxy_connect_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_connect_timeout) directive. The default is ``60s``. | ``string`` | No | +|``nextUpstream`` | If a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. See the [proxy_next_upstream](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream) directive. The default is ``true``. | bool | No | +|``nextUpstreamTries`` | The number of tries for passing a connection to the next server. See the [proxy_next_upstream_tries](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries) directive. The default is ``0``. | ``int`` | No | +|``nextUpstreamTimeout`` | The time allowed to pass a connection to the next server. See the [proxy_next_upstream_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout) directive. The default us ``0``. | ``string`` | No | +{{}} + +### SessionParameters + +The session parameters define various parameters for TCP connections and UDP sessions. + +```yaml +sessionParameters: + timeout: 50s +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``timeout`` | The timeout between two successive read or write operations on client or proxied server connections. See [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) directive. The default is ``10m``. | ``string`` | No | +{{}} + +### Action + +The action defines an action to perform for a client connection/datagram. + +In the example below, client connections/datagrams are passed to an upstream `dns-app`: + +```yaml +action: + pass: dns-app +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``pass`` | Passes connections/datagrams to an upstream. The upstream with that name must be defined in the resource. | ``string`` | Yes | +{{}} + +## Using TransportServer + +You can use the usual `kubectl` commands to work with TransportServer resources, similar to Ingress resources. + +For example, the following command creates a TransportServer resource defined in `transport-server-passthrough.yaml` with the name `secure-app`: + +```shell +kubectl apply -f transport-server-passthrough.yaml +``` +```text +transportserver.k8s.nginx.org/secure-app created +``` + +You can get the resource by running: + +```shell +kubectl get transportserver secure-app +``` +```text +NAME AGE +secure-app 46sm +``` + +In the kubectl get and similar commands, you can also use the short name `ts` instead of `transportserver`. + +### Using Snippets + +Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure [access control](http://nginx.org/en/docs/stream/ngx_stream_access_module.html) in a TransportServer: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: TransportServer +metadata: + name: cafe +spec: + host: cafe.example.com + serverSnippets: | + deny 192.168.1.1; + allow 192.168.1.0/24; + upstreams: + - name: tea + service: tea-svc + port: 80 +``` + +Snippets can also be specified for a stream. In the example below, we use snippets to [limit the number of connections](https://nginx.org/en/docs/stream/ngx_stream_limit_conn_module.html): + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: TransportServer +metadata: + name: cafe +spec: + host: cafe.example.com + streamSnippets: limit_conn_zone $binary_remote_addr zone=addr:10m; + serverSnippets: limit_conn addr 1; + upstreams: + - name: tea + service: tea-svc + port: 80 +``` + +{{< note >}} To configure snippets in the `stream` context, use `stream-snippets` ConfigMap key. {{< /note >}} + +For additional information, view the [Advanced configuration with Snippets]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) topic. + +### Validation + +Two types of validation are available for the TransportServer resource: + +- *Structural validation* by the `kubectl` and Kubernetes API server. +- *Comprehensive validation* by NGINX Ingress Controller. + +#### Structural Validation + +The custom resource definition for the TransportServer includes structural OpenAPI schema which describes the type of every field of the resource. + +If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f transport-server-passthrough.yaml + ``` + ```text + error: error validating "transport-server-passthrough.yaml": error validating data: ValidationError(TransportServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.TransportServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f transport-server-passthrough.yaml --validate=false + ``` + ```text + The TransportServer "secure-app" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: + spec.upstreams.port in body must be of type integer: "string" + ``` + +If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. + +#### Comprehensive Validation + +NGINX Ingress Controller validates the fields of a TransportServer resource. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can check if NGINX Ingress Controller successfully applied the configuration for a TransportServer. For our example `secure-app` TransportServer, we can run: + +```shell +kubectl describe ts secure-app +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/secure-app was added or updated +``` + +Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a TransportServer `secure-app` with a pass action that references a non-existing upstream, you will get : + +```shell +kubectl describe ts secure-app +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 2s nginx-ingress-controller TransportServer default/secure-app is invalid and was rejected: spec.action.pass: Not found: "some-app" +``` + +Note how the events section includes a Warning event with the Rejected reason. + +**Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. + +## Customization via ConfigMap + +The [ConfigMap]({{< ref "/nic/configuration/global-configuration/configmap-resource.md" >}}) keys (except for `stream-snippets`, `stream-log-format`, `resolver-addresses`, `resolver-ipv6`, `resolver-valid` and `resolver-timeout`) do not affect TransportServer resources. diff --git a/content/includes/nic/configuration/virtualserver-and-virtualserverroute-resources.md b/content/includes/nic/configuration/virtualserver-and-virtualserverroute-resources.md new file mode 100644 index 000000000..1acd05b02 --- /dev/null +++ b/content/includes/nic/configuration/virtualserver-and-virtualserverroute-resources.md @@ -0,0 +1,1101 @@ +--- +title: VirtualServer and VirtualServerRoute resources +toc: true +weight: 700 +docs: DOCS-599 +--- + +This document is reference material for the VirtualServer and VirtualServerRoute resources used by F5 NGINX Ingress Controller. + +VirtualServer and VirtualServerRoute resources are load balancing configurations recommended as an alternative to the Ingress resource. + +They enable use cases not supported with the Ingress resource, such as traffic splitting and advanced content-based routing. The resources are implemented as [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources) for specific use cases. + +--- + +## VirtualServer specification + +The VirtualServer resource defines load balancing configuration for a domain name, such as `example.com`. Below is an example of such configuration: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: cafe +spec: + host: cafe.example.com + listener: + http: http-8083 + https: https-8443 + tls: + secret: cafe-secret + gunzip: on + upstreams: + - name: tea + service: tea-svc + port: 80 + - name: coffee + service: coffee-svc + port: 80 + routes: + - path: /tea + action: + pass: tea + - path: /coffee + action: + pass: coffee + - path: ~ ^/decaf/.*\\.jpg$ + action: + pass: coffee + - path: = /green/tea + action: + pass: tea +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. The ``host`` value needs to be unique among all Ingress and VirtualServer resources. See also [Handling Host and Listener Collisions](/nginx-ingress-controller/configuration/host-and-listener-collisions). | ``string`` | Yes | +|``listener`` | Sets a custom HTTP and/or HTTPS listener. Valid fields are `listener.http` and `listener.https`. Each field must reference the name of a valid listener defined in a GlobalConfiguration resource | [listener](#virtualserverlistener) | No | +|``tls`` | The TLS termination configuration. | [tls](#virtualservertls) | No | +|``gunzip`` | Enables or disables [decompression](https://docs.nginx.com/nginx/admin-guide/web-server/compression/) of gzipped responses for clients. Allowed values “on”/“off”, “true”/“false” or “yes”/“no”. If the ``gunzip`` value is not set, it defaults to ``off``. | ``boolean`` | No | +|``externalDNS`` | The externalDNS configuration for a VirtualServer. | [externalDNS](#virtualserverexternaldns) | No | +|``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer. | ``string`` | No | +|``policies`` | A list of policies. | [[]policy](#virtualserverpolicy) | No | +|``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | +|``routes`` | A list of routes. | [[]route](#virtualserverroute) | No | +|``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServer resource. | ``string`` | No | +|``internalRoute`` | Specifies if the VirtualServer resource is an internal route or not. | ``boolean`` | No | +|``http-snippets`` | Sets a custom snippet in the http context. | ``string`` | No | +|``server-snippets`` | Sets a custom snippet in server context. Overrides the ``server-snippets`` ConfigMap key. | ``string`` | No | +{{}} + +### VirtualServer.TLS + +The tls field defines TLS configuration for a VirtualServer. For example: + +```yaml +secret: cafe-secret +redirect: + enable: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the VirtualServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the host of the VirtualServer. If the secret is not specified but [wildcard TLS secret](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments#cmdoption-wildcard-tls-secret) is configured, NGINX will use the wildcard secret for TLS termination. | ``string`` | No | +|``redirect`` | The redirect configuration of the TLS for a VirtualServer. | [tls.redirect](#virtualservertlsredirect) | No | ### VirtualServer.TLS.Redirect | +|``cert-manager`` | The cert-manager configuration of the TLS for a VirtualServer. | [tls.cert-manager](#virtualservertlscertmanager) | No | ### VirtualServer.TLS.CertManager | +{{}} + +### VirtualServer.TLS.Redirect + +The redirect field configures a TLS redirect for a VirtualServer: + +```yaml +enable: true +code: 301 +basedOn: scheme +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables a TLS redirect for a VirtualServer. The default is ``False``. | ``boolean`` | No | +|``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | +|``basedOn`` | The attribute of a request that NGINX will evaluate to send a redirect. The allowed values are ``scheme`` (the scheme of the request) or ``x-forwarded-proto`` (the ``X-Forwarded-Proto`` header of the request). The default is ``scheme``. | ``string`` | No | ### VirtualServer.Policy | +{{}} + +### VirtualServer.TLS.CertManager + +The cert-manager field configures x509 automated Certificate management for VirtualServer resources using cert-manager (cert-manager.io). Please see the [cert-manager configuration documentation](https://cert-manager.io/docs/configuration/) for more information on deploying and configuring Issuers. Example: + +```yaml +cert-manager: + cluster-issuer: "my-issuer-name" +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``issuer`` | the name of an Issuer. An Issuer is a cert-manager resource which describes the certificate authority capable of signing certificates. The Issuer must be in the same namespace as the VirtualServer resource. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | +|``cluster-issuer`` | the name of a ClusterIssuer. A ClusterIssuer is a cert-manager resource which describes the certificate authority capable of signing certificates. It does not matter which namespace your VirtualServer resides, as ClusterIssuers are non-namespaced resources. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | +|``issuer-kind`` | The kind of the external issuer resource, for example AWSPCAIssuer. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | +|``issuer-group`` | The API group of the external issuer controller, for example awspca.cert-manager.io. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | +|``common-name`` | This field allows you to configure spec.commonName for the Certificate to be generated. This configuration adds a CN to the x509 certificate. | ``string`` | No | +|``duration`` | This field allows you to configure spec.duration field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | +|``renew-before`` | this annotation allows you to configure spec.renewBefore field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | +|``usages`` | This field allows you to configure spec.usages field for the Certificate to be generated. Pass a string with comma-separated values i.e. ``key agreement,digital signature, server auth``. An exhaustive list of supported key usages can be found in the [the cert-manager api documentation](https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.KeyUsage). | ``string`` | No | +|``issue-temp-cert`` | When ``true``, ask cert-manager for a [temporary self-signed certificate](https://cert-manager.io/docs/usage/certificate/#temporary-certificates-while-issuing) pending the issuance of the Certificate. This allows HTTPS-only servers to use ACME HTTP01 challenges when the TLS secret does not exist yet. | ``boolean`` | No | +{{}} + +### VirtualServer.Listener +The listener field defines a custom HTTP and/or HTTPS listener. +The respective listeners used must reference the name of a listener defined using a [GlobalConfiguration](/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/) resource. +For example: +```yaml +http: http-8083 +https: https-8443 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``http`` | The name of am HTTP listener defined in a [GlobalConfiguration](/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/) resource. | ``string`` | No | +|``https`` | The name of an HTTPS listener defined in a [GlobalConfiguration](/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/) resource. | ``string`` | No | +{{}} + +### VirtualServer.ExternalDNS + +The externalDNS field configures controlling DNS records dynamically for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Please see the [ExternalDNS configuration documentation](https://kubernetes-sigs.github.io/external-dns/) for more information on deploying and configuring ExternalDNS and Providers. Example: + +```yaml +enable: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables ExternalDNS integration for a VirtualServer resource. The default is ``false``. | ``string`` | No | +|``labels`` | Configure labels to be applied to the Endpoint resources that will be consumed by ExternalDNS. | ``map[string]string`` | No | +|``providerSpecific`` | Configure provider specific properties which holds the name and value of a configuration which is specific to individual DNS providers. | [[]ProviderSpecific](#virtualserverexternaldnsproviderspecific) | No | +|``recordTTL`` | TTL for the DNS record. This defaults to 0 if not defined. See [the ExternalDNS TTL documentation for provider-specific defaults](https://kubernetes-sigs.github.io/external-dns/v0.14.2/ttl/#providers) | ``int64`` | No | +|``recordType`` | The record Type that should be created, e.g. "A", "AAAA", "CNAME". This is automatically computed based on the external endpoints if not defined. | ``string`` | No | +{{}} + +### VirtualServer.ExternalDNS.ProviderSpecific + +The providerSpecific field of the externalDNS block allows the specification of provider specific properties which is a list of key value pairs of configurations which are specific to individual DNS providers. Example: + +```yaml +- name: my-name + value: my-value +- name: my-name2 + value: my-value2 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the key value pair. | ``string`` | Yes | +|``value`` | The value of the key value pair. | ``string`` | Yes | +{{}} + +### VirtualServer.Policy + +The policy field references a [Policy resource](/nginx-ingress-controller/configuration/policy-resource/) by its name and optional namespace. For example: + +```yaml +name: access-control +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of a policy. If the policy doesn't exist or invalid, NGINX will respond with an error response with the `500` status code. | ``string`` | Yes | +|``namespace`` | The namespace of a policy. If not specified, the namespace of the VirtualServer resource is used. | ``string`` | No | +{{}} + +### VirtualServer.Route + +The route defines rules for matching client requests to actions like passing a request to an upstream. For example: + +```yaml + path: /tea + action: + pass: tea +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``path`` | The path of the route. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix (must start with ``/`` ) or an exact match (must start with ``=`` ), the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. The path must be unique among the paths of all routes of the VirtualServer. Check the [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive for more information. | ``string`` | Yes | +|``policies`` | A list of policies. The policies override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying Policies](/nginx-ingress-controller/configuration/policy-resource/#applying-policies) for more details. | [[]policy](#virtualserverpolicy) | No | +|``action`` | The default action to perform for a request. | [action](#action) | No | +|``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer route. | ``string`` | No | +|``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | +|``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | +|``route`` | The name of a VirtualServerRoute resource that defines this route. If the VirtualServerRoute belongs to a different namespace than the VirtualServer, you need to include the namespace. For example, ``tea-namespace/tea``. | ``string`` | No | +|``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | +|``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` ConfigMap key. | ``string`` | No | +{{}} + +\* -- a route must include exactly one of the following: `action`, `splits`, or `route`. + +## VirtualServerRoute specification + +The VirtualServerRoute resource defines a route for a VirtualServer. It can consist of one or multiple subroutes. The VirtualServerRoute is an alternative to [Mergeable Ingress types](/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration). + +In the example below, the VirtualServer `cafe` from the namespace `cafe-ns` defines a route with the path `/coffee`, which is further defined in the VirtualServerRoute `coffee` from the namespace `coffee-ns`. + +VirtualServer: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: cafe + namespace: cafe-ns +spec: + host: cafe.example.com + upstreams: + - name: tea + service: tea-svc + port: 80 + routes: + - path: /tea + action: + pass: tea + - path: /coffee + route: coffee-ns/coffee +``` + +VirtualServerRoute: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServerRoute +metadata: + name: coffee + namespace: coffee-ns +spec: + host: cafe.example.com + upstreams: + - name: latte + service: latte-svc + port: 80 + - name: espresso + service: espresso-svc + port: 80 + subroutes: + - path: /coffee/latte + action: + pass: latte + - path: /coffee/espresso + action: + pass: espresso +``` + +Note that each subroute must have a `path` that starts with the same prefix (here `/coffee`), which is defined in the route of the VirtualServer. Additionally, the `host` in the VirtualServerRoute must be the same as the `host` of the VirtualServer. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. Must be the same as the ``host`` of the VirtualServer that references this resource. | ``string`` | Yes | +|``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | +|``subroutes`` | A list of subroutes. | [[]subroute](#virtualserverroutesubroute) | No | +|``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServerRoute resource. Must be the same as the ``ingressClassName`` of the VirtualServer that references this resource. | ``string``_ | No | +{{}} + +### VirtualServerRoute.Subroute + +The subroute defines rules for matching client requests to actions like passing a request to an upstream. For example: + +```yaml +path: /coffee +action: + pass: coffee +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``path`` | The path of the subroute. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix, the path must start with the same path as the path of the route of the VirtualServer that references this resource. In the case of an exact or regex match, the path must be the same as the path of the route of the VirtualServer that references this resource. A matching path of the route of the VirtualServer but in different type is not accepted, e.g. a regex path (`~/match`) cannot be used with a prefix path in VirtualServer (`/match`) In the case of a prefix or an exact match, the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. The path must be unique among the paths of all subroutes of the VirtualServerRoute. | ``string`` | Yes | +|``policies`` | A list of policies. The policies override *all* policies defined in the route of the VirtualServer that references this resource. The policies also override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying Policies](/nginx-ingress-controller/configuration/policy-resource/#applying-policies) for more details. | [[]policy](#virtualserverpolicy) | No | +|``action`` | The default action to perform for a request. | [action](#action) | No | +|``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServerRoute subroute. | ``string`` | No | +|``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | +|``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | +|``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | +|``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` of the VirtualServer (if set) or the ``location-snippets`` ConfigMap key. | ``string`` | No | +{{}} + +\* -- a subroute must include exactly one of the following: `action` or `splits`. + +## Common VirtualServer and VirtualServerRoute specifications + +### Upstream + +The upstream defines a destination for the routing configuration. For example: + +```yaml +name: tea +service: tea-svc +subselector: + version: canary +port: 80 +lb-method: round_robin +fail-timeout: 10s +max-fails: 1 +max-conns: 32 +keepalive: 32 +connect-timeout: 30s +read-timeout: 30s +send-timeout: 30s +next-upstream: "error timeout non_idempotent" +next-upstream-timeout: 5s +next-upstream-tries: 10 +client-max-body-size: 2m +tls: + enable: true +``` + +**Note**: The WebSocket protocol is supported without any additional configuration. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | +|``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). The service must belong to the same namespace as the resource. If the service doesn't exist, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. For NGINX Plus only, services of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) are also supported (check the [prerequisites](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services#prerequisites) ). | ``string`` | Yes | +|``subselector`` | Selects the pods within the service using label keys and values. By default, all pods of the service are selected. Note: the specified labels are expected to be present in the pods when they are created. If the pod labels are updated, NGINX Ingress Controller will not see that change until the number of the pods is changed. | ``map[string]string`` | No | +|``use-cluster-ip`` | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method`` and ``next-upstream``) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | ``boolean`` | No | +|``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. The port must fall into the range ``1..65535``. | ``uint16`` | Yes | +|``lb-method`` | The load [balancing method](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method). To use the round-robin method, specify ``round_robin``. The default is specified in the ``lb-method`` ConfigMap key. | ``string`` | No | +|``fail-timeout`` | The time during which the specified number of unsuccessful attempts to communicate with an upstream server should happen to consider the server unavailable. See the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the server directive. The default is set in the ``fail-timeout`` ConfigMap key. | ``string`` | No | +|``max-fails`` | The number of unsuccessful attempts to communicate with an upstream server that should happen in the duration set by the ``fail-timeout`` to consider the server unavailable. See the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the server directive. The default is set in the ``max-fails`` ConfigMap key. | ``int`` | No | +|``max-conns`` | The maximum number of simultaneous active connections to an upstream server. See the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the server directive. By default there is no limit. Note: if keepalive connections are enabled, the total number of active and idle keepalive connections to an upstream server may exceed the ``max_conns`` value. | ``int`` | No | +|``keepalive`` | Configures the cache for connections to upstream servers. The value ``0`` disables the cache. See the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. The default is set in the ``keepalive`` ConfigMap key. | ``int`` | No | +|``connect-timeout`` | The timeout for establishing a connection with an upstream server. See the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) directive. The default is specified in the ``proxy-connect-timeout`` ConfigMap key. | ``string`` | No | +|``read-timeout`` | The timeout for reading a response from an upstream server. See the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) directive. The default is specified in the ``proxy-read-timeout`` ConfigMap key. | ``string`` | No | +|``send-timeout`` | The timeout for transmitting a request to an upstream server. See the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) directive. The default is specified in the ``proxy-send-timeout`` ConfigMap key. | ``string`` | No | +|``next-upstream`` | Specifies in which cases a request should be passed to the next upstream server. See the [proxy_next_upstream](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) directive. The default is ``error timeout``. | ``string`` | No | +|``next-upstream-timeout`` | The time during which a request can be passed to the next upstream server. See the [proxy_next_upstream_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) directive. The ``0`` value turns off the time limit. The default is ``0``. | ``string`` | No | +|``next-upstream-tries`` | The number of possible tries for passing a request to the next upstream server. See the [proxy_next_upstream_tries](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) directive. The ``0`` value turns off this limit. The default is ``0``. | ``int`` | No | +|``client-max-body-size`` | Sets the maximum allowed size of the client request body. See the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. The default is set in the ``client-max-body-size`` ConfigMap key. | ``string`` | No | +|``tls`` | The TLS configuration for the Upstream. | [tls](#upstreamtls) | No | +|``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive. Note: this feature is supported only in NGINX Plus. | [healthcheck](#upstreamhealthcheck) | No | +|``slow-start`` | The slow start allows an upstream server to gradually recover its weight from 0 to its nominal value after it has been recovered or became available or when the server becomes available after a period of time it was considered unavailable. By default, the slow start is disabled. See the [slow_start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start) parameter of the server directive. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods and will be ignored. | ``string`` | No | +|``queue`` | Configures a queue for an upstream. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request. By default, no queue is configured. Note: this feature is supported only in NGINX Plus. | [queue](#upstreamqueue) | No | +|``buffering`` | Enables buffering of responses from the upstream server. See the [proxy_buffering](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) directive. The default is set in the ``proxy-buffering`` ConfigMap key. | ``boolean`` | No | +|``buffers`` | Configures the buffers used for reading a response from the upstream server for a single connection. | [buffers](#upstreambuffers) | No | +|``buffer-size`` | Sets the size of the buffer used for reading the first part of a response received from the upstream server. See the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) directive. The default is set in the ``proxy-buffer-size`` ConfigMap key. | ``string`` | No | +|``ntlm`` | Allows proxying requests with NTLM Authentication. See the [ntlm](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ntlm) directive. In order for NTLM authentication to work, it is necessary to enable keepalive connections to upstream servers using the ``keepalive`` field. Note: this feature is supported only in NGINX Plus.| ``boolean`` | No | +|``type`` |The type of the upstream. Supported values are ``http`` and ``grpc``. The default is ``http``. For gRPC, it is necessary to enable HTTP/2 in the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#listeners) and configure TLS termination in the VirtualServer. | ``string`` | No | +|``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | +|``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | +{{}} + +### Upstream.Buffers + +The buffers field configures the buffers used for reading a response from the upstream server for a single connection: + +```yaml +number: 4 +size: 8K +``` + +See the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive for additional information. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``number`` | Configures the number of buffers. The default is set in the ``proxy-buffers`` ConfigMap key. | ``int`` | Yes | +|``size`` | Configures the size of a buffer. The default is set in the ``proxy-buffers`` ConfigMap key. | ``string`` | Yes | +{{}} + +### Upstream.TLS + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables HTTPS for requests to upstream servers. The default is ``False`` , meaning that HTTP will be used. Note: by default, NGINX will not verify the upstream server certificate. To enable the verification, configure an [EgressMTLS Policy](/nginx-ingress-controller/configuration/policy-resource/#egressmtls). | ``boolean`` | No | +{{}} + +### Upstream.Queue + +The queue field configures a queue. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request: + +```yaml +size: 10 +timeout: 60s +``` + +See [`queue`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#queue) directive for additional information. + +Note: This feature is supported only in NGINX Plus. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``size`` | The size of the queue. | ``int`` | Yes | +|``timeout`` | The timeout of the queue. A request cannot be queued for a period longer than the timeout. The default is ``60s``. | ``string`` | No | +{{}} + +### Upstream.Healthcheck + +The Healthcheck defines an [active health check](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/). In the example below we enable a health check for an upstream and configure all the available parameters, including the `slow-start` parameter combined with [`mandatory` and `persistent`](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#mandatory-health-checks): + +```yaml +name: tea +service: tea-svc +port: 80 +slow-start: 30s +healthCheck: + enable: true + path: /healthz + interval: 20s + jitter: 3s + fails: 5 + passes: 5 + port: 8080 + tls: + enable: true + connect-timeout: 10s + read-timeout: 10s + send-timeout: 10s + headers: + - name: Host + value: my.service + statusMatch: "! 500" + mandatory: true + persistent: true + keepalive-time: 60s +``` + +Note: This feature is supported only in NGINX Plus. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | +|``path`` | The path used for health check requests. The default is ``/``. This not configurable for gRPC type upstreams. | ``string`` | No | +|``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | +|``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | +|``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | +|``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | +|``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | +|``tls`` | The TLS configuration used for health check requests. By default, the ``tls`` field of the upstream is used. | [upstream.tls](#upstreamtls) | No | +|``connect-timeout`` | The timeout for establishing a connection with an upstream server. By default, the ``connect-timeout`` of the upstream is used. | ``string`` | No | +|``read-timeout`` | The timeout for reading a response from an upstream server. By default, the ``read-timeout`` of the upstream is used. | ``string`` | No | +|``send-timeout`` | The timeout for transmitting a request to an upstream server. By default, the ``send-timeout`` of the upstream is used. | ``string`` | No | +|``headers`` | The request headers used for health check requests. NGINX Plus always sets the ``Host`` , ``User-Agent`` and ``Connection`` headers for health check requests. | [[]header](#header) | No | +|``statusMatch`` | The expected response status codes of a health check. By default, the response should have status code 2xx or 3xx. Examples: ``"200"`` , ``"! 500"`` , ``"301-303 307"``. See the documentation of the [match](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html?#match) directive. This not supported for gRPC type upstreams. | ``string`` | No | +|``grpcStatus`` | The expected [gRPC status code](https://github.com/grpc/grpc/blob/master/doc/statuscodes.md#status-codes-and-their-use-in-grpc) of the upstream server response to the [Check method](https://github.com/grpc/grpc/blob/master/doc/health-checking.md). Configure this field only if your gRPC services do not implement the gRPC health checking protocol. For example, configure ``12`` if the upstream server responds with `12 (UNIMPLEMENTED)` status code. Only valid on gRPC type upstreams. | ``int`` | No | +|``grpcService`` | The gRPC service to be monitored on the upstream server. Only valid on gRPC type upstreams. | ``string`` | No | +|``mandatory`` | Require every newly added server to pass all configured health checks before NGINX Plus sends traffic to it. If this is not specified, or is set to false, the server will be initially considered healthy. When combined with [slow-start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start), it gives a new server more time to connect to databases and “warm up” before being asked to handle their full share of traffic. | ``bool`` | No | +|``persistent`` | Set the initial “up” state for a server after reload if the server was considered healthy before reload. Enabling persistent requires that the mandatory parameter is also set to `true`. | ``bool`` | No | +|``keepalive-time`` | Enables [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) connections for health checks and specifies the time during which requests can be processed through one keepalive connection. The default is ``60s``. | ``string`` | No | +{{}} + +### Upstream.SessionCookie + +The SessionCookie field configures session persistence which allows requests from the same client to be passed to the same upstream server. The information about the designated upstream server is passed in a session cookie generated by NGINX Plus. + +In the example below, we configure session persistence with a session cookie for an upstream and configure all the available parameters: + +```yaml +name: tea +service: tea-svc +port: 80 +sessionCookie: + enable: true + name: srv_id + path: / + expires: 1h + domain: .example.com + httpOnly: false + secure: true + samesite: strict +``` + +See the [`sticky`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html?#sticky) directive for additional information. The session cookie corresponds to the `sticky cookie` method. + +Note: This feature is supported only in NGINX Plus. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables session persistence with a session cookie for an upstream server. The default is ``false``. | ``boolean`` | No | +|``name`` | The name of the cookie. | ``string`` | Yes | +|``path`` | The path for which the cookie is set. | ``string`` | No | +|``expires`` | The time for which a browser should keep the cookie. Can be set to the special value ``max`` , which will cause the cookie to expire on ``31 Dec 2037 23:55:55 GMT``. | ``string`` | No | +|``domain`` | The domain for which the cookie is set. | ``string`` | No | +|``httpOnly`` | Adds the ``HttpOnly`` attribute to the cookie. | ``boolean`` | No | +|``secure`` | Adds the ``Secure`` attribute to the cookie. | ``boolean`` | No | +|``samesite`` | Adds the ``SameSite`` attribute to the cookie. The allowed values are: ``strict``, ``lax``, ``none`` | ``string`` | No | +{{}} + +### Header + +The header defines an HTTP Header: + +```yaml +name: Host +value: example.com +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. | ``string`` | No | +{{}} + +### Action + +The action defines an action to perform for a request. + +In the example below, client requests are passed to an upstream `coffee`: + +```yaml + path: /coffee + action: + pass: coffee +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``pass`` | Passes requests to an upstream. The upstream with that name must be defined in the resource. | ``string`` | No | +|``redirect`` | Redirects requests to a provided URL. | [action.redirect](#actionredirect) | No | +|``return`` | Returns a preconfigured response. | [action.return](#actionreturn) | No | +|``proxy`` | Passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). | [action.proxy](#actionproxy) | No | +{{}} + +\* -- an action must include exactly one of the following: `pass`, `redirect`, `return` or `proxy`. + +### Action.Redirect + +The redirect action defines a redirect to return for a request. + +In the example below, client requests are passed to a url `http://www.nginx.com`: + +```yaml +redirect: + url: http://www.nginx.com + code: 301 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` , ``$http_x_forwarded_proto`` , ``$request_uri`` , ``$host``. Variables must be enclosed in curly braces. For example: ``${host}${request_uri}``. | ``string`` | Yes | +|``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | +{{}} + +### Action.Return + +The return action defines a preconfigured response for a request. + +In the example below, NGINX will respond with the preconfigured response for every request: + +```yaml +return: + code: 200 + type: text/plain + body: "Hello World\n" + headers: + - name: x-coffee + value: espresso +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``code`` | The status code of the response. The allowed values are: ``2XX``, ``4XX`` or ``5XX``. The default is ``200``. | ``int`` | No | +|``type`` | The MIME type of the response. The default is ``text/plain``. | ``string`` | No | +|``body`` | The body of the response. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``Request is ${request_uri}\n``. | ``string`` | Yes | +|``headers`` | The custom headers of the response. | [[]Action.Return.Header](#actionreturnheader) | No | +{{}} + +\* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing` and `$connections_waiting`. + +### Action.Return.Header + +The header defines an HTTP Header for a canned response in an actionReturn: + +```yaml +name: x-coffee +value: espresso +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. | ``string`` | Yes | +{{}} + +### Action.Proxy + +The proxy action passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). + +In the example below, the request URI is rewritten to `/`, and the request and the response headers are modified: + +```yaml +proxy: + upstream: coffee + requestHeaders: + pass: true + set: + - name: My-Header + value: Value + - name: Client-Cert + value: ${ssl_client_escaped_cert} + responseHeaders: + add: + - name: My-Header + value: Value + - name: IC-Nginx-Version + value: ${nginx_version} + always: true + hide: + - x-internal-version + ignore: + - Expires + - Set-Cookie + pass: + - Server + rewritePath: / +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``upstream`` | The name of the upstream which the requests will be proxied to. The upstream with that name must be defined in the resource. | ``string`` | Yes | +|``requestHeaders`` | The request headers modifications. | [action.Proxy.RequestHeaders](#actionproxyrequestheaders) | No | +|``responseHeaders`` | The response headers modifications. | [action.Proxy.ResponseHeaders](#actionproxyresponseheaders) | No | +|``rewritePath`` | The rewritten URI. If the route path is a regular expression -- starts with `~` -- the `rewritePath` can include capture groups with ``$1-9``. For example `$1` for the first group, and so on. For more information, check the [rewrite](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/rewrites) example. | ``string`` | No | +{{}} + +### Action.Proxy.RequestHeaders + +The RequestHeaders field modifies the headers of the request to the proxied upstream server. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``pass`` | Passes the original request headers to the proxied upstream server. See the [proxy_pass_request_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_headers) directive for more information. Default is true. | ``bool`` | No | +|``set`` | Allows redefining or appending fields to present request headers passed to the proxied upstream servers. See the [proxy_set_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive for more information. | [[]header](#actionproxyrequestheaderssetheader) | No | +{{}} + +### Action.Proxy.RequestHeaders.Set.Header + +The header defines an HTTP Header: + +```yaml +name: My-Header +value: My-Value +``` + +It is possible to override the default value of the `Host` header, which NGINX Ingress Controller sets to [`$host`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_host): + +```yaml +name: Host +value: example.com +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | +{{}} + +\* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` (NGINX Plus only) and `$jwt_header_` (NGINX Plus only). + +### Action.Proxy.ResponseHeaders + +The ResponseHeaders field modifies the headers of the response to the client. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``hide`` | The headers that will not be passed* in the response to the client from a proxied upstream server. See the [proxy_hide_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directive for more information. | ``[]string`` | No | +|``pass`` | Allows passing the hidden header fields* to the client from a proxied upstream server. See the [proxy_pass_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directive for more information. | ``[]string`` | No | +|``ignore`` | Disables processing of certain headers** to the client from a proxied upstream server. See the [proxy_ignore_headers](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ignore_headers) directive for more information. | ``[]string`` | No | +|``add`` | Adds headers to the response to the client. | [[]addHeader](#addheader) | No | +{{}} + +\* -- Default hidden headers are: `Date`, `Server`, `X-Pad` and `X-Accel-...`. + +\** -- The following fields can be ignored: `X-Accel-Redirect`, `X-Accel-Expires`, `X-Accel-Limit-Rate`, `X-Accel-Buffering`, `X-Accel-Charset`, `Expires`, `Cache-Control`, `Set-Cookie` and `Vary`. + +### AddHeader + +The addHeader defines an HTTP Header with an optional `always` field: + +```yaml +name: My-Header +value: My-Value +always: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | +|``always`` | If set to true, add the header regardless of the response status code**. Default is false. See the [add_header](http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header) directive for more information. | ``bool`` | No | +{{}} + +\* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` (NGINX Plus only) and `$jwt_header_` (NGINX Plus only). + +{{< note >}} If `always` is false, the response header is added only if the response status code is any of `200`, `201`, `204`, `206`, `301`, `302`, `303`, `304`, `307` or `308`. {{< /note >}} + +### Split + +The split defines a weight for an action as part of the splits configuration. + +In the example below NGINX passes 80% of requests to the upstream `coffee-v1` and the remaining 20% to `coffee-v2`: + +```yaml +splits: +- weight: 80 + action: + pass: coffee-v1 +- weight: 20 + action: + pass: coffee-v2 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``weight`` | The weight of an action. Must fall into the range ``0..100``. The sum of the weights of all splits must be equal to ``100``. | ``int`` | Yes | +|``action`` | The action to perform for a request. | [action](#action) | Yes | +{{}} + +### Match + +The match defines a match between conditions and an action or splits. + +In the example below, NGINX routes requests with the path `/coffee` to different upstreams based on the value of the cookie `user`: + +- `user=john` -> `coffee-future` +- `user=bob` -> `coffee-deprecated` +- If the cookie is not set or not equal to either `john` or `bob`, NGINX routes to `coffee-stable` + +```yaml +path: /coffee +matches: +- conditions: + - cookie: user + value: john + action: + pass: coffee-future +- conditions: + - cookie: user + value: bob + action: + pass: coffee-deprecated +action: + pass: coffee-stable +``` + +In the next example, NGINX routes requests based on the value of the built-in [`$request_method` variable](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_method), which represents the HTTP method of a request: + +- all POST requests -> `coffee-post` +- all non-POST requests -> `coffee` + +```yaml +path: /coffee +matches: +- conditions: + - variable: $request_method + value: POST + action: + pass: coffee-post +action: + pass: coffee +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``conditions`` | A list of conditions. Must include at least 1 condition. | [[]condition](#condition) | Yes | +|``action`` | The action to perform for a request. | [action](#action) | No | +|``splits`` | The splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | +{{}} + +{{< note >}} A match must include exactly one of the following: `action` or `splits`. {{< /note >}} + +### Condition + +The condition defines a condition in a match. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``header`` | The name of a header. Must consist of alphanumeric characters or ``-``. | ``string`` | No | +|``cookie`` | The name of a cookie. Must consist of alphanumeric characters or ``_``. | ``string`` | No | +|``argument`` | The name of an argument. Must consist of alphanumeric characters or ``_``. | ``string`` | No | +|``variable`` | The name of an NGINX variable. Must start with ``$``. See the list of the supported variables below the table. | ``string`` | No | +|``value`` | The value to match the condition against. How to define a value is shown below the table. | ``string`` | Yes | +{{}} + +{{< note >}} a condition must include exactly one of the following: `header`, `cookie`, `argument` or `variable`. {{< /note >}} + +Supported NGINX variables: `$args`, `$http2`, `$https`, `$remote_addr`, `$remote_port`, `$query_string`, `$request`, `$request_body`, `$request_uri`, `$request_method`, `$scheme`. Find the documentation for each variable [here](https://nginx.org/en/docs/varindex.html). + +The value supports two kinds of matching: + +- *Case-insensitive string comparison*. For example: + - `john` -- case-insensitive matching that succeeds for strings, such as `john`, `John`, `JOHN`. + - `!john` -- negation of the case-insensitive matching for john that succeeds for strings, such as `bob`, `anything`, `''` (empty string). +- *Matching with a regular expression*. Note that NGINX supports regular expressions compatible with those used by the Perl programming language (PCRE). For example: + - `~^yes` -- a case-sensitive regular expression that matches any string that starts with `yes`. For example: `yes`, `yes123`. + - `!~^yes` -- negation of the previous regular expression that succeeds for strings like `YES`, `Yes123`, `noyes`. (The negation mechanism is not part of the PCRE syntax). + - `~*no$` -- a case-insensitive regular expression that matches any string that ends with `no`. For example: `no`, `123no`, `123NO`. + +{{< note >}} A value must not include any unescaped double quotes (`"`) and must not end with an unescaped backslash (`\`). For example, the following are invalid values: `some"value`, `somevalue\`. {{< /note >}} + +### ErrorPage + +The errorPage defines a custom response for a route for the case when either an upstream server responds with (or NGINX generates) an error status code. The custom response can be a redirect or a canned response. See the [error_page](https://nginx.org/en/docs/http/ngx_http_core_module.html#error_page) directive for more information. + +```yaml +path: /coffee +errorPages: +- codes: [502, 503] + redirect: + code: 301 + url: https://nginx.org +- codes: [404] + return: + code: 200 + body: "Original resource not found, but success!" +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``codes`` | A list of error status codes. | ``[]int`` | Yes | +|``redirect`` | The redirect action for the given status codes. | [errorPage.Redirect](#errorpageredirect) | No | +|``return`` | The canned response action for the given status codes. | [errorPage.Return](#errorpagereturn) | No | +{{}} + +{{< note >}} An errorPage must include exactly one of the following: `return` or `redirect`. {{< /note >}} + +### ErrorPage.Redirect + +The redirect defines a redirect for an errorPage. + +In the example below, NGINX responds with a redirect when a response from an upstream server has a 404 status code. + +```yaml +codes: [404] +redirect: + code: 301 + url: ${scheme}://cafe.example.com/error.html +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | +|``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` and ``$http_x_forwarded_proto``. Variables must be enclosed in curly braces. For example: ``${scheme}``. | ``string`` | Yes | +{{}} + +### ErrorPage.Return + +The return defines a canned response for an errorPage. + +In the example below, NGINX responds with a canned response when a response from an upstream server has either 401 or 403 status code. + +```yaml +codes: [401, 403] +return: + code: 200 + type: application/json + body: | + {\"msg\": \"You don't have permission to do this\"} + headers: + - name: x-debug-original-statuses + value: ${upstream_status} +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``code`` | The status code of the response. The default is the status code of the original response. | ``int`` | No | +|``type`` | The MIME type of the response. The default is ``text/html``. | ``string`` | No | +|``body`` | The body of the response. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | Yes | +|``headers`` | The custom headers of the response. | [[]errorPage.Return.Header](#errorpagereturnheader) | No | +{{}} + +### ErrorPage.Return.Header + +The header defines an HTTP Header for a canned response in an errorPage: + +```yaml +name: x-debug-original-statuses +value: ${upstream_status} +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | No | +{{}} + +## Using VirtualServer and VirtualServerRoute + +You can use the usual `kubectl` commands to work with VirtualServer and VirtualServerRoute resources, similar to Ingress resources. + +For example, the following command creates a VirtualServer resource defined in `cafe-virtual-server.yaml` with the name `cafe`: + +```shell +kubectl apply -f cafe-virtual-server.yaml +``` +```text +virtualserver.k8s.nginx.org "cafe" created +``` + +You can get the resource by running: + +```shell +kubectl get virtualserver cafe +``` +```text +NAME STATE HOST IP PORTS AGE +cafe Valid cafe.example.com 12.13.23.123 [80,443] 3m +``` + +In `kubectl get` and similar commands, you can use the short name `vs` instead of `virtualserver`. + +Similarly, for VirtualServerRoute you can use `virtualserverroute` or the short name `vsr`. + +### Using Snippets + +Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure several NGINX features in a VirtualServer: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: cafe + namespace: cafe +spec: + http-snippets: | + limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s; + proxy_cache_path /tmp keys_zone=one:10m; + host: cafe.example.com + tls: + secret: cafe-secret + server-snippets: | + limit_req zone=mylimit burst=20; + upstreams: + - name: tea + service: tea-svc + port: 80 + - name: coffee + service: coffee-svc + port: 80 + routes: + - path: /tea + location-snippets: | + proxy_cache one; + proxy_cache_valid 200 10m; + action: + pass: tea + - path: /coffee + action: + pass: coffee +``` + +For additional information, view the [Advanced configuration with Snippets]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) topic. + +### Validation + +Two types of validation are available for VirtualServer and VirtualServerRoute resources: + +- *Structural validation* by the `kubectl` and Kubernetes API server. +- *Comprehensive validation* by NGINX Ingress Controller. + +#### Structural Validation + +The custom resource definitions for VirtualServer and VirtualServerRoute include structural OpenAPI schema which describes the type of every field of those resources. + +If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f cafe-virtual-server.yaml + ``` + ```text + error: error validating "cafe-virtual-server.yaml": error validating data: ValidationError(VirtualServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.VirtualServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f cafe-virtual-server.yaml --validate=false + ``` + ```text + The VirtualServer "cafe" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: + spec.upstreams.port in body must be of type integer: "string" + ``` + +If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. + +#### Comprehensive Validation + +NGINX Ingress Controller validates the fields of the VirtualServer and VirtualServerRoute resources. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can check if NGINX Ingress Controller successfully applied the configuration for a VirtualServer. For our example `cafe` VirtualServer, we can run: + +```shell +kubectl describe vs cafe +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated +``` + +Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a VirtualServer `cafe` with two upstream with the same name `tea`, you will get: + +```shell +kubectl describe vs cafe +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 12s nginx-ingress-controller VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" +``` + +Note how the events section includes a Warning event with the Rejected reason. + +Additionally, this information is also available in the `status` field of the VirtualServer resource. Note the Status section of the VirtualServer: + +```shell +kubectl describe vs cafe +``` +```text +... +Status: + External Endpoints: + Ip: 12.13.23.123 + Ports: [80,443] + Message: VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" + Reason: Rejected + State: Invalid +``` + +NGINX Ingress Controller validates VirtualServerRoute resources in a similar way. + +**Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. + +## Customization using ConfigMap + +You can customize the NGINX configuration for VirtualServer and VirtualServerRoutes resources using the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource). Most of the ConfigMap keys are supported, with the following exceptions: + +- `proxy-hide-headers` +- `proxy-pass-headers` +- `hsts` +- `hsts-max-age` +- `hsts-include-subdomains` +- `hsts-behind-proxy` +- `redirect-to-https` +- `ssl-redirect` diff --git a/content/includes/nic/installation/create-common-resources.md b/content/includes/nic/installation/create-common-resources.md new file mode 100644 index 000000000..7cfe90e48 --- /dev/null +++ b/content/includes/nic/installation/create-common-resources.md @@ -0,0 +1,27 @@ +--- +docs: DOCS-1464 +--- + +In this section, you'll create resources that most NGINX Ingress Controller installations require: + +1. (Optional) Create a secret for the default NGINX server's TLS certificate and key. Complete this step only if you're using the [default server TLS secret]({{< ref "/nic/configuration/global-configuration/command-line-arguments#cmdoption-default-server-tls-secret.md" >}}) command-line argument. If you're not, feel free to skip this step. + + By default, the server returns a _404 Not Found_ page for all requests when no ingress rules are set up. Although we provide a self-signed certificate and key for testing purposes, we recommend using your own certificate. + + ```shell + kubectl apply -f examples/shared-examples/default-server-secret/default-server-secret.yaml + ``` + +2. Create a ConfigMap to customize your NGINX settings: + + ```shell + kubectl apply -f deployments/common/nginx-config.yaml + ``` + +3. Create an `IngressClass` resource. NGINX Ingress Controller won't start without an `IngressClass` resource. + + ```shell + kubectl apply -f deployments/common/ingress-class.yaml + ``` + + If you want to make this NGINX Ingress Controller instance your cluster's default, uncomment the `ingressclass.kubernetes.io/is-default-class` annotation. This action will auto-assign `IngressClass` to new ingresses that don't specify an `ingressClassName`. diff --git a/content/includes/nic/installation/create-custom-resources.md b/content/includes/nic/installation/create-custom-resources.md new file mode 100644 index 000000000..d7ad9ec27 --- /dev/null +++ b/content/includes/nic/installation/create-custom-resources.md @@ -0,0 +1,49 @@ +--- +docs: DOCS-1463 +--- + +To make sure your NGINX Ingress Controller pods reach the `Ready` state, you'll need to create custom resource definitions (CRDs) for various components. + +Alternatively, you can disable this requirement by setting the `-enable-custom-resources` command-line argument to `false`. + +There are two ways you can install the custom resource definitions: + +1. Using a URL to apply a single CRD yaml file, which we recommend. +1. Applying your local copy of the CRD yaml files, which requires you to clone the repository. + +The core custom CRDs are the following: + +- [VirtualServer and VirtualServerRoute]({{< ref "/nic/configuration/virtualserver-and-virtualserverroute-resources.md" >}}) +- [TransportServer]({{< ref "/nic/configuration/transportserver-resource.md" >}}) +- [Policy]({{< ref "/nic/configuration/policy-resource.md" >}}) +- [GlobalConfiguration]({{< ref "/nic/configuration/global-configuration/globalconfiguration-resource.md" >}}) + +{{}} + +{{%tab name="Install CRDs from single YAML"%}} + +```shell +kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v{{< nic-version >}}/deploy/crds.yaml +``` + +{{%/tab%}} + +{{%tab name="Install CRDs after cloning the repo"%}} + +{{< note >}} If you are installing the CRDs this way, ensure you have first cloned the repository. {{< /note >}} + +{{< note >}} Please make sure to read the steps outlined in [Upgrade to V4](https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/upgrade-to-v4/#update-custom-resource-apiversion) before running the CRD upgrade and perform the steps if applicable. +{{< /note >}} + + +```shell +kubectl apply -f config/crd/bases/k8s.nginx.org_virtualservers.yaml +kubectl apply -f config/crd/bases/k8s.nginx.org_virtualserverroutes.yaml +kubectl apply -f config/crd/bases/k8s.nginx.org_transportservers.yaml +kubectl apply -f config/crd/bases/k8s.nginx.org_policies.yaml +kubectl apply -f config/crd/bases/k8s.nginx.org_globalconfigurations.yaml +``` + +{{%/tab%}} + +{{}} diff --git a/content/includes/nic/installation/deploy-controller.md b/content/includes/nic/installation/deploy-controller.md new file mode 100644 index 000000000..5723c73ae --- /dev/null +++ b/content/includes/nic/installation/deploy-controller.md @@ -0,0 +1,10 @@ +--- +docs: DOCS-1462 +--- + +You have two options for deploying NGINX Ingress Controller: + +- **Deployment**. Choose this method for the flexibility to dynamically change the number of NGINX Ingress Controller replicas. +- **DaemonSet**. Choose this method if you want NGINX Ingress Controller to run on all nodes or a subset of nodes. + +Before you start, update the [command-line arguments]({{< ref "/nic/configuration/global-configuration/command-line-arguments.md" >}}) for the NGINX Ingress Controller container in the relevant manifest file to meet your specific requirements. diff --git a/content/includes/nic/installation/download-jwt.md b/content/includes/nic/installation/download-jwt.md new file mode 100644 index 000000000..d89c65a43 --- /dev/null +++ b/content/includes/nic/installation/download-jwt.md @@ -0,0 +1,10 @@ +--- +docs: "DOCS-000" +--- + +1. Log in to [MyF5](https://my.f5.com/manage/s/). +2. Go to **My Products & Plans > Subscriptions** to see your active subscriptions. +3. Find your NGINX products or services subscription, and select the **Subscription ID** for details. +4. Download the **JSON Web Token (JWT)** from the subscription page. + +{{< note >}} The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. A regular NGINX Plus instance JWT must be used. {{< /note >}} diff --git a/content/includes/nic/installation/jwt-password-note.md b/content/includes/nic/installation/jwt-password-note.md new file mode 100644 index 000000000..147c67af3 --- /dev/null +++ b/content/includes/nic/installation/jwt-password-note.md @@ -0,0 +1,11 @@ +--- +docs: +--- + +{{< note >}} For security, follow these practices with JSON Web Tokens (JWTs), passwords, and shell history: + +1. **JWTs:** JWTs are sensitive information. Store them securely. Delete them after use to prevent unauthorized access. + +1. **Shell history:** Commands that include JWTs or passwords are recorded in the history of your shell, in plain text. Clear your shell history after running such commands. For example, if you use bash, you can delete commands in your `~/.bash_history` file. Alternatively, you can run the `history -c` command to erase your shell history. + +Follow these practices to help ensure the security of your system and data. {{< /note >}} diff --git a/content/includes/nic/installation/manifests/daemonset.md b/content/includes/nic/installation/manifests/daemonset.md new file mode 100644 index 000000000..c19376eb2 --- /dev/null +++ b/content/includes/nic/installation/manifests/daemonset.md @@ -0,0 +1,21 @@ +--- +docs: DOCS-1465 +--- + +For additional context on managing containers using Kubernetes DaemonSets, refer to the official Kubernetes [DaemonSets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) documentation. + +When you deploy NGINX Ingress Controller as a DaemonSet, Kubernetes creates an Ingress Controller pod on every node in the cluster. + +- For NGINX, run: + + ```shell + kubectl apply -f deployments/daemon-set/nginx-ingress.yaml + ``` + +- For NGINX Plus, run: + + ```shell + kubectl apply -f deployments/daemon-set/nginx-plus-ingress.yaml + ``` + + Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. diff --git a/content/includes/nic/installation/manifests/deployment.md b/content/includes/nic/installation/manifests/deployment.md new file mode 100644 index 000000000..3786a2256 --- /dev/null +++ b/content/includes/nic/installation/manifests/deployment.md @@ -0,0 +1,21 @@ +--- +docs: DOCS-1467 +--- + +For additional context on managing containers using Kubernetes Deployments, refer to the official Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) documentation. + +When you deploy NGINX Ingress Controller as a Deployment, Kubernetes automatically sets up a single NGINX Ingress Controller pod. + +- For NGINX, run: + + ```shell + kubectl apply -f deployments/deployment/nginx-ingress.yaml + ``` + +- For NGINX Plus, run: + + ```shell + kubectl apply -f deployments/deployment/nginx-plus-ingress.yaml + ``` + + Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. diff --git a/content/includes/nic/installation/manifests/verify-pods-are-running.md b/content/includes/nic/installation/manifests/verify-pods-are-running.md new file mode 100644 index 000000000..9f11fa7b0 --- /dev/null +++ b/content/includes/nic/installation/manifests/verify-pods-are-running.md @@ -0,0 +1,10 @@ +--- +docs: DOCS-1466 +--- + + +To confirm the NGINX Ingress Controller pods are operational, run: + +```shell +kubectl get pods --namespace=nginx-ingress +``` diff --git a/content/includes/nic/rbac/set-up-rbac.md b/content/includes/nic/rbac/set-up-rbac.md new file mode 100644 index 000000000..a8bd5a176 --- /dev/null +++ b/content/includes/nic/rbac/set-up-rbac.md @@ -0,0 +1,33 @@ +--- +docs: DOCS-1468 +--- + +{{}}To complete these steps you need admin access to your cluster. Refer to to your Kubernetes platform's documentation to set up admin access. For Google Kubernetes Engine (GKE), you can refer to their [Role-Based Access Control guide](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control).{{}} + +1. Create a namespace and a service account: + + ```shell + kubectl apply -f deployments/common/ns-and-sa.yaml + ``` + +2. Create a cluster role and binding for the service account: + + ```shell + kubectl apply -f deployments/rbac/rbac.yaml + ``` + +
+ +If you're planning to use NGINX App Protect or NGINX App Protect DoS, additional roles and bindings are needed. + +1. (NGINX App Protect only) Create the *App Protect* role and binding: + + ```shell + kubectl apply -f deployments/rbac/ap-rbac.yaml + ``` + +2. (NGINX App Protect DoS only) Create the *App Protect DoS* role and binding: + + ```shell + kubectl apply -f deployments/rbac/apdos-rbac.yaml + ``` diff --git a/content/nic/_index.md b/content/nic/_index.md new file mode 100644 index 000000000..8d93ce681 --- /dev/null +++ b/content/nic/_index.md @@ -0,0 +1,5 @@ +--- +title: NGINX Ingress Controller +menu: docs +url: /nginx-ingress-controller/ +--- diff --git a/content/nic/community.md b/content/nic/community.md new file mode 100644 index 000000000..4a10a6083 --- /dev/null +++ b/content/nic/community.md @@ -0,0 +1,23 @@ +--- +title: Community and contributing +weight: 2200 +nd-content-type: reference +nd-product: NIC +nd-docs: DOCS-1447 +--- + +There are a few ways to get involved with the F5 NGINX Ingress Controller community and contribute to the project. + +# Community + +- Our [GitHub issues page](https://github.com/nginx/kubernetes-ingress/issues) or [GitHub discussions page](https://github.com/nginx/kubernetes-ingress/discussions) offers more space for an asynchronous technical discussion. + +- You can can also get help through the [NGINX Community Forum](https://community.nginx.org/). + +# Contribute + +Please see our [contributing guide](https://github.com/nginx/kubernetes-ingress/blob/main/CONTRIBUTING.md) to get involved with code or documentation. + +# License + +[Apache License, Version 2.0](https://github.com/nginx/kubernetes-ingress/blob/main/LICENSE) diff --git a/content/nic/configuration/_index.md b/content/nic/configuration/_index.md new file mode 100644 index 000000000..5554b2f83 --- /dev/null +++ b/content/nic/configuration/_index.md @@ -0,0 +1,4 @@ +--- +title: Configuration +weight: 1400 +--- diff --git a/content/nic/configuration/access-control.md b/content/nic/configuration/access-control.md new file mode 100644 index 000000000..54fee5f91 --- /dev/null +++ b/content/nic/configuration/access-control.md @@ -0,0 +1,120 @@ +--- +title: Deploy a Policy for access control +weight: 900 +toc: true +docs: DOCS-000 +--- + +This topic describes how to use F5 NGINX Ingress Controller to apply and update a Policy for access control. It demonstrates it using an example application and a [VirtualServer custom resource]({{< ref "/nic/configuration/virtualserver-and-virtualserverroute-resources.md" >}}). + +--- + +## Before you begin + +You should have a [working NGINX Ingress Controller]({{< ref "/nic/installation/installing-nic/installation-with-helm.md" >}}) instance. + +For ease of use in shell commands, set two shell variables: + +1. The public IP address for your NGINX Ingress Controller instance. + +```shell +IC_IP= +``` + +2. The HTTP port of the same instance. + +```shell +IC_HTTP_PORT= +``` + +--- + +## Deploy the example application + +Create the file _webapp.yaml_ with the following contents: + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/webapp.yaml" >}} + +Apply it using `kubectl`: + +```shell +kubectl apply -f webapp.yaml +``` + +--- + +## Deploy a Policy to create a deny rule + +Create a file named _access-control-policy-deny.yaml_. The highlighted _deny_ field will be used by the example application, and should be changed to the subnet of your machine. + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/access-control-policy-deny.yaml" "hl_lines=7-8" >}} + +Apply the policy: + +```shell +kubectl apply -f access-control-policy-deny.yaml +``` + +--- + +## Configure load balancing + +Create a file named _virtual-server.yaml_ for the VirtualServer resource. The _policies_ field references the access control Policy created in the previous section. + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/virtual-server.yaml" "hl_lines=7-8" >}} + +Apply the policy: + +```shell +kubectl apply -f virtual-server.yaml +``` + +--- + +## Test the example application + +Use `curl` to attempt to access the application: + +```shell +curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT +``` +```text + +403 Forbidden + +

403 Forbidden

+ + +``` + +The *403* response is expected, successfully blocking your machine. + +--- + +## Update the Policy to create an allow rule + +Update the Policy with the file _access-control-policy-allow.yaml_, setting the _allow_ field to the subnet of your machine. + +{{< ghcode "https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/access-control-policy-allow.yaml" "hl_lines=7-8" >}} + +Apply the Policy: + +```shell +kubectl apply -f access-control-policy-allow.yaml +``` + +---- + +## Verify the Policy update + +Attempt to access the application again: + +```shell +curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT +``` +```text +Server address: 10.64.0.13:8080 +Server name: webapp-5cbbc7bd78-wf85w +``` + +The successful response demonstrates that the policy has been updated. diff --git a/content/nic/configuration/configuration-examples.md b/content/nic/configuration/configuration-examples.md new file mode 100644 index 000000000..0a0e6e3bb --- /dev/null +++ b/content/nic/configuration/configuration-examples.md @@ -0,0 +1,13 @@ +--- +docs: DOCS-584 +doctypes: +- '' +title: Configuration examples +toc: true +weight: 400 +--- + +Our [GitHub repo](https://github.com/nginx/kubernetes-ingress) includes a number of configuration examples: + +- [*Examples of Custom Resources*](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources) show how to advanced NGINX features by using VirtualServer, VirtualServerRoute, TransportServer and Policy Custom Resources. +- [*Examples of Ingress Resources*](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources) show how to use advanced NGINX features in Ingress resources with annotations. diff --git a/content/nic/configuration/global-configuration/_index.md b/content/nic/configuration/global-configuration/_index.md new file mode 100644 index 000000000..4f71e2f5f --- /dev/null +++ b/content/nic/configuration/global-configuration/_index.md @@ -0,0 +1,8 @@ +--- +title: Global configuration +description: +weight: 100 +menu: + docs: + parent: NGINX Ingress Controller +--- diff --git a/content/nic/configuration/global-configuration/command-line-arguments.md b/content/nic/configuration/global-configuration/command-line-arguments.md new file mode 100644 index 000000000..79d8f3f02 --- /dev/null +++ b/content/nic/configuration/global-configuration/command-line-arguments.md @@ -0,0 +1,685 @@ +--- +docs: DOCS-585 +doctypes: +- '' +title: Command-line arguments +toc: true +weight: 100 +--- + +F5 NGINX Ingress Controller supports several command-line arguments, which are set based on installation method: + +- If you're using *Kubernetes Manifests* to install NGINX Ingress Controller, modify the Manifests to set the command-line arguments. View the [Installation with Manifests]({{}}) topic for more information. +- If you're using *Helm* to install NGINX Ingress Controller, modify the parameters of the Helm chart to set the command-line arguments. View the [Installation with Helm]({{}}) topic for more information. + + + +### -enable-snippets + +Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources. + +Default `false`. + + + +--- + +### -default-server-tls-secret `` + +Secret with a TLS certificate and key for TLS termination of the default server. + +- If not set, certificate and key in the file `/etc/nginx/secrets/default` are used. +- If `/etc/nginx/secrets/default` doesn't exist, NGINX Ingress Controller will configure NGINX to reject TLS connections to the default server. +- If a secret is set, but NGINX Ingress Controller is not able to fetch it from Kubernetes API, or it is not set and NGINX Ingress Controller fails to read the file "/etc/nginx/secrets/default", NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -wildcard-tls-secret `` + +A Secret with a TLS certificate and key for TLS termination of every Ingress/VirtualServer host for which TLS termination is enabled but the Secret is not specified. + +- If the argument is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection +- If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -enable-custom-resources + +Enables custom resources. + +Default `true`. + + + + +--- + +### -enable-oidc + +Enables OIDC policies. + +Default `false`. + + + +--- + +### -enable-leader-election + +Enables Leader election to avoid multiple replicas of the controller reporting the status of Ingress, VirtualServer and VirtualServerRoute resources -- only one replica will report status. +Default `true`. + +See [-report-ingress-status](#cmdoption-report-ingress-status) flag. + + + +--- + +### -enable-tls-passthrough + +Enable TLS Passthrough on port 443. + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -tls-passthrough-port `` + +Set the port for TLS Passthrough. +Format: `[1024 - 65535]` (default `443`) + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -enable-cert-manager + +Enable x509 automated certificate management for VirtualServer resources using cert-manager (cert-manager.io). + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -enable-external-dns + +Enable integration with ExternalDNS for configuring public DNS entries for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + +--- + +### -external-service `` + +Specifies the name of the service with the type LoadBalancer through which the NGINX Ingress Controller pods are exposed externally. The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. + +For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). + + + +--- + +### -ingresslink `` + +Specifies the name of the IngressLink resource, which exposes the NGINX Ingress Controller pods via a BIG-IP system. The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. + +For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). + + + +--- + +### -global-configuration `` + +A GlobalConfiguration resource for global configuration of NGINX Ingress Controller. + +Format: `/` + +Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). + + + +--- + +### -health-status + +Adds a location "/nginx-health" to the default server. The location responds with the 200 status code for any request. + +Useful for external health-checking of NGINX Ingress Controller. + + + +--- + +### -health-status-uri `` + +Sets the URI of health status location in the default server. Requires [-health-status](#cmdoption-health-status). (default `/nginx-health`) + + + +--- + +### -ingress-class `` + +The `-ingress-class` argument refers to the name of the resource `kind: IngressClass`. An IngressClass resource with a name equal to the class must be deployed. Otherwise, NGINX Ingress Controller will fail to start. +NGINX Ingress Controller will only process Ingress resources that belong to its class (Whose `ingressClassName` value matches the value of `-ingress-class`), skipping the ones without it. It will also process all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the `ingressClassName` field. + +Default `nginx`. + + + +--- + +### -ingress-template-path `` + +Path to the ingress NGINX configuration template for an ingress resource. Default for NGINX is `nginx.ingress.tmpl`; default for NGINX Plus is `nginx-plus.ingress.tmpl`. + + + +--- + +### -leader-election-lock-name `` + +Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. + +Requires [-enable-leader-election](#cmdoption-enable-leader-election). + + + +--- + +### -log_backtrace_at `` + +When logging hits line `file:N`, emit a stack trace. + + + +--- + +### -main-template-path `` + +Path to the main NGINX configuration template. + +- Default for NGINX is `nginx.tmpl`. +- Default for NGINX Plus is `nginx-plus.tmpl`. + + + +--- + +### -nginx-configmaps `` + +A ConfigMap resource for customizing NGINX configuration. If a ConfigMap is set, but NGINX Ingress Controller is not able to fetch it from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -mgmt-configmap `` + +The Management ConfigMap resource is used for customizing the NGINX mgmt block. If using NGINX Plus, a Management ConfigMap must be set. If NGINX Ingress Controller is not able to fetch it from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -nginx-debug + +Enable debugging for NGINX. Uses the nginx-debug binary. Requires 'error-log-level: debug' in the ConfigMap. + + + +--- + +### -nginx-plus + +Enable support for NGINX Plus. + + + +--- + +### -nginx-reload-timeout `` + +Timeout in milliseconds which NGINX Ingress Controller will wait for a successful NGINX reload after a change or at the initial start. + +Default is 60000. + + + +--- + +### -nginx-status + +Enable the NGINX stub_status, or the NGINX Plus API. + +Default `true`. + + + +--- + +### -nginx-status-allow-cidrs `` + +Add IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. + +Separate multiple IP/CIDR by commas. (default `127.0.0.1,::1`) + + + +--- + +### -nginx-status-port `` + +Set the port where the NGINX stub_status or the NGINX Plus API is exposed. + +Format: `[1024 - 65535]` (default `8080`) + + + +--- + +### -proxy `` + +{{< warning >}} This argument is intended for testing purposes only. {{< /warning >}} + +Use a proxy server to connect to Kubernetes API started with `kubectl proxy`. + +NGINX Ingress Controller does not start NGINX and does not write any generated NGINX configuration files to disk. + + + +--- + +### -report-ingress-status + +Updates the address field in the status of Ingress resources. + +Requires the [-external-service](#cmdoption-external-service) or [-ingresslink](#cmdoption-ingresslink) flag, or the `external-status-address` key in the ConfigMap. + + + +--- + +### -transportserver-template-path `` + +Path to the TransportServer NGINX configuration template for a TransportServer resource. + +- Default for NGINX is `nginx.transportserver.tmpl`. +- Default for NGINX Plus is `nginx-plus.transportserver.tmpl`. + + + +--- + +### -log-level `` + +Log level for Ingress Controller logs. Allowed values: fatal, error, warn, info, debug, trace. + +- Default is `info`. + + + +--- + +### -log-format `` + +Log format for Ingress Controller logs. Allowed values: glog, json, text. + +- Default is `glog`. + + + +--- + +### -version + +Print the version, git-commit hash and build date and exit. + + + +--- + +### -virtualserver-template-path `` + +Path to the VirtualServer NGINX configuration template for a VirtualServer resource. + +- Default for NGINX is `nginx.virtualserver.tmpl`. +- Default for NGINX Plus is `nginx-plus.virtualserver.tmpl`. + + + + +--- + +### -vmodule `` + +A comma-separated list of pattern=N settings for file-filtered logging. + + + +--- + +### -watch-namespace `` + +Comma separated list of namespaces NGINX Ingress Controller should watch for resources. By default NGINX Ingress Controller watches all namespaces. Mutually exclusive with "watch-namespace-label". + + + +--- + +### -watch-namespace-label `` + +Configures NGINX Ingress Controller to watch only those namespaces with label foo=bar. By default NGINX Ingress Controller watches all namespaces. Mutually exclusive with "watch-namespace". + + + +--- + +### -watch-secret-namespace `` + +Comma separated list of namespaces NGINX Ingress Controller should watch for secrets. If this arg is not configured, NGINX Ingress Controller watches the same namespaces for all resources, see "watch-namespace" and "watch-namespace-label". All namespaces included with this argument must be part of either `-watch-namespace` or `-watch-namespace-label`. + + + +--- + +### -enable-prometheus-metrics + +Enables exposing NGINX or NGINX Plus metrics in the Prometheus format. + + + +--- + +### -prometheus-metrics-listen-port `` + +Sets the port where the Prometheus metrics are exposed. + +Format: `[1024 - 65535]` (default `9113`) + + + +--- + +### -prometheus-tls-secret `` + +A Secret with a TLS certificate and key for TLS termination of the Prometheus metrics endpoint. + +- If the argument is not set, the Prometheus endpoint will not use a TLS connection. +- If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. + + + +--- + +### -enable-service-insight + +Exposes the Service Insight endpoint for Ingress Controller. + + + +--- + +### -service-insight-listen-port `` + +Sets the port where the Service Insight is exposed. + +Format: `[1024 - 65535]` (default `9114`) + + + +--- + +### -service-insight-tls-secret `` + +A Secret with a TLS certificate and key for TLS termination of the Service Insight endpoint. + +- If the argument is not set, the Service Insight endpoint will not use a TLS connection. +- If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. + +Format: `/` + + + +--- + +### -spire-agent-address `` + +Specifies the address of a running Spire agent. **For use with NGINX Service Mesh only**. + +- If the argument is set, but NGINX Ingress Controller is unable to connect to the Spire Agent, NGINX Ingress Controller will fail to start. + + + + +--- + +### -enable-internal-routes + +Enable support for internal routes with NGINX Service Mesh. **For use with NGINX Service Mesh only**. + +Requires [-spire-agent-address](#cmdoption-spire-agent-address). + +- If the argument is set, but `spire-agent-address` is not provided, NGINX Ingress Controller will fail to start. + + + +--- + +### -enable-latency-metrics + +Enable collection of latency metrics for upstreams. +Requires [-enable-prometheus-metrics](#cmdoption-enable-prometheus-metrics). + + + +--- + +### -enable-app-protect + +Enables support for App Protect. + +Requires [-nginx-plus](#cmdoption-nginx-plus). + +- If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-log-level `` + +Sets log level for App Protect. Allowed values: fatal, error, warn, info, debug, trace. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect](#cmdoption-enable-app-protect). + +- If the argument is set, but `nginx-plus` and `enable-app-protect` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -enable-app-protect-dos + +Enables support for App Protect DoS. + +Requires [-nginx-plus](#cmdoption-nginx-plus). + +- If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-debug + +Enable debugging for App Protect DoS. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-max-daemons + +Max number of ADMD instances. + +Default `1`. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-max-workers + +Max number of nginx processes to support. + +Default `Number of CPU cores in the machine`. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -app-protect-dos-memory + +RAM memory size to consume in MB + +Default `50% of free RAM in the container or 80MB, the smaller`. + +Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). + +- If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. + + + +--- + +### -ready-status + +Enables the readiness endpoint `/nginx-ready`. The endpoint returns a success code when NGINX has loaded all the config after the startup. + +Default `true`. + + + +--- + +### -ready-status-port + +The HTTP port for the readiness endpoint. + +Format: `[1024 - 65535]` (default `8081`) + +--- + +### -disable-ipv6 + +Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack. + +Default `false`. + + + +--- + +### -default-http-listener-port + +Sets the port for the HTTP `default_server` listener. + +Default `80`. + + + +--- + +### -default-https-listener-port + +Sets the port for the HTTPS `default_server` listener. + +Default `443`. + + + +--- + +### -ssl-dynamic-reload + +Used to activate or deactivate lazy loading for SSL Certificates. + +The default value is `true`. + + + +--- + +### -weight-changes-dynamic-reload + +Enables the ability to change the weight distribution of two-way split clients without reloading NGINX. + +Requires [-nginx-plus](#cmdoption-nginx-plus). + +Using this feature may require increasing `map_hash_bucket_size`, `map_hash_max_size`, `variable_hash_bucket_size`, and `variable_hash_max_size` in the ConfigMap based on the number of two-way splits. + +The default value is `false`. + +- If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will ignore the flag. + + + +--- + +### -enable-telemetry-reporting + +Enable gathering and reporting of software telemetry. + +The default value is `true`. + + + +--- + +### -agent + +Enable NGINX Agent which can used with `-enable-app-protect` to send events to Security Monitoring. + +The default value is `false`. + + + +--- + +### -agent-instance-group + +Specify the instance group name to use for the NGINX Ingress Controller deployment when using `-agent`. + + diff --git a/content/nic/configuration/global-configuration/configmap-resource.md b/content/nic/configuration/global-configuration/configmap-resource.md new file mode 100644 index 000000000..711ac7065 --- /dev/null +++ b/content/nic/configuration/global-configuration/configmap-resource.md @@ -0,0 +1,250 @@ +--- +title: ConfigMap resources +weight: 300 +toc: true +type: how-to +product: NIC +docs: DOCS-586 +--- + +When using F5 NGINX Ingress Controller, you can customize or fine tune NGINX behavior using ConfigMap resources. Examples include setting the number of worker processes or customizing the access log format. + +## Using ConfigMap + +1. The [Installation with Manifests]({{< ref "/nic/installation/installing-nic/installation-with-manifests.md" >}}) documentation deploy an empty ConfigMap while the default installation manifests specify it in the command-line arguments of the Ingress Controller. However, if you customized the manifests, to use ConfigMap, make sure to specify the ConfigMap resource to use the [command-line arguments]({{< ref "/nic/configuration/global-configuration/command-line-arguments" >}}) of NGINX Ingress Controller. + +1. Create a ConfigMap file with the name *nginx-config.yaml* and set the values +that make sense for your setup: + + ```yaml + kind: ConfigMap + apiVersion: v1 + metadata: + name: nginx-config + namespace: nginx-ingress + data: + proxy-connect-timeout: "10s" + proxy-read-timeout: "10s" + client-max-body-size: "2m" + ``` + + See the section [Summary of ConfigMap Keys](#summary-of-configmap-keys) for the explanation of the available ConfigMap keys (such as `proxy-connect-timeout` in this example). + +1. Create a new (or update the existing) ConfigMap resource: + + ```shell + kubectl apply -f nginx-config.yaml + ``` + + The NGINX configuration will be updated. + +--- + +## ConfigMap and Ingress annotations + +ConfigMap applies globally, meaning that it affects every Ingress resource. In contrast, annotations always apply to their Ingress resource. Annotations can override some ConfigMap keys: an example is that the `nginx.org/proxy-connect-timeout` annotations overrides the `proxy-connect-timeout` ConfigMap key. + +For more information, view the [Advanced configuration with annotations]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-annotations" >}}) topic. + +--- + +## ConfigMap and VirtualServer/VirtualServerRoute resources + +The ConfigMap affects every VirtualServer and VirtualServerRoute resources. However, the fields of those resources allow overriding some ConfigMap keys. For example, the `connect-timeout` field of the `upstream` overrides the `proxy-connect-timeout` ConfigMap key. + +For more information, view the [VirtualServer and VirtualServerRoute resources]({{< ref "/nic/configuration/virtualserver-and-virtualserverroute-resources" >}}) topic. + +--- + +## ConfigMap keys + +### Ingress Controller (Unrelated to NGINX Configuration) + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*external-status-address* | Sets the address to be reported in the status of Ingress resources. Requires the *-report-status* command-line argument. Overrides the *-external-service* argument. | N/A | [Reporting resource status]({{< ref "/nic/configuration/global-configuration/reporting-resources-status" >}}) | +{{}} + +--- + +### General customization + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | +|*proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | +|*proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | +|*client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | +|*proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | +|*proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | +|*proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | +|*proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | +|*set-real-ip-from* | Sets the value of the [set_real_ip_from](https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from) directive. | N/A | | +|*real-ip-header* | Sets the value of the [real_ip_header](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header) directive. | *X-Real-IP* | | +|*real-ip-recursive* | Enables or disables the [real_ip_recursive](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_recursive) directive. | *False* | | +|*default-server-return* | Configures the [return](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return) directive in the default server, which handles a client request if none of the hosts of Ingress or VirtualServer resources match. The default value configures NGINX to return a 404 error page. You can configure a fixed response or a redirect. For example, *default-server-return: 302 https://nginx.org* will redirect a client to *https://nginx.org*. | *404* | | +|*server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. Additionally, with the NGINX Plus, you can specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | +|*worker-processes* | Sets the value of the [worker_processes](https://nginx.org/en/docs/ngx_core_module.html#worker_processes) directive. | *auto* | | +|*worker-rlimit-nofile* | Sets the value of the [worker_rlimit_nofile](https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile) directive. | N/A | | +|*worker-connections* | Sets the value of the [worker_connections](https://nginx.org/en/docs/ngx_core_module.html#worker_connections) directive. | *1024* | | +|*worker-cpu-affinity* | Sets the value of the [worker_cpu_affinity](https://nginx.org/en/docs/ngx_core_module.html#worker_cpu_affinity) directive. | N/A | | +|*worker-shutdown-timeout* | Sets the value of the [worker_shutdown_timeout](https://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout) directive. | N/A | | +|*server-names-hash-bucket-size* | Sets the value of the [server_names_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size) directive. | *256* | | +|*server-names-hash-max-size* | Sets the value of the [server_names_hash_max_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_max_size) directive. | *1024* | | +|*map-hash-bucket-size* | Sets the value of the [map_hash_bucket_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size) directive.| *256* | | +|*map-hash-max-size* | Sets the value of the [map_hash_max_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_max_size) directive. | *2048* | | +|*resolver-addresses* | Sets the value of the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) addresses. Note: If you use a DNS name (for example, *kube-dns.kube-system.svc.cluster.local* ) as a resolver address, NGINX Plus will resolve it using the system resolver during the start and on every configuration reload. If the name cannot be resolved or the DNS server doesn't respond, NGINX Plus will fail to start or reload. To avoid this, we recommend using IP addresses as resolver addresses instead of DNS names. Supported in NGINX Plus only. | N/A | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*resolver-ipv6* | Enables IPv6 resolution in the resolver. Supported in NGINX Plus only. | *True* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*resolver-valid* | Sets the time NGINX caches the resolved DNS records. Supported in NGINX Plus only. | TTL value of a DNS record | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*resolver-timeout* | Sets the [resolver_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver_timeout) for name resolution. Supported in NGINX Plus only. | *30s* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services). | +|*keepalive-timeout* | Sets the value of the [keepalive_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) directive. | *75s* | | +|*keepalive-requests* | Sets the value of the [keepalive_requests](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests) directive. | *1000* | | +|*variables-hash-bucket-size* | Sets the value of the [variables_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_bucket_size) directive. | *256* | | +|*variables-hash-max-size* | Sets the value of the [variables-hash-max-size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_max_size) directive. | *1024* | | +{{}} + +--- + +### Logging + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*error-log-level* | Sets the global [error log level](https://nginx.org/en/docs/ngx_core_module.html#error_log) for NGINX. | *notice* | | +|*access-log* | Sets the directive [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). A syslog destination is the only valid value. The value will be set to its default in-case user tries to configure it with location other than a syslog. +| ``/dev/stdout main`` | ``syslog:server=localhost:514`` | +|*access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). | *False* | | +|*default-server-access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) for the default server. If access log is disabled globally (*access-log-off: "True"*), then the default server access log is always disabled. | *False* | | +|*log-format* | Sets the custom [log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for HTTP and HTTPS traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx.tmpl) for the access log. | [Custom Log Format](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/shared-examples/custom-log-format). | +|*log-format-escaping* | Sets the characters escaping for the variables of the log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | +|*stream-log-format* | Sets the custom [log format](https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format) for TCP, UDP, and TLS Passthrough traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx.tmpl). | | +|*stream-log-format-escaping* | Sets the characters escaping for the variables of the stream log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | +{{}} + +--- + +### Request URI/Header manipulation + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: *"nginx.org/proxy-hide-headers": "header-a,header-b"* | N/A | | +|*proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: *"nginx.org/proxy-pass-headers": "header-a,header-b"* | N/A | | +{{}} + +--- + +### Auth and SSL/TLS + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*redirect-to-https* | Sets the 301 redirect rule based on the value of the *http_x_forwarded_proto* header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of the Ingress Controller — see [115](https://github.com/nginx/kubernetes-ingress/issues/115) | *False* | | +|*ssl-redirect* | Sets an unconditional 301 redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS. | *True* | | +|*hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/) : the HSTS header is added to the responses from backends. The *preload* directive is included in the header. | *False* | | +|*hsts-max-age* | Sets the value of the *max-age* directive of the HSTS header. | *2592000* (1 month) | | +|*hsts-include-subdomains* | Adds the *includeSubDomains* directive to the HSTS header. | *False* | | +|*hsts-behind-proxy* | Enables HSTS based on the value of the *http_x_forwarded_proto* request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of the Ingress Controller. Note: to control redirection from HTTP to HTTPS configure the *nginx.org/redirect-to-https* annotation. | *False* | | +|*ssl-protocols* | Sets the value of the [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) directive. | *TLSv1 TLSv1.1 TLSv1.2* | | +|*ssl-prefer-server-ciphers* | Enables or disables the [ssl_prefer_server_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers) directive. | *False* | | +|*ssl-ciphers* | Sets the value of the [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) directive. | *HIGH:!aNULL:!MD5* | | +|*ssl-dhparam-file* | Sets the content of the dhparam file. The controller will create the file and set the value of the [ssl_dhparam](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam) directive with the path of the file. | N/A | | +{{}} + +--- + +### Listeners + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*http2* | Enables HTTP/2 in servers with SSL enabled. | *False* | | +|*proxy-protocol* | Enables PROXY Protocol for incoming connections. | *False* | [Proxy Protocol](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/shared-examples/proxy-protocol). | +{{}} + +--- + +### Backend services (Upstreams) + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*lb-method* | Sets the [load balancing method](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method). To use the round-robin method, specify *"round_robin"*. | *"random two least_conn"* | | +|*max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the *server* directive. | *1* | | +|*upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. For NGINX, the special value 0 disables the shared memory zones. For NGINX Plus, shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *256k* for NGINX, *512k* for NGINX Plus | | +|*fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the *server* directive. | *10s* | | +|*keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that *proxy_set_header Connection "";* is added to the generated configuration when the value > 0. | *0* | | +{{}} + +--- + +### Zone Sync + +Zone Sync enables the [ngx_stream_zone_sync_module](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) in NGINX Ingress Controller when NGINX Plus is used. Multiple replicas are required to effectively utililise this functionality. More information is available in the [How NGINX Plus Performs Zone Synchronization](https://docs.nginx.com/nginx/admin-guide/high-availability/zone_sync_details/) topic. + +Zone synchronization with TLS for NGINX Ingress Controller is not yet available with ConfigMap. If you would like to enable Zone Sync with TLS, please remove `zone-sync` from ConfigMap and add Zone Sync parameters via [`stream-snippets`]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) similar to [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-config.yaml) and adding the [zone_sync_ssl directive](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl) along with any other TLS parameters to the `stream-snippets`. + +You will also need to manually add the headless service, such as in [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml). + +{{< caution >}} +If you previously installed OIDC or used the `zone_sync` directive with `stream-snippets` in [v4.0.1](https://github.com/nginx/kubernetes-ingress/tree/v4.0.1) or earlier, and you plan to enable the `zone-sync` ConfigMap key, the `zone_sync` directive should be removed from `stream-snippets`. + +If you encounter the error `error [emerg] 13#13: "zone_sync" directive is duplicate in /etc/nginx/nginx.conf:164` it is likely due to `zone_sync` being enabled in both `stream-snippets` and the ConfigMap. Once upgraded, remove the [old headless service](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml) deployed for OIDC. +{{< /caution >}} + + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*zone-sync* | Enables zone synchronization between NGINX Ingress Controller Pods. This autogenerates a [zone_sync_server](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_server) and a headless service using the `ReplicaSet` or `DaemonSet` name. Please note that this headless service will be automatically cleaned up when uninstalling via Helm or by removing the value from the ConfigMap. The headless service will need to be manually removed if the `controller.customConfigMap` value is set via Helm or the deployment is uninstalled via Manifests. Each Ingress Controller manages its own headless service. NGINX Plus Required. | *False* | | +|*zone-sync-port* | Specifies the optional port on which NGINX Ingress Controller listens for zone sync traffic. NGINX Plus & `zone-sync` Required. | *12345* | | +|*zone-sync-resolver-addresses* | Configures optional addresses used in the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync. This field takes a comma separated list of addresses. NGINX Plus & `zone-sync` Required | `kube-dns.kube-system.svc.cluster.local` | | +|*zone-sync-resolver-ipv6* | Configures whether the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will look up IPv6 addresses. NGINX Plus & `zone-sync` Required | `true` | | +|*zone-sync-resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will override the TTL value of responses from nameservers with. NGINX Plus & `zone-sync` Required | `5s` | | +{{}} + +--- + +### Snippets and custom templates + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*main-snippets* | Sets a custom snippet in main context. | N/A | | +|*http-snippets* | Sets a custom snippet in http context. | N/A | | +|*location-snippets* | Sets a custom snippet in location context. | N/A | | +|*server-snippets* | Sets a custom snippet in server context. | N/A | | +|*stream-snippets* | Sets a custom snippet in stream context. | N/A | [Support for TCP/UDP Load Balancing](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/tcp-udp). | +|*main-template* | Sets the main NGINX configuration template. | By default the template is read from the file in the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +|*ingress-template* | Sets the NGINX configuration template for an Ingress resource. | By default the template is read from the file on the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +|*virtualserver-template* | Sets the NGINX configuration template for an VirtualServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +|*transportserver-template* | Sets the NGINX configuration template for a TransportServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). | +{{}} + +--- + +### Modules + +{{}} +|ConfigMap Key | Description | Default | Example | +| ---| ---| ---| --- | +|*otel-exporter-endpoint* | OTLP/gRPC endpoint that will accept [OpenTelemetry](https://opentelemetry.io) data. Set `otel-trace-in-http` to *"true"* to enable OpenTelemetry at the global level. | N/A | *"https://otel-collector:4317"* | +|*otel-exporter-header-name* | The name of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-value` required. | N/A | *"X-custom-header"* | +|*otel-exporter-header-value* | The value of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-name` required. | N/A | *"custom-value"* | +|*otel-service-name* | Sets the `service.name` attribute of the OTel resource. `otel-exporter-endpoint` required. | N/A | *"nginx-ingress-controller:nginx"* | +| *otel-trace-in-http* | Enables [OpenTelemetry](https://opentelemetry.io) globally (for all Ingress, VirtualServer and VirtualServerRoute resources). Set this to *"false"* to enable OpenTelemetry for individual routes with snippets. `otel-exporter-endpoint` required. | *"false"* | *"true"* | +|*opentracing* | Removed in v5.0.0. Enables [OpenTracing](https://opentracing.io) globally (for all Ingress, VirtualServer and VirtualServerRoute resources). Note: requires the Ingress Controller image with OpenTracing module and a tracer. See the [docs]({{< ref "/nic/installation/integrations/opentracing.md" >}}) for more information. | *False* | | +|*opentracing-tracer* | Removed in v5.0.0. Sets the path to the vendor tracer binary plugin. | N/A | | +|*opentracing-tracer-config* | Removed in v5.0.0. Sets the tracer configuration in JSON format. | N/A | | +|*app-protect-compressed-requests-action* | Sets the *app_protect_compressed_requests_action* [global directive](/nginx-app-protect/configuration/#global-directives). | *drop* | | +|*app-protect-cookie-seed* | Sets the *app_protect_cookie_seed* [global directive](/nginx-app-protect/configuration/#global-directives). | Random automatically generated string | | +|*app-protect-failure-mode-action* | Sets the *app_protect_failure_mode_action* [global directive](/nginx-app-protect/configuration/#global-directives). | *pass* | | +|*app-protect-cpu-thresholds* | Sets the *app_protect_cpu_thresholds* [global directive](/nginx-app-protect/configuration/#global-directives). | *high=100 low=100* | | +|*app-protect-physical-memory-util-thresholds* | Sets the *app_protect_physical_memory_util_thresholds* [global directive](/nginx-app-protect/configuration/#global-directives). | *high=100 low=100* | | +|`app-protect-reconnect-period-seconds` | Sets the `app_protect_reconnect_period_seconds` [global directive](/nginx-app-protect/configuration/#global-directives). | `5` | | +|*app-protect-dos-log-format* | Sets the custom [log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for Dos Access log traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | `, vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, policy_name=$app_protect_dos_policy_name, dos_version=$app_protect_dos_version, ip_tls=$remote_addr:$app_protect_dos_tls_fp,` | | +|*app-protect-dos-log-format-escaping* | Sets the characters escaping for the variables of the stream log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | +|*app-protect-dos-arb-fqdn* | Sets the *app-protect-dos-arb-fqdn* [directive](/nginx-app-protect-dos/directives-and-policy/learn-about-directives-and-policy/#arbitrator-fqdn-directive-app_protect_dos_arb_fqdn). | *svc-appprotect-dos-arb* | | +{{}} diff --git a/content/nic/configuration/global-configuration/custom-templates.md b/content/nic/configuration/global-configuration/custom-templates.md new file mode 100644 index 000000000..589ce5951 --- /dev/null +++ b/content/nic/configuration/global-configuration/custom-templates.md @@ -0,0 +1,11 @@ +--- +docs: DOCS-587 +doctypes: +- '' +title: Custom templates +toc: true +weight: 500 +--- + + +F5 NGINX Ingress Controller uses templates to generate NGINX configuration for Ingress resources, VirtualServer resources and the main NGINX configuration file. You can customize the templates and apply them via the ConfigMap. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/shared-examples/custom-templates). diff --git a/content/nic/configuration/global-configuration/globalconfiguration-resource.md b/content/nic/configuration/global-configuration/globalconfiguration-resource.md new file mode 100644 index 000000000..84b4e4002 --- /dev/null +++ b/content/nic/configuration/global-configuration/globalconfiguration-resource.md @@ -0,0 +1,183 @@ +--- +docs: DOCS-588 +doctypes: +- '' +title: GlobalConfiguration resource +toc: true +weight: 200 +--- + +This page explains how to use the GlobalConfiguration resource to define the global configuration parameters of F5 NGINX Ingress Controller. + +The resource supports configuring listeners for TCP and UDP load balancing, and is implemented as a [Custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +Listeners are required by [TransportServer resources]({{< ref "/nic/configuration/transportserver-resource.md" >}}) and can be used to [configure custom listeners for VirtualServers]({{< ref "/nic/tutorials/virtual-server-with-custom-listener-ports.md" >}}). + +--- + +## Prerequisites + +When [installing NGINX Ingress Controller using Manifests]({{< ref "/nic/installation/installing-nic/installation-with-manifests.md" >}}), you need to reference a GlobalConfiguration resource in the [`-global-configuration`](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments#cmdoption-global-configuration) command-line argument. NGINX Ingress Controller only needs one GlobalConfiguration resource. + +--- + +## GlobalConfiguration specification + +The GlobalConfiguration resource defines the global configuration parameters of the Ingress Controller. Below is an example: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: GlobalConfiguration +metadata: + name: nginx-configuration + namespace: nginx-ingress +spec: + listeners: + - name: dns-udp + port: 5353 + protocol: UDP + - name: dns-tcp + port: 5353 + protocol: TCP + - name: http-8083 + port: 8083 + protocol: HTTP + - name: https-8443 + port: 8443 + protocol: HTTP + ssl: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +| *listeners* | A list of listeners. | [listener](#listener) | No | +{{}} + +### Listener + +The `listeners:` key defines a listener (a combination of a protocol and a port) that NGINX will use to accept traffic for a [TransportServer]({{< ref "/nic/configuration/transportserver-resource.md" >}}) and a [VirtualServer]({{< ref "/nic/configuration/virtualserver-and-virtualserverroute-resources.md" >}}): + +```yaml +- name: dns-tcp + port: 5353 + protocol: TCP +- name: http-8083 + port: 8083 + protocol: HTTP +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +| *name* | The name of the listener. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``listener-123`` are valid. The name must be unique among all listeners. The name ``tls-passthrough`` is reserved for the built-in TLS Passthrough listener and cannot be used. | *string* | Yes | +| *port* | The port of the listener. The port must fall into the range ``1..65535`` with the following exceptions: ``80``, ``443``, the [status port](/nginx-ingress-controller/logging-and-monitoring/status-page), the [Prometheus metrics port](/nginx-ingress-controller/logging-and-monitoring/prometheus). Among all listeners, only a single combination of a port-protocol is allowed. | *int* | Yes | +| *protocol* | The protocol of the listener. Supported values: ``TCP``, ``UDP`` and ``HTTP``. | *string* | Yes | +| *ssl* | Configures the listener with SSL. This is currently only supported for ``HTTP`` listeners. Default value is ``false`` | *bool* | No | +| *ipv4* | Specifies the IPv4 address to listen on. | *string* | No | +| *ipv6* | Specifies the IPv6 address to listen on. | *string* | No | + +{{}} + +--- + +## Using GlobalConfiguration + +You can use the usual `kubectl` commands to work with a GlobalConfiguration resource. + +For example, the following command creates a GlobalConfiguration resource defined in `global-configuration.yaml` with the name `nginx-configuration`: + +```shell +kubectl apply -f global-configuration.yaml +``` +```shell +globalconfiguration.k8s.nginx.org/nginx-configuration created +``` + +Assuming the namespace of the resource is `nginx-ingress`, you can get the resource by running: + +```shell +kubectl get globalconfiguration nginx-configuration -n nginx-ingress +``` +```shell +NAME AGE +nginx-configuration 13s +``` + +With `kubectl get` and similar commands, you can use the short name `gc` instead of `globalconfiguration`. + +--- + +### Validation + +Two types of validation are available for the GlobalConfiguration resource: + +- *Structural validation* by `kubectl` and Kubernetes API server. +- *Comprehensive validation* by NGINX Ingress Controller. + + +#### Structural validation + +The custom resource definition for the GlobalConfiguration includes structural OpenAPI schema which describes the type of every field of the resource. + +If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of a listener), `kubectl` and Kubernetes API server will reject such a resource: + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f global-configuration.yaml + ``` + ```text + error: error validating "global-configuration.yaml": error validating data: ValidationError(GlobalConfiguration.spec.listeners[0].port): invalid type for org.nginx.k8s.v1.GlobalConfiguration.spec.listeners.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f global-configuration.yaml --validate=false + ``` + ```text + The GlobalConfiguration "nginx-configuration" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: + spec.listeners.port in body must be of type integer: "string" + ``` + +If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. + +#### Comprehensive validation + +NGINX Ingress Controller validates the fields of a GlobalConfiguration resource. If a GlobalConfiguration resource is partially invalid, NGINX Ingress Controller use the valid listeners and emit events about invalid listeners. + +You can check if the Ingress Controller successfully applied the configuration for a GlobalConfiguration. For our `nginx-configuration` GlobalConfiguration, we can run: + +```shell +kubectl describe gc nginx-configuration -n nginx-ingress +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Updated 11s nginx-ingress-controller GlobalConfiguration nginx-ingress/nginx-configuration was updated +``` + +The events section includes a Normal event with the Updated reason that informs us that the configuration was successfully applied. + +If you create a GlobalConfiguration `nginx-configuration` with two or more listeners that have the same protocol UDP and port 53, you will get: + +```shell +kubectl describe gc nginx-configuration -n nginx-ingress +``` +```text +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Updated 55s nginx-ingress-controller GlobalConfiguration nginx-ingress/nginx-configuration was updated + Warning AddedOrUpdatedWithError 6s nginx-ingress-controller GlobalConfiguration nginx-ingress/nginx-configuration is invalid and was rejected: spec.listeners: Duplicate value: "Duplicated port/protocol combination 53/UDP" +``` + +The events section includes a Warning event with the AddedOrUpdatedWithError reason. + + +## Using IPV4 and IPV6 Addresses with GlobalConfiguration + +You can customize the IPv4 and IPv6 Address listeners in the global configuration and apply them to your VirtualServer resources. See the corresponding example [here](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/custom-ip-listeners/virtualserver/) diff --git a/content/nic/configuration/global-configuration/mgmt-configmap-resource.md b/content/nic/configuration/global-configuration/mgmt-configmap-resource.md new file mode 100644 index 000000000..e1d9a120c --- /dev/null +++ b/content/nic/configuration/global-configuration/mgmt-configmap-resource.md @@ -0,0 +1,50 @@ +--- +docs: DOCS-586 +doctypes: +- '' +title: Management ConfigMap resource +toc: true +weight: 300 +--- + +When using F5 NGINX Ingress Controller with NGINX Plus, it is required to pass a [command line argument]({{< ref "/nic/configuration/global-configuration/command-line-arguments" >}}) to NGINX Ingress Controller, `--mgmt-configmap=` which specifies the ConfigMap to use. The minimal required ConfigMap must have a `license-token-secret-name` key. Helm users will not need to create this map or pass the argument, it will be created with a Helm install. + +--- + +1. Create a ConfigMap file with the name *nginx-config-mgmt.yaml* and set the values +that make sense for your setup: + + ```yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: nginx-config-mgmt + namespace: nginx-ingress + data: + license-token-secret-name: "license-token" + ``` +1. Create a new (or update the existing) ConfigMap resource: + + ```shell + kubectl apply -f nginx-config-mgmt.yaml + ``` + + The [NGINX Management](https://nginx.org/en/docs/ngx_mgmt_module.html) block configuration will be updated. +--- +## Management ConfigMap keys + +{{}} +|ConfigMap Key | Description | Default | +| ---| ---| ---| +|*license-token-secret-name* | Configures the secret used in the [license_token](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.com/license` with the base64 encoded JWT in the `license.jwt` key. | N/A | +|*ssl-verify* | Configures the [ssl_verify](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_verify) directive, which enables or disables verification of the usage reporting endpoint certificate. | `true` | +|*enforce-initial-report* | Configures the [enforce_initial_report](https://nginx.org/en/docs/ngx_mgmt_module.html#enforce_initial_report) directive, which enables or disables the 180-day grace period for sending the initial usage report. | `false` | +|*usage-report-endpoint* | Configures the endpoint of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This is used to configure the endpoint NGINX uses to send usage reports to NIM. | `product.connect.nginx.com` | +|*usage-report-interval* | Configures the interval of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This specifies the frequency that usage reports are sent. This field takes an [NGINX time](https://nginx.org/en/docs/syntax.html). | `1h` | +|*usage-report-proxy-host* | Configures the host name of the [proxy](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy) directive with optional port. | N/A | +|*ssl-trusted-certificate-secret-name* | Configures the secret used to create the file(s) referenced the in [ssl_trusted_certifcate](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_trusted_certificate), and [ssl_crl](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.org/ca`, where the `ca.crt` key contains a base64 encoded trusted cert, and the optional `ca.crl` key can contain a base64 encoded CRL. If the optional `ca.crl` key is supplied, it will configure the NGINX `ssl_crl` directive. | N/A | +|*ssl-certificate-secret-name* | Configures the secret used to create the `ssl_certificate` and `ssl_certificate_key` directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `kubernetes.io/tls`| N/A | +|*resolver-addresses* | Configures addresses used in the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive. This field takes a comma separated list of addresses. | N/A | +|*resolver-ipv6* | Configures whether the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will look up IPv6 addresses. | `true` | +|*resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will override the TTL value of responses from nameservers with. | N/A | +{{}} diff --git a/content/nic/configuration/global-configuration/reporting-resources-status.md b/content/nic/configuration/global-configuration/reporting-resources-status.md new file mode 100644 index 000000000..8e36733eb --- /dev/null +++ b/content/nic/configuration/global-configuration/reporting-resources-status.md @@ -0,0 +1,195 @@ +--- +docs: DOCS-589 +doctypes: +- '' +title: Reporting resource status +toc: true +weight: 600 +--- + +This page describes how to view the status of resources managed by F5 NGINX Ingress Controller. + +## Ingress resources + +An Ingress resource status includes the address (an IP address or a DNS name), through which the hosts of that Ingress resource are publicly accessible. + +You can see the address in the output of the `kubectl get ingress` command, in the ADDRESS column, as shown below: + +```shell +kubectl get ingresses +``` +```text +NAME HOSTS ADDRESS PORTS AGE +cafe-ingress cafe.example.com 12.13.23.123 80, 443 2m +``` + +NGINX Ingress Controller must be configured to report an Ingress status: + +1. Use the command-line flag `-report-ingress-status`. +1. Define a source for an external address. This can be either of: + 1. A user defined address, specified in the `external-status-address` ConfigMap key. + 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. + +View the [ConfigMap keys](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) and [Command-line arguments](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments) topics for more information. + +{{< note >}} NGINX Ingress Controller does not clear the status of Ingress resources when it is being shut down. {{< /note >}} + +## VirtualServer and VirtualServerRoute resources + +A VirtualServer or VirtualServerRoute resource includes the status field with information about the state of the resource and the IP address, through which the hosts of that resource are publicly accessible. + +You can see the status in the output of the `kubectl get virtualservers` or `kubectl get virtualserverroutes` commands as shown below: + +```shell +kubectl get virtualservers +``` +```text + NAME STATE HOST IP PORTS AGE + cafe Valid cafe.example.com 12.13.23.123 [80,443] 34s +``` + +To see an external hostname address associated with a VirtualServer resource, use the `-o wide` option: + +```shell +kubectl get virtualservers -o wide +``` +```text + NAME STATE HOST IP EXTERNALHOSTNAME PORTS AGE + cafe Valid cafe.example.com ae430f41a1a0042908655abcdefghijkl-12345678.eu-west-2.elb.amazonaws.com [80,443] 106s +``` + +{{< note >}} If there are multiple addresses, only the first one is shown. {{< /note >}} + +In order to see additional addresses or extra information about the `Status` of the resource, use the following command: + +```shell +kubectl describe virtualserver +``` +```text +... +Status: + External Endpoints: + Ip: 12.13.23.123 + Ports: [80,443] + Message: Configuration for cafe/cafe was added or updated + Reason: AddedOrUpdated + State: Valid +``` + +### Status specification + +The following fields are reported in both VirtualServer and VirtualServerRoute status: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +|*State* | Current state of the resource. Can be ``Valid``, ``Warning`` an ``Invalid``. For more information, refer to the ``message`` field. | *string* | +|*Reason* | The reason of the last update. | *string* | +|*Message* | Additional information about the state. | *string* | +|*ExternalEndpoints* | A list of external endpoints for which the hosts of the resource are publicly accessible. | *[externalEndpoint](#externalendpoint)* | +{{}} + +The *ReferencedBy* field is reported for the VirtualServerRoute status only: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +| *ReferencedBy* | The VirtualServer that references this VirtualServerRoute. Format as ``namespace/name`` | *string* | +{{}} + +### externalEndpoint + +{{}} +|Field | Description | Type | +| ---| ---| --- | +|``IP`` | The external IP address. | ``string`` | +|``Hostname`` | The external LoadBalancer Hostname address. | ``string`` | +|``Ports`` | A list of external ports. | ``string`` | +{{}} + +NGINX Ingress Controller must be configured to report a VirtualServer or VirtualServerRoute status: + +1. If you want NGINX Ingress Controller to report the `externalEndpoints`, define a source for an external address (The rest of the fields will be reported without the external address configured). This can be: + 1. A user defined address, specified in the `external-status-address` ConfigMap key. + 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. + +View the [ConfigMap keys](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) and [Command-line arguments](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments) topics for more information. + +{{< note >}} NGINX Ingress Controller does not clear the status of VirtualServer and VirtualServerRoute resources when it is being shut down. {{< /note >}} + +## Policy resources + +A Policy resource includes the status field with information about the state of the resource. + +You can see the status in the output of the `kubectl get policy` command as shown below: + +```shell +kubectl get policy +``` +```text + NAME STATE AGE + webapp-policy Valid 30s +``` + +In order to see additional addresses or extra information about the `Status` of the resource, use the following command: + +```shell +kubectl describe policy +``` +```text +... +Status: + Message: Configuration for default/webapp-policy was added or updated + Reason: AddedOrUpdated + State: Valid +``` + +### Status specification + +The following fields are reported in Policy status: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +|``State`` | Current state of the resource. Can be ``Valid`` or ``Invalid``. For more information, refer to the ``message`` field. | ``string`` | +|``Reason`` | The reason of the last update. | ``string`` | +|``Message`` | Additional information about the state. | ``string`` | +{{}} + +## TransportServer resources + +A TransportServer resource includes the status field with information about the state of the resource. + +You can see the status in the output of the `kubectl get transportserver` command as shown below: + +```shell +kubectl get transportserver +``` +```text + NAME STATE REASON AGE + dns-tcp Valid AddedOrUpdated 47m +``` + +To see additional addresses or extra information about the `Status` of the resource, use the following command: + +```shell +kubectl describe transportserver +``` +```text +Status: + Message: Configuration for default/dns-tcp was added or updated + Reason: AddedOrUpdated + State: Valid +``` + +### Status specification + +The following fields are reported in TransportServer status: + +{{}} +|Field | Description | Type | +| ---| ---| --- | +| *State* | Current state of the resource. Can be ``Valid``, ``Warning`` or ``Invalid``. For more information, refer to the ``message`` field. | *string* | +| *Reason* | The reason of the last update. | *string* | +| *Message* | Additional information about the state. | *string* | +{{}} diff --git a/content/nic/configuration/host-and-listener-collisions.md b/content/nic/configuration/host-and-listener-collisions.md new file mode 100644 index 000000000..502ef830c --- /dev/null +++ b/content/nic/configuration/host-and-listener-collisions.md @@ -0,0 +1,168 @@ +--- +title: Host and Listener collisions +toc: true +weight: 800 +docs: DOCS-590 +--- + +This document explains how F5 NGINX Ingress Controller handles host and listener collisions between resources. + +--- + +## Winner Selection Algorithm + +If multiple resources contend for the same host or listener, NGINX Ingress Controller will pick the winner based on the `creationTimestamp` of the resources: the oldest resource will win. In case there are more than one oldest resource (their `creationTimestamp` is the same), NGINX Ingress Controller will choose the resource with the lexicographically smallest `uid`. + +{{< note >}} The `creationTimestamp` and `uid` fields are part of the [ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/) resource. {{< /note >}} + +--- + +## Host collisions + +A host collision occurs when multiple Ingress, VirtualServer, and TransportServer (configured for TLS Passthrough) resources configure the same `host`. NGINX Ingress Controller has two strategies for handling host collisions: + +- Choosing a single "winner" resource to handle the host. +- Merging the configuration of the conflicting resources. + +--- + +### Choosing the winner + +Consider the following two resources: + +- `cafe-ingress` Ingress: + + ```yaml + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: cafe-ingress + spec: + ingressClassName: nginx + rules: + - host: cafe.example.com + . . . + ``` + +- `cafe-virtual-server` VirtualServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: VirtualServer + metadata: + name: cafe-virtual-server + spec: + host: cafe.example.com + . . . + ``` + +If a user creates both resources in the cluster, a host collision will occur. NGINX Ingress Controller will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). + +If `cafe-virtual-server` was created first, it will win the host `cafe.example.com` and NGINX Ingress Controller will reject `cafe-ingress`. This will be reflected in the events and in the resource's status field: + +```shell +kubectl describe vs cafe-virtual-server +``` +```text +... +Status: + ... + Message: Configuration for default/cafe-virtual-server was added or updated + Reason: AddedOrUpdated + State: Valid +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 9s nginx-ingress-controller Configuration for default/cafe-virtual-server was added or updated +``` + +```shell +kubectl describe ingress cafe-ingress +``` +```text +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 66s nginx-ingress-controller All hosts are taken by other resources +``` + +Similarly, if `cafe-ingress` was created first, it will win `cafe.example.com` and NGINX Ingress Controller will reject `cafe-virtual-server`. + +{{< note >}} You can configure multiple hosts for Ingress resources, and its possible that an Ingress resource can be the winner for some of its hosts and a loser for the others. + +For example, if `cafe-ingress` had an additional rule host rule for `pub.example.com`, NGINX Ingress Controller would not reject the Ingress. Instead, it would allow `cafe-ingress` to handle `pub.example.com`. {{< /note >}} + +--- + +### Merging configuration for the same host + +It is possible to merge configuration for multiple Ingress resources for the same host. One common use case for this approach is distributing resources across multiple namespaces. + +The [Cross-namespace configuration]({{< ref "/nic/configuration/ingress-resources/cross-namespace-configuration.md">}}) topic has more information. + +It is *not* possible to merge the configurations for multiple VirtualServer resources for the same host. However, you can split the VirtualServers into multiple VirtualServerRoute resources, which a single VirtualServer can then reference. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/cross-namespace-configuration) on GitHub. + +It is *not* possible to merge configuration for multiple TransportServer resources. + +--- + +## Listener/Host collisions + +Listener/Host collisions occur when multiple TransportServer resources (configured for TCP/UDP load balancing) specify the same combination of `spec.listener.name` and `spec.host`. + +The combination of `spec.listener.name` and `spec.host` must be unique among all TransportServer resources. If two TransportServer resources specify the same spec.listener.name and spec.host, one of them will be rejected to prevent conflicts. In the case where spec.host is not specified, it is considered an empty string. + +NGINX Ingress Controller will choose the winner, which will own that listener and host combination. + +--- + +### Choosing the winner + +Consider the following two resources: + +- `tcp-1` TransportServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: tcp-1 + spec: + host: dns.example.com + listener: + name: dns-tcp + protocol: TCP + . . . + ``` + +- `tcp-2` TransportServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: tcp-2 + spec: + host: dns.example.com + listener: + name: dns-tcp + protocol: TCP + . . . + ``` + +If a user creates both resources in the cluster, a listener collision will occur. As a result, NGINX Ingress Controller will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). + +In our example, if `tcp-1` was created first, it will win the listener `dns-tcp` and NGINX Ingress Controller will reject `tcp-2`. This will be reflected in the events and in the resource's status field: + +```shell +kubectl describe ts tcp-2 +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 10s nginx-ingress-controller Listener dns-tcp is taken by another resource +``` + +Similarly, if `tcp-2` was created first, it will win `dns-tcp` and NGINX Ingress Controller will reject `tcp-1`. diff --git a/content/nic/configuration/ingress-resources/_index.md b/content/nic/configuration/ingress-resources/_index.md new file mode 100644 index 000000000..8a2b5b282 --- /dev/null +++ b/content/nic/configuration/ingress-resources/_index.md @@ -0,0 +1,8 @@ +--- +title: Ingress resources +description: +weight: 200 +menu: + docs: + parent: NGINX Ingress Controller +--- diff --git a/content/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md b/content/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md new file mode 100644 index 000000000..ee3dbc860 --- /dev/null +++ b/content/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md @@ -0,0 +1,225 @@ +--- +docs: DOCS-591 +doctypes: +- '' +title: Advanced configuration with Annotations +toc: true +weight: 200 +--- + +This topic explains how to enable advanced features in F5 NGINX Ingress Controller with Annotations. + +The Ingress resource can use basic NGINX features such as host or path-based routing and TLS termination. Advanced features like rewriting the request URI or inserting additional response headers can be enabled with Annotations. + +Outside of advanced features, Annotations are necessary for customizing NGINX behavior such as setting the value of connection timeouts. + +Customization is also available through the [ConfigMap]({{< ref "/nic/configuration/global-configuration/configmap-resource.md" >}}) resources: Annotations take priority. + +## Using Annotations + +This example uses Annotations to customize the configuration for an Ingress resource: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: cafe-ingress-with-annotations + annotations: + nginx.org/proxy-connect-timeout: "30s" + nginx.org/proxy-read-timeout: "20s" + nginx.org/client-max-body-size: "4m" + nginx.org/server-snippets: | + location / { + return 302 /coffee; + } +spec: + rules: + - host: cafe.example.com + http: + paths: + - path: /tea + pathType: Prefix + backend: + service: + name: tea-svc + port: + number: 80 + - path: /coffee + pathType: Prefix + backend: + service: + name: coffee-svc + port: + number: 80 +``` + +## Validation + +NGINX Ingress Controller validates the annotations of Ingress resources. If an Ingress is invalid, NGINX Ingress Controller will reject it: the Ingress will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can check if NGINX Ingress Controller successfully applied the configuration for an Ingress resource. For the example `cafe-ingress-with-annotations` Ingress, you can run: + +```shell +kubectl describe ing cafe-ingress-with-annotations +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/cafe-ingress-with-annotations was added or updated +``` + +The events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid Ingress, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create an Ingress `cafe-ingress-with-annotations`, with an annotation `nginx.org/redirect-to-https` set to `yes please` instead of `true`, you will get: + +```shell +kubectl describe ing cafe-ingress-with-annotations +``` +```text +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 13s nginx-ingress-controller annotations.nginx.org/redirect-to-https: Invalid value: "yes please": must be a boolean +``` + +Note how the events section includes a Warning event with the Rejected reason. + +{{< note >}} If you make an existing Ingress invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. {{< /note >}} + +The `nginx.com/jwt-token` Ingress annotation has limited validation. + +## Summary of Annotations + +The table below summarizes the available annotations. + +{{< note >}} Annotations that start with `nginx.com` are only supported with NGINX Plus. {{< /note >}} + +### General customization + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/proxy-connect-timeout* | *proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | +| *nginx.org/proxy-read-timeout* | *proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | +| *nginx.org/proxy-send-timeout* | *proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | +| *nginx.org/client-max-body-size* | *client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | +| *nginx.org/proxy-buffering* | *proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | +| *nginx.org/proxy-buffers* | *proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | +| *nginx.org/proxy-buffer-size* | *proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | +| *nginx.org/proxy-max-temp-file-size* | *proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | +| *nginx.org/server-tokens* | *server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. Additionally, with the NGINX Plus, you can specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | +| *nginx.org/path-regex* | N/A | Enables regular expression modifiers for Ingress path parameter. This translates to the NGINX [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive. You can specify one of these values: "case_sensitive", "case_insensitive", or "exact". The annotation is applied to the entire Ingress resource and its paths. While using Master and Minion Ingresses i.e. Mergeable Ingresses, this annotation can be specified on Minion types. The `path-regex` annotation specified on Master is ignored, and has no effect on paths defined on Minions. | N/A | [path-regex](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/path-regex) | +{{}} + +### Request URI/Header Manipulation + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/proxy-hide-headers* | *proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: ``"nginx.org/proxy-hide-headers": "header-a,header-b"* | N/A | | +| *nginx.org/proxy-pass-headers* | *proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: ``"nginx.org/proxy-pass-headers": "header-a,header-b"* | N/A | | +| *nginx.org/rewrites* | N/A | Configures URI rewriting using [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive. | N/A | [rewrites](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/rewrites) | +|*nginx.org/proxy-set-headers* | N/A | Enables customization of proxy headers and values using the [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. Example: ``"nginx.org/proxy-set-headers": "header-a: valueA,header-b: valueB,header-c: valueC"`` | N/A | [Proxy Set Headers](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/proxy-set-headers). | +{{}} + +### Auth and SSL/TLS + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/redirect-to-https* | *redirect-to-https* | Sets the 301 redirect rule based on the value of the ``http_x_forwarded_proto* header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of NGINX Ingress Controller — see [115](https://github.com/nginx/kubernetes-ingress/issues/115) | *False* | | +| *ingress.kubernetes.io/ssl-redirect* | *ssl-redirect* | Sets an unconditional 301 redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS. | *True* | | +| *nginx.org/hsts* | *hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/)\ : the HSTS header is added to the responses from backends. The ``preload* directive is included in the header. | *False* | | +| *nginx.org/hsts-max-age* | *hsts-max-age* | Sets the value of the ``max-age* directive of the HSTS header. | *2592000* (1 month) | | +| *nginx.org/hsts-include-subdomains* | *hsts-include-subdomains* | Adds the ``includeSubDomains* directive to the HSTS header. | *False* | | +| *nginx.org/hsts-behind-proxy* | *hsts-behind-proxy* | Enables HSTS based on the value of the ``http_x_forwarded_proto* request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of NGINX Ingress Controller. Note: to control redirection from HTTP to HTTPS configure the ``nginx.org/redirect-to-https* annotation. | *False* | | +| *nginx.org/basic-auth-secret* | N/A | Specifies a Secret resource with a user list for HTTP Basic authentication. | N/A | | +| *nginx.org/basic-auth-realm* | N/A | Specifies a realm. | N/A | | +| *nginx.com/jwt-key* | N/A | Specifies a Secret resource with keys for validating JSON Web Tokens (JWTs). | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +| *nginx.com/jwt-realm* | N/A | Specifies a realm. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +| *nginx.com/jwt-token* | N/A | Specifies a variable that contains a JSON Web Token. | By default, a JWT is expected in the ``Authorization* header as a Bearer Token. | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +| *nginx.com/jwt-login-url* | N/A | Specifies a URL to which a client is redirected in case of an invalid or missing JWT. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/jwt). | +{{}} + +### Listeners + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/listen-ports* | N/A | Configures HTTP ports that NGINX will listen on. | *[80]* | | +| *nginx.org/listen-ports-ssl* | N/A | Configures HTTPS ports that NGINX will listen on. | *[443]* | | +{{}} + +### Backend services (Upstreams) + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/lb-method* | *lb-method* | Sets the [load balancing method](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method). To use the round-robin method, specify ``"round_robin"``. | *"random two least_conn"* | | +| *nginx.org/ssl-services* | N/A | Enables HTTPS or gRPC over SSL when connecting to the endpoints of services. | N/A | [ssl-services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/ssl-services) | +| *nginx.org/grpc-services* | N/A | Enables gRPC for services. Note: requires HTTP/2 (see ``http2* ConfigMap key); only works for Ingresses with TLS termination enabled. | N/A | [grpc-services](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/grpc-services) | +| *nginx.org/websocket-services* | N/A | Enables WebSocket for services. | N/A | [websocket](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/websocket) | +| *nginx.org/max-fails* | *max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the ``server* directive. | *1* | | +| *nginx.org/max-conns* | N\A | Sets the value of the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the ``server* directive. | *0* | | +| *nginx.org/upstream-zone-size* | *upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. For NGINX, the special value 0 disables the shared memory zones. For NGINX Plus, shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *256K* | | +| *nginx.org/fail-timeout* | *fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the ``server* directive. | *10s* | | +| *nginx.com/sticky-cookie-services* | N/A | Configures session persistence. | N/A | [session-persistence](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/session-persistence) | +| *nginx.org/keepalive* | *keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that ``proxy_set_header Connection "";* is added to the generated configuration when the value > 0. | *0* | | +| *nginx.com/health-checks* | N/A | Enables active health checks. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks) | +| *nginx.com/health-checks-mandatory* | N/A | Configures active health checks as mandatory. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks) | +| *nginx.com/health-checks-mandatory-queue* | N/A | When active health checks are mandatory, creates a queue where incoming requests are temporarily stored while NGINX Plus is checking the health of the endpoints after a configuration reload. | *0* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks) | +| *nginx.com/slow-start* | N/A | Sets the upstream server [slow-start period](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#server-slow-start). By default, slow-start is activated after a server becomes [available](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#passive-health-checks) or [healthy](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#active-health-checks). To enable slow-start for newly-added servers, configure [mandatory active health checks](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/health-checks). | *"0s"* | | +| *nginx.org/use-cluster-ip* | N/A | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method* and ``next-upstream``) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | *False* | | +{{}} + +### Rate limiting + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/limit-req-rate* | N/A | Enables request-rate-limiting for this ingress by creating a [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) and matching [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) for each location. All servers/locations of one ingress share the same zone. Must have unit r/s or r/m. | N/A | 200r/s | +| *nginx.org/limit-req-key* | N/A | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ${}. | ${binary_remote_addr} | ${binary_remote_addr} | +| *nginx.org/limit-req-zone-size* | N/A | Configures the size of the created [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone). | 10m | 20m | +| *nginx.org/limit-req-delay* | N/A | Configures the delay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | 0 | 100 | +| *nginx.org/limit-req-no-delay* | N/A | Configures the nodelay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | false | true | +| *nginx.org/limit-req-burst* | N/A | Configures the burst-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | N/A | 100 | +| *nginx.org/limit-req-dry-run* | N/A | Enables the dry run mode. In this mode, the rate limit is not actually applied, but the number of excessive requests is accounted as usual in the shared memory zone. | false | true | +| *nginx.org/limit-req-log-level* | N/A | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are info, notice, warn or error. | error | info | +| *nginx.org/limit-req-reject-code* | N/A | Sets the status code to return in response to rejected requests. Must fall into the range 400..599. | 429 | 503 | +| *nginx.org/limit-req-scale* | N/A | Enables a constant rate-limit by dividing the configured rate by the number of nginx-ingress pods currently serving traffic. This adjustment ensures that the rate-limit remains consistent, even as the number of nginx-pods fluctuates due to autoscaling. Note: This will not work properly if requests from a client are not evenly distributed accross all ingress pods (sticky sessions, long lived TCP-Connections with many requests etc.). In such cases using [zone-sync]({{< ref "/nic/configuration/global-configuration/configmap-resource.md#zone-sync" >}}) instead would give better results. Enabling `zone-sync` will suppress this setting. | false | true | +{{}} + +### Snippets and custom templates + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *nginx.org/location-snippets* | *location-snippets* | Sets a custom snippet in location context. | N/A | | +| *nginx.org/server-snippets* | *server-snippets* | Sets a custom snippet in server context. | N/A | | +{{}} + +### App Protect WAF {#app-protect} + +{{< note >}} The App Protect annotations only work if the App Protect WAF module is [installed]({{< ref "/nic/installation/integrations/app-protect-waf/installation.md" >}}). {{< /note >}} + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *appprotect.f5.com/app-protect-policy* | N/A | The name of the App Protect Policy for the Ingress Resource. Format is ``namespace/name``. If no namespace is specified, the same namespace of the Ingress Resource is used. If not specified but ``appprotect.f5.com/app-protect-enable* is true, a default policy id applied. If the referenced policy resource does not exist, or policy is invalid, this annotation will be ignored, and the default policy will be applied. | N/A | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-enable* | N/A | Enable App Protect for the Ingress Resource. | *False* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-security-log-enable* | N/A | Enable the [security log](/nginx-app-protect/troubleshooting/#app-protect-logging-overview) for App Protect. | *False* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-security-log* | N/A | The App Protect log configuration for the Ingress Resource. Format is ``namespace/name``. If no namespace is specified, the same namespace as the Ingress Resource is used. If not specified the default is used which is: filter: ``illegal``, format: ``default``. Multiple configurations can be specified in a comma separated list. Both log configurations and destinations list (see below) must be of equal length. Configs and destinations are paired by the list indices. | N/A | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +| *appprotect.f5.com/app-protect-security-log-destination* | N/A | The destination of the security log. For more information check the [DESTINATION argument](/nginx-app-protect/troubleshooting/#app-protect-logging-overview). Multiple destinations can be specified in a comma-separated list. Both log configurations and destinations list (see above) must be of equal length. Configs and destinations are paired by the list indices. | *syslog:server=localhost:514* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-waf) | +{{}} + +### App Protect DoS + +{{< note >}} The App Protect DoS annotations only work if the App Protect DoS module is [installed]({{< ref "/nic/installation/integrations/app-protect-dos/installation.md" >}}). {{< /note >}} + +{{}} +|Annotation | ConfigMap Key | Description | Default | Example | +| ---| ---| ---| ---| --- | +| *appprotectdos.f5.com/app-protect-dos-resource* | N/A | Enable App Protect DoS for the Ingress Resource by specifying a [DosProtectedResource]({{< ref "/nic/installation/integrations/app-protect-dos/dos-protected.md" >}}). | N/A | [app-protect-dos](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/app-protect-dos) | +{{}} diff --git a/content/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md b/content/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md new file mode 100644 index 000000000..0060a7e3b --- /dev/null +++ b/content/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md @@ -0,0 +1,128 @@ +--- +docs: DOCS-592 +doctypes: +- '' +title: Advanced configuration with Snippets +toc: true +weight: 400 +--- + +Snippets allow you to insert raw NGINX config into different contexts of the NGINX configurations that F5 NGINX Ingress Controller generates. + +Snippets are intended for advanced NGINX users who need more control over the generated NGINX configuration, and can be used in cases where Annotations and ConfigMap entries would not apply. + + + +## Disadvantages of snippets + +Snippets are configured [using Annotations]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#snippets-and-custom-templates" >}}), but are disabled by default due to their complexity. They are also available through the [ConfigMap]({{< ref "/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates" >}}) resource. + +To use snippets, set the [`enable-snippets`]({{< ref "/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-snippets" >}}) command-line argument. + +Snippets have the following disadvantages: + +- *Complexity*. Snippets require you to: + - Understand NGINX configuration primitives and implement a correct NGINX configuration. + - Understand how NGINX Ingress Controller generates NGINX configuration so that a snippet doesn't interfere with the other features in the configuration. +- *Decreased robustness*. An incorrect snippet can invalidate NGINX configuration, causing reload failures. Until the snippet is fixed, it will prevent any new configuration updates, including updates for the other Ingress resources. +- *Security implications*. Snippets give access to NGINX configuration primitives, which are not validated by NGINX Ingress Controller. For example, a snippet can configure NGINX to serve the TLS certificates and keys used for TLS termination for Ingress resources. + +{{< note >}} If the NGINX configuration includes an invalid snippet, NGINX will continue to operate with the last valid configuration. {{< /note >}} + +## Using snippets + +The example below shows how to use snippets to customize the NGINX configuration template using annotations. + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: cafe-ingress-with-snippets + annotations: + nginx.org/server-snippets: | + location / { + return 302 /coffee; + } + nginx.org/location-snippets: | + add_header my-test-header test-value; +spec: + rules: + - host: cafe.example.com + http: + paths: + - path: /tea + pathType: Prefix + backend: + service: + name: tea-svc + port: + number: 80 + - path: /coffee + pathType: Prefix + backend: + service: + name: coffee-svc + port: + number: 80 +``` + +These snippets generate the following NGINX configuration: + +{{< note >}} The example is shortened for conciseness. {{< /note >}} + +```nginx +server { + listen 80; + + + location / { + return 302 /coffee; + } + + + location /coffee { + proxy_http_version 1.1; + + + add_header my-test-header test-value; + ... + proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-coffee-svc-80; + } + + location /tea { + proxy_http_version 1.1; + + add_header my-test-header test-value; + ... + proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-tea-svc-80; + } +} +``` + +## Troubleshooting + +If a snippet includes an invalid NGINX configuration, NGINX Ingress Controller will fail to reload NGINX. The error will be reported in NGINX Ingress Controller logs and an event with the error will be associated with the Ingress resource: + +An example of an error from the logs: + +```text +[emerg] 31#31: unknown directive "badd_header" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54 +Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cafe-ingress-with-snippets", UID:"f9656dc9-63a6-41dd-a499-525b0e0309bb", APIVersion:"extensions/v1beta1", ResourceVersion:"2322030", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithError' Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" +stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" +finished with error: exit status 1 +``` + +An example of an event with an error (you can view events associated with the Ingress by running `kubectl describe -n nginx-ingress ingress nginx-ingress`): + +```text +Events: +Type Reason Age From Message +---- ------ ---- ---- ------- +Normal AddedOrUpdated 52m (x3 over 61m) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated +finished with error: exit status 1 +Warning AddedOrUpdatedWithError 54s (x2 over 89s) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" +stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" +finished with error: exit status 1 +``` + +Additionally, to help troubleshoot snippets, a number of Prometheus metrics show the stats about failed reloads – `controller_nginx_last_reload_status` and `controller_nginx_reload_errors_total`. diff --git a/content/nic/configuration/ingress-resources/basic-configuration.md b/content/nic/configuration/ingress-resources/basic-configuration.md new file mode 100644 index 000000000..8d1948693 --- /dev/null +++ b/content/nic/configuration/ingress-resources/basic-configuration.md @@ -0,0 +1,107 @@ +--- +title: Basic configuration +weight: 100 +toc: true +type: reference +product: NIC +docs: DOCS-593 +--- + +This document shows a basic Ingress resource definition for F5 NGINX Ingress Controller. It load balances requests for two services as part of a single application. + +{{< ghcode `https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/ingress-resources/complete-example/cafe-ingress.yaml`>}} + +Here is a breakdown of what this Ingress resource definition means: + +- The `metadata.name` field defines the name of the resource `cafe‑ingress`. +- The `spec.tls` field sets up SSL/TLS termination: + - The `hosts` field applies the certificate and key to the `cafe.example.com` host. + - The `secretName` references a secret resource by its name, `cafe‑secret`. The secret must belong to the same namespace as the Ingress, of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that hold the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls>). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the hosts to which the secret is applied. +- The `spec.rules` field defines a host with the domain name `cafe.example.com`. +- The `paths` field defines two path‑based rules: + - The rule with the path `/tea` instructs NGINX to distribute the requests with the `/tea` URI among the pods of the *tea* service, which is deployed with the name `tea‑svc` in the cluster. + - The rule with the path `/coffee` instructs NGINX to distribute the requests with the `/coffee` URI among the pods of the *coffee* service, which is deployed with the name `coffee‑svc` in the cluster. + - Both rules instruct NGINX to distribute the requests to `port 80` of the corresponding service (the `servicePort` field). + +To learn more about the Ingress resource, view [the official Kubernetes documentation for Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/). + +{{< note >}} For complete instructions on deploying Ingress and Secret resources in the cluster, see the [complete example](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/complete-example) in the GitHub repository. {{< /note >}} + +--- + +## New features available in Kubernetes 1.18 + +Starting from Kubernetes 1.18, you can use the following new features: + +- The host field supports wildcard domain names, such as `*.example.com`. +- The path supports different matching rules with the new field `pathType`, which takes the following values: `Prefix` for prefix-based matching, `Exact` for exact matching and `ImplementationSpecific`, which is the default type and is the same as `Prefix`. For example: + + ```yaml {hl_lines=[2, 7, 14]} + - path: /tea + pathType: Prefix + backend: + serviceName: tea-svc + servicePort: 80 + - path: /tea/green + pathType: Exact + backend: + service: + name: tea-svc + port: + number: 80 + - path: /coffee + pathType: ImplementationSpecific + backend: + service: + name: coffee-svc + port: + number: 80 + ``` + +- The `ingressClassName` field is now supported: + + ```yaml {hl_lines=[6]} + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: cafe-ingress + spec: + ingressClassName: nginx + tls: + - hosts: + - cafe.example.com + secretName: cafe-secret + rules: + - host: cafe.example.com + . . . + ``` + + When using this field you need to create the `IngressClass` resource with the corresponding `name`. View the [Create common resources]({{< ref "/nic/installation/installing-nic/installation-with-manifests.md#create-common-resources" >}}) section of the Installation with Manifests topic for more information. + +--- + +## Restrictions + +NGINX Ingress Controller imposes the following restrictions on Ingress resources: + +- When defining an Ingress resource, the `host` field is required. +- The `host` value needs to be unique among all Ingress and VirtualServer resources unless the Ingress resource is a [mergeable minion]({{< ref "/nic/configuration/ingress-resources/cross-namespace-configuration.md" >}}). View the [Host and Listener collisions]({{< ref "/nic/configuration/host-and-listener-collisions.md" >}}) topic for more information. +- The `path` field in `spec.rules[].http.paths[]` is required for `Exact` and `Prefix` `pathTypes`. +- The ImplementationSpecific `pathType` is treated as equivalent to `Prefix` `pathType`, with the exception that when this `pathType` is configured, the `path` field in `spec.rules[].http.paths[]` is not mandatory. `path` defaults to `/` if not set but the `pathType` is set to ImplementationSpecific. + +--- + +## Advanced configuration + +NGINX Ingress Controller generates NGINX configuration by executing a template file that contains the configuration options. + +These options are set with the Ingress resource and NGINX Ingress Controller's ConfigMap. + +The Ingress resource only allows you to use basic NGINX features: host and path-based routing and TLS termination. + +For advanced configuration, you have two options: + +- [Annotations]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md" >}}) can be used to rewrite request URIs or inserting additional response headers. +- [Snippets]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-snippets" >}}) can be used to insert raw NGINX configuration, changing generated files. + +Additionally, it is possible to customize the template, described in the [Custom templates]({{< ref "/nic/configuration/global-configuration/custom-templates.md" >}}) topic. diff --git a/content/nic/configuration/ingress-resources/cross-namespace-configuration.md b/content/nic/configuration/ingress-resources/cross-namespace-configuration.md new file mode 100644 index 000000000..3e92b6403 --- /dev/null +++ b/content/nic/configuration/ingress-resources/cross-namespace-configuration.md @@ -0,0 +1,14 @@ +--- +docs: DOCS-594 +doctypes: +- '' +title: Cross-namespace configuration +toc: true +weight: 500 +--- + +This topic explains how to spread Ingress configuration across different namespaces in F5 NGINX Ingress Controller. + +You can spread the Ingress configuration for a common host across multiple Ingress resources using Mergeable Ingress resources. Such resources can belong to the *same* or *different* namespaces. This enables easier management when using a large number of paths. See the [Mergeable Ingress Resources](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/mergeable-ingress-types) example in our GitHub repo. + +As an alternative to Mergeable Ingress resources, you can use [VirtualServer and VirtualServerRoute resources](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/) for cross-namespace configuration. See the [Cross-Namespace Configuration](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/cross-namespace-configuration) example in our GitHub repo. diff --git a/content/nic/configuration/ingress-resources/custom-annotations.md b/content/nic/configuration/ingress-resources/custom-annotations.md new file mode 100644 index 000000000..8bbd136b0 --- /dev/null +++ b/content/nic/configuration/ingress-resources/custom-annotations.md @@ -0,0 +1,146 @@ +--- +docs: DOCS-595 +doctypes: +- '' +title: Custom annotations +toc: true +weight: 300 +--- + +This topic explains how you can use custom annotations with F5 NGINX Ingress Controller. + +Custom annotations enable you to quickly extend the Ingress resource to support many advanced features of NGINX, such as rate limiting, caching, etc. + +## Overview + +NGINX Ingress Controller supports a number of annotations for the Ingress resource that fine tune NGINX configuration (for example, connection timeouts) or enable additional features (for example, JWT validation). The complete list of annotations is available [here](/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations). + +The annotations are provided only for the most common features and use cases, meaning that not every NGINX feature or a customization option is available through the annotations. Additionally, even if an annotation is available, it might not give you the satisfactory level of control of a particular NGINX feature. + +Custom annotations allow you to add an annotation for an NGINX feature that is not available as a regular annotation. In contrast with regular annotations, to add a custom annotation, you don't need to modify the Ingress Controller source code -- just modify the template. Additionally, with a custom annotation, you get full control of how the feature is implemented in NGINX configuration. + +## Usage + +The Ingress Controller generates NGINX configuration for Ingress resources by executing a configuration template. See [NGINX template](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx.ingress.tmpl) or [NGINX Plus template](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/internal/configs/version1/nginx-plus.ingress.tmpl). + +To support custom annotations, the template has access to the information about the Ingress resource - its *name*, *namespace* and *annotations*. It is possible to check if a particular annotation present in the Ingress resource and conditionally insert NGINX configuration directives at multiple NGINX contexts - `http`, `server`, `location` or `upstream`. Additionally, you can get the value that is set to the annotation. + +Consider the following excerpt from the template, which was extended to support two custom annotations: + +```jinja2 +# This is the configuration for {{$.Ingress.Name}}/{{$.Ingress.Namespace}} + +{{if index $.Ingress.Annotations "custom.nginx.org/feature-a"}} +# Insert config for feature A if the annotation is set +{{end}} + +{{with $value := index $.Ingress.Annotations "custom.nginx.org/feature-b"}} +# Insert config for feature B if the annotation is set +# Print the value assigned to the annotation: {{$value}} +{{end}} +``` + +Consider the following Ingress resource and note how we set two annotations: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: example-ingress + namespace: production + annotations: + custom.nginx.org/feature-a: "on" + custom.nginx.org/feature-b: "512" +spec: + rules: + - host: example.com + . . . +``` + +Assuming that the Ingress Controller is using that customized template, it will generate a config for the Ingress resource that will include the following part, generated by our template excerpt: + +```yaml +# This is the configuration for cafe-ingress/default + +# Insert config for feature A if the annotation is set + + + +# Insert config for feature B if the annotation is set +# Print the value assigned to the annotation: 512 +``` + +**Notes**: + +- You can customize the template to insert you custom annotations via [custom templates](/nginx-ingress-controller/configuration/global-configuration/custom-templates). +- The Ingress Controller uses go templates to generate NGINX config. You can read more information about go templates [here](https://golang.org/pkg/text/template/). + +See the examples in the next section that use custom annotations to configure NGINX features. + +### Custom Annotations with Mergeable Ingress Resources + +A Mergeable Ingress resource consists of multiple Ingress resources - one master and one or several minions. Read more about Mergeable Ingress resources [here](/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration). + +If you'd like to use custom annotations with Mergeable Ingress resources, please keep the following in mind: + +- Custom annotations can be used in the Master and in Minions. For Minions, you can access them in the template only when processing locations. + + If you access `$.Ingress` anywhere in the Ingress template, you will get the master Ingress resource. To access a Minion Ingress resource, use `$location.MinionIngress`. However, it is only available when processing locations: + + ```jinja2 + {{range $location := $server.Locations}} + location {{$location.Path}} { + {{with $location.MinionIngress}} + # location for minion {{$location.MinionIngress.Namespace}}/{{$location.MinionIngress.Name}} + {{end}} + } {{end}} + ``` + + **Note**: `$location.MinionIngress` is a pointer. When a regular Ingress resource is processed in the template, the value of the pointer is `nil`. Thus, it is important that you check that `$location.MinionIngress` is not `nil` as in the example above using the `with` action. + +- Minions do not inherent custom annotations of the master. + +### Helper Functions + +Helper functions can be used in the Ingress template to parse the values of custom annotations. + +{{% table %}} +| Function | Input Arguments | Return Arguments | Description | +| ---| ---| ---| --- | +| ``split`` | ``s, sep string`` | ``[]string`` | Splits the string ``s`` into a slice of strings separated by the ``sep``. | +| ``trim`` | ``s string`` | ``string`` | Trims the trailing and leading whitespace from the string ``s``. | +| ``contains`` | ``s, substr string`` | ``bool`` | Tests whether the string ``substr`` is a substring of the string ``s``. | +| ``hasPrefix`` | ``s, prefix string`` | ``bool`` | Tests whether the string ``prefix`` is a prefix of the string ``s``. | +| ``hasSuffix`` | ``s, suffix string`` | ``bool`` | Tests whether the string ``suffix`` is a suffix of the string ``s``. | +| ``toLower`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their lower case. | +| ``toUpper`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their upper case. | +| ``replaceAll`` | ``s, old, new string`` | ``string`` | Replaces all occurrences of ``old`` with ``new`` in the string ``s``. | +{{% /table %}} + +Consider the following custom annotation `custom.nginx.org/allowed-ips`, which expects a comma-separated list of IP addresses: + +```yaml +annotations: + custom.nginx.org/allowed-ips: "192.168.1.3, 10.0.0.13" +``` + + The helper functions can parse the value of the `custom.nginx.org/allowed-ips` annotation, so that in the template you can use each IP address separately. Consider the following template excerpt: + +```jinja2 +{{range $ip := split (index $.Ingress.Annotations "custom.nginx.org/allowed-ips") ","}} + allow {{trim $ip}}; +{{end}} +deny all; +``` + +The template excerpt will generate the following configuration: + +``` +allow 192.168.1.3; +allow 10.0.0.13; +deny all; +``` + +## Example + +See the [custom annotations example](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/examples/ingress-resources/custom-annotations). diff --git a/content/nic/configuration/policy-resource.md b/content/nic/configuration/policy-resource.md new file mode 100644 index 000000000..4188188ef --- /dev/null +++ b/content/nic/configuration/policy-resource.md @@ -0,0 +1,947 @@ +--- +title: Policy resources +weight: 500 +toc: true +type: how-to +product: NIC +docs: DOCS-596 +--- + +The Policy resource allows you to configure features like access control and rate-limiting, which you can add to your [VirtualServer and VirtualServerRoute resources](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/). + +The resource is implemented as a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +This document is the reference documentation for the Policy resource. An example of a Policy for access control is available in our [GitHub repository](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/examples/custom-resources/access-control). + +## Prerequisites + +Policies work together with [VirtualServer and VirtualServerRoute resources](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/), which you need to create separately. + +## Policy Specification + +Below is an example of a policy that allows access for clients from the subnet `10.0.0.0/8` and denies access for any other clients: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: Policy +metadata: + name: allow-localhost +spec: + accessControl: + allow: + - 10.0.0.0/8 +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``accessControl`` | The access control policy based on the client IP address. | [accessControl](#accesscontrol) | No | +|``ingressClassName`` | Specifies which instance of NGINX Ingress Controller must handle the Policy resource. | ``string`` | No | +|``rateLimit`` | The rate limit policy controls the rate of processing requests per a defined key. | [rateLimit](#ratelimit) | No | +|``apiKey`` | The API Key policy configures NGINX to authorize requests which provide a valid API Key in a specified header or query param. | [apiKey](#apikey) | No | +|``basicAuth`` | The basic auth policy configures NGINX to authenticate client requests using HTTP Basic authentication credentials. | [basicAuth](#basicauth) | No | +|``jwt`` | The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. | [jwt](#jwt) | No | +|``ingressMTLS`` | The IngressMTLS policy configures client certificate verification. | [ingressMTLS](#ingressmtls) | No | +|``egressMTLS`` | The EgressMTLS policy configures upstreams authentication and certificate verification. | [egressMTLS](#egressmtls) | No | +|``waf`` | The WAF policy configures WAF and log configuration policies for [NGINX AppProtect]({{< ref "/nic/installation/integrations/app-protect-waf/configuration.md" >}}) | [WAF](#waf) | No | +{{% /table %}} + +\* A policy must include exactly one policy. + +### AccessControl + +The access control policy configures NGINX to deny or allow requests from clients with the specified IP addresses/subnets. + +For example, the following policy allows access for clients from the subnet `10.0.0.0/8` and denies access for any other clients: + +```yaml +accessControl: + allow: + - 10.0.0.0/8 +``` + +In contrast, the policy below does the opposite: denies access for clients from `10.0.0.0/8` and allows access for any other clients: + +```yaml +accessControl: + deny: + - 10.0.0.0/8 +``` +{{< note >}} + +The feature is implemented using the NGINX [ngx_http_access_module](http://nginx.org/en/docs/http/ngx_http_access_module.html). NGINX Ingress Controller access control policy supports either allow or deny rules, but not both (as the module does). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``allow`` | Allows access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | +|``deny`` | Denies access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | \* an accessControl must include either `allow` or `deny`. | +{{% /table %}} + +#### AccessControl Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple access control policies. For example, here we reference two policies, each with configured allow lists: + +```yaml +policies: +- name: allow-policy-one +- name: allow-policy-two +``` + +When you reference more than one access control policy, NGINX Ingress Controller will merge the contents into a single allow list or a single deny list. + +Referencing both allow and deny policies, as shown in the example below, is not supported. If both allow and deny lists are referenced, NGINX Ingress Controller uses just the allow list policies. + +```yaml +policies: +- name: deny-policy +- name: allow-policy-one +- name: allow-policy-two +``` + +### RateLimit + +The rate limit policy configures NGINX to limit the processing rate of requests. + +For example, the following policy will limit all subsequent requests coming from a single IP address once a rate of 10 requests per second is exceeded: + +```yaml +rateLimit: + rate: 10r/s + zoneSize: 10M + key: ${binary_remote_addr} +``` +{{< note >}} + +The feature is implemented using the NGINX [ngx_http_limit_req_module](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html). + +{{< /note >}} + +{{< note >}} + +When the [Zone Sync feature]({{< ref "/nic/configuration/global-configuration/configmap-resource.md#zone-sync" >}}) is enabled with NGINX Plus, the rate limiting zone will be synchronized across all replicas in the cluster. This means all replicas are aware of the requests that have been rate limited by other replicas in the cluster. + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``rate`` | The rate of requests permitted. The rate is specified in requests per second (r/s) or requests per minute (r/m). | ``string`` | Yes | +|``key`` | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ``${}``. For example: ``${binary_remote_addr}``. Accepted variables are ``$binary_remote_addr``, ``$request_uri``,``$request_method``, ``$url``, ``$http_``, ``$args``, ``$arg_``, ``$cookie_``, ``$jwt_claim_``. | ``string`` | Yes | +|``zoneSize`` | Size of the shared memory zone. Only positive values are allowed. Allowed suffixes are ``k`` or ``m``, if none are present ``k`` is assumed. | ``string`` | Yes | +|``delay`` | The delay parameter specifies a limit at which excessive requests become delayed. If not set all excessive requests are delayed. | ``int`` | No | +|``noDelay`` | Disables the delaying of excessive requests while requests are being limited. Overrides ``delay`` if both are set. | ``bool`` | No | +|``burst`` | Excessive requests are delayed until their number exceeds the ``burst`` size, in which case the request is terminated with an error. | ``int`` | No | +|``dryRun`` | Enables the dry run mode. In this mode, the rate limit is not actually applied, but the number of excessive requests is accounted as usual in the shared memory zone. | ``bool`` | No | +|``logLevel`` | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are ``info``, ``notice``, ``warn`` or ``error``. Default is ``error``. | ``string`` | No | +|``rejectCode`` | Sets the status code to return in response to rejected requests. Must fall into the range ``400..599``. Default is ``503``. | ``int`` | No | +|``scale`` | Enables a constant rate-limit by dividing the configured rate by the number of nginx-ingress pods currently serving traffic. This adjustment ensures that the rate-limit remains consistent, even as the number of nginx-pods fluctuates due to autoscaling. **This will not work properly if requests from a client are not evenly distributed across all ingress pods** (Such as with sticky sessions, long lived TCP Connections with many requests, and so forth). In such cases using [zone-sync]({{< ref "/nic/configuration/global-configuration/configmap-resource.md#zone-sync" >}}) instead would give better results. Enabling `zone-sync` will suppress this setting. | ``bool`` | No | +|``condition`` | Add a condition to a rate-limit policy. | [ratelimit.condition](#ratelimitcondition) | No | +{{% /table %}} + +{{< note >}} + +For each policy referenced in a VirtualServer and/or its VirtualServerRoutes, NGINX Ingress Controller will generate a single rate limiting zone defined by the [`limit_req_zone`](http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive. If two VirtualServer resources reference the same policy, NGINX Ingress Controller will generate two different rate limiting zones, one zone per VirtualServer. + +{{< /note >}} + +#### RateLimit Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple rate limit policies. For example, here we reference two policies: + +```yaml +policies: +- name: rate-limit-policy-one +- name: rate-limit-policy-two +``` + +When you reference more than one rate limit policy, NGINX Ingress Controller will configure NGINX to use all referenced rate limits. When you define multiple policies, each additional policy inherits the `dryRun`, `logLevel`, and `rejectCode` parameters from the first policy referenced (`rate-limit-policy-one`, in the example above). + +### RateLimit.Condition + +RateLimit.Condition defines a condition for a rate limit policy. For example: + +```yaml +condition: + jwt: + claim: user_details.level + match: premium + default: true +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``jwt`` | defines a JWT condition to rate limit against. | [ratelimit.condition.jwt](#ratelimitconditionjwt) | No | +|``default`` | sets the rate limit in this policy to be the default if no conditions are met. In a group of policies with the same JWT condition, only one policy can be the default. | ``bool`` | No | +{{% /table %}} + +The rate limit policy with condition is designed to be used in combination with one or more rate limit policies. For example, multiple rate limit policies with [RateLimit.Condition.JWT](#ratelimitconditionjwt) can be used to apply different tiers of rate limit based on the value of a JWT claim. For a practical example of tiered rate limiting by the value of a JWT claim, see the example in our [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/rate-limit-tiered-jwt-claim/README.md). + +### RateLimit.Condition.JWT +{{< note >}} + +This feature is only available with NGINX Plus. + +{{< /note >}} + +RateLimit.Condition.JWT defines a condition for a rate limit by JWT claim. For example, here we define a condition for a rate limit policy that only applies to requests with a JWT claim `user_details.level` with a value `premium`: + +```yaml +jwt: + claim: user_details.level + match: premium +``` + +The rate limit policy will only apply to requests that contain a JWT with the specified claim and value. For example, the following JWT payload will match the JWT condition: + +```json +{ + "user_details": { + "level": "premium" + }, + "sub": "client1" +} +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``claim`` | Claim is the JWT claim to be rate limit by. Nested claims should be separated by ".". | ``string`` | Yes | +|``match`` | the value of the claim to match against. | ``string`` | Yes | +{{% /table %}} + +### APIKey + +The API Key auth policy configures NGINX to authorize client requests based on the presence of a valid API Key in a header or query param specified in the policy. + +{{< note >}} + +The feature is implemented using NGINX [ngx_http_auth_request_module](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html) and [NGINX JavaScript (NJS)](https://nginx.org/en/docs/njs/). + +{{< /note >}} + +The policies' API keys are securely stored using SHA-256 hashing. When a client sends an API Key, it is hashed by NJS and then compared to the hashed API Key in the NGINX config. + +If the hashed keys match, the NGINX JavaScript (NJS) subrequest issues a 204 No Content response to the `auth_request` directive, indicating successful authorization. Conversely, if no API Key is provided in the specified header or query parameter, a 401 Unauthorized response is returned. Similarly, if an invalid key is presented in the expected header or query parameter, a 403 Forbidden response is issued, denying access. + +It is possible to use the [errorPages](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#errorpage) property on a route, to change the default behaviour of 401 or 403 errors. + +At least one header or query param is required. + +The policy below configures NGINX Ingress Controller to require the API Key `password` in the header "my-header". + +```yaml +apiKey: + suppliedIn: + header: + - "my-header" + clientSecret: api-key-secret +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: api-key-secret +type: nginx.org/apikey +data: + client1: cGFzc3dvcmQ= # password +``` + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``suppliedIn`` | `header` or `query`. | | Yes | +|``suppliedIn.header`` | An array of headers that the API Key may appear in. | ``string[]`` | No | +|``suppliedIn.query`` | An array of query params that the API Key may appear in. | ``string[]`` | No | +|``clientSecret`` | The name of the Kubernetes secret that stores the API Key(s). It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/apikey``, and the API Key(s) must be stored in a key: val format where each key is a unique clientID and each value is a unique base64 encoded API Key | ``string`` | Yes | +{{% /table %}} + +{{}}An APIKey Policy must include a minimum of one of the `suppliedIn.header` or `suppliedIn.query` parameters. Both can also be supplied.{{}} + +#### APIKey Merging Behavior + +A VirtualServer or VirtualServerRoute can be associated with only one API Key policy per route or subroute. However, it is possible to replace an API Key policy from a higher-level with a different policy defined on a more specific route. + +For example, a VirtualServer can implement different API Key policies at various levels. In the configuration below, the server-wide api-key-policy-server applies to /backend1 for authorization, as it lacks a more specific policy. Meanwhile, /backend2 uses the api-key-policy-route defined at the route level. + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: virtual-server +spec: + host: virtual-server.example.com + policies: + - name: api-key-policy-server + upstreams: + - name: backend2 + service: backend2-svc + port: 80 + - name: backend1 + service: backend1-svc + port: 80 + routes: + - path: /backend1 + action: + pass: backend1 + - path: /backend2 + action: + pass: backend2 + policies: + - name: api-key-policy-route +``` + +### BasicAuth + +The basic auth policy configures NGINX to authenticate client requests using the [HTTP Basic authentication scheme](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication). + +For example, the following policy will reject all requests that do not include a valid username/password combination in the HTTP header `Authentication` + +```yaml +basicAuth: + secret: htpasswd-secret + realm: "My API" +``` +{{< note >}} +The feature is implemented using the NGINX [ngx_http_auth_basic_module](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html). +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of the Kubernetes secret that stores the Htpasswd configuration. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/htpasswd``, and the config must be stored in the secret under the key ``htpasswd``, otherwise the secret will be rejected as invalid. | ``string`` | Yes | +|``realm`` | The realm for the basic authentication. | ``string`` | No | +{{% /table %}} + +#### BasicAuth Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple basic auth policies. However, only one can be applied. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: basic-auth-policy-one +- name: basic-auth-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `basic-auth-policy-one`, and ignores `basic-auth-policy-two`. + +### JWT Using Local Kubernetes Secret + +{{< note >}} + +This feature is only available with NGINX Plus. + +{{< /note >}} + +The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. + +The following example policy will reject all requests that do not include a valid JWT in the HTTP header `token`: + +```yaml +jwt: + secret: jwk-secret + realm: "My API" + token: $http_token +``` + +You can pass the JWT claims and JOSE headers to the upstream servers. For example: + +```yaml +action: + proxy: + upstream: webapp + requestHeaders: + set: + - name: user + value: ${jwt_claim_user} + - name: alg + value: ${jwt_header_alg} +``` + +We use the `requestHeaders` of the [Action.Proxy](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#actionproxy) to set the values of two headers that NGINX will pass to the upstream servers. + +The value of the `${jwt_claim_user}` variable is the `user` claim of a JWT. For other claims, use `${jwt_claim_name}`, where `name` is the name of the claim. Note that nested claims and claims that include a period (`.`) are not supported. Similarly, use `${jwt_header_name}` where `name` is the name of a header. In our example, we use the `alg` header. + +{{< note >}} + +This feature is implemented using the NGINX Plus [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of the Kubernetes secret that stores the JWK. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/jwk``, and the JWK must be stored in the secret under the key ``jwk``, otherwise the secret will be rejected as invalid. | ``string`` | Yes | +|``realm`` | The realm of the JWT. | ``string`` | Yes | +|``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | +{{% /table %}} + +#### JWT Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple JWT policies. However, only one can be applied: every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: jwt-policy-one +- name: jwt-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `jwt-policy-one`, and ignores `jwt-policy-two`. + +### JWT Using JWKS From Remote Location + +{{< note >}} + +This feature is only available with NGINX Plus. + +{{< /note >}} + +The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens, allowing import of the keys (JWKS) for JWT policy by means of a URL (for a remote server or an identity provider) as a result they don't have to be copied and updated to the IC pod. + +The following example policy will reject all requests that do not include a valid JWT in the HTTP header fetched from the identity provider: + +```yaml +jwt: + realm: MyProductAPI + token: $http_token + jwksURI: + keyCache: 1h +``` + +{{< note >}} + +This feature is implemented using the NGINX Plus directive [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) under [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``jwksURI`` | The remote URI where the request will be sent to retrieve JSON Web Key set| ``string`` | Yes | +|``keyCache`` | Enables in-memory caching of JWKS (JSON Web Key Sets) that are obtained from the ``jwksURI`` and sets a valid time for expiration. | ``string`` | Yes | +|``realm`` | The realm of the JWT. | ``string`` | Yes | +|``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | +{{% /table %}} + +{{< note >}} + +Content caching is enabled by default for each JWT policy with a default time of 12 hours. + +This is done to ensure to improve resiliency by allowing the JWKS (JSON Web Key Set) to be retrieved from the cache even when it has expired. + +{{< /note >}} + +#### JWT Merging Behavior + +This behavior is similar to using a local Kubernetes secret where a VirtualServer/VirtualServerRoute can reference multiple JWT policies. However, only one can be applied: every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: jwt-policy-one +- name: jwt-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `jwt-policy-one`, and ignores `jwt-policy-two`. + +### IngressMTLS + +The IngressMTLS policy configures client certificate verification. + +For example, the following policy will verify a client certificate using the CA certificate specified in the `ingress-mtls-secret`: + +```yaml +ingressMTLS: + clientCertSecret: ingress-mtls-secret + verifyClient: "on" + verifyDepth: 1 +``` + +Below is an example of the `ingress-mtls-secret` using the secret type `nginx.org/ca` + +```yaml +kind: Secret +metadata: + name: ingress-mtls-secret +apiVersion: v1 +type: nginx.org/ca +data: + ca.crt: +``` + +A VirtualServer that references an IngressMTLS policy must: + +- Enable [TLS termination](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualservertls). +- Reference the policy in the VirtualServer [`spec`](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserver-specification). It is not allowed to reference an IngressMTLS policy in a [`route`](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserverroute) or in a VirtualServerRoute [`subroute`](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserverroutesubroute). + +If the conditions above are not met, NGINX will send the `500` status code to clients. + +You can pass the client certificate details, including the certificate, to the upstream servers. For example: + +```yaml +action: + proxy: + upstream: webapp + requestHeaders: + set: + - name: client-cert-subj-dn + value: ${ssl_client_s_dn} # subject DN + - name: client-cert + value: ${ssl_client_escaped_cert} # client certificate in the PEM format (urlencoded) +``` + +We use the `requestHeaders` of the [Action.Proxy](/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#actionproxy) to set the values of the two headers that NGINX will pass to the upstream servers. See the [list of embedded variables](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#variables) that are supported by the `ngx_http_ssl_module`, which you can use to pass the client certificate details. + +{{< note >}} + + The feature is implemented using the NGINX [ngx_http_ssl_module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html). + + {{< /note >}} + +#### Using a Certificate Revocation List + +The IngressMTLS policy supports configuring at CRL for your policy. +This can be done in one of two ways. + +{{< note >}} + + Only one of these configurations options can be used at a time. + +{{< /note >}} + +1. Adding the `ca.crl` field to the `nginx.org/ca` secret type, which accepts a base64 encoded certificate revocation list (crl). + Example YAML: + +```yaml +kind: Secret +metadata: + name: ingress-mtls-secret +apiVersion: v1 +type: nginx.org/ca +data: + ca.crt: + ca.crl: +``` + +2. Adding the `crlFileName` field to your IngressMTLS policy spec with the name of the CRL file. + +{{< note >}} + +This configuration option should only be used when using a CRL that is larger than 1MiB. + +Otherwise we recommend using the `nginx.org/ca` secret type for managing your CRL. + +{{< /note >}} + +Example YAML: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: Policy +metadata: + name: ingress-mtls-policy +spec: +ingressMTLS: + clientCertSecret: ingress-mtls-secret + crlFileName: webapp.crl + verifyClient: "on" + verifyDepth: 1 +``` + +**IMPORTANT NOTE** +When configuring a CRL with the `ingressMTLS.crlFileName` field, there is additional context to keep in mind: + +1. NGINX Ingress Controller will expect the CRL, in this case `webapp.crl`, will be in `/etc/nginx/secrets`. A volume mount will need to be added to NGINX Ingress Controller deployment add your CRL to `/etc/nginx/secrets` +2. When updating the content of your CRL (e.g a new certificate has been revoked), NGINX will need to be reloaded to pick up the latest changes. Depending on your environment this may require updating the name of your CRL and applying this update to your `ingress-mtls.yaml` policy to ensure NGINX picks up the latest CRL. + +Please refer to the Kubernetes documentation on [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) to find the best implementation for your environment. + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``clientCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``, otherwise the secret will be rejected as invalid. | ``string`` | Yes | +|``verifyClient`` | Verification for the client. Possible values are ``"on"``, ``"off"``, ``"optional"``, ``"optional_no_ca"``. The default is ``"on"``. | ``string`` | No | +|``verifyDepth`` | Sets the verification depth in the client certificates chain. The default is ``1``. | ``int`` | No | +|``crlFileName`` | The file name of the Certificate Revocation List. NGINX Ingress Controller will look for this file in `/etc/nginx/secrets` | ``string`` | No | +{{% /table %}} + +#### IngressMTLS Merging Behavior + +A VirtualServer can reference only a single IngressMTLS policy. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: ingress-mtls-policy-one +- name: ingress-mtls-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `ingress-mtls-policy-one`, and ignores `ingress-mtls-policy-two`. + +### EgressMTLS + +The EgressMTLS policy configures upstreams authentication and certificate verification. + +For example, the following policy will use `egress-mtls-secret` to authenticate with the upstream application and `egress-trusted-ca-secret` to verify the certificate of the application: + +```yaml +egressMTLS: + tlsSecret: egress-mtls-secret + trustedCertSecret: egress-trusted-ca-secret + verifyServer: on + verifyDepth: 2 +``` + +{{< note >}} + +The feature is implemented using the NGINX [ngx_http_proxy_module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html). + +{{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``tlsSecret`` | The name of the Kubernetes secret that stores the TLS certificate and key. It must be in the same namespace as the Policy resource. The secret must be of the type ``kubernetes.io/tls``, the certificate must be stored in the secret under the key ``tls.crt``, and the key must be stored under the key ``tls.key``, otherwise the secret will be rejected as invalid. | ``string`` | No | +|``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``, otherwise the secret will be rejected as invalid. | ``string`` | No | +|``verifyServer`` | Enables verification of the upstream HTTPS server certificate. | ``bool`` | No | +|``verifyDepth`` | Sets the verification depth in the proxied HTTPS server certificates chain. The default is ``1``. | ``int`` | No | +|``sessionReuse`` | Enables reuse of SSL sessions to the upstreams. The default is ``true``. | ``bool`` | No | +|``serverName`` | Enables passing of the server name through ``Server Name Indication`` extension. | ``bool`` | No | +|``sslName`` | Allows overriding the server name used to verify the certificate of the upstream HTTPS server. | ``string`` | No | +|``ciphers`` | Specifies the enabled ciphers for requests to an upstream HTTPS server. The default is ``DEFAULT``. | ``string`` | No | +|``protocols`` | Specifies the protocols for requests to an upstream HTTPS server. The default is ``TLSv1 TLSv1.1 TLSv1.2``. | ``string`` | No | > Note: the value of ``ciphers`` and ``protocols`` is not validated by NGINX Ingress Controller. As a result, NGINX can fail to reload the configuration. To ensure that the configuration for a VirtualServer/VirtualServerRoute that references the policy was successfully applied, check its [status](/nginx-ingress-controller/configuration/global-configuration/reporting-resources-status/#virtualserver-and-virtualserverroute-resources). The validation will be added in the future releases. | +{{% /table %}} + +#### EgressMTLS Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple EgressMTLS policies. However, only one can be applied. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: egress-mtls-policy-one +- name: egress-mtls-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `egress-mtls-policy-one`, and ignores `egress-mtls-policy-two`. + +### OIDC + +{{< tip >}} + +This feature is disabled by default. To enable it, set the [enable-oidc]({{< ref "/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-oidc" >}}) command-line argument of NGINX Ingress Controller. + +{{< /tip >}} + +The OIDC policy configures NGINX Plus as a relying party for OpenID Connect authentication. + +For example, the following policy will use the client ID `nginx-plus` and the client secret `oidc-secret` to authenticate with the OpenID Connect provider `https://idp.example.com`: + +```yaml +spec: + oidc: + clientID: nginx-plus + clientSecret: oidc-secret + authEndpoint: https://idp.example.com/openid-connect/auth + tokenEndpoint: https://idp.example.com/openid-connect/token + jwksURI: https://idp.example.com/openid-connect/certs + endSessionEndpoint: https://idp.example.com/openid-connect/logout + postLogoutRedirectURI: / + accessTokenEnable: true + pkceEnable: false +``` + +NGINX Plus will pass the ID of an authenticated user to the backend in the HTTP header `username`. + +{{< note >}} + +The feature is implemented using the [reference implementation](https://github.com/nginxinc/nginx-openid-connect/) of NGINX Plus as a relying party for OpenID Connect authentication. + +{{< /note >}} + +#### Prerequisites + +In order to use OIDC, you need to enable [zone synchronization](https://docs.nginx.com/nginx/admin-guide/high-availability/zone_sync/). If you don't set up zone synchronization, NGINX Plus will fail to reload. +You also need to configure a resolver, which NGINX Plus will use to resolve the IDP authorization endpoint. You can find an example configuration [in our GitHub repository](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/examples/custom-resources/oidc#step-7---configure-nginx-plus-zone-synchronization-and-resolver). + +{{< warning >}} + +The configuration in the example doesn't enable TLS and the synchronization between the replica happens in clear text. This could lead to the exposure of tokens. + +{{< /warning >}} + +#### Limitations + +The OIDC policy defines a few internal locations that can't be customized: `/_jwks_uri`, `/_token`, `/_refresh`, `/_id_token_validation`, `/logout`. In addition, as explained below, `/_codexch` is the default value for redirect URI, and `/_logout` is the default value for post logout redirect URI, both of which can be customized. Specifying one of these locations as a route in the VirtualServer or VirtualServerRoute will result in a collision and NGINX Plus will fail to reload. + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``clientID`` | The client ID provided by your OpenID Connect provider. | ``string`` | Yes | +|``clientSecret`` | The name of the Kubernetes secret that stores the client secret provided by your OpenID Connect provider. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/oidc``, and the secret under the key ``client-secret``, otherwise the secret will be rejected as invalid. If PKCE is enabled, this should be not configured. | ``string`` | Yes | +|``authEndpoint`` | URL for the authorization endpoint provided by your OpenID Connect provider. | ``string`` | Yes | +|``authExtraArgs`` | A list of extra URL arguments to pass to the authorization endpoint provided by your OpenID Connect provider. Arguments must be URL encoded, multiple arguments may be included in the list, for example ``[ arg1=value1, arg2=value2 ]`` | ``string[]`` | No | +|``tokenEndpoint`` | URL for the token endpoint provided by your OpenID Connect provider. | ``string`` | Yes | +|``endSessionEndpoint`` | URL provided by your OpenID Connect provider to request the end user be logged out. | ``string`` | No | +|``jwksURI`` | URL for the JSON Web Key Set (JWK) document provided by your OpenID Connect provider. | ``string`` | Yes | +|``scope`` | List of OpenID Connect scopes. The scope ``openid`` always needs to be present and others can be added concatenating them with a ``+`` sign, for example ``openid+profile+email``, ``openid+email+userDefinedScope``. The default is ``openid``. | ``string`` | No | +|``redirectURI`` | Allows overriding the default redirect URI. The default is ``/_codexch``. | ``string`` | No | +|``postLogoutRedirectURI`` | URI to redirect to after the logout has been performed. Requires ``endSessionEndpoint``. The default is ``/_logout``. | ``string`` | No | +|``zoneSyncLeeway`` | Specifies the maximum timeout in milliseconds for synchronizing ID/access tokens and shared values between Ingress Controller pods. The default is ``200``. | ``int`` | No | +|``accessTokenEnable`` | Option of whether Bearer token is used to authorize NGINX to access protected backend. | ``boolean`` | No | +|``pkceEnable`` | Switches Proof Key for Code Exchange on. The OpenID client needs to be in public mode. `clientSecret` is not used in this mode. | ``boolean`` | No | +{{% /table %}} + +{{< note >}} + +Only one OIDC policy can be referenced in a VirtualServer and its VirtualServerRoutes. However, the same policy can still be applied to different routes in the VirtualServer and VirtualServerRoutes. + +{{< /note >}} + +#### OIDC Merging Behavior + +A VirtualServer/VirtualServerRoute can reference only a single OIDC policy. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: oidc-policy-one +- name: oidc-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `oidc-policy-one`, and ignores `oidc-policy-two`. + +## Using Policy + +You can use the usual `kubectl` commands to work with Policy resources, just as with built-in Kubernetes resources. + +For example, the following command creates a Policy resource defined in `access-control-policy-allow.yaml` with the name `webapp-policy`: + +```shell +kubectl apply -f access-control-policy-allow.yaml + +policy.k8s.nginx.org/webapp-policy configured +``` + +You can get the resource by running: + +```shell +kubectl get policy webapp-policy + +NAME AGE +webapp-policy 27m +``` + +For `kubectl get` and similar commands, you can also use the short name `pol` instead of `policy`. + +### WAF {#waf} + +{{< note >}} The feature is implemented using the NGINX Plus [NGINX App Protect WAF Module](https://docs.nginx.com/nginx-app-protect/configuration/). {{< /note >}} + +The WAF policy configures NGINX Plus to secure client requests using App Protect WAF policies. + +For example, the following policy will enable the referenced APPolicy. You can configure multiple APLogConfs with log destinations: + +```yaml +waf: + enable: true + apPolicy: "default/dataguard-alarm" + securityLogs: + - enable: true + apLogConf: "default/logconf" + logDest: "syslog:server=syslog-svc.default:514" + - enable: true + apLogConf: "default/logconf" + logDest: "syslog:server=syslog-svc-secondary.default:514" +``` + +{{< note >}} The field `waf.securityLog` is deprecated and will be removed in future releases.It will be ignored if `waf.securityLogs` is populated. {{< /note >}} + +{{% table %}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables NGINX App Protect WAF. | ``bool`` | Yes | +|``apPolicy`` | The [App Protect WAF policy]({{< ref "/nic/installation/integrations/app-protect-waf/configuration.md#waf-policies" >}}) of the WAF. Accepts an optional namespace. Mutually exclusive with ``apBundle``. | ``string`` | No | +|``apBundle`` | The [App Protect WAF policy bundle]({{< ref "/nic/installation/integrations/app-protect-waf/configuration.md#waf-bundles" >}}). Mutually exclusive with ``apPolicy``. | ``string`` | No | +|``securityLog.enable`` | Enables security log. | ``bool`` | No | +|``securityLog.apLogConf`` | The [App Protect WAF log conf]({{< ref "/nic/installation/integrations/app-protect-waf/configuration.md#waf-logs" >}}) resource. Accepts an optional namespace. Only works with ``apPolicy``. | ``string`` | No | +|``securityLog.apLogBundle`` | The [App Protect WAF log bundle]({{< ref "/nic/installation/integrations/app-protect-waf/configuration.md#waf-bundles" >}}) resource. Only works with ``apBundle``. | ``string`` | No | +|``securityLog.logDest`` | The log destination for the security log. Only accepted variables are ``syslog:server=:``, ``stderr``, ````. | ``string`` | No | +{{% /table %}} + +#### WAF Merging Behavior + +A VirtualServer/VirtualServerRoute can reference multiple WAF policies. However, only one can be applied. Every subsequent reference will be ignored. For example, here we reference two policies: + +```yaml +policies: +- name: waf-policy-one +- name: waf-policy-two +``` + +In this example NGINX Ingress Controller will use the configuration from the first policy reference `waf-policy-one`, and ignores `waf-policy-two`. + +### Applying Policies + +You can apply policies to both VirtualServer and VirtualServerRoute resources. For example: + +- VirtualServer: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: VirtualServer + metadata: + name: cafe + namespace: cafe + spec: + host: cafe.example.com + tls: + secret: cafe-secret + policies: # spec policies + - name: policy1 + upstreams: + - name: coffee + service: coffee-svc + port: 80 + routes: + - path: /tea + policies: # route policies + - name: policy2 + namespace: cafe + route: tea/tea + - path: /coffee + policies: # route policies + - name: policy3 + namespace: cafe + action: + pass: coffee + ``` + + For VirtualServer, you can apply a policy: + * to all routes (spec policies) + * to a specific route (route policies) + + Route policies of the *same type* override spec policies. In the example above, if the type of the policies `policy-1` and `policy-3` is `accessControl`, then for requests to `cafe.example.com/coffee`, NGINX will apply `policy-3`. + + The overriding is enforced by NGINX: the spec policies are implemented in the `server` context of the config, and the route policies are implemented in the `location` context. As a result, the route policies of the same type win. + +- VirtualServerRoute, which is referenced by the VirtualServer above: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: VirtualServerRoute + metadata: + name: tea + namespace: tea + spec: + host: cafe.example.com + upstreams: + - name: tea + service: tea-svc + port: 80 + subroutes: # subroute policies + - path: /tea + policies: + - name: policy4 + namespace: tea + action: + pass: tea + ``` + + For VirtualServerRoute, you can apply a policy to a subroute (subroute policies). + + Subroute policies of the same type override spec policies. In the example above, if the type of the policies `policy-1` (in the VirtualServer) and `policy-4` is `accessControl`, then for requests to `cafe.example.com/tea`, NGINX will apply `policy-4`. As with the VirtualServer, the overriding is enforced by NGINX. + + Subroute policies always override route policies no matter the types. For example, the policy `policy-2` in the VirtualServer route will be ignored for the subroute `/tea`, because the subroute has its own policies (in our case, only one policy `policy4`). If the subroute didn't have any policies, then the `policy-2` would be applied. This overriding is enforced by NGINX Ingress Controller -- the `location` context for the subroute will either have route policies or subroute policies, but not both. + +### Invalid Policies + +NGINX will treat a policy as invalid if one of the following conditions is met: + +- The policy doesn't pass the [comprehensive validation](#comprehensive-validation). +- The policy isn't present in the cluster. +- The policy doesn't meet its type-specific requirements. For example, an `ingressMTLS` policy requires TLS termination enabled in the VirtualServer. + +For an invalid policy, NGINX returns the 500 status code for client requests with the following rules: + +- If a policy is referenced in a VirtualServer `route` or a VirtualServerRoute `subroute`, then NGINX will return the 500 status code for requests for the URIs of that route/subroute. +- If a policy is referenced in the VirtualServer `spec`, then NGINX will return the 500 status code for requests for all URIs of that VirtualServer. + +If a policy is invalid, the VirtualServer or VirtualServerRoute will have the [status](/nginx-ingress-controller/configuration/global-configuration/reporting-resources-status#virtualserver-and-virtualserverroute-resources) with the state `Warning` and the message explaining why the policy wasn't considered invalid. + +### Validation + +Two types of validation are available for the Policy resource: + +- *Structural validation*, done by `kubectl` and the Kubernetes API server. +- *Comprehensive validation*, done by NGINX Ingress Controller. + +#### Structural Validation + +The custom resource definition for the Policy includes a structural OpenAPI schema, which describes the type of every field of the resource. + +If you try to create (or update) a resource that violates the structural schema -- for example, the resource uses a string value instead of an array of strings in the `allow` field -- `kubectl` and the Kubernetes API server will reject the resource. + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f access-control-policy-allow.yaml + + error: error validating "access-control-policy-allow.yaml": error validating data: ValidationError(Policy.spec.accessControl.allow): invalid type for org.nginx.k8s.v1.Policy.spec.accessControl.allow: got "string", expected "array"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f access-control-policy-allow.yaml --validate=false + + The Policy "webapp-policy" is invalid: spec.accessControl.allow: Invalid value: "string": spec.accessControl.allow in body must be of type array: "string" + ``` + +If a resource passes structural validation, then NGINX Ingress Controller's comprehensive validation runs. + +#### Comprehensive Validation + +NGINX Ingress Controller validates the fields of a Policy resource. If a resource is invalid, NGINX Ingress Controller will reject it. The resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can use `kubectl` to check whether or not NGINX Ingress Controller successfully applied a Policy configuration. For our example `webapp-policy` Policy, we can run: + +```shell +kubectl describe pol webapp-policy + +. . . +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated +``` + +Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a Policy `webapp-policy` with an invalid IP `10.0.0.` in the `allow` field, you will get: + +```shell +kubectl describe policy webapp-policy + +. . . +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 7s nginx-ingress-controller Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP +``` + +Note how the events section includes a Warning event with the Rejected reason. + +Additionally, this information is also available in the `status` field of the Policy resource. Note the Status section of the Policy: + +```shell +kubectl describe pol webapp-policy + +. . . +Status: + Message: Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP + Reason: Rejected + State: Invalid +``` + +**Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it. diff --git a/content/nic/configuration/security.md b/content/nic/configuration/security.md new file mode 100644 index 000000000..f0b74f6a3 --- /dev/null +++ b/content/nic/configuration/security.md @@ -0,0 +1,102 @@ +--- +docs: DOCS-597 +doctypes: +- '' +title: Security recommendations +toc: true +weight: 300 +--- + +F5 NGINX Ingress Controller follows Kubernetes best practices: this page outlines configuration specific to NGINX Ingress Controller you may require, including links to examples in the [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples). + +For general guidance, we recommend the official Kubernetes documentation for [Securing a Cluster](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/). + +## Kubernetes recommendations + +### RBAC and Service Accounts + +Kubernetes uses [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) to control the resources and operations available to different types of users. + +NGINX Ingress Controller requires RBAC to configure a [ServiceUser](https://kubernetes.io/docs/concepts/security/service-accounts/#default-service-accounts), and provides least privilege access in its standard deployment configurations: + +- [Helm](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/deployments/rbac/rbac.yaml) +- [Manifests](https://github.com/nginx/kubernetes-ingress/blob/v{{< nic-version >}}/deployments/rbac/rbac.yaml) + +By default, the ServiceAccount has access to all Secret resources in the cluster. + +### Secrets + +[Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) are required by NGINX Ingress Controller for certificates and privacy keys, which Kubernetes stores unencrypted by default. We recommend following the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) to store these Secrets using at-rest encryption. + + +## NGINX Ingress Controller recommendations + +### Configure root filesystem as read-only + +{{< caution >}} + This feature is compatible with [NGINX App Protect WAFv5](https://docs.nginx.com/nginx-app-protect-waf/v5/). It is not compatible with [NGINX App Protect WAFv4](https://docs.nginx.com/nginx-app-protect-waf/v4/) or [NGINX App Protect DoS](https://docs.nginx.com/nginx-app-protect-dos/). +{{< /caution >}} + +NGINX Ingress Controller is designed to be resilient against attacks in various ways, such as running the service as non-root to avoid changes to files. We recommend setting filesystems on all containers to read-only, this includes `nginx-ingress-controller`, though also includes `waf-enforcer` and `waf-config-mgr` when NGINX App Protect WAFv5 is in use. This is so that the attack surface is further reduced by limiting changes to binaries and libraries. + +This is not enabled by default, but can be enabled with **Helm** using the [**readOnlyRootFilesystem**]({{< ref "/nic/installation/installing-nic/installation-with-helm.md#configuration" >}}) argument in security contexts on all containers: `nginx-ingress-controller`, `waf_enforcer` and `waf_config_mgr`. + +For **Manifests**, uncomment the following sections of the deployment and add sections for `waf-enforcer` and `waf-config-mgr` containers: + +- `readOnlyRootFilesystem: true` +- The entire **volumeMounts** section +- The entire **initContainers** section + +The block below shows the code you will look for: + +```yaml +# volumes: +# - name: nginx-etc +# emptyDir: {} +# - name: nginx-cache +# emptyDir: {} +# - name: nginx-lib +# emptyDir: {} +# - name: nginx-log +# emptyDir: {} +. +. +. +# readOnlyRootFilesystem: true +. +. +. +# volumeMounts: +# - mountPath: /etc/nginx +# name: nginx-etc +# - mountPath: /var/cache/nginx +# name: nginx-cache +# - mountPath: /var/lib/nginx +# name: nginx-lib +# - mountPath: /var/log/nginx +# name: nginx-log +``` + +- Add **waf-enforcer** and **waf-config-mgr** container sections +- Add `readOnlyFilesystem: true` in both containers security context sections + +### Prometheus + +If Prometheus metrics are [enabled]({{< ref "/nic/logging-and-monitoring/prometheus.md" >}}), we recommend [using HTTPS]({{< ref "/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-prometheus-tls-secret" >}}). + +### Snippets + +Snippets allow raw NGINX configuration to be inserted into resources. They are intended for advanced NGINX users and could create vulnerabilities in a cluster if misused. + +Snippets are disabled by default. To use snippets, set the [**enable-snippets**]({{< ref"/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-snippets" >}}) command-line argument. + +{{< caution >}} + Snippets are **always** enabled for ConfigMap. +{{< /caution >}} + +For more information, read the following: + +- [Advanced configuration using Snippets]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) +- [Using Snippets with VirtualServer/VirtualServerRoute]({{< ref "/nic/configuration/virtualserver-and-virtualserverroute-resources.md#using-snippets" >}}) +- [Using Snippets with TransportServer]({{< ref "/nic/configuration/transportserver-resource.md#using-snippets" >}}) +- [ConfigMap snippets and custom templates]({{< ref "/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates" >}}) diff --git a/content/nic/configuration/transportserver-resource.md b/content/nic/configuration/transportserver-resource.md new file mode 100644 index 000000000..b725ea2e6 --- /dev/null +++ b/content/nic/configuration/transportserver-resource.md @@ -0,0 +1,416 @@ +--- +title: TransportServer resources +toc: true +weight: 600 +docs: DOCS-598 +--- + +This document is reference material for the TransportServer resource used by F5 NGINX Ingress Controller. + +The TransportServer resource allows you to configure TCP, UDP, and TLS Passthrough load balancing. The resource is implemented as a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources) for specific use cases. + +## Prerequisites + +- For TCP and UDP, the TransportServer resource must be used in conjunction with the [GlobalConfiguration resource]({{< ref "/nic/configuration/global-configuration/globalconfiguration-resource.md" >}}), which must be created separately. +- For TLS Passthrough, make sure to enable the [`-enable-tls-passthrough`]({{< ref "/nic/configuration/global-configuration/command-line-arguments#cmdoption-enable-tls-passthrough.md" >}}) command-line argument of NGINX Ingress Controller. + +## TransportServer Specification + +The TransportServer resource defines load balancing configuration for TCP, UDP, or TLS Passthrough traffic. Below are a few examples: + +- TCP load balancing: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: dns-tcp + spec: + host: dns.example.com + listener: + name: dns-tcp + protocol: TCP + tls: + secret: cafe-secret + upstreams: + - name: dns-app + service: dns-service + port: 5353 + action: + pass: dns-app + ``` + +- UDP load balancing: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: dns-udp + spec: + listener: + name: dns-udp + protocol: UDP + upstreams: + - name: dns-app + service: dns-service + port: 5353 + upstreamParameters: + udpRequests: 1 + udpResponses: 1 + action: + pass: dns-app + ``` + +- TLS passthrough load balancing: + + ```yaml + apiVersion: k8s.nginx.org/v1 + kind: TransportServer + metadata: + name: secure-app + spec: + listener: + name: tls-passthrough + protocol: TLS_PASSTHROUGH + host: app.example.com + upstreams: + - name: secure-app + service: secure-app + port: 8443 + action: + pass: secure-app + ``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``listener`` | The listener on NGINX that will accept incoming connections/datagrams. | [listener](#listener) | Yes | +|``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. Wildcard domains like ``*.example.com`` are not allowed. When specified, NGINX will use this host for SNI-based routing. For TLS Passthrough, this field is required. For TCP with TLS termination, specifying the host enables SNI routing and requires specifying a TLS secret.| ``string`` | No | +|``tls`` | The TLS termination configuration. Not supported for TLS Passthrough load balancing. | [tls](#tls) | No | +|``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | Yes | +|``upstreamParameters`` | The upstream parameters. | [upstreamParameters](#upstreamparameters) | No | +|``action`` | The action to perform for a client connection/datagram. | [action](#action) | Yes | +|``ingressClassName`` | Specifies which Ingress Controller must handle the TransportServer resource. | ``string`` | No | +|``streamSnippets`` | Sets a custom snippet in the ``stream`` context. | ``string`` | No | +|``serverSnippets`` | Sets a custom snippet in the ``server`` context. | ``string`` | No | +{{}} + +\* -- Required for TLS Passthrough load balancing. + +### Listener + +The listener field references a listener that NGINX will use to accept incoming traffic for the TransportServer. For TCP and UDP, the listener must be defined in the [GlobalConfiguration resource]({{< ref "/nic/configuration/global-configuration/globalconfiguration-resource.md" >}}). When referencing a listener, both the name and the protocol must match. For TLS Passthrough, use the built-in listener with the name `tls-passthrough` and the protocol `TLS_PASSTHROUGH`. + +The combination of ``spec.listener.name`` and ``spec.host`` must be unique among all TransportServers. If two TransportServers specify the same combination of ``spec.listener.name`` and ``spec.host``, one of them will be rejected to prevent conflicts. In the case where no host is specified, it is considered an empty string. + +An example: + +```yaml +listener: + name: dns-udp + protocol: UDP +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the listener. | ``string`` | Yes | +|``protocol`` | The protocol of the listener. | ``string`` | Yes | +{{}} + +### TLS + +The tls field defines TLS configuration for a TransportServer. When using TLS termination (not TLS Passthrough), you can specify the host field to enable SNI-based routing, allowing multiple applications to share the same listener port and be distinguished by the TLS SNI hostname. Each application can use its own TLS certificate and key specified via the secret field. + +```yaml +secret: cafe-secret +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the TransportServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). | ``string`` | No | +{{}} + +### Upstream + +The upstream defines a destination for the TransportServer. For example: + +```yaml +name: secure-app +service: secure-app +port: 8443 +maxFails: 3 +maxConns: 100 +failTimeout: 30s +loadBalancingMethod: least_conn +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | +|``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). The service must belong to the same namespace as the resource. If the service doesn't exist, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. | ``string`` | Yes | +|``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. The port must fall into the range ``1..65535``. | ``int`` | Yes | +|``maxFails`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_fails) of unsuccessful attempts to communicate with the server that should happen in the duration set by the failTimeout parameter to consider the server unavailable. The default ``1``. | ``int`` | No | +|``maxConns`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_conns) of maximum connections to the proxied server. Default value is zero, meaning there is no limit. The default is ``0``. | ``int`` | No | +|``failTimeout`` | Sets the [time](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#fail_timeout) during which the specified number of unsuccessful attempts to communicate with the server should happen to consider the server unavailable and the period of time the server will be considered unavailable. The default is ``10s``. | ``string`` | No | +|``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive. Note: this feature is supported only in NGINX Plus. | [healthcheck](#upstreamhealthcheck) | No | +|``loadBalancingMethod`` | The method used to load balance the upstream servers. By default, connections are distributed between the servers using a weighted round-robin balancing method. See the [upstream](http://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) section for available methods and their details. | ``string`` | No | +|``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | +|``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | +{{}} + +### Upstream.Healthcheck + +The Healthcheck defines an [active health check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html?#health_check). In the example below we enable a health check for an upstream and configure all the available parameters: + +```yaml +name: secure-app +service: secure-app +port: 8443 +healthCheck: + enable: true + interval: 20s + timeout: 30s + jitter: 3s + fails: 5 + passes: 5 + port: 8080 +``` + +{{< note >}} This feature is only supported with NGINX Plus. {{< /note >}} + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | +|``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | +|``timeout`` | This overrides the timeout set by [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) which is set in `SessionParameters` for health checks. The default value is ``5s``. | ``string`` | No | +|``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | +|``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | +|``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | +|``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | +|``match`` | Controls the data to send and the response to expect for the healthcheck. | [match](#upstreamhealthcheckmatch) | No | +{{}} + +### Upstream.Healthcheck.Match + +The match controls the data to send and the response to expect for the healthcheck: + +```yaml +match: + send: 'GET / HTTP/1.0\r\nHost: localhost\r\n\r\n' + expect: "~200 OK" +``` + +Both `send` and `expect` fields can contain hexadecimal literals with the prefix `\x` followed by two hex digits, for example, `\x80`. + +See the [match](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) directive for details. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``send`` | A string to send to an upstream server. | ``string`` | No | +|``expect`` | A literal string or a regular expression that the data obtained from the server should match. The regular expression is specified with the preceding ``~*`` modifier (for case-insensitive matching), or the ``~`` modifier (for case-sensitive matching). NGINX Ingress Controller validates a regular expression using the RE2 syntax. | ``string`` | No | +{{}} + +### UpstreamParameters + +The upstream parameters define various parameters for the upstreams: + +```yaml +upstreamParameters: + udpRequests: 1 + udpResponses: 1 + connectTimeout: 60s + nextUpstream: true + nextUpstreamTimeout: 50s + nextUpstreamTries: 1 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``udpRequests`` | The number of datagrams, after receiving which, the next datagram from the same client starts a new session. See the [proxy_requests](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_requests) directive. The default is ``0``. | ``int`` | No | +|``udpResponses`` | The number of datagrams expected from the proxied server in response to a client datagram. See the [proxy_responses](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses) directive. By default, the number of datagrams is not limited. | ``int`` | No | +|``connectTimeout`` | The timeout for establishing a connection with a proxied server. See the [proxy_connect_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_connect_timeout) directive. The default is ``60s``. | ``string`` | No | +|``nextUpstream`` | If a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. See the [proxy_next_upstream](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream) directive. The default is ``true``. | bool | No | +|``nextUpstreamTries`` | The number of tries for passing a connection to the next server. See the [proxy_next_upstream_tries](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries) directive. The default is ``0``. | ``int`` | No | +|``nextUpstreamTimeout`` | The time allowed to pass a connection to the next server. See the [proxy_next_upstream_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout) directive. The default us ``0``. | ``string`` | No | +{{}} + +### SessionParameters + +The session parameters define various parameters for TCP connections and UDP sessions. + +```yaml +sessionParameters: + timeout: 50s +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``timeout`` | The timeout between two successive read or write operations on client or proxied server connections. See [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) directive. The default is ``10m``. | ``string`` | No | +{{}} + +### Action + +The action defines an action to perform for a client connection/datagram. + +In the example below, client connections/datagrams are passed to an upstream `dns-app`: + +```yaml +action: + pass: dns-app +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``pass`` | Passes connections/datagrams to an upstream. The upstream with that name must be defined in the resource. | ``string`` | Yes | +{{}} + +## Using TransportServer + +You can use the usual `kubectl` commands to work with TransportServer resources, similar to Ingress resources. + +For example, the following command creates a TransportServer resource defined in `transport-server-passthrough.yaml` with the name `secure-app`: + +```shell +kubectl apply -f transport-server-passthrough.yaml +``` +```text +transportserver.k8s.nginx.org/secure-app created +``` + +You can get the resource by running: + +```shell +kubectl get transportserver secure-app +``` +```text +NAME AGE +secure-app 46sm +``` + +In the kubectl get and similar commands, you can also use the short name `ts` instead of `transportserver`. + +### Using Snippets + +Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure [access control](http://nginx.org/en/docs/stream/ngx_stream_access_module.html) in a TransportServer: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: TransportServer +metadata: + name: cafe +spec: + host: cafe.example.com + serverSnippets: | + deny 192.168.1.1; + allow 192.168.1.0/24; + upstreams: + - name: tea + service: tea-svc + port: 80 +``` + +Snippets can also be specified for a stream. In the example below, we use snippets to [limit the number of connections](https://nginx.org/en/docs/stream/ngx_stream_limit_conn_module.html): + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: TransportServer +metadata: + name: cafe +spec: + host: cafe.example.com + streamSnippets: limit_conn_zone $binary_remote_addr zone=addr:10m; + serverSnippets: limit_conn addr 1; + upstreams: + - name: tea + service: tea-svc + port: 80 +``` + +{{< note >}} To configure snippets in the `stream` context, use `stream-snippets` ConfigMap key. {{< /note >}} + +For additional information, view the [Advanced configuration with Snippets]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) topic. + +### Validation + +Two types of validation are available for the TransportServer resource: + +- *Structural validation* by the `kubectl` and Kubernetes API server. +- *Comprehensive validation* by NGINX Ingress Controller. + +#### Structural Validation + +The custom resource definition for the TransportServer includes structural OpenAPI schema which describes the type of every field of the resource. + +If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f transport-server-passthrough.yaml + ``` + ```text + error: error validating "transport-server-passthrough.yaml": error validating data: ValidationError(TransportServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.TransportServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f transport-server-passthrough.yaml --validate=false + ``` + ```text + The TransportServer "secure-app" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: + spec.upstreams.port in body must be of type integer: "string" + ``` + +If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. + +#### Comprehensive Validation + +NGINX Ingress Controller validates the fields of a TransportServer resource. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can check if NGINX Ingress Controller successfully applied the configuration for a TransportServer. For our example `secure-app` TransportServer, we can run: + +```shell +kubectl describe ts secure-app +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/secure-app was added or updated +``` + +Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a TransportServer `secure-app` with a pass action that references a non-existing upstream, you will get : + +```shell +kubectl describe ts secure-app +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 2s nginx-ingress-controller TransportServer default/secure-app is invalid and was rejected: spec.action.pass: Not found: "some-app" +``` + +Note how the events section includes a Warning event with the Rejected reason. + +**Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. + +## Customization via ConfigMap + +The [ConfigMap]({{< ref "/nic/configuration/global-configuration/configmap-resource.md" >}}) keys (except for `stream-snippets`, `stream-log-format`, `resolver-addresses`, `resolver-ipv6`, `resolver-valid` and `resolver-timeout`) do not affect TransportServer resources. diff --git a/content/nic/configuration/virtualserver-and-virtualserverroute-resources.md b/content/nic/configuration/virtualserver-and-virtualserverroute-resources.md new file mode 100644 index 000000000..1acd05b02 --- /dev/null +++ b/content/nic/configuration/virtualserver-and-virtualserverroute-resources.md @@ -0,0 +1,1101 @@ +--- +title: VirtualServer and VirtualServerRoute resources +toc: true +weight: 700 +docs: DOCS-599 +--- + +This document is reference material for the VirtualServer and VirtualServerRoute resources used by F5 NGINX Ingress Controller. + +VirtualServer and VirtualServerRoute resources are load balancing configurations recommended as an alternative to the Ingress resource. + +They enable use cases not supported with the Ingress resource, such as traffic splitting and advanced content-based routing. The resources are implemented as [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). + +The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources) for specific use cases. + +--- + +## VirtualServer specification + +The VirtualServer resource defines load balancing configuration for a domain name, such as `example.com`. Below is an example of such configuration: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: cafe +spec: + host: cafe.example.com + listener: + http: http-8083 + https: https-8443 + tls: + secret: cafe-secret + gunzip: on + upstreams: + - name: tea + service: tea-svc + port: 80 + - name: coffee + service: coffee-svc + port: 80 + routes: + - path: /tea + action: + pass: tea + - path: /coffee + action: + pass: coffee + - path: ~ ^/decaf/.*\\.jpg$ + action: + pass: coffee + - path: = /green/tea + action: + pass: tea +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. The ``host`` value needs to be unique among all Ingress and VirtualServer resources. See also [Handling Host and Listener Collisions](/nginx-ingress-controller/configuration/host-and-listener-collisions). | ``string`` | Yes | +|``listener`` | Sets a custom HTTP and/or HTTPS listener. Valid fields are `listener.http` and `listener.https`. Each field must reference the name of a valid listener defined in a GlobalConfiguration resource | [listener](#virtualserverlistener) | No | +|``tls`` | The TLS termination configuration. | [tls](#virtualservertls) | No | +|``gunzip`` | Enables or disables [decompression](https://docs.nginx.com/nginx/admin-guide/web-server/compression/) of gzipped responses for clients. Allowed values “on”/“off”, “true”/“false” or “yes”/“no”. If the ``gunzip`` value is not set, it defaults to ``off``. | ``boolean`` | No | +|``externalDNS`` | The externalDNS configuration for a VirtualServer. | [externalDNS](#virtualserverexternaldns) | No | +|``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer. | ``string`` | No | +|``policies`` | A list of policies. | [[]policy](#virtualserverpolicy) | No | +|``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | +|``routes`` | A list of routes. | [[]route](#virtualserverroute) | No | +|``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServer resource. | ``string`` | No | +|``internalRoute`` | Specifies if the VirtualServer resource is an internal route or not. | ``boolean`` | No | +|``http-snippets`` | Sets a custom snippet in the http context. | ``string`` | No | +|``server-snippets`` | Sets a custom snippet in server context. Overrides the ``server-snippets`` ConfigMap key. | ``string`` | No | +{{}} + +### VirtualServer.TLS + +The tls field defines TLS configuration for a VirtualServer. For example: + +```yaml +secret: cafe-secret +redirect: + enable: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the VirtualServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the host of the VirtualServer. If the secret is not specified but [wildcard TLS secret](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments#cmdoption-wildcard-tls-secret) is configured, NGINX will use the wildcard secret for TLS termination. | ``string`` | No | +|``redirect`` | The redirect configuration of the TLS for a VirtualServer. | [tls.redirect](#virtualservertlsredirect) | No | ### VirtualServer.TLS.Redirect | +|``cert-manager`` | The cert-manager configuration of the TLS for a VirtualServer. | [tls.cert-manager](#virtualservertlscertmanager) | No | ### VirtualServer.TLS.CertManager | +{{}} + +### VirtualServer.TLS.Redirect + +The redirect field configures a TLS redirect for a VirtualServer: + +```yaml +enable: true +code: 301 +basedOn: scheme +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables a TLS redirect for a VirtualServer. The default is ``False``. | ``boolean`` | No | +|``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | +|``basedOn`` | The attribute of a request that NGINX will evaluate to send a redirect. The allowed values are ``scheme`` (the scheme of the request) or ``x-forwarded-proto`` (the ``X-Forwarded-Proto`` header of the request). The default is ``scheme``. | ``string`` | No | ### VirtualServer.Policy | +{{}} + +### VirtualServer.TLS.CertManager + +The cert-manager field configures x509 automated Certificate management for VirtualServer resources using cert-manager (cert-manager.io). Please see the [cert-manager configuration documentation](https://cert-manager.io/docs/configuration/) for more information on deploying and configuring Issuers. Example: + +```yaml +cert-manager: + cluster-issuer: "my-issuer-name" +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``issuer`` | the name of an Issuer. An Issuer is a cert-manager resource which describes the certificate authority capable of signing certificates. The Issuer must be in the same namespace as the VirtualServer resource. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | +|``cluster-issuer`` | the name of a ClusterIssuer. A ClusterIssuer is a cert-manager resource which describes the certificate authority capable of signing certificates. It does not matter which namespace your VirtualServer resides, as ClusterIssuers are non-namespaced resources. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | +|``issuer-kind`` | The kind of the external issuer resource, for example AWSPCAIssuer. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | +|``issuer-group`` | The API group of the external issuer controller, for example awspca.cert-manager.io. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | +|``common-name`` | This field allows you to configure spec.commonName for the Certificate to be generated. This configuration adds a CN to the x509 certificate. | ``string`` | No | +|``duration`` | This field allows you to configure spec.duration field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | +|``renew-before`` | this annotation allows you to configure spec.renewBefore field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | +|``usages`` | This field allows you to configure spec.usages field for the Certificate to be generated. Pass a string with comma-separated values i.e. ``key agreement,digital signature, server auth``. An exhaustive list of supported key usages can be found in the [the cert-manager api documentation](https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.KeyUsage). | ``string`` | No | +|``issue-temp-cert`` | When ``true``, ask cert-manager for a [temporary self-signed certificate](https://cert-manager.io/docs/usage/certificate/#temporary-certificates-while-issuing) pending the issuance of the Certificate. This allows HTTPS-only servers to use ACME HTTP01 challenges when the TLS secret does not exist yet. | ``boolean`` | No | +{{}} + +### VirtualServer.Listener +The listener field defines a custom HTTP and/or HTTPS listener. +The respective listeners used must reference the name of a listener defined using a [GlobalConfiguration](/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/) resource. +For example: +```yaml +http: http-8083 +https: https-8443 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``http`` | The name of am HTTP listener defined in a [GlobalConfiguration](/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/) resource. | ``string`` | No | +|``https`` | The name of an HTTPS listener defined in a [GlobalConfiguration](/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/) resource. | ``string`` | No | +{{}} + +### VirtualServer.ExternalDNS + +The externalDNS field configures controlling DNS records dynamically for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Please see the [ExternalDNS configuration documentation](https://kubernetes-sigs.github.io/external-dns/) for more information on deploying and configuring ExternalDNS and Providers. Example: + +```yaml +enable: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables ExternalDNS integration for a VirtualServer resource. The default is ``false``. | ``string`` | No | +|``labels`` | Configure labels to be applied to the Endpoint resources that will be consumed by ExternalDNS. | ``map[string]string`` | No | +|``providerSpecific`` | Configure provider specific properties which holds the name and value of a configuration which is specific to individual DNS providers. | [[]ProviderSpecific](#virtualserverexternaldnsproviderspecific) | No | +|``recordTTL`` | TTL for the DNS record. This defaults to 0 if not defined. See [the ExternalDNS TTL documentation for provider-specific defaults](https://kubernetes-sigs.github.io/external-dns/v0.14.2/ttl/#providers) | ``int64`` | No | +|``recordType`` | The record Type that should be created, e.g. "A", "AAAA", "CNAME". This is automatically computed based on the external endpoints if not defined. | ``string`` | No | +{{}} + +### VirtualServer.ExternalDNS.ProviderSpecific + +The providerSpecific field of the externalDNS block allows the specification of provider specific properties which is a list of key value pairs of configurations which are specific to individual DNS providers. Example: + +```yaml +- name: my-name + value: my-value +- name: my-name2 + value: my-value2 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the key value pair. | ``string`` | Yes | +|``value`` | The value of the key value pair. | ``string`` | Yes | +{{}} + +### VirtualServer.Policy + +The policy field references a [Policy resource](/nginx-ingress-controller/configuration/policy-resource/) by its name and optional namespace. For example: + +```yaml +name: access-control +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of a policy. If the policy doesn't exist or invalid, NGINX will respond with an error response with the `500` status code. | ``string`` | Yes | +|``namespace`` | The namespace of a policy. If not specified, the namespace of the VirtualServer resource is used. | ``string`` | No | +{{}} + +### VirtualServer.Route + +The route defines rules for matching client requests to actions like passing a request to an upstream. For example: + +```yaml + path: /tea + action: + pass: tea +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``path`` | The path of the route. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix (must start with ``/`` ) or an exact match (must start with ``=`` ), the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. The path must be unique among the paths of all routes of the VirtualServer. Check the [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive for more information. | ``string`` | Yes | +|``policies`` | A list of policies. The policies override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying Policies](/nginx-ingress-controller/configuration/policy-resource/#applying-policies) for more details. | [[]policy](#virtualserverpolicy) | No | +|``action`` | The default action to perform for a request. | [action](#action) | No | +|``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer route. | ``string`` | No | +|``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | +|``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | +|``route`` | The name of a VirtualServerRoute resource that defines this route. If the VirtualServerRoute belongs to a different namespace than the VirtualServer, you need to include the namespace. For example, ``tea-namespace/tea``. | ``string`` | No | +|``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | +|``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` ConfigMap key. | ``string`` | No | +{{}} + +\* -- a route must include exactly one of the following: `action`, `splits`, or `route`. + +## VirtualServerRoute specification + +The VirtualServerRoute resource defines a route for a VirtualServer. It can consist of one or multiple subroutes. The VirtualServerRoute is an alternative to [Mergeable Ingress types](/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration). + +In the example below, the VirtualServer `cafe` from the namespace `cafe-ns` defines a route with the path `/coffee`, which is further defined in the VirtualServerRoute `coffee` from the namespace `coffee-ns`. + +VirtualServer: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: cafe + namespace: cafe-ns +spec: + host: cafe.example.com + upstreams: + - name: tea + service: tea-svc + port: 80 + routes: + - path: /tea + action: + pass: tea + - path: /coffee + route: coffee-ns/coffee +``` + +VirtualServerRoute: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServerRoute +metadata: + name: coffee + namespace: coffee-ns +spec: + host: cafe.example.com + upstreams: + - name: latte + service: latte-svc + port: 80 + - name: espresso + service: espresso-svc + port: 80 + subroutes: + - path: /coffee/latte + action: + pass: latte + - path: /coffee/espresso + action: + pass: espresso +``` + +Note that each subroute must have a `path` that starts with the same prefix (here `/coffee`), which is defined in the route of the VirtualServer. Additionally, the `host` in the VirtualServerRoute must be the same as the `host` of the VirtualServer. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. Must be the same as the ``host`` of the VirtualServer that references this resource. | ``string`` | Yes | +|``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | +|``subroutes`` | A list of subroutes. | [[]subroute](#virtualserverroutesubroute) | No | +|``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServerRoute resource. Must be the same as the ``ingressClassName`` of the VirtualServer that references this resource. | ``string``_ | No | +{{}} + +### VirtualServerRoute.Subroute + +The subroute defines rules for matching client requests to actions like passing a request to an upstream. For example: + +```yaml +path: /coffee +action: + pass: coffee +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``path`` | The path of the subroute. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix, the path must start with the same path as the path of the route of the VirtualServer that references this resource. In the case of an exact or regex match, the path must be the same as the path of the route of the VirtualServer that references this resource. A matching path of the route of the VirtualServer but in different type is not accepted, e.g. a regex path (`~/match`) cannot be used with a prefix path in VirtualServer (`/match`) In the case of a prefix or an exact match, the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. The path must be unique among the paths of all subroutes of the VirtualServerRoute. | ``string`` | Yes | +|``policies`` | A list of policies. The policies override *all* policies defined in the route of the VirtualServer that references this resource. The policies also override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying Policies](/nginx-ingress-controller/configuration/policy-resource/#applying-policies) for more details. | [[]policy](#virtualserverpolicy) | No | +|``action`` | The default action to perform for a request. | [action](#action) | No | +|``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServerRoute subroute. | ``string`` | No | +|``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | +|``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | +|``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | +|``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` of the VirtualServer (if set) or the ``location-snippets`` ConfigMap key. | ``string`` | No | +{{}} + +\* -- a subroute must include exactly one of the following: `action` or `splits`. + +## Common VirtualServer and VirtualServerRoute specifications + +### Upstream + +The upstream defines a destination for the routing configuration. For example: + +```yaml +name: tea +service: tea-svc +subselector: + version: canary +port: 80 +lb-method: round_robin +fail-timeout: 10s +max-fails: 1 +max-conns: 32 +keepalive: 32 +connect-timeout: 30s +read-timeout: 30s +send-timeout: 30s +next-upstream: "error timeout non_idempotent" +next-upstream-timeout: 5s +next-upstream-tries: 10 +client-max-body-size: 2m +tls: + enable: true +``` + +**Note**: The WebSocket protocol is supported without any additional configuration. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | +|``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). The service must belong to the same namespace as the resource. If the service doesn't exist, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. For NGINX Plus only, services of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) are also supported (check the [prerequisites](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/ingress-resources/externalname-services#prerequisites) ). | ``string`` | Yes | +|``subselector`` | Selects the pods within the service using label keys and values. By default, all pods of the service are selected. Note: the specified labels are expected to be present in the pods when they are created. If the pod labels are updated, NGINX Ingress Controller will not see that change until the number of the pods is changed. | ``map[string]string`` | No | +|``use-cluster-ip`` | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method`` and ``next-upstream``) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | ``boolean`` | No | +|``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. The port must fall into the range ``1..65535``. | ``uint16`` | Yes | +|``lb-method`` | The load [balancing method](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method). To use the round-robin method, specify ``round_robin``. The default is specified in the ``lb-method`` ConfigMap key. | ``string`` | No | +|``fail-timeout`` | The time during which the specified number of unsuccessful attempts to communicate with an upstream server should happen to consider the server unavailable. See the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the server directive. The default is set in the ``fail-timeout`` ConfigMap key. | ``string`` | No | +|``max-fails`` | The number of unsuccessful attempts to communicate with an upstream server that should happen in the duration set by the ``fail-timeout`` to consider the server unavailable. See the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the server directive. The default is set in the ``max-fails`` ConfigMap key. | ``int`` | No | +|``max-conns`` | The maximum number of simultaneous active connections to an upstream server. See the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the server directive. By default there is no limit. Note: if keepalive connections are enabled, the total number of active and idle keepalive connections to an upstream server may exceed the ``max_conns`` value. | ``int`` | No | +|``keepalive`` | Configures the cache for connections to upstream servers. The value ``0`` disables the cache. See the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. The default is set in the ``keepalive`` ConfigMap key. | ``int`` | No | +|``connect-timeout`` | The timeout for establishing a connection with an upstream server. See the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) directive. The default is specified in the ``proxy-connect-timeout`` ConfigMap key. | ``string`` | No | +|``read-timeout`` | The timeout for reading a response from an upstream server. See the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) directive. The default is specified in the ``proxy-read-timeout`` ConfigMap key. | ``string`` | No | +|``send-timeout`` | The timeout for transmitting a request to an upstream server. See the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) directive. The default is specified in the ``proxy-send-timeout`` ConfigMap key. | ``string`` | No | +|``next-upstream`` | Specifies in which cases a request should be passed to the next upstream server. See the [proxy_next_upstream](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) directive. The default is ``error timeout``. | ``string`` | No | +|``next-upstream-timeout`` | The time during which a request can be passed to the next upstream server. See the [proxy_next_upstream_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) directive. The ``0`` value turns off the time limit. The default is ``0``. | ``string`` | No | +|``next-upstream-tries`` | The number of possible tries for passing a request to the next upstream server. See the [proxy_next_upstream_tries](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) directive. The ``0`` value turns off this limit. The default is ``0``. | ``int`` | No | +|``client-max-body-size`` | Sets the maximum allowed size of the client request body. See the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. The default is set in the ``client-max-body-size`` ConfigMap key. | ``string`` | No | +|``tls`` | The TLS configuration for the Upstream. | [tls](#upstreamtls) | No | +|``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive. Note: this feature is supported only in NGINX Plus. | [healthcheck](#upstreamhealthcheck) | No | +|``slow-start`` | The slow start allows an upstream server to gradually recover its weight from 0 to its nominal value after it has been recovered or became available or when the server becomes available after a period of time it was considered unavailable. By default, the slow start is disabled. See the [slow_start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start) parameter of the server directive. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods and will be ignored. | ``string`` | No | +|``queue`` | Configures a queue for an upstream. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request. By default, no queue is configured. Note: this feature is supported only in NGINX Plus. | [queue](#upstreamqueue) | No | +|``buffering`` | Enables buffering of responses from the upstream server. See the [proxy_buffering](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) directive. The default is set in the ``proxy-buffering`` ConfigMap key. | ``boolean`` | No | +|``buffers`` | Configures the buffers used for reading a response from the upstream server for a single connection. | [buffers](#upstreambuffers) | No | +|``buffer-size`` | Sets the size of the buffer used for reading the first part of a response received from the upstream server. See the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) directive. The default is set in the ``proxy-buffer-size`` ConfigMap key. | ``string`` | No | +|``ntlm`` | Allows proxying requests with NTLM Authentication. See the [ntlm](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ntlm) directive. In order for NTLM authentication to work, it is necessary to enable keepalive connections to upstream servers using the ``keepalive`` field. Note: this feature is supported only in NGINX Plus.| ``boolean`` | No | +|``type`` |The type of the upstream. Supported values are ``http`` and ``grpc``. The default is ``http``. For gRPC, it is necessary to enable HTTP/2 in the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#listeners) and configure TLS termination in the VirtualServer. | ``string`` | No | +|``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | +|``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | +{{}} + +### Upstream.Buffers + +The buffers field configures the buffers used for reading a response from the upstream server for a single connection: + +```yaml +number: 4 +size: 8K +``` + +See the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive for additional information. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``number`` | Configures the number of buffers. The default is set in the ``proxy-buffers`` ConfigMap key. | ``int`` | Yes | +|``size`` | Configures the size of a buffer. The default is set in the ``proxy-buffers`` ConfigMap key. | ``string`` | Yes | +{{}} + +### Upstream.TLS + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables HTTPS for requests to upstream servers. The default is ``False`` , meaning that HTTP will be used. Note: by default, NGINX will not verify the upstream server certificate. To enable the verification, configure an [EgressMTLS Policy](/nginx-ingress-controller/configuration/policy-resource/#egressmtls). | ``boolean`` | No | +{{}} + +### Upstream.Queue + +The queue field configures a queue. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request: + +```yaml +size: 10 +timeout: 60s +``` + +See [`queue`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#queue) directive for additional information. + +Note: This feature is supported only in NGINX Plus. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``size`` | The size of the queue. | ``int`` | Yes | +|``timeout`` | The timeout of the queue. A request cannot be queued for a period longer than the timeout. The default is ``60s``. | ``string`` | No | +{{}} + +### Upstream.Healthcheck + +The Healthcheck defines an [active health check](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/). In the example below we enable a health check for an upstream and configure all the available parameters, including the `slow-start` parameter combined with [`mandatory` and `persistent`](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#mandatory-health-checks): + +```yaml +name: tea +service: tea-svc +port: 80 +slow-start: 30s +healthCheck: + enable: true + path: /healthz + interval: 20s + jitter: 3s + fails: 5 + passes: 5 + port: 8080 + tls: + enable: true + connect-timeout: 10s + read-timeout: 10s + send-timeout: 10s + headers: + - name: Host + value: my.service + statusMatch: "! 500" + mandatory: true + persistent: true + keepalive-time: 60s +``` + +Note: This feature is supported only in NGINX Plus. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | +|``path`` | The path used for health check requests. The default is ``/``. This not configurable for gRPC type upstreams. | ``string`` | No | +|``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | +|``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | +|``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | +|``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | +|``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | +|``tls`` | The TLS configuration used for health check requests. By default, the ``tls`` field of the upstream is used. | [upstream.tls](#upstreamtls) | No | +|``connect-timeout`` | The timeout for establishing a connection with an upstream server. By default, the ``connect-timeout`` of the upstream is used. | ``string`` | No | +|``read-timeout`` | The timeout for reading a response from an upstream server. By default, the ``read-timeout`` of the upstream is used. | ``string`` | No | +|``send-timeout`` | The timeout for transmitting a request to an upstream server. By default, the ``send-timeout`` of the upstream is used. | ``string`` | No | +|``headers`` | The request headers used for health check requests. NGINX Plus always sets the ``Host`` , ``User-Agent`` and ``Connection`` headers for health check requests. | [[]header](#header) | No | +|``statusMatch`` | The expected response status codes of a health check. By default, the response should have status code 2xx or 3xx. Examples: ``"200"`` , ``"! 500"`` , ``"301-303 307"``. See the documentation of the [match](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html?#match) directive. This not supported for gRPC type upstreams. | ``string`` | No | +|``grpcStatus`` | The expected [gRPC status code](https://github.com/grpc/grpc/blob/master/doc/statuscodes.md#status-codes-and-their-use-in-grpc) of the upstream server response to the [Check method](https://github.com/grpc/grpc/blob/master/doc/health-checking.md). Configure this field only if your gRPC services do not implement the gRPC health checking protocol. For example, configure ``12`` if the upstream server responds with `12 (UNIMPLEMENTED)` status code. Only valid on gRPC type upstreams. | ``int`` | No | +|``grpcService`` | The gRPC service to be monitored on the upstream server. Only valid on gRPC type upstreams. | ``string`` | No | +|``mandatory`` | Require every newly added server to pass all configured health checks before NGINX Plus sends traffic to it. If this is not specified, or is set to false, the server will be initially considered healthy. When combined with [slow-start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start), it gives a new server more time to connect to databases and “warm up” before being asked to handle their full share of traffic. | ``bool`` | No | +|``persistent`` | Set the initial “up” state for a server after reload if the server was considered healthy before reload. Enabling persistent requires that the mandatory parameter is also set to `true`. | ``bool`` | No | +|``keepalive-time`` | Enables [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) connections for health checks and specifies the time during which requests can be processed through one keepalive connection. The default is ``60s``. | ``string`` | No | +{{}} + +### Upstream.SessionCookie + +The SessionCookie field configures session persistence which allows requests from the same client to be passed to the same upstream server. The information about the designated upstream server is passed in a session cookie generated by NGINX Plus. + +In the example below, we configure session persistence with a session cookie for an upstream and configure all the available parameters: + +```yaml +name: tea +service: tea-svc +port: 80 +sessionCookie: + enable: true + name: srv_id + path: / + expires: 1h + domain: .example.com + httpOnly: false + secure: true + samesite: strict +``` + +See the [`sticky`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html?#sticky) directive for additional information. The session cookie corresponds to the `sticky cookie` method. + +Note: This feature is supported only in NGINX Plus. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``enable`` | Enables session persistence with a session cookie for an upstream server. The default is ``false``. | ``boolean`` | No | +|``name`` | The name of the cookie. | ``string`` | Yes | +|``path`` | The path for which the cookie is set. | ``string`` | No | +|``expires`` | The time for which a browser should keep the cookie. Can be set to the special value ``max`` , which will cause the cookie to expire on ``31 Dec 2037 23:55:55 GMT``. | ``string`` | No | +|``domain`` | The domain for which the cookie is set. | ``string`` | No | +|``httpOnly`` | Adds the ``HttpOnly`` attribute to the cookie. | ``boolean`` | No | +|``secure`` | Adds the ``Secure`` attribute to the cookie. | ``boolean`` | No | +|``samesite`` | Adds the ``SameSite`` attribute to the cookie. The allowed values are: ``strict``, ``lax``, ``none`` | ``string`` | No | +{{}} + +### Header + +The header defines an HTTP Header: + +```yaml +name: Host +value: example.com +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. | ``string`` | No | +{{}} + +### Action + +The action defines an action to perform for a request. + +In the example below, client requests are passed to an upstream `coffee`: + +```yaml + path: /coffee + action: + pass: coffee +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``pass`` | Passes requests to an upstream. The upstream with that name must be defined in the resource. | ``string`` | No | +|``redirect`` | Redirects requests to a provided URL. | [action.redirect](#actionredirect) | No | +|``return`` | Returns a preconfigured response. | [action.return](#actionreturn) | No | +|``proxy`` | Passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). | [action.proxy](#actionproxy) | No | +{{}} + +\* -- an action must include exactly one of the following: `pass`, `redirect`, `return` or `proxy`. + +### Action.Redirect + +The redirect action defines a redirect to return for a request. + +In the example below, client requests are passed to a url `http://www.nginx.com`: + +```yaml +redirect: + url: http://www.nginx.com + code: 301 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` , ``$http_x_forwarded_proto`` , ``$request_uri`` , ``$host``. Variables must be enclosed in curly braces. For example: ``${host}${request_uri}``. | ``string`` | Yes | +|``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | +{{}} + +### Action.Return + +The return action defines a preconfigured response for a request. + +In the example below, NGINX will respond with the preconfigured response for every request: + +```yaml +return: + code: 200 + type: text/plain + body: "Hello World\n" + headers: + - name: x-coffee + value: espresso +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``code`` | The status code of the response. The allowed values are: ``2XX``, ``4XX`` or ``5XX``. The default is ``200``. | ``int`` | No | +|``type`` | The MIME type of the response. The default is ``text/plain``. | ``string`` | No | +|``body`` | The body of the response. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``Request is ${request_uri}\n``. | ``string`` | Yes | +|``headers`` | The custom headers of the response. | [[]Action.Return.Header](#actionreturnheader) | No | +{{}} + +\* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing` and `$connections_waiting`. + +### Action.Return.Header + +The header defines an HTTP Header for a canned response in an actionReturn: + +```yaml +name: x-coffee +value: espresso +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. | ``string`` | Yes | +{{}} + +### Action.Proxy + +The proxy action passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). + +In the example below, the request URI is rewritten to `/`, and the request and the response headers are modified: + +```yaml +proxy: + upstream: coffee + requestHeaders: + pass: true + set: + - name: My-Header + value: Value + - name: Client-Cert + value: ${ssl_client_escaped_cert} + responseHeaders: + add: + - name: My-Header + value: Value + - name: IC-Nginx-Version + value: ${nginx_version} + always: true + hide: + - x-internal-version + ignore: + - Expires + - Set-Cookie + pass: + - Server + rewritePath: / +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``upstream`` | The name of the upstream which the requests will be proxied to. The upstream with that name must be defined in the resource. | ``string`` | Yes | +|``requestHeaders`` | The request headers modifications. | [action.Proxy.RequestHeaders](#actionproxyrequestheaders) | No | +|``responseHeaders`` | The response headers modifications. | [action.Proxy.ResponseHeaders](#actionproxyresponseheaders) | No | +|``rewritePath`` | The rewritten URI. If the route path is a regular expression -- starts with `~` -- the `rewritePath` can include capture groups with ``$1-9``. For example `$1` for the first group, and so on. For more information, check the [rewrite](https://github.com/nginx/kubernetes-ingress/tree/v{{< nic-version >}}/examples/custom-resources/rewrites) example. | ``string`` | No | +{{}} + +### Action.Proxy.RequestHeaders + +The RequestHeaders field modifies the headers of the request to the proxied upstream server. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``pass`` | Passes the original request headers to the proxied upstream server. See the [proxy_pass_request_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_headers) directive for more information. Default is true. | ``bool`` | No | +|``set`` | Allows redefining or appending fields to present request headers passed to the proxied upstream servers. See the [proxy_set_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive for more information. | [[]header](#actionproxyrequestheaderssetheader) | No | +{{}} + +### Action.Proxy.RequestHeaders.Set.Header + +The header defines an HTTP Header: + +```yaml +name: My-Header +value: My-Value +``` + +It is possible to override the default value of the `Host` header, which NGINX Ingress Controller sets to [`$host`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_host): + +```yaml +name: Host +value: example.com +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | +{{}} + +\* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` (NGINX Plus only) and `$jwt_header_` (NGINX Plus only). + +### Action.Proxy.ResponseHeaders + +The ResponseHeaders field modifies the headers of the response to the client. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``hide`` | The headers that will not be passed* in the response to the client from a proxied upstream server. See the [proxy_hide_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directive for more information. | ``[]string`` | No | +|``pass`` | Allows passing the hidden header fields* to the client from a proxied upstream server. See the [proxy_pass_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directive for more information. | ``[]string`` | No | +|``ignore`` | Disables processing of certain headers** to the client from a proxied upstream server. See the [proxy_ignore_headers](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ignore_headers) directive for more information. | ``[]string`` | No | +|``add`` | Adds headers to the response to the client. | [[]addHeader](#addheader) | No | +{{}} + +\* -- Default hidden headers are: `Date`, `Server`, `X-Pad` and `X-Accel-...`. + +\** -- The following fields can be ignored: `X-Accel-Redirect`, `X-Accel-Expires`, `X-Accel-Limit-Rate`, `X-Accel-Buffering`, `X-Accel-Charset`, `Expires`, `Cache-Control`, `Set-Cookie` and `Vary`. + +### AddHeader + +The addHeader defines an HTTP Header with an optional `always` field: + +```yaml +name: My-Header +value: My-Value +always: true +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | +|``always`` | If set to true, add the header regardless of the response status code**. Default is false. See the [add_header](http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header) directive for more information. | ``bool`` | No | +{{}} + +\* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` (NGINX Plus only) and `$jwt_header_` (NGINX Plus only). + +{{< note >}} If `always` is false, the response header is added only if the response status code is any of `200`, `201`, `204`, `206`, `301`, `302`, `303`, `304`, `307` or `308`. {{< /note >}} + +### Split + +The split defines a weight for an action as part of the splits configuration. + +In the example below NGINX passes 80% of requests to the upstream `coffee-v1` and the remaining 20% to `coffee-v2`: + +```yaml +splits: +- weight: 80 + action: + pass: coffee-v1 +- weight: 20 + action: + pass: coffee-v2 +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``weight`` | The weight of an action. Must fall into the range ``0..100``. The sum of the weights of all splits must be equal to ``100``. | ``int`` | Yes | +|``action`` | The action to perform for a request. | [action](#action) | Yes | +{{}} + +### Match + +The match defines a match between conditions and an action or splits. + +In the example below, NGINX routes requests with the path `/coffee` to different upstreams based on the value of the cookie `user`: + +- `user=john` -> `coffee-future` +- `user=bob` -> `coffee-deprecated` +- If the cookie is not set or not equal to either `john` or `bob`, NGINX routes to `coffee-stable` + +```yaml +path: /coffee +matches: +- conditions: + - cookie: user + value: john + action: + pass: coffee-future +- conditions: + - cookie: user + value: bob + action: + pass: coffee-deprecated +action: + pass: coffee-stable +``` + +In the next example, NGINX routes requests based on the value of the built-in [`$request_method` variable](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_method), which represents the HTTP method of a request: + +- all POST requests -> `coffee-post` +- all non-POST requests -> `coffee` + +```yaml +path: /coffee +matches: +- conditions: + - variable: $request_method + value: POST + action: + pass: coffee-post +action: + pass: coffee +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``conditions`` | A list of conditions. Must include at least 1 condition. | [[]condition](#condition) | Yes | +|``action`` | The action to perform for a request. | [action](#action) | No | +|``splits`` | The splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | +{{}} + +{{< note >}} A match must include exactly one of the following: `action` or `splits`. {{< /note >}} + +### Condition + +The condition defines a condition in a match. + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``header`` | The name of a header. Must consist of alphanumeric characters or ``-``. | ``string`` | No | +|``cookie`` | The name of a cookie. Must consist of alphanumeric characters or ``_``. | ``string`` | No | +|``argument`` | The name of an argument. Must consist of alphanumeric characters or ``_``. | ``string`` | No | +|``variable`` | The name of an NGINX variable. Must start with ``$``. See the list of the supported variables below the table. | ``string`` | No | +|``value`` | The value to match the condition against. How to define a value is shown below the table. | ``string`` | Yes | +{{}} + +{{< note >}} a condition must include exactly one of the following: `header`, `cookie`, `argument` or `variable`. {{< /note >}} + +Supported NGINX variables: `$args`, `$http2`, `$https`, `$remote_addr`, `$remote_port`, `$query_string`, `$request`, `$request_body`, `$request_uri`, `$request_method`, `$scheme`. Find the documentation for each variable [here](https://nginx.org/en/docs/varindex.html). + +The value supports two kinds of matching: + +- *Case-insensitive string comparison*. For example: + - `john` -- case-insensitive matching that succeeds for strings, such as `john`, `John`, `JOHN`. + - `!john` -- negation of the case-insensitive matching for john that succeeds for strings, such as `bob`, `anything`, `''` (empty string). +- *Matching with a regular expression*. Note that NGINX supports regular expressions compatible with those used by the Perl programming language (PCRE). For example: + - `~^yes` -- a case-sensitive regular expression that matches any string that starts with `yes`. For example: `yes`, `yes123`. + - `!~^yes` -- negation of the previous regular expression that succeeds for strings like `YES`, `Yes123`, `noyes`. (The negation mechanism is not part of the PCRE syntax). + - `~*no$` -- a case-insensitive regular expression that matches any string that ends with `no`. For example: `no`, `123no`, `123NO`. + +{{< note >}} A value must not include any unescaped double quotes (`"`) and must not end with an unescaped backslash (`\`). For example, the following are invalid values: `some"value`, `somevalue\`. {{< /note >}} + +### ErrorPage + +The errorPage defines a custom response for a route for the case when either an upstream server responds with (or NGINX generates) an error status code. The custom response can be a redirect or a canned response. See the [error_page](https://nginx.org/en/docs/http/ngx_http_core_module.html#error_page) directive for more information. + +```yaml +path: /coffee +errorPages: +- codes: [502, 503] + redirect: + code: 301 + url: https://nginx.org +- codes: [404] + return: + code: 200 + body: "Original resource not found, but success!" +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``codes`` | A list of error status codes. | ``[]int`` | Yes | +|``redirect`` | The redirect action for the given status codes. | [errorPage.Redirect](#errorpageredirect) | No | +|``return`` | The canned response action for the given status codes. | [errorPage.Return](#errorpagereturn) | No | +{{}} + +{{< note >}} An errorPage must include exactly one of the following: `return` or `redirect`. {{< /note >}} + +### ErrorPage.Redirect + +The redirect defines a redirect for an errorPage. + +In the example below, NGINX responds with a redirect when a response from an upstream server has a 404 status code. + +```yaml +codes: [404] +redirect: + code: 301 + url: ${scheme}://cafe.example.com/error.html +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | +|``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` and ``$http_x_forwarded_proto``. Variables must be enclosed in curly braces. For example: ``${scheme}``. | ``string`` | Yes | +{{}} + +### ErrorPage.Return + +The return defines a canned response for an errorPage. + +In the example below, NGINX responds with a canned response when a response from an upstream server has either 401 or 403 status code. + +```yaml +codes: [401, 403] +return: + code: 200 + type: application/json + body: | + {\"msg\": \"You don't have permission to do this\"} + headers: + - name: x-debug-original-statuses + value: ${upstream_status} +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``code`` | The status code of the response. The default is the status code of the original response. | ``int`` | No | +|``type`` | The MIME type of the response. The default is ``text/html``. | ``string`` | No | +|``body`` | The body of the response. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | Yes | +|``headers`` | The custom headers of the response. | [[]errorPage.Return.Header](#errorpagereturnheader) | No | +{{}} + +### ErrorPage.Return.Header + +The header defines an HTTP Header for a canned response in an errorPage: + +```yaml +name: x-debug-original-statuses +value: ${upstream_status} +``` + +{{}} +|Field | Description | Type | Required | +| ---| ---| ---| --- | +|``name`` | The name of the header. | ``string`` | Yes | +|``value`` | The value of the header. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | No | +{{}} + +## Using VirtualServer and VirtualServerRoute + +You can use the usual `kubectl` commands to work with VirtualServer and VirtualServerRoute resources, similar to Ingress resources. + +For example, the following command creates a VirtualServer resource defined in `cafe-virtual-server.yaml` with the name `cafe`: + +```shell +kubectl apply -f cafe-virtual-server.yaml +``` +```text +virtualserver.k8s.nginx.org "cafe" created +``` + +You can get the resource by running: + +```shell +kubectl get virtualserver cafe +``` +```text +NAME STATE HOST IP PORTS AGE +cafe Valid cafe.example.com 12.13.23.123 [80,443] 3m +``` + +In `kubectl get` and similar commands, you can use the short name `vs` instead of `virtualserver`. + +Similarly, for VirtualServerRoute you can use `virtualserverroute` or the short name `vsr`. + +### Using Snippets + +Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure several NGINX features in a VirtualServer: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: cafe + namespace: cafe +spec: + http-snippets: | + limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s; + proxy_cache_path /tmp keys_zone=one:10m; + host: cafe.example.com + tls: + secret: cafe-secret + server-snippets: | + limit_req zone=mylimit burst=20; + upstreams: + - name: tea + service: tea-svc + port: 80 + - name: coffee + service: coffee-svc + port: 80 + routes: + - path: /tea + location-snippets: | + proxy_cache one; + proxy_cache_valid 200 10m; + action: + pass: tea + - path: /coffee + action: + pass: coffee +``` + +For additional information, view the [Advanced configuration with Snippets]({{< ref "/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md" >}}) topic. + +### Validation + +Two types of validation are available for VirtualServer and VirtualServerRoute resources: + +- *Structural validation* by the `kubectl` and Kubernetes API server. +- *Comprehensive validation* by NGINX Ingress Controller. + +#### Structural Validation + +The custom resource definitions for VirtualServer and VirtualServerRoute include structural OpenAPI schema which describes the type of every field of those resources. + +If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: + +- Example of `kubectl` validation: + + ```shell + kubectl apply -f cafe-virtual-server.yaml + ``` + ```text + error: error validating "cafe-virtual-server.yaml": error validating data: ValidationError(VirtualServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.VirtualServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false + ``` + +- Example of Kubernetes API server validation: + + ```shell + kubectl apply -f cafe-virtual-server.yaml --validate=false + ``` + ```text + The VirtualServer "cafe" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: + spec.upstreams.port in body must be of type integer: "string" + ``` + +If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. + +#### Comprehensive Validation + +NGINX Ingress Controller validates the fields of the VirtualServer and VirtualServerRoute resources. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. + +You can check if NGINX Ingress Controller successfully applied the configuration for a VirtualServer. For our example `cafe` VirtualServer, we can run: + +```shell +kubectl describe vs cafe +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated +``` + +Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. + +If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a VirtualServer `cafe` with two upstream with the same name `tea`, you will get: + +```shell +kubectl describe vs cafe +``` +```text +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Rejected 12s nginx-ingress-controller VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" +``` + +Note how the events section includes a Warning event with the Rejected reason. + +Additionally, this information is also available in the `status` field of the VirtualServer resource. Note the Status section of the VirtualServer: + +```shell +kubectl describe vs cafe +``` +```text +... +Status: + External Endpoints: + Ip: 12.13.23.123 + Ports: [80,443] + Message: VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" + Reason: Rejected + State: Invalid +``` + +NGINX Ingress Controller validates VirtualServerRoute resources in a similar way. + +**Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. + +## Customization using ConfigMap + +You can customize the NGINX configuration for VirtualServer and VirtualServerRoutes resources using the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource). Most of the ConfigMap keys are supported, with the following exceptions: + +- `proxy-hide-headers` +- `proxy-pass-headers` +- `hsts` +- `hsts-max-age` +- `hsts-include-subdomains` +- `hsts-behind-proxy` +- `redirect-to-https` +- `ssl-redirect` diff --git a/content/nic/glossary.md b/content/nic/glossary.md new file mode 100644 index 000000000..d76f9e3db --- /dev/null +++ b/content/nic/glossary.md @@ -0,0 +1,30 @@ +--- +description: null +docs: DOCS-1446 +title: Glossary +weight: 10000 +--- + +This is a glossary of terms related to F5 NGINX Ingress Controller and Kubernetes as a whole. + +--- + +## Ingress {#ingress} + +_Ingress_ refers to an _Ingress Resource_, a Kubernetes API object which allows access to [Services](https://kubernetes.io/docs/concepts/services-networking/service/) within a cluster. They are managed by an [Ingress Controller]({{< ref "/nic/glossary.md#ingress-controller">}}). + +_Ingress_ resources enable the following functionality: + +- **Load balancing**, extended through the use of Services +- **Content-based routing**, using hosts and paths +- **TLS/SSL termination**, based on hostnames + +For additional information, please read the official [Kubernetes Ingress Documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/). + +--- + +## Ingress Controller {#ingress-controller} + +*Ingress Controllers* are applications within a Kubernetes cluster that enable [Ingress]({{< ref "/nic/glossary.md#ingress">}}) resources to function. They are not automatically deployed with a Kubernetes cluster, and can vary in implementation based on intended use, such as load balancing algorithms for Ingress resources. + +[The design of NGINX Ingress Controller]({{< ref "/nic/overview/design.md">}}) explains the technical details of NGINX Ingress Controller. diff --git a/content/nic/installation/_index.md b/content/nic/installation/_index.md new file mode 100644 index 000000000..4ed49bfcb --- /dev/null +++ b/content/nic/installation/_index.md @@ -0,0 +1,8 @@ +--- +title: Installation +description: +weight: 400 +menu: + docs: + parent: NGINX Ingress Controller +--- diff --git a/content/nic/installation/build-nginx-ingress-controller.md b/content/nic/installation/build-nginx-ingress-controller.md new file mode 100644 index 000000000..4f37d8f25 --- /dev/null +++ b/content/nic/installation/build-nginx-ingress-controller.md @@ -0,0 +1,203 @@ +--- +title: Build NGINX Ingress Controller +toc: true +weight: 400 +type: how-to +product: NIC +docs: DOCS-1453 +--- + +This document describes how to build an F5 NGINX Ingress Controller image from source code and upload it to a private Docker registry. + +It also includes information on the Makefile targets and variables. + +{{}} If you do not need to build a custom image, see the [pre-built image options](#pre-built-images) at the end of this guide. {{}} + +## Before you begin + +To get started, you need the following software installed on your machine: + +- [Docker v19.03 or higher](https://docs.docker.com/engine/release-notes/19.03/) +- [GNU Make](https://www.gnu.org/software/make/) +- [git](https://git-scm.com/) +- [OpenSSL](https://www.openssl.org/), optionally, if you would like to generate a self-signed certificate and a key for the default server. +- For NGINX Plus users, download the certificate (_nginx-repo.crt_) and key (_nginx-repo.key_) from [MyF5](https://my.f5.com). + +Although NGINX Ingress Controller is written in Golang, you don't need to have Golang installed. + +You can download the precompiled binary file or build NGINX Ingress Controller in a Docker container. + +--- + +## Prepare the environment {#prepare-environment} + +Get your system ready for building and pushing the NGINX Ingress Controller image. + +1. Sign in to your private registry. Replace `` with the path to your own private registry. + + ```shell + docker login + ``` + +2. Clone the NGINX Ingress Controller GitHub repository. Replace `` with the version of NGINX Ingress Controller you want. + + ```shell + git clone https://github.com/nginx/kubernetes-ingress.git --branch + cd kubernetes-ingress + ``` + + For instance if you want to clone version v{{< nic-version >}}, the commands to run would be: + + ```shell + git clone https://github.com/nginx/kubernetes-ingress.git --branch v{{< nic-version >}} + cd kubernetes-ingress + ``` + +--- + +## Build the NGINX Ingress Controller image {#build-image} + +After setting up your environment, follow these steps to build the NGINX Ingress Controller image. + +{{< note >}} If you have a local Golang environment and want to build the binary yourself, remove `TARGET=download` from the make commands. If you don't have Golang but still want to build the binary, use `TARGET=container`. {{< /note >}} + +### For NGINX + +Build the image. Replace `` with your private registry's path. + +- For a Debian-based image: + + ```shell + make debian-image PREFIX=/nginx-ingress TARGET=download + ``` + +- For an Alpine-based image: + + ```shell + make alpine-image PREFIX=/nginx-ingress TARGET=download + ``` + +**What to expect**: The image is built and tagged with a version number, which is derived from the `VERSION` variable in the [_Makefile_](#makefile-details). This version number is used for tracking and deployment purposes. + +### For NGINX Plus + +Place your NGINX Plus license files (_nginx-repo.crt_ and _nginx-repo.key_) in the project's root folder. To verify they're in place, run: + +```shell +ls nginx-repo.* +``` + +You should see: + +```text +nginx-repo.crt nginx-repo.key +``` + +Build the image. Replace `` with your private registry's path. + +```shell +make debian-image-plus PREFIX=/nginx-plus-ingress TARGET=download +``` + +
+ +**What to expect**: The image is built and tagged with a version number, which is derived from the `VERSION` variable in the [_Makefile_](#makefile-details). This version number is used for tracking and deployment purposes. + +{{}} If a patch for NGINX Plus is released, make sure to rebuild your image to get the latest version. If your system is caching the Docker layers and not updating the packages, add `DOCKER_BUILD_OPTIONS="--pull --no-cache"` to the make command. {{}} + +--- + +## Push the image to your private registry {#push-image} + +Once you've successfully built the NGINX or NGINX Plus Ingress Controller image, the next step is to upload it to your private Docker registry. This makes the image available for deployment to your Kubernetes cluster. + +### For NGINX + +Upload the NGINX image. If you're using a custom tag, append `TAG=your-tag` to the command. Replace `` with your private registry's path. + +```shell +make push PREFIX=/nginx-ingress +``` + +### For NGINX Plus + +Upload the NGINX Plus image. Like with the NGINX image, if you're using a custom tag, add `TAG=your-tag` to the end of the command. Replace `` with your private registry's path. + +```shell +make push PREFIX=/nginx-plus-ingress +``` + +--- + +## Makefile details {#makefile-details} + +This section provides comprehensive information on the targets and variables available in the _Makefile_. These targets and variables allow you to customize how you build, tag, and push your NGINX or NGINX Plus images. + +### Key Makefile targets {#key-makefile-targets} + +{{}}To view available _Makefile_ targets, run `make` with no target or type `make help`.{{}} + +Key targets include: + +{{}} +|
Target | Description | +|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| _build_ | Creates the NGINX Ingress Controller binary with your local Go environment. | +| _alpine-image_ | Builds an Alpine-based image with NGINX. | +| _alpine-image-plus_ | Builds an Alpine-based image with NGINX Plus. | +| _alpine-image-plus-fips_ | Builds an Alpine-based image with NGINX Plus and FIPS. | +| _alpine-image-nap-v5-plus-fips_ | Builds an Alpine-based image with NGINX Plus, the [NGINX App Protect WAF v5](/nginx-app-protect/) module and FIPS. | +| _debian-image_ | Builds a Debian-based image with NGINX. | +| _debian-image-plus_ | Builds a Debian-based image with NGINX Plus. | +| _debian-image-nap-plus_ | Builds a Debian-based image with NGINX Plus and the [NGINX App Protect WAF](/nginx-app-protect/) module. | +| _debian-image-nap-v5-plus_ | Builds a Debian-based image with NGINX Plus and the [NGINX App Protect WAF v5](/nginx-app-protect/) module. | +| _debian-image-dos-plus_ | Builds a Debian-based image with NGINX Plus and the [NGINX App Protect DoS](/nginx-app-protect-dos/) module. | +| _debian-image-nap-dos-plus_ | Builds a Debian-based image with NGINX Plus, [NGINX App Protect WAF](/nginx-app-protect/) and [NGINX App Protect DoS](/nginx-app-protect-dos/) modules. | +| _ubi-image_ | Builds a UBI-based image with NGINX for [OpenShift](https://www.openshift.com/) clusters. | +| _ubi-image-plus_ | Builds a UBI-based image with NGINX Plus for [OpenShift](https://www.openshift.com/) clusters. | +| _ubi-image-nap-plus_ | Builds a UBI-based image with NGINX Plus and the [NGINX App Protect WAF](/nginx-app-protect/) module for [OpenShift](https://www.openshift.com/) clusters. | +| _ubi-image-nap-v5-plus_ | Builds a UBI-based image with NGINX Plus and the [NGINX App Protect WAF v5](/nginx-app-protect/) module for [OpenShift](https://www.openshift.com/) clusters. | +| _ubi-image-dos-plus_ | Builds a UBI-based image with NGINX Plus and the [NGINX App Protect DoS](/nginx-app-protect-dos/) module for [OpenShift](https://www.openshift.com/) clusters. | +| _ubi-image-nap-dos-plus_ |

Builds a UBI-based image with NGINX Plus, [NGINX App Protect WAF](/nginx-app-protect/) and the [NGINX App Protect DoS](/nginx-app-protect-dos/) module for [OpenShift](https://www.openshift.com/) clusters.

**Important**: Save your RHEL organization and activation keys in a file named _rhel_license_ at the project root.

For instance:

RHEL_ORGANIZATION=1111111
RHEL_ACTIVATION_KEY=your-key
| +{{}} + +--- + +### Additional useful targets {#other-makefile-targets} + +A few other useful targets: + +{{}} +|
Target
| Description | +|---------------------------------------|---------------| +| _push_ | Pushes the built image to the Docker registry. Configures with `PREFIX` and `TAG`. | +| _all_ | Runs `test`, `lint`, `verify-codegen`, `update-crds`, and `debian-image`. Stops and reports an error if any of these targets fail. | +| _test_ | Runs unit tests. | +{{
}} + +### Makefile variables you can customize {#makefile-variables} + +The _Makefile_ includes several key variables. You have the option to either modify these variables directly in the _Makefile_ or override them when you run the `make` command. + +{{}} +|
Variable
| Description | +|-----------------------------------------|---------------| +| _ARCH_ | Defines the architecture for the image and binary. The default is `amd64`, but you can also use `arm64`. | +| _PREFIX_ | Gives the image its name. The default is `nginx/nginx-ingress`. | +| _TAG_ | Adds a tag to the image. This is often the version of NGINX Ingress Controller. | +| _DOCKER\_BUILD\_OPTIONS_ | Allows for additional [options](https://docs.docker.com/engine/reference/commandline/build/#options) during the `docker build` process, like `--pull`. | +| _TARGET_ |

Determines the build environment. NGINX Ingress Controller compiles locally in a Golang environment by default. Ensure the NGINX Ingress Controller repo resides in your `$GOPATH` if you select this option.

Alternatively, you can set `TARGET=container` to build using a Docker [Golang](https://hub.docker.com/_/golang/) container. To skip compiling the binary if you're on a specific tag or the latest `main` branch commit, set `TARGET=download`.

| +{{
}} + +--- + +## Alternatives to building your own image {#pre-built-images} + +If you prefer not to build your own NGINX Ingress Controller image, you can use pre-built images. Here are your options: + +**NGINX Ingress Controller**: Download the image `nginx/nginx-ingress` from [DockerHub](https://hub.docker.com/r/nginx/nginx-ingress) or [GitHub](https://github.com/nginx/kubernetes-ingress/pkgs/container/kubernetes-ingress). + +**NGINX Plus Ingress Controller**: You have two options for this: + +- Download the image using your NGINX Ingress Controller subscription certificate and key. View the [Get NGINX Ingress Controller from the F5 Registry]({{< ref "/nic/installation/nic-images/get-registry-image" >}}) topic. +- Use your NGINX Ingress Controller subscription JWT token to get the image. View the [Get the NGINX Ingress Controller image with JWT]({{< ref "/nic/installation/nic-images/get-image-using-jwt.md" >}}) topic. diff --git a/content/nic/installation/create-license-secret.md b/content/nic/installation/create-license-secret.md new file mode 100644 index 000000000..91be0f9f9 --- /dev/null +++ b/content/nic/installation/create-license-secret.md @@ -0,0 +1,155 @@ +--- +title: Create a license Secret +toc: true +weight: 300 +nd-content-type: how-to +nd-product: NIC +--- + +This document explains how to create and use a license secret for F5 NGINX Ingress Controller. + +# Overview + +NGINX Plus Ingress Controller requires a valid JSON Web Token (JWT) to download the container image from the F5 registry. From version 4.0.0, this JWT token is also required to run NGINX Plus. + +This requirement is part of F5’s broader licensing program and aligns with industry best practices. The JWT will streamline subscription renewals and usage reporting, helping you manage your NGINX Plus subscription more efficiently. The [telemetry](#telemetry) data we collect helps us improve our products and services to better meet your needs. + +The JWT is required for validating your subscription and reporting telemetry data. For environments connected to the internet, telemetry is automatically sent to F5’s licensing endpoint. In offline environments, telemetry is routed through [NGINX Instance Manager]({{< ref "/nim/" >}}). By default usage is reported every hour and also whenever NGINX is reloaded. + +{{< note >}} Read the [subscription licenses topic]({{< ref "/solutions/about-subscription-licenses.md#for-internet-connected-environments" >}}) for a list of IPs associated with F5's licensing endpoint (`product.connect.nginx.com`). {{}} + +## Set up your NGINX Plus license + +### Download the JWT + +{{< include "/nic/installation/download-jwt.md" >}} + +### Create the Secret + +The JWT needs to be configured before deploying NGINX Ingress Controller. The JWT will be stored in a Kubernetes Secret of type `nginx.com/license`, and can be created with the following command. + +```shell +kubectl create secret generic license-token --from-file=license.jwt= --type=nginx.com/license -n +``` +You can now delete the downloaded `.jwt` file. + +{{< note >}} +The Secret needs to be in the same Namespace as the NGINX Ingress Controller Pod(s). +{{}} + +{{< include "/nic/installation/jwt-password-note.md" >}} + +### Use the NGINX Plus license Secret + +If using a name other than the default `license-token`, provide the name of this Secret when installing NGINX Ingress Controller: + +{{}} + +{{%tab name="Helm"%}} + +Specify the Secret name using the `controller.mgmt.licenseTokenSecretName` Helm value. + +For detailed guidance on creating the Management block via Helm, refer to the [Helm configuration documentation]({{< ref "/nic/installation/installing-nic/installation-with-helm/#configuration" >}}). + +{{% /tab %}} + +{{%tab name="Manifests"%}} + +Specify the Secret name in the `license-token-secret-name` Management ConfigMap key. + +For detailed guidance on creating the Management ConfigMap, refer to the [Management ConfigMap Resource Documentation]({{< ref "/nic/configuration/global-configuration/mgmt-configmap-resource/" >}}). + +{{% /tab %}} + +{{}} + +If you are reporting to the default licensing endpoint, then you can now proceed with [installing NGINX Ingress Controller]({{< ref "nic/installation/installing-nic/" >}}). Otherwise, follow the steps below to configure reporting to NGINX Instance Manager + +### Create report for NGINX Instance Manager {#nim} + +If you are deploying NGINX Ingress Controller in an "air-gapped" environment you will need to report to [NGINX Instance Manager]({{< ref "/nim/" >}}) instead of the default licensing endpoint. + +First, you must specify the endpoint of your NGINX Instance Manager. + +{{}} + +{{%tab name="Helm"%}} + +Specify the endpoint using the `controller.mgmt.usageReport.endpoint` helm value. + +{{% /tab %}} + +{{%tab name="Manifests"%}} + +Specify the endpoint in the `usage-report-endpoint` Management ConfigMap key. + +{{% /tab %}} + +{{}} + +#### Configure SSL certificates and SSL trusted certificates {#nim-cert} + +To configure SSL certificates or SSL trusted certificates, extra steps are necessary. + +To use Client Auth with NGINX Instance Manager, first create a Secret of type `kubernetes.io/tls` in the same namespace as the NGINX Ingress Controller pods. + +```shell +kubectl create secret tls ssl-certificate --cert= --key= -n +``` + +To provide a SSL trusted certificate, and an optional Certificate Revocation List, create a Secret of type `nginx.org/ca` in the Namespace that the NIC Pod(s) are in. + +```shell +kubectl create secret generic ssl-trusted-certificate \ + --from-file=ca.crt= \ + --from-file=ca.crl= \ # optional + --type=nginx.org/ca +``` + +Providing an optional CRL (certificate revocation list) will configure the [`ssl_crl`](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directive. + +{{}} + +{{%tab name="Helm"%}} + +Specify the SSL certificate Secret name using the `controller.mgmt.sslCertificateSecretName` Helm value. + +Specify the SSL trusted certificate Secret name using the `controller.mgmt.sslTrustedCertificateSecretName` Helm value. + +{{% /tab %}} + +{{%tab name="Manifests"%}} + +Specify the SSL certificate Secret name in the `ssl-certificate-secret-name` management ConfigMap key. + +Specify the SSL trusted certificate Secret name in the `ssl-trusted-certificate-secret-name` management ConfigMap key. + +{{% /tab %}} + +{{}} + +
+ +Once these Secrets are created and configured, you can now [install NGINX Ingress Controller ]({{< ref "/nic/installation/installing-nic/" >}}). + + +## What’s reported and how it’s protected {#telemetry} + +NGINX Plus reports the following data every hour by default: + +- **NGINX version and status**: The version of NGINX Plus running on the instance. +- **Instance UUID**: A unique identifier for each NGINX Plus instance. +- **Traffic data**: + - **Bytes received from and sent to clients**: HTTP and stream traffic volume between clients and NGINX Plus. + - **Bytes received from and sent to upstreams**: HTTP and stream traffic volume between NGINX Plus and upstream servers. + - **Client connections**: The number of accepted client connections (HTTP and stream traffic). + - **Requests handled**: The total number of HTTP requests processed. +- **NGINX uptime**: The number of reloads and worker connections during uptime. +- **Usage report timestamps**: Start and end times for each usage report. +- **Kubernetes node details**: Information about Kubernetes nodes. + +### Security and privacy of reported data + +All communication between your NGINX Plus instances, NGINX Instance Manager, and F5’s licensing endpoint (`product.connect.nginx.com`) is protected using **SSL/TLS** encryption. + +Only **operational metrics** are reported — no **personally identifiable information (PII)** or **sensitive customer data** is transmitted. diff --git a/content/nic/installation/ingress-nginx.md b/content/nic/installation/ingress-nginx.md new file mode 100644 index 000000000..cdace0791 --- /dev/null +++ b/content/nic/installation/ingress-nginx.md @@ -0,0 +1,546 @@ +--- +docs: DOCS-1469 +doctypes: +- tutorial +tags: +- docs +title: Migrate from Ingress-NGINX Controller to NGINX Ingress Controller +toc: true +weight: 500 +--- + +This document describes how to migrate from the community-maintained Ingress-NGINX Controller to F5 NGINX Ingress Controller. + +## Overview + +This page explains two different ways to migrate from the community-maintained [Ingress-NGINX Controller](https://github.com/kubernetes/ingress-nginx) project to NGINX Ingress Controller: using NGINX's Ingress Resources or with Kubernetes's built-in Ingress Resources. This is typically because of implementation differences, and to take advantage of features such as [NGINX Plus integration]({{}}). + +The information in this guide is extracted from a free eBook called "_Kubernetes Ingress Controller Deployment and Security with NGINX_", which can be downloaded from the [NGINX Library](https://www.nginx.com/resources/library/kubernetes-ingress-controller-deployment-security-nginx/). + +## Before you begin + +To complete the instructions in this guide, you need the following: + +- A working knowledge of [Ingress Controllers]({{}}). +- An [NGINX Ingress Controller installation]({{}}) on the same host as an existing Ingress-NGINX Controller. + +There are two primary paths for migrating between the community Ingress-NGINX Controller to NGINX Ingress Controller: + +- Using NGINX Ingress Resources +- Using Kubernetes Ingress Resources. + +## Migration with NGINX Ingress resources +This path uses Kubernetes Ingress Resources to set root permissions, then NGINX Ingress Resources for configuration using custom resource definitions (CRDs): + +* [VirtualServer and VirtualServerRoute]({{}}) +* [TransportServer]({{}}) +* [GlobalConfiguration]({{}}) +* [Policy]({{}}) + +### Configuring SSL termination and HTTP path-based routing +The following two code examples correspond to a Kubernetes Ingress Resource and an [NGINX VirtualServer Resource]({{}}). Although the syntax and indentation is different, they accomplish the same basic Ingress functions, used for SSL termination and Layer 7 path-based routing. + +**Kubernetes Ingress Resource** +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: nginx-test +spec: + tls: + - hosts: + - foo.bar.com + secretName: tls-secret + rules: + - host: foo.bar.com + http: + paths: + - path: /login + backend: + serviceName: login-svc + servicePort: 80 + - path: /billing + serviceName: billing-svc + servicePort: 80 +``` + +**NGINX VirtualServer Resource** +```yaml +apiVersion: networking.k8s.io/v1 +kind: VirtualServer +metadata: + name: nginx-test +spec: + host: foo.bar.com + tls: + secret: tls-secret + upstreams: + - name: login + service: login-svc + port: 80 + - name: billing + service: billing-svc + port: 80 + routes: + - path: /login + action: + pass: login + - path: /billing + action: + pass: billing +``` + +### Configuring TCP/UDP load balancing and TLS passthrough +NGINX Ingress Controller exposes TCP and UDP services using [TransportServer]({{}}) and [GlobalConfiguration]({{}}) resources. These resources provide a broad range of options for TCP/UDP and TLS Passthrough load balancing. By contrast, the community Ingress-NGINX Controller exposes TCP/UDP services by using a Kubernetes ConfigMap object. + +--- + +### Convert Ingress-NGINX Controller annotations to NGINX Ingress resources +Kubernetes deployments often need to extend basic Ingress rules for advanced use cases such as canary and blue-green deployments, traffic throttling, and ingress-egress traffic manipulation. The community Ingress-NGINX Controller implements many of these using Kubernetes annotations with custom Lua extensions. + +These custom Lua extensions are intended for specific NGINX Ingress resource definitions and may not be as granular as required for advanced use cases. The following examples show how to convert these annotations into NGINX Ingress Controller Resources. + +--- + +#### Canary deployments +Canary and blue-green deployments allow you to push code changes to production environments without disrupting existing users. NGINX Ingress Controller runs them on the data plane: to migrate from the community Ingress-NGINX Controller, you must map the latter's annotations to [VirtualServer and VirtualServerRoute resources]({{}}). + +The Ingress-NGINX Controller evaluates canary annotations in the following order: + +1. _nginx.ingress.kubernetes.io/canary-by-header_ +1. _nginx.ingress.kubernetes.io/canary-by-cookie_ +1. _nginx.ingress.kubernetes.io/canary-by-weight_ + +For NGINX Ingress Controller to evalute them the same way, they must appear in the same order in the VirtualServer or VirtualServerRoute Manifest. + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/canary: "true" +nginx.ingress.kubernetes.io/canary-by-header: "httpHeader" +``` + +**NGINX Ingress Controller** +```yaml +matches: +- conditions: + - header: httpHeader + value: never + action: + pass: echo + - header: httpHeader + value: always + action: + pass: echo-canary +action: + pass: echo +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/canary: "true" +nginx.ingress.kubernetes.io/canary-by-header: "httpHeader" +nginx.ingress.kubernetes.io/canary-by-header-value: "my-value" +``` + +**NGINX Ingress Controller** +```yaml +matches: +- conditions: + - header: httpHeader + value: my-value + action: + pass: echo-canary +action: + pass: echo +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/canary: "true" +nginx.ingress.kubernetes.io/canary-by-cookie: "cookieName" +``` + +**NGINX Ingress Controller** +```yaml +matches: +- conditions: + - cookie: cookieName + value: never + action: + pass: echo + - cookie: cookieName + value: always + action: + pass: echo-canary +action: + pass: echo +``` + +--- + +#### Traffic control +Environments using microservices tend to use extensive traffic-control policies to manage ephemeral applications using circuit breaking and rate and connection limiting to prevent error conditions due to unhealthy states or abnormal behavior. + +The following examples map Ingress-NGINX Controller annotations to NGINX [VirtualServer and VirtualServerRoute resources]({{}}) for rate limiting, custom HTTP errors, custom default backend and URI rewriting. + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/custom-http-errors: "code" + +nginx.ingress.kubernetes.io/default-backend: "default-svc" +``` + +**NGINX Ingress Controller** +```yaml +errorPages: +- codes: [code] + redirect: + code: 301 + url: default-svc +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/limit-connections: "number" +``` + +**NGINX Ingress Controller** +```yaml +http-snippets: | + limit_conn_zone $binary_remote_addr zone=zone_name:size; +routes: +- path: /path + location-snippets: | + limit_conn zone_name number; +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/limit-rate: "number" +nginx.ingress.kubernetes.io/limit-rate-after: "number" +``` + +**NGINX Ingress Controller** +```yaml +location-snippets: | + limit_rate number; + + limit_rate_after number; +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/limit-rpm: "number" +nginx.ingress.kubernetes.io/limit-burst-multiplier: "multiplier" +``` + +**NGINX Ingress Controller** +```yaml +rateLimit: + rate: numberr/m + + burst: number * multiplier + key: ${binary_remote_addr} + zoneSize: size +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/limit-rps: "number" +nginx.ingress.kubernetes.io/limit-burst-multiplier: "multiplier" +``` + +**NGINX Ingress Controller** +```yaml +rateLimit: + rate: numberr/s + + burst: number * multiplier + key: ${binary_remote_addr} + zoneSize: size +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/limit-whitelist: "CIDR" +``` + +**NGINX Ingress Controller** +```yaml +http-snippets: | +server-snippets: | +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/rewrite-target: "URI" +``` + +**NGINX Ingress Controller** +```yaml +rewritePath: "URI" +``` + +There are four Ingress-NGINX Controller annotations without NGINX Ingress resource fields yet: they must be handled using snippets. + +- _nginx.ingress.kubernetes.io/limit-connections_ +- _nginx.ingress.kubernetes.io/limit-rate_ +- _nginx.ingress.kubernetes.io/limit-rate-after_ +- _nginx.ingress.kubernetes.io/limit-whitelist_ + +--- + +#### Header manipulation +Manipulating HTTP headers is useful in many cases, as they contain information that is important and relevant to systems involved in HTTP transactions. The community Ingress-NGINX Controller supports enabling and configuring cross-origin resource sharing (CORS) headings used by AJAX applications, where front-end Javascript code interacts with backend applications or web servers. + +These code blocks show how the Ingress-NGINX annotations correspond to NGINX Ingress Controller [VirtualServer and VirtualServerRoute resources]({{}}). + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/enable-cors: "true" +nginx.ingress.kubernetes.io/cors-allow-credentials: "true" + +nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For" + +nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS" + +nginx.ingress.kubernetes.io/cors-allow-origin: "*" + +nginx.ingress.kubernetes.io/cors-max-age: "seconds" +``` + +**NGINX Ingress Controller** +```yaml +responseHeaders: + add: + - name: Access-Control-Allow-Credentials + value: "true" + - name: Access-Control-Allow-Headers + value: "X-Forwarded-For" + - name: Access-Control-Allow-Methods + value: "PUT, GET, POST, OPTIONS" + - name: Access-Control-Allow-Origin + value: "*" + - name: Access-Control-Max-Age + value: "seconds" +``` + +--- + +#### Proxying and load balancing +NGINX Ingress Controller has multiple proxy and load balancing functionalities you may want to configure based on the use case, such as configuring the load balancing algorithm and the timeout and buffering settings for proxied connections. + +This table shows how Ingress-NGINX Controller annotations map to statements in the upstream field for [VirtualServer and VirtualServerRoute resources]({{}}), covering load balancing, proxy timeout, proxy buffering and connection routing for a services' ClusterIP address and port. + +{{< bootstrap-table "table table-bordered table-striped table-responsive" >}} +| Ingress-NGINX Controller | NGINX Ingress Controller | +| ------------------------ | ------------------------ | +| _nginx.ingress.kubernetes.io/load-balance_ | _lb-method_ | +| _nginx.ingress.kubernetes.io/proxy-buffering_ | _buffering_ | +| _nginx.ingress.kubernetes.io/proxy-buffers-number_ | _buffers_ | +| _nginx.ingress.kubernetes.io/proxy-buffer-size_ | _buffers_ | +| _nginx.ingress.kubernetes.io/proxy-connect-timeout_ | _connect-timeout_ | +| _nginx.ingress.kubernetes.io/proxy-next-upstream_ | _next-upstream_ | +| _nginx.ingress.kubernetes.io/proxy-next-upstream-timeout_ | _next-upstream-timeout_ | +| _nginx.ingress.kubernetes.io/proxy-read-timeout_ | _read-timeout_ | +| _nginx.ingress.kubernetes.io/proxy-send-timeout_ | _send-timeout_ | +| _nginx.ingress.kubernetes.io/service-upstream_ | _use-cluster-ip_ | +{{% /bootstrap-table %}} + +#### mTLS authentication + +mTLS authentication is a way of enforcing mutual authentication on traffic entering and exiting a cluster (north-sourth traffic). This secure form of communication is common within a service mesh, commonly used in strict zero-trust environments. + +NGINX Ingress Controller layer can handle mTLS authentication for end systems through the presentation of valid certificates for external connections. It accomplishes this through [Policy]({{}}) resources, which correspond to Ingress-NGINX Controller annotations for [client certificate authentication](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-certificate-authentication) and [backend certificate authentication](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-certificate-authentication). + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/auth-tls-secret: secretName +nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" +nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1" +``` + +**NGINX Ingress Controller** +```yaml +ingressMTLS: + clientCertSecret: secretName + verifyClient: "on" + + verifyDepth: 1 +``` + +--- + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/proxy-ssl-secret: "secretName" +nginx.ingress.kubernetes.io/proxy-ssl-verify: "on|off" +nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "1" +nginx.ingress.kubernetes.io/proxy-ssl-protocols: "TLSv1.2" +nginx.ingress.kubernetes.io/proxy-ssl-ciphers: "DEFAULT" +nginx.ingress.kubernetes.io/proxy-ssl-name: "server-name" +nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on|off" +``` + +**NGINX Ingress Controller** +```yaml +egressMTLS: + tlsSecret: secretName + + verifyServer: true|false + + verifyDepth: 1 + + protocols: TLSv1.2 + + ciphers: DEFAULT + + sslName: server-name + + serverName: true|false +``` + +--- + +#### Session persistence with NGINX Plus +With [NGINX Plus]({{}}), you can use [Policy]({{}}) resources for session persistence, which have corresponding annotations for the community Ingress-NGINX Controller. + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/affinity: "cookie" +nginx.ingress.kubernetes.io/session-cookie-name: "cookieName" +nginx.ingress.kubernetes.io/session-cookie-expires: "x" +nginx.ingress.kubernetes.io/session-cookie-path: "/route" +nginx.ingress.kubernetes.io/session-cookie-secure: "true" +``` + +**NGINX Ingress Controller** +```yaml +sessionCookie: + enable: true + + name: cookieName + + expires: xh + + path: /route + + secure: true +``` + +## Migration with Kubernetes Ingress resources +The other option for migrating from the community Ingress-NGINX Controller to NGINX Ingress Controller is using only [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) and [ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/) from standard Kubernetes resources, potentially relying on [mergeable Ingress types](https://github.com/nginx/kubernetes-ingress/tree/main/examples/ingress-resources/mergeable-ingress-types). + +This ensures that all configuration is kept in the Ingress object. + +{{< warning >}} +You should avoid altering the `spec` field of the Ingress resource when taking this option. Ingress-NGINX Controller and NGINX Ingress Controller differ slightly in their implementations: changing the Kubernetes Ingress can create incompatibility issues. +{{< /warning >}} + +### Advanced configuration with annotations +This table maps the Ingress-NGINX Controller annotations to NGINX Ingress Controller's equivalent annotations, and the respective NGINX Directive. + +{{< bootstrap-table "table table-bordered table-striped table-responsive" >}} +| Ingress-NGINX Controller | NGINX Ingress Controller | NGINX Directive | +| ------------------------ | ------------------------ | --------------- | +| [_nginx.ingress.kubernetes.io/configuration-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#configuration-snippet) | [_nginx.org/location-snippets_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#snippets-and-custom-templates) | N/A | +| [_nginx.ingress.kubernetes.io/load-balance_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-nginx-load-balancing) (1) | [_nginx.org/lb-method_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#backend-services-upstreams) | [_random two least_conn_](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#random) | +| [_nginx.ingress.kubernetes.io/proxy-buffering_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-buffering) | [_nginx.org/proxy-buffering_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) | [_proxy_buffering_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) | +| [_nginx.ingress.kubernetes.io/proxy-buffers-number_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-buffers-number) | [_nginx.org/proxy-buffers_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) | [_proxy_buffers_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) | +| [_nginx.ingress.kubernetes.io/proxy-buffer-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-buffer-size) | [_nginx.org/proxy-buffer-size_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) | [_proxy_buffer_size_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) | +| [_nginx.ingress.kubernetes.io/proxy-connect-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-connect-timeout_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) | [_proxy_connect_timeout_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) | +| [_nginx.ingress.kubernetes.io/proxy-read-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-read-timeout_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) | [_proxy_read_timeout_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) | +| [_nginx.ingress.kubernetes.io/proxy-send-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-send-timeout_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) | [_proxy_send_timeout_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) | +| [_nginx.ingress.kubernetes.io/rewrite-target_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rewrite) | [_nginx.org/rewrites_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#request-uriheader-manipulation) | [_rewrite_](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite) | +| [_nginx.ingress.kubernetes.io/server-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-snippet)| [_nginx.org/server-snippets_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#snippets-and-custom-templates) | N/A | +| [_nginx.ingress.kubernetes.io/ssl-redirect_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-side-https-enforcement-through-redirect) | [_ingress.kubernetes.io/ssl-redirect_](https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#auth-and-ssltls) | N/A (2) | +{{% /bootstrap-table %}} + +1. Ingress-NGINX Controller implements some of its load balancing algorithms with Lua, which may not have an equivalent in NGINX Ingress Controller. +1. To redirect HTTP (80) traffic to HTTPS (443), NGINX Ingress Controller uses built-in NGINX `if` conditions while Ingress-NGINX Controller uses Lua. + +The following two snippets outline Ingress-NGINX Controller annotations that correspond to annotations for NGINX Ingress Controller with NGINX Plus. + +**Ingress-NGINX Controller** +```yaml +nginx.ingress.kubernetes.io/affinity: "cookie" +nginx.ingress.kubernetes.io/session-cookie-name: "cookie_name" +nginx.ingress.kubernetes.io/session-cookie-expires: "seconds" +nginx.ingress.kubernetes.io/session-cookie-path: "/route" +``` + +**NGINX Ingress Controller (with NGINX Plus)** +```yaml +nginx.com/sticky-cookie-services: "serviceName=example-svc cookie_name expires=time path=/route" +``` + +{{< note >}} +NGINX Ingress Controller has additional annotations for features using NGINX Plus that have no Ingress-NGINX Controller equivalent, such as active health checks and authentication using JSON Web Tokens (JWTs). +{{< /note >}} + +### Global configuration with ConfigMaps + +This table maps the Ingress-NGINX Controller ConfigMap keys to NGINX Ingress Controller's equivalent ConfigMap keys. + + + +{{< bootstrap-table "table table-bordered table-striped table-responsive" >}} +| Ingress-NGINX Controller | NGINX Ingress Controller | +| ------------------------ | ------------------------ | +| [_disable-access-log_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#disable-access-log) | [_access-log-off_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#logging) | +| [_error-log-level_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#error-log-level) | [_error-log-level_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#logging) | +| [_hsts_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#hsts) | [_hsts_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#auth-and-ssltls) | +| [_hsts-include-subdomains_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#hsts-include-subdomains) | [_hsts-include-subdomains_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#auth-and-ssltls) | +| [_hsts-max-age_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#hsts-max-age) | [_hsts-max-age_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#auth-and-ssltls) | +| [_http-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#http-snippet) | [_http-snippets_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#snippets-and-custom-templates) | +| [_keep-alive_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#keep-alive) | [_keepalive-timeout_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_keep-alive-requests_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#keep-alive-requests) | [_keepalive-requests_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_load-balance_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#load-balance) | [_lb-method_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#backend-services-upstreams) | +| [_location-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#location-snippet) | [_location-snippets_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#snippets-and-custom-templates) | +| [_log-format-escape-json_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#log-format-escape-json) | [_log-format-escaping: "json"_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#logging) | +| [_log-format-stream_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#log-format-stream) | [_stream-log-format_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#logging) | +| [_log-format-upstream_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#log-format-upstream) | [_log-format_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#logging) | +| [_main-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#main-snippet) | [_main-snippets_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#snippets-and-custom-templates) | +| [_max-worker-connections_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#max-worker-connections) | [_worker-connections_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_max-worker-open-files_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#max-worker-open-files) | [_worker-rlimit-nofile_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-body-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-body-size) | [_client-max-body-size_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-buffering_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-buffering) | [_proxy-buffering_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-buffers-number_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-buffers-number) | [_proxy-buffers: number size_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-buffer-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-buffer-size) | [_proxy-buffers: number size_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-connect-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-connect-timeout) | [_proxy-connect-timeout_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-read-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-read-timeout) | [_proxy-read-timeout_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-send-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-send-timeout) | [_proxy-send-timeout_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_server-name-hash-bucket-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#server-name-hash-bucket-size) | [_server-names-hash-bucket-size_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_proxy-headers-hash-max-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-headers-hash-max-size) | [_server-names-hash-max-size_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_server-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#server-snippet) | [_server-snippets_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#snippets-and-custom-templates) | +| [_server-tokens _](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#server-tokens) | [_server-tokens_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_ssl-ciphers_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-ciphers) | [_ssl-ciphers_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#auth-and-ssltls) | +| [_ssl-dh-param_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-dh-param) | [_ssl-dhparam-file_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#auth-and-ssltls) | +| [_ssl-protocols_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-protocols) | [_ssl-protocols_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#auth-and-ssltls) | +| [_ssl-redirect_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-redirect) | [_ssl-redirect_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#auth-and-ssltls) | +| [_upstream-keepalive-connections_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#upstream-keepalive-connections) | [_keepalive_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#backend-services-upstreams) | +| [_use-http2_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-http2) | [_http2_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#listeners) | +| [_use-proxy-protocol_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol) | [_proxy-protocol_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#listeners) | +| [_variables-hash-bucket-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#variables-hash-bucket-size) | [_variables-hash-bucket-size_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_worker-cpu-affinity_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#worker-cpu-affinity) | [_worker-cpu-affinity_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_worker-processes_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#worker-processes) | [_worker-processes_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +| [_worker-shutdown-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#worker-shutdown-timeout) | [_worker-shutdown-timeole_](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#general-customization) | +{{% /bootstrap-table %}} diff --git a/content/nic/installation/installing-nic/_index.md b/content/nic/installation/installing-nic/_index.md new file mode 100644 index 000000000..ad70f8f42 --- /dev/null +++ b/content/nic/installation/installing-nic/_index.md @@ -0,0 +1,5 @@ +--- +title: Install NGINX Ingress Controller +description: +weight: 100 +--- diff --git a/content/nic/installation/installing-nic/deploy-with-nap-using-helm.md b/content/nic/installation/installing-nic/deploy-with-nap-using-helm.md new file mode 100644 index 000000000..93cf807fd --- /dev/null +++ b/content/nic/installation/installing-nic/deploy-with-nap-using-helm.md @@ -0,0 +1,343 @@ +--- +title: Install NGINX Ingress Controller and NGINX App Protect WAF with Docker and Helm +toc: true +weight: 500 +type: how-to +product: NIC +--- + +This document describes how to build a local F5 NGINX App Protect WAF v5 Docker image with NGINX Plus Ingress +Controller, which can be used to compile WAF policies. + +This is accomplished with the following steps: + +- Prepare license secrets to enable a Kubernetes deployment +- Use a NGINX App Protect WAF Docker image to transform a policy JSON file into a compiled bundle +- Configure PersistentVolumes so the deployed NGINX App Protect WAF instance can access the compiled bundle +- Deploy NGINX Plus Ingress Controller with NGINX App Protect +- Test example services to validate that the WAF policies work + +--- + +## Prepare Secrets and credentials + +1. Download your NGINX Ingress Controller subscription's JSON Web Token, SSL Certificate, and Private Key from MyF5. + You can use the same JSON Web Token, Certificate, and Key as NGINX Plus in your MyF5 portal. +1. Rename the files to the following: + - `nginx-repo.crt` + - `nginx-repo.key` + - `nginx-repo.jwt` +1. Log in to the Docker registry using the contents of the JSON Web Token file: + ```shell + docker login private-registry.nginx.com --username=$(cat nginx-repo.jwt) --password=none + ``` + +--- + +## Compile WAF Policy from JSON to Bundle + +Pull the `waf-compiler` image with: + +```shell +docker pull private-registry.nginx.com/nap/waf-compiler:5.6.0 +``` + +Download the [provided WAF Policy JSON](https://raw.githubusercontent.com/nginx/kubernetes-ingress/main/tests/data/ap-waf-v5/wafv5.json): + +```shell +curl -L https://raw.githubusercontent.com/nginx/kubernetes-ingress/main/tests/data/ap-waf-v5/wafv5.json -o /tmp/wafv5.json +``` + +Use your pulled NAP Docker image (`private-registry.nginx.com/nap/waf-compiler:5.6.0`) to compile the policy bundle: + +```shell +# Using your newly created image +docker run --rm \ + -v /tmp:/tmp \ + private-registry.nginx.com/nap/waf-compiler:5.6.0 \ + -p /tmp/wafv5.json \ + -o /tmp/compiled_policy.tgz +``` + +Move the downloaded JSON and compiled policy to your workspace: + +```shell +mv /tmp/wafv5.json $(pwd)/wafv5.json +mv /tmp/compiled_policy.tgz $(pwd)/compiled_policy.tgz +``` + +After this command, your workspace should contain: + +``` +├── nginx-repo.crt +├── nginx-repo.key +├── nginx-repo.jwt +├── wafv5.json +└── compiled_policy.tgz +``` + +--- + +## Create the persistent volume and claim to store the policy bundle + +Save the following configuration data as `pvc.yaml` in the same directory. + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: task-pv-volume + labels: + type: local +spec: + storageClassName: manual + capacity: + storage: 1Gi + accessModes: + - ReadWriteOnce + hostPath: + path: "/tmp/" + +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: task-pv-claim +spec: + storageClassName: manual + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +``` + +This sets up a 1Gi disk and attaches a claim to it that you will reference in the deployment chart. + +Create these with: + +```shell +kubectl apply -f pvc.yaml +``` + +Verify that the persistent volume and claim are created: + +```shell +# For the persistent volume +kubectl get pv + +# For the persistent volume claim +kubectl get pvc +``` + +## Deploy NGINX Plus NIC Controller with NAP Enabled using Helm + +Add the official NGINX Helm repository: + +```shell +helm repo add nginx-stable https://helm.nginx.com/stable +helm repo update +``` + +Create Kubernetes Docker and licensing secrets: +```shell +kubectl create secret \ + docker-registry regcred \ + --docker-server=private-registry.nginx.com \ + --docker-username=$(cat nginx-repo.jwt) \ + --docker-password=none + +kubectl create secret \ + generic license-token \ + --from-file=license.jwt=./nginx-repo.jwt \ + --type=nginx.com/license +``` + +Install the required CRDs for NGINX Ingress Controller: + +```shell +kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v5.0.0/deploy/crds.yaml +``` + +Using Helm, install NGINX Ingress Controller + +```shell +helm upgrade --install nic nginx-stable/nginx-ingress \ + --set controller.image.repository="private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress" \ + --set controller.image.tag="5.0.0-alpine-fips" \ + --set controller.nginxplus=true \ + --set controller.appprotect.enable=true \ + --set controller.appprotect.v5=true \ + --set-json 'controller.appprotect.volumes=[ + {"name":"app-protect-bd-config","emptyDir":{}}, + {"name":"app-protect-config","emptyDir":{}}, + {"name":"app-protect-bundles","persistentVolumeClaim":{"claimName":"task-pv-claim"}} + ]' \ + --set controller.serviceAccount.imagePullSecretName=regcred \ + --set 'controller.volumeMounts[0].name=app-protect-bundles' \ + --set 'controller.volumeMounts[0].mountPath=/etc/app_protect/bundles/' + +``` + +Verify deployment success: + +```shell +kubectl get pods +``` + +--- + +## Copy the policy bundle into the running instance + +Get the name of the pod from the `kubectl get pods` command above. + +Copy the file into the `nginx-ingress` container within the pod: + +```shell +kubectl cp ./compiled_policy.tgz \ + :/etc/app_protect/bundles/compiled_policy.tgz \ + -c nginx-ingress +``` + +Replace `` with the actual name of the pod, for example: + +```shell +kubectl cp ./compiled_policy.tgz \ + nic-nginx-ingress-controller-9bd89589d-j925h:/etc/app_protect/bundles/compiled_policy.tgz \ + -c nginx-ingress +``` + +Confirm that the policy file is in the pod. The following command should list `compiled_policy.tgz`. + +```shell +kubectl exec --stdin --tty \ + -c nginx-ingress \ + \ + -- ls -la /etc/app_protect/bundles +``` + +## Confirm that the WAF policies work + +Save the following kubernetes config file as `webapp.yaml`: + +```yaml +apiVersion: k8s.nginx.org/v1 +kind: VirtualServer +metadata: + name: webapp +spec: + host: webapp.example.com + policies: + - name: waf-policy + upstreams: + - name: webapp + service: webapp-svc + port: 80 + routes: + - path: / + action: + pass: webapp +--- +apiVersion: k8s.nginx.org/v1 +kind: Policy +metadata: + name: waf-policy +spec: + waf: + enable: true + apBundle: "compiled_policy.tgz" +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: webapp +spec: + replicas: 1 + selector: + matchLabels: + app: webapp + template: + metadata: + labels: + app: webapp + spec: + containers: + - name: webapp + image: nginxdemos/nginx-hello:plain-text + ports: + - containerPort: 8080 +--- +apiVersion: v1 +kind: Service +metadata: + name: webapp-svc +spec: + ports: + - port: 80 + targetPort: 8080 + protocol: TCP + name: http + selector: + app: webapp +``` + +Create the services with +```shell +kubectl apply -f webapp.yaml +``` + +Confirm that the services have started with + +```shell +kubectl get pods +``` + +### Save the public IP and PORT in environment variables + +Get the public IP and port of your instance with the following command: + +```shell +kubectl get svc +``` + +Save them in the following environment variables: + +```shell +IC_IP=XXX.YYY.ZZZ.III +IC_HTTP_PORT= +``` + +### Validate that the WAF works + +Send a valid request to the deployed application: + +```shell +curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT/ +``` + +```text +Server address: 10.92.2.13:8080 +Server name: webapp-7b7dfbff54-dtxzt +Date: 18/Apr/2025:19:39:18 +0000 +URI: / +Request ID: 4f378a01fb8a36ae27e2c3059d264527 +``` + +And send one that should be rejected + +```shell +curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP "http://webapp.example.com:$IC_HTTP_PORT/