From a646098bd3cf3c063b051c16b41c5ffccc603fed Mon Sep 17 00:00:00 2001 From: Rowena Date: Fri, 7 Mar 2025 17:50:25 +0100 Subject: [PATCH 1/8] feat(lb): continue troubleshooting --- faq/loadbalancer.mdx | 16 +- .../troubleshooting/http-errors.mdx | 139 ++++++++++++++++++ .../troubleshooting/k8s-errors.mdx | 72 +++++++++ .../troubleshooting/random-problems.mdx | 135 +++++++++++++++++ 4 files changed, 361 insertions(+), 1 deletion(-) create mode 100644 pages/load-balancer/troubleshooting/http-errors.mdx create mode 100644 pages/load-balancer/troubleshooting/k8s-errors.mdx create mode 100644 pages/load-balancer/troubleshooting/random-problems.mdx diff --git a/faq/loadbalancer.mdx b/faq/loadbalancer.mdx index 084cb38e57..c48dcabeed 100644 --- a/faq/loadbalancer.mdx +++ b/faq/loadbalancer.mdx @@ -27,6 +27,8 @@ Yes: Check out our documentation on: By default, each [public](/load-balancer/concepts/#accessibility) Load Balancer is created automatically with a flexible IPv4 address. This is a public IP that can be held in your account even after you delete your Load Balancer. You can optionally also add an IPv6 address. +**This IP address is fixed and does not risk changing once attached to your Load Balancer**. + Your frontend listens to your Load Balancer's public flexible IP address. In case of a failure of the Load Balancer, a replica Load Balancer is immediately spawned and deployed, and the IP address is automatically rerouted to this replica. This is done automatically, by the Load Balancer control subsystems. When you delete a Load Balancer, you can choose to keep its flexible IP(s) in your account, to reuse later with a new Load Balancer. These flexible IPs are not compatible with other Scaleway products (e.g. Instances, Elastic Metal servers, Public Gateways): each resource has its own set of flexible IPs. @@ -35,6 +37,12 @@ When you delete a Load Balancer, you can choose to keep its flexible IP(s) in yo Each Load Balancer can have one public IPv4 address and one public IPv6 address. Currently, it is not possible to assign more than one of each type of IP to a given Load Balancer. +## How can I move my Instance's flexible IP address to my Load Balancer? + +This is not possible: flexible IPs are scoped to the resource-type that they were created for. You can move a flexible IP between different Instances, but not move it to a Load Balancer. + +Watch this space for resource-agnostic flexible IPs in the future. + ## Do Load Balancers support external IPv6 connections? Yes, Load Balancer supports both IPv4 and IPv6 addresses at the frontend. IPv6 can also be used to communicate between the Load Balancer and your backend servers. @@ -60,6 +68,8 @@ To take advantage of multi-cloud, you must choose a compatible Load Balancer off All protocols based on `TCP` are supported. This includes `database`, `HTTP`, `LDAP`, `IMAP` and so on. You can also specify `HTTP` to benefit from support and features that are exclusive to this protocol. +Scaleway Load Balancer does not currently support `UDP`. + ## Is it possible to add security to restrict access to a URL or port on the Load Balancer? Yes, you can restrict the use of a `TCP` port or `HTTP` URL using ACLs. Find more information in our [ACL documentation](/load-balancer/how-to/create-manage-acls/). @@ -78,4 +88,8 @@ A health check is one of the core concepts for a well-functioning Load Balancer. ## Can I set up a caching service for my load balanced application? -Yes, this is possible with Scaleway's [Edge Services](/edge-services/) product, currently in Public Beta. By creating an Edge Services pipeline for your Load Balancer, you can access Edge Services caching service reduce load on your origin. \ No newline at end of file +Yes, this is possible with Scaleway's [Edge Services](/edge-services/) product, currently in Public Beta. By creating an Edge Services pipeline for your Load Balancer, you can access Edge Services caching service reduce load on your origin. + +## How can I add extra security such as a firewall or anti-DDOS to my Load Balancer? + +This will be available soon via [Edge Services](/edge-services/), watch this space. \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/http-errors.mdx b/pages/load-balancer/troubleshooting/http-errors.mdx new file mode 100644 index 0000000000..bf16b8af8d --- /dev/null +++ b/pages/load-balancer/troubleshooting/http-errors.mdx @@ -0,0 +1,139 @@ +--- +meta: + title: I am experiencing HTTP errors with my Load Balancer + description: Troubleshoot HTTP errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get your application back up and running. +content: + h1: I am experiencing HTTP errors with my Load Balancer + paragraph: Troubleshoot HTTP errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get your application back up and running. +tags: load-balancer http-errors bad-request +dates: + validation: 2025-03-06 + posted: 2025-03-06 +categories: + - network +--- + +You may experience HTTP errors when attempting to connect to an application served via your Load Balancer. + +This page helps you find solutions to some of these most common errors. + +### I'm experiencing a 400 Bad Request error when accessing my application through my Load Balancer + +Check if [Proxy Protocol](load-balancer/reference-content/configuring-backends/#proxy-protocol) is enabled on your Load Balancer. If your backend server is not configured to handle Proxy Protocol headers correctly, it may reject the requests. + +Try [disabling Proxy Protocol](/load-balancer/how-to/manage-frontends-and-backends/#how-to-edit-backends-and-health-checks) on your Load Balancer to see if it resolves the issue. + +If the issue is resolved when Proxy Protocol is disabled, [ensure your backend server is correctly configured for Proxy Protocol](/tutorials/proxy-protocol-v2-load-balancer/) before re-enabling. + +--- + +### I'm experiencing a 503 Service Unavailable error when trying to access my application through my Load Balancer (OR GENERALLY UNABLE TO CONNECT TO APP) + +- Check the health of backend servers +- Check Load Balancer is not overloaded and exceeding its bandwidth (Cockpit) +- Port and protocol configuration (check Load Balancer is forwarding to the right backend port and protocol) +- Check backend application logs for errors like crashes, timeouts or rate limiting +- Scaling limits: ensure the backend has enough instances/resources to handle incoming requests + +## I'm experiencing SSL protocol or secure connection errors + +ERR_SSL_PROTOCOL_ERROR" (Chrome) +"SSL_ERROR_PROTOCOL_VERSION_ALERT" (Firefox) +"This site can't provide a secure connection" (Edge) +OpenSSL/3.0.14: error:0A00010B:SSL routines::wrong version number + +Ensure that you have correctly configured a certificate (link to doc) +Link to SSL offloading/passthrough etc. +Don't confused proxy protocol and SSL passthrough + +## I'm getting HTTP 404: IP not owned by Scaleway + +About how you need to use SCW backend servers, or else Multicloud + +HTTP Error 400: The Port 80 Frontend Must be Associated to an HTTP Backend + +If you're experiencing issues with your load balancer and receiving the error "HTTP error 400: The port 80 frontend must be associated to an HTTP backend" when trying to obtain an SSL certificate, this troubleshooting guide may help. Common symptoms include errors when trying to access the load balancer or when attempting to obtain an SSL certificate. + +Suggested Solutions: + + Verify that your backend is configured to use the HTTP protocol. Ensure that the backend is set up to accept HTTP connections from the frontend. + Check the frontend configuration: When creating or editing your frontend, make sure it is linked to the correct backend that is configured for HTTP. + Create or update the backend: If the backend is not already configured for HTTP, modify or create a new one. When creating a new backend, ensure that the protocol is set to HTTP. + Verify the ports: Ensure that the frontend is configured to listen on port 80 and that the backend is configured to use the correct port for your service. + Consult the load balancer's documentation for specific instructions on configuring frontends and backends, as well as obtaining SSL certificates. + + +## I'm experiencing a 413 Request Entity Too Large error when sending upload requests through my Load Balancer + +If you have a Kubernetes Load Balancer with an Nginx Ingress Controller, you may need to edit the `proxy-body-size`. Replace the value `50m` with teh correct desired value for your use-case. + +``` +kubectl edit cm ingress-nginx-controller -n ingress-nginx +proxy-body-size: "50m" +``` + +## I'm experiencing a 503 Service Unavailable error when accessing my application through my Load Balancer, even though the backend server passes health checks + +This issue can occur for several reasons, including: + +- **Application does not accept traffic on the expected domain or path**: Health checks generally target a specific endpoint e.g. `/health`, but your application may be rejecting requests on other paths. Test direct access to your backend using the expected request path e.g. with a curl. If your application expects a specific `Host` header, ensure the Load Balancer is configured to + +Issue +You receive a 503 Service Unavailable error when accessing your application through the Load Balancer, even though the backend server passes health checks. + +Possible Causes and Solutions +1. Application does not accept traffic on the expected domain or path +Health checks usually target a specific endpoint (e.g., /health), but your application may reject requests to other paths. +If the application enforces host-based routing, requests from the Load Balancer may not be handled correctly. +✅ Solution: + +Test direct access to your backend using the expected request path: +sh +curl -I http://:/ + +If this request returns 503, check application logs to see why it is rejecting real requests. +If your application expects a specific Host header, configure the Load Balancer to send it: + +sh +curl -I -H "Host: yourapp.example.com" http://: + +2. Rate limiting or connection exhaustion +The backend may have rate limits, thread limits, or connection pool limits that allow health check requests but block real client traffic. +Health checks are typically lightweight, so they may not trigger these limits. +✅ Solution: + +Check backend logs for rate-limiting errors (e.g., 429 Too Many Requests or resource exhaustion warnings). +Increase rate limits or connection limits in your application. +If using a database, ensure it allows enough connections to handle real traffic. +3. Protocol mismatch between Load Balancer and backend server +If the Load Balancer sends HTTP requests but the backend expects HTTPS, it may reject requests while still responding to health checks. +Similarly, if the backend uses HTTP/2 or WebSockets, ensure the Load Balancer supports it. +✅ Solution: + +Verify the protocol used by the Load Balancer. If the backend requires HTTPS, update the Load Balancer to send HTTPS requests. +If the backend requires WebSockets or HTTP/2, confirm that the Load Balancer is configured to allow them. +4. Backend server returns a 503 for real requests +The health check endpoint may be returning a cached success response while real requests trigger an application failure. +✅ Solution: + +Compare health check behavior with real requests: +sh +Copy +Edit +curl -I http://:/ +If health checks succeed but real requests fail, investigate backend logs for service errors. +Check for dependency failures (e.g., database timeouts or API failures). +5. Load Balancer is overloaded or misconfigured +The Load Balancer may run out of available connections or use a backend selection policy that allows a server to pass health checks but not receive traffic. +If connection draining is enabled, some backend instances may be marked as "healthy" but not receiving traffic. +✅ Solution: + +Check if your Load Balancer logs show failed request routing or connection timeouts. +Ensure that instances in the backend pool are not in a draining state. +If sticky sessions are enabled, verify that they are routing clients correctly. +Next Steps +Enable debug logging on the backend to capture rejected requests. +Test with a different backend instance to rule out instance-specific issues. +Check Load Balancer logs for patterns of failed requests. + +---- \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/k8s-errors.mdx b/pages/load-balancer/troubleshooting/k8s-errors.mdx new file mode 100644 index 0000000000..fa9db69a2c --- /dev/null +++ b/pages/load-balancer/troubleshooting/k8s-errors.mdx @@ -0,0 +1,72 @@ +--- +meta: + title: I am experiencing problems with my Kubernetes Load Balancer + description: Troubleshoot problems with your Kubernetes Load Balancer +content: + h1: I am experiencing problems with my Kubernetes Load Balancer + paragraph: Troubleshoot problems with your Kubernetes Load Balancer +tags: load-balancer kubernetes annotations +dates: + validation: 2025-03-06 + posted: 2025-03-06 +categories: + - network +--- + +You may experience errors when attempting to configure your Kubernetes Load Balancer. + +This page helps you find solutions to some of the most common problems. + +### I can't add a certificate to my Kubernetes Load Balancer + +Do it via annotations, not the console + +If you're also using Terraform, then: +- For SSL certificate management via terraform, see +https://registry.terraform.io/providers/scaleway/scaleway/latest/docs/resources/lb_certificate +And in your "ingress_nginx" manifest ensure you have a cert-manager that you'll need to deploy via terraform. +https://registry.terraform.io/modules/terraform-iaac/cert-manager/kubernetes/latest + +## I'm experiencing connectivity issues with my Kubernetes Load Balancer + +Symptoms: + +- Load balancer not connecting to nodes in Kapsule cluster. +- Application inaccessible from internet. +- Health checks failing for some nodes. + +Troubleshooting Steps: + +- Ensure that you provisioned and configured your Load Balancer via Kubernetes nad not via the Scaleway console, which provokes unexpected behaviours and errors, including certificate addition +- Check `externalTrafficPolicy` setting. If it is set to "Local" instead of "Cluster", this could be causing the issue. Change the policy to "Cluster". +- Try enabling or disabling Cloudflare's Proxy Mode, which may be affecting connectivity +- Verify that the required service is running on all nodes. If it is missing from some nodes, this could be causing health checks to fail. + +## I'm using Kubernetes cluster with Traefik2 proxy, when I check the logs of Traefik DaemonSet the load balancer internal IP appears instead of real client IP. + +the engineers have told us that you need to use the proxy protocol annotation to retrieve the real source IP. + +The doc for the implem is here: https://www.scaleway.com/en/docs/tutorials/proxy-protocol-v2-load-balancer/ + +## My Load Balancer is not connecting to my Kapsule cluster nodes, resulting in my application being inaccessible from the internet. + +Potential Causes and Solutions: + +- Incorrect externalTrafficPolicy setting : Check if the externalTrafficPolicy setting in your Kubernetes configuration is set to "Local". If so, try changing it to "Cluster" to allow all nodes to expose the service and route traffic to the correct node. +- Cloudflare proxy mode enabled : If you're using Cloudflare as your DNS, CDN, WAF, etc., try disabling the proxy mode to ensure that the load balancer receives traffic from your IP address instead of Cloudflare's IP addresses. +- Nodes not running pods : Verify that all nodes in your Kapsule cluster are running pods associated with the service. If not, ensure that the pods are properly deployed and running on all nodes. +- Load balancer configuration issues : Check the load balancer configuration to ensure that it's properly set up to route traffic to all nodes in the cluster. + +Additional Troubleshooting Steps: + +- Check the load balancer logs for any errors or warnings that may indicate the cause of the issue. +- Verify that the Kapsule cluster nodes are properly configured and running. +- Test the application by accessing it from a different location or network to rule out any issues with your local network or internet connection. + +## My k8s Load Balancer is not behaving as expected + +Do not attempt to create a Load Balancer manually and then add your ingress controller or Kubernetes nodes as backend server. + +This results in unexpected behaviours, HTTP errors and incorrect traffic handling. + +You must provision your Load Balancer via Kubernetes itself: see our [dedicated documentation](TODO). \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/random-problems.mdx b/pages/load-balancer/troubleshooting/random-problems.mdx new file mode 100644 index 0000000000..51d8160d27 --- /dev/null +++ b/pages/load-balancer/troubleshooting/random-problems.mdx @@ -0,0 +1,135 @@ +--- +meta: + title: I am experiencing RANDOM errors with my Load Balancer + description: Troubleshoot RANDOM errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get everything back up and running. +content: + h1: I am experiencing RANDOM errors with my Load Balancer + paragraph: Troubleshoot RANDOM errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get everything back up and running. +tags: load-balancer TODO +dates: + validation: 2025-03-06 + posted: 2025-03-06 +categories: + - network +--- + +## BACKEND SERVERS + +## When I try to add my backend server's IP address to the Load Balancer, I get an error + +CHECK THE ERROR +You have to have multicloud or be using Scaleway resources + +## When I want to add my instance's private IP to the balancer's backend configuration, it tells me that this IP doesn't exist. + +This error occurs because your IPs range is not in the private blocks IPs as describe in RFC1918. + +https://en.wikipedia.org/wiki/Private_network + +Scaleway allow only our public IPs or private IP that match the RFC1918 for the LB backend service. + +## How do I attach my Intsances to my LB via their private IPs via Terraform? + +https://admin.internal.scaleway.com/tickets?modal-id=ticket-details&ticket-id=1464043 + +## My Load Balancer can't communicate with Elastic Metal backend servers over a Private Network + +Ensure that the Elastic Metal servers have been correctly configured for the Privat Network. Note that additional steps are required. + + +## CERTIFICATE + +## I'm experiencing issues with my wildcard certificate not working and receiving HTTPS errors when trying to use a custom certificate, + +Common symptoms include certificates not being recognized or validated, or errors when trying to access subdomains. + +Suggested Solutions: + +- Verify that your wildcard certificate covers the correct domain and subdomains. Check the certificate's subject alternative name (SAN) or common name (CN) to ensure it matches the domain you're trying to secure. +- Ensure that your certificate is properly configured and installed on your load balancer. Check the load balancer's documentation for specific instructions on uploading and configuring certificates. +- If you're using a subdomain, make sure your certificate covers the correct subdomain. For example, if you're trying to secure 'subdomain.example.com', your certificate should cover '*.example.com'. +- Check the certificate's validity dates and ensure it's not expired or not yet valid. +- Consult the load balancer's documentation and the certificate issuer's documentation for specific troubleshooting steps and requirements. + +# I'm experiencing DNS errors when creating an SSL certificate + +You may be experiencing error messages such as + +``` +invalid argument(s): dns_name does not respect constraint / does not resolve to your Load Balancer IP +``` + +- Verify that the DNS entry is correctly set up and points to the Load Balancer IP address. +- After updating DNS entries, Wait for 30-60 minutes after updating the DNS entry to allow for propagation delay. +- Try generating the SSL certificate again after the propagation delay has passed. +- If the error persists, check the DNS entry using a tool like dig to ensure it is resolving correctly. + +## I am experiencing errors creating Let's Encrypt SSL certificate with Load Balancer + +You may be receiving "HTTP error 400: The port 80 frontend must be associated to an HTTP backend" error. + +- Verify that the backend is configured to use the HTTP protocol. +- Ensure that the frontend is correctly associated with the HTTP backend. +- Check the frontend configuration to ensure it is linked to the correct HTTP backend. +- If the backend is not already configured for HTTP, update or create a new backend with the HTTP protocol specified. +- Verify that the frontend is listening on port 80 and the backend is configured for the correct port corresponding to your service. +- Review the Load Balancer documentation on configuring frontends for additional guidance: https://www.scaleway.com/en/docs/load-balancer/reference-content/configuring-frontends/ + +## I'm experiencing issues with SSL certificates not being resolved when accessing my Kubernetes load balancers from within the cluster + +Key Points to Consider: + +- When accessing load balancers from within the cluster, Kubernetes may skip the load balancer and forward traffic in L4 to the application, causing SSL issues. +- To resolve this issue, you need to set up specific annotations on your load balancer service to instruct Kubernetes to use the hostname instead of the IP address. + + +Suggested Solution: + +- Add the following annotation to your load balancer service: `service.beta.kubernetes.io/scw-loadbalancer-use-hostname: "true"` +- This annotation will instruct Kubernetes to use the hostname of the load balancer instead of the IP address, allowing the SSL certificates to be resolved correctly. + +Refer to the Scaleway documentation for more information on load balancer annotations: https://github.com/scaleway/scaleway-cloud-controller-manager/blob/master/docs/loadbalancer-annotations.md#servicebetakubernetesioscw-loadbalancer-use-hostname + Ensure that your Kubernetes dependencies are up-to-date and that you have the latest ca-certificates installed. + Verify that your container image is correctly configured and that the SSL certificates are properly managed within the cluster or on the Scaleway side. + + +## I am experiencing issues with my load balancer that has MySQL backend servers, where connections are failing and SSL certificate generation is unsuccessful. + +- Disable Proxy Protocol : Check if the proxy protocol is enabled in your backend. If so, disable it, as it may be incompatible with your database setup. +- Verify Database Configuration : Double-check your database configuration, including the host, port, username, and password, to ensure they are correct and match the load balancer settings. +- Check Load Balancer Routing : Verify that the load balancer is correctly routing requests to your database instances. + + +Issue 2: SSL Certificate Generation + +If this is the case, you can try explicitly adding an entry for letsencrypt of the type @ IN CAA 0 issue ‘letsencrypt.org’, as shown here = https://letsencrypt.org/docs/caa/ + +- Check CAA Records : Your domain's CAA (Certificate Authority Authorization) records may be blocking the generation of the SSL certificate. If necessary, add a CAA record for Let's Encrypt with the format @ IN CAA 0 issue 'letsencrypt.org'. +- Verify Domain Configuration : Ensure that your domain is correctly configured to point to the load balancer. + + +## COCKPIT + +## I can't see any Layer 7 metrics in my Grafana dashboard + +This is normal if your Load Balancer is in TCP mode. + +## I can't see any Load Balancer logs in Grafana + +Is this normal??? + + + +## CONFIG + +## My Load Balancer's private IP address (172.16.8.2) is appearing in the backend application's logs, instead of the real client IP address. + +Use the proxy protocol annotation to retrieve the real source IP. The Scaleway support team provided documentation on how to implement this: https://www.scaleway.com/en/docs/tutorials/proxy-protocol-v2-load-balancer/ . This allowed the client to correctly log the real client IP address in the Traefik logs. + +## Security rules not being applied as expected, and I am having difficulties in filtering incoming traffic through my LB + +Key Points to Consider: + +- Security groups only apply to public IPs, not private IPs. If your instance behind a Load Balancer only has a private IP, the security group rules will not be applied. +- Security group rules will still apply to Instances serving a Load Balancer as long as they are attached by a public IP not a private one. Verify that your security group rules are correctly configured and propagated to the instances. Check that the rules correspond to the required ports, protocols, and IP addresses. Ensure that the instance is associated with the correct security group and that there are no conflicting rules blocking the traffic. +- To filter incoming traffic to your backend servers as it passes through a load balancer, you should apply security rules directly to the load balancer. Load balancers do not perform specific filtering; they redirect all authorized traffic to the backend servers, unless ACLs are configured. Consult the Scaleway documentation for more information on security groups and load balancer ACLs: https://www.scaleway.com/en/docs/instances/concepts/#security-group and https://www.scaleway.com/en/docs/load-balancer/how-to/create-manage-acls/ \ No newline at end of file From e7d8d52986c16172bb1c3e2f78f645bade1e4f8f Mon Sep 17 00:00:00 2001 From: Rowena Date: Mon, 10 Mar 2025 18:03:32 +0100 Subject: [PATCH 2/8] feat(lb): add more troubleshooting --- .../troubleshooting/certificates.mdx | 92 ++++++++++++ .../troubleshooting/configuration.mdx | 98 ++++++++++++ .../http-connection-errors.mdx | 75 ++++++++++ .../troubleshooting/http-errors.mdx | 139 ------------------ .../troubleshooting/k8s-errors.mdx | 24 ++- .../troubleshooting/random-problems.mdx | 135 ----------------- 6 files changed, 288 insertions(+), 275 deletions(-) create mode 100644 pages/load-balancer/troubleshooting/certificates.mdx create mode 100644 pages/load-balancer/troubleshooting/configuration.mdx create mode 100644 pages/load-balancer/troubleshooting/http-connection-errors.mdx delete mode 100644 pages/load-balancer/troubleshooting/http-errors.mdx delete mode 100644 pages/load-balancer/troubleshooting/random-problems.mdx diff --git a/pages/load-balancer/troubleshooting/certificates.mdx b/pages/load-balancer/troubleshooting/certificates.mdx new file mode 100644 index 0000000000..3a061281e9 --- /dev/null +++ b/pages/load-balancer/troubleshooting/certificates.mdx @@ -0,0 +1,92 @@ +--- +meta: + title: I am having problems with my Load Balancer's certificate + description: Troubleshoot errors that you may experience when creating an SSL/TLS certificate, adding it to your Load Balancer frontend, or successfully handling HTTPS connections. +content: + h1: I am having problems with my Load Balancer's certificate + paragraph: Troubleshoot errors that you may experience when creating an SSL/TLS certificate, adding it to your Load Balancer frontend, or successfully handling HTTPS connections. +tags: load-balancer certificate ssl tls dns +dates: + validation: 2025-03-10 + posted: 2025-03-10 +categories: + - network +--- + +## I'm experiencing DNS errors when adding an SSL/TLS certificate + +You may be trying to [create or upload](/load-balancer/how-to/add-certificate/) a certificate for your Load Balancer, and receive the following error message: + +``` +invalid argument(s): dns_name does not respect constraint, does not resolve to your Load Balancer IP +``` + +### Cause + +The domain name specified does not resolve to the Load Balancer's public IP address. + +### Solutions + +Try the following steps: + +- Ensure that a DNS record exists, pointing this domain to the Load Balancer's public IP address. +- Ensure that you have correctly typed the domain name, with no typos or errors. +- If you created the DNS record very recently, DNS propagation might not yet be complete. Wait for 30-60 minutes and try again, to see if the issue resolves itself. +- If you are trying to upload a custom certificate: + - Check the certificate's validity dates and ensure it's not expired or not yet valid. + - If the certificate has wildcards, ensure it covers the correct domain and subdomains. For example, if your certificate covers `*.example.com`, you can use it to secure `subdomain.example.com` but not `sub.subdomain.example.com`. Check the [IETF documentation](https://www.ietf.org/rfc/rfc2818.txt). +- If the error persists, check the DNS entry using a tool like `dig`, to ensure it is resolving correctly. + + +## I am experiencing HTTP errors when generating a Let's Encrypt SSL/TLS certificate + +You may be trying to [generate a Let's Encrypt certificate](/load-balancer/how-to/add-certificate/#how-to-generate-and-add-a-lets-encrypt-certificate) for your Load Balancer, and receive the following error message: + +``` +HTTP error 400: The port 80 frontend must be associated to an HTTP backend +``` + +### Cause + +Let's Encrypt certificates cannot be created for Load Balancers which have a frontend listening on port 80, but are attached to a **TCP** backend. This is because the Let's Encrypt challenge would fail. + +### Solution: + +Ensure that your Load Balancer has either: +- An HTTP-protocol backend attached to a frontend listening on port 80, or +- A TCP-protocol backend attached to a frontend listening on a port other than 80 + +Alternatively, create and import your own [custom certificate](/load-balancer/how-to/add-certificate/#how-to-import-a-certificate) for your Load Balancer, rather than generating a Let's Encrypt certificate via Scaleway. + +## I added a certificate to my Kubernetes Load Balancer via the Scaleway console, but it is not working correctly + +You may have used the Scaleway console attach a certificate to your Kubernetes Kapsule Load Balancer, and then find that the SSL certificate does not work as expected afterwards, with connections lost and HTTPS traffic dropped. + +### Cause + +Kubernetes Kapsule is a managed service, as are the Load Balancers created as part of the cluster. +Modifying a Kubernetes Load Balancer via the Scaleway console results in non-permanent modifications which are not known to the Kubernetes Kapsule service, and therefore end up being overwritten. + +### Solution + +Always modify Kubernetes Load Balancers via the cluster's Cloud Controller Manager (CCM), using [Load Balancer annotations](/kubernetes/reference-content/using-load-balancer-annotations/). + +The specific annotation to use can be found in the [Scaleway CCM documentation](https://github.com/scaleway/scaleway-cloud-controller-manager/blob/master/docs/loadbalancer-annotations.md#servicebetakubernetesioscw-loadbalancer-certificate-ids). + + +## I have a different problem related to my Load Balancer SSL/TLS certificate + +Check the following documentation: + +- [How to add an SSL/TLS certificate](/load-balancer/how-to/add-certificate/) +- [Setting up SSL bridging, offloading or passthrough](/load-balancer/reference-content/ssl-bridging-offloading-passthrough/) +- [Load Balancer API Documentation: Certificates](https://www.scaleway.com/en/developers/api/load-balancer/zoned-api/#path-certificate-get-an-ssltls-certificate) +- [Load Balancer Terraform Documentation: Certificates](https://registry.terraform.io/providers/scaleway/scaleway/latest/docs/resources/lb_certificate) + + + + + + + + \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/configuration.mdx b/pages/load-balancer/troubleshooting/configuration.mdx new file mode 100644 index 0000000000..0373164efe --- /dev/null +++ b/pages/load-balancer/troubleshooting/configuration.mdx @@ -0,0 +1,98 @@ +--- +meta: + title: I am having problems configuring my Load Balancer + description: Troubleshoot problems that you may experience when configuring your Load Balancer, such as adding backend servers, setting up Private Networks and dealing with security concerns. +content: + h1: I am having problems configuring my Load Balancer + paragraph: Troubleshoot problems that you may experience when configuring your Load Balancer, such as adding backend servers, setting up Private Networks and dealing with security concerns. +tags: load-balancer configuration backend server error security ip +dates: + validation: 2025-03-06 + posted: 2025-03-06 +categories: + - network +--- + +If your problem concerns any of the following, see our specific documentation pages: + +- [Troubleshooting certificate configuration](/load-balancer/troubleshooting/certificates/) +- [Setting up SSL bridging, offloading or passthrough](/load-balancer/reference-content/ssl-bridging-offloading-passthrough/) +- [Troubleshooting connection and HTTP errors](/load-balancer/troubleshooting/http-connection-errors/) +- General advice and help for configuring [frontends](/load-balancer/reference-content/configuring-frontends/), [backends](/load-balancer/reference-content/configuring-backends/) and [health checks](/load-balancer/reference-content/configuring-health-checks/) +- [Creating and configuring a Kubernetes Load Balancer](/kubernetes/reference-content/kubernetes-load-balancer/) + +## When adding a backend server to my Load Balancer, I get an error that the IP is not owned by Scaleway` + +You may be trying to [add a backend server](/load-balancer/how-to/create-frontends-backends/#configuring-traffic-management) to your Load Balancer's backend, and experience the following error: + +`HTTP 404: IP not owned by Scaleway` + +### Cause + +You are trying to add the IP address of a backend server that is not owned by Scaleway (i.e. is not a Scaleway resource such as an Instance, Elastic Metal server or Managed Database.) + +### Solution + +Only certain Load Balancer types (L and XL) are compatible with non-Scaleway resources as backend servers. This is indicated as "Multi-cloud provider" compatibility in the [Load Balancer creation form](https://console.scaleway.com/load-balancer/lbs/create). + +Either: + +- [Resize](/load-balancer/how-to/resize-lb/) your Load Balancer to a type that is compatible with multi-cloud backend servers, or +- Use only Scaleway resources as backend servers for your Load Balancer + +## When adding a backend server via its private IP address, I get an error saying this IP doesn't exist + +You may be trying to [add a backend server](/load-balancer/how-to/create-frontends-backends/#configuring-traffic-management) to your Load Balancer's backend using the server's private IP address, and experience an error message saying that the IP doesn't exist. + +### Cause + +You are entering an incorrect IP address for your resource, or using private IP address that is outside the standard range for private networks. + +### Solution + +- Check that you are entering the correct [private IP address](/vpc/how-to/attach-resources-to-pn/#how-to-view-the-resources-ip-address) for your resource, and that it is attached to the same Private Network as the Load Balancer. +- Verify that you are using a private IP address that is within the standard ranges used for private networks as described in [RFC1918](https://en.wikipedia.org/wiki/Private_network#Private_IPv4_addresses). Only IP addresses from within one of these ranges are supported by Scaleway Load Balancer. + + +## My Load Balancer's Elastic Metal backend servers added via private IPs are all down + +You may find that your Elastic Metal backend servers, which were added to your Load Balancer's backend via their private IP addresses, are all marked as `DOWN` as soon as you add them, and you are unable to work out why they are failing their health checks. + +### Cause + +The Load Balancer is unable to successfully communicate with the Elastic Metal backend servers over the Private Network, resulting in failed health checks, due to a configuration problem. + +### Solution + +- Check that you are entering the correct [private IP address](/vpc/how-to/attach-resources-to-pn/#how-to-view-the-resources-ip-address) for your Elastic Metal server, and that it is attached to the same Private Network as the Load Balancer. +- Elastic Metal servers require additional manual configuration of their network interface, unlike Instances and other resource types. Ensure you have [followed the necessary configuration steps](/elastic-metal/how-to/use-private-networks/#how-to-configure-the-network-interface-on-your-elastic-metal-server-for-private-networks). + +Ensure that the Elastic Metal servers have been correctly configured for the Privat Network. Note that additional steps are required. + + +## My Load Balancer's IP address is appearing in the backend application's logs, instead of the real client IP address. + +You may find that as requests are passed from the client, through the Load Balancer, to your backend servers, that the client's original IP address is replaced with the Load Balancer's IP address in your backend application's logs. This is problematic if you need the original IP address for localization, security or other purposes. + +### Cause + +Proxy Protocol has not been activated on your Load Balancer, meaning that information about the original client's connection is not being passed through to the backend servers. + +### Solution + +Activate [Proxy Protocol](/load-balancer/reference-content/configuring-backends/#proxy-protocol) on your Load Balancer, and ensure that your backend server is [correctly configured](/tutorials/proxy-protocol-v2-load-balancer/) to handle the activation of this protocol. + +## Security rules not being applied as expected, and I am having difficulties in filtering incoming traffic through my Load Balancer + +You may find that traffic is not being filtered as expected via your Load Balancer, and that Instances in your backend are not dropping unauthorized traffic as expected. + +### Cause + +Instance Security Groups and/or Load Balancer ACLs are incorrectly configured. + +### Solutions + +Instance [Security Groups](instances/how-to/use-security-groups/) should still filter public traffic arriving on your backend server Instances, as long as that traffic is arriving over the public interface, i.e. the Instance in question is attached to the Load Balancer via its public IP and not private IP. +- Ensure that your Instance is attached via its public IP address. If your Instance behind a Load Balancer is attached via a private IP address, the Security Group rules will not be applied. +- Double check your [Security Group rules](/instances/how-to/use-security-groups/#how-to-choose-security-group-settings), and that they correspond to the required ports, protocols and IP addresses configured for your Load Balancer +- To filter incoming traffic to your backend servers **as it passes through the Load Balancer**, use [Load Balancer ACLs](/load-balancer/how-to/create-manage-acls/). diff --git a/pages/load-balancer/troubleshooting/http-connection-errors.mdx b/pages/load-balancer/troubleshooting/http-connection-errors.mdx new file mode 100644 index 0000000000..403d4c97ab --- /dev/null +++ b/pages/load-balancer/troubleshooting/http-connection-errors.mdx @@ -0,0 +1,75 @@ +--- +meta: + title: I am experiencing connection problems and HTTP errors with my Load Balancer + description: Troubleshoot connection problems and HTTP errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get your application back up and running. +content: + h1: I am experiencing connection problems and HTTP errors with my Load Balancer + paragraph: Troubleshoot connection problems and HTTP errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get your application back up and running. +tags: load-balancer http-errors bad-request +dates: + validation: 2025-03-06 + posted: 2025-03-06 +categories: + - network +--- + +You may experience connection problems and HTTP errors when attempting to connect to an application served via your Load Balancer. + +This page helps you find solutions to some of these most common errors. + +### I am getting a 400 Bad Request error when accessing my application through my Load Balancer + +You may find that when attempting to connect to the domain linked to your Load Balancer / the application being served by your backend servers, you receive a `400 Bad Request` error. + +## Cause + +400 Bad Request errors occur when the backend servers cannot process a request due to client-side issues, or an incompatibility in the way that requests are passed through the Load Balancer and received by the backend server. + +## Solutions + +- Try accessing your application directly, and not through your Load Balancer, to eliminate the possibility that the problem does not come from the Load Balancer. Use a tool such as `cURL` or Postman to compare headers and body content to check how the Load Balancer is modifying requests. +- Check your Load Balancer's [logs](/load-balancer/how-to/monitor-lb-cockpit/#how-to-view-and-understand-your-load-balancer-logs) for any additional information about the way the request was handled. +- Verify your certificate and [SSL bridging/offloading/passthrough](/load-balancer/reference-content/ssl-bridging-offloading-passthrough/) settings. For example, if SSL is terminated at the Load Balancer, but the backend expects HTTPS, requests may be rejected. +- Check if [Proxy Protocol](load-balancer/reference-content/configuring-backends/#proxy-protocol) is enabled on your Load Balancer. If your backend server is not configured to handle Proxy Protocol headers correctly, it may reject the requests. Try [disabling Proxy Protocol](/load-balancer/how-to/manage-frontends-and-backends/#how-to-edit-backends-and-health-checks) on your Load Balancer to see if it resolves the issue. If the issue is resolved when Proxy Protocol is disabled, [ensure your backend server is correctly configured for Proxy Protocol](/tutorials/proxy-protocol-v2-load-balancer/) before re-enabling. + +### I am getting a 503 Service Unavailable error when trying to access my application through my Load Balancer + +You may find that when attempting to connect to the domain linked to your Load Balancer / the application being served by your backend servers, you receive a `503 Service Unavailable` error. + +### Cause + +503 Service Unavailable errors occur when backend servers are unable to handle requests due to overload or maintenance issues. It indicates that the server cannot currently fulfill the request, but may be able to in the future. + +### Solutions + +- Check the health of your backend servers. If the servers are failing their health checks, this is likely to be the reason for the error. Investigate the reason for the failing health check, and either make the necessary changes to the servers so they are able to successfully respond to health checks, or [modify your health check settings](/load-balancer/reference-content/configuring-health-checks/) as necessary. +- Check that your Load Balancer is not exceeding its bandwidth. Each Load Balancer type has a [maximum bandwidth](https://www.scaleway.com/en/pricing/network/#load-balancer) it can handle. If you are exceeding this bandwidth, a 503 error is likely. [Check your Load Balancer's metrics](/load-balancer/how-to/monitor-lb-cockpit/), and [resize your Load Balancer](/load-balancer/how-to/resize-lb/) if necessary. +- Check your Load Balancer's [backend protection settings](/load-balancer/reference-content/configuring-backends/#backend-protection), and compare with [Cockpit data](/load-balancer/how-to/monitor-lb-cockpit/). If backend protection compared to request/connection volume is set in such a way that all backend servers are becoming overloaded, you may need to add additional backend servers or adjust your backend protection settings. + + +## I am getting SSL protocol errors when trying to access my application through my Load Balancer + +You may find that when attempting to connect to the domain linked to your Load Balancer / the application being served by your backend servers, you receive an error similar to one of the following + +``` +ERR_SSL_PROTOCOL_ERROR +``` + +``` +SSL_ERROR_PROTOCOL_VERSION_ALERT +``` + +``` +This site can't provide a secure connection +OpenSSL/3.0.14: error:0A00010B:SSL routines::wrong version number +``` + +### Cause + +There is a mismatch between the SSL/TLS protocol versions or configurations between the client and the Load Balancer / its backend servers. This can prevent you from being able to establish a secure connection to your application. + +### Solution + +- Check that the client or backend servers are not using older SSL protocols such as SSLv2 or SSLv3, which are considered insecure. +- Ensure that you have correctly configured [SSL bridging, offloading or passthrough](/load-balancer/reference-content/ssl-bridging-offloading-passthrough/) on your Load Balancer, depending on your use case. +- Ensure that you have not confused activation of the [Proxy Protocol](/load-balancer/reference-content/configuring-backends/#proxy-protocol) setting as anything to do with SSL bridging, offloading or passthrough, as it is unrelated. \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/http-errors.mdx b/pages/load-balancer/troubleshooting/http-errors.mdx deleted file mode 100644 index bf16b8af8d..0000000000 --- a/pages/load-balancer/troubleshooting/http-errors.mdx +++ /dev/null @@ -1,139 +0,0 @@ ---- -meta: - title: I am experiencing HTTP errors with my Load Balancer - description: Troubleshoot HTTP errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get your application back up and running. -content: - h1: I am experiencing HTTP errors with my Load Balancer - paragraph: Troubleshoot HTTP errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get your application back up and running. -tags: load-balancer http-errors bad-request -dates: - validation: 2025-03-06 - posted: 2025-03-06 -categories: - - network ---- - -You may experience HTTP errors when attempting to connect to an application served via your Load Balancer. - -This page helps you find solutions to some of these most common errors. - -### I'm experiencing a 400 Bad Request error when accessing my application through my Load Balancer - -Check if [Proxy Protocol](load-balancer/reference-content/configuring-backends/#proxy-protocol) is enabled on your Load Balancer. If your backend server is not configured to handle Proxy Protocol headers correctly, it may reject the requests. - -Try [disabling Proxy Protocol](/load-balancer/how-to/manage-frontends-and-backends/#how-to-edit-backends-and-health-checks) on your Load Balancer to see if it resolves the issue. - -If the issue is resolved when Proxy Protocol is disabled, [ensure your backend server is correctly configured for Proxy Protocol](/tutorials/proxy-protocol-v2-load-balancer/) before re-enabling. - ---- - -### I'm experiencing a 503 Service Unavailable error when trying to access my application through my Load Balancer (OR GENERALLY UNABLE TO CONNECT TO APP) - -- Check the health of backend servers -- Check Load Balancer is not overloaded and exceeding its bandwidth (Cockpit) -- Port and protocol configuration (check Load Balancer is forwarding to the right backend port and protocol) -- Check backend application logs for errors like crashes, timeouts or rate limiting -- Scaling limits: ensure the backend has enough instances/resources to handle incoming requests - -## I'm experiencing SSL protocol or secure connection errors - -ERR_SSL_PROTOCOL_ERROR" (Chrome) -"SSL_ERROR_PROTOCOL_VERSION_ALERT" (Firefox) -"This site can't provide a secure connection" (Edge) -OpenSSL/3.0.14: error:0A00010B:SSL routines::wrong version number - -Ensure that you have correctly configured a certificate (link to doc) -Link to SSL offloading/passthrough etc. -Don't confused proxy protocol and SSL passthrough - -## I'm getting HTTP 404: IP not owned by Scaleway - -About how you need to use SCW backend servers, or else Multicloud - -HTTP Error 400: The Port 80 Frontend Must be Associated to an HTTP Backend - -If you're experiencing issues with your load balancer and receiving the error "HTTP error 400: The port 80 frontend must be associated to an HTTP backend" when trying to obtain an SSL certificate, this troubleshooting guide may help. Common symptoms include errors when trying to access the load balancer or when attempting to obtain an SSL certificate. - -Suggested Solutions: - - Verify that your backend is configured to use the HTTP protocol. Ensure that the backend is set up to accept HTTP connections from the frontend. - Check the frontend configuration: When creating or editing your frontend, make sure it is linked to the correct backend that is configured for HTTP. - Create or update the backend: If the backend is not already configured for HTTP, modify or create a new one. When creating a new backend, ensure that the protocol is set to HTTP. - Verify the ports: Ensure that the frontend is configured to listen on port 80 and that the backend is configured to use the correct port for your service. - Consult the load balancer's documentation for specific instructions on configuring frontends and backends, as well as obtaining SSL certificates. - - -## I'm experiencing a 413 Request Entity Too Large error when sending upload requests through my Load Balancer - -If you have a Kubernetes Load Balancer with an Nginx Ingress Controller, you may need to edit the `proxy-body-size`. Replace the value `50m` with teh correct desired value for your use-case. - -``` -kubectl edit cm ingress-nginx-controller -n ingress-nginx -proxy-body-size: "50m" -``` - -## I'm experiencing a 503 Service Unavailable error when accessing my application through my Load Balancer, even though the backend server passes health checks - -This issue can occur for several reasons, including: - -- **Application does not accept traffic on the expected domain or path**: Health checks generally target a specific endpoint e.g. `/health`, but your application may be rejecting requests on other paths. Test direct access to your backend using the expected request path e.g. with a curl. If your application expects a specific `Host` header, ensure the Load Balancer is configured to - -Issue -You receive a 503 Service Unavailable error when accessing your application through the Load Balancer, even though the backend server passes health checks. - -Possible Causes and Solutions -1. Application does not accept traffic on the expected domain or path -Health checks usually target a specific endpoint (e.g., /health), but your application may reject requests to other paths. -If the application enforces host-based routing, requests from the Load Balancer may not be handled correctly. -✅ Solution: - -Test direct access to your backend using the expected request path: -sh -curl -I http://:/ - -If this request returns 503, check application logs to see why it is rejecting real requests. -If your application expects a specific Host header, configure the Load Balancer to send it: - -sh -curl -I -H "Host: yourapp.example.com" http://: - -2. Rate limiting or connection exhaustion -The backend may have rate limits, thread limits, or connection pool limits that allow health check requests but block real client traffic. -Health checks are typically lightweight, so they may not trigger these limits. -✅ Solution: - -Check backend logs for rate-limiting errors (e.g., 429 Too Many Requests or resource exhaustion warnings). -Increase rate limits or connection limits in your application. -If using a database, ensure it allows enough connections to handle real traffic. -3. Protocol mismatch between Load Balancer and backend server -If the Load Balancer sends HTTP requests but the backend expects HTTPS, it may reject requests while still responding to health checks. -Similarly, if the backend uses HTTP/2 or WebSockets, ensure the Load Balancer supports it. -✅ Solution: - -Verify the protocol used by the Load Balancer. If the backend requires HTTPS, update the Load Balancer to send HTTPS requests. -If the backend requires WebSockets or HTTP/2, confirm that the Load Balancer is configured to allow them. -4. Backend server returns a 503 for real requests -The health check endpoint may be returning a cached success response while real requests trigger an application failure. -✅ Solution: - -Compare health check behavior with real requests: -sh -Copy -Edit -curl -I http://:/ -If health checks succeed but real requests fail, investigate backend logs for service errors. -Check for dependency failures (e.g., database timeouts or API failures). -5. Load Balancer is overloaded or misconfigured -The Load Balancer may run out of available connections or use a backend selection policy that allows a server to pass health checks but not receive traffic. -If connection draining is enabled, some backend instances may be marked as "healthy" but not receiving traffic. -✅ Solution: - -Check if your Load Balancer logs show failed request routing or connection timeouts. -Ensure that instances in the backend pool are not in a draining state. -If sticky sessions are enabled, verify that they are routing clients correctly. -Next Steps -Enable debug logging on the backend to capture rejected requests. -Test with a different backend instance to rule out instance-specific issues. -Check Load Balancer logs for patterns of failed requests. - ----- \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/k8s-errors.mdx b/pages/load-balancer/troubleshooting/k8s-errors.mdx index fa9db69a2c..c746cc298f 100644 --- a/pages/load-balancer/troubleshooting/k8s-errors.mdx +++ b/pages/load-balancer/troubleshooting/k8s-errors.mdx @@ -69,4 +69,26 @@ Do not attempt to create a Load Balancer manually and then add your ingress cont This results in unexpected behaviours, HTTP errors and incorrect traffic handling. -You must provision your Load Balancer via Kubernetes itself: see our [dedicated documentation](TODO). \ No newline at end of file +You must provision your Load Balancer via Kubernetes itself: see our [dedicated documentation](TODO). + +## My certificate is not being resolved when accessing my Kubernetes Load Balancer from within the cluster + +You may be able to reach applications from outside your cluster, but when trying to reach your Load Balancer from inside your Kapsule cluster, experience the following error message: + +``` +routines:ss3_get_record:wrong version number:../ssl/record/ssl3_record.c:331 +``` + +## Cause + +The Load Balancer is not properly configured to handle requests from within the cluster, specifically it is not using the hostname to route requests. + +## Solution: + +Add an [annotation](TODO) to the Load Balancer configuration, to use the hostname to route requests: + +``` +service.beta.kubernetes.io/scw-loadbalancer-use-hostname: "true" +``` + +By adding this annotation, the Load Balancer will use the hostname to route requests from within the cluster. \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/random-problems.mdx b/pages/load-balancer/troubleshooting/random-problems.mdx deleted file mode 100644 index 51d8160d27..0000000000 --- a/pages/load-balancer/troubleshooting/random-problems.mdx +++ /dev/null @@ -1,135 +0,0 @@ ---- -meta: - title: I am experiencing RANDOM errors with my Load Balancer - description: Troubleshoot RANDOM errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get everything back up and running. -content: - h1: I am experiencing RANDOM errors with my Load Balancer - paragraph: Troubleshoot RANDOM errors that you may experience when accessing applications served via your Load Balancer. Learn how to resolve common problems and get everything back up and running. -tags: load-balancer TODO -dates: - validation: 2025-03-06 - posted: 2025-03-06 -categories: - - network ---- - -## BACKEND SERVERS - -## When I try to add my backend server's IP address to the Load Balancer, I get an error - -CHECK THE ERROR -You have to have multicloud or be using Scaleway resources - -## When I want to add my instance's private IP to the balancer's backend configuration, it tells me that this IP doesn't exist. - -This error occurs because your IPs range is not in the private blocks IPs as describe in RFC1918. - -https://en.wikipedia.org/wiki/Private_network - -Scaleway allow only our public IPs or private IP that match the RFC1918 for the LB backend service. - -## How do I attach my Intsances to my LB via their private IPs via Terraform? - -https://admin.internal.scaleway.com/tickets?modal-id=ticket-details&ticket-id=1464043 - -## My Load Balancer can't communicate with Elastic Metal backend servers over a Private Network - -Ensure that the Elastic Metal servers have been correctly configured for the Privat Network. Note that additional steps are required. - - -## CERTIFICATE - -## I'm experiencing issues with my wildcard certificate not working and receiving HTTPS errors when trying to use a custom certificate, - -Common symptoms include certificates not being recognized or validated, or errors when trying to access subdomains. - -Suggested Solutions: - -- Verify that your wildcard certificate covers the correct domain and subdomains. Check the certificate's subject alternative name (SAN) or common name (CN) to ensure it matches the domain you're trying to secure. -- Ensure that your certificate is properly configured and installed on your load balancer. Check the load balancer's documentation for specific instructions on uploading and configuring certificates. -- If you're using a subdomain, make sure your certificate covers the correct subdomain. For example, if you're trying to secure 'subdomain.example.com', your certificate should cover '*.example.com'. -- Check the certificate's validity dates and ensure it's not expired or not yet valid. -- Consult the load balancer's documentation and the certificate issuer's documentation for specific troubleshooting steps and requirements. - -# I'm experiencing DNS errors when creating an SSL certificate - -You may be experiencing error messages such as - -``` -invalid argument(s): dns_name does not respect constraint / does not resolve to your Load Balancer IP -``` - -- Verify that the DNS entry is correctly set up and points to the Load Balancer IP address. -- After updating DNS entries, Wait for 30-60 minutes after updating the DNS entry to allow for propagation delay. -- Try generating the SSL certificate again after the propagation delay has passed. -- If the error persists, check the DNS entry using a tool like dig to ensure it is resolving correctly. - -## I am experiencing errors creating Let's Encrypt SSL certificate with Load Balancer - -You may be receiving "HTTP error 400: The port 80 frontend must be associated to an HTTP backend" error. - -- Verify that the backend is configured to use the HTTP protocol. -- Ensure that the frontend is correctly associated with the HTTP backend. -- Check the frontend configuration to ensure it is linked to the correct HTTP backend. -- If the backend is not already configured for HTTP, update or create a new backend with the HTTP protocol specified. -- Verify that the frontend is listening on port 80 and the backend is configured for the correct port corresponding to your service. -- Review the Load Balancer documentation on configuring frontends for additional guidance: https://www.scaleway.com/en/docs/load-balancer/reference-content/configuring-frontends/ - -## I'm experiencing issues with SSL certificates not being resolved when accessing my Kubernetes load balancers from within the cluster - -Key Points to Consider: - -- When accessing load balancers from within the cluster, Kubernetes may skip the load balancer and forward traffic in L4 to the application, causing SSL issues. -- To resolve this issue, you need to set up specific annotations on your load balancer service to instruct Kubernetes to use the hostname instead of the IP address. - - -Suggested Solution: - -- Add the following annotation to your load balancer service: `service.beta.kubernetes.io/scw-loadbalancer-use-hostname: "true"` -- This annotation will instruct Kubernetes to use the hostname of the load balancer instead of the IP address, allowing the SSL certificates to be resolved correctly. - -Refer to the Scaleway documentation for more information on load balancer annotations: https://github.com/scaleway/scaleway-cloud-controller-manager/blob/master/docs/loadbalancer-annotations.md#servicebetakubernetesioscw-loadbalancer-use-hostname - Ensure that your Kubernetes dependencies are up-to-date and that you have the latest ca-certificates installed. - Verify that your container image is correctly configured and that the SSL certificates are properly managed within the cluster or on the Scaleway side. - - -## I am experiencing issues with my load balancer that has MySQL backend servers, where connections are failing and SSL certificate generation is unsuccessful. - -- Disable Proxy Protocol : Check if the proxy protocol is enabled in your backend. If so, disable it, as it may be incompatible with your database setup. -- Verify Database Configuration : Double-check your database configuration, including the host, port, username, and password, to ensure they are correct and match the load balancer settings. -- Check Load Balancer Routing : Verify that the load balancer is correctly routing requests to your database instances. - - -Issue 2: SSL Certificate Generation - -If this is the case, you can try explicitly adding an entry for letsencrypt of the type @ IN CAA 0 issue ‘letsencrypt.org’, as shown here = https://letsencrypt.org/docs/caa/ - -- Check CAA Records : Your domain's CAA (Certificate Authority Authorization) records may be blocking the generation of the SSL certificate. If necessary, add a CAA record for Let's Encrypt with the format @ IN CAA 0 issue 'letsencrypt.org'. -- Verify Domain Configuration : Ensure that your domain is correctly configured to point to the load balancer. - - -## COCKPIT - -## I can't see any Layer 7 metrics in my Grafana dashboard - -This is normal if your Load Balancer is in TCP mode. - -## I can't see any Load Balancer logs in Grafana - -Is this normal??? - - - -## CONFIG - -## My Load Balancer's private IP address (172.16.8.2) is appearing in the backend application's logs, instead of the real client IP address. - -Use the proxy protocol annotation to retrieve the real source IP. The Scaleway support team provided documentation on how to implement this: https://www.scaleway.com/en/docs/tutorials/proxy-protocol-v2-load-balancer/ . This allowed the client to correctly log the real client IP address in the Traefik logs. - -## Security rules not being applied as expected, and I am having difficulties in filtering incoming traffic through my LB - -Key Points to Consider: - -- Security groups only apply to public IPs, not private IPs. If your instance behind a Load Balancer only has a private IP, the security group rules will not be applied. -- Security group rules will still apply to Instances serving a Load Balancer as long as they are attached by a public IP not a private one. Verify that your security group rules are correctly configured and propagated to the instances. Check that the rules correspond to the required ports, protocols, and IP addresses. Ensure that the instance is associated with the correct security group and that there are no conflicting rules blocking the traffic. -- To filter incoming traffic to your backend servers as it passes through a load balancer, you should apply security rules directly to the load balancer. Load balancers do not perform specific filtering; they redirect all authorized traffic to the backend servers, unless ACLs are configured. Consult the Scaleway documentation for more information on security groups and load balancer ACLs: https://www.scaleway.com/en/docs/instances/concepts/#security-group and https://www.scaleway.com/en/docs/load-balancer/how-to/create-manage-acls/ \ No newline at end of file From fa741464071bf2847fe37ac744fdaf7218885599 Mon Sep 17 00:00:00 2001 From: Rowena Date: Tue, 11 Mar 2025 10:05:38 +0100 Subject: [PATCH 3/8] fix(lb): add troubleshooting --- menu/navigation.json | 16 ++++ .../troubleshooting/certificates.mdx | 6 +- .../troubleshooting/configuration.mdx | 27 ++++--- .../http-connection-errors.mdx | 12 +-- .../troubleshooting/k8s-errors.mdx | 77 ++++++------------- 5 files changed, 63 insertions(+), 75 deletions(-) diff --git a/menu/navigation.json b/menu/navigation.json index 113aff08b7..07439d986f 100644 --- a/menu/navigation.json +++ b/menu/navigation.json @@ -3597,6 +3597,22 @@ }, { "items": [ + { + "label": "I am having problems configuring my Load Balancer", + "slug": "configuration" + }, + { + "label": "I am experiencing connection problems and HTTP errors with my Load Balancer", + "slug": "http-connection-errors" + }, + { + "label": "I am having problems with my Load Balancer's certificate", + "slug": "certificates" + }, + { + "label": "I am experiencing problems with my Kubernetes Load Balancer", + "slug": "k8s-errors" + }, { "label": "Load Balancer Limitations", "slug": "load-balancer-limitations" diff --git a/pages/load-balancer/troubleshooting/certificates.mdx b/pages/load-balancer/troubleshooting/certificates.mdx index 3a061281e9..9e8de96f90 100644 --- a/pages/load-balancer/troubleshooting/certificates.mdx +++ b/pages/load-balancer/troubleshooting/certificates.mdx @@ -25,7 +25,7 @@ invalid argument(s): dns_name does not respect constraint, does not res The domain name specified does not resolve to the Load Balancer's public IP address. -### Solutions +### Solution Try the following steps: @@ -53,8 +53,8 @@ Let's Encrypt certificates cannot be created for Load Balancers which have a fro ### Solution: Ensure that your Load Balancer has either: -- An HTTP-protocol backend attached to a frontend listening on port 80, or -- A TCP-protocol backend attached to a frontend listening on a port other than 80 +- An HTTP-protocol-backend attached to a frontend listening on port 80, or +- A TCP-protocol-backend attached to a frontend listening on a port other than 80 Alternatively, create and import your own [custom certificate](/load-balancer/how-to/add-certificate/#how-to-import-a-certificate) for your Load Balancer, rather than generating a Let's Encrypt certificate via Scaleway. diff --git a/pages/load-balancer/troubleshooting/configuration.mdx b/pages/load-balancer/troubleshooting/configuration.mdx index 0373164efe..8176543d3b 100644 --- a/pages/load-balancer/troubleshooting/configuration.mdx +++ b/pages/load-balancer/troubleshooting/configuration.mdx @@ -21,7 +21,7 @@ If your problem concerns any of the following, see our specific documentation pa - General advice and help for configuring [frontends](/load-balancer/reference-content/configuring-frontends/), [backends](/load-balancer/reference-content/configuring-backends/) and [health checks](/load-balancer/reference-content/configuring-health-checks/) - [Creating and configuring a Kubernetes Load Balancer](/kubernetes/reference-content/kubernetes-load-balancer/) -## When adding a backend server to my Load Balancer, I get an error that the IP is not owned by Scaleway` +## When adding a backend server to my Load Balancer, I get the message: IP is not owned by Scaleway You may be trying to [add a backend server](/load-balancer/how-to/create-frontends-backends/#configuring-traffic-management) to your Load Balancer's backend, and experience the following error: @@ -29,18 +29,18 @@ You may be trying to [add a backend server](/load-balancer/how-to/create-fronten ### Cause -You are trying to add the IP address of a backend server that is not owned by Scaleway (i.e. is not a Scaleway resource such as an Instance, Elastic Metal server or Managed Database.) +You are trying to add the IP address of a backend server that is not owned by Scaleway (i.e. is not a Scaleway resource such as an Instance, Elastic Metal server or Managed Database). ### Solution -Only certain Load Balancer types (L and XL) are compatible with non-Scaleway resources as backend servers. This is indicated as "Multi-cloud provider" compatibility in the [Load Balancer creation form](https://console.scaleway.com/load-balancer/lbs/create). +Only certain Load Balancer types (L and XL) allow you to add non-Scaleway resources as backend servers. This is indicated as "Multi-cloud provider" compatibility in the [Load Balancer creation form](https://console.scaleway.com/load-balancer/lbs/create). Either: - [Resize](/load-balancer/how-to/resize-lb/) your Load Balancer to a type that is compatible with multi-cloud backend servers, or - Use only Scaleway resources as backend servers for your Load Balancer -## When adding a backend server via its private IP address, I get an error saying this IP doesn't exist +## When adding a backend server via its private IP address, I get the message: IP doesn't exist You may be trying to [add a backend server](/load-balancer/how-to/create-frontends-backends/#configuring-traffic-management) to your Load Balancer's backend using the server's private IP address, and experience an error message saying that the IP doesn't exist. @@ -56,19 +56,18 @@ You are entering an incorrect IP address for your resource, or using private IP ## My Load Balancer's Elastic Metal backend servers added via private IPs are all down -You may find that your Elastic Metal backend servers, which were added to your Load Balancer's backend via their private IP addresses, are all marked as `DOWN` as soon as you add them, and you are unable to work out why they are failing their health checks. +You may add Elastic Metal backend servers to your Load Balancer using their private IP address, and find they are marked as `DOWN` as soon as you add them. You are unable to work out why they are failing their health checks. ### Cause -The Load Balancer is unable to successfully communicate with the Elastic Metal backend servers over the Private Network, resulting in failed health checks, due to a configuration problem. +The Load Balancer is unable to successfully communicate with the Elastic Metal backend servers over the Private Network, resulting in failed health checks. ### Solution -- Check that you are entering the correct [private IP address](/vpc/how-to/attach-resources-to-pn/#how-to-view-the-resources-ip-address) for your Elastic Metal server, and that it is attached to the same Private Network as the Load Balancer. +- Check that your health checks and backend servers are correctly configured to work together. +- Check that you have entered the correct [private IP address](/vpc/how-to/attach-resources-to-pn/#how-to-view-the-resources-ip-address) for your Elastic Metal server, and that it is attached to the same Private Network as the Load Balancer. - Elastic Metal servers require additional manual configuration of their network interface, unlike Instances and other resource types. Ensure you have [followed the necessary configuration steps](/elastic-metal/how-to/use-private-networks/#how-to-configure-the-network-interface-on-your-elastic-metal-server-for-private-networks). -Ensure that the Elastic Metal servers have been correctly configured for the Privat Network. Note that additional steps are required. - ## My Load Balancer's IP address is appearing in the backend application's logs, instead of the real client IP address. @@ -80,7 +79,7 @@ Proxy Protocol has not been activated on your Load Balancer, meaning that inform ### Solution -Activate [Proxy Protocol](/load-balancer/reference-content/configuring-backends/#proxy-protocol) on your Load Balancer, and ensure that your backend server is [correctly configured](/tutorials/proxy-protocol-v2-load-balancer/) to handle the activation of this protocol. +Activate [Proxy Protocol](/load-balancer/reference-content/configuring-backends/#proxy-protocol) on your Load Balancer, and ensure that your backend server is [correctly configured](/tutorials/proxy-protocol-v2-load-balancer/) to handle this protocol. ## Security rules not being applied as expected, and I am having difficulties in filtering incoming traffic through my Load Balancer @@ -90,9 +89,9 @@ You may find that traffic is not being filtered as expected via your Load Balanc Instance Security Groups and/or Load Balancer ACLs are incorrectly configured. -### Solutions +### Solution -Instance [Security Groups](instances/how-to/use-security-groups/) should still filter public traffic arriving on your backend server Instances, as long as that traffic is arriving over the public interface, i.e. the Instance in question is attached to the Load Balancer via its public IP and not private IP. +Instance [Security Groups](instances/how-to/use-security-groups/) should still filter public traffic arriving on your backend server Instances, as long as that traffic is arriving over the public interface. This means the Instance in question must be attached to the Load Balancer via its public IP and not any private IP. - Ensure that your Instance is attached via its public IP address. If your Instance behind a Load Balancer is attached via a private IP address, the Security Group rules will not be applied. -- Double check your [Security Group rules](/instances/how-to/use-security-groups/#how-to-choose-security-group-settings), and that they correspond to the required ports, protocols and IP addresses configured for your Load Balancer -- To filter incoming traffic to your backend servers **as it passes through the Load Balancer**, use [Load Balancer ACLs](/load-balancer/how-to/create-manage-acls/). +- Double check your [Security Group rules](/instances/how-to/use-security-groups/#how-to-choose-security-group-settings), to verify that they correspond to the required ports, protocols and IP addresses configured for your Load Balancer +- To filter incoming traffic to your backend servers **as it passes through the Load Balancer**, use [Load Balancer ACLs](/load-balancer/how-to/create-manage-acls/). \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/http-connection-errors.mdx b/pages/load-balancer/troubleshooting/http-connection-errors.mdx index 403d4c97ab..32bc318b56 100644 --- a/pages/load-balancer/troubleshooting/http-connection-errors.mdx +++ b/pages/load-balancer/troubleshooting/http-connection-errors.mdx @@ -17,22 +17,22 @@ You may experience connection problems and HTTP errors when attempting to connec This page helps you find solutions to some of these most common errors. -### I am getting a 400 Bad Request error when accessing my application through my Load Balancer +## I am getting a 400 Bad Request error when accessing my application through my Load Balancer You may find that when attempting to connect to the domain linked to your Load Balancer / the application being served by your backend servers, you receive a `400 Bad Request` error. -## Cause +### Cause 400 Bad Request errors occur when the backend servers cannot process a request due to client-side issues, or an incompatibility in the way that requests are passed through the Load Balancer and received by the backend server. -## Solutions +### Solutions - Try accessing your application directly, and not through your Load Balancer, to eliminate the possibility that the problem does not come from the Load Balancer. Use a tool such as `cURL` or Postman to compare headers and body content to check how the Load Balancer is modifying requests. -- Check your Load Balancer's [logs](/load-balancer/how-to/monitor-lb-cockpit/#how-to-view-and-understand-your-load-balancer-logs) for any additional information about the way the request was handled. +- Check your Load Balancer's [Grafana dashboard](/load-balancer/how-to/monitor-lb-cockpit/) for any additional information about the way the request was handled. - Verify your certificate and [SSL bridging/offloading/passthrough](/load-balancer/reference-content/ssl-bridging-offloading-passthrough/) settings. For example, if SSL is terminated at the Load Balancer, but the backend expects HTTPS, requests may be rejected. - Check if [Proxy Protocol](load-balancer/reference-content/configuring-backends/#proxy-protocol) is enabled on your Load Balancer. If your backend server is not configured to handle Proxy Protocol headers correctly, it may reject the requests. Try [disabling Proxy Protocol](/load-balancer/how-to/manage-frontends-and-backends/#how-to-edit-backends-and-health-checks) on your Load Balancer to see if it resolves the issue. If the issue is resolved when Proxy Protocol is disabled, [ensure your backend server is correctly configured for Proxy Protocol](/tutorials/proxy-protocol-v2-load-balancer/) before re-enabling. -### I am getting a 503 Service Unavailable error when trying to access my application through my Load Balancer +## I am getting a 503 Service Unavailable error when trying to access my application through my Load Balancer You may find that when attempting to connect to the domain linked to your Load Balancer / the application being served by your backend servers, you receive a `503 Service Unavailable` error. @@ -42,7 +42,7 @@ You may find that when attempting to connect to the domain linked to your Load B ### Solutions -- Check the health of your backend servers. If the servers are failing their health checks, this is likely to be the reason for the error. Investigate the reason for the failing health check, and either make the necessary changes to the servers so they are able to successfully respond to health checks, or [modify your health check settings](/load-balancer/reference-content/configuring-health-checks/) as necessary. +- Check the health of your backend servers. If the servers are failing their health checks, this is likely to be the reason for the error. Investigate the reason for the failing health check, and reconfigure your server or [health check settings](/load-balancer/reference-content/configuring-health-checks/) as necessary. - Check that your Load Balancer is not exceeding its bandwidth. Each Load Balancer type has a [maximum bandwidth](https://www.scaleway.com/en/pricing/network/#load-balancer) it can handle. If you are exceeding this bandwidth, a 503 error is likely. [Check your Load Balancer's metrics](/load-balancer/how-to/monitor-lb-cockpit/), and [resize your Load Balancer](/load-balancer/how-to/resize-lb/) if necessary. - Check your Load Balancer's [backend protection settings](/load-balancer/reference-content/configuring-backends/#backend-protection), and compare with [Cockpit data](/load-balancer/how-to/monitor-lb-cockpit/). If backend protection compared to request/connection volume is set in such a way that all backend servers are becoming overloaded, you may need to add additional backend servers or adjust your backend protection settings. diff --git a/pages/load-balancer/troubleshooting/k8s-errors.mdx b/pages/load-balancer/troubleshooting/k8s-errors.mdx index c746cc298f..a1f9335be6 100644 --- a/pages/load-balancer/troubleshooting/k8s-errors.mdx +++ b/pages/load-balancer/troubleshooting/k8s-errors.mdx @@ -1,10 +1,10 @@ --- meta: title: I am experiencing problems with my Kubernetes Load Balancer - description: Troubleshoot problems with your Kubernetes Load Balancer + description: Troubleshoot problems with your Kubernetes Load Balancer. Discover the solutions to common errors, and get your Load Balancer service back up and running. content: h1: I am experiencing problems with my Kubernetes Load Balancer - paragraph: Troubleshoot problems with your Kubernetes Load Balancer + paragraph: Troubleshoot problems with your Kubernetes Load Balancer. Discover the solutions to common errors, and get your Load Balancer service back up and running. tags: load-balancer kubernetes annotations dates: validation: 2025-03-06 @@ -13,63 +13,31 @@ categories: - network --- -You may experience errors when attempting to configure your Kubernetes Load Balancer. +If you are experiencing errors with your Kubernetes Kapsule Load Balancer. This page helps you find solutions to some of the most common problems. -This page helps you find solutions to some of the most common problems. + +You should **never** try to create or modify a Kubernetes Kapsule's Load Balancer via the Scaleway console, the API, or any other devtools. -### I can't add a certificate to my Kubernetes Load Balancer +This leads to unexpected and unreliable behaviour, as the Kluster's **C**loud **C**ontroller **M**anager is unaware of the Load Balancer and attempts to overwrite configurations made in the console. -Do it via annotations, not the console - -If you're also using Terraform, then: -- For SSL certificate management via terraform, see -https://registry.terraform.io/providers/scaleway/scaleway/latest/docs/resources/lb_certificate -And in your "ingress_nginx" manifest ensure you have a cert-manager that you'll need to deploy via terraform. -https://registry.terraform.io/modules/terraform-iaac/cert-manager/kubernetes/latest +Always provision and modify Kubernetes Load Balancers via the [CCM](/kubernetes/reference-content/kubernetes-load-balancer/#creating-a-load-balancer-for-your-cluster-overview). Use [annotations](/kubernetes/reference-content/kubernetes-load-balancer/#defining-your-load-balancers-configuration-via-annotations) to modify configure your cluster’s Load Balancer. + ## I'm experiencing connectivity issues with my Kubernetes Load Balancer -Symptoms: - -- Load balancer not connecting to nodes in Kapsule cluster. -- Application inaccessible from internet. -- Health checks failing for some nodes. - -Troubleshooting Steps: - -- Ensure that you provisioned and configured your Load Balancer via Kubernetes nad not via the Scaleway console, which provokes unexpected behaviours and errors, including certificate addition -- Check `externalTrafficPolicy` setting. If it is set to "Local" instead of "Cluster", this could be causing the issue. Change the policy to "Cluster". -- Try enabling or disabling Cloudflare's Proxy Mode, which may be affecting connectivity -- Verify that the required service is running on all nodes. If it is missing from some nodes, this could be causing health checks to fail. - -## I'm using Kubernetes cluster with Traefik2 proxy, when I check the logs of Traefik DaemonSet the load balancer internal IP appears instead of real client IP. - -the engineers have told us that you need to use the proxy protocol annotation to retrieve the real source IP. +You may find that your Load Balancer is not connecting to nodes in your Kapsule cluster, meaning that health checks are failing and your application is inaccessible from the internet -The doc for the implem is here: https://www.scaleway.com/en/docs/tutorials/proxy-protocol-v2-load-balancer/ +### Cause -## My Load Balancer is not connecting to my Kapsule cluster nodes, resulting in my application being inaccessible from the internet. +A configuration issue is preventing successful communication between your Load Balancer and the cluster's nodes. -Potential Causes and Solutions: +### Solutions -- Incorrect externalTrafficPolicy setting : Check if the externalTrafficPolicy setting in your Kubernetes configuration is set to "Local". If so, try changing it to "Cluster" to allow all nodes to expose the service and route traffic to the correct node. -- Cloudflare proxy mode enabled : If you're using Cloudflare as your DNS, CDN, WAF, etc., try disabling the proxy mode to ensure that the load balancer receives traffic from your IP address instead of Cloudflare's IP addresses. -- Nodes not running pods : Verify that all nodes in your Kapsule cluster are running pods associated with the service. If not, ensure that the pods are properly deployed and running on all nodes. -- Load balancer configuration issues : Check the load balancer configuration to ensure that it's properly set up to route traffic to all nodes in the cluster. - -Additional Troubleshooting Steps: - -- Check the load balancer logs for any errors or warnings that may indicate the cause of the issue. -- Verify that the Kapsule cluster nodes are properly configured and running. -- Test the application by accessing it from a different location or network to rule out any issues with your local network or internet connection. - -## My k8s Load Balancer is not behaving as expected - -Do not attempt to create a Load Balancer manually and then add your ingress controller or Kubernetes nodes as backend server. - -This results in unexpected behaviours, HTTP errors and incorrect traffic handling. +- Ensure that you provisioned and configured your Load Balancer via Kubernetes nad not via the Scaleway console, which provokes unexpected behaviors and errors. +- Verify that the required service is running on all nodes. If it is missing from some nodes, this could be causing health checks to fail. +- Check your cluster's `externalTrafficPolicy` setting. If it is set to `Local` instead of `Cluster`, this could be causing the issue. Change the policy to `Cluster`. +- Try enabling or disabling Cloudflare's Proxy Mode, which may be affecting connectivity -You must provision your Load Balancer via Kubernetes itself: see our [dedicated documentation](TODO). ## My certificate is not being resolved when accessing my Kubernetes Load Balancer from within the cluster @@ -79,16 +47,21 @@ You may be able to reach applications from outside your cluster, but when trying routines:ss3_get_record:wrong version number:../ssl/record/ssl3_record.c:331 ``` -## Cause +### Cause -The Load Balancer is not properly configured to handle requests from within the cluster, specifically it is not using the hostname to route requests. +The Load Balancer is not properly configured to handle requests from within the cluster. Specifically, it is not using the hostname to route requests. ## Solution: -Add an [annotation](TODO) to the Load Balancer configuration, to use the hostname to route requests: +Add an [annotation](/kubernetes/reference-content/kubernetes-load-balancer/#defining-your-load-balancers-configuration-via-annotations) to the Load Balancer configuration, to use the hostname to route requests: ``` service.beta.kubernetes.io/scw-loadbalancer-use-hostname: "true" ``` -By adding this annotation, the Load Balancer will use the hostname to route requests from within the cluster. \ No newline at end of file +By adding this annotation, the Load Balancer will use the hostname to route requests from within the cluster. + +## I am experiencing a different problem + +- Check the Load Balancer troubleshooting in the [Kubernetes Kapsule documentation](/kubernetes/reference-content/kubernetes-load-balancer/#troubleshooting-kubernetes-load-balancers) +- Read the full documentation on [creating and configuring Kubernetes Load Balancers](/kubernetes/reference-content/kubernetes-load-balancer/) From f69e79afaff06c8383832037d5575bdfdb70f6dc Mon Sep 17 00:00:00 2001 From: Rowena Jones <36301604+RoRoJ@users.noreply.github.com> Date: Tue, 11 Mar 2025 12:06:10 +0100 Subject: [PATCH 4/8] Update pages/load-balancer/troubleshooting/k8s-errors.mdx --- pages/load-balancer/troubleshooting/k8s-errors.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/load-balancer/troubleshooting/k8s-errors.mdx b/pages/load-balancer/troubleshooting/k8s-errors.mdx index a1f9335be6..a166e21122 100644 --- a/pages/load-balancer/troubleshooting/k8s-errors.mdx +++ b/pages/load-balancer/troubleshooting/k8s-errors.mdx @@ -18,7 +18,7 @@ If you are experiencing errors with your Kubernetes Kapsule Load Balancer. This You should **never** try to create or modify a Kubernetes Kapsule's Load Balancer via the Scaleway console, the API, or any other devtools. -This leads to unexpected and unreliable behaviour, as the Kluster's **C**loud **C**ontroller **M**anager is unaware of the Load Balancer and attempts to overwrite configurations made in the console. +This leads to unexpected and unreliable behaviour, as the cluster's **C**loud **C**ontroller **M**anager is unaware of the Load Balancer and attempts to overwrite configurations made in the console. Always provision and modify Kubernetes Load Balancers via the [CCM](/kubernetes/reference-content/kubernetes-load-balancer/#creating-a-load-balancer-for-your-cluster-overview). Use [annotations](/kubernetes/reference-content/kubernetes-load-balancer/#defining-your-load-balancers-configuration-via-annotations) to modify configure your cluster’s Load Balancer. From 87037fba0f0b99817f69708179b9293a8db8c001 Mon Sep 17 00:00:00 2001 From: Rowena Jones <36301604+RoRoJ@users.noreply.github.com> Date: Wed, 12 Mar 2025 10:07:27 +0100 Subject: [PATCH 5/8] Apply suggestions from code review Co-authored-by: Benedikt Rollik Co-authored-by: Jessica <113192637+jcirinosclwy@users.noreply.github.com> Co-authored-by: ldecarvalho-doc <82805470+ldecarvalho-doc@users.noreply.github.com> --- pages/load-balancer/troubleshooting/certificates.mdx | 8 -------- pages/load-balancer/troubleshooting/configuration.mdx | 8 ++++---- pages/load-balancer/troubleshooting/k8s-errors.mdx | 8 ++++---- 3 files changed, 8 insertions(+), 16 deletions(-) diff --git a/pages/load-balancer/troubleshooting/certificates.mdx b/pages/load-balancer/troubleshooting/certificates.mdx index 9e8de96f90..d01a23f878 100644 --- a/pages/load-balancer/troubleshooting/certificates.mdx +++ b/pages/load-balancer/troubleshooting/certificates.mdx @@ -82,11 +82,3 @@ Check the following documentation: - [Setting up SSL bridging, offloading or passthrough](/load-balancer/reference-content/ssl-bridging-offloading-passthrough/) - [Load Balancer API Documentation: Certificates](https://www.scaleway.com/en/developers/api/load-balancer/zoned-api/#path-certificate-get-an-ssltls-certificate) - [Load Balancer Terraform Documentation: Certificates](https://registry.terraform.io/providers/scaleway/scaleway/latest/docs/resources/lb_certificate) - - - - - - - - \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/configuration.mdx b/pages/load-balancer/troubleshooting/configuration.mdx index 8176543d3b..aeb287b462 100644 --- a/pages/load-balancer/troubleshooting/configuration.mdx +++ b/pages/load-balancer/troubleshooting/configuration.mdx @@ -87,11 +87,11 @@ You may find that traffic is not being filtered as expected via your Load Balanc ### Cause -Instance Security Groups and/or Load Balancer ACLs are incorrectly configured. +Instance security groups and/or Load Balancer ACLs are incorrectly configured. ### Solution -Instance [Security Groups](instances/how-to/use-security-groups/) should still filter public traffic arriving on your backend server Instances, as long as that traffic is arriving over the public interface. This means the Instance in question must be attached to the Load Balancer via its public IP and not any private IP. -- Ensure that your Instance is attached via its public IP address. If your Instance behind a Load Balancer is attached via a private IP address, the Security Group rules will not be applied. -- Double check your [Security Group rules](/instances/how-to/use-security-groups/#how-to-choose-security-group-settings), to verify that they correspond to the required ports, protocols and IP addresses configured for your Load Balancer +Instance [security groups](instances/how-to/use-security-groups/) should still filter public traffic arriving on your backend server Instances, as long as that traffic is arriving over the public interface. This means the Instance in question must be attached to the Load Balancer via its public IP and not any private IP. +- Ensure that your Instance is attached via its public IP address. If your Instance behind a Load Balancer is attached via a private IP address, the security group rules will not be applied. +- Double check your [security group rules](/instances/how-to/use-security-groups/#how-to-choose-security-group-settings), to verify that they correspond to the required ports, protocols, and IP addresses configured for your Load Balancer - To filter incoming traffic to your backend servers **as it passes through the Load Balancer**, use [Load Balancer ACLs](/load-balancer/how-to/create-manage-acls/). \ No newline at end of file diff --git a/pages/load-balancer/troubleshooting/k8s-errors.mdx b/pages/load-balancer/troubleshooting/k8s-errors.mdx index a166e21122..e6613e8628 100644 --- a/pages/load-balancer/troubleshooting/k8s-errors.mdx +++ b/pages/load-balancer/troubleshooting/k8s-errors.mdx @@ -16,11 +16,11 @@ categories: If you are experiencing errors with your Kubernetes Kapsule Load Balancer. This page helps you find solutions to some of the most common problems. -You should **never** try to create or modify a Kubernetes Kapsule's Load Balancer via the Scaleway console, the API, or any other devtools. +You should **never** try to create or modify a Kubernetes Kapsule's Load Balancer via the Scaleway console, the API, or any other developer tools. This leads to unexpected and unreliable behaviour, as the cluster's **C**loud **C**ontroller **M**anager is unaware of the Load Balancer and attempts to overwrite configurations made in the console. -Always provision and modify Kubernetes Load Balancers via the [CCM](/kubernetes/reference-content/kubernetes-load-balancer/#creating-a-load-balancer-for-your-cluster-overview). Use [annotations](/kubernetes/reference-content/kubernetes-load-balancer/#defining-your-load-balancers-configuration-via-annotations) to modify configure your cluster’s Load Balancer. +Always provision and modify Kubernetes Load Balancers via the [CCM](/kubernetes/reference-content/kubernetes-load-balancer/#creating-a-load-balancer-for-your-cluster-overview). Use [annotations](/kubernetes/reference-content/kubernetes-load-balancer/#defining-your-load-balancers-configuration-via-annotations) to configure your cluster’s Load Balancer. ## I'm experiencing connectivity issues with my Kubernetes Load Balancer @@ -33,10 +33,10 @@ A configuration issue is preventing successful communication between your Load B ### Solutions -- Ensure that you provisioned and configured your Load Balancer via Kubernetes nad not via the Scaleway console, which provokes unexpected behaviors and errors. +- Ensure that you provisioned and configured your Load Balancer via Kubernetes and not via the Scaleway console, which provokes unexpected behaviors and errors. - Verify that the required service is running on all nodes. If it is missing from some nodes, this could be causing health checks to fail. - Check your cluster's `externalTrafficPolicy` setting. If it is set to `Local` instead of `Cluster`, this could be causing the issue. Change the policy to `Cluster`. -- Try enabling or disabling Cloudflare's Proxy Mode, which may be affecting connectivity +- Try enabling or disabling Cloudflare's Proxy Mode, which may be affecting connectivity. ## My certificate is not being resolved when accessing my Kubernetes Load Balancer from within the cluster From 8084acdff8c7c175f009ee712c6856123b282c20 Mon Sep 17 00:00:00 2001 From: Rowena Jones <36301604+RoRoJ@users.noreply.github.com> Date: Fri, 14 Mar 2025 14:19:32 +0100 Subject: [PATCH 6/8] Update pages/load-balancer/troubleshooting/configuration.mdx --- pages/load-balancer/troubleshooting/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/load-balancer/troubleshooting/configuration.mdx b/pages/load-balancer/troubleshooting/configuration.mdx index aeb287b462..8e065b6007 100644 --- a/pages/load-balancer/troubleshooting/configuration.mdx +++ b/pages/load-balancer/troubleshooting/configuration.mdx @@ -81,7 +81,7 @@ Proxy Protocol has not been activated on your Load Balancer, meaning that inform Activate [Proxy Protocol](/load-balancer/reference-content/configuring-backends/#proxy-protocol) on your Load Balancer, and ensure that your backend server is [correctly configured](/tutorials/proxy-protocol-v2-load-balancer/) to handle this protocol. -## Security rules not being applied as expected, and I am having difficulties in filtering incoming traffic through my Load Balancer +## Security rules are not being applied as expected, and I am having difficulties in filtering incoming traffic through my Load Balancer You may find that traffic is not being filtered as expected via your Load Balancer, and that Instances in your backend are not dropping unauthorized traffic as expected. From e09fee878da1fdf6c4eb6af485f87488c6e367d1 Mon Sep 17 00:00:00 2001 From: Rowena Jones <36301604+RoRoJ@users.noreply.github.com> Date: Fri, 14 Mar 2025 14:19:56 +0100 Subject: [PATCH 7/8] Apply suggestions from code review --- pages/load-balancer/troubleshooting/k8s-errors.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/load-balancer/troubleshooting/k8s-errors.mdx b/pages/load-balancer/troubleshooting/k8s-errors.mdx index e6613e8628..c4759e86ee 100644 --- a/pages/load-balancer/troubleshooting/k8s-errors.mdx +++ b/pages/load-balancer/troubleshooting/k8s-errors.mdx @@ -13,7 +13,7 @@ categories: - network --- -If you are experiencing errors with your Kubernetes Kapsule Load Balancer. This page helps you find solutions to some of the most common problems. +If you are experiencing errors with your Kubernetes Kapsule Load Balancer, this page may help you find solutions to some of the most common problems. You should **never** try to create or modify a Kubernetes Kapsule's Load Balancer via the Scaleway console, the API, or any other developer tools. From 2cff2e9f131de23d20aca75edf97608bc74bb4a0 Mon Sep 17 00:00:00 2001 From: Rowena Date: Fri, 14 Mar 2025 14:36:26 +0100 Subject: [PATCH 8/8] fix(lb): add index and links --- .../load-balancer/how-to/add-certificate.mdx | 4 ++ .../kubernetes-load-balancer.mdx | 1 + pages/load-balancer/troubleshooting/index.mdx | 70 +++++++++++++++++-- 3 files changed, 71 insertions(+), 4 deletions(-) diff --git a/pages/load-balancer/how-to/add-certificate.mdx b/pages/load-balancer/how-to/add-certificate.mdx index 5ceb5d6c04..6d8872fb29 100644 --- a/pages/load-balancer/how-to/add-certificate.mdx +++ b/pages/load-balancer/how-to/add-certificate.mdx @@ -200,4 +200,8 @@ Use the button to delete unwanted certificates. If you have more than one certificate for your Load Balancer, it will automatically use the first in the list, regardless of its expiry date. Ensure that you delete any expired certificates, and that the current certificate you want to use is first in the list. +## Troubleshooting certificate problems + +See our [dedicated troubleshooting documentation](/load-balancer/troubleshooting/certifiates/.) + diff --git a/pages/load-balancer/reference-content/kubernetes-load-balancer.mdx b/pages/load-balancer/reference-content/kubernetes-load-balancer.mdx index 0a3450bba3..5eeb947bd7 100644 --- a/pages/load-balancer/reference-content/kubernetes-load-balancer.mdx +++ b/pages/load-balancer/reference-content/kubernetes-load-balancer.mdx @@ -24,6 +24,7 @@ Full documentation for creating and configuring a Load Balancer for your cluster - [Managing Load Balancer IPs](/kubernetes/reference-content/managing-load-balancer-ips/) - [Using Load Balancer annotations](/kubernetes/reference-content/using-load-balancer-annotations/) - [Exposing a Kubernetes Kapsule ingress controller service with a Load Balancer](/kubernetes/reference-content/lb-ingress-controller/) +- [Troubleshooting Kubernetes Load Balancers](/load-balancer/troubleshooting/k8s-errors/) You may also find the following resources helpful: diff --git a/pages/load-balancer/troubleshooting/index.mdx b/pages/load-balancer/troubleshooting/index.mdx index 6256d357ce..9ad9c65b08 100644 --- a/pages/load-balancer/troubleshooting/index.mdx +++ b/pages/load-balancer/troubleshooting/index.mdx @@ -1,8 +1,70 @@ --- meta: - title: Load Balancers - Troubleshooting - description: Load Balancers Troubleshooting + title: Load Balancer - Troubleshooting + description: Find troubleshooting resources for resolving issues with Scaleway Load Balancers. content: - h1: Load Balancers - Troubleshooting - paragraph: Load Balancers Troubleshooting + h1: Load Balancer - Troubleshooting + paragraph: Find troubleshooting resources for resolving issues with Scaleway Load Balancers. +dates: + posted: 2025-03-14 + validation: 2025-03-14 +categories: + - network --- + + + +## Featured Pages + + + + + + + + + + + + +## Load Balancer troubleshooting pages + +- [I am having problems with my Load Balancer's certificate](/load-balancer/troubleshooting/certificates/) +- [I am having problems configuring my Load Balancer](/load-balancer/troubleshooting/configuration/) +- [I am experiencing connection problems and HTTP errors with my Load Balancer](/load-balancer/troubleshooting/http-connection-errors/) +- [I am experiencing problems with my Kubernetes Load Balancer](/load-balancer/troubleshooting/k8s-errors/) +- [Load Balancer Limitations](/load-balancer/troubleshooting/load-balancer-limitations/)