Skip to content

Commit 9f71ce3

Browse files
SlimJ4SlimJ4
committed
Guide Creation
Copied from the Public Cloud Load Balancer, need to finish adapting to the OVHcloud Load Balancer product
1 parent 88e83fa commit 9f71ce3

File tree

2 files changed

+437
-0
lines changed

2 files changed

+437
-0
lines changed

‎pages/network/load_balancer/use_api_logs_2_customers/guide.en-gb.md

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
---
2+
title: OVHcloud Load Balancer TCP / HTTP / HTTPS Logs Forwarding
3+
excerpt: Find out how to forward your logs from an OVHcloud Load Balancer to Logs Data Platform
4+
updated: 2024-09-25
5+
---
6+
 
7+
## Objective
8+
9+
The purpose of this guide is to show you how to enable the forwarding of logs from your OVHcloud Load Balancer to Logs Data Platform (LDP), a platform that helps you store, archive, query and visualize your logs.
10+
If you would like to find out more about Logs Data Platform before reading this guide, please refer to the [Logs Data Platform introduction guide](/pages/manage_and_operate/observability/logs_data_platform/getting_started_introduction_to_LDP).
11+
12+
> [!primary]
13+
> In order to use this feature, you must first make a refresh call to the Load Balancers whose logs you want to collect.
14+
> This is necessary to ensure that the logging format of the OVHcloud Load Balancer is up to date.
15+
>
16+
17+
## Glossary
18+
19+
- **Logs Data Platform:** a fully managed and secured log management platform by OVHcloud. For more information, consult the [Logs Data Platform](https://www.ovhcloud.com/en-gb/logs-data-platform/) service page.
20+
- **Data Stream:** a logical partition of logs which you create in an LDP account and which you will use when ingesting, viewing or querying your logs. Multiple sources can be stored in the same data stream, and it is the unit that can be used to define a log pipeline (retention policy, archiving, live streaming, etc.), access rights and alert policies.
21+
- **Logs forwarding:** a feature integrated into an OVHcloud product to ingest the logs of its services into a *Data Stream* of an LDP account in the same OVHcloud account. This feature must be activated by the customer and per service.
22+
- **Logs forwarding Subscription:** when enabling the logs forwarding for a given OVHcloud service to a given LDP *Data Stream*, a *Subscription* is created and attached to the *Data Stream* for further management by the customer.
23+
24+
## Requirements
25+
26+
- A Logs Data Platform (LDP) account with at least one active *Stream* configured. This guide will walk you through all the necessary steps: [Quick start for Logs Data Platform](/pages/manage_and_operate/observability/logs_data_platform/getting_started_quick_start).
27+
- If you are not familiar with all the LDP *Stream* configuration possibilities, simply create a new one with the default options (indexing & websocket enabled, long-term storage disabled) for the purpose of this guide.
28+
- An up-and-running [OVHcloud Load Balancer](/pages/network/load_balancer/use_presentation).
29+
- Both the LDP account and the OVHcloud Load Balancer account must belong to the same OVHcloud account.
30+
31+
## Concepts & limits
32+
33+
> [!warning]
34+
> To date, the logs of the **UDP** listeners are not forwarded.
35+
>
36+
37+
**What are the logs of a OVHcloud Load Balancer?**
38+
39+
The forwarded logs are generated by [HAproxy](https://en.wikipedia.org/wiki/HAProxy){.external} (the opensource component used for load balancing).
40+
41+
### Log content for TCP based listeners : `TCP`, `HTTP`
42+
43+
| Field name | Description | Type |
44+
|------------|-------------|---------|
45+
| accept_date | The timestamp at which the request/connection was made | datetime (with millisecond resolution) e.g. 25/Mar/2024:14:07:19.536 |
46+
| bytes_read_int | The number of bytes read by the server | Integer |
47+
| bytes_uploaded_int | The number of bytes sent by the server to the client | Integer |
48+
| client_ip | IP address of the client that initiated the TCP connection to the Load Balancer | IP |
49+
| client_ip_city_name | The city computed by Geoip from `client_ip`| String (e.g. `Lille`) |
50+
| client_ip_country_code | The ISO 3166 A-2 country code computed by Logstash Geoip from `client_ip` ISO | XX (e.g. `FR`) |
51+
| client_ip_geolocation | The latitude, longitude computed by Logstash Geoip from `client_ip`| x.x,y.y (e.g. `50.624,3.0511`) XST6Y7U899O0|
52+
| client_port_int | TCP port of the client that initiated the TCP connection to the Load Balancer | Integer |
53+
| listener_id | The ID of the listener that received the request/connection | uuid |
54+
| load_balancer_id | The ID of the Load Balancer that received the request/connection | uuid |
55+
| member | The member to which the request / connection was sent | uuid |
56+
| message | The original log message | String |
57+
| pool | The pool that processed the request / connection | uuid |
58+
| project_id | The ID of the Public Cloud project to which the Load Balancer belongs | uuid |
59+
| region | The Public Cloud region to which the load balancer belongs | String |
60+
| tcp_total_session_duration_time_int | The time in milliseconds the TCP session has been opened when this request is made | Integer |
61+
| termination_state | The session termination indicator: 2 letters for TCP, 4 letters for HTTP all the details on [HAProxy documentation page](https://docs.haproxy.org/2.6/configuration.html#8.5){.external}| String (e.g. "----") |
62+
| timestamp | The timestamp at which the log has been issued | String |
63+
64+
| Field name | Description | Type |
65+
|------------|-------------|---------|
66+
| service_name | The name of the Load Balancer that received the request/connection | String |
67+
| date_time | The timestamp at which the request/connection was made | datetime (with millisecond resolution) e.g. 25/Mar/2024:14:07:19.536 |
68+
| zone | The Public Cloud region to which the load balancer belongs | String |
69+
| client_ip | IP address of the client that initiated the TCP connection to the Load Balancer | IP |
70+
| client_port_int | TCP port of the client that initiated the TCP connection to the Load Balancer | Integer |
71+
| frontend_name | Name of the frontend of your OVH
72+
| frontend_ip |
73+
| frontend_port_int |
74+
| backend_name |
75+
| backend_source_ip |
76+
| backend_source_port_int |
77+
| server_ip |
78+
| server_port_int |
79+
| termination_state | The session termination indicator: 2 letters for TCP, 4 letters for HTTP, all the details on [HAProxy documentation page](https://docs.haproxy.org/2.6/configuration.html#8.5){.external}| String (e.g. "----") |
80+
| bytes_read_int |
81+
| bytes_uploaded_int |
82+
| time_duration_int | The time in milliseconds the TCP session has been opened when this request is made | Integer |
83+
84+
### Additional content for `HTTP` & `TERMINATED_HTTPS` listeners
85+
86+
| Field name | Description | Type |
87+
|------------|-------------|---------|
88+
| captured_request_headers | The headers of the HTTP request e.g. "User-Agent" | String |
89+
| http_request | The resource of the HTTP request e.g. "/index.html" | String |
90+
| http_status_code_int | The returned HTTP status e.g. "200" | Integer |
91+
92+
93+
## Instructions
94+
95+
Note that the forwarding activation is free of charge, but you will be charged for the usage of the Logs Data Platform service as per the standard price plan. For LDP pricing, refer to this [page](https://www.ovhcloud.com/en-gb/logs-data-platform/).
96+
97+
### Enabling Load Balancer Log Forwarding using the OVHcloud Control Panel
98+
99+
This feature is not yet available in the Control Panel.
100+
101+
### Enabling Audit Log Forwarding using APIs
102+
103+
You will have to define the targeted *Stream* of one of your LDP accounts to which you want your logs forwarded. The enablement of the forwarding will create a subscription for this stream id.
104+
105+
You can retrieve the API specifications in the [OVH API Portal](https://eu.api.ovh.com/console-preview/?section=%2Fdbaas%2Flogs&branch=v1#post-/dbaas/logs/-serviceName-/output/graylog/stream).
106+
107+
#### Step 1 - Retrieve your target Stream (and ID)
108+
109+
List data streams of your Logs Data Platform account (enter your LDP ID in the form ldp-xx-xxxx into the field "serviceName"):
110+
111+
> [!api]
112+
>
113+
> @api {v1} /dbaas/logs GET /dbaas/logs/{serviceName}/output/graylog/stream
114+
>
115+
116+
Get the details of a data stream:
117+
118+
> [!api]
119+
>
120+
> @api {v1} /dbaas/logs GET /dbaas/logs/{serviceName}/output/graylog/stream/{streamId}
121+
>
122+
123+
#### Step 2 - Create your subscription
124+
125+
Use the following API call to create a subscription:
126+
127+
> [!api]
128+
>
129+
> @api {v1} /cloud POST /cloud/project/{serviceName}/region/{regionName}/loadbalancing/loadbalancer/{loadBalancerId}/log/subscription
130+
>
131+
132+
You will need to replace:
133+
134+
- **serviceName**: this is the internal name of your Load Balancer, you can find it in the Load Balancer management page in the OVHcloud Control Panel or using the [dedicated API](https://eu.api.ovh.com/console/?section=%2FipLoadbalancing&branch=v1#get-/ipLoadbalancing) call.
135+
136+
The POST request has a payload that requires:
137+
138+
- `kind`: the kind of log you want to forward, either "http" or "tcp". You can find available kinds using the [dedicated API](https://eu.api.ovh.com/console/?section=%2FipLoadbalancing&branch=v1#get-/ipLoadbalancing/-serviceName-/log/kind) call.
139+
- `streamId`: the target data stream of your LDP account where you want your Load Balancer logs to be forwarded to.
140+
141+
```shell
142+
POST /cloud/project/{serviceName}/region/{regionName}/loadbalancing/loadbalancer/{loadBalancerId}/log/subscription
143+
{
144+
"kind": "string", // Currently the only supported value is 'haproxy'.
145+
"streamId": "18d602ec-af40-4000-8e59-41ecc8c23f80" // The streamID of the targeted Stream.
146+
}
147+
```
148+
149+
You will get in response an `operationId`:
150+
151+
```shell
152+
{
153+
"operationId": "f550aa1c-89ab-4b1a-81ae-4fba4959966f",
154+
"serviceName": "ldp-xxxxx"
155+
}
156+
```
157+
158+
You can use the `operationId` to retrieve the `subscriptionId` for further management purposes using the following api call:
159+
160+
> [!api]
161+
>
162+
> @api {v1} /dbaas/logs GET /dbaas/logs/{serviceName}/operation/{operationId}
163+
>
164+
165+
Alternatively, once the operation is finished, the subscriptions can be retrieved using the following API call:
166+
167+
> [!api]
168+
>
169+
> @api {v1} /cloud GET /cloud/project/{serviceName}/region/{regionName}/loadbalancing/loadbalancer/{loadBalancerId}/log/subscription/
170+
>
171+
172+
Once you have the `subscriptionId`, you can get the details using the following API call:
173+
174+
> [!api]
175+
>
176+
> @api {v1} /cloud GET /cloud/project/{serviceName}/region/{regionName}/loadbalancing/loadbalancer/{loadBalancerId}/log/subscription/{subscriptionId}
177+
>
178+
179+
```shell
180+
GET /cloud/project/{serviceName}/region/{regionName}/loadbalancing/loadbalancer/{loadBalancerId}/log/subscription/{subscriptionId}
181+
182+
{
183+
"createdAt": "2024-01-31T15:45:25.286Z",
184+
"kind": "string",
185+
"resource": {
186+
"name": "string",
187+
"type": "string"
188+
},
189+
"serviceName": "string",
190+
"streamId": "string",
191+
"subscriptionId": "18d60324-b260-4000-83db-b484f4db6e80",
192+
"updatedAt": "2024-01-31T15:45:25.286Z"
193+
}
194+
```
195+
196+
### How to use OVHcloud Load Balancer logs?
197+
198+
Now that your logs are ingested and stored in your Logs Data Platform data stream, you can query your logs and build dashboards to have a graphical representation of your logs using the web-based UI of Graylog.
199+
200+
- In the OVHcloud Control Panel, retrieve the LDP username (ex: logs-xxxx) and its password in your Logs Data Platform account home page. You can refer to the [Quick start guide for Logs Data Platform](/pages/manage_and_operate/observability/logs_data_platform/getting_started_quick_start).
201+
- Open the Graylog web-ui. You can retrieve the link in your account home page or using your Access point depending on your account region (for example: Gravelines region is https://gra1.logs.ovh.com/).
202+
- Log into Graylog using your Logs Data Platform Username and Password.
203+
- Search through your logs across the data stream of your Logs Data Platform account. You can refer to [Graylog writing search queries](https://go2docs.graylog.org/4-x/making_sense_of_your_log_data/writing_search_queries.html){.external} documentation for details on search syntax.
204+
205+
Refer to the following documentation: [Logs Data Platform - Visualizing, querying and exploiting your logs](/products/observability-logs-data-platform-visualizing-querying-exploiting) for more details about how to use your logs with Logs Data Platform, including how to:
206+
207+
- setup alerts
208+
- view the logs in real time through a WebSocket
209+
- build visualization with OpenSearch Dashboards
210+
- integrate with OpenSearch API
211+
- connect with Grafana
212+
 
213+
### How to manage your subscriptions?
214+
215+
At any time, you can retrieve the subscriptions attached to your Logs Data Platform data stream and choose to disable the forwarding by cancelling your subscription on your stream, so that your Logs Data Platform stream no longer receives your audit logs.
216+
217+
Note that this does not delete the logs stored before the subscription was cancelled, as the data stored in a log stream is immutable unless you delete the entire stream.
218+
219+
To delete your subscription you can use the following API call:
220+
221+
> [!api]
222+
>
223+
> @api {v1} /cloud DELETE /cloud/project/{serviceName}/region/{regionName}/loadbalancing/loadbalancer/{loadBalancerId}/log/subscription/{subscriptionId}
224+
>
225+
226+
## Go further
227+
228+
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for assisting you on your specific use case of your project.
229+
230+
Join our community of users on <https://community.ovh.com/en/>.

0 commit comments

Comments
 (0)