Skip to content

Commit bcedc93

Browse files
MichaelKatsoulisMichaelKatsoulis
authored
chore: Add user_agent.original, destination.address, destination.port, url.domain to ELB access logs (#43141)
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR enhances ELB access log parsing in the `awslogsencodingextension` to include additional HTTP and network-related fields: - ALB and CLB - `user_agent.original` - Original User-Agent header value - NLB, ALB and CLB - `destination.address` - Target destination address - `destination.port ` - Target destination port - ALB and NLB - `url.domain` - Domain from the request URL These fields provide better observability for ELB access logs by capturing important HTTP request metadata that was previously missing.
1 parent 093e302 commit bcedc93

File tree

9 files changed

+245
-27
lines changed

9 files changed

+245
-27
lines changed

‎.chloggen/chore_add-fields-elb-logs.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: 'enhancement'
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
7+
component: 'extension/encoding/awslogsencodingextension'
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: 'Add user_agent.original, destination.address, destination.port, url.domain to ELB access logs'
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues: [43141]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext:
19+
20+
# If your change doesn't affect end users or the exported elements of any package,
21+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
22+
# Optional: The change log or logs in which this entry should be included.
23+
# e.g. '[user]' or '[user, api]'
24+
# Include 'user' if the change is relevant to end users.
25+
# Include 'api' if there is a change to a library API.
26+
# Default: '[user]'
27+
change_logs: []

‎extension/encoding/awslogsencodingextension/README.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -282,27 +282,27 @@ ELB access log record fields are mapped this way in the resulting OpenTelemetry
282282
> AWS Fields are according to [documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html).
283283

284284

285-
| **AWS Field** | **OpenTelemetry Field(s)** |
286-
|------------------------------|-----------------------------------------------------------------------|
287-
| type | `network.protocol.name` |
288-
| time | Log timestamp |
285+
| **AWS Field** | **OpenTelemetry Field(s)** |
286+
|------------------------------|----------------------------------------------------------------------|
287+
| type | `network.protocol.name` |
288+
| time | Log timestamp |
289289
| elb | `cloud.resource_id` |
290290
| client:port | `client.address`, `client.port` |
291291
| received_bytes | `http.request.size` |
292292
| sent_bytes | `http.response.size` |
293-
| "request" | `url.full`, `http.request.method`, `network.protocol.version` |
293+
| "request" | `url.full`, `http.request.method`, `network.protocol.version` |
294294
| ssl_cipher | `tls.cipher` |
295295
| ssl_protocol | `tls.protocol.version` |
296296
| elb_status_code | `aws.elb.status.code` |
297+
| user_agent | `user_agent.original` |
298+
| domain_name | `url.domain` |
297299
| target:port | _Currently not supported_ |
298300
| request_processing_time | _Currently not supported_ |
299301
| target_processing_time | _Currently not supported_ |
300302
| response_processing_time | _Currently not supported_ |
301303
| target_status_code | _Currently not supported_ |
302-
| "user_agent" | _Currently not supported_ |
303304
| target_group_arn | _Currently not supported_ |
304305
| "trace_id" | _Currently not supported_ |
305-
| "domain_name" | _Currently not supported_ |
306306
| "chosen_cert_arn" | _Currently not supported_ |
307307
| matched_rule_priority | _Currently not supported_ |
308308
| request_creation_time | _Currently not supported_ |
@@ -319,26 +319,26 @@ ELB access log record fields are mapped this way in the resulting OpenTelemetry
319319

320320
> AWS Fields are according to [documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html#access-log-entry-format).
321321

322-
| **AWS Field** | **OpenTelemetry Field(s)** |
322+
| **AWS Field** | **OpenTelemetry Field(s)** |
323323
|------------------------------|-------------------------------------------------------------|
324324
| type | `network.protocol.name` |
325325
| version | `network.protocol.version` |
326326
| time | Log timestamp |
327327
| elb | `cloud.resource_id` |
328-
| listener | `aws.elb.tls.listener.resource_id` |
328+
| listener | `aws.elb.tls.listener.resource_id` |
329329
| client:port | `client.address`, `client.port` |
330+
| destination:port | `destination.address`, `destination.port` |
330331
| received_bytes | `http.request.size` |
331332
| sent_bytes | `http.response.size` |
332333
| tls_cipher | `tls.cipher` |
333334
| tls_protocol_version | `tls.protocol.version` |
334-
| destination:port | _Currently not supported_ |
335+
| domain_name | `url.domain` |
335336
| connection_time | _Currently not supported_ |
336337
| tls_handshake_time | _Currently not supported_ |
337338
| incoming_tls_alert | _Currently not supported_ |
338339
| chosen_cert_arn | _Currently not supported_ |
339340
| chosen_cert_serial | _Currently not supported_ |
340341
| tls_named_group | _Currently not supported_ |
341-
| domain_name | _Currently not supported_ |
342342
| alpn_fe_protocol | _Currently not supported_ |
343343
| alpn_be_protocol | _Currently not supported_ |
344344
| alpn_client_preference_list | _Currently not supported_ |
@@ -352,16 +352,16 @@ ELB access log record fields are mapped this way in the resulting OpenTelemetry
352352
|-----------------------|--------------------------------------------------------------------------------------------|
353353
| time | Log timestamp |
354354
| elb | `cloud.resource_id` |
355-
| client:port | `client.address`, `client.port` |
355+
| client:port | `client.address`, `client.port` |
356356
| elb_status_code | `aws.elb.status.code` |
357357
| backend_status_code | `aws.elb.backend.status.code` |
358358
| received_bytes | `http.request.size` |
359359
| sent_bytes | `http.response.size` |
360360
| "request" | `url.full`, `http.request.method`, `network.protocol.name`, `network.protocol.version` |
361361
| ssl_cipher | `tls.cipher` |
362362
| ssl_protocol | `tls.protocol.version` |
363-
| backend:port | _Currently not supported_ |
363+
| user_agent | `user_agent.original` |
364+
| backend:port | _Currently not supported_ |
364365
| request_processing_time | _Currently not supported_ |
365366
| backend_processing_time | _Currently not supported_ |
366367
| response_processing_time | _Currently not supported_ |
367-
| user_agent | _Currently not supported_ |

‎extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/elb.go‎

Lines changed: 57 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import (
77
"errors"
88
"fmt"
99
"io"
10+
"net"
1011
"strconv"
1112
"strings"
1213
"time"
@@ -30,7 +31,9 @@ type CLBAccessLogRecord struct {
3031
ELB string // The name of the load balancer
3132
ClientIP string // Client IP
3233
ClientPort int64 // Client port
33-
BackendIPPort string // The IP address and port of the registered instance that processed this request, or -
34+
BackendIPPort string // Backend IP:Port or -
35+
BackendIP string // Backend IP split from BackendIPPort
36+
BackendPort int64 // Backend port split from BackendIPPort
3437
RequestProcessingTime float64 // Time taken to process the request in seconds (HTTP/TCP)
3538
BackendProcessingTime float64 // Time taken for the registered instance to respond
3639
ResponseProcessingTime float64 // Time taken to send the response to the client
@@ -59,7 +62,7 @@ func convertTextToCLBAccessLogRecord(fields []string) (CLBAccessLogRecord, error
5962
record := CLBAccessLogRecord{
6063
Time: fields[0], // Timestamp
6164
ELB: fields[1], // Load balancer name
62-
BackendIPPort: fields[3], // Backend IP:Port
65+
BackendIPPort: fields[3], // Backend IP:Port or -
6366
ELBStatusCode: 0, // Placeholder for ELB status code
6467
BackendStatusCode: 0, // Placeholder for Backend status code
6568
UserAgent: fields[12], // User-Agent
@@ -68,10 +71,25 @@ func convertTextToCLBAccessLogRecord(fields []string) (CLBAccessLogRecord, error
6871
}
6972

7073
// Process the fields for numerical values (convenient to parse from string)
71-
record.ClientIP = strings.Split(fields[2], ":")[0]
72-
if record.ClientPort, err = safeConvertStrToInt(strings.Split(fields[2], ":")[1]); err != nil {
74+
var clientPort string
75+
if record.ClientIP, clientPort, err = net.SplitHostPort(fields[2]); err != nil {
76+
return record, fmt.Errorf("could not parse client IP:Port %s: %w", fields[2], err)
77+
}
78+
if record.ClientPort, err = safeConvertStrToInt(clientPort); err != nil {
7379
return record, fmt.Errorf("could not convert client port to integer: %w", err)
7480
}
81+
82+
// Parse BackendIPPort into BackendIP and BackendPort
83+
if record.BackendIPPort != unknownField {
84+
var backendPort string
85+
if record.BackendIP, backendPort, err = net.SplitHostPort(record.BackendIPPort); err != nil {
86+
return record, fmt.Errorf("could not parse backend IP:Port %s: %w", record.BackendIPPort, err)
87+
}
88+
if record.BackendPort, err = safeConvertStrToInt(backendPort); err != nil {
89+
return record, fmt.Errorf("could not convert backend port to integer: %w", err)
90+
}
91+
}
92+
7593
if record.RequestProcessingTime, err = safeConvertStrToFloat(fields[4]); err != nil {
7694
return record, fmt.Errorf("could not convert request processing time to float: %w", err)
7795
}
@@ -121,7 +139,8 @@ type NLBAccessLogRecord struct {
121139
Listener string // Resource ID of the TLS listener for the connection
122140
ClientIP string // Client IP
123141
ClientPort int64 // Client port
124-
DestinationIPPort string // The destination IP and port of the target
142+
DestinationIP string // Destination IP
143+
DestinationPort int64 // Destination port
125144
ConnectionTime int64 // Total time for the connection to complete, in milliseconds
126145
TLSHandshakeTime int64 // Time for the TLS handshake to complete, in milliseconds, or -
127146
ReceivedBytes int64 // Count of bytes received by the load balancer from the client, after decryption
@@ -156,7 +175,6 @@ func convertTextToNLBAccessLogRecord(fields []string) (NLBAccessLogRecord, error
156175
Time: fields[2], // Timestamp
157176
ELB: fields[3], // Load balancer resource ID
158177
Listener: fields[4], // Listener ID
159-
DestinationIPPort: fields[6], // Destination IP and port
160178
TLSHandshakeTime: 0, // TLSHandshakeTime placeholder value
161179
IncomingTLSAlert: fields[11], // Incoming TLS alert
162180
ChosenCertARN: fields[12], // Chosen certificate ARN
@@ -172,11 +190,22 @@ func convertTextToNLBAccessLogRecord(fields []string) (NLBAccessLogRecord, error
172190
}
173191

174192
// Processing additional fields if applicable
175-
record.ClientIP = strings.Split(fields[5], ":")[0]
176-
if record.ClientPort, err = safeConvertStrToInt(strings.Split(fields[5], ":")[1]); err != nil {
193+
var clientPort string
194+
if record.ClientIP, clientPort, err = net.SplitHostPort(fields[5]); err != nil {
195+
return record, fmt.Errorf("could not parse client IP:Port %s: %w", fields[5], err)
196+
}
197+
if record.ClientPort, err = safeConvertStrToInt(clientPort); err != nil {
177198
return record, fmt.Errorf("could not convert client port to integer: %w", err)
178199
}
179200

201+
var destinationPort string
202+
if record.DestinationIP, destinationPort, err = net.SplitHostPort(fields[6]); err != nil {
203+
return record, fmt.Errorf("could not parse destination IP:Port %s: %w", fields[6], err)
204+
}
205+
if record.DestinationPort, err = safeConvertStrToInt(destinationPort); err != nil {
206+
return record, fmt.Errorf("could not convert destination port to integer: %w", err)
207+
}
208+
180209
if record.ConnectionTime, err = safeConvertStrToInt(fields[7]); err != nil {
181210
return record, fmt.Errorf("could not convert connection time to integer: %w", err)
182211
}
@@ -205,7 +234,9 @@ type ALBAccessLogRecord struct {
205234
ELB string // Load balancer resource ID
206235
ClientIP string // Client IP
207236
ClientPort int64 // Client port
208-
TargetIPPort string // Target IP and port
237+
TargetIPPort string // Target IP:Port or -
238+
TargetIP string // Target IP
239+
TargetPort int64 // Target port
209240
RequestProcessingTime string // Time taken to process the request in seconds
210241
TargetProcessingTime string // Time taken for the target to process the request in seconds
211242
ResponseProcessingTime string // Time taken to send the response to the client in seconds
@@ -269,10 +300,25 @@ func convertTextToALBAccessLogRecord(fields []string) (ALBAccessLogRecord, error
269300
Classification: fields[27],
270301
ClassificationReason: fields[28],
271302
}
272-
record.ClientIP = strings.Split(fields[3], ":")[0]
273-
if record.ClientPort, err = safeConvertStrToInt(strings.Split(fields[3], ":")[1]); err != nil {
303+
var clientPort string
304+
if record.ClientIP, clientPort, err = net.SplitHostPort(fields[3]); err != nil {
305+
return record, fmt.Errorf("could not parse client IP:Port %s: %w", fields[3], err)
306+
}
307+
if record.ClientPort, err = safeConvertStrToInt(clientPort); err != nil {
274308
return record, fmt.Errorf("could not convert client port to integer: %w", err)
275309
}
310+
311+
// Parse TargetIPPort into TargetIP and TargetPort
312+
if record.TargetIPPort != unknownField {
313+
var targetPort string
314+
if record.TargetIP, targetPort, err = net.SplitHostPort(record.TargetIPPort); err != nil {
315+
return record, fmt.Errorf("could not parse target IP:Port %s: %w", record.TargetIPPort, err)
316+
}
317+
if record.TargetPort, err = safeConvertStrToInt(targetPort); err != nil {
318+
return record, fmt.Errorf("could not convert target port to integer: %w", err)
319+
}
320+
}
321+
276322
if record.ELBStatusCode, err = safeConvertStrToInt(fields[8]); err != nil {
277323
return record, fmt.Errorf("could not convert elb status code to integer: %w", err)
278324
}

‎extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/testdata/alb_al_valid_logs.log‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
1-
https 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 10.0.0.1:80 0.086 0.048 0.037 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.46.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337281-1d84f3d73c47ec4e58577259" "www.example.com" "arn:aws:acm:us-east-2:123456789012:certificate/12345678-1234-1234-1234-123456789012" 1 2018-07-02T22:22:48.364000Z "authenticate,forward" "-" "-" "10.0.0.1:80" "200" "-" "-" TID_1234abcd5678ef90
1+
https 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 10.0.0.1:80 0.086 0.048 0.037 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.46.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337281-1d84f3d73c47ec4e58577259" "www.example.com" "arn:aws:acm:us-east-2:123456789012:certificate/12345678-1234-1234-1234-123456789012" 1 2018-07-02T22:22:48.364000Z "authenticate,forward" "-" "-" "10.0.0.1:80" "200" "-" "-" TID_1234abcd5678ef90
2+
https 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 [fe80::202:b3ff:fe1e:8329]:443 [2001:db8::1]:80 0.086 0.048 0.037 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.46.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337281-1d84f3d73c47ec4e58577259" "www.example.com" "arn:aws:acm:us-east-2:123456789012:certificate/12345678-1234-1234-1234-123456789012" 1 2018-07-02T22:22:48.364000Z "authenticate,forward" "-" "-" "10.0.0.1:80" "200" "-" "-" TID_1234abcd5678ef90

0 commit comments

Comments
 (0)