Skip to content
This repository was archived by the owner on Sep 2, 2025. It is now read-only.

Commit 573e4cc

Browse files
aurbiztondo-splunkaurbiztondo-splunk
committed
Logs update
1 parent 6ec7538 commit 573e4cc

File tree

1 file changed

+3
-389
lines changed

1 file changed

+3
-389
lines changed

‎gdi/get-data-in/connect/aws/aws-logs.rst‎

Lines changed: 3 additions & 389 deletions
Original file line numberDiff line numberDiff line change
@@ -7,395 +7,9 @@ Collect logs from your AWS services
77
.. meta::
88
:description: Collect logs from your AWS services in Splunk.
99

10-
Log collection is not available in Splunk Observability Cloud. Instead, you can use the Data Manager to send AWS logs to Splunk Platform. Learn how at :new-page:`Onboard AWS in Data Manager <https://docs.splunk.com/Documentation/DM/1.8.2/User/AWSAbout>`. If you have a Log Observer entitlement, you must transition to Splunk Cloud Platform or Splunk Enterprise. Read more at :ref:`lo-transition`.
10+
Log collection is not available in Splunk Observability Cloud.
1111

12-
To query logs, use Log Observer Connect. See more at :ref:`logs-intro-logconnect`.
12+
Instead, you can use the Data Manager to send AWS logs to Splunk Platform. Learn how at :new-page:`Onboard AWS in Data Manager <https://docs.splunk.com/Documentation/DM/1.8.2/User/AWSAbout>`. To query logs, use Log Observer Connect. See more at :ref:`logs-intro-logconnect`.
1313

14-
.. note::
15-
16-
If you experience any issues getting logs from AWS, see :ref:`aws-ts-logs`.
17-
18-
Which services can you collect logs from?
19-
============================================
20-
21-
You can collect logs from the following services:
22-
23-
- Services storing their logs in Cloudwatch. Logs are stored in log groups which start with ``aws/<servicename>``. For example: ``aws/lambda``, ``aws/rds``, or ``aws/eks``
24-
- WAF CloudWatch logs
25-
- API Gateway execution logs
26-
- AWS Glue continuous logs if a default log group name is used
27-
- Network and Application Load Balancers access logs (classic Load Balancers not supported) from S3
28-
- S3 access logs from S3
29-
- Redshift access logs from S3
30-
- CloudFront access logs from S3
31-
32-
Limitations
33-
-----------------------------------
34-
35-
The following restrictions apply:
36-
37-
- Log sync can only be activated for a single AWS integration per AWS account. Note the integration can cover multiple services and regions.
38-
39-
- Deployment in China or Gov regions requires additional manual steps. See :new-page:`the available CloudFormation templates on GitHub <https://github.com/signalfx/aws-cloudformation-templates>`.
40-
41-
.. _aws-logs-unsupported:
42-
43-
Collect logs from unsupported services
44-
==================================================================================
45-
46-
CloudWatch log groups also store logs from unsupported services. If you want to capture those logs, add ``/aws/<namespace>`` to the list of custom namespaces in the integration object. While this option is not available in the Splunk Observability UI, you can easily do it using :ref:`API <aws-logs-api>`, or by adding :ref:`subscription filters <aws-logs-filter>`.
47-
48-
.. _aws-logs-api:
49-
50-
Collect logs using API
51-
-----------------------------------
52-
53-
To capture logs from unsupported services using the API, follow these steps:
54-
55-
1. Use a ``GET`` request to retrieve existing integration object:
56-
57-
.. code-block:: none
58-
59-
curl https://app.<realm>.signalfx.com/v2/integration/<integrationId> \
60-
-H 'x-sf-token: <user API access token>'
61-
62-
2. Update the retrieved object by adding or modifying the ``customNamespaceSyncRules`` field by executing:
63-
64-
.. code-block:: none
65-
66-
{
67-
"customNamespaceSyncRules": [
68-
{
69-
"namespace": "aws/<namespace>"
70-
}
71-
],
72-
"enabled": true,
73-
"id": "E1c1_huAAAA",
74-
}
75-
76-
- Namespaces must use lowercase only
77-
- Some fields are omitted for brevity
78-
79-
3. Use a ``PUT`` request to update your integration:
80-
81-
.. code-block:: none
82-
83-
curl https://app.<realm>.signalfx.com/v2/integration/<integrationId> \
84-
-X PUT \
85-
-X 'x-sf-token: <user API access token>' \
86-
-X 'content-type: application/json' \
87-
--data-raw '<updated integration JSON here>'
88-
89-
90-
.. _aws-logs-filter:
91-
92-
Collect logs manually with subscription filters
93-
-----------------------------------------------------------
94-
95-
Instead of sending all logs to Splunk Observability Cloud, you can forward logs only from selected CloudWatch log groups by adding a subscription filter. To add a filter, follow these steps:
96-
97-
#. Install the ``splunk-aws-logs-collector`` lambda using the :ref:`CloudFormation template <aws-cloudformation>`. Don't select ``CloudWatch Logs`` as a data type to ingest in the AWS integration.
98-
99-
#. Create a subscription filter to invoke the lambda as a CloudWatch log group subscriber for any log groups you want to forward logs from.
100-
101-
.. caution:: Do not name your filters ``Splunk Log Collector``. It's a reserved name for Splunk-managed subscriptions and they'll be removed automatically.
102-
103-
Log metadata
104-
============================
105-
106-
Log events from AWS services are enriched with relevant metadata. Some of the metadata is common :ref:`to all services <aws-logs-meta-common>`, while some other is :ref:`service-specific <aws-logs-meta-service>`.
107-
108-
.. _aws-logs-meta-common:
109-
110-
Common metadata
111-
-----------------------------------
112-
113-
.. list-table::
114-
:header-rows: 1
115-
:widths: 30, 40, 40
116-
117-
* - Field name
118-
- Description
119-
- Example
120-
121-
* - awsAccountId
122-
- The AWS Account ID of the resource that produced the logs
123-
- awsAccountId: 123456790
124-
125-
* - region
126-
- The AWS region of the resource that produced the logs
127-
- region: us-east-1
128-
129-
* - logForwarder
130-
- The name and version of ``aws-log-collector`` that sends these logs
131-
- logForwarder: splunk_aws_log_forwarder:1.0.1
132-
133-
.. _aws-logs-meta-service:
134-
135-
Service-specific metadata
136-
-----------------------------------
137-
138-
Services that store logs in CloudWatch Logs
139-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
140-
141-
.. list-table::
142-
:header-rows: 1
143-
:widths: 30, 40, 40
144-
145-
* - Field name
146-
- Description
147-
- Example
148-
149-
* - host
150-
- Same as ``logGroup``, unless overridden by service specific host
151-
- logGroup: /aws/lambda/my_function
152-
153-
* - logGroup
154-
- Source CloudWatch log group name
155-
- logGroup: /aws/lambda/my_function
156-
157-
* - logStream
158-
- Source CloudWatch log stream name
159-
- logStream: 2020/07/31/[1]e46fcdcac7094436bd846edb431a3f1
160-
161-
* - source
162-
- Service name
163-
- source: lambda
164-
165-
* - sourcetype
166-
- `aws`: prefixed service name
167-
- sourcetype: aws:lambda
168-
169-
API Gateway, ApplicationELB, CloudFront, EKS, Lambda, NetworkELB, RDS, Redshift, S3
170-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
171-
172-
.. list-table::
173-
:header-rows: 1
174-
:widths: 30, 40, 40
175-
176-
* - Field name
177-
- Description
178-
- Example
179-
180-
* - [aws_tag_name]
181-
- AWS tags associated with the resource that generated logs
182-
- name: my_func_name
183-
env: prod
184-
myCustomTag: someValue
185-
186-
API Gateway
187-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
188-
189-
.. list-table::
190-
:header-rows: 1
191-
:widths: 30, 40, 40
192-
193-
* - Field name
194-
- Description
195-
- Example
196-
197-
* - arn
198-
- API gateway ARN
199-
- arn: arn:aws:apigateway:us-east-1::/restapis/kgiqlx3nok/stages/prod
200-
201-
* - host
202-
- arn
203-
- host: arn:aws:apigateway:us-east-1::/restapis/kgiqlx3nok/stages/prod
204-
205-
* - apiGatewayStage
206-
- The API Gateway Stage name
207-
- apiGatewayStage: prod
208-
209-
* - apiGatewayId
210-
- The API Gateway ID
211-
- apiGatewayId: kgiqlx3nok
212-
213-
Application Load Balancer
214-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
215-
216-
.. list-table::
217-
:header-rows: 1
218-
:widths: 30, 40, 40
219-
220-
* - Field name
221-
- Description
222-
- Example
223-
224-
* - elbArn
225-
- Load balancer ARN
226-
- elbArn:
227-
arn:aws:elasticloadbalancing:us-east-1:1234567890:loadbalancer/app/my-loadbalancer/50dc6c495c0c9188
228-
229-
* - targetGroupArn
230-
- Target group ARN (when available)
231-
- targetGroupArn:
232-
arn:aws:elasticloadbalancing:us-east-1:1234567890:loadbalancer/app/my-loadbalancer/50dc6c495c0c9188
233-
234-
CloudFront
235-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
236-
237-
.. list-table::
238-
:header-rows: 1
239-
:widths: 30, 40, 40
240-
241-
* - Field name
242-
- Description
243-
- Example
244-
245-
* - distributionArn
246-
- CloudFront distribution ARN
247-
- distributionArn:
248-
arn:aws:cloudfront::1234567890:distribution/EMLARXS9EXAMPLE
249-
250-
EKS
251-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
252-
253-
.. list-table::
254-
:header-rows: 1
255-
:widths: 30, 40, 40
256-
257-
* - Field name
258-
- Description
259-
- Example
260-
261-
* - arn
262-
- EKS cluster ARN
263-
- arn:
264-
arn:aws:rds:us-east-1:123456790:cluster/test-eks-cluster
265-
266-
* - host
267-
- EKS cluster host
268-
- host: test-eks-cluster
269-
270-
* - eksClusterName
271-
- The EKS cluster name
272-
- eksClusterName: test-eks-cluster
273-
274-
Lambda
275-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
276-
277-
.. list-table::
278-
:header-rows: 1
279-
:widths: 30, 40, 40
280-
281-
* - Field name
282-
- Description
283-
- Example
284-
285-
* - arn
286-
- The ARN of the lambda function that generated the logs
287-
- arn:
288-
arn:aws:lambda:us-east-1:123456790:function:my_function
289-
290-
* - host
291-
- Lambda host
292-
- host: arn:aws:lambda:us-east-1:123456790:function:my_function
293-
294-
* - functionName
295-
- The name of the lambda
296-
- functionName: my_function
297-
298-
Network Load Balancer
299-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
300-
301-
.. list-table::
302-
:header-rows: 1
303-
:widths: 30, 40, 40
304-
305-
* - Field name
306-
- Description
307-
- Example
308-
309-
* - elbArn
310-
- Load balancer ARN
311-
- elbArn:
312-
arn:aws:elasticloadbalancing:us-east-1:1234567890:loadbalancer/net/my-netlb/c6e77e28c25b2234
313-
314-
RDS PostgreSQL
315-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
316-
317-
.. list-table::
318-
:header-rows: 1
319-
:widths: 30, 40, 40
320-
321-
* - Field name
322-
- Description
323-
- Example
324-
325-
* - arn
326-
- DB host ARN
327-
- arn:
328-
arn:aws:rds:us-east-1:123456790:db:druid-lab0
329-
330-
* - host
331-
- The host of RDS
332-
- host: druid-lab0
333-
334-
* - dbType
335-
- The type of DB
336-
- dbType: postgresql
337-
338-
RDS, other than PostgreSQL
339-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
340-
341-
.. list-table::
342-
:header-rows: 1
343-
:widths: 30, 40, 40
344-
345-
* - Field name
346-
- Description
347-
- Example
348-
349-
* - arn
350-
- DB host ARN
351-
- arn:
352-
arn:aws:rds:us-east-1:123456790:db:test-database-1
353-
354-
* - host
355-
- The host of RDS
356-
- host: test-database-1
357-
358-
* - dbLogName
359-
- The name of the RDS log
360-
- dbLogName: error
361-
362-
Redshift
363-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
364-
365-
.. list-table::
366-
:header-rows: 1
367-
:widths: 30, 40, 40
368-
369-
* - Field name
370-
- Description
371-
- Example
372-
373-
* - clusterArn
374-
- Redshift cluster ARN
375-
- clusterArn:
376-
arn:aws:redshift:us-east-1:1234567890:cluster:redshift-cluster-1
377-
378-
* - logType
379-
- Redshift log type. Possible: connectionlog, useractivitylog, or userlog
380-
- logType: userlog
381-
382-
S3
383-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
384-
385-
.. list-table::
386-
:header-rows: 1
387-
:widths: 30, 40, 40
388-
389-
* - Field name
390-
- Description
391-
- Example
392-
393-
* - bucketArn
394-
- S3 bucket ARN
395-
- bucketArn:
396-
arn:aws:s3:::my-bucket
397-
398-
* - objectArn
399-
- S3 object ARN (when available)
400-
- objectArn: arn:aws:s3:::my-bucket/sample.jpeg
14+
If you have a Log Observer entitlement, you must transition to Splunk Cloud Platform or Splunk Enterprise. Read more at :ref:`lo-transition`.
40115

0 commit comments

Comments
 (0)