Skip to content
This repository was archived by the owner on Sep 2, 2025. It is now read-only.

Commit d1d7bd3

Browse files
mbechtold-splunkmbechtold-splunk
authored
Merge pull request #1737 from splunk/mbechtold-6073-zc-advanced-config
[6073] Advanced configuration doc for zero config
2 parents 8a1403c + c23ddfa commit d1d7bd3

File tree

4 files changed

+255
-2
lines changed

4 files changed

+255
-2
lines changed

‎gdi/opentelemetry/auto-instrumentation/advanced-config-auto-instrumentation.rst‎

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
.. _advanced-config-auto-instrumentation:
2+
3+
*************************************************************************
4+
Advanced customization for zero config auto instrumentation
5+
*************************************************************************
6+
7+
.. meta::
8+
:description: Learn how to customize Splunk zero config auto instrumentation for advanced scenarios.
9+
10+
Learn how to customize Splunk zero config auto instrumentation for advanced scenarios.
11+
12+
Through advanced customization, you can achieve the following tasks:
13+
14+
* :ref:`Change instrumentation version <change-zeroconfig-version>`
15+
* :ref:`Override auto instrumentation settings <override-zeroconfig-settings>`
16+
* :ref:`Use auto instrumentation with gateway mode <deploy-in-gateway>`
17+
18+
.. _change-zeroconfig-version:
19+
20+
Change instrumentation versions (Kubernetes only)
21+
====================================================
22+
23+
By default, the Splunk Distribution of OpenTelemetry Collector uses the latest version of the related instrumentation library. To change the version of your instrumentation library, follow these steps:
24+
25+
#. Open your ``values.yaml`` file.
26+
#. In the instrumentation language section, change the ``tag`` value to match your desired version. The following example changes the Java instrumentation version to ``v1.27.0``.
27+
28+
.. code-block:: yaml
29+
:emphasize-lines: 14
30+
31+
clusterName: myCluster
32+
splunkObservability:
33+
realm: <splunk-realm>
34+
accessToken: <splunk-access-token>
35+
environment: prd
36+
certmanager:
37+
enabled: true
38+
operator:
39+
enabled: true
40+
instrumentation:
41+
spec:
42+
java:
43+
repository: ghcr.io/signalfx/splunk-otel-java/splunk-otel-java
44+
tag: v1.27.0
45+
46+
#. Reinstall the Splunk OTel Collector Chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.
47+
48+
.. code-block:: bash
49+
50+
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml
51+
52+
.. note:: If you don't see the ``java`` field in your ``values.yaml`` file, then you have to add the field and the ``repository`` value before changing the version. The repository value is always ``ghcr.io/signalfx/splunk-otel-java/splunk-otel-java``.
53+
54+
See the following pages for information about previous versions for each language:
55+
56+
* Java: :ref:`java-otel-requirements`
57+
* Node.js: :ref:`nodejs-otel-requirements`
58+
* .NET: :ref:`dotnet-otel-requirements`
59+
60+
.. _override-zeroconfig-settings:
61+
62+
Override auto instrumentation settings
63+
====================================================
64+
65+
You can override default auto instrumentation settings to use features for profiling and runtime metrics collection.
66+
67+
Activate AlwaysOn Profiling
68+
----------------------------------------------------
69+
70+
You can activate CPU and memory profiling by updating the environment variables for your instrumentation. AlwaysOn Profiling continuously collects stack traces so that you can discover which lines of code are slowing your processes down. To learn more about Profiling, see :ref:`profiling-intro`.
71+
72+
.. tabs::
73+
74+
.. tab:: Linux
75+
76+
To activate profiling globally, add the ``--enable-profiler`` flag upon installation for CPU profiling, or ``--enable-profiler-memory`` flag for memory profiling. For example:
77+
78+
.. code-block:: bash
79+
:emphasize-lines: 4
80+
81+
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
82+
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
83+
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
84+
--enable-profiler --enable-profiler-memory
85+
86+
You can also activate profiling for individual languages. By using this approach, you can determine which languages profiling gathers call stacks from.
87+
88+
Follow these steps to activate AlwaysOn Profiling for an individual language:
89+
90+
#. Open the <language>.conf file located in the ``/etc/splunk/zeroconfig`` directory.
91+
#. Set the environment variable ``SPLUNK_PROFILER_ENABLED=true`` for CPU profiling, and ``SPLUNK_PROFILER_MEMORY_ENABLED=true`` for memory profiling.
92+
#. Restart your applications.
93+
94+
.. note:: If you're using ``systemd``, the environment variables are instead located in ``/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf``.
95+
96+
To sample call stacks from a specific interval, change the ``SPLUNK_PROFILER_CALL_STACK_INTERVAL`` setting to your desired interval in milliseconds. The default value is ``10000``.
97+
98+
For example, ``SPLUNK_PROFILER_CALL_STACK_INTERVAL=5000`` sets the call stack interval to 5000 milliseconds.
99+
100+
.. tab:: Kubernetes
101+
102+
You can configure AlwaysOn Profiling in Kubernetes by editing the values.yaml file for the Helm Chart.
103+
104+
Follow these steps to activate Profiling for a language:
105+
106+
#. Open the values.yaml file.
107+
#. In the ``operator.instrumentation.spec.<language>.env`` section, add the ``SPLUNK_PROFILER_ENABLED=true`` environment variable and the ``SPLUNK_PROFILER_CALL_STACK_INTERVAL`` environment variable.
108+
109+
For example, the following values.yaml file configures AlwaysOn Profiling to sample call stacks from a 5000 millisecond interval:
110+
111+
.. code-block:: yaml
112+
113+
operator:
114+
enabled: true
115+
instrumentation:
116+
spec:
117+
nodejs:
118+
env:
119+
# Activates AlwaysOn Profiling for Node.js
120+
- name: SPLUNK_PROFILER_ENABLED
121+
value: true
122+
# Samples call stacks from a 5000 millisecond interval.
123+
# If excluded, samples from a 10000 millisecond interval by default.
124+
- name: SPLUNK_PROFILER_CALL_STACK_INTERVAL
125+
value: 5000
126+
127+
#. Reinstall the Splunk OTel Collector Chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.
128+
129+
.. code-block:: bash
130+
131+
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml
132+
133+
Activate runtime metrics collection (Linux only)
134+
----------------------------------------------------
135+
136+
You can activate runtime metrics collection by updating the environment variables for your instrumentation. This setting configures the agent to collect additional metrics from your application.
137+
138+
To activate runtime metrics globally, add the ``--enable-metrics`` flag upon installation. For example:
139+
140+
.. code-block:: bash
141+
:emphasize-lines: 4
142+
143+
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
144+
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
145+
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
146+
--enable-metrics
147+
148+
You can also activate runtime metrics collection for individual languages. By using this approach, you can determine which runtime metrics the Collector sends at a language-level.
149+
150+
To activate runtime metrics collection for an individual language, follow these steps:
151+
152+
#. Open the <language>.conf file located in the ``/etc/splunk/zeroconfig`` directory.
153+
#. Set the environment variable ``SPLUNK_METRICS_ENABLED=true``.
154+
#. Restart your applications.
155+
156+
.. _deploy-in-gateway:
157+
158+
Use auto instrumentation with gateway mode
159+
===========================================================
160+
161+
The Splunk OTel Collector Chart uses the agent mode by default. Activating gateway mode deploys an instance of the OpenTelemetry Collector in a separate container, and this instance collects data from the entire cluster.
162+
163+
To learn more about the gateway mode, see :ref:`collector-gateway-mode`.
164+
165+
.. tabs::
166+
167+
.. tab:: Linux
168+
169+
You can't directly deploy auto instrumentation in gateway mode for Linux, but you can send collected data from auto instrumentation to an existing gateway deployment.
170+
171+
Follow these steps to send data to a gateway deployment of the OpenTelemetry Collector:
172+
173+
#. Open the <language>.conf file located in ``/etc/splunk/zeroconfig``.
174+
#. Set the environment variable ``OTEL_EXPORTER_OTLP_ENDPOINT=<gateway_endpoint>`` where <gateway_endpoint> is the port of your gateway deployment.
175+
#. Restart your applications.
176+
177+
The auto instrumentation now sends data to your gateway deployment.
178+
179+
.. tab:: Kubernetes
180+
181+
You can send data to an endpoint of a Collector instance running in gateway mode.
182+
183+
Follow these steps to send data to a gateway endpoint:
184+
185+
#. Open the values.yaml file.
186+
#. Set the ``operator.instrumentation.spec.exporter.endpoint`` value to the gateway endpoint. For example:
187+
188+
.. code-block:: yaml
189+
:emphasize-lines: 13
190+
191+
clusterName: myCluster
192+
splunkObservability:
193+
realm: <splunk-realm>
194+
accessToken: <splunk-access-token>
195+
environment: prd
196+
certmanager:
197+
enabled: true
198+
operator:
199+
enabled: true
200+
instrumentation:
201+
spec:
202+
exporter:
203+
endpoint: <gateway-endpoint>
204+
205+
#. Reinstall the Splunk OTel Collector Chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.
206+
207+
.. code-block:: bash
208+
209+
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml
210+
211+
You can also automatically send data to a deployed gateway endpoint by setting ``agent.enabled: false`` and ``gateway.enabled: true`` in your values.yaml file. For example:
212+
213+
.. code-block:: yaml
214+
:emphasize-lines: 12, 14
215+
216+
clusterName: myCluster
217+
splunkObservability:
218+
realm: <splunk-realm>
219+
accessToken: <splunk-access-token>
220+
environment: prd
221+
certmanager:
222+
enabled: true
223+
operator:
224+
enabled: true
225+
226+
agent:
227+
enabled: false
228+
gateway:
229+
enabled: true
230+
231+
Using this configuration, auto instrumentation automatically sends data to a running gateway endpoint.
232+
233+
234+
Additional settings
235+
===================================
236+
237+
There are many other settings you can customize in zero configuration auto instrumentation.
238+
239+
For a list of settings that you can change for each language, see the following resources:
240+
241+
.. list-table::
242+
:header-rows: 1
243+
:width: 100
244+
245+
* - Language
246+
- Resource
247+
* - Java
248+
- :ref:`advanced-java-otel-configuration`
249+
* - Node.js
250+
- :ref:`advanced-nodejs-otel-configuration`
251+
252+
For a complete list of settings that you can modify in values.yaml for Kubernetes, see :new-page:`https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/values.yaml`.

‎gdi/opentelemetry/auto-instrumentation/auto-instrumentation-java-k8s.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -340,7 +340,7 @@ You can also use the methods shown in :ref:`zeroconfig-java-traces` to configure
340340
341341
kubectl set env deployment/<my-deployment> OTEL_RESOURCE_ATTRIBUTES=build.id=feb2023_v2
342342
343-
See :ref:`advanced-java-otel-configuration` for the full list of supported environment variables.
343+
See :ref:`advanced-config-auto-instrumentation` for more information.
344344

345345
.. _troubleshooting-zeroconfig-java-k8s:
346346

‎gdi/opentelemetry/auto-instrumentation/auto-instrumentation-nodejs-k8s.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -343,7 +343,7 @@ For example, if you want every span to include the key-value pair ``build.id=feb
343343
344344
You can also use the methods shown in :ref:`zeroconfig-nodejs-traces` to configure your instrumentation with the ``OTEL_RESOURCE_ATTRIBUTES`` environment variable and other environment variables.
345345

346-
See :ref:`advanced-nodejs-otel-configuration` for the full list of supported environment variables.
346+
See :ref:`advanced-config-auto-instrumentation` for more information.
347347

348348
.. _troubleshooting-zeroconfig-nodejs-k8s:
349349

‎gdi/opentelemetry/zero-config.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Splunk OpenTelemetry Zero Configuration Auto Instrumentation
1515
Java <auto-instrumentation/auto-instrumentation-java>
1616
.NET <auto-instrumentation/auto-instrumentation-dotnet>
1717
Node.js <auto-instrumentation/auto-instrumentation-nodejs>
18+
Advanced customization <auto-instrumentation/advanced-config-auto-instrumentation>
1819

1920
Splunk OpenTelemetry Zero Configuration Auto Instrumentation automatically instruments your back-end applications and services to capture and report distributed traces and metrics to the Splunk Distribution of OpenTelemetry Collector, and then on to Splunk APM.
2021

0 commit comments

Comments
 (0)