Skip to content

Commit 354c547

Browse files
jbachorikclaude
andcommitted
feat(profiling): Add OTLP profiles core infrastructure
Add profiling-otel module with core infrastructure for JFR to OTLP profiles conversion: - Dictionary tables for OTLP compression (StringTable, FunctionTable, LocationTable, StackTable, LinkTable, AttributeTable) - ProtobufEncoder for hand-coded protobuf wire format encoding - OtlpProtoFields constants for OTLP profiles proto field numbers - Unit tests for all dictionary tables and encoder - Architecture documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 17c7fcf commit 354c547

File tree

16 files changed

+2186
-0
lines changed

16 files changed

+2186
-0
lines changed
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
plugins {
2+
`java-library`
3+
}
4+
5+
apply(from = "$rootDir/gradle/java.gradle")
6+
7+
dependencies {
8+
implementation("io.btrace", "jafar-parser", "0.0.1-SNAPSHOT")
9+
implementation(project(":internal-api"))
10+
11+
testImplementation(libs.bundles.junit5)
12+
testImplementation(libs.bundles.jmc)
13+
}
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# OTLP Profiles Writer - Architecture & Implementation Journal
2+
3+
## Overview
4+
5+
This module provides a JFR to OTLP/profiles format converter. It reads JFR recordings via the `RecordingData` abstraction and produces OTLP-compliant profile data in both binary protobuf and JSON formats.
6+
7+
## OTLP Profiles Format
8+
9+
Based on: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/profiles/v1development/profiles.proto
10+
11+
### Key Architectural Concepts
12+
13+
1. **Dictionary-based Compression**: OTLP profiles use shared dictionary tables to minimize wire size. All repeated data (strings, functions, locations, stacks, links, attributes) is stored once in dictionary tables and referenced by integer indices.
14+
15+
2. **Index 0 Semantics**: In all dictionary tables, index 0 is reserved for "null/unset" values. Index 0 should never be dereferenced - it represents the absence of a value.
16+
17+
3. **Sample Identity**: A sample's identity is the tuple `{stack_index, set_of(attribute_indices), link_index}`. Samples with the same identity should be aggregated.
18+
19+
### Message Hierarchy
20+
21+
```
22+
ProfilesData
23+
├── dictionary: ProfilesDictionary (shared across all profiles)
24+
│ ├── string_table[] - interned strings
25+
│ ├── function_table[] - function metadata
26+
│ ├── location_table[] - stack frame locations
27+
│ ├── mapping_table[] - binary/library mappings
28+
│ ├── stack_table[] - call stacks (arrays of location indices)
29+
│ ├── link_table[] - trace context links
30+
│ └── attribute_table[] - key-value attributes
31+
32+
└── resource_profiles[]
33+
└── scope_profiles[]
34+
└── profiles[]
35+
├── sample_type: ValueType
36+
├── period_type: ValueType
37+
├── samples[]
38+
│ ├── stack_index -> stack_table
39+
│ ├── attribute_indices[] -> attribute_table
40+
│ ├── link_index -> link_table
41+
│ ├── values[]
42+
│ └── timestamps_unix_nano[]
43+
└── time_unix_nano, duration_nano, profile_id, etc.
44+
```
45+
46+
## Package Structure
47+
48+
```
49+
com.datadog.profiling.otel/
50+
├── dictionary/ # Dictionary table implementations
51+
│ ├── StringTable # String interning
52+
│ ├── FunctionTable # Function deduplication
53+
│ ├── LocationTable # Stack frame deduplication
54+
│ ├── StackTable # Call stack deduplication
55+
│ ├── LinkTable # Trace link deduplication
56+
│ └── AttributeTable # Attribute deduplication
57+
58+
├── proto/ # Protobuf encoding
59+
│ ├── ProtobufEncoder # Wire format encoder
60+
│ └── OtlpProtoFields # Field number constants
61+
62+
└── (future: converter, writer classes)
63+
```
64+
65+
## JFR Event to OTLP Mapping
66+
67+
| JFR Event Type | OTLP Profile Type | Value Type | Unit |
68+
|----------------|-------------------|------------|------|
69+
| `datadog.ExecutionSample` | cpu | count | samples |
70+
| `datadog.MethodSample` | wall | count | samples |
71+
| `datadog.ObjectSample` | alloc-samples | bytes | bytes |
72+
| `jdk.JavaMonitorEnter` | lock-contention | duration | nanoseconds |
73+
| `jdk.JavaMonitorWait` | lock-contention | duration | nanoseconds |
74+
75+
## Implementation Details
76+
77+
### Phase 1: Core Infrastructure (Completed)
78+
79+
#### Dictionary Tables
80+
81+
All dictionary tables follow a common pattern:
82+
- Index 0 reserved for null/unset (pre-populated in constructor)
83+
- `intern()` method returns existing index or adds new entry
84+
- `get()` method retrieves entry by index
85+
- `reset()` method clears table to initial state
86+
- HashMap-based deduplication for O(1) lookup
87+
88+
**StringTable**: Simple string interning. Null and empty strings map to index 0.
89+
90+
**FunctionTable**: Functions identified by composite key (nameIndex, systemNameIndex, filenameIndex, startLine). All indices reference StringTable.
91+
92+
**LocationTable**: Locations represent stack frames. Key is (mappingIndex, address, functionIndex, line, column). Supports multiple Line entries for inlined functions.
93+
94+
**StackTable**: Stacks are arrays of location indices. Uses Arrays.hashCode/equals for array-based key comparison. Makes defensive copies of input arrays.
95+
96+
**LinkTable**: Links connect samples to trace spans. Stores 16-byte traceId and 8-byte spanId. Provides convenience method for 64-bit DD trace/span IDs.
97+
98+
**AttributeTable**: Supports STRING, BOOL, INT, DOUBLE value types. Key includes (keyIndex, valueType, value, unitIndex).
99+
100+
#### ProtobufEncoder
101+
102+
Hand-coded protobuf wire format encoder without external dependencies:
103+
104+
- **Wire Types**: VARINT (0), FIXED64 (1), LENGTH_DELIMITED (2), FIXED32 (5)
105+
- **Varint Encoding**: Variable-length integers, 7 bits per byte, MSB indicates continuation
106+
- **ZigZag Encoding**: For signed varints, maps negative numbers to positive
107+
- **Fixed Encoding**: Little-endian for fixed32/fixed64
108+
- **Length-Delimited**: Length prefix (varint) followed by content
109+
- **Nested Messages**: Written to temporary buffer to compute length first
110+
111+
Key methods:
112+
- `writeVarint()`, `writeFixed64()`, `writeFixed32()`
113+
- `writeTag()` - combines field number and wire type
114+
- `writeString()`, `writeBytes()` - length-delimited
115+
- `writeNestedMessage()` - for sub-messages
116+
- `writePackedVarintField()`, `writePackedFixed64Field()` - for repeated fields
117+
118+
#### OtlpProtoFields
119+
120+
Constants for all OTLP protobuf field numbers, organized by message type. Enables type-safe field references without magic numbers.
121+
122+
### Phase 2: JFR Parsing & CPU Profile (In Progress)
123+
124+
(To be documented as implementation progresses)
125+
126+
## Testing Strategy
127+
128+
- **Unit Tests**: Each dictionary table and encoder method tested independently
129+
- **Integration Tests**: End-to-end conversion with JMC JFR Writer API for creating test recordings
130+
- **Round-trip Validation**: Verify protobuf output can be parsed correctly
131+
132+
## Dependencies
133+
134+
- `jafar-parser` - JFR parsing library
135+
- `internal-api` - RecordingData abstraction
136+
- `libs.bundles.jmc` - JMC libraries for test JFR creation (test scope)
137+
- `libs.bundles.junit5` - Testing framework (test scope)
Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
package com.datadog.profiling.otel.dictionary;
2+
3+
import java.util.ArrayList;
4+
import java.util.HashMap;
5+
import java.util.List;
6+
import java.util.Map;
7+
import java.util.Objects;
8+
9+
/**
10+
* Attribute deduplication table for OTLP profiles. Index 0 is reserved for the null/unset
11+
* attribute. Attributes are key-value pairs with optional unit.
12+
*/
13+
public final class AttributeTable {
14+
15+
/** Attribute value types. */
16+
public enum ValueType {
17+
STRING,
18+
BOOL,
19+
INT,
20+
DOUBLE
21+
}
22+
23+
/** Immutable key for attribute lookup. */
24+
private static final class AttributeKey {
25+
final int keyIndex;
26+
final ValueType valueType;
27+
final Object value;
28+
final int unitIndex;
29+
30+
AttributeKey(int keyIndex, ValueType valueType, Object value, int unitIndex) {
31+
this.keyIndex = keyIndex;
32+
this.valueType = valueType;
33+
this.value = value;
34+
this.unitIndex = unitIndex;
35+
}
36+
37+
@Override
38+
public boolean equals(Object o) {
39+
if (this == o) return true;
40+
if (o == null || getClass() != o.getClass()) return false;
41+
AttributeKey that = (AttributeKey) o;
42+
return keyIndex == that.keyIndex
43+
&& unitIndex == that.unitIndex
44+
&& valueType == that.valueType
45+
&& Objects.equals(value, that.value);
46+
}
47+
48+
@Override
49+
public int hashCode() {
50+
return Objects.hash(keyIndex, valueType, value, unitIndex);
51+
}
52+
}
53+
54+
/** Attribute entry stored in the table. */
55+
public static final class AttributeEntry {
56+
public final int keyIndex;
57+
public final ValueType valueType;
58+
public final Object value;
59+
public final int unitIndex;
60+
61+
AttributeEntry(int keyIndex, ValueType valueType, Object value, int unitIndex) {
62+
this.keyIndex = keyIndex;
63+
this.valueType = valueType;
64+
this.value = value;
65+
this.unitIndex = unitIndex;
66+
}
67+
68+
public String getStringValue() {
69+
return valueType == ValueType.STRING ? (String) value : null;
70+
}
71+
72+
public Boolean getBoolValue() {
73+
return valueType == ValueType.BOOL ? (Boolean) value : null;
74+
}
75+
76+
public Long getIntValue() {
77+
return valueType == ValueType.INT ? (Long) value : null;
78+
}
79+
80+
public Double getDoubleValue() {
81+
return valueType == ValueType.DOUBLE ? (Double) value : null;
82+
}
83+
}
84+
85+
private final List<AttributeEntry> attributes;
86+
private final Map<AttributeKey, Integer> attributeToIndex;
87+
88+
public AttributeTable() {
89+
attributes = new ArrayList<>();
90+
attributeToIndex = new HashMap<>();
91+
// Index 0 is reserved for null/unset attribute
92+
attributes.add(new AttributeEntry(0, ValueType.STRING, "", 0));
93+
}
94+
95+
/**
96+
* Interns a string attribute and returns its index.
97+
*
98+
* @param keyIndex index into string table for attribute key
99+
* @param value string value
100+
* @param unitIndex index into string table for unit (0 = no unit)
101+
* @return the index of the interned attribute
102+
*/
103+
public int internString(int keyIndex, String value, int unitIndex) {
104+
if (keyIndex == 0) {
105+
return 0;
106+
}
107+
return intern(keyIndex, ValueType.STRING, value, unitIndex);
108+
}
109+
110+
/**
111+
* Interns a boolean attribute and returns its index.
112+
*
113+
* @param keyIndex index into string table for attribute key
114+
* @param value boolean value
115+
* @param unitIndex index into string table for unit (0 = no unit)
116+
* @return the index of the interned attribute
117+
*/
118+
public int internBool(int keyIndex, boolean value, int unitIndex) {
119+
if (keyIndex == 0) {
120+
return 0;
121+
}
122+
return intern(keyIndex, ValueType.BOOL, value, unitIndex);
123+
}
124+
125+
/**
126+
* Interns an integer attribute and returns its index.
127+
*
128+
* @param keyIndex index into string table for attribute key
129+
* @param value integer value
130+
* @param unitIndex index into string table for unit (0 = no unit)
131+
* @return the index of the interned attribute
132+
*/
133+
public int internInt(int keyIndex, long value, int unitIndex) {
134+
if (keyIndex == 0) {
135+
return 0;
136+
}
137+
return intern(keyIndex, ValueType.INT, value, unitIndex);
138+
}
139+
140+
/**
141+
* Interns a double attribute and returns its index.
142+
*
143+
* @param keyIndex index into string table for attribute key
144+
* @param value double value
145+
* @param unitIndex index into string table for unit (0 = no unit)
146+
* @return the index of the interned attribute
147+
*/
148+
public int internDouble(int keyIndex, double value, int unitIndex) {
149+
if (keyIndex == 0) {
150+
return 0;
151+
}
152+
return intern(keyIndex, ValueType.DOUBLE, value, unitIndex);
153+
}
154+
155+
private int intern(int keyIndex, ValueType valueType, Object value, int unitIndex) {
156+
AttributeKey key = new AttributeKey(keyIndex, valueType, value, unitIndex);
157+
Integer existing = attributeToIndex.get(key);
158+
if (existing != null) {
159+
return existing;
160+
}
161+
162+
int index = attributes.size();
163+
attributes.add(new AttributeEntry(keyIndex, valueType, value, unitIndex));
164+
attributeToIndex.put(key, index);
165+
return index;
166+
}
167+
168+
/**
169+
* Returns the attribute entry at the given index.
170+
*
171+
* @param index the index
172+
* @return the attribute entry
173+
* @throws IndexOutOfBoundsException if index is out of bounds
174+
*/
175+
public AttributeEntry get(int index) {
176+
return attributes.get(index);
177+
}
178+
179+
/**
180+
* Returns the number of attributes (including the null attribute at index 0).
181+
*
182+
* @return the size of the attribute table
183+
*/
184+
public int size() {
185+
return attributes.size();
186+
}
187+
188+
/**
189+
* Returns the list of all attribute entries.
190+
*
191+
* @return the list of attribute entries
192+
*/
193+
public List<AttributeEntry> getAttributes() {
194+
return attributes;
195+
}
196+
197+
/** Resets the table to its initial state with only the null attribute at index 0. */
198+
public void reset() {
199+
attributes.clear();
200+
attributeToIndex.clear();
201+
attributes.add(new AttributeEntry(0, ValueType.STRING, "", 0));
202+
}
203+
}

0 commit comments

Comments
 (0)