JFR Context Carrier Event

[jbachorik]

Quick links

• Contextual Event Prototype Branch

---

Rationale

It is proved to be beneficial to be able to connect JFR events with custom context. What the context actually is depends on the application or library and is usually quite domain specific.

From the point of view of a distributed tracer provider the context could be a combination of trace id and span id. But it as well may be a REST endpoint or SQL call, or a subsystem ID, or really anything that can help making sense out of the huge volumes of data we are collecting nowadays.

Design

While the most straightforward design is to use something similar to Go labelling, where an arbitrary list of key-value tuples representing the context is attached to each thread it might be possible to achieve the similar effect using more JFR-like approach.

@egahlin has provided a good writeup in JDK-8325465

Minimal implementation

The writeup is opening up a possibility of more enhanced event filtering but to keep things simple, let's start with a very minimal implementation.

There should be a possibility to annotate a JFR event type as 'context carrier' - all fields defined for that event will be considered as the context. Because context can be 'set' for very short periods this could lead to swamping the recording with context events without any additional value - they would not be associated with any 'regular' events. In order to prevent this swamping the context carrier events should emitted only if there is at least one other event committed between the calls to begin() and end() of the context carrier event.

This implementation will require introducing a new annotation to designate the context carrier event type.
The proposed annotation name @ContextCarrier and the annotation will be in jdk.jfr.experimental packaget until it is stable enough that it can be considered to be promoted to public API. The jdk.jfr.experimental package will not be exported and as such, in order to use this new annotation the user will have to make sure the proper visibility and reachability by eg. command line arguments.

Context stack

The context is attached to a particular thread. This also means that there might be more 'contexts' set for a thread - these contexts will form a stack and will live in the JFR thread local instance, already available from a JavaThread.

For the minimal implementation, the context will be represented by a tuple of the context carrier event type id and a boolean flag determining whether another event was committed while the context was active. Let's call this tuple the 'context frame'.

Therefore, a call to begin() for an event marked @ContextCarrier annotation will effectively push the context frame on stack and, similarly, a call to end() will pop the context frame from the stack.

The assumption is that the context stack depth will be rather small, but we might add some safety bars to eg. stop tracking context when reaching certain depth.

Emitting context

The context stack will be integrated with the event writers - both, the Java event writer and the native writer. When an event is written, the context stack will be inspected and if not empty, the top frame will be modified such that boolean part is true, meaning the context is with events.

There are a few native event types which are related to other thread than the one on which they are being written (eg. ExecutionSample etc.) and these need to handled separately such that the proper thread context stack is used.

When the end() method is called for the context carrier event, it will pop the stack frame and store the value of the 'with events' boolean in an implicit event field. This field will then be checked in shouldCommit() method, effectively preventing committing the context event if it was not with events.

Evolvability

The proposed design and implementation leaves the doors open for the further evolution.

Making certain event types context aware and not emit when there is no context present is quite trivial - these events would just need to have their shouldCommit() method to check for the context stack not being empty.

Similarly, the context carrier events could be made to respond only to certain event types by defining an event type ids to check against when setting the 'with events' flag.

However, I would propose postponing the implementation of the more advanced filterings after we have the initial implementation done - the additional flexibility also means additional perf hit and the benefit/cost ratio must be evaluated.

[egahlin]

If it's more straightforward with "something similar to Go labelling", go ahead, but such a mechanism should not be tied to JFR, but be an API that third-party libraries and applications can use to provide context on JVMs and JDK images that don't support JFR. Labels should be accessible through means other than JFR, for example, in logs or in a debugger. Without this capability, it is unlikely that the API will be widely adopted. It would just be an unnecessary large API surface to maintain.

Since its inception, the JFR approach has been to provide contexts using events.

Events that occur in the same thread between the start and end of a context event are associated with that context. Such events are already in use today, but what we have lacked is means to mark those events as special so that tools like JMC or the 'jfr tool' can offer contextual visualization. For example, this could include a context view where events are organized per context, or displaying context values in an event tooltip or a property page. Currently, there is the @Relational annotation, which provides information that two events are related.

As a part of a "@Contextual" annotation, there could be filtering capabilities to reduce the recording size and CPU usage. This filtering would not come as a Java API, but be implemented using the existing event settings system. It can be configured via command line, on a running application using jcmd, programmatically from Java, or remotely using the FlightRecorderMXBean. Three use cases have been identified where filtering could be beneficial 1) throttling, 2) thresholding, and 3) triggering.

[jbachorik]

Ok, I am confused now.

I thought that the whole purpose of the @Contextual was to mark event types for special handling when the event would be committed only if other events happened on the same thread between begin() and end() of the contextual event were called.

Here is an example how I understood the concept was supposed to work (and I am leaving out the throttling part because it opens up a number of other challenges, eg. we don't have a truly generic throttling implementation readily available as of now).

MyContextEvent ctx1 = new MyContextEvent(1234);
ctx1.begin();
new MySimpleEvent().commit();
ctx1.end();
ctx1.commit(); // ctx1 will get committed here as there are events 'enveloped' by the context event

MyContextEvent ctx2 = new MyContextEvent(4321);
ctx2.begin();
ctx2.end();
ctx2.commit(); // ctx2 will not get committed because there are no 'enveloped' events

For a @Contextual event, what effect would if-context have? That the event would be committable only if there is another contextual event in progress on the same thread? What would be the use case for this?

As for the selector - from what I read it looked like the selector would have been used to make a non-contextual event sensitive to context. Especially with select=if-context which would mean that the event will be committed only if there is an enveloping context event present.

Again, here is the code example with comments explaining the expected outcomes.

// MySimpleEvent is configured with `select=if-context`
MyContextEvent ctx1 = new MyContextEvent(1234);
ctx1.begin();
new MySimpleEvent().commit(); // the event will get committed - there is enveloping context here
ctx1.end();
ctx1.commit();

new MySimpleEvent().commit(); // the event will not get committed - there is no enveloping context here

Can, you, perhaps, add a code snippet showing how if-outlier is supposed to work? I want to completely understand the requirements before I try incorporating it in the prototype.

Thanks!

[egahlin]

There seems to be two discussions:

1. How event settings for the triggering use case (what you describe above) should look like, so you can build a prototype

2. The design of the event settings, for all use cases (throttling, triggering, thresholding, user-defined selection, ...)

When it comes to 1, implement the settings however you like, and see what works. Take that information back into the second discussion. Don't read too much into what I wrote in the JIRA issue. It was just some first thoughts to illustrate. Now, I'm thinking enablement could have a third state, "true", "false" and "triggered", but not sure it's a good idea. This would be the semantics:

  {"triggered", "triggered"} -> "triggered"
  {"true", "triggered"} -> "true"
  {"false", "triggered"} -> "triggered"

[jbachorik]

I have implemented the prototype (https://github.com/skogsluft/jdk-skogsluft/tree/feature/jfr_context_1) and split it into three separate commits:

• Minimal support for contextual events - it is adding support for managing and tracking open contexts (contextual events) and also for activating those events by the fact of committing of other non-requstable events while the contextual event is open
• Internal @Contextual annotation - syntactic sugar to make defining contextual events as easy as placing that annotation on the event definition class
• Initial selector support - adds an initial support for select config option and @Selector annotation, allowing to configure an event type to be emitted only if it is within an active context

The reason for this is to be able to move in small, incremental steps, which are completely self-contained and provide the expected functionality. Another motive would be attempting to backport the minimal support to at least JDK 21 - I know this will be contentious but, nevertheless, we could figure this out and the benefits will far outweigh the risks, especially because we will be able to get some real-life user feedback on the usefulness of the contextual event concept (we are definitely not going to have many customers running on the latest JDK release - in best case the latest LTS).

Examples

Examples assume that jdk.jfr.internal package will be 'force-exported' by jdk.jfr module by the user.

Minimal support

    @Label("My Context Event")
    @Name("jdk.TestContext")
    @Registered
    @Enabled
    public static class MyContextEvent extends Event {
        @Label("spanId")
        private String spanId;

        private transient long ctxId = 0;

        public MyContextEvent(String spanId) {
            this.spanId = spanId;
        }

        public void doBegin() {
            ctxId = JVM.openContext();
        }

        public void doEnd() {
            ctxId = JVM.closeContext() - ctxId;
        }

        public void doCommit() {
            if (shouldCommit() && ctxId != 0) {
                commit();
            }
        }
    }

    @Label("My Work Event")
    @Name("jdk.TestWork")
    @Registered
    @Enabled
    public static class MyWorkEvent extends Event {
    }
    
   public static void main(String ... args) {
     var ctx = new MyContextEvent("123");
     ctx.doBegin(); // open context
     new MyWorkEvent().commit(); // commit work event
     ctx.doEnd(); 
     ctx.doCommit();
   }

Annotated contextual event

This time the author of a contextual event does not need to add special methods to handle context open/close operations (also, guarded commit).

    @Label("My Context Event")
    @Name("jdk.TestContext")
    @Registered
    @Enabled
    @Contextual
    public static class MyContextEvent extends Event {
        @Label("spanId")
        private String spanId;
        public MyContextEvent(String spanId) {
            this.spanId = spanId;
        }
    }

    @Label("My Work Event")
    @Name("jdk.TestWork")
    @Registered
    @Enabled
    public static class MyWorkEvent extends Event {
    }
    
   public static void main(String ... args) {
     var ctx = new MyContextEvent("123");
     ctx.begin(); // open context
     new MyWorkEvent().commit(); // commit work event
     ctx.end(); 
     ctx.commit();
   }

Context with selector

@Label("My Context Event")
    @Name("jdk.TestContext")
    @Registered
    @Enabled
    @Contextual
    public static class MyContextEvent extends Event {
        @Label("spanId")
        private String spanId;
        public MyContextEvent(String spanId) {
            this.spanId = spanId;
        }
    }
    @Label("My Work Event")
    @Name("jdk.TestWork")
    @Registered
    @Enabled
    @Selector("if-context")
    public static class MyWorkEvent extends Event {
    }
    
    public static void main(String ... args) {
     var ctx = new MyContextEvent("123");
     new MyWorkEvent().commit(); // this event will not be emitted because the selector requires context
     ctx.begin(); // open context
     new MyWorkEvent().commit(); // commit work event
     ctx.end(); 
     ctx.commit();
   }

[egahlin]

I don't think enhancements should be backported and especially not experimental ones. Also, the trigger use case can be simulated by post-processing a file where events with @contextual are removed unless another event happens simultaneously in the same thread. See RecordingFile::write.

I think there are some questions regarding the design (starting with triggering use case) that should be answered/discussed or written down.

• How will carrier threads be handled?
• What constitutes a trigger, start of an event, end of an event, or both within begin() and end()
• How is misuse of the API handled, for example, calling begin() twice on a contextual event, or end() before begin().
• How are nesting of contextual events handled?
• Are contextual events reserved for Java events, or could there be a need for native as well, for example, GC events.

[jbachorik]

Hi Erik, thanks for the response!

Yeah, the backport mention was just a stab in the dark. I know it most probably won't happen ...
Unfortunately, the simulation is not feasible to be used as a replacement for the proper implementation - either we would have to ship potentially extremely large recordings over network or we would have do the post-processing inside a customer's JVM. Neither of those options is acceptable.

The design questions are valid and definitely need to be answered before having a 'packaged' solution with a nice annotation etc.
For the first step of just allowing an external user to write their own contextual event implementation by having access to the virtual event counter per thread in order to be able to quickly determine whether some events were committed during the period of interest, it is completely up to the implementor itself. The internal access is meant to provide read-only view and there is no chance of affecting JFR by the user.

For further steps, let me elaborate

Virtual Threads/Carrier Threads

At this moment I don't see a cheap way to make the contextual events specific to virtual threads. The problem is that when a virtual thread with open context is unmounted we would need to persist the state of the flag, determining whether the event is enveloping other events, somehow along the virtual thread state.
I will need to think more about this - if you or someone else has any ideas how to achieve this without incurring high overhead, both in terms of processing and memory, I am open to suggestions.

Triggers

By 'trigger' you mean the fact that there is another event emitted on a thread while a particular contextual event is 'open' (between begin/end)? If that's the case I would assume that trigger must be 'commit' - AFAICU, it is perfectly legal for an event to go through both begin and end but still not to be committed, correct?

API invariants handling

Currently, there are no handrails. But, calling openContext() and closeContext() are non-destructible operations regarding the context.

For the future I would propose following semantics:

• A single contextual event can not span multiple threads
• Calling begin() is idempotent
• Calling end() is idempotent
• When no end() has been called explicitly it will be called implicitly by commit() (this is the current state in all events)
• When end() is called (either explicitly or implicitly) without at least one previous call to begin() the contextual event will be considered empty and not written out

Nesting of contextual events

Solely for the triggering purposes, the nesting does not play any role. A contextual event will get triggered as soon as the virtual pointer to the event buffer has moved since begin() - regardless of the nesting level.

For selectors, it will be necessary to at least track the nesting level, such that we know when there is at least one contextual event active. I know we have been talking about advanced filtering based on event type or even the attribute values, but I would rather leave that for a followup enhancement because it will have to solve many complexities.
In order to implement the nesting level tracking, we can have a single counter incremented when a contextual event is open and decremented when it is closed. The counter must be incremented and decremented only once per event instance and call to begin()/end() respectively.

Native contextual events?

I haven't thought about this - surely, I haven't had a use case for native contextual events.
If it becomes desirable to support contextual native events it should be fairly easy to implement the feature, considering that all the building blocks will be in place. However, I would rather focus on Java custom events first and ,once the concept is validated, extending to native events.

[egahlin]

Virtual Threads/Carrier Threads
Could be that a general mechanism is needed to carry JFR related information across carrier threads. We ran into the same issue with the Metrics feature and decided to postpone until VT had been checked in.

Invariants
Sounds reasonable.

What would happen if a contextual event is used across different threads, for example, begin() called in Thread A and end() called Thread B?

I would rather leave that for a followup enhancement because it will have to solve many complexities.

I'm fine with leaving the topic for now, but I think we should think it over from a design perspective (doesn't mean we need to implement it)

[jbachorik]

Could be that a general mechanism is needed to carry JFR related information across carrier threads.
Indeed, this would be the optimal solution. Having the support firmly embedded in the virtual thread itself could help to reduce the overhead.

At the moment I created a very rough PoC of the virtual thread support which is adding 8 bytes to each virtual thread such that it is possible to reliably track the JFR context per virtual thread.

What would happen if a contextual event is used across different threads, for example, begin() called in Thread A and end() called Thread B?

Logging an error and ignoring the context is probably best we can do. This would leave the nesting counter in inconsistent state - but we could make the counter multi-thread safe, for the price of memory barriers etc.

---

I am not trying to brush off the advanced filter - in fact, I did spend a bit of time thinking about it - but it quickly boils down to keeping a stack of rather complex context objects per virtual thread. We need the stack such that all selectable events can walk all the nested contexts and match their filters (which might be rather complex boolean operations, I guess) and decide whether they should be committed.

I am rather afraid of the unbounded memory requirements when combined with virtual threads - this would make the advanced filtering non-feasible. But if you have some alternatives in mind, please, don't hesitate to share!