Skip to content

Commit 38d52a1

Browse files
chr1sj0nesGoogle-ML-Automation
authored andcommitted
[mosaic_gpu] Force flush all cupti activity, then unsubscribe.
With default flushing, it is possible for events to be missed. We should only unsubscribe after we are finished with cupti. PiperOrigin-RevId: 737939327
1 parent 34cd5b0 commit 38d52a1

File tree

1 file changed

+4
-3
lines changed

1 file changed

+4
-3
lines changed

jaxlib/mosaic/gpu/mosaic_gpu_ext.cc

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -238,11 +238,12 @@ NB_MODULE(_mosaic_gpu_ext, m) {
238238
"failed to enable tracking of kernel activity by CUPTI");
239239
});
240240
m.def("_cupti_get_timings", []() {
241+
THROW_IF_CUPTI_ERROR(
242+
cuptiActivityFlushAll(CUPTI_ACTIVITY_FLAG_FLUSH_FORCED),
243+
"failed to flush CUPTI activity buffers");
244+
THROW_IF_CUPTI_ERROR(cuptiFinalize(), "failed to detach CUPTI");
241245
THROW_IF_CUPTI_ERROR(cuptiUnsubscribe(profiler_state.subscriber),
242246
"failed to unsubscribe from CUPTI");
243-
THROW_IF_CUPTI_ERROR(cuptiActivityFlushAll(CUPTI_ACTIVITY_FLAG_NONE),
244-
"failed to flush CUPTI activity buffers");
245-
THROW_IF_CUPTI_ERROR(cuptiFinalize(), "failed to detach CUPTI");
246247
return profiler_state.timings;
247248
});
248249
}

0 commit comments

Comments
 (0)