[bug] [Instrumentation.Hangfire] Cannot use RecordQueueLatency instrumentation option together with Hangfire Mongo storage provider

[marcin-brzozowski]

Component

OpenTelemetry.Instrumentation.Hangfire

Package Version

Package Name	Version
OpenTelemetry.Instrumentation.Hangfire	1.14.0-beta.1

Runtime Version

net10.0

Description

When configuring HangFire instrumentation with option RecordQueueLatency set to true, the HangfirePendingDurationFilterAttribute filter is added to the job processing pipeline. This filter adds a Parameter to the job definition called OpenTelemetry.EnqueuedAt.

There's a problem with the naming of this parameter when used with certain Hangfire storage providers. I've reproduced it with Hangfire.Mongo. The problem is that in Mongo, the syntax with dot (.) between properties indicates nesting. As a result, the property get's stored in Mongo as:

"OpenTelemetry": {
   "EnqueuedAt": "1764604466290"
}

Then, in Hangfire client, when the parameters are being read from the storage during job scheduling & execution, the client tries to deserialize the OpenTelemetry field into a string, essentially trying to deserialize { "EnqueuedAt" : "1764594496238" } object into a string, which fails with the following exception:

Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonString'.Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonString'.

Steps to Reproduce

Minimal code snippet, requires Mongo db running locally:

using Hangfire;
using Hangfire.Mongo;
using Hangfire.Mongo.Migration.Strategies;
using Hangfire.Mongo.Migration.Strategies.Backup;
using MongoDB.Driver;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);

const string databaseName = "HangfireDb";

builder.Services.AddOpenApi();

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource =>
    {
        resource.AddService(serviceName: "HangFireInstrumentation", serviceVersion: "1.0.0");
    })
    .WithMetrics(metrics =>
    {
        metrics
            .AddHangfireInstrumentation(options => { options.RecordQueueLatency = true; })
            .AddOtlpExporter(opts => { opts.Endpoint = new Uri("http://127.0.0.1:4317"); });
    });
builder.Services.AddSingleton<IMongoClient>(_ =>
{
    var mongoConnectionString = builder.Configuration.GetConnectionString("MongoStore");
    var mongoUrlBuilder = new MongoUrlBuilder(mongoConnectionString) { DatabaseName = databaseName };
    var clientSettings = MongoClientSettings.FromUrl(mongoUrlBuilder.ToMongoUrl());
    var mongoClient = new MongoClient(clientSettings);
    mongoClient.GetDatabase(databaseName);
    return mongoClient;
});
builder.Services.AddHangfire((sp, config) =>
{
    config
        .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
        .UseSimpleAssemblyNameTypeSerializer()
        .UseRecommendedSerializerSettings();

        var mongoClient = sp.GetRequiredService<IMongoClient>();
        
        config.UseMongoStorage(mongoClient, databaseName, new MongoStorageOptions
        {
            MigrationOptions = new MongoMigrationOptions
            {
                MigrationStrategy = new MigrateMongoMigrationStrategy(),
                BackupStrategy = new CollectionMongoBackupStrategy()
            },
            CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.TailNotificationsCollection,
            Prefix = "hangfire.mongo",
            CheckConnection = true,
        });
});

builder.Services.AddHangfireServer(options =>
{
    options.Queues = ["default"];
});


var app = builder.Build();

app.UseHttpsRedirection();

app.MapPost("/enqueue-job", (IBackgroundJobClient backgroundJobClient, JobRequest jobRequest) =>
{
    backgroundJobClient.Enqueue(() =>
   
        Console.WriteLine(jobRequest.Message)
    );
    return Results.Ok("Job enqueued");
});

app.Run();


public class JobRequest
{
    public string Message { get; set; }
}

Expected Result

RecordQueueLatency instrumentation option can be successfully used with Hangfire.Mongo storage provider.

Actual Result

Exception is thrown:

Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonString'.Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonString'.

When RecordQueueLatency is set to true

Additional Context

I suggest renaming the OpenTelemetry.EnqueuedAt property to opentelemetry_enqueued_at, which simplifies the naming so that it doesn't accidently introduce nesting.

Similar concept is used e.g. in HangfireInstrumentationConstants.ActivityContextKey which is named opentelemetry_activity_context and is stored in Mongo store as BsonString field instead of BsonDocument.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Tagging component owner(s).

@fred2u