fix: improve telemetry spans

[avilagaston9]

Improve Telemetry Spans

Motivation

We found that sometimes our Batcher tries to cancel batches that were actually included in the net, calling the batcherTaskCreationFailed endpoint, which finalizes the trace and prevents the Aggregator from registering its spans in the trace.

Description

• Stops finalizing the trace when a batcherTaskCreationFailed occurs.
• Converts all events into spans to visualize them in real-time, eliminating the need to wait for the parent span to finish.

Observations

On a real batcherTaskCreationFailed, the Aggregator won't receive the new task, and the trace will remain unfinished. Furthermore, the trace metadata won't be removed from the Telemetry server store. Despite that, we will be able to visualize the orphans spans with a warning that their parent ID is invalid.

#1477 was created to address this issue.

How To Test

1. Check that everything is working normally:

Run anvil, all Aligned components with one or more operators and start telemetry:

make telemetry_full_start

Go to jaeger and explore the generated traces.

2. Check the scenario addressed in this PR:

Change the Batcher create_new_task_retryable function in batcher/aligned-batcher/src/retry/batcher_retryables.rs:165 to return an error after receiving the receipt:

 // timeout to prevent a deadlock while waiting for the transaction to be included in a block.
    let _result = timeout(Duration::from_millis(transaction_wait_timeout), pending_tx)
        .await
        .map_err(|e| {
            warn!("Error while waiting for batch inclusion: {e}");
            RetryError::Permanent(BatcherError::ReceiptNotFoundError)
        })?
        .map_err(|e| {
            warn!("Error while waiting for batch inclusion: {e}");
            RetryError::Permanent(BatcherError::ReceiptNotFoundError)
        })?
        .ok_or(RetryError::Permanent(BatcherError::ReceiptNotFoundError));
    Err(RetryError::Permanent(BatcherError::ReceiptNotFoundError))

Then, start all components again and you should be able to see the Aggregator spans even when the Batcher sends Batcher - Task Creation Failed

Type of change

Please delete options that are not relevant.

• Bug fix
• Refactor

Checklist

• “Hotfix” to testnet, everything else to staging
• Linked to Github Issue
• This change depends on code or research by an external entity
  • Acknowledgements were updated to give credit
• Unit tests added
• This change requires new documentation.
  • Documentation has been added/updated.
• This change is an Optimization
  • Benchmarks added/run
• Has a known issue
  • Remove unfinished traces from the Telemetry store #1477
• If your PR changes the Operator compatibility (Ex: Upgrade prover versions)
  • This PR adds compatibility for operator for both versions and do not change batcher/docs/examples
  • This PR updates batcher and docs/examples to the newer version. This requires the operator are already updated to be compatible

[PatStiles]

Ran on local devnet and observed the Aggregator spans even when Batcher - Task Creation Failed was thrown.

[Oppen]

I don't think a span for every event is the right solution here. I believe we still want real spans, just not a single one. The times for the spans seem off as well, and it looks like the aggregator sees the future here:

Still, if it works better than what we have I think we should merge it and review the solution later.

[avilagaston9]

I don't think a span for every event is the right solution here. I believe we still want real spans, just not a single one. The times for the spans seem off as well, and it looks like the aggregator sees the future here: Still, if it works better than what we have I think we should merge it and review the solution later.

The problem with the events is that they are associated with a parent span, but if for some reason we don't finalize it, we lose all the associated events.
It seems that the task event arrives at the Aggregator before the receipt arrives at the Batcher. This behavior is also observed in staging, but I don't consider this a problem.
IMO, we should separate this into two traces in the future, one for each component, removing the dependency between these two. I have created #1477 to revisit this solution later.
@Oppen

[MarcosNicolau]

Tested on macos and it is working well.

[avilagaston9]

Addressed in #1670 .