Skip to content

observability.md

Latest commit

 

History

History
215 lines (169 loc) · 9.21 KB

observability.md

File metadata and controls

215 lines (169 loc) · 9.21 KB
Raw

Observability

This advanced guide describes the tracing contract for applications that plan Arrow schemas, write Arrow RecordBatch values to SQL Server, or wrap those writes inside a higher-level workflow. New users do not need tracing to complete a write; start with Getting Started.

Arrow SQL Server emits spans and events through the tracing crate. It does not install a global subscriber, choose an output sink, configure filters, or send telemetry to a vendor. Applications own subscriber setup and decide where events go.

Subscriber Ownership

Library code only emits telemetry. It is silent unless the application has a subscriber installed by the time Arrow SQL Server code runs.

For a small application or test, one possible setup is:

[dependencies]
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
use tracing_subscriber::EnvFilter;

fn init_tracing_for_example() {
    tracing_subscriber::fmt()
        .with_env_filter(EnvFilter::new(
            "info,arrow_sql_server=debug,tiberius_raw_bulk::protocol=info",
        ))
        .init();
}

Production applications should choose their own subscriber, filter policy, and export pipeline. For example, a service might send tracing data through an OpenTelemetry layer to Grafana, Datadog, or another backend. That setup belongs to the application, not to Arrow SQL Server.

Target And Levels

All crate-owned instrumentation uses the tracing target:

arrow_sql_server

The tiberius-raw-bulk dependency emits sanitized SQL Server client and TDS protocol telemetry under:

tiberius_raw_bulk::protocol

That protocol target covers lower-level phases such as connection setup, TLS negotiation, login, bulk-load protocol operations, packet write summaries, server token summaries, and SQL Browser named-instance resolution. It is emitted by tiberius-raw-bulk, not re-wrapped by Arrow SQL Server.

Recommended filters:

Level Meaning
info Lifecycle summaries such as schema planning, writer initialization, batch writes, and finish.
debug Direct raw backend summaries such as measured encoded bytes and planned row ranges.
error Sanitized failures with phase, summary, and diagnostic codes.
trace Reserved for future high-volume details. The current writer path does not require trace-level events.

The crate intentionally avoids per-row logs and raw payload logs.

Span And Event Contract

The table below lists the current tracing vocabulary emitted by the Arrow-to-SQL Server write path. Names are useful for downstream filters, dashboards, and tests, but they are not exposed as public Rust constants.

Area Span Events Level
Schema planning arrow_sql_server.schema_planning arrow_sql_server.schema_planning.started, arrow_sql_server.schema_planning.completed, arrow_sql_server.schema_planning.failed info, error
Writer initialization arrow_sql_server.writer_initialization arrow_sql_server.writer_initialization.started, arrow_sql_server.writer_initialization.completed, arrow_sql_server.writer_initialization.failed info, error
Target metadata validation arrow_sql_server.writer_initialization arrow_sql_server.target_metadata_validation.started, arrow_sql_server.target_metadata_validation.completed, arrow_sql_server.target_metadata_validation.failed info, error
Batch write arrow_sql_server.batch_write arrow_sql_server.batch_write.started, arrow_sql_server.batch_write.completed, arrow_sql_server.batch_write.failed info, error
Direct raw encoding arrow_sql_server.batch_write arrow_sql_server.direct_raw.measured, arrow_sql_server.direct_raw.ranges_planned, arrow_sql_server.direct_raw.failed debug, error
Direct raw packet write arrow_sql_server.batch_write arrow_sql_server.direct_raw.packet_write.completed, arrow_sql_server.direct_raw.failed debug, error
Finish arrow_sql_server.finish arrow_sql_server.finish.started, arrow_sql_server.finish.completed, arrow_sql_server.finish.failed info, error

Direct raw encoding and packet write events are emitted only for WriteBackend::DirectRawBulk. Baseline and framed direct writes still emit the shared writer initialization, batch write, and finish spans.

Field Contract

Fields are structured for filtering and aggregation. The exact set depends on the phase and event.

Safe field categories:

Category Examples
Phase and event identity phase, telemetry_event
Backend names requested_backend, resolved_backend, backend
Validated SQL Server identifiers target_schema, target_table
Counts arrow_field_count, planned_column_count, batch_row_count, batch_column_count, accepted_rows_after, batches_written
Direct raw summaries encoded_row_start, encoded_row_count, encoded_byte_count, encoded_range_count
Diagnostic summaries diagnostic_count, error_diagnostic_count, warning_diagnostic_count, diagnostic_codes, error_summary
Timing summaries elapsed_us
Planning policy names string_policy, binary_policy, timezone_policy, nanosecond_policy, uint64_policy, decimal_policy, decimal256_policy, float_policy, date64_policy

Schema planning also emits data type family summaries such as arrow_data_type_families and mssql_type_families. Diagnostic summaries may include diagnostic_field_names. Field and table names are identifiers, not row values, but they are log-visible. Do not use secret material as schema, table, or column names.

Redaction Contract

Arrow SQL Server tracing does not emit:

  • SQL Server connection strings.
  • Passwords, access tokens, or authentication material.
  • Row values.
  • Raw packet bytes or full encoded row payload bytes.
  • Raw dependency debug output.
  • Arbitrary SQL text.
  • Diagnostic messages that may include detailed source text.

Failure events use sanitized error_summary values and machine-readable diagnostic_codes. Detailed diagnostics remain available from returned Error values for callers that need to inspect them in process.

For application reports, prefer Error::safe_error_info(). It returns the same safe summary shape used by tracing, plus the outer write phase, inner error kind, diagnostic codes, and structured diagnostics when the error carries them. Display stays short and redacted. Error::without_write_phase() and std::error::Error::source() are for trusted in-process matching or debug paths, not default end-user output.

Downstream Workflow Spans

Applications should add workflow, source, and output context outside Arrow SQL Server. The crate cannot know job ids, source aliases, retry ids, or orchestrator output names.

Use a parent span around writer calls:

use tracing::Instrument as _;

let output_span = tracing::info_span!(
    target: "my_app",
    "my_app.output_write",
    workflow_name = "daily_load",
    output_name = "people",
);

async {
    writer.write_batch(batch).await?;
    writer.finish().await
}
.instrument(output_span)
.await?;

With a subscriber installed, the arrow_sql_server spans are emitted inside the application span. This lets downstream systems group crate-owned writer details under the application's workflow or output context without duplicating writer internals.

During writer operations, tiberius-raw-bulk protocol events are emitted under the active arrow_sql_server writer spans. A typical write trace can therefore look like:

my_app.output_write
  -> arrow_sql_server.writer_initialization
    -> protocol.connection.connect
    -> protocol.tls.negotiation
    -> protocol.login.flow
  -> arrow_sql_server.batch_write
    -> protocol.bulk_load.request
    -> protocol.bulk_load.packet.written
  -> arrow_sql_server.finish
    -> protocol.token.done

Collectors should keep the targets distinct: arrow_sql_server describes Arrow schema planning and writer lifecycle semantics, while tiberius_raw_bulk::protocol describes SQL Server client and TDS protocol semantics.

When a workflow writes multiple outputs, create one parent span per output or per logical workflow step. Keep source-specific context in the application span or its fields rather than expecting Arrow SQL Server to infer it.

Known Gaps

  • Arrow SQL Server does not install a subscriber or exporter.
  • Arrow SQL Server does not emit row-level telemetry.
  • Arrow SQL Server does not emit raw packet bytes, raw row payload bytes, or arbitrary SQL text.
  • Direct raw writer events report safe row and byte summaries. Lower-level packet summaries are emitted by tiberius-raw-bulk protocol tracing.
  • SQL Server engine behavior is outside both client libraries. Server-side execution, waits, locks, IO, and query plans require SQL Server DMVs, Extended Events, Query Store, or separate profiling queries.
  • Workflow ids, output names, source aliases, retries, transactions, and orchestration status must be supplied by downstream applications.
  • The bench-profile feature exposes benchmark-only profiling hooks. It is separate from the normal tracing contract and is not required for production observability.