Sending data to a self-hosted custom collector

[GiamBoscaro]

When I try to start my node application, I get this error as soon as the instrumentation.js file is preloaded.
I am using Node 16 inside Docker container (node:16-slim official image).

ReferenceError: crypto is not defined
    at V (/usr/src/app/node_modules/logfire/dist/index.cjs:1:2221)
    at m.generateTraceId (/usr/src/app/node_modules/logfire/dist/index.cjs:1:2139)
    at Tracer.startSpan (/usr/src/app/node_modules/@opentelemetry/sdk-trace-base/build/src/Tracer.js:71:41)
    at ProxyTracer.startSpan (/usr/src/app/node_modules/@opentelemetry/api/build/src/trace/ProxyTracer.js:32:34)
    at w (/usr/src/app/node_modules/@pydantic/logfire-api/dist/index.cjs:1:6201)
    at Module.ue [as reportError] (/usr/src/app/node_modules/@pydantic/logfire-api/dist/index.cjs:1:6916)
    at process.<anonymous> (/usr/src/app/node_modules/logfire/dist/index.cjs:1:3715)
    at process.emit (node:events:513:28)
    at process._fatalException (node:internal/process/execution:146:13)

// instrumentation.js
const logfire = require('logfire');
const os = require('os');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
const config = require('./config');
const { version } = require('./package.json');

logfire.configure({
  serviceName: os.hostname(),
  sendToLogfire: false,
  serviceVersion: version,
  environment: config.NODE_ENV,
  nodeAutoInstrumentations: {
    '@opentelemetry/instrumentation-express': ExpressInstrumentation,
    '@opentelemetry/instrumentation-http': HttpInstrumentation,
  },
  advanced: {
    baseUrl: config.TRACING_BASE_URL,
  },
});

I am connecting to a local instance of Alloy, and then I see the Traces in grafana. Everything is working normally in the Python logfire SDK, but getting these errors in the Javascript SDK

[petyosi]

I'm reasonably confident that the problems you experience are related to Node v16. It has reached end-of-life on September 11, 2023 (source). Do you experience the same issues with the current Node LTS (22.15)?

[GiamBoscaro]

I'm reasonably confident that the problems you experience are related to Node v16. It has reached end-of-life on September 11, 2023 (source). Do you experience the same issues with the current Node LTS (22.15)?

Unfortunately our services are running on Node 16 and I cannot upgrade to newer versions at the moment. I did find that enabling the flag --experimental-global-webcrypto makes the crypto package available globally. With this I can actually preload the logfire instrumentation without errors (after solving a small problem in my package.json).
So I have verified that it loads, now I need to figure out why there is nothing appearing in Grafana Tempo.
Just to be sure, is my instrumentation.js correctly configured? Also the baseUrl parameter should be just http://alloy:4318 or should I add /v1/traces? In the Logfire SDK for Python I do specify the endpoint as http://alloy:4318/v1/traces

[petyosi]

Perhaps you should check the logs of Alloy to verify that it actually accepts the payload or something goes wrong. You can also use the diagLogLevel configuration option to make the otel-js SDK report what it is doing.

[GiamBoscaro]

Perhaps you should check the logs of Alloy to verify that it actually accepts the payload or something goes wrong. You can also use the diagLogLevel configuration option to make the otel-js SDK report what it is doing.

Yes this is what I am doing but even the verbose logs are not very useful. The only thing I get is a timeout error, this is why I was asking about the correct url format, that maybe I am setting wrongly.

{"stack":"Error: Timeout\n at Timeout._onTimeout (/usr/src/app/node_modules/@opentelemetry/sdk-trace-base/build/src/export/BatchSpanProcessorBase.js:138:24)\n at listOnTimeout (node:internal/timers:559:17)\n at processTimers (node:internal/timers:502:7)","message":"Timeout","name":"Error"}

[petyosi]

The base url should not include /v1/traces.

[GiamBoscaro]

The base url should not include /v1/traces.

I checked on your sources code and have seen that indeed you create the full url later in traceExporterUrl: ${baseUrl}/${TRACE_ENDPOINT_PATH}.
I did do some testing with any combination of the baseUrl, and at the moment using http://alloy:4318 (4318 being the http server), but I still get the timeout.
I have checked the networking and that is ok, the alloy container is reachable and the APIs are working. I would really like to see a better error on what is happening, but even with a log level of 9999 I cannot see anything more.

I also noticed the typo in advanced?: AdancedLogfireConfigOptions

[petyosi]

@GiamBoscaro does your setup work with the regular opentelemetry-js instrumentation?

[GiamBoscaro]

@GiamBoscaro does your setup work with the regular opentelemetry-js instrumentation?

Tested this today, and yes it is actually working:

const sdk = new NodeSDK({
  serviceName: os.hostname(),
  traceExporter: new OTLPTraceExporter({
    url: `${config.ALLOY_BASE_URL}/v1/traces`,
    headers: {},
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: `${config.ALLOY_BASE_URL}/v1/metrics`,
      headers: {},
      concurrencyLimit: 1,
    }),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

I guess then there is something going on in the Logfire-js SDK

[petyosi]

Question, do you have a Logfire token configured in your env? Otherwise, the SDK will not send any data, as our service endpoint won't accept it.

[GiamBoscaro]

Question, do you have a Logfire token configured in your env? Otherwise, the SDK will not send any data, as our service endpoint won't accept it.

Nono, no token in my environment and I tried also to explicitely set token: null in the config.

[petyosi]

I guess that's the problem then. You need to set a token, and it should not be an empty string/null/undefined.

[GiamBoscaro]

I guess that's the problem then. You need to set a token, and it should not be an empty string/null/undefined.

Is it mandatory? Even if I am pushing the traces into a self-hosted collector? In Logfire Python SDK that is not required. I did try anyway at it not working, setting a random token.

I am trying to check the source code and run the example, and at the moment also the example is not working in my environment. I got some doubts here:

const baseUrl = !sendToLogfire || !token ? '' : logfireApi.resolveBaseUrl(process.env, cnf.advanced?.baseUrl, token)

Here sendToLogfire is false as I do not want to send my data to the Cloud.
Then if token is null, the url will be an empty string. So you are right that without a token it will not work, even if imho it does not make sense. Thus, I set a random token but then resolveBaseUrl will be called and generate some url that is not really a valid url for my environment.

I suppose the right way to do this would be something like:
const baseUrl = !sendToLogfire || !token ? cnf.advanced?.baseUrl : logfireApi.resolveBaseUrl(process.env, cnf.advanced?.baseUrl, token)

Anyway I changed this locally to

const baseUrl = cnf.advanced?.baseUrl;

Still does not work though. I checked then into traceExporter.ts and I have see that:

export function traceExporter(): SpanExporter {
  if (!logfireConfig.sendToLogfire) {
    return new VoidTraceExporter()
  }

  const token = logfireConfig.token
  if (!token) {
    throw new Error('Logfire token is required')
  }

  return new OTLPTraceExporter({
    headers: logfireConfig.authorizationHeaders,
    url: logfireConfig.traceExporterUrl,
  })
}

the only way to get a OTLPTraceExporter back is to have sendToLogfire = true and a not null token. But this again will not work if I am using a self-hosted collector. I then removed all these checks and returned only the OTLPTraceExporter with my custom url.

I have also seen that in the sdk.ts the traces are exported using by setting a processor:

const sdk = new NodeSDK({
  autoDetectResources: false,
  contextManager,
  idGenerator: new ULIDGenerator(),
  instrumentations: [getNodeAutoInstrumentations(logfireConfig.nodeAutoInstrumentations)],
  metricReader: logfireConfig.metrics === false ? undefined : periodicMetricReader(),
  resource,
  spanProcessors: [processor, ...logfireConfig.additionalSpanProcessors], // here
  textMapPropagator: propagator,
});

that is different from my working config where I had actually set the traceExporter parameter and never set any spanProcessors. But I don't know if this makes a difference, I don't know the various way of configuring the SDK.

I am still working on this since even after these experiments the example is still not working 🤕

[petyosi]

After consulting with the Python SDK authors, I believe that the JS SDK needs additional modification to support custom, non-Logfire connectors. Reference to the Python PR: pydantic/logfire#351.

[GiamBoscaro]

After consulting with the Python SDK authors, I believe that the JS SDK needs additional modification to support custom, non-Logfire connectors. Reference to the Python PR: pydantic/logfire#351.

Uh ok this makes much more sense now. So I will need to manually configure the opentelemetry SDK for the time being