Markdown specification based on schema

[thompson-tomo]

Progresses #255

This introduces a markdown based version of the specification which is generated via tooling which consumes the schema definition.

Entry point is: docs/spec/README.md and obviously work is needing to be done to set titles etc.

[jack-berg]

Hey sorry for never engaging on this. I just got back from parental leave. I like the idea, but decided we needed to go in the direction of custom tooling, because there's a variety of information we want to capture / display which doesn't fit neatly into the JSON schema. Also left a comment on the issue here: #255 (comment)

Now that I'm back, I'd love to engage on this more with you if you have any other thoughts / ideas!

[thompson-tomo]

No worries at all @jack-berg

I have been thinking about the documentation recently after reading the new doc as well as considering a recent spec proposal.

As it stands looking at the current doc and seeing your post about stability what I think would be the nicest option is if we had a structure as follows in the docs/spec folder:

• README - Gives an introduction
• data-model.md - Shows the file definition
• Profiles - folder to store all profile definitions
  • {{xyz}}.md - Description of a profile including the json schema with these files being generated based on scripts

If I use logging as an example I would have the following profiles:

• BatchLogRecordProcessor
• SimpleLogRecordProcessor
• LoggerMatcherAndConfig

Those profiles have an explicit stability and defined seperately meaning the core data model is reduced hence easier to Stabilize.

Let me know if you want to discuss anything.

[jack-berg]

data-model.md - Shows the file definition

You mean the JSON schema definition itself?

Profiles - folder to store all profile definitions

Not familiar with what you mean by a profile. JSON schema uses the vocabulary of types and properties. E.g. OpenTelemetryConfiguration is type with properties file_format, disabled, etc.

I suspect by profile you mean type. Assuming this, you're advocating for an organization of one file per type, as opposed to a single large file with all types as currently modeled in schema-docs.md? Personally, I lean towards the higher information density of keeping everything in one file. Of course, there are usability issues when the file gets too large, but typically markdown renderers have tooling to assist with navigation, like github's outline view:

If you feel strongly about this, open an issue and I'm happy to change course if that's the consensus.

Those profiles have an explicit stability and defined seperately meaning the core data model is reduced hence easier to Stabilize.

Yup we differentiate between stable / experimental types and properties using the convention documented here:

• Experimental types start with Experimental*, e.g. ExperimentalLoggerConfig
• Experimental properties end with the suffix */development, e.g. .instrumentation/development

We're allowed to make breaking changes to experimental types and properties after stabilization, but not to stable types and properties.

[thompson-tomo]

My mistake I meant policy as spoken about in open-telemetry/opentelemetry-specification#4738

I would have the data model define the core without the extensions.

The problem I had with the doc is that is overwhelming to read.

Also what I am thinking of is how can we enable easy extension to the schema like we have with semantic conventions and enable seamless promotion to stable.

Let's see what I can come up with and draft and enable further discussion.

[jack-berg]

Also what I am thinking of is how can we enable easy extension to the schema like we have with semantic conventions and enable seamless promotion to stable.

That's something that's come up to and which I've been happy to punt on so far. Concrete use cases for extensibility include:

• SDK extension points (exporters, processors, etc) which have their own configuration schema
• Instrumentation configuration, which have some bits of configuration which are shared across languages and libraries (e.g. the set of HTTP request headers to record) and some bits which are language or library specific.

So far, all we have to address this is a "schema on read" approach. SDK extension points / instrumentation are provided a programmatic representation of a generic YAML mapping node (referred to as ConfigProperties in the spec), and are responsible for extracting bits of information and validation / translating to whatever schema they need to initialize.

This is fine, but can be cumbersome and feel repetitive. In java, I've been working on some lightweight tooling to make this easier / idiomatic for the ecosystem.

Its worth opening an issue to track / discuss holistically.