Add Support for `gen_ai` semantic-convention breaking changes in v1.37.0

[Mikaayenson]

Is your feature request related to a problem?

The gen_ai fields are not using the latest semantic convention. It looks like this was supposed to be done in this PR https://github.com/open-telemetry/opentelemetry-python/pull/4731/changes#diff-25ace29f4af9b016a152aad40987138b245a5dcee698581ecdcb3fb106455b58 but using the latest sdk version v1.39.1, it appears the gen_ai fields are still using the only semantic convention (labels.gen_ai_prompt).

Describe the solution you'd like

In the v1.37.0 release https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.37.0, it added breaking changers for gen_ai fields. Can we double check that the fields are properly ingested and follow the latest convention.

Describe alternatives you've considered

No response

Additional Context

No response

Would you like to implement a fix?

None

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[susan-shu-c]

Here's an example document using the Python OTel SDK:

{
  "_index": ".ds-logs-apm.app.unknown_service-default-2026.01.12-000001",
  "_id": "Co8htJsBSwdo5i2ty46R",
  "_version": 1,
  "_source": {
    "observer": {
      "hostname": "ea8811c724ce",
      "type": "apm-server",
      "version": "9.2.3"
    },
    "agent": {
      "name": "opentelemetry/python/elastic",
      "version": "1.11.0"
    },
    "trace": {
      "id": "abce5e50a4f842f448f0222f0a96497c"
    },
    "@timestamp": "2026-01-12T21:34:31.302Z",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "apm.app.unknown_service"
    },
    "service": {
      "node": {
        "name": "1a003503-c9d5-4d0a-9f5e-44288fd61a65"
      },
      "framework": {
        "name": "openlit.otel.tracing"
      },
      "name": "unknown_service",
      "runtime": {
        "name": "cpython",
        "version": "3.12.12"
      },
      "language": {
        "name": "python"
      }
    },
    "host": {
      "os": {
        "type": "macos",
        "version": "25.2.0",
        "platform": "darwin"
      }
    },
    "event": {
      "kind": "event",
      "dataset": "apm.app.unknown_service"
    },
    "message": "gen_ai.content.prompt",
    "transaction": {
      "id": "6a65d5b1a8cac03a"
    },
    "labels": {
      "process_runtime_description": "3.12.12 (main, Dec 26 2025, 09:10:55) [Clang 17.0.0 (clang-1700.6.3.2)]",
      "gen_ai_prompt": "system: You are a helpful assistant.\nuser: I am going to Paris, what should I see?\nassistant: Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the worlds largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, its no wonder that Paris is one of the most popular tourist destinations in the world.\nuser: What is so great about #1?",
      "telemetry_auto_version": "0.60b1"
    },
    "span": {
      "id": "6a65d5b1a8cac03a"
    }
  },
  "fields": {
    "labels.process_runtime_description": [
      "3.12.12 (main, Dec 26 2025, 09:10:55) [Clang 17.0.0 (clang-1700.6.3.2)]"
    ],
    "service.node.name": [
      "1a003503-c9d5-4d0a-9f5e-44288fd61a65"
    ],
    "service.language.name": [
      "python"
    ],
    "service.node.name.text": [
      "1a003503-c9d5-4d0a-9f5e-44288fd61a65"
    ],
    "transaction.id": [
      "6a65d5b1a8cac03a"
    ],
    "labels.telemetry_auto_version": [
      "0.60b1"
    ],
    "trace.id": [
      "abce5e50a4f842f448f0222f0a96497c"
    ],
    "agent.name.text": [
      "opentelemetry/python/elastic"
    ],
    "host.os.version": [
      "25.2.0"
    ],
    "agent.name": [
      "opentelemetry/python/elastic"
    ],
    "event.kind": [
      "event"
    ],
    "host.os.type": [
      "macos"
    ],
    "service.name": [
      "unknown_service"
    ],
    "service.framework.name": [
      "openlit.otel.tracing"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "service.language.name.text": [
      "python"
    ],
    "service.runtime.name": [
      "cpython"
    ],
    "service.runtime.name.text": [
      "cpython"
    ],
    "service.runtime.version": [
      "3.12.12"
    ],
    "span.id": [
      "6a65d5b1a8cac03a"
    ],
    "message": [
      "gen_ai.content.prompt"
    ],
    "data_stream.type": [
      "logs"
    ],
    "observer.hostname": [
      "ea8811c724ce"
    ],
    "service.framework.name.text": [
      "openlit.otel.tracing"
    ],
    "labels.gen_ai_prompt": [
      "system: You are a helpful assistant.\nuser: I am going to Paris, what should I see?\nassistant: Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the worlds largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, its no wonder that Paris is one of the most popular tourist destinations in the world.\nuser: What is so great about #1?"
    ],
    "@timestamp": [
      "2026-01-12T21:34:31.302Z"
    ],
    "host.os.platform": [
      "darwin"
    ],
    "observer.type": [
      "apm-server"
    ],
    "observer.version": [
      "9.2.3"
    ],
    "service.name.text": [
      "unknown_service"
    ],
    "data_stream.dataset": [
      "apm.app.unknown_service"
    ],
    "agent.version": [
      "1.11.0"
    ],
    "event.dataset": [
      "apm.app.unknown_service"
    ]
  }
}

[EsraaKamel11]

Thanks for the detailed report and links.

I can take this on. I’ll:

1. Reproduce on opentelemetry-python v1.39.1 and confirm current emitted gen_ai attributes (e.g., labels.gen_ai_prompt).
2. Compare against semantic-conventions v1.37.0 gen_ai breaking changes.
3. Update the SDK to emit the latest gen_ai attribute keys and add tests to prevent regressions.

If that sounds good, please assign the issue to me and I’ll follow up with a PR.