Skip to content

Add new buffering options to all OP destinations #2747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .generated-info
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"spec_repo_commit": "69305be",
"generated": "2025-08-05 16:20:40.645"
"spec_repo_commit": "884871f",
"generated": "2025-08-05 21:31:29.263"
}
91 changes: 91 additions & 0 deletions .generator/schemas/v2/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4946,6 +4946,8 @@ components:
description: Optional prefix for blobs written to the container.
example: logs/
type: string
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
container_name:
description: The name of the Azure Blob Storage container to store logs
in.
Expand Down Expand Up @@ -24998,6 +25000,8 @@ components:
description: The `microsoft_sentinel` destination forwards logs to Microsoft
Sentinel.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
client_id:
description: Azure AD client ID used for authentication.
example: a1b2c3d4-5678-90ab-cdef-1234567890ab
Expand Down Expand Up @@ -26630,6 +26634,8 @@ components:
properties:
auth:
$ref: '#/components/schemas/ObservabilityPipelineAmazonOpenSearchDestinationAuth'
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
bulk_index:
description: The index to write logs to.
example: logs-index
Expand Down Expand Up @@ -26708,6 +26714,8 @@ components:
description: S3 bucket name.
example: error-logs
type: string
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
id:
description: Unique identifier for the destination component.
example: amazon-s3-destination
Expand Down Expand Up @@ -26827,6 +26835,28 @@ components:
role session.
type: string
type: object
ObservabilityPipelineBufferOptions:
description: Configuration for buffer settings on destination components.
oneOf:
- $ref: '#/components/schemas/ObservabilityPipelineDiskBufferOptions'
- $ref: '#/components/schemas/ObservabilityPipelineMemoryBufferOptions'
- $ref: '#/components/schemas/ObservabilityPipelineMemoryBufferSizeOptions'
ObservabilityPipelineBufferOptionsDiskType:
default: disk
description: The type of the buffer that will be configured, a disk buffer.
enum:
- disk
type: string
x-enum-varnames:
- DISK
ObservabilityPipelineBufferOptionsMemoryType:
default: memory
description: The type of the buffer that will be configured, a memory buffer.
enum:
- memory
type: string
x-enum-varnames:
- MEMORY
ObservabilityPipelineConfig:
description: Specifies the pipeline's configuration, including its sources,
processors, and destinations.
Expand Down Expand Up @@ -26982,6 +27012,8 @@ components:
ObservabilityPipelineDatadogLogsDestination:
description: The `datadog_logs` destination forwards logs to Datadog Log Management.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
id:
description: The unique identifier for this component.
example: datadog-logs-destination
Expand Down Expand Up @@ -27083,12 +27115,27 @@ components:
type: string
x-enum-varnames:
- DEDUPE
ObservabilityPipelineDiskBufferOptions:
description: Options for configuring a disk buffer.
properties:
max_size:
description: Maximum size of the disk buffer.
example: 4096
format: int64
maximum: 536870912000
minimum: 268435488
type: integer
type:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptionsDiskType'
type: object
ObservabilityPipelineElasticsearchDestination:
description: The `elasticsearch` destination writes logs to an Elasticsearch
cluster.
properties:
api_version:
$ref: '#/components/schemas/ObservabilityPipelineElasticsearchDestinationApiVersion'
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
bulk_index:
description: The index to write logs to in Elasticsearch.
example: logs-index
Expand Down Expand Up @@ -27573,6 +27620,8 @@ components:
properties:
auth:
$ref: '#/components/schemas/ObservabilityPipelineGcpAuth'
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
customer_id:
description: The Google Chronicle customer ID.
example: abcdefg123456789
Expand Down Expand Up @@ -27639,6 +27688,8 @@ components:
description: Name of the GCS bucket.
example: error-logs
type: string
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
id:
description: Unique identifier for the destination component.
example: gcs-destination
Expand Down Expand Up @@ -27949,6 +28000,32 @@ components:
type: string
x-enum-varnames:
- LOGSTASH
ObservabilityPipelineMemoryBufferOptions:
description: Options for configuring a memory buffer by byte size.
properties:
max_size:
description: Maximum size of the disk buffer.
example: 4096
format: int64
maximum: 13743895347
minimum: 1
type: integer
type:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptionsMemoryType'
type: object
ObservabilityPipelineMemoryBufferSizeOptions:
description: Options for configuring a memory buffer by queue length.
properties:
max_events:
description: The `ObservabilityPipelineMemoryBufferSizeOptions` `max_events`.
example: 500
format: int64
maximum: 268435456
minimum: 1
type: integer
type:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptionsMemoryType'
type: object
ObservabilityPipelineMetadataEntry:
description: A custom metadata entry.
properties:
Expand All @@ -27972,6 +28049,8 @@ components:
ObservabilityPipelineNewRelicDestination:
description: The `new_relic` destination sends logs to the New Relic platform.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
id:
description: The unique identifier for this component.
example: new-relic-destination
Expand Down Expand Up @@ -28109,6 +28188,8 @@ components:
ObservabilityPipelineOpenSearchDestination:
description: The `opensearch` destination writes logs to an OpenSearch cluster.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
bulk_index:
description: The index to write logs to.
example: logs-index
Expand Down Expand Up @@ -28683,6 +28764,8 @@ components:
description: The `rsyslog` destination forwards logs to an external `rsyslog`
server over TCP or UDP using the syslog protocol.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
id:
description: The unique identifier for this component.
example: rsyslog-destination
Expand Down Expand Up @@ -29157,6 +29240,8 @@ components:
ObservabilityPipelineSentinelOneDestination:
description: The `sentinel_one` destination sends logs to SentinelOne.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
id:
description: The unique identifier for this component.
example: sentinelone-destination
Expand Down Expand Up @@ -29241,6 +29326,8 @@ components:
'
example: true
type: boolean
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
encoding:
$ref: '#/components/schemas/ObservabilityPipelineSplunkHecDestinationEncoding'
id:
Expand Down Expand Up @@ -29354,6 +29441,8 @@ components:
ObservabilityPipelineSumoLogicDestination:
description: The `sumo_logic` destination forwards logs to Sumo Logic.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
encoding:
$ref: '#/components/schemas/ObservabilityPipelineSumoLogicDestinationEncoding'
header_custom_fields:
Expand Down Expand Up @@ -29457,6 +29546,8 @@ components:
description: The `syslog_ng` destination forwards logs to an external `syslog-ng`
server over TCP or UDP using the syslog protocol.
properties:
buffer:
$ref: '#/components/schemas/ObservabilityPipelineBufferOptions'
id:
description: The unique identifier for this component.
example: syslog-ng-destination
Expand Down
42 changes: 42 additions & 0 deletions docs/datadog_api_client.v2.model.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11855,6 +11855,27 @@ datadog\_api\_client.v2.model.observability\_pipeline\_aws\_auth module
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_buffer\_options module
-----------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_buffer_options
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_buffer\_options\_disk\_type module
-----------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_buffer_options_disk_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_buffer\_options\_memory\_type module
-------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_buffer_options_memory_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_config module
--------------------------------------------------------------------

Expand Down Expand Up @@ -11953,6 +11974,13 @@ datadog\_api\_client.v2.model.observability\_pipeline\_dedupe\_processor\_type m
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_disk\_buffer\_options module
-----------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_disk_buffer_options
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_elasticsearch\_destination module
----------------------------------------------------------------------------------------

Expand Down Expand Up @@ -12296,6 +12324,20 @@ datadog\_api\_client.v2.model.observability\_pipeline\_logstash\_source\_type mo
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_memory\_buffer\_options module
-------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_memory_buffer_options
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_memory\_buffer\_size\_options module
-------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_memory_buffer_size_options
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_metadata\_entry module
-----------------------------------------------------------------------------

Expand Down
25 changes: 25 additions & 0 deletions src/datadog_api_client/v2/model/azure_storage_destination.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,28 @@


if TYPE_CHECKING:
from datadog_api_client.v2.model.observability_pipeline_buffer_options import ObservabilityPipelineBufferOptions
from datadog_api_client.v2.model.azure_storage_destination_type import AzureStorageDestinationType
from datadog_api_client.v2.model.observability_pipeline_disk_buffer_options import (
ObservabilityPipelineDiskBufferOptions,
)
from datadog_api_client.v2.model.observability_pipeline_memory_buffer_options import (
ObservabilityPipelineMemoryBufferOptions,
)
from datadog_api_client.v2.model.observability_pipeline_memory_buffer_size_options import (
ObservabilityPipelineMemoryBufferSizeOptions,
)


class AzureStorageDestination(ModelNormal):
@cached_property
def openapi_types(_):
from datadog_api_client.v2.model.observability_pipeline_buffer_options import ObservabilityPipelineBufferOptions
from datadog_api_client.v2.model.azure_storage_destination_type import AzureStorageDestinationType

return {
"blob_prefix": (str,),
"buffer": (ObservabilityPipelineBufferOptions,),
"container_name": (str,),
"id": (str,),
"inputs": ([str],),
Expand All @@ -32,6 +44,7 @@ def openapi_types(_):

attribute_map = {
"blob_prefix": "blob_prefix",
"buffer": "buffer",
"container_name": "container_name",
"id": "id",
"inputs": "inputs",
Expand All @@ -45,6 +58,13 @@ def __init__(
inputs: List[str],
type: AzureStorageDestinationType,
blob_prefix: Union[str, UnsetType] = unset,
buffer: Union[
ObservabilityPipelineBufferOptions,
ObservabilityPipelineDiskBufferOptions,
ObservabilityPipelineMemoryBufferOptions,
ObservabilityPipelineMemoryBufferSizeOptions,
UnsetType,
] = unset,
**kwargs,
):
"""
Expand All @@ -53,6 +73,9 @@ def __init__(
:param blob_prefix: Optional prefix for blobs written to the container.
:type blob_prefix: str, optional

:param buffer: Configuration for buffer settings on destination components.
:type buffer: ObservabilityPipelineBufferOptions, optional

:param container_name: The name of the Azure Blob Storage container to store logs in.
:type container_name: str

Expand All @@ -67,6 +90,8 @@ def __init__(
"""
if blob_prefix is not unset:
kwargs["blob_prefix"] = blob_prefix
if buffer is not unset:
kwargs["buffer"] = buffer
super().__init__(kwargs)

self_.container_name = container_name
Expand Down
Loading
Loading