Add support for hybrid lambda architecture for both buffered and streaming responses

[reakaleek]

Context

Preparing for the hybrid Lambda architecture to support streaming LLM responses. This PR adds OpenTelemetry tracing to AskAI operations and updates the Lambda deployment workflow and runtime configuration to support dual deployment (streaming-optimized + API Gateway-optimized).

Changes

OpenTelemetry Tracing:

• Added activity tracing to AskAiUsecase with GenAI semantic conventions
• Track conversation IDs, token counts, prompts, and agent metadata
• Extended IStreamTransformer with AgentId and AgentProvider properties
• Added streaming pipeline activities in StreamTransformerBase
• Configured OTel sources: Elastic.Documentation.Api.AskAi and StreamTransformer

Lambda Runtime Configuration:

• Support dual mode: Lambda Web Adapter (response streaming) and API Gateway HTTP API
• Switch from RestApi to HttpApi event source (HTTP API v2 payload format)
• Conditional Kestrel configuration based on AWS_LWA_INVOKE_MODE env var
• Added Amazon.Lambda.APIGatewayEvents package dependency

Deployment Workflow:

• Upload artifact to S3 once at ${ENVIRONMENT}/${GITHUB_SHA}/api-lambda.zip
• Deploy to both lambda-stream-optimized and lambda-api-gateway-optimized functions
• Tag Lambda versions with git SHA in description for traceability
• Enhanced deployment logs showing versions and commit for both functions