[exporter/awss3exporter] add s3_base_prefix parameter (#42660)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Adds a new optional parameter `s3_base_prefix` to the `awss3exporter`
which is used as the root path for all files uploaded.

This new parameter is helpful when using the
`resource_attrs_to_s3/s3_prefix` override ability but you still want a
root path to be used for all file uploads.

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes #42661

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Added unit tests and ran the otel collector locally using a config file
that specifies an `awss3` exporter with the new `s3_base_prefix`
parameter

<!--Describe the documentation added.-->
#### Documentation
Updated the readme and added new configuration examples

<!--Please delete paragraphs that you did not use before submitting.-->

Author: genehynson

.chloggen/ghynson_s3_base_path.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: exporter/awss3exporter
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: adds configuration field 's3_base_prefix' to be able to set a base path for all S3 file uploads
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [42661]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: [user]

exporter/awss3exporter/README.md

@@ -24,7 +24,8 @@ The following exporter configuration parameters are supported.
 |:--------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|
 | `region`                  | AWS region.                                                                                                                                                                                                                | "us-east-1"                                 |
 | `s3_bucket`               | S3 bucket                                                                                                                                                                                                                  |                                             |
-| `s3_prefix`               | prefix for the S3 key (root directory inside bucket).                                                                                                                                                                      |                                             |
+| `s3_base_prefix`          | root prefix for the S3 key applied to all files.                                                                               |                                             |
+| `s3_prefix`               | prefix for the S3 key that can be overridden dynamically by `resource_attrs_to_s3` parameter.                                                                                                                                                                      |                                             |
 | `s3_partition_format`     | filepath formatting for the partition; See [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) for format specification.                                                                                 | "year=%Y/month=%m/day=%d/hour=%H/minute=%M" |
 | `s3_partition_timezone`   | timezone used to format partition                                                                                                                                                                                          | Local                                       |
 | `role_arn`                | the Role ARN to be assumed                                                                                                                                                                                                 |                                             |
@@ -145,6 +146,27 @@ metric/YYYY/MM/DD/HH/mm
 Optionally along with `s3_partition_format` you can provide `s3_partition_timezone` as name from IANA Time Zone 
 database to change default local timezone to custom, for example `UTC` or `Europe/London`.
 
+## Base Path Configuration
+
+The `s3_base_prefix` option allows you to specify a root path inside the bucket that is not overridden by `resource_attrs_to_s3`. If provided, `s3_prefix` will be appended to this base path.
+
+```yaml
+exporters:
+  awss3:
+    s3uploader:
+      region: 'eu-central-1'
+      s3_bucket: 'databucket'
+      s3_base_prefix: 'environment/prod'
+      s3_prefix: 'metric'
+      s3_partition_format: '%Y/%m/%d/%H/%M'
+```
+
+In this case, logs and traces would be stored in the following path format.
+
+```console
+environment/prod/metric/YYYY/MM/DD/HH/mm
+```
+
 ## Data routing based on resource attributes
 When `resource_attrs_to_s3/s3_bucket` or `resource_attrs_to_s3/s3_prefix` is configured, the S3 bucket and/or prefix are dynamically derived from specified resource attributes in your data.
 If the attribute values are unavailable, the bucket and prefix will fall back to the values defined in `s3uploader/s3_bucket` and `s3uploader/s3_prefix` respectively.
@@ -170,6 +192,42 @@ databucket/metric/YYYY/MM/DD/HH/mm
 ...
 ```
 
+## Base Path with Resource Attributes
+
+When using both `s3_base_prefix` and `resource_attrs_to_s3/s3_prefix`, the `s3_base_prefix` is always used while `s3_prefix` can be dynamically overridden by resource attributes.
+
+```yaml
+exporters:
+  awss3:
+    s3uploader:
+      region: 'eu-central-1'
+      s3_bucket: 'databucket'
+      s3_base_prefix: 'environment/prod'
+      s3_prefix: 'default-metric'
+      s3_partition_format: '%Y/%m/%d/%H/%M'
+    resource_attrs_to_s3:
+      s3_prefix: "com.awss3.prefix"
+```
+
+In this configuration:
+- **Base Prefix**: `environment/prod` (always included)
+- **Prefix**: Dynamically set from resource attribute `com.awss3.prefix` if available, otherwise falls back to `default-metric`
+
+**Path format examples:**
+
+```console
+# When resource attribute com.awss3.prefix = "service-a/metrics"
+environment/prod/service-a/metrics/YYYY/MM/DD/HH/mm
+
+# When resource attribute com.awss3.prefix = "service-b/logs"  
+environment/prod/service-b/logs/YYYY/MM/DD/HH/mm
+
+# When resource attribute is unavailable (fallback)
+environment/prod/default-metric/YYYY/MM/DD/HH/mm
+```
+
+This allows you to maintain consistent organizational structure (via base path) while dynamically routing different data types or services to specific subdirectories.
+
 ## Retry
 
 Standard is the default retryer implementation used by service clients. See the [retry](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry) package documentation for details on what errors are considered as retryable by the standard retryer implementation.

exporter/awss3exporter/config.go

@@ -25,7 +25,9 @@ type S3UploaderConfig struct {
 	Region string `mapstructure:"region"`
 	// S3Bucket is the bucket name to be uploaded to.
 	S3Bucket string `mapstructure:"s3_bucket"`
-	// S3Prefix is the key (directory) prefix to written to inside the bucket
+	// S3BasePrefix is the root key (directory) prefix used to write the file.
+	S3BasePrefix string `mapstructure:"s3_base_prefix"`
+	// S3Prefix is the key (directory) prefix to write to inside the bucket. Appended to S3BasePrefix if provided.
 	S3Prefix string `mapstructure:"s3_prefix"`
 	// S3PartitionFormat is used to provide the rollup on how data is written. Uses [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) formatting.
 	S3PartitionFormat string `mapstructure:"s3_partition_format"`

exporter/awss3exporter/config_test.go

@@ -557,3 +557,84 @@ func TestConfigS3UniqueKeyFunc(t *testing.T) {
 	}, e,
 	)
 }
+
+func TestConfigS3BasePrefix(t *testing.T) {
+	factories, err := otelcoltest.NopFactories()
+	assert.NoError(t, err)
+
+	factory := NewFactory()
+	factories.Exporters[factory.Type()] = factory
+	cfg, err := otelcoltest.LoadConfigAndValidate(
+		filepath.Join("testdata", "config-s3_base_prefix.yaml"), factories)
+
+	require.NoError(t, err)
+	require.NotNil(t, cfg)
+
+	e := cfg.Exporters[component.MustNewID("awss3")].(*Config)
+	queueCfg := exporterhelper.NewDefaultQueueConfig()
+	queueCfg.Enabled = false
+	timeoutCfg := exporterhelper.TimeoutConfig{
+		Timeout: 5 * time.Second,
+	}
+
+	assert.Equal(t, &Config{
+		S3Uploader: S3UploaderConfig{
+			Region:            "us-east-1",
+			S3Bucket:          "foo",
+			S3Prefix:          "bar",
+			S3BasePrefix:      "base/prefix",
+			S3PartitionFormat: "year=%Y/month=%m/day=%d/hour=%H/minute=%M",
+			Endpoint:          "http://endpoint.com",
+			StorageClass:      "STANDARD",
+			RetryMode:         DefaultRetryMode,
+			RetryMaxAttempts:  DefaultRetryMaxAttempts,
+			RetryMaxBackoff:   DefaultRetryMaxBackoff,
+		},
+		QueueSettings:   queueCfg,
+		TimeoutSettings: timeoutCfg,
+		MarshalerName:   "otlp_json",
+	}, e,
+	)
+}
+
+func TestConfigS3BasePrefixWithResourceAttrs(t *testing.T) {
+	factories, err := otelcoltest.NopFactories()
+	assert.NoError(t, err)
+
+	factory := NewFactory()
+	factories.Exporters[factory.Type()] = factory
+	cfg, err := otelcoltest.LoadConfigAndValidate(
+		filepath.Join("testdata", "config-s3_base_prefix_with_resource_attrs.yaml"), factories)
+
+	require.NoError(t, err)
+	require.NotNil(t, cfg)
+
+	e := cfg.Exporters[component.MustNewID("awss3")].(*Config)
+	queueCfg := exporterhelper.NewDefaultQueueConfig()
+	queueCfg.Enabled = false
+	timeoutCfg := exporterhelper.TimeoutConfig{
+		Timeout: 5 * time.Second,
+	}
+
+	assert.Equal(t, &Config{
+		S3Uploader: S3UploaderConfig{
+			Region:            "us-east-1",
+			S3Bucket:          "foo",
+			S3Prefix:          "default-metric",
+			S3BasePrefix:      "environment/prod",
+			S3PartitionFormat: "year=%Y/month=%m/day=%d/hour=%H/minute=%M",
+			Endpoint:          "http://endpoint.com",
+			StorageClass:      "STANDARD",
+			RetryMode:         DefaultRetryMode,
+			RetryMaxAttempts:  DefaultRetryMaxAttempts,
+			RetryMaxBackoff:   DefaultRetryMaxBackoff,
+		},
+		QueueSettings:   queueCfg,
+		TimeoutSettings: timeoutCfg,
+		MarshalerName:   "otlp_json",
+		ResourceAttrsToS3: ResourceAttrsToS3{
+			S3Prefix: "com.awss3.prefix",
+		},
+	}, e,
+	)
+}

exporter/awss3exporter/internal/upload/partition.go

@@ -7,6 +7,7 @@ import (
 	"math/rand/v2"
 	"path"
 	"strconv"
+	"strings"
 	"time"
 
 	"github.com/google/uuid"
@@ -19,8 +20,12 @@ var compressionFileExtensions = map[configcompression.Type]string{
 }
 
 type PartitionKeyBuilder struct {
+	// PartitionBasePrefix defines the root S3
+	// directory (key) prefix used to write the file.
+	PartitionBasePrefix string
 	// PartitionPrefix defines the S3 directory (key)
-	// prefix used to write the file
+	// prefix used to write the file.
+	// Appended to PartitionBasePrefix if provided.
 	PartitionPrefix string
 	// PartitionFormat is used to separate values into
 	// different time buckets.
@@ -57,14 +62,24 @@ func (pki *PartitionKeyBuilder) bucketKeyPrefix(ts time.Time, overridePrefix str
 	if overridePrefix != "" {
 		prefix = overridePrefix
 	}
+
+	var pathParts []string
+
+	if pki.PartitionBasePrefix != "" {
+		pathParts = append(pathParts, pki.PartitionBasePrefix)
+	}
+
 	if prefix != "" {
-		prefix += "/"
+		pathParts = append(pathParts, prefix)
 	}
+
 	location := pki.PartitionTimeLocation
 	if location == nil {
 		location = time.Local
 	}
-	return prefix + timefmt.Format(ts.In(location), pki.PartitionFormat)
+	pathParts = append(pathParts, timefmt.Format(ts.In(location), pki.PartitionFormat))
+
+	return strings.Join(pathParts, "/")
 }
 
 func (pki *PartitionKeyBuilder) fileName() string {

exporter/awss3exporter/internal/upload/partition_test.go

@@ -102,6 +102,69 @@ func TestPartitionKeyInputsNewPartitionKey(t *testing.T) {
 			expect:         "/foo-prefix1/year=2024/month=01/day=24/hour=06/minute=40/signal-output-service-01_pod2_fixed.metrics.gz",
 			overridePrefix: "/foo-prefix1",
 		},
+		{
+			name: "base path only",
+			inputs: &PartitionKeyBuilder{
+				PartitionBasePrefix: "base/path",
+				PartitionFormat:     "year=%Y/month=%m/day=%d/hour=%H/minute=%M",
+				FilePrefix:          "signal-output-",
+				Metadata:            "service-01_pod2",
+				FileFormat:          "metrics",
+				UniqueKeyFunc: func() string {
+					return "fixed"
+				},
+			},
+			expect:         "base/path/year=2024/month=01/day=24/hour=06/minute=40/signal-output-service-01_pod2_fixed.metrics",
+			overridePrefix: "",
+		},
+		{
+			name: "base path with prefix",
+			inputs: &PartitionKeyBuilder{
+				PartitionBasePrefix: "base/path",
+				PartitionPrefix:     "telemetry",
+				PartitionFormat:     "year=%Y/month=%m/day=%d/hour=%H/minute=%M",
+				FilePrefix:          "signal-output-",
+				Metadata:            "service-01_pod2",
+				FileFormat:          "metrics",
+				UniqueKeyFunc: func() string {
+					return "fixed"
+				},
+			},
+			expect:         "base/path/telemetry/year=2024/month=01/day=24/hour=06/minute=40/signal-output-service-01_pod2_fixed.metrics",
+			overridePrefix: "",
+		},
+		{
+			name: "base path with prefix and override",
+			inputs: &PartitionKeyBuilder{
+				PartitionBasePrefix: "base/path",
+				PartitionPrefix:     "telemetry",
+				PartitionFormat:     "year=%Y/month=%m/day=%d/hour=%H/minute=%M",
+				FilePrefix:          "signal-output-",
+				Metadata:            "service-01_pod2",
+				FileFormat:          "metrics",
+				UniqueKeyFunc: func() string {
+					return "fixed"
+				},
+			},
+			expect:         "base/path/override/year=2024/month=01/day=24/hour=06/minute=40/signal-output-service-01_pod2_fixed.metrics",
+			overridePrefix: "override",
+		},
+		{
+			name: "base path with empty prefix",
+			inputs: &PartitionKeyBuilder{
+				PartitionBasePrefix: "base/path",
+				PartitionPrefix:     "",
+				PartitionFormat:     "year=%Y/month=%m/day=%d/hour=%H/minute=%M",
+				FilePrefix:          "signal-output-",
+				Metadata:            "service-01_pod2",
+				FileFormat:          "metrics",
+				UniqueKeyFunc: func() string {
+					return "fixed"
+				},
+			},
+			expect:         "base/path/year=2024/month=01/day=24/hour=06/minute=40/signal-output-service-01_pod2_fixed.metrics",
+			overridePrefix: "",
+		},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()

exporter/awss3exporter/s3_writer.go

@@ -101,6 +101,7 @@ func newUploadManager(
 	return upload.NewS3Manager(
 		conf.S3Uploader.S3Bucket,
 		&upload.PartitionKeyBuilder{
+			PartitionBasePrefix:   conf.S3Uploader.S3BasePrefix,
 			PartitionPrefix:       conf.S3Uploader.S3Prefix,
 			PartitionFormat:       conf.S3Uploader.S3PartitionFormat,
 			PartitionTimeLocation: s3PartitionTimeLocation,

exporter/awss3exporter/testdata/config-s3_base_prefix.yaml

@@ -0,0 +1,26 @@
+receivers:
+  nop:
+
+exporters:
+  awss3:
+    sending_queue:
+      enabled: false
+    timeout: 5s
+
+    s3uploader:
+        region: 'us-east-1'
+        s3_bucket: 'foo'
+        s3_base_prefix: 'base/prefix'
+        s3_prefix: 'bar'
+        s3_partition_format: 'year=%Y/month=%m/day=%d/hour=%H/minute=%M'
+        endpoint: "http://endpoint.com"
+
+processors:
+  nop:
+
+service:
+  pipelines:
+    traces:
+      receivers: [nop]
+      processors: [nop]
+      exporters: [awss3]

exporter/awss3exporter/testdata/config-s3_base_prefix_with_resource_attrs.yaml

@@ -0,0 +1,29 @@
+receivers:
+  nop:
+
+exporters:
+  awss3:
+    sending_queue:
+      enabled: false
+    timeout: 5s
+
+    s3uploader:
+        region: 'us-east-1'
+        s3_bucket: 'foo'
+        s3_base_prefix: 'environment/prod'
+        s3_prefix: 'default-metric'
+        s3_partition_format: 'year=%Y/month=%m/day=%d/hour=%H/minute=%M'
+        endpoint: "http://endpoint.com"
+    
+    resource_attrs_to_s3:
+        s3_prefix: "com.awss3.prefix"
+
+processors:
+  nop:
+
+service:
+  pipelines:
+    traces:
+      receivers: [nop]
+      processors: [nop]
+      exporters: [awss3]