PutBucketLifecycleConfiguration writing fails with MalformedXML

[rabarar]

Describe the bug

when attempting to Put a lifecycle configuration onto a bucket, it fails with:

Got an error creating the policy:
operation error S3: PutBucketLifecycleConfiguration, https response error StatusCode: 400, RequestID: M2506HYABH1AT7BK, HostID: ttF1JGprIhRDwOkz8e22C4EQR3CoUFvWyFau4z6fckhzzOH9aIGB25HtqZH7cxPy1DOTqD5lzlc=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

The input looks as follows:

  input := &s3.PutBucketLifecycleConfigurationInput{
        Bucket: bucket,
        LifecycleConfiguration: &types.BucketLifecycleConfiguration{
            Rules: []types.LifecycleRule{
                {       
                    Status: types.ExpirationStatusEnabled,
                    AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
                        DaysAfterInitiation: 1,
                    },  
                    Expiration: &types.LifecycleExpiration{
                        ExpiredObjectDeleteMarker: true,
                        Days:                      0,
                        Date:                      nil,
                    },
                    NoncurrentVersionExpiration:  nil,
                    NoncurrentVersionTransitions: []types.NoncurrentVersionTransition{},
                    ID:                           &lifecyleID,
                    Prefix:                       nil,
                    Transitions:                  []types.Transition{},
                },
            },
        }, 
    }

Expected Behavior

Expect it to write the lifecycle configuration with the instantiated input values

Current Behavior

failure with:

Got an error creating the policy:
operation error S3: PutBucketLifecycleConfiguration, https response error StatusCode: 400, RequestID: M2506HYABH1AT7BK, HostID: ttF1JGprIhRDwOkz8e22C4EQR3CoUFvWyFau4z6fckhzzOH9aIGB25HtqZH7cxPy1DOTqD5lzlc=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

Reproduction Steps

Use this input:

   input := &s3.PutBucketLifecycleConfigurationInput{
        Bucket: bucket,
        LifecycleConfiguration: &types.BucketLifecycleConfiguration{
            Rules: []types.LifecycleRule{
                {   
                    Status: types.ExpirationStatusEnabled,
                    AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
                        DaysAfterInitiation: 1,
                    },  
                    Expiration: &types.LifecycleExpiration{
                        ExpiredObjectDeleteMarker: true,
                        Days:                      0,  
                        Date:                      nil,
                    },  
                    NoncurrentVersionExpiration:  nil,
                    NoncurrentVersionTransitions: []types.NoncurrentVersionTransition{},
                    ID:                           &lifecyleID,
                    Prefix:                       nil,
                    Transitions:                  []types.Transition{},
                },  
            },  
        },  
    }   

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"

Compiler and Version used

go version go1.18.3 darwin/arm64

Operating System and version

Mac OSX 12.4

[skmcgrail]

Could you follow the instructions in the Developer Guide to enable client logging with the aws.LogRequestWithBody option enabled? That would help us understand the serialized XML that is being sent to the service for your particular request. Thanks!

[rabarar]

Here’s the serialized XML on the request body: <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><AbortIncompleteMultipartUpload><DaysAfterInitiation>1</DaysAfterInitiation></AbortIncompleteMultipartUpload><Expiration><ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker></Expiration><ID>cleanup853211</ID><Status>Enabled</Status></Rule></LifecycleConfiguration>

…

On Jun 8, 2022, at 12:44 PM, Sean McGrail ***@***.***> wrote: Could you follow the instructions in the Developer Guide <https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/logging/> to enable client logging with the aws.LogRequestWithBody option enabled? That would help us understand the serialized XML that is being sent to the service for your particular request. Thanks! — Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6HWUPSKYUINIBMK2VTVODEWNANCNFSM5X7GOUEQ>. You are receiving this because you authored the thread.

[skmcgrail]

Thank you for the logging information @rabarar ,

I was also able to reproduce this on my end as well in both the V2 and V1 Go SDKs, and the AWS CLI. Will follow-up this investigation with the service team.

[skmcgrail]

I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not. If you provide an empty filter at the CLI it will work correctly, same with the V1 SDK (provide a pointer to a structure).

Works

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>1</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
        <Expiration>
            <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
        </Expiration>
        <Filter />
        <ID>abortMulti</ID>
        <Status>Enabled</Status>
    </Rule>
</LifecycleConfiguration>

Doesn't Work

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>1</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
        <Expiration>
            <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
        </Expiration>
        <ID>abortMulti</ID>
        <Status>Enabled</Status>
    </Rule>
</LifecycleConfiguration>

Unfortunately the Go V2 SDK uses a union type, so you can't quite set this in exactly the same way.

[rabarar]

as an aside, attempting to reference the Filter type LifecycleRuleAndOperator in types fails to resolve and is inconsistent with the go-docs.

…

On Jun 9, 2022, at 12:52 PM, Sean McGrail ***@***.***> wrote: I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not. Works <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> <Expiration> <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker> </Expiration> <Filter /> <ID>abortMulti</ID> <Status>Enabled</Status> </Rule> </LifecycleConfiguration> Doesn't Work <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> <Expiration> <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker> </Expiration> <ID>abortMulti</ID> <Status>Enabled</Status> </Rule> </LifecycleConfiguration> — Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6EXR3D2CN2TO6HCKZLVOIOOPANCNFSM5X7GOUEQ>. You are receiving this because you were mentioned.

[skmcgrail]

This code snippet does appear to work, though the serialization is different. It would provide a temporary work around at least:

_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{
		Bucket: aws.String("mcgrails-test-data"),
		LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					Filter: &types.LifecycleRuleFilterMemberPrefix{Value: ""},
					ID:     aws.String("abortMulti"),
				},
			},
		},
	})

This serializes as:

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
    <AbortIncompleteMultipartUpload>
        <DaysAfterInitiation>1</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
    <Expiration>
        <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
    </Expiration>
    <Filter>
        <Prefix></Prefix>
    </Filter>
    <ID>abortMulti</ID>
    <Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>

[skmcgrail]

This seems to be broken in the S3 Control model as well, for different, currently unknown, reasons...

client := s3control.NewFromConfig(cfg)
_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3control.PutBucketLifecycleConfigurationInput{
	AccountId: aws.String("accountId"),
	Bucket:    aws.String("mcgrails-test-data"),
	LifecycleConfiguration: &types.LifecycleConfiguration{
		Rules: []types.LifecycleRule{
			{
				Status: types.ExpirationStatusEnabled,
				Filter: &types.LifecycleRuleFilter{},
				AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
					DaysAfterInitiation: 1,
				},
				Expiration: &types.LifecycleExpiration{
					ExpiredObjectDeleteMarker: true,
				},
				ID: aws.String("abortMulti"),
			},
		},
	},
})

Returns

<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse>
    <Error>
        <Code>InvalidURI</Code>
        <Message>Couldn't parse the specified URI.</Message>
        <URI>bucket/mcgrails-test-data/lifecycleconfiguration</URI>
    </Error>
    <RequestId>Z2AGJDNK1QRRVC4V</RequestId>
    <HostId>3pXY5FhtMiP+Bq0w6wEU786mNkWl4ROwekWZ9JOusTZ5UlKkZZictU0+FkpbZlVwY2oa43KmzdQ=</HostId>
</ErrorResponse>

Seeing the same behavior with the Go V1 SDK and CLI.