Topic / namespace retention policy isn't removed when the retention_policy is removed

[lhotari]

Expected behavior

It would be expected that when topic's or namespace's retention policy is removed that this change would be applied in the Pulsar cluster.

Actual behavior

When the topic or namespace retention policy is removed, this has no effect on the Pulsar cluster side

Additional details

It looks like there's nothing present to remove the retention policy at the topic level:

terraform-provider-pulsar/pulsar/resource_pulsar_topic.go

Lines 995 to 1041 in ab1aecf

	func updateRetentionPolicies(d *schema.ResourceData, meta interface{}, topicName *utils.TopicName) error {
	client := getClientFromMeta(meta).Topics()
	retentionPoliciesConfig := d.Get("retention_policies").(*schema.Set)
	if retentionPoliciesConfig.Len() == 0 {
	return nil
	}
	if !topicName.IsPersistent() {
	return errors.New("ERROR_UPDATE_RETENTION_POLICIES: SetRetention: " +
	"unsupported set retention policies for non-persistent topic")
	}
	if retentionPoliciesConfig.Len() > 0 {
	var policies utils.RetentionPolicies
	data := retentionPoliciesConfig.List()[0].(map[string]interface{})
	policies.RetentionTimeInMinutes = data["retention_time_minutes"].(int)
	policies.RetentionSizeInMB = int64(data["retention_size_mb"].(int))
	// First apply the retention policy
	if err := client.SetRetention(*topicName, policies); err != nil {
	if !isIgnorableTopicPolicyError(err) {
	return fmt.Errorf("ERROR_UPDATE_RETENTION_POLICIES: SetRetention: %w", err)
	} else {
	return backoff.Permanent(fmt.Errorf("ERROR_UPDATE_RETENTION_POLICIES: SetRetention: %w", err))
	}
	}
	// Then verify it was successfully applied using the new polling mechanism
	return waitForTopicConfigUpdate(d, topicName, "RETENTION_POLICIES", func() (bool, error) {
	// Get the current retention policy and check if it matches what we set
	ret, err := client.GetRetention(*topicName, true)
	if err != nil {
	return false, fmt.Errorf("ERROR_UPDATE_RETENTION_POLICIES: GetRetention: %w", err)
	}
	// Check if the values match what we expect
	if ret.RetentionTimeInMinutes != policies.RetentionTimeInMinutes ||
	ret.RetentionSizeInMB != policies.RetentionSizeInMB {
	return false, nil
	}
	return true, nil
	})
	}
	return nil
	}

.

It would be expected to submit DELETE to the REST endpoint /{tenant}/{namespace}/{topic}/retention to remove the topic level retention config.
pulsar-client-go contains RemoveRetention for this purpose.

There are no references to RemoveRetention in the code base. That's why it is assumed that this problem exists also for namespace policy.

The PR apache/pulsar-client-go#1433 made a change to return nil if retention isn't set for topic / namespace. That is most likely useful in resolving this issue.

There's a new test file for testing resetting topic policy configs to defaults, that's https://github.com/streamnative/terraform-provider-pulsar/blob/master/pulsar/resource_pulsar_topic_defaults_test.go . It could be useful to add the test for removing the retention policy there. A similar test would be needed for removing the retention policy from the namespace policy.