Adding unfollow action in ism to invoke stop replication for ccr

[aggarwalShivani]

Issue #, if available: #726

This is to add support for unfollow feature in ism.
It depends on two PRs already raised and under-review in common-utils project as well as CCR project. Detailed information about the proposed solution is explained there.

Description of changes:

1. Imported replication utilities from common-utils project
2. Added a new action "unfollow" for ism policies. It invokes the stop-replication utility from common-utils.
3. Added one integration test - (with installation of ccr plugin in testClusters.integTest )

CheckList:

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[aggarwalShivani]

Hi,

1. As the other prior two PRs in common-utils and ccr are not yet merged, the checks on CI would not go through. Also, in build.gradle, locally placed artifacts for these projects are being used. I shall remove those changes once the artifacts are merged and available in repo.
2. I've also added a UT (AttemptUnfollowStepsTest.kt) - but that is WIP as its not working correctly, as I've discussed with @bowenlan-amzn. I've tried two approaches to mock the response value, by using mockStatic (to mock ReplicationInterface) and mock spy (to mock AttemptUnfollowStep.performStopAction() alone) but havent been successful.
   Would like to hear if there any suggestions in this area.

Looking forward for the review, Thanks!

[aggarwalShivani]

Hi Reviewers,
As suggested by @bowenlan-amzn, I'm sharing some details on how one can bring up the environment with ccr and ism for this use-case. I've done this for a linux environment.

Environment setup details

1. Setup Opensearch

$ git clone https://github.com/opensearch-project/OpenSearch.git 
$ cd Opensearch
### Ensure JAVA_HOME, PATH are set. 
$ ./gradlew assemble
$ export OPENSEARCH_HOME=<opensearch-home-dir>   ## where you wish to setup Opensearch
$ export OPENSEARCH_BUILD=distribution/archives/linux-tar/build/install/opensearch-$(./gradlew properties -q | grep -E '^version:' | awk '{print $2}')
$ cp -Rf $OPENSEARCH_BUILD/* $OPENSEARCH_HOME

(Above steps can be avoided if you already have opensearch setup)
Installation of plugins:

 $ cd $OPENSEARCH_HOME
 $ ./bin/opensearch-plugin install <location-of-opensearch-job-scheduler-3.0.0.0-SNAPSHOT.zip>
 $ ./bin/opensearch-plugin install <location-of-opensearch-cross-cluster-replication-3.0.0.0-SNAPSHOT.zip>
 $ ./bin/opensearch-plugin install <location-of-opensearch-index-management-3.0.0.0-SNAPSHOT.zip>

For ex. bin/opensearch-plugin install file:///home/username/ccr/opensearch-cross-cluster-replication-3.0.0.0-SNAPSHOT.zip

2. Setup Configs
We need to bring up one cluster as the leader and other as the follower (that replicates the leader indices).
This small snippet is a function to generate opensearch.yml config files for the two clusters. Run this from $OPENSEARCH_HOME location.

configure_opensearch() {
	 role=$1
	 mkdir -p ccrtest/$role/data/  mkdir -p ccrtest/$role/logs
	 cp config/opensearch.yml config/opensearch-$role.yml
	 echo "
	 plugins.security.disabled: true
	 cluster.name: ${role}Cluster
	 path.data: $OPENSEARCH_HOME/ccrtest/${role}/data
	 path.logs: $OPENSEARCH_HOME/ccrtest/${role}/logs
	
	 " >> config/opensearch-${role}.yml
}
	
// Invoke the function twice. It generates files opensearch-leader.yml and  opensearch-follower.yml at $OPENSEARCH_HOME/config.
configure_opensearch "leader"
configure_opensearch "follower"

(This is using the default opensearch.yml file and just adding minimal required configs for CCR)

3. Run Opensearch clusters
   Leader cluster as 127.0.0.1:9200, follower as 127.0.0.1:9201 using the generated config files respectively in step 2.

4. Start Replication using CCR REST APIs
   Create an index in leader cluster, and setup replication in follower cluster.

5. Start ISM
   Create an ism policy to run the action "unfollow" on the desired indices.

For steps 3-5, I've added the steps in a shell script for convenience - test-unfollow-script.txt. (Changed the extension to .txt as .sh files cannot be uploaded here)

I hope that helps : )

[bowenlan-amzn]

I read the CCR doc and see if we stop relication, we won't be able to resume anymore.
In ISM, probably the normal workflow is after the leader index rollover, we can then stop the replication

I'm wondering how do we know that so to prevent early stopping the replication.

not requiring this since probably we can just mention this caveat in the documentation, and give a long waiting time before stop relication in the follower cluster

[aggarwalShivani]

Yes, replication cannot be resumed on an index after stopping (or pausing more than 12h).

Hmm, the major use-case we had identified in ISM+CCR case was - say ISM is setup in both leader and follower clusters, for deletion and other housekeeping operations in the respective clusters.
In the follower cluster, even if the user has setup a policy and intends to delete some-pattern* indices, it would not be allowed as they would be still read-only due to ongoing replication, which needs to be stopped first.

So in such cases, users could chain the actions to be preceeded by stop-replication first (before any other write actions).

[bowenlan-amzn]

Understood.

I'm just a little worried about one part: the follower index cannot know whether leader index finishes indexing, so it may stop relication early unexpectedly

but this can be waited for community feedback whether it's needed

[bowenlan-amzn]

remove

[aggarwalShivani]

Sure 👍

I had added this to get some suggestions on writing mock-uts for AttemptUnfollowStep.kt as I have tried with two different approaches using a function performStopAction() and directly invoking ReplicationPluginInterface.suspendUntil, but to no success.

Do you have any suggestions on either partially mocking AttempUnfollowStep or mocking ReplicationPluginInterface.stopReplication in this case?

[bowenlan-amzn]

Wondering did you try mock the client behavior like this, I suppose it would be pretty similar since both return acknowledge response

index-management/src/test/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/AttemptRolloverStepTests.kt

Lines 172 to 205 in 4d8ef69

	private fun getIndicesAdminClient(
	rolloverResponse: RolloverResponse?,
	aliasResponse: AcknowledgedResponse?,
	rolloverException: Exception?,
	aliasException: Exception?,
	): IndicesAdminClient {
	assertTrue(
	"Must provide one and only one response or exception",
	(rolloverResponse != null).xor(rolloverException != null),
	)
	assertTrue(
	"Must provide one and only one response or exception",
	(aliasResponse != null).xor(aliasException != null),
	)
	return mock {
	doAnswer { invocationOnMock ->
	val listener = invocationOnMock.getArgument<ActionListener<AcknowledgedResponse>>(1)
	if (rolloverResponse != null) {
	listener.onResponse(rolloverResponse)
	} else {
	listener.onFailure(rolloverException)
	}
	}.whenever(this.mock).rolloverIndex(any(), any())
	doAnswer { invocationOnMock ->
	val listener = invocationOnMock.getArgument<ActionListener<AcknowledgedResponse>>(1)
	if (aliasResponse != null) {
	listener.onResponse(aliasResponse)
	} else {
	listener.onFailure(aliasException)
	}
	}.whenever(this.mock).aliases(any(), any())
	}
	}

[aggarwalShivani]

Hey,
Yes I did try this. But there is a slight difference in these two classes, which is the trouble.

AttemptRolloverStep executes the rollover like this

val response: RolloverResponse = context.client.admin().indices().suspendUntil { rolloverIndex(request, it) }

In AttemptRolloverStepTests, context.client has been mocked in a way that when the rolloverIndex() is invoked by the test, it would return the value as directed by the mocked function.

But in the AttemptUnfollowStep.kt class, the execute() method invokes it like this

val response: AcknowledgedResponse =
                ReplicationPluginInterface.suspendUntil {
                    this.stopReplication(
                        context.client as NodeClient,
                        stopIndexReplicationRequestObj,
                        it,
                    )
                }

Here, I can mock the context.client and other params, but the test needs to mock ReplicationPluginInterface and ReplicationPluginInterface.suspendUntil to ultimately mock the response of stopReplication() function. If that does not happen, it would run the actual implementation and not the mocked one - which does not work in the UT.
I'm unable to mock ReplicationPluginInterface as it is static. Tried another approach using mock spy (to mock AttemptUnfollowStep.performStopAction() partially), and that hasn't worked either.

[aggarwalShivani]

Hi @bowenlan-amzn , Any thoughts here?

[bowenlan-amzn]

@aggarwalShivani it's published from this action
https://github.com/opensearch-project/cross-cluster-replication/blob/e4f2143f294c32936f9d3652f3db4a7eaefd274a/.github/workflows/maven-publish.yml

would you mind just raise a PR in ccr and update this file to be same as the ISM one
We can see this action never run in ccr https://github.com/opensearch-project/cross-cluster-replication/actions

[aggarwalShivani]

Thanks for the suggestion @bowenlan-amzn.
Noticed that the github action file is there in 2.x branch but not in main branch of CCR.
For main, I have created a PR in CCR for this
opensearch-project/cross-cluster-replication#1509. Should see if this helps resolving the issue.

[aggarwalShivani]

Hi all
Status update:
Changes for this are ready to be pushed. However, we have a blocking issue recently. Details below:

• The common-utils project has moved to using opensearch-3.0.0-alpha1-SNAPSHOT. One breaking change here is that classes like AcknowledgedRequest, AcknowledgedResponse etc are moved from org.opensearch.action.support.master to org.opensearch.action.support.clustermanager.
• ccr and ism projects are still using opensearch-3.0.0-SNAPSHOT and classes from old org.opensearch.action.support.master package. This inconsistency results in incompatibility issues while using these classes.
• As discussed over slack with @bowenlan-amzn and @ankitkala, it is suggested to raise PRs on 2.x branches first, until the development issue with the main branches is resolved.
• For this change, I've raised a backport PR on 2.x branch ([2.x] Adding new stop-replication action in ism #1370)

[bowenlan-amzn]

Have reviewed all the PRs, and setup my local run on 2.19.1 and able to succeed.
Great job! Thank you for seeing it through to the end!

[aggarwalShivani]

Have reviewed all the PRs, and setup my local run on 2.19.1 and able to succeed. Great job! Thank you for seeing it through to the end!

Thank You for the review and valuable feedback throughout
For the main branch, all review comments received so far have been addressed and PRs are ready for review - this and common-utils and ccr too.

Pls note:
1] CI Checks in ccr and ism would fail until common-utils is merged.
2] For CCR plugin publishing though, still need to see if this change is sufficient.

[bowenlan-amzn]

Thank you very much!