Implementing Finalize Backfill Hook for All Backfila Clients by mminkoffs · Pull Request #358 · cashapp/backfila

mminkoffs · 2023-12-01T00:26:06Z

No description provided.

client-dynamodb-v2/src/main/kotlin/app/cash/backfila/client/dynamodbv2/DynamoDbBackfill.kt

shellderp · 2023-12-01T17:40:12Z

service/src/main/kotlin/app/cash/backfila/service/runner/statemachine/BatchAwaiter.kt

        backfillRunner.factory.queryFactory,
      )
-      if (partitions.all { it.run_state == BackfillState.COMPLETE }) {
+      partitions


we should do these server side changes in a separate PR, and get them deployed before making the client side changes. Otherwise, a client developer might implement the finalize method when it doesn't do anything yet.

Also, there are problems with this code I mentioned in the previous PR. I will see if I can take a look at writing this correctly in the next few days

Sounds good - let me know a commit when you're done and we can either integrate it with this PR or externally!

Maybe we want a COMPLETE_&_FINALIZED state to make that clearer?

@shellderp just curious if any progress was made here?

I know how to code it up, read the version and retry both transactions if the 2nd read a different version. I can do this in a followup if you get the code merged as is (just add tests in this PR)

client/src/main/proto/app/cash/backfila/client_service.proto

mpawliszyn

We need much more testing as well.

Also the embedded backfila has to be updated as well.

Overall though it seems to be heading in a good direction.

mpawliszyn · 2023-12-04T14:12:23Z

service/src/main/kotlin/app/cash/backfila/service/runner/statemachine/BatchAwaiter.kt

+        when (e) {
+          is HttpException ->
+            when (e.code()) {
+              404 -> logger.info { "No finalization endpoint found for ${backfillRunner.backfillName}, skipping" }


This should show an event in the backfill.

I wonder if we want something to indicate versions for a backfill. Like what Backfills have what capabilities.

We might want an event even in the success case?

mpawliszyn · 2023-12-04T14:13:18Z

service/src/main/kotlin/app/cash/backfila/service/runner/statemachine/BatchAwaiter.kt

      )

-      // If all states are COMPLETE the whole backfill will be completed.
+      // If all states are COMPLETE, for every other partition, the whole backfill will be completed.


Should we have a state that is Finalizing?

I don't think the extra complexity is needed

mpawliszyn · 2023-12-04T14:15:08Z

client/src/main/kotlin/app/cash/backfila/client/BackfillConfig.kt

  val dryRun: Boolean,
 )
+
+data class FinalizeBackfillConfig<Param : Any>(


I wonder if we need better names and/or structure for these.

PartitionlessBackfillConfig? Or WholeBackfillConfig?

This is config that is meant to represent the whole run. WholeRunBackfillConfig? Also add the accessor above. And probably an accessor to Prepare here too.

I'm ok with it as is

mpawliszyn · 2023-12-05T15:13:34Z

service/src/main/kotlin/app/cash/backfila/service/runner/statemachine/BatchAwaiter.kt

        backfillRunner.factory.queryFactory,
      )
-      if (partitions.all { it.run_state == BackfillState.COMPLETE }) {
+      partitions


Maybe we want a COMPLETE_&_FINALIZED state to make that clearer?

mpawliszyn · 2023-12-05T15:17:20Z

client-base/src/test/kotlin/app/cash/backfila/client/fixedset/FixedSetBackfill.kt

  abstract fun runOne(row: FixedSetRow, backfillConfig: BackfillConfig<Param>)
+
+  /**
+   * Override this to do any work after the backfill completes.


FYI This is a test only implementation.

mpawliszyn · 2023-12-05T15:23:51Z

client-base/src/test/kotlin/app/cash/backfila/client/fixedset/FixedSetBackfillOperator.kt

+    val config = parametersOperator.constructBackfillConfig(request)
+    backfill.finalize(config)
+
+    return FinalizeBackfillResponse.Builder()


I like the // Return empty 200 to indicate it was successful similar comment above. Does it make sense to make that more apparant?

mpawliszyn · 2023-12-05T15:26:36Z

client-s3/src/main/kotlin/app/cash/backfila/client/s3/S3DatasourceBackfill.kt

  abstract val recordStrategy: RecordStrategy<R>
+
+  /**
+   * Override this to do any work after the backfill completes.


I think we need to update the comment a bit:
Override this to do any final work after the backfill completes. Only one successful call is expected in your distributed system and this call must be idempotent.

Does that make sense?

mpawliszyn · 2023-12-05T15:31:06Z

...kfill/src/main/kotlin/app/cash/backfila/service/selfbackfill/BackfillRegisteredParameters.kt

 ) : HibernateBackfill<DbRegisteredBackfill, Id<DbRegisteredBackfill>, NoParameters>() {

-  override fun runOne(id: Id<DbRegisteredBackfill>, config: BackfillConfig<NoParameters>) {
+  override fun runOne(pkey: Id<DbRegisteredBackfill>, config: BackfillConfig<NoParameters>) {


undo the changes to this class

mpawliszyn · 2023-12-05T15:34:27Z

service/src/main/kotlin/app/cash/backfila/service/runner/BackfillRunner.kt

    }
  }

+  suspend fun finalize(state: FinalizeState) {


@shellderp is this right wrt coroutines?

mpawliszyn · 2023-12-05T15:35:00Z

service/src/main/kotlin/app/cash/backfila/service/runner/statemachine/BatchAwaiter.kt

  }

-  private fun completePartition() {
+  private suspend fun completePartition() {


Also this, is it right wrt coroutines? @shellderp

mminkoffs added 3 commits November 30, 2023 16:21

Implementing Finalize logic for Dynamov2 client

91037c5

More implementation

73d0e05

More implementation x2

1cca7cb

mmontgomery-square reviewed Dec 1, 2023

View reviewed changes

client-dynamodb-v2/src/main/kotlin/app/cash/backfila/client/dynamodbv2/DynamoDbBackfill.kt Show resolved Hide resolved

mminkoffs marked this pull request as ready for review December 1, 2023 15:59

shellderp reviewed Dec 1, 2023

View reviewed changes

client/src/main/proto/app/cash/backfila/client_service.proto Show resolved Hide resolved

mpawliszyn reviewed Dec 5, 2023

View reviewed changes

Conversation

mminkoffs commented Dec 1, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mpawliszyn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants