pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr failed

[cockroach-teamcity]

pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test.TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr failed with artifacts on master @ 6fb3814bbb5fc0cd6365d21d884ed2dbd7bdf6f8:

          SELECT count(*)=$successfulStageCount*2 FROM  multiregion_db.public.table_regional_by_row@idx_v
          ----
          true
    framework.go:562: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:49:
          stage-query phase=PostCommitNonRevertiblePhase stage=:
          SELECT index_name FROM crdb_internal.table_indexes WHERE descriptor_name='table_regional_by_row' ORDER BY index_name;
          ----
          idx_v
          table_regional_by_row_pkey
          table_regional_by_row_rowid_key
    test_server_shim.go:93: cluster virtualization disabled due to issue: #156308 (expected label: C-test-failure)
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 10.000516156s
    test_server_shim.go:93: cluster virtualization disabled due to issue: #156308 (expected label: C-test-failure)
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 20.000681027s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 30.001220943s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 40.00133484s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 50.001553977s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m0.001590192s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m10.001668243s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m20.006235411s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m30.009314828s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m40.013037702s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m50.060644168s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m0.066143734s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m10.066271693s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m20.080328116s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m30.080606352s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m40.080733808s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m50.108741603s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 3m0.11252418s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 3m10.112917219s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 3m20.114853454s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 3m30.115004884s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 3m40.115116542s
    datadriven.go:357: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11934/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test_/sctestbackupmrccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 3m50.115174442s
    panic.go:615: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/4ba3032e14bb8edea9365598514cf290/logTestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr2238141748
--- FAIL: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr (230.49s)
=== RUN   TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24/post_1_rollback_stage_exec/restore_database_with_schema_only
    backup.go:624: post-RESTORE schema change was expected to not succeed
--- FAIL: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24/post_1_rollback_stage_exec/restore_database_with_schema_only (28.04s)
=== RUN   TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24
    test_server_shim.go:93: cluster virtualization disabled due to issue: #156308 (expected label: C-test-failure)
--- FAIL: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24 (161.36s)
=== RUN   TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24/post_1_rollback_stage_exec
--- FAIL: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24/post_1_rollback_stage_exec (56.36s)
=== RUN   TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24/post_1_rollback_stage_exec/restore_database
    backup.go:624: post-RESTORE schema change was expected to not succeed
--- FAIL: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_16_of_24/post_1_rollback_stage_exec/restore_database (26.59s)

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr failed #160314 pkg/ccl/schemachangerccl/sctestbackupmrccl/sctestbackupmrccl_test: TestBackupRollbacksMixedVersion_multiregion_alter_table_alter_primary_key_rbr failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-25.4]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-58649

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.

[bghal]

Still need to refactor the test to avoid the timeout -- seems like #159914 and #160344 were not enough

[fqazi]

Fixed via: #161646