Bug#37023549 (1/5) Improve NdbApi SPJ scan protocol timeout handling

Backport to 7.6

Add error insert to SPJ to allow a 'hard' timeout + failure to
close to be simulated.

Change-Id: Ie2479be2496ff4eb8ebe32547d8a8cb513708d26

Bug#37023549 (2/5) Improve NdbApi SPJ scan protocol timeout handling

Add SPJ scan testing

SPJ scans are added to the test as they have a different implementation
in the API and so need to be covered separately.

Due to bugs in SPJ, this test causes data node failure without
further fixes.

Change-Id: I1b7dd00a3bbaaa88aaf689de2ffc617afd041f62

Bug#37023549 (3/5) Improve NdbApi SPJ scan protocol timeout handling

Modify TC to disconnect bad APIs

TC detects in many cases when an API sends a signal which is
unexpected due the state discovered on an ApiConnectRecord.

Some cases are understood and expected, and others are not,
resulting in assertion failures in TC -> data node failure.

These can be useful for finding logic errors but can have
a large impact on users.

This patch modifies TC to handle some of these situations
differently :
 - Log information about the mismatch
 - Disconnect the API(s) involved
   (API sending the signal, API owning the ConnectRecord)

This will result in :
  - The API(s) being disconnected by all data nodes
  - API failure handling on all data nodes performing cleanup
    of the API's transaction objects, continuing with commit
    or rollback + then releasing them
  - APIs having to reinitialise their connection state to
    the cluster.

API disconnect + reconnect is generally less work, quicker
and less risky for system availability than a data node
restart.

This change in functionality is tested to some extent by
the SPJ timeout scenario in ndb_scan_protocol_timeout, but
other situations leading to state mismatch may exist.

Change-Id: Id88df14141983b4fef71e74ce275d6b4e26c6c58

Bug#37023549 (4/5) Improve NdbApi SPJ scan protocol timeout handling

On timeout, make SPJ API mark scans as needing close

SPJ API is aligned to the (fixed) behaviour of NdbApi on protocol
timeout handling - marking scans as needing close at the kernel.

This causes close() to attempt to close the scan, which will
succeed in cases where the scan has timed out due to LOAD.

Additionally, SPJ API is modified to log when these protocol
timeout handling paths are used, and what the result is.

The SPJ tests in ndb_scan_protocol_timeout for the 'LOAD' case
now pass with no leaks, but the BUG case needs a further fix.

Change-Id: I90e4421228f3db9c2de1de7ebb48ae81123abf68

Bug#37023549 (5/5) Improve NdbApi SPJ scan protocol timeout handling

SPJ Timeout Release on Close timeout

In the case where there is a timeout attempting to close an SPJ scan
at the kernel, this patch sets the ReleaseOnClose variable so that
the kernel side ApiConnectRecord will not be reused.

This aligns with the normal scan behaviour in the close-failed-timeout
case.

The ndb_scan_protocol_timeout testcase result is updated with the
expected results for SPJ scan timeouts due to load + bugs

Change-Id: I04b199759415c3e0d8f6f6546551c0271fac768c

Author: frazerclement

mysql-test/suite/ndb/r/ndb_scan_protocol_timeout.result

@@ -304,6 +304,70 @@ Leaks
 select count(1) as ops from ndbinfo.cluster_operations;
 ops
 0
+
+-------------------------------
+SPJ request timeout due to load
+-------------------------------
+Standalone
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b where t1.a > 70;
+ERROR HY000: Got error 4008 'Receive from NDB failed' from NDBCLUSTER
+Check pk lookups
+Clear error condition
+Check transaction leaks
+Leaks
+0
+select count(1) as ops from ndbinfo.cluster_operations;
+ops
+0
+
+-------------------------------
+SPJ request timeout due to load
+-------------------------------
+As part of a stateful transaction
+begin;
+insert into test.t1 values (54,54);
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b where t1.a > 70;
+ERROR HY000: Got error 4008 'Receive from NDB failed' from NDBCLUSTER
+Clear error condition
+Check pk lookups
+Check transaction leaks
+Leaks
+0
+select count(1) as ops from ndbinfo.cluster_operations;
+ops
+0
+
+------------------------------
+SPJ request timeout due to bug
+------------------------------
+Standalone
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b;
+ERROR HY000: Got error 4008 'Receive from NDB failed' from NDBCLUSTER
+Check pk lookups
+Clear error condition
+Check transaction leaks
+Leaks
+1
+select count(1) as ops from ndbinfo.cluster_operations;
+ops
+0
+
+-------------------------------
+SPJ request timeout due to bug
+------------------------------
+As part of a stateful transaction
+begin;
+insert into test.t1 values (54,54);
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b;
+ERROR HY000: Got error 4008 'Receive from NDB failed' from NDBCLUSTER
+Check pk lookups
+Clear error condition
+Check transaction leaks
+Leaks
+1
+select count(1) as ops from ndbinfo.cluster_operations;
+ops
+0
 SET SESSION debug=@save_debug;
 Normal requests
 ---------------

mysql-test/suite/ndb/t/ndb_scan_protocol_timeout.test

@@ -2,6 +2,7 @@
 --echo # We are using some debug-only features in this test
 -- source include/have_debug.inc
 -- source have_ndb_error_insert.inc
+-- source include/have_query_cache_disabled.inc
 
 #
 # Test of scan protocol timeout behaviour
@@ -37,6 +38,12 @@
 #                 the ability of scan + transaction
 #                 close() to cleanup a scan
 #
+#                 17123 (SPJ)
+#                 SPJ blocks SCAN_FRAGCONF processing
+#                 This stalls nextResult() and also
+#                 the ability of scan + transaction
+#                 close() to cleanup
+#
 # Timeout is expected to be mostly a result of
 # operations taking too long, and occasionally a
 # result of bugs
@@ -45,9 +52,9 @@
 # 'scan taking too long' and 8124 to represent
 # 'problem closing scan / bugs'.
 #
-# Ordered (e.g. by API) and unordered scans are taken
-# separately as they have separate implementations
-# inside NdbApi
+# Ordered (e.g. by API) and unordered scans and SPJ
+# scans are taken separately as they have separate
+# implementations inside NdbApi
 #
 # A .cnf file is used to avoid TDDT getting in the
 # way of API timeout testing
@@ -441,6 +448,169 @@ eval select $api_conn_count - $start_api_conn_count as Leaks;
 select count(1) as ops from ndbinfo.cluster_operations;
 #select * from ndbinfo.cluster_operations;
 
+--echo
+--echo -------------------------------
+--echo SPJ request timeout due to load
+--echo -------------------------------
+--echo Standalone
+
+# Error insert 5112 causes SPJ request timeout, but close is unaffected
+--exec $NDB_MGM -e "all error 5112" >> $NDB_TOOLS_OUTPUT
+
+--sorted_result
+--error 1296
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b where t1.a > 70;
+
+# Checking pk lookups checks that all usable ApiConnectRecords can
+# be used for a different transaction
+--echo Check pk lookups
+--disable_query_log
+--disable_result_log
+--let $i=0
+while ($i < $keycount)
+{
+  --eval select * from test.t1 where a=$i;
+  --inc $i
+}
+--enable_result_log
+--enable_query_log
+
+--echo Clear error condition
+--exec $NDB_MGM -e "all error 0" >> $NDB_TOOLS_OUTPUT
+--sleep $STABILISATION_SECS
+
+--echo Check transaction leaks
+--source ndb_get_api_connect_count.inc
+--disable_query_log
+eval select $api_conn_count - $start_api_conn_count as Leaks;
+--let $start_api_conn_count = $api_conn_count
+--enable_query_log
+select count(1) as ops from ndbinfo.cluster_operations;
+
+--echo
+--echo -------------------------------
+--echo SPJ request timeout due to load
+--echo -------------------------------
+--echo As part of a stateful transaction
+
+# Error insert 5112 causes SPJ request timeout, but close is unaffected
+--exec $NDB_MGM -e "all error 5112" >> $NDB_TOOLS_OUTPUT
+
+begin;
+insert into test.t1 values (54,54);
+
+--sorted_result
+--error 1296
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b where t1.a > 70;
+
+--echo Clear error condition
+--exec $NDB_MGM -e "all error 0" >> $NDB_TOOLS_OUTPUT
+--sleep $STABILISATION_SECS
+
+# Checking pk lookups checks that all usable ApiConnectRecords can
+# be used for a different transaction
+--echo Check pk lookups
+--disable_query_log
+--disable_result_log
+--let $i=0
+while ($i < $keycount)
+{
+  --eval select * from test.t1 where a=$i;
+  --inc $i
+}
+--enable_result_log
+--enable_query_log
+
+--echo Check transaction leaks
+--source ndb_get_api_connect_count.inc
+--disable_query_log
+eval select $api_conn_count - $start_api_conn_count as Leaks;
+--let $start_api_conn_count = $api_conn_count
+--enable_query_log
+select count(1) as ops from ndbinfo.cluster_operations;
+
+--echo
+--echo ------------------------------
+--echo SPJ request timeout due to bug
+--echo ------------------------------
+--echo Standalone
+
+# Error insert 17123 causes SPJ request and close() to timeout
+--exec $NDB_MGM -e "all error 17123" >> $NDB_TOOLS_OUTPUT
+
+--sorted_result
+--error 1296
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b;
+
+# Checking pk lookups checks that all usable ApiConnectRecords can
+# be used for a different transaction
+--echo Check pk lookups
+--disable_query_log
+--disable_result_log
+--let $i=0
+while ($i < $keycount)
+{
+  --eval select * from test.t1 where a=$i;
+  --inc $i
+}
+--enable_result_log
+--enable_query_log
+
+--echo Clear error condition
+--exec $NDB_MGM -e "all error 0" >> $NDB_TOOLS_OUTPUT
+--sleep $STABILISATION_SECS
+
+--echo Check transaction leaks
+--source ndb_get_api_connect_count.inc
+--disable_query_log
+eval select $api_conn_count - $start_api_conn_count as Leaks;
+--let $start_api_conn_count = $api_conn_count
+--enable_query_log
+select count(1) as ops from ndbinfo.cluster_operations;
+
+
+--echo
+--echo -------------------------------
+--echo SPJ request timeout due to bug
+--echo ------------------------------
+--echo As part of a stateful transaction
+# Error insert 17123 causes SPJ request and close() to timeout
+--exec $NDB_MGM -e "all error 17123" >> $NDB_TOOLS_OUTPUT
+
+begin;
+insert into test.t1 values (54,54);
+--sorted_result
+--error 1296
+select t1.b, t2.b from test.t1 join test.t2 on t2.a = t1.b;
+
+# Checking pk lookups checks that all usable ApiConnectRecords can
+# be used for a different transaction
+--echo Check pk lookups
+--disable_query_log
+--disable_result_log
+--let $i=0
+while ($i < $keycount)
+{
+  --eval select * from test.t1 where a=$i;
+  --inc $i
+}
+--enable_result_log
+--enable_query_log
+
+--echo Clear error condition
+--exec $NDB_MGM -e "all error 0" >> $NDB_TOOLS_OUTPUT
+--sleep $STABILISATION_SECS
+
+--echo Check transaction leaks
+--source ndb_get_api_connect_count.inc
+--disable_query_log
+eval select $api_conn_count - $start_api_conn_count as Leaks;
+--let $start_api_conn_count = $api_conn_count
+--enable_query_log
+select count(1) as ops from ndbinfo.cluster_operations;
+
+
+
 SET SESSION debug=@save_debug;
 
 # Show that after all of the above, with error insertions disabled,
@@ -469,4 +639,3 @@ select count(1) as ops from ndbinfo.cluster_operations;
 drop table test.t1;
 drop table test.t2;
 
---remove_file $NDB_TOOLS_OUTPUT

storage/ndb/src/kernel/blocks/ERROR_codes.txt

@@ -38,7 +38,7 @@ Next DBTUX 12010
 Next SUMA 13060
 Next LGMAN 15002
 Next TSMAN 16002
-Next DBSPJ 17000
+Next DBSPJ 17124
 Next TRIX 18004
 Next DBUTIL 19002
 

storage/ndb/src/kernel/blocks/dbspj/DbspjMain.cpp

@@ -1,5 +1,5 @@
 /*
-   Copyright (c) 2012, 2023, Oracle and/or its affiliates.
+   Copyright (c) 2012, 2024, Oracle and/or its affiliates.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License, version 2.0,
@@ -3758,6 +3758,15 @@ Dbspj::execSCAN_FRAGCONF(Signal* signal)
                      DBLQH);
 #endif
 
+  if (ERROR_INSERTED(17123)) {
+    jam();
+    g_eventLogger->info(
+        "Dbspj %u : Error insert stalling SCAN_FRAGCONF for 0.5s", instance());
+    sendSignalWithDelay(reference(), GSN_SCAN_FRAGCONF, signal, 500,
+                        signal->getLength());
+    return;
+  }
+
   Ptr<ScanFragHandle> scanFragHandlePtr;
   ndbrequire(getGuardedPtr(scanFragHandlePtr, conf->senderData));
   Ptr<TreeNode> treeNodePtr;

storage/ndb/src/kernel/blocks/dbtc/Dbtc.hpp

@@ -2051,10 +2051,11 @@ class Dbtc
   // Generated statement blocks
   void warningHandlerLab(Signal* signal, int line);
   void systemErrorLab(Signal* signal, int line);
-  void sendSignalErrorRefuseLab(Signal* signal);
+  void handleSignalStateProblem(Signal *signal,
+                                ApiConnectRecordPtr apiConnectptr,
+                                NodeId signalNodeId, Uint32 context);
   void scanTabRefLab(Signal* signal, Uint32 errCode);
   void diFcountReqLab(Signal* signal, ScanRecordPtr);
-  void signalErrorRefuseLab(Signal* signal);
   void abort080Lab(Signal* signal);
   void sendKeyInfoTrain(Signal* signal,
                         BlockReference TBRef,