fix(consensus)!: refactor execution fork handling, better finalization perf (tari-project#1425)

Description
---
fix(consensus)!: refactor execution fork handling, better finalization
perf

Motivation and Context
---
Finalizing big blocks (>2000 finalizing txs) would take around a minute
in tests. This PR reduces this to < 10s for similar payloads.

The finalizing step simply inserts the transaction ID into an index to
mark it as finalized.

How Has This Been Tested?
---
Manually

What process can a PR reviewer use to test or verify this change?
---
Works as before

Breaking Changes
---

- [ ] None
- [x] Requires data directory to be deleted
- [ ] Other - Please specify

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added grouping support for databases and shard group display in the
database inspector UI.
- Introduced explicit transaction execution querying, locking, and
finalization methods in storage layers.
- Added new transaction validation result struct to clarify sequencing
logic.
- Added TypeScript interfaces for new account info and wallet
transaction types.

- **Improvements**
- Consolidated and simplified transaction and execution data models,
removing legacy fields and unifying execution handling.
- Enhanced consensus and transaction pool logic for clearer transaction
sequencing, validation, and execution preparation.
- Streamlined abort reason handling with a refined and reduced set of
abort causes.
  - Centralized leader fee calculation logic for consistency.
- Updated transaction execution and block execution models to use richer
block and transaction identifiers.
- Improved error handling and logging in consensus and transaction
processing flows.
  - Refined transaction finalization and decision assertions in tests.
- Replaced explicit unwraps with safer access patterns in transaction
result handling.
- Replaced `chrono` with `time` crate for timestamp handling across
storage and template management modules.
- Updated web UI and RPC handlers to reflect new transaction execution
data structures.
- Simplified transaction handling by removing legacy executed
transaction abstractions.
- Added utility functions for current timestamp retrieval and
encoding/decoding of datetime values.
- Fixed template selection logic for swap and faucet templates in the
tariswap test bench.
- Improved transaction pool insertion to accept explicit decision
parameters.
- Refined leader timeout suspension handling in consensus proposal
processing.
  - Updated consensus no-vote handling to propagate detailed reasons.
- Enhanced transaction pool insertion to accept explicit decision
parameters.
- Adjusted transaction execution and block transaction execution models
to explicitly store leaf blocks and transaction IDs.
- Consolidated transaction execution retrieval and finalization logic in
storage layers.
- Simplified transaction record by removing execution and finalization
fields, delegating to storage queries.
- Updated CLI, bindings, and clients to align with new transaction
execution APIs and data structures.
  - Improved internal documentation and code comments for clarity.

- **Bug Fixes**
- Removed duplicate transaction input validation during validator node
bootstrap.

- **Refactor**
- Standardized transaction execution method signatures and storage keys,
adopting richer block and transaction identifiers.
- Updated test harnesses, CLI tools, and bindings to align with new
transaction execution APIs and data structures.
- Removed SQL-related configuration and code from consensus test
builders and validators.
- Modularized transaction preparation logic by input type and improved
lock acquisition strategies.
- Refactored consensus transaction manager and HotStuff logic to unify
transaction record handling.
- Cleaned up abort handling and removed redundant methods from
transaction and execution models.
- Reorganized database column families and queries to include block
height and improve indexing.
- Simplified and unified deserialization and serialization logic in
wallet storage.
  - Removed executed transaction module and related legacy abstractions.
- Renamed pacemaker failure suspension methods to timeout suspension for
clarity.
- Added TypeScript bindings for new types and updated existing bindings
to match data model changes.

- **Chores**
  - Removed unused dependencies including `chrono` and `strum_macros`.
  - Updated package scripts to separate formatting and build steps.
  - Cleaned up legacy code paths and imports.
- Removed SQL database configuration from consensus test builders and
validators.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: sdbondi

Cargo.lock

@@ -29,15 +29,14 @@ pub trait TransactionExecutor {
 
     fn execute(
         &self,
-        transaction: Transaction,
+        transaction: &Transaction,
         state_store: ReadOnlyMemoryStateStore,
         virtual_substates: VirtualSubstates,
     ) -> Result<ExecutionOutput, Self::Error>;
 }
 
 #[derive(Debug, Clone)]
 pub struct ExecutionOutput {
-    pub transaction: Transaction,
     pub result: ExecuteResult,
 }
 
@@ -106,7 +105,7 @@ where TTemplateProvider: TemplateProvider<Template = LoadedTemplate>
 
     fn execute(
         &self,
-        transaction: Transaction,
+        transaction: &Transaction,
         state_store: ReadOnlyMemoryStateStore,
         virtual_substates: VirtualSubstates,
     ) -> Result<ExecutionOutput, Self::Error> {
@@ -136,7 +135,7 @@ where TTemplateProvider: TemplateProvider<Template = LoadedTemplate>
         );
         let result = processor.execute(transaction.clone())?;
 
-        Ok(ExecutionOutput { transaction, result })
+        Ok(ExecutionOutput { result })
     }
 }
 

applications/tari_dan_app_utilities/src/transaction_executor.rs

@@ -372,12 +372,7 @@ pub async fn handle_get_all(
         .wallet_sdk()
         .transaction_api()
         .fetch_all(req.status, req.component)?;
-    Ok(TransactionGetAllResponse {
-        transactions: transactions
-            .into_iter()
-            .map(|tx| (tx.transaction, tx.finalize, tx.status, tx.last_update_time))
-            .collect(),
-    })
+    Ok(TransactionGetAllResponse { transactions })
 }
 
 pub async fn handle_get_result(

applications/tari_dan_wallet_daemon/src/handlers/transaction.rs

@@ -139,6 +139,7 @@ pub async fn handle_claim_validator_fees(
     let max_fee = req.max_fee.unwrap_or(DEFAULT_FEE);
 
     let transaction = transaction_builder(context)
+        .with_dry_run(req.dry_run)
         .with_fee_instructions_builder(|builder| {
             let mut bucket_names = vec![];
             fee_pool_addresses

applications/tari_dan_wallet_daemon/src/handlers/validator.rs

@@ -40,14 +40,7 @@ import StatusChip from "../../Components/StatusChip";
 import { DataTableCell } from "../../Components/StyledComponents";
 import { useGetAllTransactions } from "../../api/hooks/useTransactions";
 import { emptyRows, handleChangePage, handleChangeRowsPerPage } from "../../utils/helpers";
-import { useAccountsGet } from "../../api/hooks/useAccounts";
-import {
-  Account,
-  FinalizeResult,
-  substateIdToString,
-  Transaction,
-  TransactionStatus,
-} from "@tari-project/typescript-bindings";
+import { Account, substateIdToString, WalletTransaction } from "@tari-project/typescript-bindings";
 
 export default function Transactions({ account }: { account: Account }) {
   const [page, setPage] = useState(0);
@@ -77,12 +70,12 @@ export default function Transactions({ account }: { account: Account }) {
             <TableBody>
               {data?.transactions
                 ?.slice(page * rowsPerPage, page * rowsPerPage + rowsPerPage)
-                .map(([t, result, status, _s]: [Transaction, FinalizeResult | null, TransactionStatus, string]) => {
-                  const tx = t.V1;
-                  if (!tx?.id) {
-                    return <></>;
-                  }
-
+                .map((transaction: WalletTransaction) => {
+                  const {
+                    transaction: { V1: tx },
+                    finalize: result,
+                    status,
+                  } = transaction;
                   const hash = tx.id;
                   return (
                     <TableRow key={hash}>

applications/tari_dan_wallet_daemon/web_ui/src/routes/Transactions/Transactions.tsx

@@ -113,7 +113,7 @@ where TSubstateCache: SubstateCache + 'static
 
         // execute the payload in the WASM engine and return the result
         let exec_output = task::block_in_place(|| {
-            payload_processor.execute(transaction, state_store.into_read_only(), virtual_substates)
+            payload_processor.execute(&transaction, state_store.into_read_only(), virtual_substates)
         })?;
 
         Ok(exec_output.result)

applications/tari_indexer/src/dry_run/processor.rs

@@ -37,7 +37,11 @@ use tari_dan_app_utilities::{keypair::RistrettoKeypair, substate_file_cache::Sub
 use tari_dan_common_types::{optional::Optional, public_key_to_peer_id, Epoch, PeerAddress, SubstateRequirement};
 use tari_dan_engine::{template::TemplateModuleLoader, wasm::WasmModule};
 use tari_dan_p2p::TariMessagingSpec;
-use tari_dan_storage::{consensus_models::Decision, global::GlobalDb};
+use tari_dan_storage::{
+    consensus_models::Decision,
+    global::GlobalDb,
+    time::{PrimitiveDateTime, UtcDateTime},
+};
 use tari_dan_storage_sqlite::{error::SqliteStorageError, global::SqliteGlobalDbAdapter};
 use tari_engine_types::{FromByteType, ToByteType};
 use tari_epoch_manager::{service::EpochManagerHandle, EpochManagerReader};
@@ -450,7 +454,7 @@ impl JsonRpcHandlers {
                     execution_result: Some(Box::new(exec_result)),
                     final_decision: Decision::Commit,
                     abort_details: None,
-                    finalized_time: Default::default(),
+                    finalized_time: now(),
                     execution_time: Default::default(),
                 },
                 transaction_id,
@@ -675,3 +679,8 @@ impl JsonRpcHandlers {
         Self::error_response(answer_id, JsonRpcErrorReason::InternalError, error)
     }
 }
+
+fn now() -> PrimitiveDateTime {
+    let now = UtcDateTime::now();
+    PrimitiveDateTime::new(now.date(), now.time())
+}

applications/tari_indexer/src/json_rpc/handlers.rs

@@ -506,7 +506,6 @@ pub fn create_mempool_transaction_validator(
         .and_then(HasInputs::new())
         .and_then(FeeTransactionValidator)
         .and_then(TransactionSignatureValidator)
-        .and_then(HasInputs::new())
         .and_then(TemplateExistsValidator::new(template_manager))
 }
 

applications/tari_validator_node/src/bootstrap.rs

@@ -6,9 +6,12 @@ use std::{collections::HashMap, sync::Arc};
 use log::info;
 use tari_consensus::traits::{BlockTransactionExecutor, BlockTransactionExecutorError};
 use tari_dan_app_utilities::transaction_executor::TransactionExecutor;
-use tari_dan_common_types::{Epoch, SubstateRequirement};
+use tari_dan_common_types::{Epoch, SubstateRequirement, VersionedSubstateId};
 use tari_dan_engine::state_store::{memory::MemoryStateStore, new_memory_store, StateWriter};
-use tari_dan_storage::{consensus_models::ExecutedTransaction, StateStore};
+use tari_dan_storage::{
+    consensus_models::{TransactionExecution, VersionedSubstateIdLockIntent},
+    StateStore,
+};
 use tari_engine_types::{
     substate::Substate,
     virtual_substate::{VirtualSubstate, VirtualSubstateId, VirtualSubstates},
@@ -70,10 +73,10 @@ where
 
     fn execute(
         &self,
-        transaction: Transaction,
+        transaction: &Transaction,
         current_epoch: Epoch,
         resolved_inputs: &HashMap<SubstateRequirement, Substate>,
-    ) -> Result<ExecutedTransaction, BlockTransactionExecutorError> {
+    ) -> Result<TransactionExecution, BlockTransactionExecutorError> {
         let id = *transaction.id();
 
         info!(target: LOG_TARGET, "Transaction {} executing. {} input(s)", id, resolved_inputs.len());
@@ -98,9 +101,27 @@ where
         // execution
         let resolved_inputs = exec_output.resolve_input_locks(resolved_inputs);
 
-        let executed = ExecutedTransaction::new(exec_output.transaction, exec_output.result, resolved_inputs);
-        info!(target: LOG_TARGET, "Transaction {} executed. {}", id,executed.result().finalize.result);
-        Ok(executed)
+        let resulting_outputs = exec_output
+            .result
+            .finalize
+            .result
+            .accept()
+            .map(|diff| {
+                diff.up_iter()
+                    .map(|(addr, substate)| {
+                        VersionedSubstateIdLockIntent::output(VersionedSubstateId::new(
+                            addr.clone(),
+                            substate.version(),
+                        ))
+                    })
+                    .collect::<Vec<_>>()
+            })
+            .unwrap_or_default();
+
+        let exec = TransactionExecution::new(exec_output.result, resolved_inputs, resulting_outputs);
+
+        info!(target: LOG_TARGET, "Transaction {} executed. {}", id, exec.result().finalize.result);
+        Ok(exec)
     }
 }
 

applications/tari_validator_node/src/consensus/block_transaction_executor.rs

@@ -126,7 +126,7 @@ impl<TStateStore: StateStore<Addr = PeerAddress> + Send + Sync> DryRunTransactio
         // execute the payload in the WASM engine and return the result
         let processor = self.payload_processor.clone();
         let exec_output = task::spawn_blocking(move || {
-            processor.execute(transaction, temp_state_store.into_read_only(), virtual_substates)
+            processor.execute(&transaction, temp_state_store.into_read_only(), virtual_substates)
         })
         .await??;
         let result = exec_output.result;