fix(consensus)!: use commit proof for foreign proposals (tari-project#1413)

Description
---
fix(consensus)!: use commit proof for foreign proposals
fix(consensus): adds new foreign proposal validations
fix(consensus): adds missing validation for QC epoch
fix(ci): ci now runs consensus tests using rocksdb
fix(state store)!: ordering bug when retreiving block diffs causing
consensus failures

Motivation and Context
---
Requires tari-project/tari#7003

Replaces the foreign proposal with a `tari_sidechain` commit proof. The
foreign proposal size is reduced by including full command data only for
shard-applicable commands. Non-applicable commands are reduced to a
32-byte hash to allow for command merkle root verification.

How Has This Been Tested?
---
Existing tests, manually

What process can a PR reviewer use to test or verify this change?
---
Foreign proposals can be viewed using the db inspector

Breaking Changes
---

- [ ] None
- [x] Requires data directory to be deleted
- [ ] Other - Please specify

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added explicit tracking of justification and commitment using quorum
certificate IDs instead of boolean flags for blocks, improving
transparency and auditability.
- Introduced new abstractions for command commit proofs and foreign
proposal handling, enhancing consensus validation and interoperability.
- Added new validation modules for local and foreign proposals,
providing stricter and clearer consensus checks.

- **Refactor**
- Refactored block, quorum certificate, and proposal data structures to
use QC IDs and encoded proofs, streamlining state management.
- Updated storage schemas and APIs to reflect new QC-based commitment
and justification, and improved foreign proposal indexing and queries.
- Simplified and modularized consensus and validation logic for
maintainability.
- Replaced direct committee member field access with accessor methods
for better encapsulation.
- Consolidated foreign proposal types into record and core data
structures for clearer separation of concerns.
- Updated commit and justify logic in consensus flow to consistently use
QC IDs.
- Replaced boolean flags with QC ID references in database schemas and
queries.
- Refactored foreign parked proposals to separate block ID, commit
proof, and pledge components.
- Simplified consensus message encoding by replacing explicit fields
with encoded blobs.
- Adjusted method names and signatures related to parked blocks and
missing transactions for clarity.
- Generalized and improved encoding/decoding of block diff keys to
include ordering and versioning.
- Replaced ownership-heavy types with reference types where appropriate
for efficiency.
- Modularized validation logic into separate modules for block and
foreign proposal validation.
  - Updated error handling and logging for better clarity and context.

- **Bug Fixes**
- Improved status determination for block displays in the web UI,
ensuring accurate commit status representation.
- Corrected committee member access and leader selection logic for more
robust consensus operation.
- Enhanced epoch validation for proposals and quorum certificates to
prevent invalid consensus states.

- **Chores**
- Updated dependencies and internal imports to align with refactored
consensus and sidechain modules.
- Incremented package versions for TypeScript bindings and JavaScript
client libraries.
- Adjusted import paths and removed unused imports for cleaner codebase.
- Removed deprecated quorum decision enum and replaced with external
sidechain import.
- Updated CI and test configurations to include integration tests and
refine test filters.

- **Tests**
- Updated and refactored tests to use new helper methods, QC ID logic,
and improved assertion clarity.
- Enhanced test coverage for foreign proposals, parked blocks, and
quorum certificates.
- Replaced manual test setups with reusable helper functions for
consistency.
- Corrected import paths and updated test data to align with refactored
data structures.

- **Documentation**
- Improved code comments and documentation for new data structures and
validation logic.
- Updated migration scripts and database schemas to reflect new
consensus tracking fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: sdbondi

.cargo/config.toml

@@ -1,3 +1,7 @@
+[profile.release-with-debug]
+inherits = "release"
+debug = true
+
 [target.aarch64-unknown-linux-gnu]
 linker = "aarch64-linux-gnu-gcc"
 

.config/nextest.toml

@@ -1,5 +1,9 @@
+[profile.default]
+default-filter = "not package(integration_tests) and not test(export_bindings)"
+
 [profile.ci]
 slow-timeout = { period = "60s", terminate-after = 4 }
+default-filter = "not package(integration_tests) and not test(export_bindings)"
 
 [profile.ci.junit]  # this can be some other profile, too
 path = "junit.xml"

.github/workflows/ci.yml

@@ -240,7 +240,7 @@ jobs:
         run: cargo test --no-run --locked --all-features --release
 
       - name: cargo test
-        run: cargo nextest run --all-features --release -E "not package(integration_tests)" --profile ci
+        run: cargo nextest run --all-features --release --profile ci
 
       - name: upload artifact
         uses: actions/upload-artifact@v4 # upload test results as artifact