Merge main to nomium-main (#22)

* tp_authority_public_key update

* lock version for cargo semver-checks

* Use shorter relative path

.. and fix some styling inconsistency

* Rewrite docs in `common_messages_sv2` ..

As part of the effort to improve Stratum V2 protocols docs, this commit
aims to improves and make the documentation more comprehensive and
accessible for contributors and end users alike.

* Add README.md ..

Use the template README used across the different Stratum V2 protocol
crates to `common_messages_sv2` crate.

* modify sv2.h to make it consistent with comments changes

* Use shorter relative path

* Add README file

* Improve Job Declaration Documentation

* Add README file

* Use shorter paths

* Improve `Template Distribution` subprotocol docs

* modify sv2.h to make it consistent with comments changes

* add binary-sv2 docs

* add binary-sv2 readme.md

* add binary-sv2 custom trait example

* add binary-sv2 no-serde derive-codec docs

* add binary-sv2 no-serde derive-codec readme

* add binary-sv2 no-serde codec readme

* add binary-sv2 no-serde codec docs

* update sv2.h for binary_sv2 docs

* change doc test in derive_codec

* Minor warning and error grammar and typo fixes

* bump jdc

* Slice docs

* Buffer docs

* Lib WriteError, Write, Buffer docs

* Back buffer pool docs

* Front buffer pool docs

* InnerMemory buffer pool docs

* PoolMode docs

* BufferPool docs

* Top buffer_pool mod docs

* Buffer pool examples

* Top level crate docs + clean up

* Update README

* Sniffer::wait_for_message_type

* fix unwrap on sv1-mining-device tcp connection..

Update roles/test-utils/mining-device-sv1/src/client.rs

Co-authored-by: jbesraa <jbesraa@gmail.com>

* remove redundant Drop implementation from ITF TemplateProvider

* Add `translator_sv2` test

* use macos-14 as  macos runner

* disable MG tproxy CI

* Have JDS ping local mempool less frequently

Otherwise Bitcoin Core -debug=rpc logging becomes too noisy.

* Use shorter paths

* Add README mining subprotocol

* Rewrite mining subprotocol docs

* Create a lib for `mining-sv2-proxy`

* Add `mining-sv2-proxy` initializer

* lock semver-checks to 37 while replacing MSRV with stable

* bump serde_v2 major to 2.0.0..

since v37, cargo semver-checks enforces that removal of features are breaking changes

therefore we need to update serde_v2 as a follow up to stratum-mining#1230

* bump binary_sv2 for serde_sv2 dependency..

since we bumpbed serde_sv2 to 2.0.0

* bump sv2_ffi for serde_sv2 dependency..

since we bumpted serde_sv2 to 2.0.0

* bump roles/Cargo.lock..

since we bumped serde_sv2 and binary_sv2

* bump const_sv2 major to 3.0.0..

since v37, cargo semver-checks enforces that removal of features are breaking changes

therefore we need to update serde_v2 as a follow up to stratum-mining#1230

* bump noise_sv2 for const_sv2 dependency..

since we bumpbed const_sv2 to 3.0.0

* bump framing_sv2 for const_sv2 dependency..

since we bumpbed const_sv2 to 3.0.0

* bump codec_sv2 for const_sv2 dependency..

since we bumpbed const_sv2 to 3.0.0

* bump subprotocols for const_sv2 dependency..

since we bumpbed const_sv2 to 3.0.0

* bump sv2_ffi for const_sv2 dependency..

since we bumpbed const_sv2 to 3.0.0

* bump roles_logic_sv2 for const_sv2 dependency..

since we bumpbed const_sv2 to 3.0.0

* bump roles due to const_sv2 bump to 3.0.0

* bump framing_sv2 major to 3.0.0..

since v37, cargo semver-checks enforces that removal of features are breaking changes

therefore we need to update framing_v2 as a follow up to stratum-mining#1230

* bump codec_sv2 for framing_sv2 dependency..

since we bumped framing_sv2 to 3.0.0

* bump framing_sv2 dendency version on roles_logic_sv2..

crate version has already been bumped before release. no need to bump again

* bump roles due to framing_sv2 bump

* bump benches due to framing_sv2 bump

* bump message_generator_sv2 due to const_sv2

* change macos-14 to macos-13

* bump buffer_sv2..

as a follow up to stratum-mining#1230

* bump buffer_sv2 dep on roles/Cargo.lock

* bump buffer_sv2 dep on utils/Cargo.lock

* bump derive_codec_sv2..

as a consequence of stratum-mining#1231

* bump derive_codec_sv2 dep on roles/Cargo.lock

* Add framing sv2 frame and handshake frame ex

* /// -> // for private types+fns

* framing mod doc cmts + clean

* Add framing README

* add codecov.yml to remove ci error

* feat: update println! and eprintln! to info using tracing cargo.

* Wait for `NewTemplate` message to arrive instead

of failing if it is not there yet when we first check

* ci: bump TP to 0.1.13

* test: increase TP log timestamp precision

Also log rpc calls.

* `common_messages_sv2` `job_declaration_sv2` `mining_sv2` `template_distribution_sv2`: remove the `no_std` feature and make them `#![no_std]` as they never use `std` anywhere.

- bump their MAJOR version because of feature removal
- bump the dependant crates PATCH version
- updates docs

* fix: clarify logging

* remove readme field from Cargo.toml of crates that dont have a README.md

* add release-libs.sh

* release-libs.yaml use 1.75.0 toolchain

* remove readme field from Cargo.toml of crates that dont have a README.md

* avoid publishing protocols crates with all-features..

publishing with all-features is overengineering:
- no_std is being deprecated
- with_serde will be deprecated soon
- for crates with relevant features, we specifically list them for publishing

* min_ntime fix to use the one sent by TP

* header_timestamp_value_assertion_in_new_extended_mining_job test addition

* bump PATCH after timestamp-bug fix (stratum-mining#1324)

* update lockfile after roles_logic patch

* Test Pool role behavior if bad `coinbase_output`..

is provided

* Remove bad pool config MG test

* Move sniffer test to a separate file

* Derive Clone+Debug for `JobDeclaratorClient`

* Derive Clone+Debug for `JobDeclaratorServer`

* Align all integration test start_* ..

functions return signature

* Make Sniffer::drop output more verbose

* Add README.md to integration-tests

* Move integration tests to separate GH action

* Rename integration tests crate..

it's important to have a descriptive name for when this is published to crates.io

* Modularize integration tests APIs..

into the following modules:
- lib: with general purpose functions (e.g.: starters)
- sniffer
- template_provider

* Remove redundant `TestPoolSv2`..

this struct is redundant and no other roles follow this pattern
we can do the initialization inside `start_pool`

* Fix variable naming on `wait_for_client`..

the `SocketAddr` is not a client, but the socket where we will listen to
also, `listner` is a typo

* Rename sniffer channel variables..

the original naming was confusing copypasta

* Create `utils.rs` in `tests-integration`

..to hold utility functions used internaly only.

* noise_sv2: keep current (std) API unchanged, add a no_std compliant API with `*_with_rng` and `*_with_now`

* generate cargo lock

---------

Co-authored-by: GitGab19 <gabriele.vernetti@protonmail.com>
Co-authored-by: plebhash <147345153+plebhash@users.noreply.github.com>
Co-authored-by: plebhash <plebhash@proton.me>
Co-authored-by: jbesraa <jbesraa@gmail.com>
Co-authored-by: bit-aloo <sshourya17@gmail.com>
Co-authored-by: Gabriele Vernetti <62447440+GitGab19@users.noreply.github.com>
Co-authored-by: Pavlenex <36959754+pavlenex@users.noreply.github.com>
Co-authored-by: RJ Rybarczyk <rj@rybar.tech>
Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
Co-authored-by: devworlds <jeffluizblockchain@gmail.com>
Co-authored-by: Georges Palauqui <georges.palauqui@gptechinno.com>
Co-authored-by: Gary Krause <gary.krause@mara.com>
Co-authored-by: Oleg Akulov <o.akulov@nomium.org>

Author: 11 people

.github/workflows/integration-tests.yaml

@@ -0,0 +1,32 @@
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+name: Integration Tests
+
+jobs:
+  ci:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os:
+          - ubuntu-latest
+        include:
+          - os: ubuntu-latest
+            target: x86_64-unknown-linux-musl
+
+    steps:
+      - name: Use stable toolchain
+        uses: actions/checkout@v4
+        with:
+          profile: minimal
+          toolchain: stable
+          override: true
+
+      - name: Roles Integration Tests
+        run: |
+          cargo test --manifest-path=roles/Cargo.toml --verbose --test '*' -- --nocapture

.github/workflows/mg.yaml

@@ -11,16 +11,6 @@ on:
       - main
 
 jobs:
-  bad-pool-config-test:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-      - name: Install cargo-llvm-cov
-        run: cargo install cargo-llvm-cov
-      - name: Run bad-pool-config-test
-        run: sh ./test/message-generator/test/bad-pool-config-test/bad-pool-config-test.sh
-
   interop-jd-translator:
     runs-on: ubuntu-latest
     steps:
@@ -167,7 +157,6 @@ jobs:
     runs-on: ubuntu-latest
     if: always()
     needs: [
-      bad-pool-config-test,
       interop-jd-translator,
       interop-proxy-with-multi-ups,
       interop-proxy-with-multi-ups-extended,
@@ -188,8 +177,7 @@ jobs:
     steps:
       - name: Aggregate Results
         run: |
-          if [ "${{ needs.bad-pool-config-test.result }}" != "success" ] ||
-          [ "${{ needs.interop-jd-translator.result }}" != "success" ] ||
+          if [ "${{ needs.interop-jd-translator.result }}" != "success" ] ||
           [ "${{ needs.interop-proxy-with-multi-ups.result }}" != "success" ] ||
           [ "${{ needs.interop-proxy-with-multi-ups-extended.result }}" != "success" ] ||
           [ "${{ needs.jds-do-not-fail-on-wrong-tsdatasucc.result }}" != "success" ] ||
@@ -209,4 +197,4 @@ jobs:
             exit 1
           else
             echo "All MG tests completed successfully"
-          fi
+          fi

.github/workflows/release-libs.yaml

@@ -1,12 +1,8 @@
 # This workflow is used to publish SV2 crates to crates.io
-# The workflow tries to update all the library crates, so if a crate is not to updated, the step will fail
-# for that each step have continue-on-error set to true.
-# Since each step can fail, the output ot the action must be manually checked to make sure that all
-# the library intended to be published are published.
-# Running cargo release in the various workspace help to prepare the version number and everything.
-# ATTENTION
-# Is very important to check the output manually cause when too many crates are updated crates.io could fail
-# and ask to rerun the action later
+# the workflow tries to publish all the library crates by running scripts/release-libs.sh
+# in case the `cargo publish` command fails, the script returns 1 and the entire workflow fails
+# the only exception is when the `cargo publish` command fails because the crate has already
+# been published, in which case the workflow continues
 
 name: Release Libs
 
@@ -28,117 +24,95 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions-rs/toolchain@v1
         with:
-            toolchain: stable
+            toolchain: 1.75.0
             override: true
       - name: Login
         run: cargo login ${{ secrets.CRATES_IO_DEPLOY_KEY }}
+
       - name: Publish crate common
-        continue-on-error: true
         run: |
-          cd common
-          cargo publish
+          ./scripts/release-libs.sh common
+
       - name: Publish crate buffer_sv2
-        continue-on-error: true
         run: |
-          cd utils/buffer
-          cargo publish
+          ./scripts/release-libs.sh utils/buffer
+
       - name: Publish crate no_serde_sv2_derive_codec
-        continue-on-error: true
         run: |
-          cd protocols/v2/binary-sv2/no-serde-sv2/derive_codec
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/binary-sv2/no-serde-sv2/derive_codec
+
       - name: Publish crate no_serde_sv2_codec
-        continue-on-error: true
         run: |
-          cd protocols/v2/binary-sv2/no-serde-sv2/codec
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/binary-sv2/no-serde-sv2/codec
+
       - name: Publish crate serde_sv2
-        continue-on-error: true
         run: |
-          cd protocols/v2/binary-sv2/serde-sv2
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/binary-sv2/serde-sv2
+
       - name: Publish crate binary_sv2
-        continue-on-error: true
         run: |
-          cd protocols/v2/binary-sv2/binary-sv2
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/binary-sv2/binary-sv2
+
       - name: Publish crate const_sv2
-        continue-on-error: true
         run: |
-          cd protocols/v2/const-sv2
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/const-sv2
+
       - name: Publish crate framing_sv2
-        continue-on-error: true
         run: |
-          cd protocols/v2/framing-sv2
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/framing-sv2
+
       - name: Publish crate noise_sv2
-        continue-on-error: true
         run: |
-          cd protocols/v2/noise-sv2
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/noise-sv2
+
       - name: Publish crate codec_sv2
-        continue-on-error: true
         run: |
-          cd protocols/v2/codec-sv2
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/codec-sv2
+
       - name: Publish crate common_messages
-        continue-on-error: true
         run: |
-          cd protocols/v2/subprotocols/common-messages
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/subprotocols/common-messages
+
       - name: Publish crate job_declaration
-        continue-on-error: true
         run: |
-          cd protocols/v2/subprotocols/job-declaration
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/subprotocols/job-declaration
+
       - name: Publish crate mining
-        continue-on-error: true
         run: |
-          cd protocols/v2/subprotocols/mining
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/subprotocols/mining
+
       - name: Publish crate template_distribution
-        continue-on-error: true
         run: |
-          cd protocols/v2/subprotocols/template-distribution
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/subprotocols/template-distribution
+
       - name: Publish crate sv2_ffi
-        continue-on-error: true
         run: |
-          cd protocols/v2/sv2-ffi
-          cargo publish --all-features
+          ./scripts/release-libs.sh protocols/v2/sv2-ffi
+
       - name: Publish crate roles_logic_sv2
-        continue-on-error: true
         run: |
-          cd protocols/v2/roles-logic-sv2
-          cargo publish
+          ./scripts/release-libs.sh protocols/v2/roles-logic-sv2
+
       - name: Publish crate v1
-        continue-on-error: true
         run: |
-          cd protocols/v1
-          cargo publish
+          ./scripts/release-libs.sh protocols/v1
+
       - name: Publish crate bip32-key-derivation
-        continue-on-error: true
         run: |
-          cd utils/bip32-key-derivation
-          cargo publish
+          ./scripts/release-libs.sh utils/bip32-key-derivation
+
       - name: Publish crate error-handling
-        continue-on-error: true
         run: |
-          cd utils/error-handling
-          cargo publish
+          ./scripts/release-libs.sh utils/error-handling
+
       - name: Publish crate key-utils
-        continue-on-error: true
         run: |
-          cd utils/key-utils
-          cargo publish
+          ./scripts/release-libs.sh utils/key-utils
+
       - name: Publish crate network_helpers_sv2
-        continue-on-error: true
         run: |
-          cd roles/roles-utils/network-helpers
-          cargo publish
+          ./scripts/release-libs.sh roles/roles-utils/network-helpers
+
       - name: Publish crate rpc_sv2
-        continue-on-error: true
         run: |
-          cd roles/roles-utils/rpc
-          cargo publish
+          ./scripts/release-libs.sh roles/roles-utils/rpc

.github/workflows/semver-check.yaml

@@ -16,10 +16,10 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v2
 
-      - name: Install Rust 1.75.0
+      - name: Install Rust stable
         uses: actions-rs/toolchain@v1
         with:
-          toolchain: 1.75.0
+          toolchain: stable
           override: true
 
       - name: Cache Cargo registry
@@ -42,7 +42,7 @@ jobs:
         run: sudo apt-get update && sudo apt-get install -y cmake
 
       - name: Install cargo-semver-checks
-        run: cargo install cargo-semver-checks --version 0.33.0 --locked
+        run: cargo install cargo-semver-checks --version 0.37.0 --locked
 
       - name: Run semver checks for common
         working-directory: common

.github/workflows/test.yaml

@@ -14,10 +14,10 @@ jobs:
     strategy:
       matrix:
         os:
-          - macos-14
+          - macos-13
           - ubuntu-latest
         include:
-          - os: macos-14
+          - os: macos-13
             target: x86_64-apple-darwin
           - os: ubuntu-latest
             target: x86_64-unknown-linux-musl
@@ -38,10 +38,6 @@ jobs:
           cargo build --manifest-path=roles/Cargo.toml
           cargo build --manifest-path=utils/Cargo.toml
 
-      - name: Roles Integration Tests
-        run: |
-          cargo test --manifest-path=roles/Cargo.toml --verbose --test '*' -- --nocapture
-
       - name: Run sv1-client-and-server example
         run: |
           cargo run --manifest-path=examples/sv1-client-and-server/Cargo.toml --bin client_and_server -- 60

benches/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "benchmark"
-version = "1.0.0"
+version = "1.0.1"
 edition = "2021"
 
 [dependencies]
@@ -10,9 +10,9 @@ async-channel = "1.4.0"
 v1 = { path="../protocols/v1", package="sv1_api", version = "^1.0.0" }
 serde_json = { version = "1.0.64", default-features = false, features = ["alloc"] }
 iai="0.1"
-mining_sv2 = { path = "../protocols/v2/subprotocols/mining", version = "^1.0.0" }
+mining_sv2 = { path = "../protocols/v2/subprotocols/mining", version = "^2.0.0" }
 roles_logic_sv2 = { path = "../protocols/v2/roles-logic-sv2", version = "^1.0.0" }
-framing_sv2 = { version = "2.0.0", path = "../protocols/v2/framing-sv2" }
+framing_sv2 = { version = "3.0.0", path = "../protocols/v2/framing-sv2" }
 serde = { version = "1.0.89", default-features = false, features = ["derive", "alloc"] }
 num-bigint = "0.4.3"
 num-traits = "0.2.15"

codecov.yaml

@@ -0,0 +1,13 @@
+coverage:
+  status:
+    project:
+      default:
+        target: auto
+        threshold: 100%
+        base: auto
+        informational: false
+    patch:
+      default:
+        target: auto
+        threshold: 100%
+        base: auto

protocols/v2/binary-sv2/binary-sv2/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "binary_sv2"
-version = "1.2.0"
+version = "1.2.1"
 authors = ["The Stratum V2 Developers"]
 edition = "2018"
 readme = "README.md"
@@ -14,7 +14,7 @@ keywords = ["stratum", "mining", "bitcoin", "protocol"]
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
-serde_sv2 = {version = "^1.0.0", path = "../serde-sv2", optional = true}
+serde_sv2 = {version = "^2.0.0", path = "../serde-sv2", optional = true}
 serde = { version = "1.0.89", features = ["derive", "alloc"], default-features = false, optional = true }
 binary_codec_sv2 = {version = "^1.0.0", path = "../no-serde-sv2/codec", optional = true}
 derive_codec_sv2 = {version = "^1.0.0", path = "../no-serde-sv2/derive_codec", optional = true}
@@ -28,4 +28,4 @@ prop_test = ["binary_codec_sv2/prop_test", "derive_codec_sv2"]
 with_buffer_pool = ["binary_codec_sv2/with_buffer_pool", "derive_codec_sv2"]
 
 [package.metadata.docs.rs]
-all-features = true
+features = ["core", "with_buffer_pool"]

protocols/v2/binary-sv2/no-serde-sv2/codec/Cargo.toml

@@ -24,4 +24,4 @@ prop_test = ["quickcheck"]
 with_buffer_pool = ["buffer_sv2"]
 
 [package.metadata.docs.rs]
-all-features = true
+features = ["with_buffer_pool"]

protocols/v2/binary-sv2/no-serde-sv2/derive_codec/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "derive_codec_sv2"
-version = "1.1.0"
+version = "1.1.1"
 authors = ["The Stratum V2 Developers"]
 edition = "2018"
 readme = "README.md"
@@ -18,6 +18,3 @@ binary_codec_sv2 = {version = "^1.0.0", path="../codec"}
 
 [lib]
 proc-macro = true
-
-[package.metadata.docs.rs]
-all-features = true