Skip to content

Conversation

@GTC6244
Copy link
Contributor

@GTC6244 GTC6244 commented Jan 21, 2026

No description provided.

Garandor and others added 30 commits November 25, 2025 15:40
* Test fix actionlint

* Update .github/workflows/lint-workflow-files.yml

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Copilot <[email protected]>
)

* Support naga-proto

* update workflow

---------

Co-authored-by: Brendon <[email protected]>
* chore(deps): bump actions/checkout from 5 to 6

Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* Test fix actionlint

* Update .github/workflows/lint-workflow-files.yml

Co-authored-by: Copilot <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GTC6244 <[email protected]>
Co-authored-by: Copilot <[email protected]>
* Address storage issues

* Update ABIs and rust bindings

* Restore function

* Update ABIs and rust bindings

* Add comment
* Removed unused and confusing stuff

* Update ABIs and rust bindings

* Mark deprecated field

* Update ABIs and rust bindings

* Fix prettier

* Update ABIs and rust bindings
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Code compatibility - ie, it builds

* cargo fmt

* Update rust/lit-node/lit-node/tests/upgrades/version_upgrades.rs

Co-authored-by: Copilot <[email protected]>

* closer

* Simple test that uses different stakers.

---------

Co-authored-by: Copilot <[email protected]>
* usd pricing script

* fix wei formatting, comment, and add a space to the log line
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v5...v6)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 6 to 7.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v6...v7)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Brendon <[email protected]>
Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4...v5)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Brendon <[email protected]>
* development state of 11/17

* clippy

Signed-off-by: Michael Lodder <[email protected]>

* blockchain abi ?

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Michael Lodder <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Brendon <[email protected]>
### TL;DR

Added a check to prevent nodes not assigned to a realm from being considered validators.

### What changed?

Added a condition that checks if `realm_id` is zero before proceeding with validator status verification. If a node has a realm ID of zero, it means the node is not assigned to any realm and therefore cannot be a validator, so the function returns `false` immediately.
### TL;DR

Improved error logging for attestation creation failures in the ChatterServer.
a the attestation creation code in the `ChatterService` implementation to use a match statement instead of an if-let pattern. This change allows for capturing and logging the specific error that occurs during attestation creation, rather than just logging a generic error message.

### How to test?

1. Run a Lit node in an environment where attestation creation might fail
2. Check the logs for more detailed error messages that now include the specific error information
3. Verify that the error message format is now `"Error creating attestation: {:?}"` with the actual error details

### Why make this change?

This change enhances debugging capabilities by providing more detailed error information when attestation creation fails. Having the specific error details in the logs will make it easier to diagnose and fix issues related to attestation creation in production environments.
* first pass at price setting script

* price setting script seems like it will work

* script works on dev and prod

* Update blockchain/contracts/scripts/generatePriceSettingTransactions.ts

Co-authored-by: Copilot <[email protected]>

* remove unused map

* regen contracts

* 100x decrease for testnets when setting prices

* divide by 10000 when setting naga-test prices

* actually, made naga-test 100000x cheaper than naga prod

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Brendon <[email protected]>
* Add spans for actions

* Add correlation_id to TracingMiddleware

* Checkout server.rs

* Extract correlation_id'

---------

Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Brendon <[email protected]>
* Add spans for actions

* Improve metrics

* first pass at adding request id to logs

* Update rust/lit-node/lit-node/src/main.rs

Co-authored-by: Copilot <[email protected]>

* change how we set the request id on the span

* clippy

* Add correlation_id to TracingMiddleware

* Checkout server.rs

* Extract correlation_id'

* Fix clippy wanrning

* separate info from gauge metrics (#44)

* Revert "Merge branch 'inf-852-fix-tracing-lit-actions' into inf-753-olly-metrics"

This reverts commit b0a6d38, reversing
changes made to 4a58992.

* Revert "Merge remote-tracking branch 'origin/feature/NODE-4860-correlation-id-in-logs' into inf-753-olly-metrics"

This reverts commit 7be7bbe, reversing
changes made to b0a6d38.

* cargo fmt

---------

Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
* fix(node): use wallet SIWE auth sig in unified ACC

* fix github workflow linting error

* fix compile error

---------

Co-authored-by: Chris Cassano <[email protected]>
…#61)

# What

Improve error handling during node startup for PeerState creation and TSS state initialization.

# Why

Lack of detailed error information when critical components fail to initialize makes it difficult to diagnose startup issues.

# Background

We wasted too much time figuring out why a newly recreated node wasn't able to join the network. If it's due to the AW registration failing, it should now be obvious from the logs

# Risks

- No functional changes to the code logic, only improved error handling
- The changes maintain the same behavior of panicking on critical initialization failures, but with better error reporting
…ro exit) (#63)

# What

Improve error handling and diagnostics for Rocket HTTP server launch failures.

# Why

Previously, Rocket launch errors (such as port bind failures) were silently ignored, making it difficult to diagnose server startup issues. This change ensures that launch failures are properly logged and propagated, causing the application to exit with a non-zero status code.

# Risks

`join_all` (wait for all futures to exit) was changed and now exits `lit-node` when a single rocket fails to launch and with nonzero exit, which could have downstream effects on systemd and other supervisors.
* add readme

* publish sdk

* put path dependency back but just specify a version

* regen lit-os cargo lockfile

* fix some build errors

* update cargo.lock files for lit-os and lit-core

---------

Co-authored-by: Brendon <[email protected]>
* Port code from the prod release PR.

* c&p

* Update rust/lit-node/lit-node/tests/common/faults.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/tests/toxiproxy/chain_faults.rs

Co-authored-by: Copilot <[email protected]>

* remove comment

* Update rust/lit-node/lit-node-testnet/src/testnet/actions.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/tests/toxiproxy/chain_faults.rs

Co-authored-by: Copilot <[email protected]>

* formatting

* cargo fmt

* ack.

* ack.

* fix fault test setup

* ignore redundant chain fault test - but keep as model.

* re-add >= to epoch/previous epoch check

---------

Co-authored-by: Copilot <[email protected]>
### TL;DR

- Configures k256 contract address as `0x029bedeacaf6821ce9a6bd7c8ac73350f24a014f`
- Updated the unused P256 contract address to a placeholder value in hardhat configuration.
kapoorabhishek24 and others added 6 commits January 13, 2026 22:07
* first pass at adding request id to logs

* Update rust/lit-node/lit-node/src/main.rs

Co-authored-by: Copilot <[email protected]>

* change how we set the request id on the span

* clippy

* Refactor spans behaviour

* Fix fallback ID

* Add Rocket fairings

* Remove redundant tracing

* optimise tests

* Address feedback

* Remove fallback UUID

* Leverage Dashmap

* build lit-os to rebuild cargo.lock

* rebuild lit actions and lit-node

* remove correlation_id from /web/job_status/v2 instrumentation since it's only on this one endpoint

* Revert "remove correlation_id from /web/job_status/v2 instrumentation since it's only on this one endpoint"

This reverts commit 7c4f5f9.

---------

Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
* draft priority-aware RPC healtcheck poller

* Update selection logic

* exclude priority from RpcEntry

* Add feedback

* Re-add empty check

---------

Co-authored-by: Abhi... <[email protected]>
* Make git_info.rs static but dynamically generated on compile

* fix constness
…des aka min threshold (#71)

* divide expected prices by number of required nodes aka min threshold

* clean up script and redundant vars
Keysets is already deployed.   All downstream *required* branches have been reviewed and merged it.  Merging to master.

* Prep for lit-peer

* WIP adding a second keyset

* prettier

* Clippy!

* fix node_restarts_without_key_material - and maybe an FSM update?

* remove keyset test ( next branch )

* Better handling of shadow splicing. (#26)

* Better handling of shadow splicing.

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Copilot <[email protected]>

* Quick ordering update for shadowOpts struct.

* abi staleness update

* blockchain staleness ?

* Handle datil old peers (#34)

* handle old peers

Signed-off-by: Michael Lodder <[email protected]>

* Open v2.1.7

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Keysets - handle multiples in DKG. (#15)

* Update test

* Update tests

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

* resync /w keysets feature.

---------

Co-authored-by: Copilot <[email protected]>

* fix tests

* cargo fmt

* cargo update / abi staleness

* Fixes to merges & clippy updates

* update tests

* Fix deploy/test JS ( which I broke with a bad merge )

* prettier....

* Feature/node 4848 test datil keys in naga (#24)

* Update test

* Update tests

* Setup

* restore datil keys into a second keyset

* remove large binaries

* remove large zip

* cargo fmt

* Updates to DKG / Epoch change processes + correct saving of data to chain.

* fix actions in testnet

* cargo fmt

* patch epoch change retries.

* cargo fmt

* Fix failed DKGs when doing restores.

* Feature/node 4874 support multiple chains at anvil level (#41)

* Update test

* Update tests

* Setup

* restore datil keys into a second keyset

* remove large binaries

* remove large zip

* cargo fmt

* Updates to DKG / Epoch change processes + correct saving of data to chain.

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* fix actions in testnet

* cargo fmt

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* Update test to interact with ImportedDatilTestnet

* remove unneded code

* refactor & remove duplications

* Refactor & add datil node secrets

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/src/tss/common/restore/eks_and_ds.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/Cargo.toml

Co-authored-by: Copilot <[email protected]>

* cargo lock cleanup

* cargo.lock update + config.rs in the resolver formatting.

* patch epoch change retries.

* cargo fmt

* Fix failed DKGs when doing restores.

* cargo fmt

* fix tests

* Logging info for CI datil instances.

* Open ports for a second instance of anvil in the yml files.

* Load hex data from test state into anvil.

* remove unused file

* comment out #if false

* take 2?

* fix test reference

* Use HEX file to load state, rather than a JSON file.

* Refactor to insure that for Datil tests we don't rely on naga cache'd chain state.

* switch lit-recovery binaries for testing

* Remove keyset_identifier as an enum and clean up tests.

* fix datil backup selection of binary.

* cargo fmt

* yml test.

* if false yml

* comment out version upgrade

* yml update

* try a var for the upgrade workflow ?

* update lit-os cargo.lock

* update gitinfo

---------

Co-authored-by: Howard Tam <[email protected]>
Co-authored-by: Copilot <[email protected]>

* Feature/node 4876 - using both chains in some tests ( adapt end user to support pkps in both chains ) (#42)

* Update test

* Update tests

* Setup

* restore datil keys into a second keyset

* remove large binaries

* remove large zip

* cargo fmt

* Updates to DKG / Epoch change processes + correct saving of data to chain.

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* fix actions in testnet

* cargo fmt

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* Update test to interact with ImportedDatilTestnet

* remove unneded code

* refactor & remove duplications

* Refactor & add datil node secrets

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/src/tss/common/restore/eks_and_ds.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/Cargo.toml

Co-authored-by: Copilot <[email protected]>

* whole lotta shaking going on.  ( endpoints changed throughout )

* PKP sign works.

* cargo clippy

* cargo lock cleanup

* git merge fixes

* cargo toml update ( to trigger cargo.lock update )

* cargo.lock update + config.rs in the resolver formatting.

* cargo fmt

* fix panic when node was kicked (realmid = 0)

* patch epoch change retries.

* cargo fmt

* Expose underlying error when peer_connect attestation fails

* Fix failed DKGs when doing restores.

* cargo fmt

* fix tests !

* fix tests

* Logging info for CI datil instances.

* Open ports for a second instance of anvil in the yml files.

* Load hex data from test state into anvil.

* Feature/price setting script (#43)

* first pass at price setting script

* price setting script seems like it will work

* script works on dev and prod

* Update blockchain/contracts/scripts/generatePriceSettingTransactions.ts

Co-authored-by: Copilot <[email protected]>

* remove unused map

* regen contracts

* 100x decrease for testnets when setting prices

* divide by 10000 when setting naga-test prices

* actually, made naga-test 100000x cheaper than naga prod

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Fix: Add `correlation_id` to `lit-actions` spans (#25)

* Add spans for actions

* Add correlation_id to TracingMiddleware

* Checkout server.rs

* Extract correlation_id'

---------

Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Refactors OsQuery metrics emission to use OpenTelemetry Gauges (#28)

* Add spans for actions

* Improve metrics

* first pass at adding request id to logs

* Update rust/lit-node/lit-node/src/main.rs

Co-authored-by: Copilot <[email protected]>

* change how we set the request id on the span

* clippy

* Add correlation_id to TracingMiddleware

* Checkout server.rs

* Extract correlation_id'

* Fix clippy wanrning

* separate info from gauge metrics (#44)

* Revert "Merge branch 'inf-852-fix-tracing-lit-actions' into inf-753-olly-metrics"

This reverts commit b0a6d38, reversing
changes made to 4a58992.

* Revert "Merge remote-tracking branch 'origin/feature/NODE-4860-correlation-id-in-logs' into inf-753-olly-metrics"

This reverts commit 7be7bbe, reversing
changes made to b0a6d38.

* cargo fmt

---------

Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Adam Reif <[email protected]>

* remove unused file

* comment out #if false

* take 2?

* fix test reference

* Use HEX file to load state, rather than a JSON file.

* Refactor to insure that for Datil tests we don't rely on naga cache'd chain state.

* switch lit-recovery binaries for testing

* Remove keyset_identifier as an enum and clean up tests.

* Fix up tests

* cargo fmt

* fix datil backup selection of binary.

* logging, shorten handshake challange value.

* fix "default" keyset bug :(

* cargo fmt

* yml test.

* if false yml

* comment out version upgrade

* yml update

* try a var for the upgrade workflow ?

* upgrade yml with default if statement.

* cargo clippy.

* Update lit-os cargo.lock

* update lit-os cargo.lock

* update gitinfo

* fix restore test to use Datil directly.

* cargo fmt

* patch so both long running restores work :(

* Update epoch change long to deal with keysets

* small shiva refactor

* difference between pkp & root key lookups when getting a keyset.

* remove datil from lit action testing.

* clean up integration test setup to account for keyset settings.

* limit caches to only active states.

* Fix Shadow splicing test, make some integration force builds

* update abis & lit-os lock file

* cargo fmt + minor cleanup

* Refactor one of the last timing related tests.

* ABI staleness, remove more testing feature flags

* Make the StakerAccountSetupMapper Send+Sync to allow Shiva to build.

* Clean up complaints.

* Update to determine key set for recovery (#58)

* update to determine key set for recovery

Signed-off-by: Michael Lodder <[email protected]>

* add test when no key set is on chain

Signed-off-by: Michael Lodder <[email protected]>

* update test

Signed-off-by: Michael Lodder <[email protected]>

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Better logging.

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Howard Tam <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Abhi... <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Michael Lodder <[email protected]>

* abi staleness + cargo fmt.

* Remove traces of custom version upgrade testing. (#57)

* fix: Invalid signature combination (#65)

* fix invalid signature combination

Signed-off-by: Michael Lodder <[email protected]>

* fix tests

Signed-off-by: Michael Lodder <[email protected]>

---------

Signed-off-by: Michael Lodder <[email protected]>

* re-instate decryption assertion

* clippy.

* remove unused val

* make openapi spec (#66)

* make openapi spec

Signed-off-by: Michael Lodder <[email protected]>

* remove "with same ip and port" test - chck /w Mike?

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: GTC6244 <[email protected]>

* Feature/node 4923 refactor upgrade tests to use local configurations (#60)

* Remove traces of custom version upgrade testing.

* First  few changes - start simplification.

* restore key_types in staking contract, allow validator cloning.

* Update to allow old version to load

* update local builds

* debugging updates

* better logging, disable network checks between epoch changes ( for now )

* clippy!

* add a filnal check.

* cargo fmt

* clippy.

* Download an install old binaries as required.

* ignore upgrades by default.

* re-include Pallas curves in deployment scripts.

* add functions to  create proper upgrade envrionment for old nodes ( remove curves, change yaml files ).

* blockchain abi stales + cargo fmt.

* disable test ( with same port )

* re-add >= to epoch change

* Update contracts to include keysets_deprecated type in tests.

* missing "deprecated" param in test

* merge master fixes

* try to update docs

* force deploy on lit-actions.

* force deploy on more lit actions.

* Feature/node 4931 implement lit actions for keysets (datil support!) (#62)

* Lit Action Server side updates /w Docs

* lit-node side updates.

* Add a few tests.

* Update rust/lit-node/lit-node-testnet/src/testnet/contracts_repo.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-actions/ext/js/02_litActionsSDK.js

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node-testnet/src/testnet/contracts_repo.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node-testnet/src/testnet/contracts_repo.rs

Co-authored-by: Copilot <[email protected]>

* copilot suggestions

* documentation update

* cargo fmt

* clipppppppyyyyy!

* Update docs?

* build docs again!

* add in missing "keysetId" values from  lit action code samples that are stored as string in mods.

* force lit-action  datil tests to run outside of caches.

* test forcing Lit Actions to recompile ( cache with sometimes using Datil seems off )

* regen docs

* Version upgrade check

* Refactor for explicit endpoint change - include keyset as part for lit-actions, which can be overwritten wtih a js param.

* documentation update

* use force_deploy for lit actions

* error handling on reading anvil file.

* read/write hex data as string + better error handling.

* better debugging.

* revert keyset in session sig issue.   ( creates invalid IPFS CID value! )

* cargo fmt

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Adam Reif <[email protected]>

* Remerge build.rs!

* cargo fmt

* more debugging.

* remerge build.rs

* cargo fmt

* Feature/node 4947 set default keyset on contract deployment (#82)

* set default realm config in the deploy.ts, and remove from test builder in node code ;-)

* prettier!

* switch function name.... which makes me wonder if my test executed!

* ABI stalensss

* add CI debbugging.

* Reset all defaults for realm #1

* Feature/node 4940 fix keyset ci times (#76)

* Make datil always available, even if not used.

* Add debugging.

* update build.rs

* debugging

* Test / Refactor / Discover - anvil cache issues.

* rename to edge testing, add a matrix

* clippy !

* test splitting out the edge related tests.

* cargo fmt

* config update (nomenclature, mainly)

* Use an updated version of datil contracts that can be properly cleared of root keys.

* remove unit test partitioning

* add debugging info

* Add keyset into ClientBuilder.

* force deploy for the child lit action function.

* fix backup-long to use only naga keysets.

* cargo fmt

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Michael Lodder <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Howard Tam <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Abhi... <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
@github-actions
Copy link

github-actions bot commented Jan 21, 2026

PASS [ 44.702s] (3/3) lit_node::test toxiproxy::perf_tests::load_with_no_latency
PASS [ 44.781s] (2/3) lit_node::test toxiproxy::perf_tests::load_with_50ms_latency_single_link
PASS [ 86.749s] (1/3) lit_node::test toxiproxy::perf_tests::load_with_50ms_latency_all_links

GTC6244 and others added 10 commits January 21, 2026 16:31
Picking up from a missed merge.

* Prep for lit-peer

* WIP adding a second keyset

* prettier

* Clippy!

* fix node_restarts_without_key_material - and maybe an FSM update?

* remove keyset test ( next branch )

* Better handling of shadow splicing. (#26)

* Better handling of shadow splicing.

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Copilot <[email protected]>

* Quick ordering update for shadowOpts struct.

* abi staleness update

* blockchain staleness ?

* Handle datil old peers (#34)

* handle old peers

Signed-off-by: Michael Lodder <[email protected]>

* Open v2.1.7

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Keysets - handle multiples in DKG. (#15)

* Update test

* Update tests

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

* resync /w keysets feature.

---------

Co-authored-by: Copilot <[email protected]>

* fix tests

* cargo fmt

* cargo update / abi staleness

* Fixes to merges & clippy updates

* update tests

* Fix deploy/test JS ( which I broke with a bad merge )

* prettier....

* Feature/node 4848 test datil keys in naga (#24)

* Update test

* Update tests

* Setup

* restore datil keys into a second keyset

* remove large binaries

* remove large zip

* cargo fmt

* Updates to DKG / Epoch change processes + correct saving of data to chain.

* fix actions in testnet

* cargo fmt

* patch epoch change retries.

* cargo fmt

* Fix failed DKGs when doing restores.

* Feature/node 4874 support multiple chains at anvil level (#41)

* Update test

* Update tests

* Setup

* restore datil keys into a second keyset

* remove large binaries

* remove large zip

* cargo fmt

* Updates to DKG / Epoch change processes + correct saving of data to chain.

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* fix actions in testnet

* cargo fmt

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* Update test to interact with ImportedDatilTestnet

* remove unneded code

* refactor & remove duplications

* Refactor & add datil node secrets

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/src/tss/common/restore/eks_and_ds.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/Cargo.toml

Co-authored-by: Copilot <[email protected]>

* cargo lock cleanup

* cargo.lock update + config.rs in the resolver formatting.

* patch epoch change retries.

* cargo fmt

* Fix failed DKGs when doing restores.

* cargo fmt

* fix tests

* Logging info for CI datil instances.

* Open ports for a second instance of anvil in the yml files.

* Load hex data from test state into anvil.

* remove unused file

* comment out #if false

* take 2?

* fix test reference

* Use HEX file to load state, rather than a JSON file.

* Refactor to insure that for Datil tests we don't rely on naga cache'd chain state.

* switch lit-recovery binaries for testing

* Remove keyset_identifier as an enum and clean up tests.

* fix datil backup selection of binary.

* cargo fmt

* yml test.

* if false yml

* comment out version upgrade

* yml update

* try a var for the upgrade workflow ?

* update lit-os cargo.lock

* update gitinfo

---------

Co-authored-by: Howard Tam <[email protected]>
Co-authored-by: Copilot <[email protected]>

* Feature/node 4876 - using both chains in some tests ( adapt end user to support pkps in both chains ) (#42)

* Update test

* Update tests

* Setup

* restore datil keys into a second keyset

* remove large binaries

* remove large zip

* cargo fmt

* Updates to DKG / Epoch change processes + correct saving of data to chain.

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* fix actions in testnet

* cargo fmt

* Add BLS decryption assertions

* Use correct keyset for assertion

* Update lit-node-testnet to work with ImportedDatilTestnet + start on PKP assertions across chain

This is incomplete.

* Add notes

* Add notes

* Update test to interact with ImportedDatilTestnet

* remove unneded code

* refactor & remove duplications

* Refactor & add datil node secrets

* Update rust/lit-node/lit-node/src/tasks/fsm/epoch_change.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/src/tss/common/restore/eks_and_ds.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/Cargo.toml

Co-authored-by: Copilot <[email protected]>

* whole lotta shaking going on.  ( endpoints changed throughout )

* PKP sign works.

* cargo clippy

* cargo lock cleanup

* git merge fixes

* cargo toml update ( to trigger cargo.lock update )

* cargo.lock update + config.rs in the resolver formatting.

* cargo fmt

* fix panic when node was kicked (realmid = 0)

* patch epoch change retries.

* cargo fmt

* Expose underlying error when peer_connect attestation fails

* Fix failed DKGs when doing restores.

* cargo fmt

* fix tests !

* fix tests

* Logging info for CI datil instances.

* Open ports for a second instance of anvil in the yml files.

* Load hex data from test state into anvil.

* Feature/price setting script (#43)

* first pass at price setting script

* price setting script seems like it will work

* script works on dev and prod

* Update blockchain/contracts/scripts/generatePriceSettingTransactions.ts

Co-authored-by: Copilot <[email protected]>

* remove unused map

* regen contracts

* 100x decrease for testnets when setting prices

* divide by 10000 when setting naga-test prices

* actually, made naga-test 100000x cheaper than naga prod

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Fix: Add `correlation_id` to `lit-actions` spans (#25)

* Add spans for actions

* Add correlation_id to TracingMiddleware

* Checkout server.rs

* Extract correlation_id'

---------

Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Refactors OsQuery metrics emission to use OpenTelemetry Gauges (#28)

* Add spans for actions

* Improve metrics

* first pass at adding request id to logs

* Update rust/lit-node/lit-node/src/main.rs

Co-authored-by: Copilot <[email protected]>

* change how we set the request id on the span

* clippy

* Add correlation_id to TracingMiddleware

* Checkout server.rs

* Extract correlation_id'

* Fix clippy wanrning

* separate info from gauge metrics (#44)

* Revert "Merge branch 'inf-852-fix-tracing-lit-actions' into inf-753-olly-metrics"

This reverts commit b0a6d38, reversing
changes made to 4a58992.

* Revert "Merge remote-tracking branch 'origin/feature/NODE-4860-correlation-id-in-logs' into inf-753-olly-metrics"

This reverts commit 7be7bbe, reversing
changes made to b0a6d38.

* cargo fmt

---------

Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Adam Reif <[email protected]>

* remove unused file

* comment out #if false

* take 2?

* fix test reference

* Use HEX file to load state, rather than a JSON file.

* Refactor to insure that for Datil tests we don't rely on naga cache'd chain state.

* switch lit-recovery binaries for testing

* Remove keyset_identifier as an enum and clean up tests.

* Fix up tests

* cargo fmt

* fix datil backup selection of binary.

* logging, shorten handshake challange value.

* fix "default" keyset bug :(

* cargo fmt

* yml test.

* if false yml

* comment out version upgrade

* yml update

* try a var for the upgrade workflow ?

* upgrade yml with default if statement.

* cargo clippy.

* Update lit-os cargo.lock

* update lit-os cargo.lock

* update gitinfo

* fix restore test to use Datil directly.

* cargo fmt

* patch so both long running restores work :(

* Update epoch change long to deal with keysets

* small shiva refactor

* difference between pkp & root key lookups when getting a keyset.

* remove datil from lit action testing.

* clean up integration test setup to account for keyset settings.

* limit caches to only active states.

* Fix Shadow splicing test, make some integration force builds

* update abis & lit-os lock file

* cargo fmt + minor cleanup

* Refactor one of the last timing related tests.

* ABI staleness, remove more testing feature flags

* Make the StakerAccountSetupMapper Send+Sync to allow Shiva to build.

* Clean up complaints.

* Update to determine key set for recovery (#58)

* update to determine key set for recovery

Signed-off-by: Michael Lodder <[email protected]>

* add test when no key set is on chain

Signed-off-by: Michael Lodder <[email protected]>

* update test

Signed-off-by: Michael Lodder <[email protected]>

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Brendon <[email protected]>

* Better logging.

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Howard Tam <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Abhi... <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Michael Lodder <[email protected]>

* abi staleness + cargo fmt.

* Remove traces of custom version upgrade testing. (#57)

* fix: Invalid signature combination (#65)

* fix invalid signature combination

Signed-off-by: Michael Lodder <[email protected]>

* fix tests

Signed-off-by: Michael Lodder <[email protected]>

---------

Signed-off-by: Michael Lodder <[email protected]>

* re-instate decryption assertion

* clippy.

* remove unused val

* make openapi spec (#66)

* make openapi spec

Signed-off-by: Michael Lodder <[email protected]>

* remove "with same ip and port" test - chck /w Mike?

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: GTC6244 <[email protected]>

* Feature/node 4923 refactor upgrade tests to use local configurations (#60)

* Remove traces of custom version upgrade testing.

* First  few changes - start simplification.

* restore key_types in staking contract, allow validator cloning.

* Update to allow old version to load

* update local builds

* debugging updates

* better logging, disable network checks between epoch changes ( for now )

* clippy!

* add a filnal check.

* cargo fmt

* clippy.

* Download an install old binaries as required.

* ignore upgrades by default.

* re-include Pallas curves in deployment scripts.

* add functions to  create proper upgrade envrionment for old nodes ( remove curves, change yaml files ).

* blockchain abi stales + cargo fmt.

* disable test ( with same port )

* re-add >= to epoch change

* Update contracts to include keysets_deprecated type in tests.

* missing "deprecated" param in test

* merge master fixes

* try to update docs

* force deploy on lit-actions.

* force deploy on more lit actions.

* Feature/node 4931 implement lit actions for keysets (datil support!) (#62)

* Lit Action Server side updates /w Docs

* lit-node side updates.

* Add a few tests.

* Update rust/lit-node/lit-node-testnet/src/testnet/contracts_repo.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-actions/ext/js/02_litActionsSDK.js

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node-testnet/src/testnet/contracts_repo.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node-testnet/src/testnet/contracts_repo.rs

Co-authored-by: Copilot <[email protected]>

* copilot suggestions

* documentation update

* cargo fmt

* clipppppppyyyyy!

* Update docs?

* build docs again!

* add in missing "keysetId" values from  lit action code samples that are stored as string in mods.

* force lit-action  datil tests to run outside of caches.

* test forcing Lit Actions to recompile ( cache with sometimes using Datil seems off )

* regen docs

* Version upgrade check

* Refactor for explicit endpoint change - include keyset as part for lit-actions, which can be overwritten wtih a js param.

* documentation update

* use force_deploy for lit actions

* error handling on reading anvil file.

* read/write hex data as string + better error handling.

* better debugging.

* revert keyset in session sig issue.   ( creates invalid IPFS CID value! )

* cargo fmt

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Adam Reif <[email protected]>

* Remerge build.rs!

* cargo fmt

* more debugging.

* remerge build.rs

* cargo fmt

* Feature/node 4947 set default keyset on contract deployment (#82)

* set default realm config in the deploy.ts, and remove from test builder in node code ;-)

* prettier!

* switch function name.... which makes me wonder if my test executed!

* ABI stalensss

* add CI debbugging.

* Reset all defaults for realm #1

* Feature/node 4940 fix keyset ci times (#76)

* Make datil always available, even if not used.

* Add debugging.

* update build.rs

* debugging

* Test / Refactor / Discover - anvil cache issues.

* rename to edge testing, add a matrix

* clippy !

* test splitting out the edge related tests.

* cargo fmt

* config update (nomenclature, mainly)

* Use an updated version of datil contracts that can be properly cleared of root keys.

* remove unit test partitioning

* add debugging info

* Add keyset into ClientBuilder.

* force deploy for the child lit action function.

* fix backup-long to use only naga keysets.

* cargo fmt

* Feature/node 4771 add test chains for non prod naga deploys (#79)

* Make datil always available, even if not used.

* Add debugging.

* update build.rs

* debugging

* Test / Refactor / Discover - anvil cache issues.

* rename to edge testing, add a matrix

* clippy !

* test splitting out the edge related tests.

* cargo fmt

* config update (nomenclature, mainly)

* Use an updated version of datil contracts that can be properly cleared of root keys.

* remove unit test partitioning

* add debugging info

* Add keyset into ClientBuilder.

* force deploy for the child lit action function.

* first pass

* Updates to make testing easier

* Simplifications of datil vs normal PKPs.

* Refactor to search for a local test configuration file, instead of duplicating tests.

* Remove duplicate tests.

* ensure datil wallets get funded.

* clean up datil PKP functions.

* Revert the proper funding amount :)

* Feature/node 4948 fix broken pkppermissions helper lookup (#81)

* quick fix  of the pkphelper contract.

* update js/ts contratc tests

* simplify and updated abi

* Update PKP Helper V2 as well.

* Prettier.

* Update ABI Staleness

* Update rust/lit-node/lit-node-testnet/src/lib.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/live_testnet_b.toml

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-node/lit-node/live_testnet_sample.toml

Co-authored-by: Copilot <[email protected]>

* fix merge issues

---------

Co-authored-by: Copilot <[email protected]>

---------

Signed-off-by: Michael Lodder <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Michael Lodder <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Howard Tam <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
Co-authored-by: Abhi... <[email protected]>
Co-authored-by: Chris Cassano <[email protected]>
# What

Improved RPC fallback selection logic to choose default-priority (0) entries when no healthy RPCs are available.

New (not-yet polled) RPCs are now initialized as `Unhealthy` instead of Healthy with a fake latency based on order of definition.

This change removes reliance on order of defining RPCs in rpc-config.yaml

followup to #73

# Why

Some of our services (prov bootstrap) request RPCs before healthcheck poller has had a chance to run. We thus have no idea which RPC from the list to return and should prefer the one with the highest likelihood of being available.

Previously, new entries that have not yet been polled were initialized as Healthy, which according to the new selection method led to the replica IP being selected (highest priority) before it could be proven to be Unhealthy by the poller, which broke prov bootstrap.  
The previous selection algorithm just selected by order of definition in this case (first RPC defined in the file) which is brittle and darkmagic-y as well.

This change ensures we use a more reliable default-priority remote RPC anytime we don't know which RPC is actually available, which is typically a public or private remote RPC to that chain and can be more safely assumed to be available.

This default (priority-0) RPC is now mandatory for every chain that might be used in such way and prio 0 now has a special meaning.

# Background

During prov bootstrap, the healthcheck poller requests a yellowstone RPC immediately on boot without waiting for a healthcheck cycle. Previously, a replica URL was selected due to high prio because not-yet polled RPCs were initialized as Healthy, which made prov boot fail.
# What

This PR marks the `rpc_healthcheck_enabled` flag as deprecated and removes its functionality from the RPC health check polling logic.

Cleanup of this flag from the config will happen in a later PR

# Why

The RPC health check functionality should always be enabled, making the toggle unnecessary. Removing the conditional check simplifies the code.

# Risks

- Malfunction of the RPC Healthchecker can lead to the node going down
- update cargo lockfile for `lit-os` workspace to fix `lit-node-operator` build
- check `lit-node-operator`clippy in CI
- fix clippy for `lit-node-operator`
* fix: Use RAII to deregister payment tracker for all requests

* refac: Cargo fmt

* test: Fix Delegation failure test scenario

* test: Debug failing session_sig_with_mgb_pkp_lit_action

* refac: Remove debug statements
### TL;DR

Bump lit_node version from 2.1.9 to 2.1.10
* works

* cargo fmt

* Update rust/lit-core/lit-observability/Cargo.toml

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-core/lit-api-core/src/context/mod.rs

Co-authored-by: Copilot <[email protected]>

* Update rust/lit-core/lit-observability/src/logging/privacy_filter.rs

Co-authored-by: Copilot <[email protected]>

* Update cargo locks and privacy mode logic in lit-api-core

* privacy-mode: proposed changes (#95)

* proposed changes

* count privacy mode requests received via otel

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Adam Reif <[email protected]>
### TL;DR

Improved error message when setting balance fails in the testnet node.

### What changed?

Enhanced the error message in `EndUser::set_balance` to include the wallet address and the amount being set when a transaction fails. This provides more context for debugging purposes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants