ft. modernise py-libp2p (#618)

* fix pyproject.toml , add ruff

* rm lock

* make progress

* add poetry lock ignore

* fix type issues

* fix tcp type errors

* fix text example - type error - wrong args

* add setuptools to dev

* test ci

* fix docs build

* fix type issues for new_swarm & new_host

* fix types in gossipsub

* fix type issues in noise

* wip: factories

* revert factories

* fix more type issues

* more type fixes

* fix: add null checks for noise protocol initialization and key handling

* corrected argument-errors in peerId and Multiaddr in peer tests

* fix: Noice - remove redundant type casts in BaseNoiseMsgReadWriter

* fix: update test_notify.py to use SwarmFactory.create_batch_and_listen, fix type hints, and comment out ClosedStream assertions

* Fix type checks for pubsub module

Signed-off-by: sukhman <sukhmansinghsaluja@gmail.com>

* Fix type checks for pubsub module-tests

Signed-off-by: sukhman <sukhmansinghsaluja@gmail.com>

* noise: add checks for uninitialized protocol and key states in PatternXX

Signed-off-by: varun-r-mallya <varunrmallya@gmail.com>

* pubsub: add None checks for optional fields in FloodSub and Pubsub

Signed-off-by: varun-r-mallya <varunrmallya@gmail.com>

* Fix type hints and improve testing

Signed-off-by: varun-r-mallya <varunrmallya@gmail.com>

* remove redundant checks

Signed-off-by: varun-r-mallya <varunrmallya@gmail.com>

* fix build issues

* add optional to trio service

* fix types

* fix type errors

* Fix type errors

Signed-off-by: varun-r-mallya <varunrmallya@gmail.com>

* fixed more-type checks in crypto and peer_data files

* wip: factories

* replaced union with optional

* fix: type-error in interp-utils and peerinfo

* replace pyright with pyrefly

* add pyrefly.toml

* wip: fix multiselect issues

* try typecheck

* base check

* mcache test fixes , typecheck ci update

* fix ci

* will this work

* minor fix

* use poetry

* fix wokflow

* use cache,fix err

* fix pyrefly.toml

* fix pyrefly.toml

* fix cache in ci

* deploy commit

* add main baseline

* update to v5

* improve typecheck ci (#14)

* fix typo

* remove holepunching code (#16)

* fix gossipsub typeerrors (#17)

* fix: ensure initiator user includes remote peer id in handshake (#15)

* fix ci (#19)

* typefix: custom_types | core/peerinfo/test_peer_info | io/abc | pubsub/floodsub | protocol_muxer/multiselect (#18)

* fix: Typefixes in PeerInfo  (#21)

* fix minor type issue (#22)

* fix type errors in pubsub (#24)

* fix: Minor typefixes in tests (#23)

* Fix failing tests for type-fixed test/pubsub (#8)

* move pyrefly & ruff to pyproject.toml & rm .project-template (#28)

* move the async_context file to tests/core

* move crypto test to crypto folder

* fix: some typefixes (#25)

* fix type errors

* fix type issues

* fix: update gRPC API usage in autonat_pb2_grpc.py (#31)

* md: typecheck ci

* rm comments

* clean up : from review suggestions

* use | None over Optional as per new python standards

* drop supporto for py3.9

* newsfragments

---------

Signed-off-by: sukhman <sukhmansinghsaluja@gmail.com>
Signed-off-by: varun-r-mallya <varunrmallya@gmail.com>
Co-authored-by: acul71 <luca.pisani@birdo.net>
Co-authored-by: kaneki003 <sakshamchauhan707@gmail.com>
Co-authored-by: sukhman <sukhmansinghsaluja@gmail.com>
Co-authored-by: varun-r-mallya <varunrmallya@gmail.com>
Co-authored-by: varunrmallya <100590632+varun-r-mallya@users.noreply.github.com>
Co-authored-by: lla-dane <abhinavagarwalla6@gmail.com>
Co-authored-by: Collins <ArtemisfowlX@protonmail.com>
Co-authored-by: Abhinav Agarwalla <120122716+lla-dane@users.noreply.github.com>
Co-authored-by: guha-rahul <52607971+guha-rahul@users.noreply.github.com>
Co-authored-by: Sukhman Singh <63765293+sukhman-sukh@users.noreply.github.com>
Co-authored-by: acul71 <34693171+acul71@users.noreply.github.com>
Co-authored-by: pacrob <5199899+pacrob@users.noreply.github.com>

Author: 9 people

.github/workflows/tox.yml

@@ -16,10 +16,10 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python: ['3.9', '3.10', '3.11', '3.12', '3.13']
+        python: ["3.10", "3.11", "3.12", "3.13"]
         toxenv: [core, interop, lint, wheel, demos]
         include:
-          - python: '3.10'
+          - python: "3.10"
             toxenv: docs
       fail-fast: false
     steps:
@@ -46,7 +46,7 @@ jobs:
     runs-on: windows-latest
     strategy:
       matrix:
-        python-version: ['3.11', '3.12', '3.13']
+        python-version: ["3.11", "3.12", "3.13"]
         toxenv: [core, wheel]
       fail-fast: false
     steps:

.gitignore

@@ -146,6 +146,9 @@ instance/
 # PyBuilder
 target/
 
+# PyRight Config
+pyrightconfig.json
+
 # Jupyter Notebook
 .ipynb_checkpoints
 
@@ -171,3 +174,7 @@ env.bak/
 
 # mkdocs documentation
 /site
+
+#lockfiles
+uv.lock
+poetry.lock

.pre-commit-config.yaml

@@ -1,59 +1,49 @@
 exclude: '.project-template|docs/conf.py|.*pb2\..*'
 repos:
--   repo: https://github.com/pre-commit/pre-commit-hooks
+  - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v5.0.0
     hooks:
-    -   id: check-yaml
-    -   id: check-toml
-    -   id: end-of-file-fixer
-    -   id: trailing-whitespace
--   repo: https://github.com/asottile/pyupgrade
-    rev: v3.15.0
-    hooks:
-    -   id: pyupgrade
-        args: [--py39-plus]
--   repo: https://github.com/psf/black
-    rev: 23.9.1
-    hooks:
-    -   id: black
--   repo: https://github.com/PyCQA/flake8
-    rev: 6.1.0
-    hooks:
-    -   id: flake8
-        additional_dependencies:
-        -   flake8-bugbear==23.9.16
-        exclude: setup.py
--   repo: https://github.com/PyCQA/autoflake
-    rev: v2.2.1
-    hooks:
-    -   id: autoflake
--   repo: https://github.com/pycqa/isort
-    rev: 5.12.0
-    hooks:
-    -   id: isort
--   repo: https://github.com/pycqa/pydocstyle
-    rev: 6.3.0
-    hooks:
-    -   id: pydocstyle
-        additional_dependencies:
-        -   tomli  # required until >= python311
--   repo: https://github.com/executablebooks/mdformat
+      - id: check-yaml
+      - id: check-toml
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+  - repo: https://github.com/asottile/pyupgrade
+    rev: v3.20.0
+    hooks:
+      - id: pyupgrade
+        args: [--py310-plus]
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.11.10
+    hooks:
+      - id: ruff
+        args: [--fix, --exit-non-zero-on-fix]
+      - id: ruff-format
+  - repo: https://github.com/executablebooks/mdformat
     rev: 0.7.22
     hooks:
-    -   id: mdformat
+      - id: mdformat
         additional_dependencies:
-        -   mdformat-gfm
--   repo: local
+          - mdformat-gfm
+  - repo: local
     hooks:
-    -   id: mypy-local
+      - id: mypy-local
         name: run mypy with all dev dependencies present
-        entry: python -m mypy -p libp2p
+        entry: mypy -p libp2p
+        language: system
+        always_run: true
+        pass_filenames: false
+  - repo: local
+    hooks:
+      - id: pyrefly-local
+        name: run pyrefly typecheck locally
+        entry: pyrefly check
         language: system
         always_run: true
         pass_filenames: false
--   repo: local
+
+  - repo: local
     hooks:
-    -   id: check-rst-files
+      - id: check-rst-files
         name: Check for .rst files in the top-level directory
         entry: python -c "import glob, sys; rst_files = glob.glob('*.rst'); sys.exit(1) if rst_files else sys.exit(0)"
         language: system

.project-template/fill_template_vars.py

@@ -7,6 +7,7 @@ help:
 	@echo "clean-pyc - remove Python file artifacts"
 	@echo "clean - run clean-build and clean-pyc"
 	@echo "dist - build package and cat contents of the dist directory"
+	@echo "fix - fix formatting & linting issues with ruff"
 	@echo "lint - fix linting issues with pre-commit"
 	@echo "test - run tests quickly with the default Python"
 	@echo "docs - generate docs and open in browser (linux-docs for version on linux)"
@@ -37,8 +38,14 @@ lint:
 		&& pre-commit run --all-files --show-diff-on-failure \
 	)
 
+fix:
+	python -m ruff check --fix
+
+typecheck:
+	pre-commit run mypy-local --all-files && pre-commit run pyrefly-local --all-files
+
 test:
-	python -m pytest tests
+	python -m pytest tests -n auto
 
 # protobufs management
 

.project-template/refill_template_vars.py

@@ -15,14 +15,24 @@
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 # sys.path.insert(0, os.path.abspath('.'))
 
+import doctest
 import os
+import sys
+from unittest.mock import MagicMock
+
+try:
+    import tomllib
+except ModuleNotFoundError:
+    # For Python < 3.11
+    import tomli as tomllib  # type: ignore (In case of >3.11 Pyrefly doesnt find tomli , which is right but a false flag)
+
+# Path to pyproject.toml (assuming conf.py is in a 'docs' subdirectory)
+pyproject_path = os.path.join(os.path.dirname(__file__), "..", "pyproject.toml")
 
-DIR = os.path.dirname(__file__)
-with open(os.path.join(DIR, "../setup.py"), "r") as f:
-    for line in f:
-        if "version=" in line:
-            setup_version = line.split('"')[1]
-            break
+with open(pyproject_path, "rb") as f:
+    pyproject_data = tomllib.load(f)
+
+setup_version = pyproject_data["project"]["version"]
 
 # -- General configuration ------------------------------------------------
 
@@ -302,7 +312,6 @@
 
 # -- Doctest configuration ----------------------------------------
 
-import doctest
 
 doctest_default_flags = (
     0
@@ -317,10 +326,9 @@
 # Mock out dependencies that are unbuildable on readthedocs, as recommended here:
 # https://docs.readthedocs.io/en/rel/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
 
-import sys
-from unittest.mock import MagicMock
 
-# Add new modules to mock here (it should be the same list as those excluded in setup.py)
+# Add new modules to mock here (it should be the same list
+# as those excluded in pyproject.toml)
 MOCK_MODULES = [
     "fastecdsa",
     "fastecdsa.encoding",
@@ -338,4 +346,4 @@
 
 # Allow duplicate object descriptions
 nitpicky = False
-nitpick_ignore = [("py:class", "type")]
+nitpick_ignore = [("py:class", "type")]

.project-template/template_vars.txt

@@ -24,9 +24,6 @@ async def main():
     insecure_transport = InsecureTransport(
         # local_key_pair: The key pair used for libp2p identity
         local_key_pair=key_pair,
-        # secure_bytes_provider: Optional function to generate secure random bytes
-        # (defaults to secrets.token_bytes)
-        secure_bytes_provider=None,  # Use default implementation
     )
 
     # Create a security options dictionary mapping protocol ID to transport

Makefile

@@ -9,8 +9,10 @@
 from libp2p.crypto.secp256k1 import (
     create_new_key_pair,
 )
-from libp2p.security.noise.transport import PROTOCOL_ID as NOISE_PROTOCOL_ID
-from libp2p.security.noise.transport import Transport as NoiseTransport
+from libp2p.security.noise.transport import (
+    PROTOCOL_ID as NOISE_PROTOCOL_ID,
+    Transport as NoiseTransport,
+)
 
 
 async def main():