Skip to content

Commit 16b8125

Browse files
emberianemberian
authored
New gossip_net backend using libp2p (#3988)
This is the revival of the tmp/cmr/net2 branch rebased onto develop. Some important user-facing changes: - No separate discovery/communication/etc ports. One port for all public daemon communications. - Automatic port forwarding with UPnP. If your local network supports UPnP, there should be no configuration required. - Local peer discovery. If your local network supports mDNS broadcast, coda daemons will automatically discover each other. This includes several daemons on the same machine- no more building peer lists! - New libp2p keypairs. These are managed the same as our key pairs with secret_file. Without configuration, key pairs are ephemeral and will disappear when the daemon restarts. (TODO: should we instead persist the keypair? does it matter for non-infrastructure?) Some important internal changes: - All daemon-daemon connections are now authenticated and confidential. - Connections are no longer transient and per-request. Individual requests get multiplexed as their own stream over the one connection between the peers. This is analogous to HTTP/2. Outgoing connections will appear to originate from the libp2p listening port, vs some transient port. Outstanding details: - Trust system needs to get augmented to track Peer.t instead of just an IP. Until then we can't implement ban_notify (#4093, #4096). - Libp2p has little per-connection structured reporting, some things we currently penalize trust for are not detected (eg opening a libp2p connection without also opening a coda RPC stream) (#4098). - New pubsub allows banning senders by peer ID. We currently don't do this but we should ban peerIDs that originated bad info and not just the IP of the whoever relayed it to us (#4096). - ~~Current pubsub validation flow goes a bit against the libp2p grain, and it's not clear to me that the current behavior will survive [this libp2p PR](libp2p/go-libp2p-kad-dht#388). There's an inline comment near the should_forward_message impl (#4097).~~ done - Connection limit enforcement (#4095) Other changes: - Rips out the last vestiges of old membership, which aren't in use. - The connection info in envelopes is much more accurate now. We shouldn't start trusting it just yet due to some future vagaries around relaying. - bump nixpkgs version Future improvements: - IPv6. There's a hardcoded IPv4 assumption in the helper around IP filtering. - Investigate libp2p autorelay. This should help nodes in restrictive networks achieve better connectivity, but has a host of problems. - Intelligent request routing. I believe we can use the "provider" feature to, at the very least, only send eg sync/bootstrap requests to nodes who believe themselves to be in sync. There are other options.
1 parent 79a007b commit 16b8125

File tree

169 files changed

+4779
-43210
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

169 files changed

+4779
-43210
lines changed

‎.circleci/config.yml‎

Lines changed: 348 additions & 79 deletions
Large diffs are not rendered by default.

‎.circleci/config.yml.jinja‎

Lines changed: 93 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ jobs:
5252
build-archive:
5353
resource_class: xlarge
5454
docker:
55-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
55+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
5656
environment:
5757
CODA_DOCKER: true
5858
HASURA_PORT: 8080
@@ -98,7 +98,7 @@ jobs:
9898
command: ./scripts/archive/build-release-archives.sh
9999
lint:
100100
docker:
101-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
101+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
102102
steps:
103103
- checkout
104104
- run:
@@ -127,9 +127,12 @@ jobs:
127127
command: ./scripts/require-ppx-coda.py
128128
lint-opt:
129129
docker:
130-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
130+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
131131
steps:
132132
- checkout
133+
- run:
134+
name: Show USER
135+
command: echo $USER
133136
- run:
134137
name: Update Submodules
135138
command: git submodule sync && git submodule update --init --recursive
@@ -151,59 +154,6 @@ jobs:
151154
name: Update branch protection rule from test configuration
152155
command: ./scripts/test.py required-status >required_status && cat required_status && ./scripts/update_branch_rule.py required_status
153156

154-
build-auxiliary:
155-
docker:
156-
- image: nixos/nix
157-
steps:
158-
- checkout
159-
- run:
160-
name: Install tools
161-
command: apk add patchelf dpkg tar
162-
- run:
163-
name: Install cachix
164-
command: nix-env -iA cachix -f https://cachix.org/api/v1/install
165-
- run:
166-
name: Build libp2p_helper using cachix
167-
command: |
168-
cachix use codaprotocol
169-
cd src/app/libp2p_helper
170-
171-
if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
172-
echo "CACHIX_SIGNING_KEY is not set"
173-
nix-build default.nix
174-
else
175-
nix-build default.nix | cachix push codaprotocol
176-
fi
177-
- run:
178-
name: Create package dir
179-
command: mkdir -p package
180-
- run:
181-
name: Collect and strip binary
182-
command: |
183-
cp src/app/libp2p_helper/result/bin/libp2p_helper package/coda-libp2p_helper
184-
chmod +w package/coda-libp2p_helper
185-
patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 package/coda-libp2p_helper
186-
- run:
187-
name: Create deb
188-
command: |
189-
mkdir -p package/deb/usr/local/sbin
190-
cp package/coda-libp2p_helper package/deb/usr/local/sbin/.
191-
mkdir -p package/deb/DEBIAN
192-
cp src/app/libp2p_helper/build/DEBIAN/control package/deb/DEBIAN/control
193-
DATE=$(date +%Y-%m-%d)
194-
sed -i "s/DATE/${DATE}/" package/deb/DEBIAN/control
195-
find src/app/libp2p_helper -type f -print0 | xargs -0 sha1sum | sort | sha1sum | cut -f 1 -d ' ' > /tmp/sha1sig
196-
HASH=$(cat /tmp/sha1sig)
197-
sed -i "s/HASH/${HASH}/" package/deb/DEBIAN/control
198-
echo "------------------------------------"
199-
cat package/deb/DEBIAN/control
200-
echo "------------------------------------"
201-
find package/deb/
202-
echo "------------------------------------"
203-
dpkg-deb -v --build package/deb/ package/coda-discovery-${HASH}.deb
204-
- store_artifacts:
205-
path: package
206-
207157
build-macos:
208158
macos:
209159
xcode: "10.2.1"
@@ -287,9 +237,9 @@ jobs:
287237
cd src/app/libp2p_helper
288238
if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
289239
echo "CACHIX_SIGNING_KEY is not set"
290-
nix-build default.nix
240+
nix-build --option sandbox false default.nix
291241
else
292-
nix-build default.nix | cachix push codaprotocol
242+
nix-build --option sandbox false default.nix | cachix push codaprotocol
293243
fi
294244

295245
### dune
@@ -353,7 +303,7 @@ jobs:
353303
build-artifacts--{{profile}}:
354304
resource_class: xlarge
355305
docker:
356-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
306+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
357307
steps:
358308
- checkout
359309
- run:
@@ -366,6 +316,22 @@ jobs:
366316
- run:
367317
name: Pin external packages
368318
command: ./scripts/pin-external-packages.sh
319+
- run:
320+
name: Install cachix
321+
command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
322+
- run:
323+
name: Build libp2p_helper using cachix
324+
command: |
325+
export USER=opam
326+
. ~/.nix-profile/etc/profile.d/nix.sh
327+
cachix use codaprotocol
328+
cd src/app/libp2p_helper
329+
if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
330+
echo "CACHIX_SIGNING_KEY is not set"
331+
nix-build --option sandbox false default.nix
332+
else
333+
nix-build --option sandbox false default.nix | cachix push codaprotocol
334+
fi
369335
- run:
370336
name: Build OCaml
371337
command: ./scripts/skip_if_only_frontend.sh bash -c 'eval `opam config env` && make build 2>&1 | tee /tmp/artifacts/buildocaml.log'
@@ -434,7 +400,7 @@ jobs:
434400
test-unit--{{profile}}:
435401
resource_class: xlarge
436402
docker:
437-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
403+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
438404
steps:
439405
- checkout
440406
- run: ulimit -c unlimited
@@ -463,6 +429,22 @@ jobs:
463429
#chmod -R 777 src/app/libp2p_helper
464430
#Do the build
465431
#sudo -H -u nix bash -c '. ~/.nix-profile/etc/profile.d/nix.sh && cachix use codaprotocol && cd src/app/libp2p_helper && nix-build default.nix'
432+
- run:
433+
name: Install cachix
434+
command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
435+
- run:
436+
name: Build libp2p_helper using cachix
437+
command: |
438+
export USER=opam
439+
. ~/.nix-profile/etc/profile.d/nix.sh
440+
cachix use codaprotocol
441+
cd src/app/libp2p_helper
442+
if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
443+
echo "CACHIX_SIGNING_KEY is not set"
444+
nix-build --option sandbox false default.nix
445+
else
446+
nix-build --option sandbox false default.nix | cachix push codaprotocol
447+
fi
466448
- run:
467449
name: Run unit tests
468450
command: ./scripts/skip_if_only_frontend.sh bash -c 'source ~/.profile && dune build --profile={{profile}} -j8 && (dune runtest src/lib --profile={{profile}} -j8 || (./scripts/link-coredumps.sh && false))'
@@ -475,13 +457,29 @@ jobs:
475457
test-unit--{{profile}}:
476458
resource_class: xlarge
477459
docker:
478-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
460+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
479461
steps:
480462
- checkout
481463
- run: ulimit -c unlimited
482464
- run:
483465
name: Update OPAM
484466
command: ./scripts/update-opam-in-docker.sh
467+
- run:
468+
name: Install cachix
469+
command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
470+
- run:
471+
name: Build libp2p_helper using cachix
472+
command: |
473+
export USER=opam
474+
. ~/.nix-profile/etc/profile.d/nix.sh
475+
cachix use codaprotocol
476+
cd src/app/libp2p_helper
477+
if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
478+
echo "CACHIX_SIGNING_KEY is not set"
479+
nix-build --option sandbox false default.nix
480+
else
481+
nix-build --option sandbox false default.nix | cachix push codaprotocol
482+
fi
485483
- run:
486484
name: Run unit tests
487485
command: ./scripts/skip_if_only_frontend.sh bash -c 'source ~/.profile && dune build --profile={{profile}} -j8 && (dune runtest src/lib --profile={{profile}} -j8 || (./scripts/link-coredumps.sh && false))'
@@ -494,12 +492,28 @@ jobs:
494492
test--{{profile}}:
495493
resource_class: large
496494
docker:
497-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
495+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
498496
steps:
499497
- checkout
500498
- run:
501499
name: Update OPAM
502500
command: ./scripts/update-opam-in-docker.sh
501+
- run:
502+
name: Install cachix
503+
command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
504+
- run:
505+
name: Build libp2p_helper using cachix
506+
command: |
507+
export USER=opam
508+
. ~/.nix-profile/etc/profile.d/nix.sh
509+
cachix use codaprotocol
510+
cd src/app/libp2p_helper
511+
if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
512+
echo "CACHIX_SIGNING_KEY is not set"
513+
nix-build --option sandbox false default.nix
514+
else
515+
nix-build --option sandbox false default.nix | cachix push codaprotocol
516+
fi
503517
{%- for test in small_curves_tests[profile] %}
504518
- run:
505519
name: Running test -- {{profile}}:{{test}}
@@ -513,12 +527,28 @@ jobs:
513527
test--{{profile}}:
514528
resource_class: xlarge
515529
docker:
516-
- image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
530+
- image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
517531
steps:
518532
- checkout
519533
- run:
520534
name: Update OPAM
521535
command: ./scripts/update-opam-in-docker.sh
536+
- run:
537+
name: Install cachix
538+
command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
539+
- run:
540+
name: Build libp2p_helper using cachix
541+
command: |
542+
export USER=opam
543+
. ~/.nix-profile/etc/profile.d/nix.sh
544+
cachix use codaprotocol
545+
cd src/app/libp2p_helper
546+
if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
547+
echo "CACHIX_SIGNING_KEY is not set"
548+
nix-build --option sandbox false default.nix
549+
else
550+
nix-build --option sandbox false default.nix | cachix push codaprotocol
551+
fi
522552
{%- for test in medium_curves_and_other_tests[profile] %}
523553
- run:
524554
name: Running test -- {{profile}}:{{test}}
@@ -537,7 +567,6 @@ workflows:
537567
jobs:
538568
- lint
539569
- lint-opt
540-
- build-auxiliary
541570
- update-branch-protection:
542571
filters:
543572
branches:

‎.dockerignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
*
22
!src/opam.export
33
!coda.deb
4+
!coda-discovery.deb
45
!src/external/**/*
5-
!src/app/kademlia-haskell
66
!src/app/libp2p_helper
77
!src/app/trace-tool

‎.gitattributes‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,5 @@ frontend/website/static/presskit.zip filter=lfs diff=lfs merge=lfs -text
1111
*.woff filter=lfs diff=lfs merge=lfs -text
1212
*.woff2 filter=lfs diff=lfs merge=lfs -text
1313
frontend/website/static/*.bc.js filter=lfs diff=lfs merge=lfs -text
14-
src/app/kademlia-haskell/packages.nix linguist-generated=true
1514
src/app/archive/archive_graphql_schema.json linguist-generated=true
1615
docs/res/block_production_fsm.dot.png filter=lfs diff=lfs merge=lfs -text

‎.github/labels.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
"color": "#9791a0"
3333
},
3434
{
35-
"name": "area-kademlia",
35+
"name": "area-libp2p",
3636
"color": "#9791a0"
3737
},
3838
{

‎.mergify.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@ pull_request_rules:
22
- name: automatically merge approved PRs with the ready-to-merge label
33
conditions:
44
- "status-success=ci/circleci: build-artifacts--testnet_postake_medium_curves"
5-
- "status-success=ci/circleci: build-auxiliary"
65
- "status-success=ci/circleci: build-wallet"
76
- "status-success=ci/circleci: lint"
87
- "status-success=ci/circleci: test--fake_hash"

‎CODEOWNERS‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
/src/app/cli/src/tests @emberian @bkase @ghost-not-in-the-shell @deepthiskumar
1919
/src/app/cli/src/coda.ml @nholland94 @bkase @emberian @imeckler
2020
/src/app/cli/src/init @nholland94 @bkase @emberian @imeckler
21-
/src/app/kademlia-haskell/ @bkase @emberian @psteckler
2221
/src/app/libp2p_helper @emberian @bkase @nholland94
2322
/src/app/reformat/ @bkase
2423
/src/app/trace-tool/ @emberian @mrmr1993
@@ -50,7 +49,6 @@
5049
/src/lib/group_map/ @rbkhmrcr @vanishreerao @imeckler
5150
/src/lib/hash_prefixes/ @rbkhmrcr @vanishreerao @imeckler
5251
/src/lib/interruptible/ @bkase @nholland94
53-
/src/lib/kademlia @bkase @emberian @psteckler
5452
/src/lib/key_value_database/ @bkase @nholland94 @psteckler
5553
/src/lib/keys_lib/ @emberian @mrmr1993 @imeckler
5654
/src/lib/ledger_catchup/ @nholland94 @ghost-not-in-the-shell

‎Makefile‎

Lines changed: 3 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,7 @@ GITLONGHASH = $(shell git rev-parse HEAD)
88
MYUID = $(shell id -u)
99
DOCKERNAME = codabuilder-$(MYUID)
1010

11-
# Unique signature of kademlia code tree
12-
KADEMLIA_SIG = $(shell cd src/app/kademlia-haskell ; find . -type f -print0 | xargs -0 sha1sum | sort | sha1sum | cut -f 1 -d ' ')
11+
# Unique signature of libp2p code tree
1312
LIBP2P_HELPER_SIG = $(shell cd src/app/libp2p_helper ; find . -type f -print0 | xargs -0 sha1sum | sort | sha1sum | cut -f 1 -d ' ')
1413

1514
ifeq ($(DUNE_PROFILE),)
@@ -61,15 +60,9 @@ clean:
6160
@rm -rf _build
6261
@rm -rf src/$(COVERAGE_DIR)
6362

64-
kademlia:
65-
@# FIXME: Bash wrap here is awkward but required to get nix-env
66-
bash -c "source ~/.profile && cd src/app/kademlia-haskell && nix-build release2.nix"
67-
6863
libp2p_helper:
6964
bash -c "source ~/.profile && cd src/app/libp2p_helper && nix-build default.nix"
7065

71-
# Alias
72-
dht: kademlia libp2p_helper
7366

7467
GENESIS_DIR := $(TMPDIR)/coda_cache_dir
7568

@@ -105,7 +98,7 @@ update-opam:
10598
macos-portable:
10699
@rm -rf _build/coda-daemon-macos/
107100
@rm -rf _build/coda-daemon-macos.zip
108-
@./scripts/macos-portable.sh src/_build/default/src/app/cli/src/coda.exe src/app/kademlia-haskell/result/bin/kademlia _build/coda-daemon-macos
101+
@./scripts/macos-portable.sh src/_build/default/src/app/cli/src/coda.exe _build/coda-daemon-macos
109102
@zip -r _build/coda-daemon-macos.zip _build/coda-daemon-macos/
110103
@echo Find coda-daemon-macos.zip inside _build/
111104

@@ -177,15 +170,6 @@ docker-toolchain-rust:
177170
echo "Repo has uncommited changes, commit first to set hash." ;\
178171
fi
179172

180-
# All in one step to build toolchain and binary for kademlia
181-
# TODO: Rename to docker-toolchain-discovery
182-
docker-toolchain-haskell:
183-
@echo "Building codaprotocol/coda:toolchain-haskell-$(KADEMLIA_SIG)" ;\
184-
docker build --file dockerfiles/Dockerfile-toolchain-haskell --tag codaprotocol/coda:toolchain-haskell-$(KADEMLIA_SIG) . ;\
185-
echo 'Extracting deb package' ;\
186-
mkdir -p _build ;\
187-
docker run --rm --entrypoint cat codaprotocol/coda:toolchain-haskell-$(KADEMLIA_SIG) /src/coda-discovery.deb > _build/coda-discovery.deb
188-
189173
update-deps:
190174
./scripts/update-toolchain-references.sh $(GITLONGHASH)
191175
make render-circleci
@@ -316,4 +300,4 @@ ml-docs:
316300
# unless there is a reason not to.
317301
# https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html
318302
# HACK: cat Makefile | egrep '^\w.*' | sed 's/:/ /' | awk '{print $1}' | grep -v myprocs | sort | xargs
319-
.PHONY: all base-docker base-googlecloud base-minikube build check-format ci-base-docker clean codaslim containerstart deb dev codabuilder kademlia coda-docker coda-googlecloud coda-minikube ocaml407-googlecloud pull-ocaml407-googlecloud reformat test test-all test-coda-block-production-sig test-coda-block-production-stake test-codapeers-sig test-codapeers-stake test-full-sig test-full-stake test-runtest test-transaction-snark-profiler-sig test-transaction-snark-profiler-stake update-deps render-circleci check-render-circleci docker-toolchain-rust toolchains doc_diagrams ml-docs macos-setup macos-setup-download macos-setup-compile
303+
.PHONY: all base-docker base-googlecloud base-minikube build check-format ci-base-docker clean codaslim containerstart deb dev codabuilder coda-docker coda-googlecloud coda-minikube ocaml407-googlecloud pull-ocaml407-googlecloud reformat test test-all test-coda-block-production-sig test-coda-block-production-stake test-codapeers-sig test-codapeers-stake test-full-sig test-full-stake test-runtest test-transaction-snark-profiler-sig test-transaction-snark-profiler-stake update-deps render-circleci check-render-circleci docker-toolchain-rust toolchains doc_diagrams ml-docs macos-setup macos-setup-download macos-setup-compile libp2p_helper

‎README-dev.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ of the repo.
5959

6060
* Pull down developer container image (~2GB download, go stretch your legs)
6161

62-
`docker pull codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e`
62+
`docker pull codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9`
6363

6464
* Create local builder image
6565

@@ -128,12 +128,12 @@ You should probably use `USEDOCKER=TRUE` unless you've done the [building withou
128128

129129
These are the most important `make` targets:
130130

131-
* `libp2p_helper`: build the libp2p helper
132131
* `build`: build everything
133132
* `docker`: build the container
134133
* `container`: restart the development container (or start it if it's not yet)
135134
* `dev`: does `docker`, `container`, and `build`
136135
* `test`: run the tests
136+
* `libp2p_helper`: build the libp2p helper
137137
* `web`: build the website, including the state explorer
138138

139139
We use the [dune](https://github.com/ocaml/dune/) buildsystem for our OCaml code.

0 commit comments

Comments
 (0)