Skip to content

Commit 3e1133c

Browse files
fix(docs): correct sharding asset paths (#573)
1 parent 1e2f1e0 commit 3e1133c

File tree

3 files changed

+29
-30
lines changed

3 files changed

+29
-30
lines changed

docs/topics/custom-domain/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Traditional replication methods rely on fixed partitions or discrete addressing.
1717

1818
For example, subscribing to a live feed might involve replicating only the latest segment (e.g., [0.999, 1]). As the stream evolves, the replication range shifts dynamically. Or when you are buffering a video, then you will replicate the segments that are being buffered and continously expand the replication range as the video progresses. See the video below
1919

20-
<video src="/topics/custom-domain/buffer.mp4" controls muted style="width: 100%" ></video>
20+
<video src="/content/docs/topics/custom-domain/buffer.mp4" controls muted style="width: 100%" ></video>
2121

2222
The timeline will turn more blue and become taller when more replicators/viewers are watching the video.
2323

docs/topics/sharding/addressing.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,24 +6,24 @@ Sharding in Peerbit is based on the content being committed. Each commit will be
66
Every change in Peerbit is committed with an explicit link to the content it depends on. This means that by following the dependencies to the root, we can get the full state.
77

88
<p align="center">
9-
<img width="400" src="./topics/sharding/p0b.png" alt="p1">
9+
<img width="400" src="./ca0b.png" alt="p1">
1010
</p>
1111

1212

1313
Every graph gets an graph id (GID).
1414
<p align="center">
15-
<img width="400" src="./topics/sharding/p7.png" alt="p1">
15+
<img width="400" src="./p7.png" alt="p1">
1616
</p>
1717

1818

1919
When graphs merge (two independent states becomes dependent), the new graph will be named the same as the graph with the longest chain.
2020

2121
<p align="center">
22-
<img width="400" src="./topics/sharding/p8.png" alt="p1">
22+
<img width="400" src="./p8.png" alt="p1">
2323
</p>
2424

2525
<p align="center">
26-
<img width="400" src="./topics/sharding/p9.png" alt="p1">
26+
<img width="400" src="./p9.png" alt="p1">
2727
</p>
2828

2929

@@ -33,46 +33,46 @@ This is important background in order to understand how replicators/content lead
3333
Imagine the commit above is made, so that the merged graph gets the label "DOG", how can we choose replicators in a fully connected network in a simple random way? (By being a replicator you have the task of storing the log and potentially also make it searchable for peers)
3434

3535
<p align="center">
36-
<img width="400" src="./topics/sharding/p1.png" alt="p1">
36+
<img width="400" src="./p1.png" alt="p1">
3737
</p>
3838

3939

4040
### 1.
4141
The first thing we need to do is to hash the labels of the peers (PeerIds) and the DOG label with a hash function (more details on this function later).
4242

4343
<p align="center">
44-
<img width="400" src="./topics/sharding/p2.png" alt="p2">
44+
<img width="400" src="./p2.png" alt="p2">
4545
</p>
4646

4747

4848
### 2.
4949
Secondly put all the hashes into a list and sort it.
5050

5151
<p align="center">
52-
<img width="400" src="./topics/sharding/p3.png" alt="p3">
52+
<img width="400" src="./p3.png" alt="p3">
5353
</p>
5454

5555

5656
### 3.
5757
Now we look up the labels from the hashes again.
5858

5959
<p align="center">
60-
<img width="400" src="./topics/sharding/p4.png" alt="p4">
60+
<img width="400" src="./p4.png" alt="p4">
6161
</p>
6262

6363

6464
### 4.
6565
Now if we want 2 replicas of our content, we can choose that the replicators are the 2 next elements in the list.
6666

6767
<p align="center">
68-
<img width="400" src="./topics/sharding/p5.png" alt="p5">
68+
<img width="400" src="./p5.png" alt="p5">
6969
</p>
7070

7171

7272
The hash function is seeded with the checksum of the content itself, so it changes for every new commit. This means that the results would differ if the content changes. E.g.
7373

7474
<p align="center">
75-
<img width="400" src="./topics/sharding/p6.png" alt="p5">
75+
<img width="400" src="./p6.png" alt="p5">
7676
</p>
7777

7878

docs/topics/sharding/sharding.md

Lines changed: 18 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ const db = peer.open("address", {
3434
For (A), we already have many solutions that work well but generally do not consider (B) and (C). For instance, in common DHT systems, we can use the identities of the participants to distribute content and pick neighbors to satisfy the minimum replication degree constraint.
3535

3636
<p align="center">
37-
<img width="800" src="./topics/sharding/p1.png" alt="p1">
37+
<img width="800" src="./p1.png" alt="p1">
3838
</p>
3939

4040

@@ -52,24 +52,24 @@ For simplicity, we consider that every peer can only have one range. And that ra
5252
A piece of data that needs to be stored will be stored at a location that depends on its hash. But instead of using the hash, we transform it into a number bounded by [0,1].
5353

5454
<p align="center">
55-
<img width="800" src="./topics/sharding/p2.png" alt="p2">
55+
<img width="800" src="./p2.png" alt="p2">
5656
</p>
5757

5858
If the vertical line intersects with a range, that peer will be responsible for replicating this data. A nice consequence of this is that peers can participate with different degrees of trust in how much work others will perform.
5959
<p align="center">
60-
<img width="800" src="./topics/sharding/p4.png" alt="p4">
60+
<img width="800" src="./p4.png" alt="p4">
6161
</p>
6262
By replicating with a factor (width) of 1, every data point will intersect the range, hence the node will always be responsible for every data point. This means that if anyone in a network creates data, it will always be sent to this peer. This is also useful property if you want to create an app where every peer always should hold the complete state locally at all times.
6363

6464
Another nice consequence of this is that if you only want to "pin" a specific data point, you only need to make your width as small as the floating points allow, to only cover that particular data point. *A line is a special case of a curve* and *pinning is a special case of range replication* (a range with width that approaches 0).
6565

6666
<p align="center">
67-
<img width="800" src="./topics/sharding/p5.png" alt="p5">
67+
<img width="800" src="./p5.png" alt="p5">
6868
</p>
6969

7070
If there is a gap, then the closest node will be chosen in the following way:
7171
<p align="center">
72-
<img width="800" src="./topics/sharding/p6.png" alt="p3">
72+
<img width="800" src="./p6.png" alt="p3">
7373
</p>
7474

7575
This means that even if the longer range is further away by measuring from the closest edge, it still needs to replicate the data due to that the transformed distance gets shorter because of the wider range. This property is important, because we wan't to make sure that someone who replicates with width 0 does not get delegated any replication work.
@@ -86,11 +86,11 @@ and to find all points, do the calculation for
8686

8787

8888
<p align="center">
89-
<img width="800" src="./topics/sharding/p7.png" alt="p7">
89+
<img width="800" src="./p7.png" alt="p7">
9090
</p>
9191

9292
<p align="center">
93-
<img width="800" src="./topics/sharding/p8.png" alt="p8">
93+
<img width="800" src="./p8.png" alt="p8">
9494
</p>
9595

9696
If you think of the content space as a circle, this would represent a rotation of `360° / min replicas`. So if `min replicas = 2` and the start point is the north pole, the second point would be the south pole.
@@ -100,28 +100,28 @@ But we will stick with the line representation because it will be easier to visu
100100
### (B) Resource Awareness
101101

102102
<p align="center">
103-
<img width="400" src="./topics/sharding/p10.png" alt="p10">
103+
<img width="400" src="./p10.png" alt="p10">
104104
</p>
105105

106106
With this in (A) place, now it is time to consider constraint (B). The innovative step here is that we adjust our width to satisfy any resource constraint. Is the memory or CPU usage too high? Just reduce the width of your responsibility until satisfied. Do you have capacity? Then perhaps it would be helpful for others if you increase your width of responsibility.
107107

108108
<p align="center">
109-
<img width="800" src="./topics/sharding/p9.png" alt="p9">
109+
<img width="800" src="./p9.png" alt="p9">
110110
</p>
111111

112112

113113
But this problem is actually more nuanced than just memory and CPU, for a healthy replication we also need to consider a few other goals.
114114

115115
<p align="center">
116-
<img width="800" src="./topics/sharding/p13.png" alt="p13">
116+
<img width="800" src="./p13.png" alt="p13">
117117
</p>
118118

119119

120120
We cannot feasibly predict the optimal width for every participant in one go because we cannot continuously share all node info (CPU, memory, and other) usage to every other node at all times. Additionally, while data is inserted, storage-limited nodes will take up less width over time, so this is a continuous process. Therefore, the solution is to work iteratively where everyone adjusts their widths in small steps, and eventually the system converges to an optimal point.
121121

122122
For clarity these iterations on what happens when you update your width over time:
123123
<p align="center">
124-
<img width="800" src="./topics/sharding/p11.png" alt="p11">
124+
<img width="800" src="./p11.png" alt="p11">
125125
</p>
126126

127127

@@ -132,7 +132,7 @@ In practice, a special form of it that comes in the form of something that is ca
132132
A simplified mathematical representation of the iterator looks like this:
133133

134134
<p align="center">
135-
<img width="800" src="./topics/sharding/p14.png" alt="p14">
135+
<img width="800" src="./p14.png" alt="p14">
136136
</p>
137137

138138

@@ -141,7 +141,7 @@ Simplified, we use [Lagrange relaxation](https://en.wikipedia.org/wiki/Lagrangia
141141
When everything functions correctly, the width will converge to a specific number for each peer over time.
142142

143143
<p align="center">
144-
<img width="800" src="./topics/sharding/p15.png" alt="p15">
144+
<img width="800" src="./p15.png" alt="p15">
145145
</p>
146146

147147
For better understanding, consider this analogy: it's as if we're trying to regulate the temperature in three houses simultaneously, where the thermostat in one house is influenced by the others. But the twist is that if one house requires less heating, another might need to compensate by heating more.
@@ -160,17 +160,17 @@ Below are illustrations of how aggregation is executed:
160160

161161
Start "local first":
162162
<p align="center">
163-
<img width="500" src="./topics/sharding/p16.png" alt="p16">
163+
<img width="500" src="./p16.png" alt="p16">
164164
</p>
165165

166166
Determine the length of your "walk":
167167
<p align="center">
168-
<img width="500" src="./topics/sharding/p17.png" alt="p17">
168+
<img width="500" src="./p17.png" alt="p17">
169169
</p>
170170

171171
Aggregate every range and note its owner, but avoid considering more than one range per "step":
172172
<p align="center">
173-
<img width="500" src="./topics/sharding/p18.png" alt="p18">
173+
<img width="500" src="./p18.png" alt="p18">
174174
</p>
175175

176176
The source code for the aggregation is accessible [here](https://github.com/dao-xyz/peerbit/blob/95420cd37cb8d2ced4733495b6901b2b5e445e01/packages/programs/data/shared-log/src/ranges.ts#L155).
@@ -182,13 +182,13 @@ Initially, we observe peers receiving segments within the content space, with st
182182

183183

184184

185-
<video src="/topics/sharding/storage-toggle.mp4" controls muted style="width: 100%" ></video>
185+
<video src="/content/docs/topics/sharding/storage-toggle.mp4" controls muted style="width: 100%" ></video>
186186

187187
When memory limitation is enabled, the ranges are observed to update accordingly to what we set the limit to. Also note on the top left "Used storage" and how that changes with the limit set.
188188

189189
Upon enabling CPU limitation, it's noticeable that minimizing a client's tab halts data replication. This occurs because a minimized tab is typically subject to significant throttling, thereby constraining processing capacity. However, once the tab is reopened, operations resume to their optimal state.
190190

191-
<video src="/topics/sharding/cpu-toggle.mp4" controls muted style="width: 100%"></video>
191+
<video src="/content/docs/topics/sharding/cpu-toggle.mp4" controls muted style="width: 100%"></video>
192192

193193
Explore this yourself and review the source code [here](https://github.com/dao-xyz/peerbit-examples/tree/master/packages/file-share).
194194

@@ -216,4 +216,3 @@ The parameters for the PID regulator might need to be adaptive, depending on net
216216
### Numerical Optimizers
217217

218218
As previously described, the resource optimization problem was solved with a PID controller, under the assumption that the problem has desirable "convex" properties. While this assumption may hold in many cases, there might be scenarios where more robust (and more resource-intensive) solvers would be preferable. For instance, when non-numerical properties and non-linear features are involved, a [Recurrent Neural Network (RNN)](https://en.wikipedia.org/wiki/Recurrent_neural_network) might perform better.
219-

0 commit comments

Comments
 (0)