Skip to content

Commit 265530b

Browse files
pinheadmzpinheadmz
authored
Merge pull request #787 from Jhoyola/node-persistence
Node data persistence in persistent volume claims
2 parents b6f0cfb + 44a102e commit 265530b

File tree

20 files changed

+382
-35
lines changed

20 files changed

+382
-35
lines changed

‎.github/workflows/test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ jobs:
4242
- graph_test.py
4343
- logging_test.py
4444
- ln_basic_test.py
45-
- ln_test.py
45+
- ln_graph_test.py
4646
- onion_test.py
4747
- plugin_test.py
4848
- rpc_test.py

‎docs/persistence.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Data Persistence
2+
3+
By default, Warnet nodes use ephemeral storage, meaning all data is lost when a pod is deleted or restarted. This document describes how to enable persistent storage to be able to use warnet for persistent development environment, such that blockchain data, wallet information, and other node state can survive pod restarts and network redeployments. This is done with Kubernetes Persistent Volume Claims (PVCs), which persist independently of pod lifecycle.
4+
5+
Persistence is available for:
6+
- **Bitcoin Core** nodes
7+
- **LND** nodes
8+
- **CLN** nodes
9+
10+
## Enabling Persistence
11+
12+
Persistence is configured per-node in the network graph definition. Add a `persistence` section to any node's configuration. This creates a new PVC for that node, which is then mounted to the appropriate data directory inside the container.
13+
14+
Also add `restartPolicy: Always` to the node's configuration to ensure that the pod is restarted if it is deleted or crashes. This is important to ensure proper restart after restart of the kubernetes cluster. If there is a risk of hard resets of the cluster, add `reindex=1` to bitcoin_core config to reindex the blockchain on startup and fix potential corrupted chain state.
15+
16+
### Bitcoin Core Node
17+
18+
```yaml
19+
bitcoin_core:
20+
image: bitcoincore-27.1:latest
21+
restartPolicy: Always
22+
persistence:
23+
enabled: true
24+
size: 20Gi # optional, default is 20Gi
25+
storageClass: "" # optional, default is cluster default storage class
26+
accessMode: ReadWriteOncePod # optional, default is ReadWriteOncePod. For compatibility with older Kubernetes versions, you may need to set this to ReadWriteOnce
27+
config: |
28+
reindex=1
29+
```
30+
31+
### Lightning Node
32+
33+
```yaml
34+
<lnd or cln>:
35+
image:
36+
tag: <node-version-tag>
37+
restartPolicy: Always
38+
persistence:
39+
enabled: true
40+
size: 10Gi # optional, default is 10Gi
41+
storageClass: "" # optional, default is cluster default storage class
42+
accessMode: ReadWriteOncePod # optional, default is ReadWriteOncePod. For compatibility with older Kubernetes versions, you may need to set this to ReadWriteOnce
43+
```
44+
45+
## Existing PVCs
46+
47+
To use custom made PVC or PVC from previous deployment, use the `existingClaim` field to reference an existing PVC by name. If the network configuration or namespace did not change, there is no need to explicitly set the `existingClaim`. The existing PVC is used by default, since its generated name matches the default pattern. To explicitly use a PVC set the name like this:
48+
49+
```yaml
50+
persistence:
51+
enabled: true
52+
existingClaim: "tank-0001.default-bitcoincore-data"
53+
```
54+
55+
The generated PVC names follow the pattern:
56+
`<pod-name>.<namespace>-<node-type>-data`
57+
58+
For example for a bitcoin core node:
59+
`tank-0001.default-bitcoincore-data`
60+
61+
And for a LND node:
62+
`tank-0001-ln.default-lnd-data`
63+
64+
Get the list of PVCs in the cluster with `kubectl get pvc -A` and delete any PVCs that are no longer needed with `kubectl delete pvc <pvc-name> -n <namespace>`.
65+
66+
## Mount Paths
67+
68+
When persistence is enabled, the following directories are persisted in the PVCs:
69+
70+
- **Bitcoin Core:** `/root/.bitcoin/`
71+
- **LND:** `/root/.lnd/`
72+
- **CLN:** `/root/.lightning/`

‎resources/charts/bitcoincore/charts/cln/templates/pod.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ spec:
5656
- mountPath: /root/.lightning/config
5757
name: config
5858
subPath: config
59+
- mountPath: /root/.lightning
60+
name: cln-data
5961
{{- with .Values.extraContainers }}
6062
{{- toYaml . | nindent 4 }}
6163
{{- end }}
@@ -80,6 +82,18 @@ spec:
8082
- configMap:
8183
name: {{ include "cln.fullname" . }}
8284
name: config
85+
- name: cln-data
86+
{{- if .Values.persistence.enabled }}
87+
{{- if .Values.persistence.existingClaim }}
88+
persistentVolumeClaim:
89+
claimName: {{ .Values.persistence.existingClaim }}
90+
{{- else }}
91+
persistentVolumeClaim:
92+
claimName: {{ include "cln.fullname" . }}.{{ .Release.Namespace }}-cln-data
93+
{{- end }}
94+
{{- else }}
95+
emptyDir: {}
96+
{{- end }}
8397
{{- with .Values.nodeSelector }}
8498
nodeSelector:
8599
{{- toYaml . | nindent 4 }}

‎resources/charts/bitcoincore/charts/cln/templates/pvc.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
2+
apiVersion: v1
3+
kind: PersistentVolumeClaim
4+
metadata:
5+
name: {{ include "cln.fullname" . }}.{{ .Release.Namespace }}-cln-data
6+
labels:
7+
{{- include "cln.labels" . | nindent 4 }}
8+
annotations:
9+
"helm.sh/resource-policy": keep
10+
spec:
11+
accessModes:
12+
- {{ .Values.persistence.accessMode }}
13+
resources:
14+
requests:
15+
storage: {{ .Values.persistence.size }}
16+
{{- if .Values.persistence.storageClass }}
17+
storageClassName: {{ .Values.persistence.storageClass }}
18+
{{- end }}
19+
{{- end }}

‎resources/charts/bitcoincore/charts/cln/values.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,15 @@ startupProbe:
8282
- "-c"
8383
- "lightning-cli createrune > /working/rune.json"
8484

85+
# Node data persistence configuration. Create persistent volume claim, or use an existing one.
86+
persistence:
87+
enabled: false
88+
storageClass: ""
89+
accessMode: ReadWriteOncePod
90+
size: 10Gi
91+
# Use existing persistent volume claim instead of creating a new one.
92+
existingClaim: ""
93+
8594
# Additional volumes on the output Deployment definition.
8695
volumes:
8796
- name: working

‎resources/charts/bitcoincore/charts/lnd/templates/pod.yaml‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,15 @@ spec:
5353
- /bin/sh
5454
- -c
5555
- |
56-
rm -rf /root/.lnd/data/chain
56+
MACAROON_PATH="/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
57+
58+
# If wallet is already initialized, unlock it and exit
59+
if [ -f "$MACAROON_PATH" ]; then
60+
until [ "$(curl --silent --insecure https://localhost:8080/v1/unlockwallet --data "{\"wallet_password\":\"AAAAAAAAAAA=\"}")" == "{}" ]; do
61+
sleep 5
62+
done
63+
exit 0
64+
fi
5765
5866
until curl --silent --insecure https://localhost:8080/v1/genseed > /tmp/genseed.json; do
5967
sleep 5
@@ -79,7 +87,7 @@ spec:
7987
- mountPath: /root/.lnd/tls.cert
8088
name: config
8189
subPath: tls.cert
82-
- name: shared-volume
90+
- name: lnd-data
8391
mountPath: /root/.lnd/
8492
{{- with .Values.extraContainers }}
8593
{{- toYaml . | nindent 4 }}
@@ -95,7 +103,7 @@ spec:
95103
- "--macaroonpath=/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
96104
- "--httplisten=0.0.0.0:{{ .Values.circuitbreaker.httpPort }}"
97105
volumeMounts:
98-
- name: shared-volume
106+
- name: lnd-data
99107
mountPath: /root/.lnd/
100108
- name: config
101109
mountPath: /tls.cert
@@ -108,8 +116,18 @@ spec:
108116
- configMap:
109117
name: {{ include "lnd.fullname" . }}
110118
name: config
111-
- name: shared-volume
119+
- name: lnd-data
120+
{{- if .Values.persistence.enabled }}
121+
{{- if .Values.persistence.existingClaim }}
122+
persistentVolumeClaim:
123+
claimName: {{ .Values.persistence.existingClaim }}
124+
{{- else }}
125+
persistentVolumeClaim:
126+
claimName: {{ include "lnd.fullname" . }}.{{ .Release.Namespace }}-lnd-data
127+
{{- end }}
128+
{{- else }}
112129
emptyDir: {}
130+
{{- end }}
113131
{{- with .Values.nodeSelector }}
114132
nodeSelector:
115133
{{- toYaml . | nindent 4 }}

‎resources/charts/bitcoincore/charts/lnd/templates/pvc.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
2+
apiVersion: v1
3+
kind: PersistentVolumeClaim
4+
metadata:
5+
name: {{ include "lnd.fullname" . }}.{{ .Release.Namespace }}-lnd-data
6+
labels:
7+
{{- include "lnd.labels" . | nindent 4 }}
8+
annotations:
9+
"helm.sh/resource-policy": keep
10+
spec:
11+
accessModes:
12+
- {{ .Values.persistence.accessMode }}
13+
resources:
14+
requests:
15+
storage: {{ .Values.persistence.size }}
16+
{{- if .Values.persistence.storageClass }}
17+
storageClassName: {{ .Values.persistence.storageClass }}
18+
{{- end }}
19+
{{- end }}

‎resources/charts/bitcoincore/charts/lnd/values.yaml‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,13 @@ image:
99
repository: lightninglabs/lnd
1010
pullPolicy: IfNotPresent
1111
# Overrides the image tag whose default is the chart appVersion.
12-
tag: "v0.19.0-beta"
12+
tag: "v0.20.1-beta"
1313

1414
imagePullSecrets: []
1515
nameOverride: ""
1616
fullnameOverride: ""
1717

18-
podLabels:
18+
podLabels:
1919
app: "warnet"
2020
mission: "lightning"
2121

@@ -65,12 +65,11 @@ resources: {}
6565
# cpu: 100m
6666
# memory: 128Mi
6767

68-
6968
livenessProbe:
7069
exec:
7170
command:
72-
- pidof
73-
- lnd
71+
- pidof
72+
- lnd
7473
failureThreshold: 3
7574
initialDelaySeconds: 60
7675
periodSeconds: 5
@@ -85,6 +84,15 @@ readinessProbe:
8584
port: 10009
8685
timeoutSeconds: 1
8786

87+
# Node data persistence configuration. Create persistent volume claim, or use an existing one.
88+
persistence:
89+
enabled: false
90+
storageClass: ""
91+
accessMode: ReadWriteOncePod
92+
size: 10Gi
93+
# Use existing persistent volume claim instead of creating a new one.
94+
existingClaim: ""
95+
8896
# Additional volumes on the output Deployment definition.
8997
volumes: []
9098
# - name: foo
@@ -128,6 +136,6 @@ defaultConfig: ""
128136
channels: []
129137

130138
circuitbreaker:
131-
enabled: false # Default to disabled
139+
enabled: false # Default to disabled
132140
image: carlakirkcohen/circuitbreaker:attackathon-test
133-
httpPort: 9235
141+
httpPort: 9235

‎resources/charts/bitcoincore/templates/pod.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,17 @@ spec:
109109
{{- toYaml . | nindent 4 }}
110110
{{- end }}
111111
- name: data
112+
{{- if .Values.persistence.enabled }}
113+
{{- if .Values.persistence.existingClaim }}
114+
persistentVolumeClaim:
115+
claimName: {{ .Values.persistence.existingClaim }}
116+
{{- else }}
117+
persistentVolumeClaim:
118+
claimName: {{ include "bitcoincore.fullname" . }}.{{ .Release.Namespace }}-bitcoincore-data
119+
{{- end }}
120+
{{- else }}
112121
emptyDir: {}
122+
{{- end }}
113123
- name: config
114124
configMap:
115125
name: {{ include "bitcoincore.fullname" . }}

‎resources/charts/bitcoincore/templates/pvc.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
2+
apiVersion: v1
3+
kind: PersistentVolumeClaim
4+
metadata:
5+
name: {{ include "bitcoincore.fullname" . }}.{{ .Release.Namespace }}-bitcoincore-data
6+
labels:
7+
{{- include "bitcoincore.labels" . | nindent 4 }}
8+
annotations:
9+
"helm.sh/resource-policy": keep
10+
spec:
11+
accessModes:
12+
- {{ .Values.persistence.accessMode }}
13+
resources:
14+
requests:
15+
storage: {{ .Values.persistence.size }}
16+
{{- if .Values.persistence.storageClass }}
17+
storageClassName: {{ .Values.persistence.storageClass }}
18+
{{- end }}
19+
{{- end }}

0 commit comments

Comments
 (0)