Horizon needs to delay starting until Stellar Core is ready #10
Description
We found in our testing that attempting to start both helm charts (core and horizon) at the same time causes errors such as gaps in the ledger.
ime="2019-05-28T13:37:15.695Z" level=info msg="Ingesting ledgers..." first_ledger=556188 last_ledger=556287 pid=1 service=ingest
time="2019-05-28T13:37:15.699Z" level=error msg="Error ingesting ledgers" err="Gap detected in stellar-core database (ledger=556188). More information: https://www.stellar.org/developers/software/known-issues.html#gaps-detected" first_ledger=556188 last_ledger=556287 pid=1 service=ingest
time="2019-05-28T13:37:16.623Z" level=info msg="Ingesting ledgers..." first_ledger=556188 last_ledger=556293 pid=1 service=ingest
time="2019-05-28T13:37:16.625Z" level=error msg="Error ingesting ledgers" err="Gap detected in stellar-core database (ledger=556188). More information: https://www.stellar.org/developers/software/known-issues.html#gaps-detected" first_ledger=556188 last_ledger=556293 pid=1 service=ingest
Delaying the start of horizon until core has passed the readiness probes seems to resolve the issue.
Install script that handles the check for readiness to be complete
nodeSeed=SCXZBWO7UYZ3TLJFLQG54MICBKRIODW7FV673B4AQINU3VLXOLRISHN7
namespace=stellar-testnet
kubectl config set-context $(kubectl config current-context) --namespace=$namespace
stellarCore=stellar-core
# Delete stellar core and reinstall
helm delete $stellarCore --purge
helm install --namespace $namespace --name $stellarCore --set nodeSeed=$nodeSeed --values stellar-core.testnet.values.yaml stellar-core
echo "Waiting for $stellarCore to be running"
## Verify everything is running before taking next step
statusCheck="NOT_STARTED"
while [ "$statusCheck" != "" ] ; do
sleep 60
statusCheck=$(kubectl get pods --namespace $namespace -o json | jq '.items[].status.phase' | grep -v "Running")
echo "Still starting $stellarCore pods $(date)"
done
stellarHorizon=stellar-horizon
helm repo update
helm dependency update stellar-horizon
helm delete $stellarHorizon --purge
helm install \
--namespace $namespace \
--name $stellarHorizon \
--set ingress.enabled=true \
--set ingress.hosts[0].name="stellar-satoshipay.us-south.containers.appdomain.cloud" \
--set ingress.hosts[0].tlsSecret="stellar-satoshipay" \
--set service.port=8000 \
--values stellar-horizon.testnet.values.yaml \
stellar-horizon