FYI: statistics/visualization script

[meowsbits]

Here's a tiny script(s) that crunches the numbers on node record counts over time. Except for spark it's pretty out-of-the-box portable.

> for f in $(find . -type f -name '*stats.sh'); do echo;echo '>>>>>>'$f;echo; cat $f; done

>>>>>>./stats.sh

#!/usr/bin/env bash

# Does basic statistics and visualization for node record counts.
# Statistics are aggregated at the DAILY granularity, traversing as far back as the second argument value.
# 
# Use: ./script.sh <nodes.json> <days>
# 
# Where 'days' is the number of days back from now.

> count.txt

for i in $(seq 1 $2); do
  commit=$(git rev-list --max-count 1 --before="$(date --date '-'"$i"' days' --iso-8601=minutes)" etccore)
  n=0
  if [[ ! -z $commit ]]; then
    git checkout $commit -- $1
    n=$(cat $1 | grep record | wc -l)
  else
    echo skipping "$i (set to 0)"
  fi
  echo "$n" >> count.txt
done

sed '1!G;h;$!d' count.txt | spark

git checkout etccore -- $1

echo latest count:$(cat $1 | grep record | wc -l)

# https://stackoverflow.com/a/9790056
perl -e 'use List::Util qw(max min sum); @a=();while(<>){$sqsum+=$_*$_; push(@a,$_)}; $n=@a;$s=sum(@a);$a=$s/@a;$m=max(@a);$mm=min(@a);$std=sqrt($sqsum/$n-($s/$n)*($s/$n));$mid=int @a/2;@srtd=sort @a;if(@a%2){$med=$srtd[$mid];}else{$med=($srtd[$mid-1]+$srtd[$mid])/2;};print "records:$n\nsum:$s\navg:$a\nstd:$std\nmed:$med\nmax:$m\min:$mm";' < count.txt | grep -v sum

rm count.txt

>>>>>>./allstats.sh

#!/usr/bin/env bash

# Use: ./allstats.sh 100 > STATS.md

for f in $(find . -type f -name '*nodes.json' | sort); do
  echo "# $f"
  echo -e \`\`\`
  ./stats.sh "$f" $1
  echo -e \`\`\`
  echo 
done

./allstats.sh 100 > STATS.md && cat STATS.md

./all.classic.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▄▄▅▇█
latest count:352
records:100
avg:245.72
std:26.7451229198896
med:246
max:372
min:200

./all.kotti.blockd.info/nodes.json

▁▁▁▁▁▃▃▃▃▃▅▅▅▅▅▅▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▆█▆▆
latest count:9
records:100
avg:8.86
std:1.32680066324977
med:7
max:12
min:7

./all.mordor.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▄▄▄▄▄▄▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▆▆▇█
latest count:16
records:100
avg:9.08
std:2.71175220106853
med:4
max:16
min:4

./les.classic.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▅▇▇▅▇█▄▁▁▁▁▁▁▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
latest count:2
records:100
avg:1.66
std:3.03387540943922
med:1
max:15
min:0

./les.kotti.blockd.info/nodes.json

▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅
latest count:1
records:100
avg:1
std:0
med:1
max:1
min:1

./les.mordor.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁████████████████████████████████████████████████████████████████████████████████████
latest count:1
records:100
avg:0.84
std:0.366606055596467
med:1
max:1
min:0