Experiments

Default evaluation technique

Applies to all experiments:

• stratified 5-fold cross validation over training/test splits;
• mean performance scores over the folds.

Single-layer perceptron optimizers

https://github.com/Wikidata/soweego/issues/285

Setting

• run: May 3 2019;
• output folder: soweego-2.eqiad.wmflabs:/srv/dev/20190503/;
• head commit: d0d390e622f2782a49a1bd0ebfc64478ed34aa0c;
• command: python -m soweego linker evaluate slp ${Dataset} ${Entity} optimizer=${Optimizer}.

Discogs band

Optimizer	Precision	Recall	F-score
sgd	.782	.945	.856
rmsprop	.801	.930	.860
nadam	.805	.925	.861
adamax	.795	.938	.861
adam	.800	.929	.860
adagrad	.802	.927	.859
adadelta	.799	.934	.861

Discogs musician

Optimizer	Precision	Recall	F-score
sgd	.815	.985	.892
rmsprop	.816	.985	.893
nadam	.816	.986	.893
adamax	.817	.985	.893
adam	.816	.985	.893
adagrad	.816	.986	.893
adadelta	.815	.986	.892

Imdb director

Optimizer	Precision	Recall	F-score
sgd	.918	.954	.936
rmsprop	.895	.954	.923
nadam	.908	.954	.930
adamax	.907	.955	.930
adam	.909	.953	.931
adagrad	.867	.950	.907
adadelta	.902	.954	.927

Imdb musician

Optimizer	Precision	Recall	F-score
sgd	.912	.927	.920
rmsprop	.913	.929	.921
nadam	.913	.929	.921
adamax	.913	.928	.921
adam	.913	.928	.921
adagrad	.873	.860	.866
adadelta	.913	.928	.921

Imdb producer

Optimizer	Precision	Recall	F-score
sgd	.917	.942	.929
rmsprop	.916	.938	.927
nadam	.916	.938	.927
adamax	.916	.940	.928
adam	.916	.938	.927
adagrad	.852	.684	.756
adadelta	.916	.939	.928

Imdb writer

Optimizer	Precision	Recall	F-score
sgd	.929	.943	.936
rmsprop	.927	.940	.934
nadam	.930	.940	.935
adamax	.930	.941	.935
adam	.930	.940	.935
adagrad	.872	.923	.896
adadelta	.931	.941	.936

Musicbrainz band

Optimizer	Precision	Recall	F-score
sgd	.952	.869	.909
rmsprop	.949	.875	.911
nadam	.949	.877	.911
adamax	.952	.871	.910
adam	.951	.875	.911
adagrad	.932	.886	.909
adadelta	.952	.874	.911

Musicbrainz musician

Optimizer	Precision	Recall	F-score
sgd	.942	.957	.949
rmsprop	.941	.958	.949
nadam	.941	.958	.949
adamax	.941	.958	.949
adam	.941	.958	.949
adagrad	.946	.953	.950
adadelta	.941	.958	.950

Max Levenshtein VS average Levenshtein

https://github.com/Wikidata/soweego/issues/176

Setting

• run: May 7 2019;
• output folder: soweego-2.eqiad.wmflabs:/srv/dev/20190507/;
• head commit: ddd5d719793ea217267413a52d1d2e5b90c341a7;
• command: python -m soweego linker evaluate ${Algorithm} ${Dataset} ${Entity}.

Discogs band

Algorithm	Precision	Recall	F-score
nb	.787	.955	.863
lsvm	.780	.960	.861
svm	.777	.963	.860
slp	.784	.954	.861
mlp	.822	.925	.870

Discogs musician

Algorithm	Precision	Recall	F-score
nb	.831	.975	.897
lsvm	.818	.985	.894
svm	.815	.985	.892
slp	.821	.983	.895
mlp	.852	.963	.904

Imdb director

Algorithm	Precision	Recall	F-score
nb	.896	.971	.932
lsvm	.919	.943	.931
svm	.911	.950	.930
slp	.917	.953	.935
mlp	.913	.964	.938

Imdb musician

Algorithm	Precision	Recall	F-score
nb	.889	.962	.924
lsvm	.917	.938	.927
svm	.904	.944	.924
slp	.924	.929	.926
mlp	.912	.951	.931

Imdb producer

Algorithm	Precision	Recall	F-score
nb	.870	.971	.918
lsvm	.920	.940	.930
svm	.923	.927	.925
slp	.914	.940	.927
mlp	.911	.956	.933

Imdb writer

Algorithm	Precision	Recall	F-score
nb	.904	.975	.938
lsvm	.936	.949	.943
svm	.932	.954	.943
slp	.938	.946	.942
mlp	.930	.963	.946

Musicbrainz band

Algorithm	Precision	Recall	F-score
nb	.821	.987	.896
lsvm	.944	.879	.910
svm	.930	.891	.910
slp	.953	.865	.907
mlp	.906	.918	.911

Musicbrainz musician

Algorithm	Precision	Recall	F-score
nb	.955	.936	.946
lsvm	.941	.963	.952
svm	.951	.938	.944
slp	.942	.957	.949
mlp	.939	.970	.954