Linkers evaluation

Setting

• run: April 11 2019 on soweego-1 VPS instance;
• output folder: /srv/dev/20190411;
• head commit: 1505429997b878568a9e24185dc3afa7ad4720eb;
• command: python -m soweego linker evaluate ${Algorithm} ${Dataset} ${Entity};
• evaluation technique: stratified 5-fold cross validation over training/test splits;
• mean performance scores over the folds.

Algorithms parameters

• Naïve Bayes (NB):
  • binarize = 0.1;
  • alpha = 0.0001;
• liblinear SVM (LSVM): default parameters as per scikit LinearSVC;
• libsvm SVM (SVM):
  • kernel = linear;
  • other parameters as per scikit SVC defaults;
• single-layer perceptron (SLP):
  • layer = fully connected (Dense);
  • activation = sigmoid;
  • optimizer = stochastic gradient descent;
  • loss = binary cross-entropy;
  • training batch size = 1,024;
  • training epochs = 100.

Performance

Algorithm	Dataset	Entity	Precision (std)	Recall (std)	F-score (std)
NB	Discogs	Band	.789 (.0031)	.941 (.0004)	.859 (.002)
LSVM	Discogs	Band	.785 (.0058)	.946 (.0029)	.858 (.0034)
SVM	Discogs	Band	.777 (.003)	.963 (.0016)	.86 (.0024)
SLP	Discogs	Band	.776 (.0041)	.956 (.0012)	.857 (.0029)
NB	Discogs	Musician	.836 (.0018)	.958 (.0012)	.893 (.0013)
SVM	Discogs	Musician	.814 (.0015)	.986 (.0003)	.892 (.001)
SLP	Discogs	Musician	.815 (.002)	.985 (.0006)	.892 (.0012)
NB	IMDb	Actor	TODO	TODO	TODO
SVM	IMDb	Actor	TODO	TODO	TODO
SLP	IMDb	Actor	TODO	TODO	TODO
NB	IMDb	Director	.897 (.00195)	.971 (.0012)	.932 (.001)
SVM	IMDb	Director	.919 (.0031)	.942 (.0019)	.93 (.002)
SLP	IMDb	Director	.867 (.0115)	.953 (.0043)	.908 (.0056)
NB	IMDb	Musician	.891 (.0042)	.96 (.0022)	.924 (.0026)
SVM	IMDb	Musician	.917 (.0043)	.937 (.0034)	.927 (.003)
SLP	IMDb	Musician	.922 (.005)	.914 (.0092)	.918 (.0055)
NB	IMDb	Producer	.871 (.0023)	.97 (.0037)	.918 (.0011)
SVM	IMDb	Producer	.92 (.005)	.938 (.0038)	.929 (.0026)
SLP	IMDb	Producer	.862 (.0609)	.914 (.0648)	.883 (.0185)
NB	IMDb	Writer	.91 (.003)	.961 (.0022)	.935 (.0022)
SVM	IMDb	Writer	.936 (.0029)	.948 (.0025)	.942 (.0026)
SLP	IMDb	Writer	.903 (.0154)	.955 (.0147)	.928 (.0047)
NB	MusicBrainz	Band	.822 (.00169)	.985 (.0008)	.896 (.001)
SVM	MusicBrainz	Band	.943 (.0019)	.888 (.0027)	.914 (.0016)
SLP	MusicBrainz	Band	.93 (.0265)	.885 (.0103)	.907 (.0082)
NB	MusicBrainz	Musician	.955 (.0009)	.936 (.0011)	.946 (.00068)
SVM	MusicBrainz	Musician	.941 (.0011)	.962 (.001)	.952 (.0004)
SLP	MusicBrainz	Musician	.943 (.0018)	.956 (.0019)	.949 (.0007)

Confidence

The following plots display the confidence scores distribution and the total predictions yielded by each algorithm on each target classification set.

Note that linear SVM is omitted since it does not output probability scores.

Axes:

• x = # predictions;
• y = confidence score.

Discogs band

NB, SVM, SLP.

Discogs musician

NB, SVM, SLP.

IMDb director

NB, SVM, SLP.

IMDb musician

NB, SVM, SLP.

IMDb producer

NB, SVM, SLP.

IMDb writer

NB, SVM, SLP.

MusicBrainz band

NB, SVM, SLP.

MusicBrainz musician

NB, SVM, SLP.

Comparison

See the plots above to have a rough idea on the amount of confident predictions.

Threshold values:

• # predictions >= 0.0000000001, i.e., equivalent to almost all matches;
• # confident >= 0.8.

Discogs band

WD items: 50,316

Measure	NB	LSVM	SVM	SLP
Precision	.789	.785	.777	.776
Recall	.941	.946	.963	.957
F-score	.859	.858	.86	.857
# predictions	820	51	94,430	91,295
# confident	219	N.A.	1,660	5,355

Discogs musician

WD items: 199,180

Measure	NB	LSVM	SVM	SLP
Precision	.836	.814	.815	.815
Recall	.958	.986	.985	.985
F-score	.893	.892	.892	.892
# predictions	3,872	200	533,301	517,450
# confident	1,101	N.A.	98,172	58,437

IMDb director

WD items: 9,249

Measure	NB	LSVM	SVM	SLP
Precision	.897	.919	.908	.867
Recall	.971	.942	.958	.953
F-score	.932	.93	.932	.908
# predictions	192	10	17,557	17,187
# confident	60	N.A.	1,616	553

IMDb musician

WD items: 217,139

Measure	NB	LSVM	SVM	SLP
Precision	.891	.917	.908	.922
Recall	.96	.937	.942	.914
F-score	.924	.927	.924	.918
# predictions	4,806	218	406,674	398,346
# confident	1,341	N.A.	21,462	7,244

IMDb producer

WD items: 2,251

Measure	NB	LSVM	SVM	SLP
Precision	.871	.92	.923	.862
Recall	.97	.938	.926	.914
F-score	.918	.929	.925	.883
# predictions	56	3	5,249	5,116
# confident	15	N.A.	507	180

IMDb writer

WD items: 16,446

Measure	NB	LSVM	SVM	SLP
Precision	.91	.936	.932	.903
Recall	.961	.948	.954	.955
F-score	.935	.942	.943	.928
# predictions	428	17	45,122	44,338
# confident	138	N.A.	2,934	1,548

MusicBrainz band

WD items: 32,658

Measure	NB	LSVM	SVM	SLP
Precision	.822	.943	.939	.93
Recall	.985	.888	.893	.885
F-score	.896	.914	.915	.907
# predictions	265	33	39,618	38,012
# confident	46	N.A.	1,475	501

MusicBrainz musician

WD items: 153,725

Measure	NB	LSVM	SVM	SLP
Precision	.955	.941	.95	.943
Recall	.936	.962	.938	.956
F-score	.946	.952	.944	.949
# predictions	2,833	154	280,029	260,530
# confident	1,212	N.A.	7,496	7,339