Linkers evaluation

Setting

• run: April 11 2019 on soweego-1 VPS instance;
• output folder: /srv/dev/20190411;
• head commit: 1505429997b878568a9e24185dc3afa7ad4720eb;
• command: python -m soweego linker evaluate ${Algorithm} ${Dataset} ${Entity};
• evaluation technique: stratified 5-fold cross validation over training/test splits;
• mean performance scores over the folds.

Algorithms parameters

• Naïve Bayes:
  • binarize = 0.1;
  • alpha = 0.0001;
• linear SVM: default parameters as per scikit LinearSVC;
• Single-layer perceptron:
  • layer = fully connected (Dense);
  • activation = sigmoid;
  • optimizer = stochastic gradient descent;
  • loss = binary cross-entropy;
  • training batch size = 1,024;
  • training epochs = 100.

Performance

Algorithm	Dataset	Entity	Precision (std)	Recall (std)	F-score (std)
NB	Discogs	Band	.789 (.0031)	.941 (.0004)	.859 (.002)
SVM	Discogs	Band	.785 (.0058)	.946 (.0029)	.858 (.0034)
SLP	Discogs	Band	.776 (.0041)	.956 (.0012)	.857 (.0029)
NB	Discogs	Musician	.836 (.0018)	.958 (.0012)	.893 (.0013)
SVM	Discogs	Musician	.814 (.0015)	.986 (.0003)	.892 (.001)
SLP	Discogs	Musician	.815 (.002)	.985 (.0006)	.892 (.0012)
NB	IMDb	Actor	TODO	TODO	TODO
SVM	IMDb	Actor	TODO	TODO	TODO
SLP	IMDb	Actor	TODO	TODO	TODO
NB	IMDb	Director	.897 (.00195)	.971 (.0012)	.932 (.001)
SVM	IMDb	Director	.919 (.0031)	.942 (.0019)	.93 (.002)
SLP	IMDb	Director	.867 (.0115)	.953 (.0043)	.908 (.0056)
NB	IMDb	Musician	.891 (.0042)	.96 (.0022)	.924 (.0026)
SVM	IMDb	Musician	.917 (.0043)	.937 (.0034)	.927 (.003)
SLP	IMDb	Musician	.922 (.005)	.914 (.0092)	.918 (.0055)
NB	IMDb	Producer	.871 (.0023)	.97 (.0037)	.918 (.0011)
SVM	IMDb	Producer	.92 (.005)	.938 (.0038)	.929 (.0026)
SLP	IMDb	Producer	.862 (.0609)	.914 (.0648)	.883 (.0185)
NB	IMDb	Writer	.91 (.003)	.961 (.0022)	.935 (.0022)
SVM	IMDb	Writer	.936 (.0029)	.948 (.0025)	.942 (.0026)
SLP	IMDb	Writer	.903 (.0154)	.955 (.0147)	.928 (.0047)
NB	MusicBrainz	Band	.822 (.00169)	.985 (.0008)	.896 (.001)
SVM	MusicBrainz	Band	.943 (.0019)	.888 (.0027)	.914 (.0016)
SLP	MusicBrainz	Band	.93 (.0265)	.885 (.0103)	.907 (.0082)
NB	MusicBrainz	Musician	.955 (.0009)	.936 (.0011)	.946 (.00068)
SVM	MusicBrainz	Musician	.941 (.0011)	.962 (.001)	.952 (.0004)
SLP	MusicBrainz	Musician	.943 (.0018)	.956 (.0019)	.949 (.0007)

Confidence

The following plots display the confidence scores distribution and the total predictions yielded by each algorithm on each target classification set.

Axes:

• x = # predictions;
• y = confidence score.

Discogs band

NB, SVM, SLP.

Discogs musician

NB, SVM, SLP.

IMDb director

NB, SVM, SLP.

IMDb musician

NB, SVM, SLP.

IMDb producer

NB, SVM, SLP.

IMDb writer

NB, SVM, SLP.

MusicBrainz band

NB, SVM, SLP.

MusicBrainz musician

NB, SVM, SLP.