Incorrect Result #2
Description
i followed all your instructions but i encounter with an error in scandata.py (a Problem with unicode characters)
some how i correct the error (i'am not sure really Correct It !) -- i think the problem is with this Code
"articleBuffer.append((id, ctitle))" that i change it to
"articleBuffer.append((id,' \u ' + ctitle))"
But "\u" append to first of each record , i delete it from records with SQL commands --
But i didn't get a suitable result, My rate in wordsim353 is 0.4 but ESA was 0.75.
my correlation SPEARMAN_RANK_CORRELATION algorithm is :
public static double spearmanRankCorrelationCoefficient(Double[] a,
Double[] b) {
check(a, b);
SortedMap<Double,Double> ranking = new TreeMap<Double,Double>();
for (int i = 0; i < a.length; ++i) {
ranking.put(a[i], b[i]);
}
double[] sortedB = new double[b.length];
for (int i = 0; i < b.length; i++) {
sortedB[i] = b[i];
}
Arrays.sort(sortedB);
Map<Double,Integer> otherRanking = new HashMap<Double,Integer>();
for (int i = 0; i < b.length; ++i) {
otherRanking.put(sortedB[i], i);
}
// keep track of the last value we saw in the key set so we can check
// for ties. If there are ties then the Pearson's product-moment
// coefficient should be returned instead.
Double last = null;
// sum of the differences in rank
double diff = 0d;
// the current rank of the element in a that we are looking at
int curRank = 0;
for (Map.Entry<Double,Double> e : ranking.entrySet()) {
Double x = e.getKey();
Double y = e.getValue();
// check that there are no tied rankings
if (last == null)
last = x;
else if (last.equals(x))
// if there was a tie, return the correlation instead.
return correlation(a,b);
else
last = x;
// determine the difference in the ranks for both values
int rankDiff = curRank - otherRanking.get(y).intValue();
diff += rankDiff * rankDiff;
curRank++;
}
return 1 - ((6 * diff) / (a.length * (a.length * a.length - 1)));
}
Please Guide me to get suitable result.
you listed some instruction in the address https://github.com/faraday/wikiprep-esa/wiki/roadmap , i want to know if you implement this instructions in your Code ?
Regards