SparseLinear model weight index access from pointer variable

[sriram7797]

Hi,

Just a doubt that index is correctly accessed in the weights pointer variable?

@registry.layers("SparseLinear.v1")
def SparseLinear(nO: Optional[int] = None, length: int = 2 ** 18):
    # NB: We can't have generic return type annotation if we want function to
    # be bound (and inspectable): https://github.com/cython/cython/issues/2753
    return Model(
        "sparse_linear",
        forward,
        init=init,
        params={"W": None, "b": None},
        dims={"nO": nO, "length": length},
    )

nO = Output dimension, determined by the number of different labels. Let's assume it to be 5 categories. (nO=5)

length = 218 = 262144**

SparseLinear model has two learnable params W(weight) of shape (nO*length) and b(bias) of shape (nO)

W = 5*262114 (Assuming each row of 262114 represents learnable vector for each class)
b = 5

  cdef void set_scoresC(float* scores,
          const uint64_t* keys, const float* values, const int32_t* lengths,
          int batch_size, int nr_out,
          const float* weights, int nr_weight) nogil:
      cdef uint32_t idx1, idx2
      cdef uint32_t hash1, hash2
      for length in lengths[:batch_size]:
          for i in range(length):
              hash1 = MurmurHash3_x86_32_uint64(keys[i], 0)
              hash2 = MurmurHash3_x86_32_uint64(keys[i], 1)
              idx1 = hash1 & (nr_weight-1)
              idx2 = hash2 & (nr_weight-1)
              value = values[i]
              for clas in range(nr_out):
                  scores[clas] += weights[idx1 + clas] * value
                  scores[clas] += weights[idx2 + clas] * value
          scores += nr_out
          keys += length
          values += length

On highlevel, (let's assume output of extract_ngrams will be passed to SparseLinear model)
key = array of hash values concatenated in single array (uigrams, bigrams, trigrams etc..,) generated from output of extract_ngrams function.
values = array of counts of corresponding key present in the doc
lengths = array of length of docs
batch_size = length of the array lengths
nr_out = nO (i.e., 5)
weights = array of size (5*262114)
nr_weight = length (i.e., 262114)

scores = array of length (batch_size,5) (output matrix)

idx1 = hash1 & (nr_weight-1) # returns modulo of the hash by 262114
idx2 = hash2 & (nr_weight-1) # returns modulo of the hash by 262114

idx1 and idx2 will not exceed the value 262114.

for clas in range(nr_out):
	scores[clas] += weights[idx1 + clas] * value
	scores[clas] += weights[idx2 + clas] * value

By conceptually understanding, for each class we need to do operation weights[idx1,clas]*value and weights[idx2,clas]*value (numpy operation in 2d matrix)
Below scores update operation is for 1st document(0th index),

scores[0,0] = weights[idx1,0]*value + weights[idx2,0]*value
scores[0,1] = weights[idx1,1]*value + weights[idx2,1]*value
scores[0,2] = weights[idx1,2]*value + weights[idx2,2]*value
scores[0,3] = weights[idx1,3]*value + weights[idx2,3]*value
scores[0,4] = weights[idx1,4]*value + weights[idx2,4]*value

weight allocation:

model.set_param("W", model.ops.alloc((nO * length,), dtype="f"))

W is pointed to pointer variable weights in set_scoresC fucntion. So value from pointer operation by accessing the index

We should be using multiplication instead of addition in the below statements,

scores[clas] += weights[idx1 + (nr_weight * clas)] * value
scores[clas] += weights[idx2 + (nr_weight * clas)] * value

because idx1 and idx2 will not be greater than 262114 and weight initialized with size (5*262114), but output of (idx1+clas) and (idx1+clas) will not be greater than (262115+5). Weights in indexes greater (262115+5) will be never be used in the process.

Please correct me if my interpretation is not correct.

[danieldk]

Thanks for reporting this! I agree that the indexing is wrong. I have created an issue for this:

explosion/thinc#752