Potential bug in timing embedding

[addtt]

Hi,

There might be a small bug here:

tensor2tensor/tensor2tensor/layers/common_attention.py

Lines 445 to 449 in ef1fcce

	log_timescale_increment = (
	math.log(float(max_timescale) / float(min_timescale)) /
	tf.maximum(tf.to_float(num_timescales) - 1, 1))
	inv_timescales = min_timescale * tf.exp(
	tf.to_float(tf.range(num_timescales)) * -log_timescale_increment)

I think in the last line the exp should be divided by min_timescale rather than multiplied, since it's inverse timescales. Usually min_timescale is 1 so it doesn't matter. But e.g. if you fix max_timescale and change min_timescale, the resulting inverse timescale corresponding to max_timescale changes.

A simpler implementation could be roughly something like this:

inv_timescales = exp(-linspace(log(min_timescale), log(max_timescale), num_timescales))

and from this one you can derive the current implementation, except with division instead of multiplication. It can be even simpler with logspace but tf seems to have this function only as experimental.

Let me know if this makes sense.

Thanks a lot!