Why is the kmeans algorithm column-oriented instead of row-oriented?

[paulhendricks]

In the docs (below), the kmeans algorithm takes a matrix where each column X[:, i] corresponds to an observed sample. This implementation goes against the idea of tidy data as well as differs from Python's scikit-learn implementation of kmeans and R's base implementation of kmeans.

Is there a good reason for this? Should this algorithm be changed from column-oriented to row-oriented so as to be consistent with R and Python as well as with the concept of tidy data?

URL: http://clusteringjl.readthedocs.io/en/stable/overview.html

Inputs

A clustering algorithm, depending on its nature, may accept an input matrix in either of the following forms:

• Sample matrix X, where each column X[:,i] corresponds to an observed sample.
• Distance matrix D, where D[i,j] indicates the distance between samples i and j, or the cost of assigning one to the other.

[ararslan]

There were discussions about this a while back (though I'm struggling to locate one now), and ultimately it came down to the original package author's personal preference. Apparently observations-as-columns is the norm in machine learning, at least according to some, and observations-as-rows is the norm in classical statistics. Personally I'm in the camp of observations being rows and have been bitten by this package's use of columns.

The problem with changing it at this point is that existing code will suddenly become very wrong, and it's not something we'd be able to catch and emit a deprecation warning for. We'd have to redesign the API and deprecate the existing API in favor of the new one.

[nalimilan]

Unless we can find another package where clustering works that way, I'd favor moving to rows as observations. A possible deprecation path would be to add a vardim argument, and require passing e.g. vardim=1 (with a default of 2) to get the current behavior without a warning. Then we would switch the default in the next release, and possibly deprecate the option later.

EDIT: see also JuliaStats/Distances.jl#35

[ararslan]

@nalimilan That sounds perfect. Simple and largely non-disruptive. Nice idea! 💯

[paulhendricks]

Sounds like a plan. I can take a first stab at implementing this and submit a pull request for your review.

[ararslan]

That sounds great, @paulhendricks. Thanks!

[paulhendricks]

Awesome. As far as I can tell, kmeans is the only clustering algorithm that needs changed to close this issue.

Below is a first pass at implementing @nalimilan 's suggestion:

const _kmeans_default_init = :kmpp
const _kmeans_default_maxiter = 100
const _kmeans_default_tol = 1.0e-6
const _kmeans_default_display = :none
const _vardim = 2

function kmeans!{T<:AbstractFloat}(X::Matrix{T}, centers::Matrix{T};
                                   weights=nothing,
                                   maxiter::Integer=_kmeans_default_maxiter,
                                   tol::Real=_kmeans_default_tol,
                                   display::Symbol=_kmeans_default_display,
                                   vardim::Integer=_vardim)

    # Here, we use vardim to assess if matrix X is
    # column-oriented (vardim == 1) or row-oriented (vardim == 2).
    # By default, X should be row-oriented. Use vardim=1
    # to recover the old behavior where X is column-oriented.
    #
    # See this issue for more details:
    # https://github.com/JuliaStats/Clustering.jl/issues/79
    info("vardim defaults to 2 in version x.x.x and will default to 1 in x.x.x.")
    info("vardim will then be deprecated in version x.x.x.")
    if vardim == 2:
      # If matrix X is row-oriented (vardim == 2), transpose
      # it to be column-oriented. When vardim is deprecated,
      # this transposition will be the first operation.
      X = transpose(X)

    # Once X is column-oriented, the rest of algorithm
    # can run as normal with no need for modification.
    m, n = size(X)
    m2, k = size(centers)
    m == m2 || throw(DimensionMismatch("Inconsistent array dimensions."))
    (2 <= k < n) || error("k must have 2 <= k < n.")

    assignments = zeros(Int, n)
    costs = zeros(T, n)
    counts = Array(Int, k)
    cweights = Array(Float64, k)

Does this look good 👍 ?

[nalimilan]

Use Base.depwarn to print warnings; and no need to say in what version it will be removed (it's always in the next major version). Also, you need to check that vardim is equal either to 1 or to 2.

[kmsquire]

Just as a point of reference, one of the original arguments for having observations as columns was that Julia arrays are in column-major order, so this allows data for individual observations to be stored contiguously.

See, e.g., http://julialang.org/blog/2013/09/fast-numeric, especially under "Write cache-friendly codes".

[nalimilan]

Right, that's probably how Dahua chose the arrangement to use. Benchmarks are presented in Distances.jl README: https://github.com/JuliaStats/Distances.jl#benchmarks

Though I wonder whether it really makes a difference since the text says the computations are done by calling BLAS's gemm. Since the array is contiguous anyway, BLAS should be able to choose the best way to perform the computation efficiently. I can't believe SciKit would use a deliberately slow convention.

Somebody would have to run the benchmarks using the rowwise convention to check this.

[kmsquire]

I can't believe SciKit would use a deliberately slow convention.

SciKit Learn has a great interface, but in my experience, it doesn't always have the fastest implementation.

More importantly, Python/numpy are row-major order by default, so matching that convention already keeps the Dara contiguous/cache friendly.

[kmsquire]

(But running the benchmarks is definitely the way to go!)

Since Julia just obtained a Transpose type, this actually shouldn't be that costly.

[nalimilan]

More importantly, Python/numpy are row-major order by default, so matching that convention already keeps the Dara contiguous/cache friendly.

I didn't know that. It's indeed a good explanation. OTOH, R is column-major, but it doesn't care so much about speed...

[nalimilan]

A confirmation that R's dist function is slow due to storing observations in rows despite being column-major: https://stat.ethz.ch/pipermail/r-devel/2017-June/074492.html

[andreasnoack]

Most relevant comments have already been made here but just wanted to point out that it seems that ML algorithms are often expressed and written by observation, i.e. for OLS that would be (∑x*x')\(∑x*y) instead of the matrix notation (X'X)\(X'y). The former can be convenient if data is just considered a stream. However, it is often wrongly argued that the former is also faster when each x is contiguous but because of (mainly) BLAS3 calls, the matrix formulation, if available, will typically be quite a bit faster than the "streaming"/"vector" formulation.