ENH: pearson correlation with population mean

[Xylambda]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

When estimating person correlation Pandas computes the sample mean of the involved variables. Although this is a reasonable approach it may lead to biases when one knows the true mean.

For instance, if I generate n number of random series and compute their autocorrelations, I see a bias due to the mean estimation and the correlation between covariance and variance.

I would like Pandas to let me pass the true mean.

Feature Description

I am not sure how to insert this into the existing API. Maybe:

def corr(method='pearson', x_mean=None, y_mean=None):
    if method == 'pearon':
        if x_mean is None:
            x_mean = compute_sample_mean_of_x
        if y_mean is None:
            y_mean = compute_sample_mean_of_y

Alternative Solutions

def pearson(x, y, x_mean=None, y_mean=None):
    
    if x_mean is None:
        x_mean = x.mean()
    
    if y_mean is None:
        y_mean = y.mean()

    x_ = x - x_mean
    y_ = y - y_mean
    
    cov = (x_ * y_).mean()
    den = ((x_ ** 2).mean() * (y_ ** 2).mean())
    return cov / np.sqrt(den)

Additional Context

I would like to point out that this is not the only bias in my problem, but I don't understand why Pandas does not let me pass the population means of my variables when I know them, expecting me to have enough data for the sample mean estimation to be unbiased.

• Bias in the Estimation of Autocorrelations

[RaghavKhemka]

Hello @Xylambda,

Current Scenario of df.corr

Pandas uses Welford's method for the variance calculation which is an incremental method where you start with the first pair of values and keep on updating the mean and cumulative covariance with every value in the row. Read here https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
The good thing about this method is you can calculate the correlation coefficient in a single pass. So if you have N rows for two set of values, the time complexity will be O(N) for calculation the correlation coefficient for the set of values. This makes df.corr very fast. Also it uses Cython which add to it's speed of calculation.

What if we add the functionality of passing the means of each column.

Given the current scenario, passing the means and using them will require changing the method and will make the corr funciton slightly slower in pandas.