Weighted mean

[changhiskhan]

No description provided.

[changhiskhan]

@wesm ok to introduce fused types yet? Guess it's not strictly necessary, but would like to enable it so dt64 ints can be used as weights

[wesm]

is this v0.11 material?

[gerigk]

great feature! when do you think you will merge this?

[jreback]

@changhiskhan I can include float32 in the fuse dtypes for compat would be great

[changhiskhan]

@jreback that would be great thanks.

@gerigk it's just pending review (and jreback's additions). It'll definitely make it in 0.12 if not before.

[jreback]

@changhiskhan out of curiosity is there any difference (perf/ease of use) for fused types vs type generation (eg generated.pyx)

[changhiskhan]

Haven't done a performance comparison but I'd be surprised if there was a substantial difference there.
fused types definitely easier to use and more natural, at least more so than the current code gen stuff in pandas.

On Mar 4, 2013, at 8:41 AM, jreback [email protected] wrote:

@changhiskhan out of curiosity is there any difference (perf/ease of use) for fused types vs type generation (eg generated.pyx)

—
Reply to this email directly or view it on GitHub.

[wesm]

I don't think there is. I would suggest migrating as much of the generated.pyx code to use fused types as soon as feasible. There are some NA-handling issues where it may not be possible to migrate everything.

[ghost]

Aren't compilation times longer?

[wesm]

We're currently doing "fused types by hand". I think the compilation times may actually be shorter because the number of Python entry points will be a lot smaller

[jreback]

@changhiskhan
I rebased a copy of your branch onto current master....2 failures
can you take a look?

[sheep-jreback-~/pandas] nosetests pandas/tests/test_frame.py 
.........................................................................................................................................................................................SSS...........................................................................................................................................................................................................................S....................S...................F......
======================================================================
FAIL: test_weighted_mean_1d_weights (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/tests/test_frame.py", line 7515, in test_weighted_mean_1d_weights
    inject_skipna=True)
  File "/mnt/home/jreback/pandas/pandas/tests/test_frame.py", line 7680, in _check_stat_op
    check_dtype=False)  # HACK: win32
  File "/mnt/home/jreback/pandas/pandas/util/testing.py", line 164, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "/mnt/home/jreback/pandas/pandas/util/testing.py", line 106, in assert_almost_equal
    assert_almost_equal(a[i], b[i], check_less_precise)
  File "/mnt/home/jreback/pandas/pandas/util/testing.py", line 137, in assert_almost_equal
    1, a / b, decimal=decimal, err_msg=err_msg(a, b), verbose=False)
  File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 460, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 5 decimals expected -0.47729 but got nan

----------------------------------------------------------------------
Ran 455 tests in 4.492s

FAILED (SKIP=5, failures=1)
[sheep-jreback-~/pandas] nosetests pandas/tests/test_series.py
...........................................................................................S..........................................................................................................................................S.............E............
======================================================================
ERROR: test_weighted_mean (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1363, in test_weighted_mean
    self._check_stat_op('mean', wgt_mean, weights=weights)
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1512, in _check_stat_op
    testit()
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1503, in testit
    assert_almost_equal(float(f(s)), float(alternate(s.values)))
  File "/mnt/home/jreback/pandas/pandas/tests/test_series.py", line 1358, in wgt_mean
    mask = notnull(weights).mul(notnull(data)).dropna().astype(bool)
  File "/mnt/home/jreback/pandas/pandas/core/series.py", line 322, in f
    raise ValueError('Lengths must be equal')
ValueError: Lengths must be equal

----------------------------------------------------------------------
Ran 257 tests in 1.208s

FAILED (SKIP=2, errors=1)
[sheep-jreback-~/pandas] 

[gerigk]

what is the status here? is it safe to use?

[gerigk]

will this feature really make it into 0.13? I seem to be the only one really interested for quite some time now ;-)

[jtratner]

i'd definitely be interested in this as well.

[jreback]

pushing to 0.14, needs some TLC

[gerigk]

still planned for 0.14? I am not technical enough to help here but I would be a huge fan of not having to write ugly weighted mean workarounds anymore :)

[jreback]

would need someone to champion this.....

[TomAugspurger]

I'll take it. I'm not a C coder, so if anyone else wants it they can have it.

[jreback]

closing as stale.

@TomAugspurger if you want to pick this up, by all means