Better handling of memory usage

[benjello]

Issue by benjello from Thursday Nov 29, 2012 at 16:25 GMT
Originally opened as https://github.com/openfisca/openfisca/issues/89

---

3 ways of doing that:

• sparse DataFrames http://pandas.pydata.org/pandas-docs/version/0.7.1/sparse.html (see france/data/sparse_test for problem with sparseSeries)
• Wait for imporvements in pandas to work with out of memory data (see roadmap http://pandas.pydata.org/developers.html )
• load/save data when needed (mode reform with survey data)

[benjello]

Comment by benjello from Friday Nov 30, 2012 at 10:21 GMT

---

SpareSeries.density is no more a problem : pandas-dev/pandas#2384

Keeping the issue opened at least until the next release of pandas

[benjello]

Comment by benjello from Friday Feb 15, 2013 at 15:41 GMT

---

Updating my first post.

• sparse DataFrames http://pandas.pydata.org/pandas-docs/version/0.7.1/sparse.html are still not very usable (see sparse_test) and SparseDataFrame should be able to handle also non float non sparse Columns pandas-dev/pandas#2873
• Wait for improvements in pandas to work with out of memory data (see roadmap http://pandas.pydata.org/developers.html ). I tried with frame_table as in http://pandas.pydata.org/pandas-docs/dev/io.html#hdf5-pytables but it doesn't work (see sparse_text example but you should before rbuild survey.h5 using build_from_sources and set the option='table' when building the tables)
• load/save data when needed (mode reform with survey data). May we should also read individual columns (in group to limit disk access) when needed and delete them when not needed. We can do that efficiently if we can infer the order of the needed column for the computation of the prestation
• Use different dataframe for individu/menage/famille/foyer to avoid sparsity.
• Slice the main data table into many chunks but make sure every menage/famille/foyer is in only one chunk
• Fine tune the dtype of Column. This should soon be the moment since this is coming ENH/BUG/DOC: allow propogation and coexistance of numeric dtypes pandas-dev/pandas#2708. For information here is the dtype of a typical input table
  bool 27
  float32 400
  float64 2
  int64 25
  We definitely should be able to lower some int. And should review the floats.

The last two points are doable. Comments welcome

[benjello]

Comment by AlexisEidelman from Wednesday May 29, 2013 at 13:07 GMT

---

To chunk dataset, I need an option subset in set_survey.
I've already done something in that sense. To be continued...

[MattiSG]

This issue is never ending (and never started ;) ).

In order to increase the readability of the current active work in progress, I close this 3-years old issue.