Skip to content

Kshedden sas7bdat perf #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: sas7bdat_perf
Choose a base branch
from
Open

Kshedden sas7bdat perf #1

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d07fb1c
Modest performance, address #12647
kshedden Mar 17, 2016
9741397
Add test data set from raderaj
kshedden Mar 19, 2016
3762954
Use encoding when reading column headers
kshedden Mar 19, 2016
0bbcdc6
More cython for performance, refactored constants
kshedden Mar 19, 2016
d1da036
Move more code to cython
kshedden Mar 19, 2016
863bb34
Decouple data decoding and decoding e.g. of column names
kshedden Mar 19, 2016
0503df1
Add two missing alignment constants
kshedden Mar 19, 2016
dc06ae8
Working on cython issues
kshedden Apr 10, 2016
230ce51
Working on cython issues
kshedden Apr 10, 2016
5d0403c
Further cythonization
kshedden Apr 10, 2016
2f61762
Added option to not decode header text
kshedden Apr 10, 2016
7adb740
added to whatsnew
kshedden Apr 11, 2016
270f6d7
Further encoding work
kshedden Apr 11, 2016
63f54a1
pep8 cleanup
kshedden Apr 11, 2016
ffb1222
PERF: some cython opts
jreback Apr 17, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions pandas/io/sas/sas7bdat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
import numpy as np
import struct
import pandas.io.sas.sas_constants as const
from pandas.io.sas.saslib import do_read
from pandas.io.sas.saslib import Parser


class _subheader_pointer(object):
Expand Down Expand Up @@ -583,7 +583,8 @@ def read(self, nrows=None):
self._byte_chunk = np.empty((nd, 8 * nrows), dtype=np.uint8)

self._current_row_in_chunk_index = 0
do_read(self, nrows)
p = Parser(self)
p.read(nrows)

rslt = self._chunk_to_dataframe()
if self.index is not None:
Expand Down
Loading