ENH: read_sas, to_sas

[benjello]

It would be really convenient to be able to at least import SAS tables into pandas dataframe. Is this planned ? Are they insurmountable issues ?

Thanks

[jreback]

can you post a link to the format? and see if any converters have been writtenin python?

obviously the idea would be to read the native format file

[cpcloud]

http://support.sas.com/techsup/technote/ts140.html

[jreback]

so looks like simple binary read/write stuff....could be done....only other question is there any license issue with doing this?

[cpcloud]

not sure. @benjello want to contact SAS and ask them?

[mikelkelle]

That technote is describing the XPORT format, which isn't the native binary format. I've never seen a published layout of the native format. Some people have partially reverse engineered it, but I've never seen a solution that could handle data sets with compression.

https://github.com/BioStatMatt/sas7bdat

[cpcloud]

@mtkni thanks had no idea.

[jreback]

aside from using SAS to actually export (e.g. csv or whatever), is there aformat that one could save that provides some interoperbility (and is openish)?

[cpcloud]

a 10 minute search suggests no but maybe someone else knows more.

[benjello]

I am not a specialist much more a potential user in heavy need of such a tool.
For now I have used alternatively StatTransfer which is not a free software
or when importing to R I used one of the method exposed here but you need to have sas installed.
I am sorry for not being able to provide ypu for more information than above.

[cpcloud]

probably could do

1. call to sas if exists
2. if doesn't exist can only read xport so try to read that (someone would need to implement the xport reader)
3. if not 1 or 2 then fail saying u need sas to read sas binary files.

[jreback]

I don't think 1) is a good idea

export data in stata to xport format or csv format

[cpcloud]

ok. just throwing it out there. i don't like calling out to other programs either, but this seems like it's going to be tough. i can implement the R code above...if u think that's a good idea...but it basically forces users to use that particular version of the format and if it ever changes we won't know until it breaks.

[cpcloud]

i wouldn't be able to test to_sas though since i don't have sas

[benjello]

I would be glad to test everything that would do the job. I have sas.

[mikelkelle]

I've spent some time on this in the past. These are my thoughts:

1. Reverse engineering the binary data set format is a difficult task and not a good priority for this project. Any solution, like the R solution, that doesn't deal with compressed data sets won't do me any good. I can't speak for others on that. I also worry about the licensing issues.
2. Given that, the only option for read_sas may be to use the sas executable to first export SAS to a format that can be read by Python.
3. The SAS transport format, as described in the tech note, is an option but it has some quirks. In particular, the native XPORT engine doesn't handle long variable names. It really hasn't evolved since SAS 6 (we're on SAS 9 now). The SAS workaround for long variable names is a set of macros that not everyone has installed.
4. I'm not convinced the XPORT format is a better option than just CSV. If it's dramatically faster it might be. I'll study that next week.
5. If the mechanism for reading SAS data sets turns out to be export to CSV and then using read_csv, then there's not much code to write that isn't SAS environment dependent. It just doesn't make for a good shared module.

As much as I wish there was a good solution to this, and as much as I'd be willing to help build it, I just don't think there is. I've built read_sas using CSV as an intermediate format. Obviously, this requires a SAS license. It takes very few lines to implement, but most of those lines are specific to our SAS environment and are not well-portable.

I will study the performance of XPORT vs CSV next week. If it's dramatically faster, then it may be worth the effort to implement. Even then, I'm not sure it's worth taking that on as part of the Pandas project. I would be interested in comments from other SAS users on that.

Just my two cents.