Fail to parse big-endian >2GB SAS files on Windows

[evanmiller]

See discussion here: #33

[evanmiller]

@vinhdizzo - Are you able to open a >2.1GB file on Windows that was created by SAS on a little-endian platform? It's possible this issue is limited to files created on big-endian platforms.

[vinhdizzo]

I don't have immediate access to SAS on a little-endian platform. I may have an old Linux laptop with SAS installed...and can look for it. Will have to roll the date back to see if that works. Will take some time though.

[evanmiller]

Will, it's not urgent, just thought that might provide a clue. I believe others have had success with large files on Windows, so for now I'm going to guess this is a created-on-big-endian issue.

[vinhdizzo]

Created a > 2GB file on a little-endian platform and I was able to read it on Windows. Thus, I think this issue is specific to > 2GB files created on a big-endian platform.

[evanmiller]

This is good to know. Can you try opening the same file on AIX and see which error you get? ("Cannot allocate vector" or "Invalid file, or file has unsupported features"). Please report the result in #33.

[evanmiller]

Does the >2.1 GB error happen immediately when you open on Windows?

[vinhdizzo]

Yes, immediately.

[evanmiller]

It's possible the AIX-related patches fixed this issue -- please try the latest ReadStat code and re-test.

Instructions for updating: https://github.com/hadley/haven#updating-readstat

[vinhdizzo]

I downloaded haven-master.zip and ReadStat-master.zip, extracted the 2, copied c and h files from ReadStat to haven, ran the unlink function in R, and used devtools to install the latest haven. I get the following errors on a few files on windows:

> d1 = read_sas('bar_big_endian.sas7bdat')
ReadStat: Failed to seek to position 2165189632 (= 8192 + 26765*80896)
ReadStat: Unable to seek within file (retval = 13): No error (errno = 0)

Error: Failed to parse bar_big_endian.sas7bdat: Unable to seek within file.
> d1 = read_sas('baz2gb.sas7bdat')
ReadStat: Failed to seek to position 2428101632 (= 8192 + 30015*80896)
ReadStat: Unable to seek within file (retval = 13): No error (errno = 0)

Error: Failed to parse baz2gb.sas7bdat: Unable to seek within file.
> d1 = read_sas('bar_little_endian.sas7bdat')
ReadStat: Failed to seek to position 2225861632 (= 8192 + 27515*80896)
ReadStat: Unable to seek within file (retval = 13): No error (errno = 0)

Error: Failed to parse bar_little_endian.sas7bdat: Unable to seek within file.

[evanmiller]

Thanks. Try this: readstat_io.c

[vinhdizzo]

This is on Windows (replaced readstat_io.c with the merged haven-master and ReadStat-master). Error compiling:

gcc -m64 -I"F:/RRO/R-3.2.2/include" -DNDEBUG    -I"F:/RRO/R-3.2.2/library/Rcpp/include" -I"F:/RRO/R-3.2.2/library/BH/include" -I"c:/applications/extsoft/include"     -O2 -Wall  -std=gnu99 -mtune=core2 -c readstat_variable.c -o readstat_variable.o
gcc -m64 -I"F:/RRO/R-3.2.2/include" -DNDEBUG    -I"F:/RRO/R-3.2.2/library/Rcpp/include" -I"F:/RRO/R-3.2.2/library/BH/include" -I"c:/applications/extsoft/include"     -O2 -Wall  -std=gnu99 -mtune=core2 -c readstat_writer.c -o readstat_writer.o
g++ -m64 -shared -s -static-libgcc -o haven.dll tmp.def DfReader.o DfWriter.o RcppExports.o readstat_bits.o readstat_convert.o readstat_dta.o readstat_dta_read.o readstat_dta_write.o readstat_error.o readstat_io.o readstat_parser.o readstat_por.o readstat_por_parse.o readstat_sas.o readstat_sav.o readstat_sav_parse.o readstat_sav_read.o readstat_sav_write.o readstat_spss.o readstat_spss_parse.o readstat_value.o readstat_variable.o readstat_writer.o -lRiconv -Lc:/applications/extsoft/lib/x64 -Lc:/applications/extsoft/lib -LF:/RRO/R-3.2.2/bin/x64 -lR
readstat_por.o:readstat_por.c:(.text+0x41b): undefined reference to `ck_hash_table_init'
readstat_por.o:readstat_por.c:(.text+0x534): undefined reference to `ck_hash_table_free'
readstat_por.o:readstat_por.c:(.text+0x9d8): undefined reference to `ck_str_hash_lookup'
readstat_por.o:readstat_por.c:(.text+0xd82): undefined reference to `ck_str_hash_insert'
collect2.exe: error: ld returned 1 exit status
no DLL was created
ERROR: compilation failed for package 'haven'
* removing 'F:/RRO/R-3.2.2/library/haven'
* restoring previous 'F:/RRO/R-3.2.2/library/haven'
Error: Command failed (1)

[evanmiller]

Do you see CKHashTable.c in the "src" directory?