Symbol not found: _libiconv

[ozak]

@ofajardo thanks for working on this!
I have been trying to make it work in a conda environment with pyton3.5 on two separate OSX systems and I always get the same error. I also tried installing in the homebrew version of python with similar results. I tried the simple pip install, also from Github, compiling from scratch...nothing. Any ideas what it could be? I do see libiconv in the conda list.

import pyreadstat
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-95ad4ef9013f> in <module>()
----> 1 import pyreadstat

~/Anaconda3/envs/GeoPython3env/lib/python3.5/site-packages/pyreadstat/__init__.py in <module>()
----> 1 from .pyreadstat import read_sas7bdat, read_xport, read_dta, read_sav, read_por, read_sas7bcat
      2 from .pyreadstat import set_value_labels, set_catalog_to_sas
      3 from ._readstat_parser import ReadstatError, metadata_container

ImportError: dlopen(/Users/user/Anaconda3/envs/GeoPython3env/lib/python3.5/site-packages/pyreadstat/pyreadstat.cpython-35m-darwin.so, 2): Symbol not found: _libiconv
  Referenced from: /Users/user/Anaconda3/envs/GeoPython3env/lib/python3.5/site-packages/pyreadstat/pyreadstat.cpython-35m-darwin.so
  Expected in: flat namespace
 in /Users/user/Anaconda3/envs/GeoPython3env/lib/python3.5/site-packages/pyreadstat/pyreadstat.cpython-35m-darwin.so

[ofajardo]

Hi @ozak

I am a bit on the dark as I do not have a machine with OSX to test. What I can tell is that libiconv is not a python library but a C library. Try installing it with homebrew to see if that solves the problem. Something like this:

https://stackoverflow.com/questions/5528839/why-does-installing-nokogiri-on-mac-os-fail-with-libiconv-is-missing/24680152

Please let me know if that solves the issue, in order to mention it in the documentation.

If that does not solve it, from the top of my head, in setup.py there is a line:
libraries.append("iconv")
Unindent it so that it gets executed, clean your env (caches) and try to install again.

The problem is that it finds it during compilation but later not anymore when importing the solution is either installing a version it can later find (first suggestion) or tell it how to find the one already there (second suggestion)

[ozak]

@ofajardo Thanks for the suggestions....Sadly, I already had libiconv installed on both homebrew and conda and that did not do the trick. It was still not able to find it. Your second suggestion also did not work. What did work is to add extra_compile_args as suggested here, i.e. in setup.py I replaced

extensions = [Extension("pyreadstat.pyreadstat",
                sources=["pyreadstat/pyreadstat" + ext] + sources,
                # this dot here is important for cython to find the pxd files
                include_dirs = [source_dir_root] + source_dirs + ["pyreadstat", "."],
                libraries=libraries,
                extra_compile_args=["-Ireadstat", "-DHAVE_ZLIB=1",
                                    "-I/ANACONDADIR/include",
                                    "-I/ANACONDADIR/lib"] ),
                Extension("pyreadstat._readstat_parser",
                sources=["pyreadstat/_readstat_parser" + ext] + sources,
                include_dirs = [source_dir_root] + source_dirs + ["pyreadstat", "."],
                libraries=libraries,
                extra_compile_args=["-Ireadstat", "-DHAVE_ZLIB=1",
                                    "-I/ANACONDADIR/include",
                                    "-I/ANACONDADIR/lib"] )]

This should solve the issue for anyone who wants to compile it. Not sure why you have the commands repeated though. Also not sure how to force it to find the ANACONDADIR or how to provide the program with the directory where this is to be found so that it could work with pip or conda. If you figure it out, it would be nice to have a conda-forge recipe so that the package becomes more easily available for others. The lack of a SAS and SPSS reader/writer is a major issue from my point of view.

[ofajardo]

@ozak brilliant! I am very happy that you managed to solve it.

I have asked another mac user and he said that he managed to install directly from pip without issue, so it seems that it is not a general problem. My guess he somehow had already libiconv on his system because of having installed it before with another application. I will see if I can discover more.
On a more generic solution, I think to find out automatically where is ANACONDADIR is not that easy, and also is not so evident where the libiconv is going to be (I am seeing different folder structures on windows and on linux and yours is different as well). Probably the user would have to tweek it anyway which is not so neat. In addition it requires that the user installs previously libiconv with conda. I will look at conda forge, but that means it would not be valid for people wanting to use virtualenv.

One simple thing that may work tough is the following: on windows what I did is to ship libiconv together with pyreadstat, in that way nothing needs to be installed in addition and pip is enough. Maybe that could be useful in this case as well? If you had some time, you could (in a new clean environment) try to install pyreadstat again, confirm that you get the error, and then just copy libiconv together with the the pyreadstat files in site packages and see if that cures the problem. If it does, then I could just add that file to the package-data as I do on windows. What do you think?

[ozak]

@ofajardo I am not sure what the issue is in that case. Since I had libiconv installed on both versions before installing pyreadstat. Not sure why it did not find it on my machines, but found it on the others. I will try what you propose once I have some spare time.

[ozak]

@ofajardo it seems it works, at least if you pip install git+https://github.com/Roche/pyreadstat.git in conda.

[ofajardo]

@ozak thanks a lot for testing.

I also tested with couple more users and it was working. For one it was OK with the system libraries, but for one it was necessary to install the libiconv from conda because the system libraries were old.

I will prepare a new Pypi package so that it can be installed with pip.

For now I will suggest in the documentation that if there are problems with libiconv a first thing to try is to install it from conda and try to compile again. Later I will check on how to do a conda pacakge/recipe as you suggested.

For the records:

In order for pyreadstat to work on Mac, it was necessary to do a static linking for iconv. Setup.py was modified for this purpose (static linking is necessary also for windows, but not for linux as iconv seems to be part of the gclib library and not a separate shared library).
Os X has that library in /usr/lib and the header file iconv.h in /usr/include. Normally pyreadstat will take it from there. If for some reason that does not work, installing libiconv with conda install libiconv and then recompiling, so that pyreadstat now links to that library, seems to solve the issue.

Useful notes:
how to check what shared libraries are loaded by pyreadstat:

DYLD_PRINT_LIBRARIES=YES python -c "import pyreadstat"

how to check a shared library version:

otool -L /usr/lib/libssl.0.9.7.dylib 

[ofajardo]

@ozak a new version is out in PyPi (0.1.8). If you get an opportunity to test it, let me know if it works.

[ofajardo]

Closed after 1 month of inactivity. If anybody else has a similar issue please open a new ticket.