A script for reproducing the initialization error

[tkf]

I think I found a reproducible script to get the error I saw sometimes. Confusingly, the key to get the error was to execute using PyCall in Julia to invoke precompilation. Before running this script, remove ~/.juliarc.jl if you have PyCall-related configuration.:

rm -rvf ~/.julia/lib/pyjulia{2,3}-v0.6 ~/.julia/lib/v0.6/PyCall.ji ~/.julia/v0.6/PyCall/deps/PYTHON ~/.julia/v0.6/PyCall/deps/deps.jl

PYTHON="$(which python3)" julia -e 'Pkg.build("PyCall"); using PyCall'
# PYTHON="$(which python3)" julia -e 'Pkg.build("PyCall")'  # this produces no error below

conda create --yes --prefix py37 python=3.7

source ~/*conda*/bin/activate ./py37
which python
python --version
python -c 'import julia; julia.Julia()'

This script produces:

Error processing line 1 of /usr/lib/python3.6/site-packages/virtualenvwrapper-4.8.2-py3.6-nspkg.pth:

Failed to import the site module
Traceback (most recent call last):
  File "/usr/lib/python3.6/site.py", line 168, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/types.py", line 171, in <module>
    import functools as _functools
  File "/usr/lib/python3.6/functools.py", line 21, in <module>
    from collections import namedtuple
  File "/usr/lib/python3.6/collections/__init__.py", line 29, in <module>
    import heapq as _heapq
  File "/usr/lib/python3.6/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site.py", line 544, in <module>
    main()
  File "/usr/lib/python3.6/site.py", line 531, in main
    known_paths = addsitepackages(known_paths)
  File "/usr/lib/python3.6/site.py", line 328, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/usr/lib/python3.6/site.py", line 207, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/usr/lib/python3.6/site.py", line 178, in addpackage
    import traceback
  File "/usr/lib/python3.6/traceback.py", line 3, in <module>
    import collections
  File "/usr/lib/python3.6/collections/__init__.py", line 29, in <module>
    import heapq as _heapq
  File "/usr/lib/python3.6/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

If I don't run using PyCall before python -c 'import julia; julia.Julia()', I don't see this error. Above error also vanishes if I don't include julia.Julia() so I think it's pyjulia specific and not virtualenvwrapper even though it is mentioned in the error message above. This error is also reported #150 and Error processing line 1 of points to another Python library. See below also.

A very confusing effect is that, if I use Python environment created by conda for configuring PyCall, i.e., if I do

rm -rvf ~/.julia/lib/pyjulia{2,3}-v0.6 ~/.julia/lib/v0.6/PyCall.ji ~/.julia/v0.6/PyCall/deps/PYTHON ~/.julia/v0.6/PyCall/deps/deps.jl

conda create --yes --prefix py36 python=3.6
PYTHON="${PWD}/py36/bin/python" julia -e 'Pkg.build("PyCall"); using PyCall'

source ~/*conda*/bin/activate ./py37
which python
python --version
python -c 'import julia; julia.Julia()'

then I get another error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 370, in __init__
    self._call(u"using PyCall")
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 412, in _call
    self.check_exception(src)
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 441, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'InitError' occurred while calling julia code:
using PyCall

in 779808b and

ERROR: could not load library "/home/takafumi/repos/watch/pyjulia/julia/fake-julia/../lib/julia/sys.so"
/home/takafumi/repos/watch/pyjulia/julia/fake-julia/../lib/julia/sys.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 416, in __init__
    self._call(u"using PyCall")
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 453, in _call
    self.check_exception(src)
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 469, in check_exception
    sprint = self.sprint
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 526, in __getattr__
    from julia import Main
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 159, in load_module
    JuliaMainModule(self, fullname))
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 68, in __init__
    self._julia = loader.julia
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 151, in julia
    self.__class__.julia = julia = Julia()
  File "/home/takafumi/repos/watch/pyjulia/julia/core.py", line 315, in __init__
    """])
  File "/home/takafumi/repos/watch/pyjulia/py37/lib/python3.7/subprocess.py", line 376, in check_output
    **kwargs).stdout
  File "/home/takafumi/repos/watch/pyjulia/py37/lib/python3.7/subprocess.py", line 468, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['julia', '-e', '\n                 println(VERSION < v"0.7.0-DEV.3073" ? JULIA_HOME : Base.Sys.BINDIR)\n                 println(Libdl.dlpath(string("lib", splitext(Base.julia_exename())[1])))\n                 println(unsafe_string(Base.JLOptions().image_file))\n                 PyCall_depsfile = Pkg.dir("PyCall","deps","deps.jl")\n                 if isfile(PyCall_depsfile)\n                    eval(Module(:__anon__),\n                        Expr(:toplevel,\n                         :(Main.Base.include($PyCall_depsfile)),\n                         :(println(pyprogramname))))\n                 else\n                    println("nowhere")\n                 end\n                 ']' returned non-zero exit status 1.

in the current head e617ad9. (Note that everything can be reproducible in the current head except the above Exception 'InitError' occurred while calling julia code part). The last exception was actually a "bug" introduced by my patch #162 (which didn't manifest unless you miss-configured pyjulia/PyCall...). pyjulia won't work anyway but the error message should be clearer. I'll fix it.

[tkf]

Of course, @stevengj already knew the answer #92 (comment)

[tkf]

Using conda makes difference since:

$ ldd ~/miniconda3/bin/python | grep libpython
$ ldd /usr/bin/python | grep libpython
        libpython3.6m.so.1.0 => /usr/lib/libpython3.6m.so.1.0 (0x00007ff5b75ab000)