Sporadic segfaults with GCC on Ubuntu Linux #110
Comments
|
I have ended making a minimal example that crashes: from __future__ import print_function
import numpy
import blosc
print("Blosc version info:", blosc.blosclib_version)
# Setting the number of threads to 3 accelerates the segfaults occurrencies
blosc.set_nthreads(3)
a = numpy.arange(1e6)
parray = blosc.compress(a, clevel=9, shuffle=blosc.SHUFFLE, cname="blosclz")
ratio = len(a) * a.itemsize * 1. / len(parray)
print("Compression: %s -> %s (%4.1fx)" % (
len(a) * a.itemsize, len(parray), ratio))With that, it is quite easy to make the python-blosc wrapper to crash: $ time for i in {1..100}; do PYTHONPATH=. python segfault.py>p ; done
Segmentation fault (core dumped)
real 0m9.803s
user 0m8.416s
sys 0m1.380sThen, during my investigations I found this:
everything is fine, even in the case 1) above. So, that's a funny situation, and after thinking about this for a good amount of time, I propose to approach this issue as follows:
Addedum: Here it follows what you can expect from using python-blosc with an external C-Blosc library: Th above also has the advantage that C-Blosc CMake infraestructure can recognize the AVX2 support by the compiler much easier. Anyway, here it is the output with python-blosc extensions compiled with -O1 flag: So, although the -O1 case still performs very well, the external library can be more than 2 GB/s faster in some cases (specially with the bitshuffle filter that takes quite a bit of advantage from AVX2). Thoughts? |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
When using GCC (tested with 4.9.3 and 5.2.1) on a Ubuntu 15.10 box one can get sporadicly but consistently segfaults when exercising the test suite enough times:
I cannot get any segfault when using clang (tested with 3.6 and 3.7). Testing on a Mac OSX box does not show any problem either (this is normal because xcode brings clang/LLVM).
A detailed investigation using valgrind does not show anything too evident, except things like:
so perhaps there is a problem with reference counting but I am not sure if this is a red herring.
Anyway, as GCC is a very important compiler this ticket has high priority.
The text was updated successfully, but these errors were encountered: