Skip to content

Distributed hello world fails when using jemalloc #1190

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
bluenote10 opened this issue Jun 20, 2017 · 10 comments
Closed

Distributed hello world fails when using jemalloc #1190

bluenote10 opened this issue Jun 20, 2017 · 10 comments

Comments

@bluenote10
Copy link
Contributor

Observation made in relation to #1179.

Running a local distributed executor like in the "hello world" example crashes if the allocator is jemalloc.

Starting an IPython shell via LD_PRELOAD=/usr/lib/libjemalloc.so.1 ipython and running the following code

from dask.distributed import Client
client = Client()

def square(x):
    return x ** 2

def neg(x):
    return -x

A = client.map(square, range(10))
B = client.map(neg, A)
total = client.submit(sum, B)
print(total.result())

crashes with ValueError: Worker not started

The code works fine using glibc.
@jreback
Copy link
Contributor

jreback commented Jun 20, 2017

xref pandas-dev/pandas#16727

@xhochy
Copy link

xhochy commented Jun 20, 2017

@jreback no, they don't seem to be related here.

Sadly I cannot reproduce this problem on my machines here (neither OSX nor Debian 7+). It seems that there is some relation to the old Ubuntu 12.04 image @bluenote10 is running on which ships with jemalloc 2.2.x and glibc 2.13.

@pitrou
Copy link
Member

pitrou commented Jun 20, 2017

I cannot reproduce on Ubuntu 16.04 either. In any case, this doesn't seem to be a Distributed bug, so I'm inclined to close this issue unless you have significant reason to believe Distributed is involved.

@bluenote10
Copy link
Contributor Author

bluenote10 commented Jun 20, 2017 •
edited
Loading

I'm not sure. Similar to the other issue it seems to affect the networking. Maybe you can make more sense of the full traceback:

tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.py", line 173, in _start_worker
    yield w._start()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 135, in _start
    response = yield self.instantiate()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 189, in instantiate
    self.process.start())
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 300, in start
    yield self._wait_until_running()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 385, in _wait_until_running
    raise ValueError("Worker not started")
ValueError: Worker not started
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-9a5cb41bb097> in <module>()
      1 from dask.distributed import Client
----> 2 client = Client()
      3 
      4 def square(x):
      5     return x ** 2

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in __init__(self, address, loop, timeout, set_as_default, scheduler_file, security, start, asynchronous, **kwargs)
    471                                      io_loop=self.loop)
    472 
--> 473         self.start(timeout=timeout, asynchronous=asynchronous)
    474 
    475         from distributed.channels import ChannelClient

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in start(self, asynchronous, **kwargs)
    570             self._started = self._start(**kwargs)
    571         else:
--> 572             sync(self.loop, self._start, **kwargs)
    573         self.status = 'running'
    574 

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/utils.pyc in sync(loop, func, *args, **kwargs)
    232         e.wait(1000000)
    233     if error[0]:
--> 234         six.reraise(*error[0])
    235     else:
    236         return result[0]

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/utils.pyc in f()
    221                 raise RuntimeError("sync() called from thread of running loop")
    222             yield gen.moment
--> 223             result[0] = yield make_coro()
    224         except Exception as exc:
    225             logger.exception(exc)

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in _start(self, timeout, **kwargs)
    614                 self.cluster = LocalCluster(loop=self.loop, start=False,
    615                                             **self._startup_kwargs)
--> 616                 yield self.cluster._start()
    617             except (OSError, socket.error) as e:
    618                 if e.errno != errno.EADDRINUSE:

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.pyc in _start(self, ip)
    146         yield self._start_all_workers(
    147             self.n_workers, ncores=self.threads_per_worker,
--> 148             services=self.worker_services, **self.worker_kwargs)
    149 
    150         self.status = 'running'

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.pyc in _start_all_workers(self, n_workers, **kwargs)
    152     @gen.coroutine
    153     def _start_all_workers(self, n_workers, **kwargs):
--> 154         yield [self._start_worker(**kwargs) for i in range(n_workers)]
    155 
    156     @gen.coroutine

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in callback(f)
    826             for f in children:
    827                 try:
--> 828                     result_list.append(f.result())
    829                 except Exception as e:
    830                     if future.done():

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.pyc in _start_worker(self, port, processes, death_timeout, **kwargs)
    171               death_timeout=death_timeout,
    172               silence_logs=self.silence_logs, **kwargs)
--> 173         yield w._start()
    174 
    175         self.workers.append(w)

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in _start(self, addr_or_port)
    133 
    134         logger.info('        Start Nanny at: %r', self.address)
--> 135         response = yield self.instantiate()
    136         if response == 'OK':
    137             assert self.worker_address

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in instantiate(self, comm)
    187             try:
    188                 yield gen.with_timeout(timedelta(seconds=self.death_timeout),
--> 189                                        self.process.start())
    190             except gen.TimeoutError:
    191                 yield self._close(timeout=self.death_timeout)

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in start(self)
    298         yield self.process.start()
    299         if self.status == 'starting':
--> 300             yield self._wait_until_running()
    301 
    302     def _on_exit(self, proc):

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1067                             exc_info = None
   1068                     else:
-> 1069                         yielded = self.gen.send(value)
   1070 
   1071                     if stack_context._state.contexts is not orig_stack_contexts:

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in _wait_until_running(self)
    383         while True:
    384             if self.status != 'starting':
--> 385                 raise ValueError("Worker not started")
    386             try:
    387                 msg = self.init_result_q.get_nowait()

ValueError: Worker not started

@pitrou
Copy link
Member

pitrou commented Jun 20, 2017

The fact that networking may be affected doesn't imply that Distributed is the culprit. Distributed is pure Python code and does not depend in any particular way on the underlying C malloc implementation. According to this message there might be a bad interaction between specific glibc and jemalloc versions.

I might add: why are you using jemalloc? Did you get specific performance improvements using it?

@bluenote10
Copy link
Contributor Author

Result of valgrind leading to the traceback

==23375== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23375==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23375==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x544CBB: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==  Address 0xffeffb998 is on thread 1's stack
==23375== 
==23854== Thread 13:
==23854== Invalid read of size 8
==23854==    at 0x4E39E01: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23854== 
==23854== Invalid read of size 8
==23854==    at 0x4E39F65: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23854== 
==23854== Invalid read of size 4
==23854==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==  Address 0x4ccccccd048d2cb0 is not stack'd, malloc'd or (recently) free'd
==23854== 
==23854== 
==23854== Process terminating with default action of signal 11 (SIGSEGV)
==23854==  General Protection Fault
==23854==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854== 
==23854== HEAP SUMMARY:
==23854==     in use at exit: 240 bytes in 3 blocks
==23854==   total heap usage: 3 allocs, 0 frees, 240 bytes allocated
==23854== 
==23856== Thread 14:
==23856== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23856==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23856==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x3a843518 is on thread 14's stack
==23856== 
==23860== Thread 15:
==23860== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23860==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23860==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==  Address 0x3b7fd518 is on thread 15's stack
==23860== 
==23864== Thread 16:
==23864== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23864==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23864==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==  Address 0x3c7fd518 is on thread 16's stack
==23864== 
==23867== Thread 17:
==23867== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23867==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23867==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==  Address 0x3d7fd518 is on thread 17's stack
==23867== 
==23856== Invalid read of size 8
==23856==    at 0x4E39E01: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23856== 
==23856== Invalid read of size 8
==23856==    at 0x4E39F65: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23856== 
==23856== Invalid read of size 4
==23856==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x4ccccccd078d2cb0 is not stack'd, malloc'd or (recently) free'd
==23856== 
==23856== 
==23856== Process terminating with default action of signal 11 (SIGSEGV)
==23856==  General Protection Fault
==23856==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856== 
==23856== HEAP SUMMARY:
==23856==     in use at exit: 240 bytes in 3 blocks
==23856==   total heap usage: 3 allocs, 0 frees, 240 bytes allocated
==23856== 
==23870== Thread 18:
==23870== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23870==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23870==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==  Address 0x3e7fd518 is on thread 18's stack
==23870== 
==23375== Thread 12:
==23375== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23375==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23375==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==  Address 0x2b3fec88 is on thread 12's stack
==23375== 
==23874== Thread 20:
==23874== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23874==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23874==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==  Address 0x407fd518 is on thread 20's stack
==23874== 
==23877== Thread 19:
==23877== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23877==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23877==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==  Address 0x3f7fd518 is on thread 19's stack
==23877== 
==23854== LEAK SUMMARY:
==23854==    definitely lost: 0 bytes in 0 blocks
==23854==    indirectly lost: 0 bytes in 0 blocks
==23854==      possibly lost: 0 bytes in 0 blocks
==23854==    still reachable: 240 bytes in 3 blocks
==23854==         suppressed: 0 bytes in 0 blocks
==23854== Reachable blocks (those to which a pointer was found) are not shown.
==23854== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==23854== 
==23854== For counts of detected and suppressed errors, rerun with: -v
==23854== Use --track-origins=yes to see where uninitialised values come from
==23854== ERROR SUMMARY: 9 errors from 4 contexts (suppressed: 0 from 0)
==23856== LEAK SUMMARY:
==23856==    definitely lost: 0 bytes in 0 blocks
==23856==    indirectly lost: 0 bytes in 0 blocks
==23856==      possibly lost: 0 bytes in 0 blocks
==23856==    still reachable: 240 bytes in 3 blocks
==23856==         suppressed: 0 bytes in 0 blocks
==23856== Reachable blocks (those to which a pointer was found) are not shown.
==23856== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==23856== 
==23856== For counts of detected and suppressed errors, rerun with: -v
==23856== Use --track-origins=yes to see where uninitialised values come from
==23856== ERROR SUMMARY: 17 errors from 5 contexts (suppressed: 0 from 0)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-9ae027076550> in <module>()
      1 from dask.distributed import Client
----> 2 client = Client()

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in __init__(self, address, loop, timeout, set_as_default, scheduler_file, security, start, asynchronous, **kwargs)
    471                                      io_loop=self.loop)
    472 
--> 473         self.start(timeout=timeout, asynchronous=asynchronous)
    474 
    475         from distributed.channels import ChannelClient

@bluenote10
Copy link
Contributor Author

I had less issues resulting from memory fragmentation with jemalloc, but I should be able to use glibc as well.

@xhochy
Copy link

xhochy commented Jun 20, 2017

@pitrou The arrow issues were solely a build environment issue that surfaced while loading a library. Once the library is successfully loaded, this glibc bug is not triggered anymore.

We're using jemalloc in general (note: I work at the same place as @bluenote10 ;) ) as it has less memory fragmentation as glibc and has a better multithreaded performance. In Apache Arrow (different use case then in this issue here), we also use it as it can provide aligned memory (re-)allocation, this enables us to use faster numeric CPU instructions.

@pitrou
Copy link
Member

pitrou commented Jun 20, 2017 •
edited
Loading

Ok. Still, I don't know what to do with this issue. Using a different memory allocator shouldn't mess network communications implemented in pure Python, unless there's something seriously wrong in low-level routines (I mean routines implemented in C either inside Python itself or inside system libraries and/or third-party C libraries).

As for the Valgrind output, the following seems fishy. But a Valgrind or glibc expert would have to dig in:


==23854== Process terminating with default action of signal 11 (SIGSEGV)
==23854==  General Protection Fault
==23854==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)

@bluenote10
Copy link
Contributor Author

Sure we can close it, if there is nothing that can be done. Just wanted to keep you posted on the issue.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@xhochy @mrocklin @jreback @pitrou @bluenote10