Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

'mp_context' keyword for initialising concurrent.futures.ProcessPoolExecutor only supported in python 3.7+ #86

Closed
Realsid opened this issue Sep 15, 2020 · 2 comments
Labels
bug Something isn't working

Comments

@Realsid
Copy link

Realsid commented Sep 15, 2020

When running the following test in google colaboratry

from fastcore.utils  import parallel
from fastcore.test import test_eq
def add_one(x, a=1): 
    time.sleep(random.random()/80)
    return x+a

inp,exp = range(50),range(1,51)
test_eq(parallel(add_one, inp, n_workers=2), exp)

I get the following error:

/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in parallel(f, items, n_workers, total, progress, pause, timeout, chunksize, *args, **kwargs)
704 "Applies func in parallel to items, using n_workers"
705 if progress is None: progress = progress_bar is not None
--> 706 with ProcessPoolExecutor(n_workers, pause=pause) as ex:
707 r = ex.map(f,items, *args, timeout=timeout, chunksize=chunksize, **kwargs)
708 if progress:

/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in init(self, max_workers, on_exc, pause, mp_context, initializer, initargs)
685 self.not_parallel = max_workers==0
686 if self.not_parallel: max_workers=1
--> 687 super().init(max_workers, mp_context=mp_context, initializer=initializer, initargs=initargs)
688
689 def map(self, f, items, timeout=None, chunksize=1, *args, **kwargs):

TypeError: init() got an unexpected keyword argument 'mp_context'

Here is the python version:

!python --version

Python 3.6.9

@Realsid
Copy link
Author

Realsid commented Sep 15, 2020

I am not using mp_context, initializer and initargs. I removed them from the definition

super().init(max_workers)

This works, but I know this is not the solution in long run. Seeking a better solution that reconciles this version difference

@jph00 jph00 added the bug Something isn't working label Sep 16, 2020
@jph00 jph00 closed this as completed in 00625f9 Sep 16, 2020
@jph00
Copy link
Contributor

jph00 commented Sep 16, 2020

Many thanks for the report :)

butchland added a commit to butchland/fastcore that referenced this issue Sep 20, 2020
…commit id #e6620759fada127a1ef4cb2336d4ad9f8c000910

This will allow the fastai package to run with Python 3.6 which doesn't have the
mp_context parameter in the list of arguments to ProcessPoolExecutor,
which was introduced only in Python 3.7.
See https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor
matjesg added a commit to matjesg/deepflash2 that referenced this issue Sep 22, 2020
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants