Scalene process cannot be used in place of multiprocessing process #651

Why-not-now · 2023-08-08T07:06:43Z

Describe the bug
I think Scalene tries to replace multiprocessing processes with its own. However, in doing so, it makes the project unable to pickle the functions

To Reproduce
Steps to reproduce the behavior:

use this in a python file

from multiprocessing import Pool


def f(x):
    return x * x


if __name__ == '__main__':
    with Pool(processes=3) as pool:
        result = pool.imap_unordered(f, range(20))

run scalene <filename>.py
raises error Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed

Desktop (please complete the following information):

OS: Windows 10
Version scalene==1.5.26, cloudpickle==2.2.1

If you have not yet tried with the repository version (python3 -m pip install git+https://github.com/plasma-umass/scalene), please try that before reporting.

Additional context
Full error:

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed
Traceback (most recent call last):
  File "C:\Users\USER\.venv\Lib\site-packages\scalene\scalene_profiler.py", line 1857, in profile_code
    exec(code, the_globals, the_locals)
  File "C:\Users\USER\<filename>.py", line 9, in <module>
    with Pool(processes=3) as pool:
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 215, 
in __init__
    self._repopulate_pool()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 306, 
in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 329, 
in _repopulate_pool_static
    w.start()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\context.py", line 336, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\popen_spawn_win32.py", line 94, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line 
60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 111, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line 
79, in duplicate
    return _winapi.DuplicateHandle(
           ^^^^^^^^^^^^^^^^^^^^^^^^

OSError: [WinError 6] The handle is invalid/EOFError: Ran out of input

The text was updated successfully, but these errors were encountered:

Why-not-now · 2023-08-08T16:46:40Z

#579, #308

I think this is linked

Why-not-now · 2023-08-08T17:13:34Z

I think I located the problem:

scalene/scalene/replacement_mp_lock.py

Lines 18 to 30 in 31b9312

    
           class ReplacementSemLock(multiprocessing.synchronize.Lock): 
        
               def __enter__(self) -> bool: 
        
                   timeout = sys.getswitchinterval() 
        
                   tident = threading.get_ident() 
        
                   while True: 
        
                       scalene.set_thread_sleeping(tident) 
        
                       acquired = self._semlock.acquire(timeout=timeout)  # type: ignore 
        
                       scalene.reset_thread_sleeping(tident) 
        
                       if acquired: 
        
                           return True 
        
               def __exit__(self, *args: Any) -> None: 
        
                   super().__exit__(*args)

Since the class is nested in the function and does not have a getstate dunder, it can't be pickled.

emeryberger · 2023-08-23T14:40:08Z

I believe this is fixed with 5e6982e. Please update to the latest version and try again!

Why-not-now · 2023-08-23T17:40:20Z

Did not work, same error still.

Why-not-now · 2023-08-23T17:40:49Z

scalene 1.5.26

Why-not-now · 2023-08-23T18:02:22Z

Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed

emeryberger · 2023-08-23T18:10:19Z

I've moved the class out of the function; can you see if that helps? Thanks!

python3 -m pip install git+https://github.com/plasma-umass/scalene@issue_651

Why-not-now · 2023-08-23T18:17:41Z

It doesn't work still. I am so confused. Maybe the version of my python may be different?

Python 3.11.4

emeryberger · 2023-08-23T19:23:34Z

I just tested it and it's actually broken for a good reason; let me see what I can do.

emeryberger · 2023-08-23T19:42:18Z

OK, I just pushed an update and tested it on Windows with your example program, and it worked! Please try again (as before, python3 -m pip install git+https://github.com/plasma-umass/scalene@issue_651)

emeryberger · 2023-08-23T21:03:24Z

You can now just install this directly from the repo (I've merged the branch). python3 -m pip install -U git+https://github.com/plasma-umass/scalene

Why-not-now · 2023-08-24T10:10:10Z

OMG YES IT WORKSSSS
Thanks so much.

Why-not-now · 2023-08-24T10:12:16Z

Sorry, just reopened so this can be marked as completed when branch is merged

Why-not-now · 2023-08-24T11:16:23Z

Okay now there is a problem when profiling this code.

from multiprocessing import Pool


def f(x):
    return x * x


if __name__ == '__main__':
    with Pool(processes=3) as pool:
        for y in pool.imap_unordered(f, range(20)):
            pass

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 130, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Documents\coding\.venv\Lib\site-packages\scalene\replacement_sem_lock.py", line 10, in <module>
    class ReplacementSemLock(multiprocessing.synchronize.Lock):
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'multiprocessing' has no attribute 'synchronize'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 130, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Documents\coding\.venv\Lib\site-packages\scalene\replacement_sem_lock.py", line 10, in <module>
    class ReplacementSemLock(multiprocessing.synchronize.Lock):
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'multiprocessing' has no attribute 'synchronize'
...```

Why-not-now · 2023-08-24T11:50:49Z

By manually importing synchronize using from multiprocessing import synchronize, init doesn't work.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 130, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Documents\coding\.venv\Lib\site-packages\scalene\replacement_sem_lock.py", line 9, in _recreate_replacement_sem_lock
    return ReplacementSemLock()
           ^^^^^^^^^^^^^^^^^^^^
TypeError: Lock.__init__() missing 1 required keyword-only argument: 'ctx'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 130, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Documents\coding\.venv\Lib\site-packages\scalene\replacement_sem_lock.py", line 9, in _recreate_replacement_sem_lock
    return ReplacementSemLock()
           ^^^^^^^^^^^^^^^^^^^^
TypeError: Lock.__init__() missing 1 required keyword-only argument: 'ctx'
......

emeryberger · 2023-08-24T12:28:59Z

OK, just pushed some more changes to the repo. Please try again (and thanks again for your patience!).

Why-not-now · 2023-08-24T13:43:41Z

I can confirm it works. Thanks for your help.

emeryberger · 2023-08-24T14:26:08Z

Thanks again for your reports and your help in nailing down this issue!

hmycroft · 2023-09-12T20:01:47Z

I'm still having a similar problem trying to profile my multiprocessing. No hurry, but any ideas?

Error in program being profiled:
Can't pickle local object 'replacement_lock..ReplacementLock'
Traceback (most recent call last):
File "C:\Python311\Lib\site-packages\scalene\scalene_profiler.py", line 1894, in profile_code
exec(code, the_globals, the_locals)
File "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.py", line 165, in
with multiprocessing.Manager() as manager:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\multiprocessing\context.py", line 57, in Manager
m.start()
File "C:\Python311\Lib\multiprocessing\managers.py", line 563, in start
self._process.start()
File "C:\Python311\Lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\multiprocessing\popen_spawn_win32.py", line 94, in init
reduction.dump(process_obj, to_child)
File "C:\Python311\Lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'replacement_lock..ReplacementLock'
Traceback (most recent call last):
File "", line 1, in
File "C:\Python311\Lib\multiprocessing\spawn.py", line 113, in spawn_main
new_handle = reduction.duplicate(pipe_handle,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\multiprocessing\reduction.py", line 79, in duplicate
return _winapi.DuplicateHandle(
^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 6] The handle is invalid

emeryberger · 2023-09-13T17:48:40Z

Can you include more details? E.g., scalene --version? If you can provide a minimal "working" example, that would be most helpful. Thanks!

emeryberger closed this as completed Aug 23, 2023

emeryberger reopened this Aug 23, 2023

emeryberger mentioned this issue Aug 23, 2023

Moved class out of function to hopefully enable pickling. #664

Closed

emeryberger mentioned this issue Aug 23, 2023

Fixes issue 597 (thread sleep time) and 651 (pickling issue with multiprocessing) #665

Merged

Why-not-now closed this as completed Aug 24, 2023

Why-not-now reopened this Aug 24, 2023

Why-not-now closed this as completed Aug 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scalene process cannot be used in place of multiprocessing process #651

Scalene process cannot be used in place of multiprocessing process #651

Why-not-now commented Aug 8, 2023 •

edited

Loading

Why-not-now commented Aug 8, 2023 •

edited

Loading

Why-not-now commented Aug 8, 2023 •

edited

Loading

emeryberger commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

emeryberger commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

emeryberger commented Aug 23, 2023

emeryberger commented Aug 23, 2023

emeryberger commented Aug 23, 2023

Why-not-now commented Aug 24, 2023

Why-not-now commented Aug 24, 2023

Why-not-now commented Aug 24, 2023 •

edited

Loading

Why-not-now commented Aug 24, 2023 •

edited

Loading

emeryberger commented Aug 24, 2023

Why-not-now commented Aug 24, 2023

emeryberger commented Aug 24, 2023

hmycroft commented Sep 12, 2023 •

edited

Loading

emeryberger commented Sep 13, 2023

Scalene process cannot be used in place of multiprocessing process #651

Scalene process cannot be used in place of multiprocessing process #651

Comments

Why-not-now commented Aug 8, 2023 • edited Loading

Why-not-now commented Aug 8, 2023 • edited Loading

Why-not-now commented Aug 8, 2023 • edited Loading

emeryberger commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

emeryberger commented Aug 23, 2023

Why-not-now commented Aug 23, 2023

emeryberger commented Aug 23, 2023

emeryberger commented Aug 23, 2023

emeryberger commented Aug 23, 2023

Why-not-now commented Aug 24, 2023

Why-not-now commented Aug 24, 2023

Why-not-now commented Aug 24, 2023 • edited Loading

Why-not-now commented Aug 24, 2023 • edited Loading

emeryberger commented Aug 24, 2023

Why-not-now commented Aug 24, 2023

emeryberger commented Aug 24, 2023

hmycroft commented Sep 12, 2023 • edited Loading

emeryberger commented Sep 13, 2023

Why-not-now commented Aug 8, 2023 •

edited

Loading

Why-not-now commented Aug 8, 2023 •

edited

Loading

Why-not-now commented Aug 8, 2023 •

edited

Loading

Why-not-now commented Aug 24, 2023 •

edited

Loading

Why-not-now commented Aug 24, 2023 •

edited

Loading

hmycroft commented Sep 12, 2023 •

edited

Loading