batchrunnerMP example?

[swirya]

Adapting the bank_reserves example to use batchrunnerMP results in the following error:
MaybeEncodingError: Error sending result: '((25, 5, 0, 0), <main.BankReservesModel object at 0x7f5d26e1a700>)'. Reason: 'AttributeError("Can't pickle local object 'BankReservesModel.init..'")'

Here's the command:
batch = BatchRunnerMP(
BankReservesModel,
nr_processes=None,
variable_parameters=br_params,
iterations=1,
max_steps=1000,
model_reporters={"Data Collector": lambda m: m.datacollector},
)
batch.run_all()

What am I missing? Thanks in advance.

[rht]

This seems to be a known problem. There are 2 solutions:

1. Use https://github.com/uqfoundation/multiprocess instead of Python stdlib's multiprocessing'. Basically, do pip install multiprocess, and then replace

   mesa/mesa/batchrunner.py

   Line 11 in d224397

   from multiprocessing import Pool, cpu_count

   with from multiprocess import Pool, cpu_count
2. Use mp.Process instead, which doesn't require pickling

The first solution is a one-line fix, but requires installing an external library. The second one uses stdlib, but requires more changes in the code. Also, the code won't look clean (note: this messiness is invisible to the user; it's messy only for Mesa developers).

@swirya if you want a quick fix, you can do the first solution.

But a permanent solution requires more discussion.

[tpike3]

@swirya I am going to close as this resolved based on @rht's comments.

Also just for your awareness, we just did a release with batch_run and this uses multiprocessing by default. Overview is in the new docs

[rht]

I think this is still an open issue. batch_run uses the same multiprocessing mechanism as the old BatchrunnerMP, i.e. imap_unordered, and so there will still be a pickling problem. My second solution, which is to use mp.Process, should work, as I had encountered this pickling problem in the past, and solved it with mp.Process.

[rht]

I tried adding a lambda function in the model reporter of examples/bank_reserves/batch_run.py. This breaks BatchRunnerMP with the pickling problem, but it somehow works in batch_run. Both use imap_unordered, but maybe because batch_run is a simple function, it works.

Closing now, because pickling lambda functions work in batch_run, and BatchRunner and BatchRunnerMP are deprecated.