python - Passing IPython parallel cluster object into custom class for batch execution -

python - Passing IPython parallel cluster object into custom class for batch execution -

i novice programmer trying utilize python scientific programming. think these posts (how work interactively-defined classes in ipython.parallel? , ipython parallel force custom object) touches on similar issue not useful me. want run code script (for pbs or sge queued schedulers) , don't know how utilize dill.

essentially, trying utilize ipython parallel cluster splitting computation defined in custom class method.

i want pass cluster object custom class instance, utilize cluster split computation operate on pieces of info defined member.

having started cluster using ipcluster (/path/to/ipcontroller-client.json), then, want run, python test_parallel.py where, test_parallel.py is class foo(object): def __init__(self): numpy import arange self.data = arange(10)*10 def a(self, y): print "in a:", y self.data[y] def parallela(self, z, cl): print "in parallela:", cl[:].map_sync(self.a, z) def seriala(self, z): print "in seriala:", map(self.a, z) if __name__ == "__main__": ipython.parallel import client f = '/path/to/security/ipcontroller-client.json' c = client(f) asdf = foo() asdf.seriala([1, 3, 5]) ## works asdf.parallela([1, 3, 5], c) ## doesn't work

the output is

$ ~/projects/parcellation$ python test_parallel.py in seriala: in a: 1 in a: 3 in a: 5 [none, none, none] in parallela: traceback (most recent phone call last): file "test_parallel.py", line 24, in <module> asdf.parallela([1, 3, 5], c) ## doesn't work file "test_parallel.py", line 11, in parallela print "in parallela:", cl[:].map_sync(self.a, z) file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/view.py", line 366, in map_sync homecoming self.map(f,*sequences,**kwargs) file "<string>", line 2, in map file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/view.py", line 66, in sync_results ret = f(self, *args, **kwargs) file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/view.py", line 624, in map homecoming pf.map(*sequences) file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/remotefunction.py", line 271, in map ret = self(*sequences) file "<string>", line 2, in __call__ file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/remotefunction.py", line 78, in sync_view_results homecoming f(self, *args, **kwargs) file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/remotefunction.py", line 243, in __call__ ar = view.apply(f, *args) file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/view.py", line 233, in apply homecoming self._really_apply(f, args, kwargs) file "<string>", line 2, in _really_apply file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/view.py", line 66, in sync_results ret = f(self, *args, **kwargs) file "<string>", line 2, in _really_apply file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/view.py", line 51, in save_ids ret = f(self, *args, **kwargs) file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/view.py", line 567, in _really_apply ident=ident) file "/usr/local/lib/python2.7/dist-packages/ipython/parallel/client/client.py", line 1263, in send_apply_request item_threshold=self.session.item_threshold, file "/usr/local/lib/python2.7/dist-packages/ipython/kernel/zmq/serialize.py", line 145, in pack_apply_message arg_bufs = flatten(serialize_object(arg, buffer_threshold, item_threshold) arg in args) file "/usr/local/lib/python2.7/dist-packages/ipython/utils/data.py", line 30, in flatten homecoming [x subseq in seq x in subseq] file "/usr/local/lib/python2.7/dist-packages/ipython/kernel/zmq/serialize.py", line 145, in <genexpr> arg_bufs = flatten(serialize_object(arg, buffer_threshold, item_threshold) arg in args) file "/usr/local/lib/python2.7/dist-packages/ipython/kernel/zmq/serialize.py", line 89, in serialize_object buffers.insert(0, pickle.dumps(cobj, pickle_protocol)) cpickle.picklingerror: can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

any help in understanding why not work, , prepare requires minimal code alter helpful.

thank you!

i figured out solution:

class foo(object): def __init__(self): numpy import arange self.data = arange(10)*10 @staticmethod def a(data, y): print "in a:", y ## doesn't produce output homecoming data[y] def parallela(self, z, cl): print "in parallela:", cl[:].map_sync(self.a, [self.data]*len(z), z) if __name__ == "__main__": ipython.parallel import client f = '/path/to/security/ipcontroller-client.json' c = client(f) asdf = foo() asdf.parallela([1, 3, 5], c)

output when above code run:

$ python test_parallel.py in parallela: [10, 30, 50]

python parallel-processing batch-processing ipython-parallel