pickle - Serialize a python function with dependencies -

pickle - Serialize a python function with dependencies -

i have tried multiple approaches pickle python function dependencies, next many recommendations on stackoverflow, (such dill, cloudpickle, etc.) seem run fundamental issue cannot figure out.

i have main module tries pickle function imported module, sends on ssh unpickled , executed @ remote machine.

so main has:

import dill (for example) import modulea serial=dill.dumps( modulea.func ) send (serial)

on remote machine:

import dill receive serial funcremote = dill.loads( serial ) funcremote()

if functions beingness pickled , sent top level functions defined in main itself, works. when in imported module, loads function fails messages of type "module modulea not found".

it appears module name pickled along function name. not see way "fix up" pickle remove dependency, or alternately, create dummy module in receiver become recipient of unpickling.

any pointers much appreciated.

--prasanna

i'm dill author. exact thing on ssh, success. currently, dill , of other serializers pickle modules reference… pass function defined in file, have ensure relevant module installed on other machine. not believe there object serializer serializes modules straight (i.e. not reference).

having said that, dill have options serialize object dependencies. example, class instances, default in dill not serialize class instances reference… class definition can serialized , send instance. in dill, can (use new feature to) serialize file handles serializing file, instead of doing reference. again, if have case of function defined in module, out-of-luck, modules serialized reference pretty darn universally.

you might able utilize dill so, however, not pickling object, extracting source , sending source code. in pathos.pp , pyina, dill used extract the source , the dependencies of object (including functions), , pass them computer/process/etc. however, since not easy thing do, dill can utilize failover of trying extract relevant import , send instead of source code.

you can understand, hopefully, messy messy thing (as noted in 1 of dependencies of function extracting below). however, asking done in pathos bundle pass code , dependencies different machines across ssh-tunneled ports.

>>> import dill >>> >>> print dill.source.importable(dill.source.importable) dill.source import importable >>> print dill.source.importable(dill.source.importable, source=true) def _closuredsource(func, alias=''): """get source code closured objects; homecoming dict of 'name' , 'code blocks'""" #fixme: entire function messy messy hack # - pollutes global namespace # - fails if name of freevars reused # - can unnecessarily duplicate function code dill.detect import freevars free_vars = freevars(func) func_vars = {} # split 'funcs' , 'non-funcs' name,obj in list(free_vars.items()): if not isfunction(obj): # source 'non-funcs' free_vars[name] = getsource(obj, force=true, alias=name) go on # source 'funcs' #…snip… …snip… …snip… …snip… …snip… # source code of objects referred obj in global scope dill.detect import globalvars obj = globalvars(obj) #xxx: don't worry alias? obj = list(getsource(_obj,name,force=true) (name,_obj) in obj.items()) obj = '\n'.join(obj) if obj else '' # combine referred-to source (global enclosing) if not obj: homecoming src if not src: homecoming obj homecoming obj + src except: if tried_import: raise tried_source = true source = not source # should never here homecoming

i imagine built around dill.detect.parents method, provides list of pointers parent object given object… , 1 reconstruct of function's dependencies objects… not implemented.

btw: found ssh tunnel, this:

>>> t = pathos.tunnel.tunnel() >>> t.connect('login.university.edu') 39322 >>> t tunnel('-q -n -l39322:login.university.edu:45075 login.university.edu')

then can work across local port zmq, or ssh, or whatever. if want ssh, pathos has built in.

python pickle dill