exception - debugging errors in python multiprocessing -

I am using pool in the multiprocessing module, Order to run the same code on different data.

It turns out that on some data my code raises an exception, but the exact line that does this is not given:

  traceback (most recent Only call in the final): File "my_wrapper_script.py", Line 366, & lt; Module & gt; Main () file "my_wrapper_script.py", line 343, main result = pool.map (process_function, folders) file "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, map returns in self. Map_async (func, iterable, chunksize) .get () file "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in self-raising._ value keyError: 'some_key'  

I know about multiprocessing.log_to_stderr () , but it seems that it is useful when issues of concurrency arise, which is not my case.

Any thoughts?

If you are using a new enough version of Python, you will actually see the real exception Print it before one example, for example, this is a sample that fails:

  Import Multi Processing Def Indoor (): Increase exception ("FAIL") def f (): Print ("HI") Internal () p = multiprocessing Pool () p.apply (f) p.close () p.join ()  

Exception in running it with Python 3.4 Is:

  multiprocessing. Pool.RemoteTraceback: "" "Trace Back (most recent call final): The file "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, worker result = (true, exception in func ("FAIL"): In the "above" * failure, the file "test.py" was the direct reason for the exception of the following exception: "test.py", line 4, "test.py" in line 4, File: Traceback (last updated call last): file "test.py ", Line 13, & lt; module & gt; P.apply (f) file" /usr/local/lib/python3.4/multiprocessing/pool.py ", line 253, in return self. Apply_async (func, args, kwds) .get () file "/ usr /local/lib/python3.4/multiprocessing/pool.py", line 599, self-raising ._value exception: unsuccessful  

If there is no alternative to using a new version, the easiest thing to do is to wrap your worker task apart from an attempt / block, which will print an exception before raising it again:

  import multiuser import traceback def (" Except: print ("Exception in worker:") traceback.print_exc () raise p = multiprocessing.Pool () p.apply (f) p. Output:  
  Hi exception in the worker: traceback (most recent call final): file "(" FAIL ") exception: FAIL traceback (as state)" test.py ", line 5, last call last): file" test.py ", line 18, & lt; module & gt; P.apply F) File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 244, in return self. Apply_async (func, args, kwds) .get () file "/ usr / local / lib / Python2.7 / multiprocessing / pool.py ", line 558, self-lifting ._value exception: unsuccessful