Tuesday, 15 January 2013

hadoop - MRJOB reducer gives no output on EMR but provides output when run in local machine -



hadoop - MRJOB reducer gives no output on EMR but provides output when run in local machine -

when execute mapreduce job on local setup desired output reducer while same code on emr not produce any. have cluster setup of 1 master , 10 core.

this output. there no error displayed

map-reduce framework map input records=3000 map output records=378 map output bytes=36054 map output materialized bytes=40448 input split bytes=1420 combine input records=0 combine output records=0 cut down input groups=179 cut down shuffle bytes=40448 cut down input records=378 cut down output records=0 spilled records=756 shuffled maps =380 failed shuffles=0 merged map outputs=380 gc time elapsed (ms)=23484 cpu time spent (ms)=125780 physical memory (bytes) snapshot=9989242880 virtual memory (bytes) snapshot=52768247808 total committed heap usage (bytes)=6517702656 shuffle errors bad_id=0 connection=0 io_error=0 wrong_length=0 wrong_map=0 wrong_reduce=0 file input format counters bytes read=711180681 file output format counters bytes written=0

following reducer code:

def reducer(self, key, val): best = -60 best_name = none lat = 0 longi = 0 yr = 0 genre = none hot, name,lat,longi,yr,genre in val: if hot > best: best = hot best_name = name lat = lat longi = longi yr = yr genre = genre yield (key,(best,best_name,lat,longi,yr,genre))

hadoop emr mrjob

No comments:

Post a Comment