source: proiecte/HadoopJUnit/hadoop-0.20.1/docs/streaming.html @ 142

Last change on this file since 142 was 120, checked in by (none), 14 years ago

Added the mail files for the Hadoop JUNit Project
  • Property svn:executable set to *
File size: 40.4 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
2<html>
3<head>
4<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
5<meta content="Apache Forrest" name="Generator">
6<meta name="Forrest-version" content="0.8">
7<meta name="Forrest-skin-name" content="pelt">
8<meta name="http-equiv" content="Content-Type">
9<meta name="content" content="text/html;">
10<meta name="charset" content="utf-8">
11<title>Hadoop Streaming</title>
12<link type="text/css" href="skin/basic.css" rel="stylesheet">
13<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
14<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
15<link type="text/css" href="skin/profile.css" rel="stylesheet">
16<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
17<link rel="shortcut icon" href="images/favicon.ico">
18</head>
19<body onload="init()">
20<script type="text/javascript">ndeSetTextSize();</script>
21<div id="top">
22<!--+
23    |breadtrail
24    +-->
25<div class="breadtrail">
26<a href="http://www.apache.org/">Apache</a> &gt; <a href="http://hadoop.apache.org/">Hadoop</a> &gt; <a href="http://hadoop.apache.org/core/">Core</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
27</div>
28<!--+
29    |header
30    +-->
31<div class="header">
32<!--+
33    |start group logo
34    +-->
35<div class="grouplogo">
36<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
37</div>
38<!--+
39    |end group logo
40    +-->
41<!--+
42    |start Project Logo
43    +-->
44<div class="projectlogo">
45<a href="http://hadoop.apache.org/core/"><img class="logoImage" alt="Hadoop" src="images/core-logo.gif" title="Scalable Computing Platform"></a>
46</div>
47<!--+
48    |end Project Logo
49    +-->
50<!--+
51    |start Search
52    +-->
53<div class="searchbox">
54<form action="http://www.google.com/search" method="get" class="roundtopsmall">
55<input value="hadoop.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp; 
56                    <input name="Search" value="Search" type="submit">
57</form>
58</div>
59<!--+
60    |end search
61    +-->
62<!--+
63    |start Tabs
64    +-->
65<ul id="tabs">
66<li>
67<a class="unselected" href="http://hadoop.apache.org/core/">Project</a>
68</li>
69<li>
70<a class="unselected" href="http://wiki.apache.org/hadoop">Wiki</a>
71</li>
72<li class="current">
73<a class="selected" href="index.html">Hadoop 0.20 Documentation</a>
74</li>
75</ul>
76<!--+
77    |end Tabs
78    +-->
79</div>
80</div>
81<div id="main">
82<div id="publishedStrip">
83<!--+
84    |start Subtabs
85    +-->
86<div id="level2tabs"></div>
87<!--+
88    |end Endtabs
89    +-->
90<script type="text/javascript"><!--
91document.write("Last Published: " + document.lastModified);
92//  --></script>
93</div>
94<!--+
95    |breadtrail
96    +-->
97<div class="breadtrail">
98
99             &nbsp;
100           </div>
101<!--+
102    |start Menu, mainarea
103    +-->
104<!--+
105    |start Menu
106    +-->
107<div id="menu">
108<div onclick="SwitchMenu('menu_1.1', 'skin/')" id="menu_1.1Title" class="menutitle">Getting Started</div>
109<div id="menu_1.1" class="menuitemgroup">
110<div class="menuitem">
111<a href="index.html">Overview</a>
112</div>
113<div class="menuitem">
114<a href="quickstart.html">Quick Start</a>
115</div>
116<div class="menuitem">
117<a href="cluster_setup.html">Cluster Setup</a>
118</div>
119<div class="menuitem">
120<a href="mapred_tutorial.html">Map/Reduce Tutorial</a>
121</div>
122</div>
123<div onclick="SwitchMenu('menu_selected_1.2', 'skin/')" id="menu_selected_1.2Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Programming Guides</div>
124<div id="menu_selected_1.2" class="selectedmenuitemgroup" style="display: block;">
125<div class="menuitem">
126<a href="commands_manual.html">Commands</a>
127</div>
128<div class="menuitem">
129<a href="distcp.html">DistCp</a>
130</div>
131<div class="menuitem">
132<a href="native_libraries.html">Native Libraries</a>
133</div>
134<div class="menupage">
135<div class="menupagetitle">Streaming</div>
136</div>
137<div class="menuitem">
138<a href="fair_scheduler.html">Fair Scheduler</a>
139</div>
140<div class="menuitem">
141<a href="capacity_scheduler.html">Capacity Scheduler</a>
142</div>
143<div class="menuitem">
144<a href="service_level_auth.html">Service Level Authorization</a>
145</div>
146<div class="menuitem">
147<a href="vaidya.html">Vaidya</a>
148</div>
149<div class="menuitem">
150<a href="hadoop_archives.html">Archives</a>
151</div>
152</div>
153<div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">HDFS</div>
154<div id="menu_1.3" class="menuitemgroup">
155<div class="menuitem">
156<a href="hdfs_user_guide.html">User Guide</a>
157</div>
158<div class="menuitem">
159<a href="hdfs_design.html">Architecture</a>
160</div>
161<div class="menuitem">
162<a href="hdfs_shell.html">File System Shell Guide</a>
163</div>
164<div class="menuitem">
165<a href="hdfs_permissions_guide.html">Permissions Guide</a>
166</div>
167<div class="menuitem">
168<a href="hdfs_quota_admin_guide.html">Quotas Guide</a>
169</div>
170<div class="menuitem">
171<a href="SLG_user_guide.html">Synthetic Load Generator Guide</a>
172</div>
173<div class="menuitem">
174<a href="libhdfs.html">C API libhdfs</a>
175</div>
176</div>
177<div onclick="SwitchMenu('menu_1.4', 'skin/')" id="menu_1.4Title" class="menutitle">HOD</div>
178<div id="menu_1.4" class="menuitemgroup">
179<div class="menuitem">
180<a href="hod_user_guide.html">User Guide</a>
181</div>
182<div class="menuitem">
183<a href="hod_admin_guide.html">Admin Guide</a>
184</div>
185<div class="menuitem">
186<a href="hod_config_guide.html">Config Guide</a>
187</div>
188</div>
189<div onclick="SwitchMenu('menu_1.5', 'skin/')" id="menu_1.5Title" class="menutitle">Miscellaneous</div>
190<div id="menu_1.5" class="menuitemgroup">
191<div class="menuitem">
192<a href="api/index.html">API Docs</a>
193</div>
194<div class="menuitem">
195<a href="jdiff/changes.html">API Changes</a>
196</div>
197<div class="menuitem">
198<a href="http://wiki.apache.org/hadoop/">Wiki</a>
199</div>
200<div class="menuitem">
201<a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a>
202</div>
203<div class="menuitem">
204<a href="releasenotes.html">Release Notes</a>
205</div>
206<div class="menuitem">
207<a href="changes.html">Change Log</a>
208</div>
209</div>
210<div id="credit"></div>
211<div id="roundbottom">
212<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
213<!--+
214  |alternative credits
215  +-->
216<div id="credit2"></div>
217</div>
218<!--+
219    |end Menu
220    +-->
221<!--+
222    |start content
223    +-->
224<div id="content">
225<div title="Portable Document Format" class="pdflink">
226<a class="dida" href="streaming.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
227        PDF</a>
228</div>
229<h1>Hadoop Streaming</h1>
230<div id="minitoc-area">
231<ul class="minitoc">
232<li>
233<a href="#Hadoop+Streaming">Hadoop Streaming</a>
234</li>
235<li>
236<a href="#How+Does+Streaming+Work">How Does Streaming Work </a>
237</li>
238<li>
239<a href="#Package+Files+With+Job+Submissions">Package Files With Job Submissions</a>
240</li>
241<li>
242<a href="#Streaming+Options+and+Usage">Streaming Options and Usage </a>
243<ul class="minitoc">
244<li>
245<a href="#Mapper-Only+Jobs">Mapper-Only Jobs </a>
246</li>
247<li>
248<a href="#Specifying+Other+Plugins+for+Jobs">Specifying Other Plugins for Jobs </a>
249</li>
250<li>
251<a href="#Large+files+and+archives+in+Hadoop+Streaming">Large files and archives in Hadoop Streaming </a>
252</li>
253<li>
254<a href="#Specifying+Additional+Configuration+Variables+for+Jobs">Specifying Additional Configuration Variables for Jobs </a>
255</li>
256<li>
257<a href="#Other+Supported+Options">Other Supported Options </a>
258</li>
259</ul>
260</li>
261<li>
262<a href="#More+usage+examples">More usage examples </a>
263<ul class="minitoc">
264<li>
265<a href="#Customizing+the+Way+to+Split+Lines+into+Key%2FValue+Pairs">Customizing the Way to Split Lines into Key/Value Pairs </a>
266</li>
267<li>
268<a href="#A+Useful+Partitioner+Class+%28secondary+sort%2C+the+-partitioner+org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner+option%29">A Useful Partitioner Class (secondary sort, the -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner option) </a>
269</li>
270<li>
271<a href="#A+Useful+Comparator+Class">A Useful Comparator Class</a>
272</li>
273<li>
274<a href="#Working+with+the+Hadoop+Aggregate+Package+%28the+-reduce+aggregate+option%29">Working with the Hadoop Aggregate Package (the -reduce aggregate option) </a>
275</li>
276<li>
277<a href="#Field+Selection+%28+similar+to+unix+%27cut%27+command%29">Field Selection ( similar to unix 'cut' command) </a>
278</li>
279</ul>
280</li>
281<li>
282<a href="#Frequently+Asked+Questions">Frequently Asked Questions </a>
283<ul class="minitoc">
284<li>
285<a href="#How+do+I+use+Hadoop+Streaming+to+run+an+arbitrary+set+of+%28semi-%29independent+tasks%3F">How do I use Hadoop Streaming to run an arbitrary set of (semi-)independent tasks? </a>
286</li>
287<li>
288<a href="#How+do+I+process+files%2C+one+per+map%3F">How do I process files, one per map? </a>
289</li>
290<li>
291<a href="#How+many+reducers+should+I+use%3F">How many reducers should I use? </a>
292</li>
293<li>
294<a href="#If+I+set+up+an+alias+in+my+shell+script%2C+will+that+work+after+-mapper%2C+i.e.+say+I+do%3A+alias+c1%3D%27cut+-f1%27.+Will+-mapper+%22c1%22+work%3F">If I set up an alias in my shell script, will that work after -mapper, i.e. say I do: alias c1='cut -f1'. Will -mapper "c1" work? </a>
295</li>
296<li>
297<a href="#Can+I+use+UNIX+pipes%3F+For+example%2C+will+-mapper+%22cut+-f1+%7C+sed+s%2Ffoo%2Fbar%2Fg%22+work%3F">Can I use UNIX pipes? For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?</a>
298</li>
299<li>
300<a href="#When+I+run+a+streaming+job+by">When I run a streaming job by distributing large executables (for example, 3.6G) through the -file option, I get a "No space left on device" error. What do I do? </a>
301</li>
302<li>
303<a href="#How+do+I+specify+multiple+input+directories%3F">How do I specify multiple input directories? </a>
304</li>
305<li>
306<a href="#How+do+I+generate+output+files+with+gzip+format%3F">How do I generate output files with gzip format? </a>
307</li>
308<li>
309<a href="#How+do+I+provide+my+own+input%2Foutput+format+with+streaming%3F">How do I provide my own input/output format with streaming? </a>
310</li>
311<li>
312<a href="#How+do+I+parse+XML+documents+using+streaming%3F">How do I parse XML documents using streaming? </a>
313</li>
314<li>
315<a href="#How+do+I+update+counters+in+streaming+applications%3F">How do I update counters in streaming applications? </a>
316</li>
317<li>
318<a href="#How+do+I+update+status+in+streaming+applications%3F">How do I update status in streaming applications? </a>
319</li>
320</ul>
321</li>
322</ul>
323</div>
324
325<a name="N10019"></a><a name="Hadoop+Streaming"></a>
326<h2 class="h3">Hadoop Streaming</h2>
327<div class="section">
328<p>
329Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example:
330</p>
331<pre class="code">
332$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
333    -input myInputDirs \
334    -output myOutputDir \
335    -mapper /bin/cat \
336    -reducer /bin/wc
337</pre>
338</div>
339
340
341<a name="N10027"></a><a name="How+Does+Streaming+Work"></a>
342<h2 class="h3">How Does Streaming Work </h2>
343<div class="section">
344<p>
345In the above example, both the mapper and the reducer are executables that read the input from stdin (line by line) and emit the output to stdout. The utility will create a Map/Reduce job, submit the job to an appropriate cluster, and monitor the progress of the job until it completes.
346</p>
347<p>
348  When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper. By default, the
349  <em>prefix of a line up to the first tab character</em> is the <strong>key</strong> and the rest of the line (excluding the tab character) will be the <strong>value</strong>.
350  If there is no tab character in the line, then entire line is considered as key and the value is null. However, this can be customized, as discussed later.
351</p>
352<p>
353When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value. However, this can be customized, as discussed later.
354</p>
355<p>
356This is the basis for the communication protocol between the Map/Reduce framework and the streaming mapper/reducer.
357</p>
358<p>
359You can supply a Java class as the mapper and/or the reducer. The above example is equivalent to:
360</p>
361<pre class="code">
362$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
363    -input myInputDirs \
364    -output myOutputDir \
365    -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
366    -reducer /bin/wc
367</pre>
368<p>User can specify <span class="codefrag">stream.non.zero.exit.is.failure</span> as
369<span class="codefrag">true</span> or <span class="codefrag">false</span> to make a streaming task that exits
370with a non-zero status to be <span class="codefrag">Failure</span> 
371or <span class="codefrag">Success</span> respectively. By default, streaming tasks exiting
372with non-zero status are considered to be failed tasks.</p>
373</div>
374
375
376<a name="N10059"></a><a name="Package+Files+With+Job+Submissions"></a>
377<h2 class="h3">Package Files With Job Submissions</h2>
378<div class="section">
379<p>
380You can specify any executable as the mapper and/or the reducer. The executables do not need to pre-exist on the machines in the cluster; however, if they don't, you will need to use "-file" option to tell the framework to pack your executable files as a part of job submission. For example:
381</p>
382<pre class="code">
383$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
384    -input myInputDirs \
385    -output myOutputDir \
386    -mapper myPythonScript.py \
387    -reducer /bin/wc \
388    -file myPythonScript.py
389</pre>
390<p>
391The above example specifies a user defined Python executable as the mapper. The option "-file myPythonScript.py" causes the python executable shipped to the cluster machines as a part of job submission.
392</p>
393<p>
394In addition to executable files, you can also package other auxiliary files (such as dictionaries, configuration files, etc) that may be used by the mapper and/or the reducer. For example:
395</p>
396<pre class="code">
397$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
398    -input myInputDirs \
399    -output myOutputDir \
400    -mapper myPythonScript.py \
401    -reducer /bin/wc \
402    -file myPythonScript.py \
403    -file myDictionary.txt
404</pre>
405</div>
406
407
408<a name="N10071"></a><a name="Streaming+Options+and+Usage"></a>
409<h2 class="h3">Streaming Options and Usage </h2>
410<div class="section">
411<a name="N10077"></a><a name="Mapper-Only+Jobs"></a>
412<h3 class="h4">Mapper-Only Jobs </h3>
413<p>
414Often, you may want to process input data using a map function only. To do this, simply set mapred.reduce.tasks to zero. The Map/Reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job.
415</p>
416<p>
417To be backward compatible, Hadoop Streaming also supports the "-reduce NONE" option, which is equivalent to "-D mapred.reduce.tasks=0".
418</p>
419<a name="N10083"></a><a name="Specifying+Other+Plugins+for+Jobs"></a>
420<h3 class="h4">Specifying Other Plugins for Jobs </h3>
421<p>
422Just as with a normal Map/Reduce job, you can specify other plugins for a streaming job:
423</p>
424<pre class="code">
425   -inputformat JavaClassName
426   -outputformat JavaClassName
427   -partitioner JavaClassName
428   -combiner JavaClassName
429</pre>
430<p>
431The class you supply for the input format should return key/value pairs of Text class. If you do not specify an input format class, the TextInputFormat is used as the default. Since the TextInputFormat returns keys of LongWritable class, which are actually not part of the input data, the keys will be discarded; only the values will be piped to the streaming mapper.
432</p>
433<p>
434The class you supply for the output format is expected to take key/value pairs of Text class. If you do not specify an output format class, the TextOutputFormat is used as the default.
435</p>
436<a name="N10096"></a><a name="Large+files+and+archives+in+Hadoop+Streaming"></a>
437<h3 class="h4">Large files and archives in Hadoop Streaming </h3>
438<p>
439The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can retrieve the host and fs_port values from the fs.default.name config variable.
440</p>
441<p>
442Here are examples of the -files option:
443</p>
444<pre class="code">
445-files hdfs://host:fs_port/user/testfile.txt#testlink
446</pre>
447<p>
448In the above example, the part of the url after # is used as the symlink name that is created in the current working directory of tasks. So the tasks will have a symlink called testlink in the cwd that points to a local copy of testfile.txt. Multiple entries can be specified as:
449</p>
450<pre class="code">
451-files hdfs://host:fs_port/user/testfile1.txt#testlink1 -files hdfs://host:fs_port/user/testfile2.txt#testlink2
452</pre>
453<p>
454The -archives option allows you to copy jars locally to the cwd of tasks and automatically unjar the files. For example:
455</p>
456<pre class="code">
457-archives hdfs://host:fs_port/user/testfile.jar#testlink3
458</pre>
459<p>
460In the example above, a symlink testlink3 is created in the current working directory of tasks. This symlink points to the directory that stores the unjarred contents of the uploaded jar file.
461</p>
462<p>
463Here's another example of the -archives option. Here, the input.txt file has two lines specifying the names of the two files: testlink/cache.txt and testlink/cache2.txt. "testlink" is a symlink to the archived directory, which has the files "cache.txt" and "cache2.txt".
464</p>
465<pre class="code">
466$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
467                  -input "/user/me/samples/cachefile/input.txt"  \
468                  -mapper "xargs cat"  \
469                  -reducer "cat"  \
470                  -output "/user/me/samples/cachefile/out" \ 
471                  -archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar#testlink' \ 
472                  -D mapred.map.tasks=1 \
473                  -D mapred.reduce.tasks=1 \
474                  -D mapred.job.name="Experiment"
475
476$ ls test_jar/
477cache.txt  cache2.txt
478
479$ jar cvf cachedir.jar -C test_jar/ .
480added manifest
481adding: cache.txt(in = 30) (out= 29)(deflated 3%)
482adding: cache2.txt(in = 37) (out= 35)(deflated 5%)
483
484$ hadoop dfs -put cachedir.jar samples/cachefile
485
486$ hadoop dfs -cat /user/me/samples/cachefile/input.txt
487testlink/cache.txt
488testlink/cache2.txt
489
490$ cat test_jar/cache.txt
491This is just the cache string
492
493$ cat test_jar/cache2.txt
494This is just the second cache string
495
496$ hadoop dfs -ls /user/me/samples/cachefile/out     
497Found 1 items
498/user/me/samples/cachefile/out/part-00000  &lt;r 3&gt;   69
499
500$ hadoop dfs -cat /user/me/samples/cachefile/out/part-00000
501This is just the cache string   
502This is just the second cache string
503
504</pre>
505<a name="N100BF"></a><a name="Specifying+Additional+Configuration+Variables+for+Jobs"></a>
506<h3 class="h4">Specifying Additional Configuration Variables for Jobs </h3>
507<p>
508You can specify additional configuration variables by using "-D  &lt;n&gt;=&lt;v&gt;". For example:
509</p>
510<pre class="code">
511$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
512    -input myInputDirs \
513    -output myOutputDir \
514    -mapper org.apache.hadoop.mapred.lib.IdentityMapper\
515    -reducer /bin/wc \
516    -D mapred.reduce.tasks=2
517</pre>
518<p>
519The -D mapred.reduce.tasks=2 in the above example specifies to use two reducers for the job.
520</p>
521<p>
522For more details on the jobconf parameters see:
523<a href="http://hadoop.apache.org/core/docs/current/mapred-default.html">mapred-default.html</a>
524</p>
525<a name="N100D6"></a><a name="Other+Supported+Options"></a>
526<h3 class="h4">Other Supported Options </h3>
527<p>
528Other options you may specify for a streaming job are described here:
529</p>
530<table class="ForrestTable" cellspacing="1" cellpadding="4">
531
532<tr>
533<th colspan="1" rowspan="1">Parameter</th><th colspan="1" rowspan="1">Optional/Required </th><th colspan="1" rowspan="1">Description </th>
534</tr>
535
536
537<tr>
538<td colspan="1" rowspan="1"> -cmdenv   name=value </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Pass env var to streaming commands </td>
539</tr>
540
541
542<tr>
543<td colspan="1" rowspan="1"> -inputreader JavaClassName </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> For backwards-compatibility: specifies a record reader class (instead of an input format class) </td>
544</tr>
545
546<tr>
547<td colspan="1" rowspan="1"> -verbose </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Verbose output </td>
548</tr>
549
550</table>
551<p>
552Streaming support Hadoop generic command line options.
553
554Supported parameters are :
555The general command line syntax is :
556<br>    bin/hadoop command [genericOptions] [commandOptions]
557</p>
558<table class="ForrestTable" cellspacing="1" cellpadding="4">
559
560<tr>
561<th colspan="1" rowspan="1">Parameter</th><th colspan="1" rowspan="1">Optional/Required </th><th colspan="1" rowspan="1">Description </th>
562</tr>
563
564
565<tr>
566<td colspan="1" rowspan="1"> -conf  configuration_file </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify an application configuration file </td>
567</tr>
568
569<tr>
570<td colspan="1" rowspan="1"> -D  property=value </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> use value for given property </td>
571</tr>
572
573<tr>
574<td colspan="1" rowspan="1"> -fs host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify a namenode </td>
575</tr>
576
577<tr>
578<td colspan="1" rowspan="1"> -jt host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify a job tracker </td>
579</tr>
580
581<tr>
582<td colspan="1" rowspan="1"> -files </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify comma separated files to be copied to the map reduce cluster </td>
583</tr>
584
585<tr>
586<td colspan="1" rowspan="1"> -archives </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify comma separated archives to be unarchived on the compute machines </td>
587</tr>
588
589<tr>
590<td colspan="1" rowspan="1">  </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1">  </td>
591</tr>
592
593<tr>
594<td colspan="1" rowspan="1"> -jt host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1">  </td>
595</tr>
596
597</table>
598<p>
599To change the local temp directory use:
600</p>
601<pre class="code">
602  -D dfs.data.dir=/tmp
603</pre>
604<p>
605To specify additional local temp directories use:
606</p>
607<pre class="code">
608   -D mapred.local.dir=/tmp/local
609   -D mapred.system.dir=/tmp/system
610   -D mapred.temp.dir=/tmp/temp
611</pre>
612<p>
613For more details on jobconf parameters see:
614<a href="http://hadoop.apache.org/core/docs/current/mapred-default.html">mapred-default.html</a>
615</p>
616<p>
617To set an environment variable in a streaming command use:
618</p>
619<pre class="code">
620-cmdenv EXAMPLE_DIR=/home/example/dictionaries/
621</pre>
622</div>
623
624
625<a name="N101BD"></a><a name="More+usage+examples"></a>
626<h2 class="h3">More usage examples </h2>
627<div class="section">
628<a name="N101C3"></a><a name="Customizing+the+Way+to+Split+Lines+into+Key%2FValue+Pairs"></a>
629<h3 class="h4">Customizing the Way to Split Lines into Key/Value Pairs </h3>
630<p>
631As noted earlier, when the Map/Reduce framework reads a line from the stdout of the mapper, it splits the line into a key/value pair. By default, the prefix of the line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value.
632</p>
633<p>
634However, you can customize this default. You can specify a field separator other than the tab character (the default), and you can specify the nth (n &gt;= 1) character rather than the first character in a line (the default) as the separator between the key and value. For example:
635</p>
636<pre class="code">
637$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
638    -input myInputDirs \
639    -output myOutputDir \
640    -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
641    -reducer org.apache.hadoop.mapred.lib.IdentityReducer \
642    -D stream.map.output.field.separator=. \
643    -D stream.num.map.output.key.fields=4
644</pre>
645<p>
646In the above example, "-D stream.map.output.field.separator=." specifies "." as the field separator for the map outputs, and the prefix up to the fourth "." in a line will be the key and the rest of the line (excluding the fourth ".") will be the value. If a line has less than four "."s, then the whole line will be the key and the value will be an empty Text object (like the one created by new Text("")).
647</p>
648<p>
649Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and "-D stream.num.reduce.output.fields=NUM" to specify the nth field separator in a line of the reduce outputs as the separator between the key and the value.
650</p>
651<p> Similarly, you can specify "stream.map.input.field.separator" and
652"stream.reduce.input.field.separator" as the input separator for map/reduce
653inputs. By default the separator is the tab character.</p>
654<a name="N101DC"></a><a name="A+Useful+Partitioner+Class+%28secondary+sort%2C+the+-partitioner+org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner+option%29"></a>
655<h3 class="h4">A Useful Partitioner Class (secondary sort, the -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner option) </h3>
656<p>
657Hadoop has a library class,
658<a href="api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html">KeyFieldBasedPartitioner</a>,
659that is useful for many applications. This class allows the Map/Reduce
660framework to partition the map outputs based on certain key fields, not
661the whole keys. For example:
662</p>
663<pre class="code">
664$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
665    -input myInputDirs \
666    -output myOutputDir \
667    -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
668    -reducer org.apache.hadoop.mapred.lib.IdentityReducer \
669    -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
670    -D stream.map.output.field.separator=. \
671    -D stream.num.map.output.key.fields=4 \
672    -D map.output.key.field.separator=. \
673    -D mapred.text.key.partitioner.options=-k1,2\
674    -D mapred.reduce.tasks=12
675</pre>
676<p>
677Here, <em>-D stream.map.output.field.separator=.</em> and <em>-D stream.num.map.output.key.fields=4</em> are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper.
678</p>
679<p>
680The map output keys of the above Map/Reduce job normally have four fields
681separated by ".". However, the Map/Reduce framework will partition the map
682outputs by the first two fields of the keys using the
683<em>-D mapred.text.key.partitioner.options=-k1,2</em> option.
684Here, <em>-D map.output.key.field.separator=.</em> specifies the separator
685for the partition. This guarantees that all the key/value pairs with the
686same first two fields in the keys will be partitioned into the same reducer.
687</p>
688<p>
689
690<em>This is effectively equivalent to specifying the first two fields as the primary key and the next two fields as the secondary. The primary key is used for partitioning, and the combination of the primary and secondary keys is used for sorting.</em> A simple illustration is shown here:
691</p>
692<p>
693Output of map (the keys)</p>
694<pre class="code">
69511.12.1.2
69611.14.2.3
69711.11.4.1
69811.12.1.1
69911.14.2.2
700
701</pre>
702<p>
703Partition into 3 reducers (the first 2 fields are used as keys for partition)</p>
704<pre class="code">
70511.11.4.1
706-----------
70711.12.1.2
70811.12.1.1
709-----------
71011.14.2.3
71111.14.2.2
712</pre>
713<p>
714Sorting within each partition for the reducer(all 4 fields used for sorting)</p>
715<pre class="code">
71611.11.4.1
717-----------
71811.12.1.1
71911.12.1.2
720-----------
72111.14.2.2
72211.14.2.3
723</pre>
724<a name="N10216"></a><a name="A+Useful+Comparator+Class"></a>
725<h3 class="h4">A Useful Comparator Class</h3>
726<p>
727Hadoop has a library class,
728<a href="api/org/apache/hadoop/mapred/lib/KeyFieldBasedComparator.html">KeyFieldBasedComparator</a>,
729that is useful for many applications. This class provides a subset of features
730provided by the Unix/GNU Sort. For example:
731</p>
732<pre class="code">
733$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
734    -input myInputDirs \
735    -output myOutputDir \
736    -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
737    -reducer org.apache.hadoop.mapred.lib.IdentityReducer \
738    -D mapred.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator \
739    -D stream.map.output.field.separator=. \
740    -D stream.num.map.output.key.fields=4 \
741    -D map.output.key.field.separator=. \
742    -D mapred.text.key.comparator.options=-k2,2nr\
743    -D mapred.reduce.tasks=12
744</pre>
745<p>
746The map output keys of the above Map/Reduce job normally have four fields
747separated by ".". However, the Map/Reduce framework will sort the
748outputs by the second field of the keys using the
749<em>-D mapred.text.key.comparator.options=-k2,2nr</em> option.
750Here, <em>-n</em> specifies that the sorting is numerical sorting and
751<em>-r</em> specifies that the result should be reversed. A simple illustration
752is shown below:
753</p>
754<p>
755Output of map (the keys)</p>
756<pre class="code">
75711.12.1.2
75811.14.2.3
75911.11.4.1
76011.12.1.1
76111.14.2.2
762</pre>
763<p>
764Sorting output for the reducer(where second field used for sorting)</p>
765<pre class="code">
76611.14.2.3
76711.14.2.2
76811.12.1.2
76911.12.1.1
77011.11.4.1
771</pre>
772<a name="N10242"></a><a name="Working+with+the+Hadoop+Aggregate+Package+%28the+-reduce+aggregate+option%29"></a>
773<h3 class="h4">Working with the Hadoop Aggregate Package (the -reduce aggregate option) </h3>
774<p>
775Hadoop has a library package called
776<a href="api/org/apache/hadoop/mapred/lib/aggregate/package-summary.html">Aggregate</a>.
777Aggregate provides a special reducer class and a special combiner class, and
778a list of simple aggregators that perform aggregations such as "sum", "max",
779"min" and so on  over a sequence of values. Aggregate allows you to define a
780mapper plugin class that is expected to generate "aggregatable items" for each
781input key/value pair of the mappers. The combiner/reducer will aggregate those
782aggregatable items by invoking the appropriate aggregators.
783</p>
784<p>
785To use Aggregate, simply specify "-reducer aggregate":
786</p>
787<pre class="code">
788$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
789    -input myInputDirs \
790    -output myOutputDir \
791    -mapper myAggregatorForKeyCount.py \
792    -reducer aggregate \
793    -file myAggregatorForKeyCount.py \
794    -D mapred.reduce.tasks=12
795</pre>
796<p>
797The python program myAggregatorForKeyCount.py looks like:
798</p>
799<pre class="code">
800#!/usr/bin/python
801
802import sys;
803
804def generateLongCountToken(id):
805    return "LongValueSum:" + id + "\t" + "1"
806
807def main(argv):
808    line = sys.stdin.readline();
809    try:
810        while line:
811            line = line[:-1];
812            fields = line.split("\t");
813            print generateLongCountToken(fields[0]);
814            line = sys.stdin.readline();
815    except "end of file":
816        return None
817if __name__ == "__main__":
818     main(sys.argv)
819</pre>
820<a name="N1025D"></a><a name="Field+Selection+%28+similar+to+unix+%27cut%27+command%29"></a>
821<h3 class="h4">Field Selection ( similar to unix 'cut' command) </h3>
822<p>
823Hadoop has a library class, org.apache.hadoop.mapred.lib.FieldSelectionMapReduce, that effectively allows you to process text data like the unix "cut" utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example:
824</p>
825<pre class="code">
826$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
827    -input myInputDirs \
828    -output myOutputDir \
829    -mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce\
830    -reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce\
831    -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
832    -D map.output.key.field.separa=. \
833    -D mapred.text.key.partitioner.options=-k1,2 \
834    -D mapred.data.field.separator=. \
835    -D map.output.key.value.fields.spec=6,5,1-3:0- \
836    -D reduce.output.key.value.fields.spec=0-2:5- \
837    -D mapred.reduce.tasks=12
838</pre>
839<p>
840The option "-D map.output.key.value.fields.spec=6,5,1-3:0-" specifies key/value selection for the map outputs. Key selection spec and value selection spec are separated by ":". In this case, the map output key will consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all fields (0- means field 0 and all
841the subsequent fields).
842</p>
843<p>
844The option "-D reduce.output.key.value.fields.spec=0-2:5-" specifies
845key/value selection for the reduce outputs. In this case, the reduce
846output key will consist of fields 0, 1, 2 (corresponding to the original
847fields 6, 5, 1). The reduce output value will consist of all fields starting
848from field 5 (corresponding to all the original fields). 
849</p>
850</div>
851
852
853<a name="N10271"></a><a name="Frequently+Asked+Questions"></a>
854<h2 class="h3">Frequently Asked Questions </h2>
855<div class="section">
856<a name="N10277"></a><a name="How+do+I+use+Hadoop+Streaming+to+run+an+arbitrary+set+of+%28semi-%29independent+tasks%3F"></a>
857<h3 class="h4">How do I use Hadoop Streaming to run an arbitrary set of (semi-)independent tasks? </h3>
858<p>
859Often you do not need the full power of Map Reduce, but only need to run multiple instances of the same program - either on different parts of the data, or on the same data, but with different parameters. You can use Hadoop Streaming to do this.
860</p>
861<a name="N10281"></a><a name="How+do+I+process+files%2C+one+per+map%3F"></a>
862<h3 class="h4">How do I process files, one per map? </h3>
863<p>
864As an example, consider the problem of zipping (compressing) a set of files across the hadoop cluster. You can achieve this using either of these methods:
865</p>
866<ol>
867
868<li> Hadoop Streaming and custom mapper script:<ul>
869 
870<li> Generate a file containing the full HDFS path of the input files. Each map task would get one file name as input.</li>
871 
872<li> Create a mapper script which, given a filename, will get the file to local disk, gzip the file and put it back in the desired output directory</li>
873
874</ul>
875</li>
876
877<li>The existing Hadoop Framework:<ul>
878   
879<li>Add these commands to your main function:
880<pre class="code">
881       FileOutputFormat.setCompressOutput(conf, true);
882       FileOutputFormat.setOutputCompressorClass(conf, org.apache.hadoop.io.compress.GzipCodec.class);
883       conf.setOutputFormat(NonSplitableTextInputFormat.class);
884       conf.setNumReduceTasks(0);
885</pre>
886</li>
887   
888<li>Write your map function:
889<pre class="code">
890
891       public void map(WritableComparable key, Writable value,
892                               OutputCollector output,
893                               Reporter reporter) throws IOException {
894            output.collect((Text)value, null);
895       }
896</pre>
897</li>
898 
899<li>Note that the output filename will not be the same as the original filename</li>
900
901</ul>
902</li>
903
904</ol>
905<a name="N102AC"></a><a name="How+many+reducers+should+I+use%3F"></a>
906<h3 class="h4">How many reducers should I use? </h3>
907<p>
908See the Hadoop Wiki for details: <a href="mapred_tutorial.html#Reducer">Reducer</a>
909
910</p>
911<a name="N102BA"></a><a name="If+I+set+up+an+alias+in+my+shell+script%2C+will+that+work+after+-mapper%2C+i.e.+say+I+do%3A+alias+c1%3D%27cut+-f1%27.+Will+-mapper+%22c1%22+work%3F"></a>
912<h3 class="h4">If I set up an alias in my shell script, will that work after -mapper, i.e. say I do: alias c1='cut -f1'. Will -mapper "c1" work? </h3>
913<p>
914Using an alias will not work, but variable substitution is allowed as shown in this example:
915</p>
916<pre class="code">
917$ hadoop dfs -cat samples/student_marks
918alice   50
919bruce   70
920charlie 80
921dan     75
922
923$ c2='cut -f2'; $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
924    -input /user/me/samples/student_marks
925    -mapper \"$c2\" -reducer 'cat' 
926    -output /user/me/samples/student_out
927    -D mapred.job.name='Experiment'
928
929$ hadoop dfs -ls samples/student_out
930Found 1 items/user/me/samples/student_out/part-00000    &lt;r 3&gt;   16
931
932$ hadoop dfs -cat samples/student_out/part-00000
93350
93470
93575
93680
937</pre>
938<a name="N102C8"></a><a name="Can+I+use+UNIX+pipes%3F+For+example%2C+will+-mapper+%22cut+-f1+%7C+sed+s%2Ffoo%2Fbar%2Fg%22+work%3F"></a>
939<h3 class="h4">Can I use UNIX pipes? For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?</h3>
940<p>
941Currently this does not work and gives an "java.io.IOException: Broken pipe" error. This is probably a bug that needs to be investigated.
942</p>
943<a name="N102D2"></a><a name="When+I+run+a+streaming+job+by"></a>
944<h3 class="h4">When I run a streaming job by distributing large executables (for example, 3.6G) through the -file option, I get a "No space left on device" error. What do I do? </h3>
945<p>
946The jar packaging happens in a directory pointed to by the configuration variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the value to a directory with more space:
947</p>
948<pre class="code">
949-D stream.tmpdir=/export/bigspace/...
950</pre>
951<a name="N102E3"></a><a name="How+do+I+specify+multiple+input+directories%3F"></a>
952<h3 class="h4">How do I specify multiple input directories? </h3>
953<p>
954You can specify multiple input directories with multiple '-input' options:
955</p>
956<pre class="code">
957 hadoop jar hadoop-streaming.jar -input '/user/foo/dir1' -input '/user/foo/dir2'
958</pre>
959<a name="N102F0"></a><a name="How+do+I+generate+output+files+with+gzip+format%3F"></a>
960<h3 class="h4">How do I generate output files with gzip format? </h3>
961<p>
962Instead of plain text files, you can generate gzip files as your generated output. Pass '-D mapred.output.compress=true -D  mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCode' as option to your streaming job.
963</p>
964<a name="N102FA"></a><a name="How+do+I+provide+my+own+input%2Foutput+format+with+streaming%3F"></a>
965<h3 class="h4">How do I provide my own input/output format with streaming? </h3>
966<p>
967At least as late as version 0.14, Hadoop does not support multiple jar files. So, when specifying your own custom classes you will have to pack them along with the streaming jar and use the custom jar instead of the default hadoop streaming jar.
968</p>
969<a name="N10304"></a><a name="How+do+I+parse+XML+documents+using+streaming%3F"></a>
970<h3 class="h4">How do I parse XML documents using streaming? </h3>
971<p>
972You can use the record reader StreamXmlRecordReader to process XML documents.
973</p>
974<pre class="code">
975hadoop jar hadoop-streaming.jar -inputreader "StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING" ..... (rest of the command)
976</pre>
977<p>
978Anything found between BEGIN_STRING and END_STRING would be treated as one record for map tasks.
979</p>
980<a name="N10315"></a><a name="How+do+I+update+counters+in+streaming+applications%3F"></a>
981<h3 class="h4">How do I update counters in streaming applications? </h3>
982<p>
983A streaming process can use the stderr to emit counter information.
984<span class="codefrag">reporter:counter:&lt;group&gt;,&lt;counter&gt;,&lt;amount&gt;</span> 
985should be sent to stderr to update the counter.
986</p>
987<a name="N10322"></a><a name="How+do+I+update+status+in+streaming+applications%3F"></a>
988<h3 class="h4">How do I update status in streaming applications? </h3>
989<p>
990A streaming process can use the stderr to emit status information.
991To set a status, <span class="codefrag">reporter:status:&lt;message&gt;</span> should be sent
992to stderr.
993</p>
994</div>
995
996</div>
997<!--+
998    |end content
999    +-->
1000<div class="clearboth">&nbsp;</div>
1001</div>
1002<div id="footer">
1003<!--+
1004    |start bottomstrip
1005    +-->
1006<div class="lastmodified">
1007<script type="text/javascript"><!--
1008document.write("Last Published: " + document.lastModified);
1009//  --></script>
1010</div>
1011<div class="copyright">
1012        Copyright &copy;
1013         2008 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
1014</div>
1015<!--+
1016    |end bottomstrip
1017    +-->
1018</div>
1019</body>
1020</html>
Note: See TracBrowser for help on using the repository browser.