source: proiecte/HadoopJUnit/hadoop-0.20.1/docs/fair_scheduler.html @ 120

Last change on this file since 120 was 120, checked in by (none), 14 years ago

Added the mail files for the Hadoop JUNit Project
  • Property svn:executable set to *
File size: 25.4 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
2<html>
3<head>
4<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
5<meta content="Apache Forrest" name="Generator">
6<meta name="Forrest-version" content="0.8">
7<meta name="Forrest-skin-name" content="pelt">
8<title>Fair Scheduler Guide</title>
9<link type="text/css" href="skin/basic.css" rel="stylesheet">
10<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
11<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
12<link type="text/css" href="skin/profile.css" rel="stylesheet">
13<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
14<link rel="shortcut icon" href="images/favicon.ico">
15</head>
16<body onload="init()">
17<script type="text/javascript">ndeSetTextSize();</script>
18<div id="top">
19<!--+
20    |breadtrail
21    +-->
22<div class="breadtrail">
23<a href="http://www.apache.org/">Apache</a> &gt; <a href="http://hadoop.apache.org/">Hadoop</a> &gt; <a href="http://hadoop.apache.org/core/">Core</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
24</div>
25<!--+
26    |header
27    +-->
28<div class="header">
29<!--+
30    |start group logo
31    +-->
32<div class="grouplogo">
33<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
34</div>
35<!--+
36    |end group logo
37    +-->
38<!--+
39    |start Project Logo
40    +-->
41<div class="projectlogo">
42<a href="http://hadoop.apache.org/core/"><img class="logoImage" alt="Hadoop" src="images/core-logo.gif" title="Scalable Computing Platform"></a>
43</div>
44<!--+
45    |end Project Logo
46    +-->
47<!--+
48    |start Search
49    +-->
50<div class="searchbox">
51<form action="http://www.google.com/search" method="get" class="roundtopsmall">
52<input value="hadoop.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp; 
53                    <input name="Search" value="Search" type="submit">
54</form>
55</div>
56<!--+
57    |end search
58    +-->
59<!--+
60    |start Tabs
61    +-->
62<ul id="tabs">
63<li>
64<a class="unselected" href="http://hadoop.apache.org/core/">Project</a>
65</li>
66<li>
67<a class="unselected" href="http://wiki.apache.org/hadoop">Wiki</a>
68</li>
69<li class="current">
70<a class="selected" href="index.html">Hadoop 0.20 Documentation</a>
71</li>
72</ul>
73<!--+
74    |end Tabs
75    +-->
76</div>
77</div>
78<div id="main">
79<div id="publishedStrip">
80<!--+
81    |start Subtabs
82    +-->
83<div id="level2tabs"></div>
84<!--+
85    |end Endtabs
86    +-->
87<script type="text/javascript"><!--
88document.write("Last Published: " + document.lastModified);
89//  --></script>
90</div>
91<!--+
92    |breadtrail
93    +-->
94<div class="breadtrail">
95
96             &nbsp;
97           </div>
98<!--+
99    |start Menu, mainarea
100    +-->
101<!--+
102    |start Menu
103    +-->
104<div id="menu">
105<div onclick="SwitchMenu('menu_1.1', 'skin/')" id="menu_1.1Title" class="menutitle">Getting Started</div>
106<div id="menu_1.1" class="menuitemgroup">
107<div class="menuitem">
108<a href="index.html">Overview</a>
109</div>
110<div class="menuitem">
111<a href="quickstart.html">Quick Start</a>
112</div>
113<div class="menuitem">
114<a href="cluster_setup.html">Cluster Setup</a>
115</div>
116<div class="menuitem">
117<a href="mapred_tutorial.html">Map/Reduce Tutorial</a>
118</div>
119</div>
120<div onclick="SwitchMenu('menu_selected_1.2', 'skin/')" id="menu_selected_1.2Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Programming Guides</div>
121<div id="menu_selected_1.2" class="selectedmenuitemgroup" style="display: block;">
122<div class="menuitem">
123<a href="commands_manual.html">Commands</a>
124</div>
125<div class="menuitem">
126<a href="distcp.html">DistCp</a>
127</div>
128<div class="menuitem">
129<a href="native_libraries.html">Native Libraries</a>
130</div>
131<div class="menuitem">
132<a href="streaming.html">Streaming</a>
133</div>
134<div class="menupage">
135<div class="menupagetitle">Fair Scheduler</div>
136</div>
137<div class="menuitem">
138<a href="capacity_scheduler.html">Capacity Scheduler</a>
139</div>
140<div class="menuitem">
141<a href="service_level_auth.html">Service Level Authorization</a>
142</div>
143<div class="menuitem">
144<a href="vaidya.html">Vaidya</a>
145</div>
146<div class="menuitem">
147<a href="hadoop_archives.html">Archives</a>
148</div>
149</div>
150<div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">HDFS</div>
151<div id="menu_1.3" class="menuitemgroup">
152<div class="menuitem">
153<a href="hdfs_user_guide.html">User Guide</a>
154</div>
155<div class="menuitem">
156<a href="hdfs_design.html">Architecture</a>
157</div>
158<div class="menuitem">
159<a href="hdfs_shell.html">File System Shell Guide</a>
160</div>
161<div class="menuitem">
162<a href="hdfs_permissions_guide.html">Permissions Guide</a>
163</div>
164<div class="menuitem">
165<a href="hdfs_quota_admin_guide.html">Quotas Guide</a>
166</div>
167<div class="menuitem">
168<a href="SLG_user_guide.html">Synthetic Load Generator Guide</a>
169</div>
170<div class="menuitem">
171<a href="libhdfs.html">C API libhdfs</a>
172</div>
173</div>
174<div onclick="SwitchMenu('menu_1.4', 'skin/')" id="menu_1.4Title" class="menutitle">HOD</div>
175<div id="menu_1.4" class="menuitemgroup">
176<div class="menuitem">
177<a href="hod_user_guide.html">User Guide</a>
178</div>
179<div class="menuitem">
180<a href="hod_admin_guide.html">Admin Guide</a>
181</div>
182<div class="menuitem">
183<a href="hod_config_guide.html">Config Guide</a>
184</div>
185</div>
186<div onclick="SwitchMenu('menu_1.5', 'skin/')" id="menu_1.5Title" class="menutitle">Miscellaneous</div>
187<div id="menu_1.5" class="menuitemgroup">
188<div class="menuitem">
189<a href="api/index.html">API Docs</a>
190</div>
191<div class="menuitem">
192<a href="jdiff/changes.html">API Changes</a>
193</div>
194<div class="menuitem">
195<a href="http://wiki.apache.org/hadoop/">Wiki</a>
196</div>
197<div class="menuitem">
198<a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a>
199</div>
200<div class="menuitem">
201<a href="releasenotes.html">Release Notes</a>
202</div>
203<div class="menuitem">
204<a href="changes.html">Change Log</a>
205</div>
206</div>
207<div id="credit"></div>
208<div id="roundbottom">
209<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
210<!--+
211  |alternative credits
212  +-->
213<div id="credit2"></div>
214</div>
215<!--+
216    |end Menu
217    +-->
218<!--+
219    |start content
220    +-->
221<div id="content">
222<div title="Portable Document Format" class="pdflink">
223<a class="dida" href="fair_scheduler.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
224        PDF</a>
225</div>
226<h1>Fair Scheduler Guide</h1>
227<div id="minitoc-area">
228<ul class="minitoc">
229<li>
230<a href="#Purpose">Purpose</a>
231</li>
232<li>
233<a href="#Introduction">Introduction</a>
234</li>
235<li>
236<a href="#Installation">Installation</a>
237</li>
238<li>
239<a href="#Configuring+the+Fair+scheduler">Configuring the Fair scheduler</a>
240</li>
241<li>
242<a href="#Administration"> Administration</a>
243</li>
244<li>
245<a href="#Implementation">Implementation</a>
246</li>
247</ul>
248</div>
249
250   
251<a name="N1000D"></a><a name="Purpose"></a>
252<h2 class="h3">Purpose</h2>
253<div class="section">
254<p>This document describes the Fair Scheduler, a pluggable
255        Map/Reduce scheduler for Hadoop which provides a way to share
256        large clusters.</p>
257</div>
258
259   
260<a name="N10017"></a><a name="Introduction"></a>
261<h2 class="h3">Introduction</h2>
262<div class="section">
263<p>Fair scheduling is a method of assigning resources to jobs
264        such that all jobs get, on average, an equal share of resources
265        over time. When there is a single job running, that job uses the
266        entire cluster. When other jobs are submitted, tasks slots that
267        free up are assigned to the new jobs, so that each job gets
268        roughly the same amount of CPU time. Unlike the default Hadoop
269        scheduler, which forms a queue of jobs, this lets short jobs finish
270        in reasonable time while not starving long jobs. It is also a
271        reasonable way to share a cluster between a number of users. Finally,
272        fair sharing can also work with job priorities - the priorities are
273        used as weights to determine the fraction of total compute time that
274        each job should get.
275      </p>
276<p>
277        The scheduler actually organizes jobs further into "pools", and
278        shares resources fairly between these pools. By default, there is a
279        separate pool for each user, so that each user gets the same share
280        of the cluster no matter how many jobs they submit. However, it is
281        also possible to set a job's pool based on the user's Unix group or
282        any other jobconf property, such as the queue name property used by
283        <a href="capacity_scheduler.html">Capacity Scheduler</a>.
284        Within each pool, fair sharing is used to share capacity between
285        the running jobs. Pools can also be given weights to share the
286        cluster non-proportionally in the config file.
287      </p>
288<p>
289        In addition to providing fair sharing, the Fair Scheduler allows
290        assigning guaranteed minimum shares to pools, which is useful for
291        ensuring that certain users, groups or production applications
292        always get sufficient resources. When a pool contains jobs, it gets
293        at least its minimum share, but when the pool does not need its full
294        guaranteed share, the excess is split between other running jobs.
295        This lets the scheduler guarantee capacity for pools while utilizing
296        resources efficiently when these pools don't contain jobs.       
297      </p>
298<p>
299        The Fair Scheduler lets all jobs run by default, but it is also
300        possible to limit the number of running jobs per user and per pool
301        through the config file. This can be useful when a user must submit
302        hundreds of jobs at once, or in general to improve performance if
303        running too many jobs at once would cause too much intermediate data
304        to be created or too much context-switching. Limiting the jobs does
305        not cause any subsequently submitted jobs to fail, only to wait in the
306        sheduler's queue until some of the user's earlier jobs finish. Jobs to
307        run from each user/pool are chosen in order of priority and then
308        submit time, as in the default FIFO scheduler in Hadoop.
309      </p>
310<p>
311        Finally, the fair scheduler provides several extension points where
312        the basic functionality can be extended. For example, the weight
313        calculation can be modified to give a priority boost to new jobs,
314        implementing a "shortest job first" policy which reduces response
315        times for interactive jobs even further.
316      </p>
317</div>
318
319   
320<a name="N10031"></a><a name="Installation"></a>
321<h2 class="h3">Installation</h2>
322<div class="section">
323<p>
324        To run the fair scheduler in your Hadoop installation, you need to put
325        it on the CLASSPATH. The easiest way is to copy the
326        <em>hadoop-*-fairscheduler.jar</em> from
327        <em>HADOOP_HOME/contrib/fairscheduler</em> to <em>HADOOP_HOME/lib</em>.
328        Alternatively you can modify <em>HADOOP_CLASSPATH</em> to include this jar, in
329        <em>HADOOP_CONF_DIR/hadoop-env.sh</em>
330     
331</p>
332<p>
333        In order to compile fair scheduler, from sources execute <em> ant
334        package</em> in source folder and copy the
335        <em>build/contrib/fair-scheduler/hadoop-*-fairscheduler.jar</em> 
336        to <em>HADOOP_HOME/lib</em>
337     
338</p>
339<p>
340       You will also need to set the following property in the Hadoop config
341       file  <em>HADOOP_CONF_DIR/mapred-site.xml</em> to have Hadoop use
342       the fair scheduler: <br>
343       
344<span class="codefrag">&lt;property&gt;</span>
345<br> 
346       
347<span class="codefrag">&nbsp;&nbsp;&lt;name&gt;mapred.jobtracker.taskScheduler&lt;/name&gt;</span>
348<br>
349       
350<span class="codefrag">&nbsp;&nbsp;&lt;value&gt;org.apache.hadoop.mapred.FairScheduler&lt;/value&gt;</span>
351<br>
352       
353<span class="codefrag">&lt;/property&gt;</span>
354     
355</p>
356<p>
357        Once you restart the cluster, you can check that the fair scheduler
358        is running by going to http://&lt;jobtracker URL&gt;/scheduler
359        on the JobTracker's web UI. A "job scheduler administration" page should
360        be visible there. This page is described in the Administration section.
361      </p>
362</div>
363   
364   
365<a name="N10070"></a><a name="Configuring+the+Fair+scheduler"></a>
366<h2 class="h3">Configuring the Fair scheduler</h2>
367<div class="section">
368<p>
369      The following properties can be set in mapred-site.xml to configure
370      the fair scheduler:
371      </p>
372<table class="ForrestTable" cellspacing="1" cellpadding="4">
373       
374<tr>
375       
376<th colspan="1" rowspan="1">Name</th><th colspan="1" rowspan="1">Description</th>
377       
378</tr>
379       
380<tr>
381       
382<td colspan="1" rowspan="1">
383          mapred.fairscheduler.allocation.file
384        </td>
385        <td colspan="1" rowspan="1">
386          Specifies an absolute path to an XML file which contains the
387          allocations for each pool, as well as the per-pool and per-user
388          limits on number of running jobs. If this property is not
389          provided, allocations are not used.<br>
390          This file must be in XML format, and can contain three types of
391          elements:
392          <ul>
393         
394<li>pool elements, which may contain elements for minMaps,
395          minReduces, maxRunningJobs (limit the number of jobs from the
396          pool to run at once),and weight (to share the cluster
397          non-proportionally with other pools).
398          </li>
399         
400<li>user elements, which may contain a maxRunningJobs to limit
401          jobs. Note that by default, there is a separate pool for each
402          user, so these may not be necessary; they are useful, however,
403          if you create a pool per user group or manually assign jobs
404          to pools.</li>
405         
406<li>A userMaxJobsDefault element, which sets the default running
407          job limit for any users whose limit is not specified.</li>
408         
409</ul>
410         
411<br>
412          Example Allocation file is listed below :<br>
413         
414<span class="codefrag">&lt;?xml version="1.0"?&gt; </span> 
415<br>
416         
417<span class="codefrag">&lt;allocations&gt;</span> 
418<br> 
419         
420<span class="codefrag">&nbsp;&nbsp;&lt;pool name="sample_pool"&gt;</span>
421<br>
422         
423<span class="codefrag">&nbsp;&nbsp;&nbsp;&nbsp;&lt;minMaps&gt;5&lt;/minMaps&gt;</span>
424<br>
425         
426<span class="codefrag">&nbsp;&nbsp;&nbsp;&nbsp;&lt;minReduces&gt;5&lt;/minReduces&gt;</span>
427<br>
428         
429<span class="codefrag">&nbsp;&nbsp;&nbsp;&nbsp;&lt;weight&gt;2.0&lt;/weight&gt;</span>
430<br>
431         
432<span class="codefrag">&nbsp;&nbsp;&lt;/pool&gt;</span>
433<br>
434         
435<span class="codefrag">&nbsp;&nbsp;&lt;user name="sample_user"&gt;</span>
436<br>
437         
438<span class="codefrag">&nbsp;&nbsp;&nbsp;&nbsp;&lt;maxRunningJobs&gt;6&lt;/maxRunningJobs&gt;</span>
439<br>
440         
441<span class="codefrag">&nbsp;&nbsp;&lt;/user&gt;</span>
442<br>
443         
444<span class="codefrag">&nbsp;&nbsp;&lt;userMaxJobsDefault&gt;3&lt;/userMaxJobsDefault&gt;</span>
445<br>
446         
447<span class="codefrag">&lt;/allocations&gt;</span>
448         
449<br>
450          This example creates a pool sample_pool with a guarantee of 5 map
451          slots and 5 reduce slots. The pool also has a weight of 2.0, meaning
452          it has a 2x higher share of the cluster than other pools (the default
453          weight is 1). Finally, the example limits the number of running jobs
454          per user to 3, except for sample_user, who can run 6 jobs concurrently.
455          Any pool not defined in the allocations file will have no guaranteed
456          capacity and a weight of 1.0. Also, any pool or user with no max
457          running jobs set in the file will be allowed to run an unlimited
458          number of jobs.
459        </td>
460       
461</tr>
462       
463<tr>
464       
465<td colspan="1" rowspan="1">
466          mapred.fairscheduler.assignmultiple
467        </td>
468        <td colspan="1" rowspan="1">
469          Allows the scheduler to assign both a map task and a reduce task
470          on each heartbeat, which improves cluster throughput when there
471          are many small tasks to run. Boolean value, default: false.
472        </td>
473       
474</tr>
475       
476<tr>
477       
478<td colspan="1" rowspan="1">
479          mapred.fairscheduler.sizebasedweight
480        </td>
481        <td colspan="1" rowspan="1">
482          Take into account job sizes in calculating their weights for fair
483          sharing.By default, weights are only based on job priorities.
484          Setting this flag to true will make them based on the size of the
485          job (number of tasks needed) as well,though not linearly
486          (the weight will be proportional to the log of the number of tasks
487          needed). This lets larger jobs get larger fair shares while still
488          providing enough of a share to small jobs to let them finish fast.
489          Boolean value, default: false.
490        </td>
491       
492</tr>
493       
494<tr>
495       
496<td colspan="1" rowspan="1">
497          mapred.fairscheduler.poolnameproperty
498        </td>
499        <td colspan="1" rowspan="1">
500          Specify which jobconf property is used to determine the pool that a
501          job belongs in. String, default: user.name (i.e. one pool for each
502          user). Some other useful values to set this to are: <br>
503         
504<ul> 
505           
506<li> group.name (to create a pool per Unix group).</li>
507           
508<li>mapred.job.queue.name (the same property as the queue name in
509            <a href="capacity_scheduler.html">Capacity Scheduler</a>).</li>
510         
511</ul>
512       
513</td>
514       
515</tr>
516       
517<tr>
518       
519<td colspan="1" rowspan="1">
520          mapred.fairscheduler.weightadjuster
521        </td>
522        <td colspan="1" rowspan="1">
523        An extensibility point that lets you specify a class to adjust the
524        weights of running jobs. This class should implement the
525        <em>WeightAdjuster</em> interface. There is currently one example
526        implementation - <em>NewJobWeightBooster</em>, which increases the
527        weight of jobs for the first 5 minutes of their lifetime to let
528        short jobs finish faster. To use it, set the weightadjuster
529        property to the full class name,
530        <span class="codefrag">org.apache.hadoop.mapred.NewJobWeightBooster</span> 
531        NewJobWeightBooster itself provides two parameters for setting the
532        duration and boost factor. <br>
533       
534<ol>
535       
536<li> 
537<em>mapred.newjobweightbooster.factor</em>
538          Factor by which new jobs weight should be boosted. Default is 3</li>
539       
540<li>
541<em>mapred.newjobweightbooster.duration</em>
542          Duration in milliseconds, default 300000 for 5 minutes</li>
543       
544</ol>
545       
546</td>
547       
548</tr>
549       
550<tr>
551       
552<td colspan="1" rowspan="1">
553          mapred.fairscheduler.loadmanager
554        </td>
555        <td colspan="1" rowspan="1">
556          An extensibility point that lets you specify a class that determines
557          how many maps and reduces can run on a given TaskTracker. This class
558          should implement the LoadManager interface. By default the task caps
559          in the Hadoop config file are used, but this option could be used to
560          make the load based on available memory and CPU utilization for example.
561        </td>
562       
563</tr>
564       
565<tr>
566       
567<td colspan="1" rowspan="1">
568          mapred.fairscheduler.taskselector:
569        </td>
570        <td colspan="1" rowspan="1">
571        An extensibility point that lets you specify a class that determines
572        which task from within a job to launch on a given tracker. This can be
573        used to change either the locality policy (e.g. keep some jobs within
574        a particular rack) or the speculative execution algorithm (select
575        when to launch speculative tasks). The default implementation uses
576        Hadoop's default algorithms from JobInProgress.
577        </td>
578       
579</tr>
580     
581</table>
582</div>
583   
584<a name="N10151"></a><a name="Administration"></a>
585<h2 class="h3"> Administration</h2>
586<div class="section">
587<p>
588      The fair scheduler provides support for administration at runtime
589      through two mechanisms:
590    </p>
591<ol>
592   
593<li>
594      It is possible to modify pools' allocations
595      and user and pool running job limits at runtime by editing the allocation
596      config file. The scheduler will reload this file 10-15 seconds after it
597      sees that it was modified.
598     </li>
599     
600<li>
601     Current jobs, pools, and fair shares  can be examined through the
602     JobTracker's web interface, at  http://&lt;jobtracker URL&gt;/scheduler.
603     On this interface, it is also possible to modify jobs' priorities or
604     move jobs from one pool to another and see the effects on the fair
605     shares (this requires JavaScript).
606     </li>
607   
608</ol>
609<p>
610      The following fields can be seen for each job on the web interface:
611     </p>
612<ul>
613     
614<li>
615<em>Submitted</em> - Date and time job was submitted.</li>
616     
617<li>
618<em>JobID, User, Name</em> - Job identifiers as on the standard
619     web UI.</li>
620     
621<li>
622<em>Pool</em> - Current pool of job. Select another value to move job to
623     another pool.</li>
624     
625<li>
626<em>Priority</em> - Current priority. Select another value to change the
627     job's priority</li>
628     
629<li>
630<em>Maps/Reduces Finished</em>: Number of tasks finished / total tasks.</li>
631     
632<li>
633<em>Maps/Reduces Running</em>: Tasks currently running.</li>
634     
635<li>
636<em>Map/Reduce Fair Share</em>: The average number of task slots that this
637     job should have at any given time according to fair sharing. The actual
638     number of tasks will go up and down depending on how much compute time
639     the job has had, but on average it will get its fair share amount.</li>
640     
641</ul>
642<p>
643     In addition, it is possible to turn on an "advanced" view for the web UI,
644     by going to http://&lt;jobtracker URL&gt;/scheduler?advanced. This view shows
645     four more columns used for calculations internally:
646     </p>
647<ul>
648     
649<li>
650<em>Maps/Reduce Weight</em>: Weight of the job in the fair sharing
651     calculations. This depends on priority and potentially also on
652     job size and job age if the <em>sizebasedweight</em> and
653     <em>NewJobWeightBooster</em> are enabled.</li>
654     
655<li>
656<em>Map/Reduce Deficit</em>: The job's scheduling deficit in machine-
657     seconds - the amount of resources it should have gotten according to
658     its fair share, minus how many it actually got. Positive deficit means
659      the job will be scheduled again in the near future because it needs to
660      catch up to its fair share. The scheduler schedules jobs with higher
661      deficit ahead of others. Please see the Implementation section of
662      this document for details.</li>
663     
664</ul>
665</div>
666   
667<a name="N101A3"></a><a name="Implementation"></a>
668<h2 class="h3">Implementation</h2>
669<div class="section">
670<p>There are two aspects to implementing fair scheduling: Calculating
671    each job's fair share, and choosing which job to run when a task slot
672    becomes available.</p>
673<p>To select jobs to run, the scheduler then keeps track of a
674    "deficit" for each job - the difference between the amount of
675     compute time it should have gotten on an ideal scheduler, and the amount
676     of compute time it actually got. This is a measure of how
677     "unfair" we've been to the job. Every few hundred
678     milliseconds, the scheduler updates the deficit of each job by looking
679     at how many tasks each job had running during this interval vs. its
680     fair share. Whenever a task slot becomes available, it is assigned to
681     the job with the highest deficit. There is one exception - if there
682     were one or more jobs who were not meeting their pool capacity
683     guarantees, we only choose among these "needy" jobs (based
684     again on their deficit), to ensure that the scheduler meets pool
685     guarantees as soon as possible.</p>
686<p>
687     The fair shares are calculated by dividing the capacity of the cluster
688     among runnable jobs according to a "weight" for each job. By
689     default the weight is based on priority, with each level of priority
690     having 2x higher weight than the next (for example, VERY_HIGH has 4x the
691     weight of NORMAL). However, weights can also be based on job sizes and ages,
692     as described in the Configuring section. For jobs that are in a pool,
693     fair shares also take into account the minimum guarantee for that pool.
694     This capacity is divided among the jobs in that pool according again to
695     their weights.
696     </p>
697<p>Finally, when limits on a user's running jobs or a pool's running jobs
698     are in place, we choose which jobs get to run by sorting all jobs in order
699     of priority and then submit time, as in the standard Hadoop scheduler. Any
700     jobs that fall after the user/pool's limit in this ordering are queued up
701     and wait idle until they can be run. During this time, they are ignored
702     from the fair sharing calculations and do not gain or lose deficit (their
703     fair share is set to zero).</p>
704</div>
705 
706</div>
707<!--+
708    |end content
709    +-->
710<div class="clearboth">&nbsp;</div>
711</div>
712<div id="footer">
713<!--+
714    |start bottomstrip
715    +-->
716<div class="lastmodified">
717<script type="text/javascript"><!--
718document.write("Last Published: " + document.lastModified);
719//  --></script>
720</div>
721<div class="copyright">
722        Copyright &copy;
723         2008 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
724</div>
725<!--+
726    |end bottomstrip
727    +-->
728</div>
729</body>
730</html>
Note: See TracBrowser for help on using the repository browser.