[120] | 1 | <?xml version="1.0"?> |
---|
| 2 | <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> |
---|
| 3 | |
---|
| 4 | <!-- Do not modify this file directly. Instead, copy entries that you --> |
---|
| 5 | <!-- wish to modify from this file into mapred-site.xml and change them --> |
---|
| 6 | <!-- there. If mapred-site.xml does not already exist, create it. --> |
---|
| 7 | |
---|
| 8 | <configuration> |
---|
| 9 | |
---|
| 10 | <property> |
---|
| 11 | <name>hadoop.job.history.location</name> |
---|
| 12 | <value></value> |
---|
| 13 | <description> If job tracker is static the history files are stored |
---|
| 14 | in this single well known place. If No value is set here, by default, |
---|
| 15 | it is in the local file system at ${hadoop.log.dir}/history. |
---|
| 16 | </description> |
---|
| 17 | </property> |
---|
| 18 | |
---|
| 19 | <property> |
---|
| 20 | <name>hadoop.job.history.user.location</name> |
---|
| 21 | <value></value> |
---|
| 22 | <description> User can specify a location to store the history files of |
---|
| 23 | a particular job. If nothing is specified, the logs are stored in |
---|
| 24 | output directory. The files are stored in "_logs/history/" in the directory. |
---|
| 25 | User can stop logging by giving the value "none". |
---|
| 26 | </description> |
---|
| 27 | </property> |
---|
| 28 | |
---|
| 29 | <!-- i/o properties --> |
---|
| 30 | |
---|
| 31 | <property> |
---|
| 32 | <name>io.sort.factor</name> |
---|
| 33 | <value>10</value> |
---|
| 34 | <description>The number of streams to merge at once while sorting |
---|
| 35 | files. This determines the number of open file handles.</description> |
---|
| 36 | </property> |
---|
| 37 | |
---|
| 38 | <property> |
---|
| 39 | <name>io.sort.mb</name> |
---|
| 40 | <value>100</value> |
---|
| 41 | <description>The total amount of buffer memory to use while sorting |
---|
| 42 | files, in megabytes. By default, gives each merge stream 1MB, which |
---|
| 43 | should minimize seeks.</description> |
---|
| 44 | </property> |
---|
| 45 | |
---|
| 46 | <property> |
---|
| 47 | <name>io.sort.record.percent</name> |
---|
| 48 | <value>0.05</value> |
---|
| 49 | <description>The percentage of io.sort.mb dedicated to tracking record |
---|
| 50 | boundaries. Let this value be r, io.sort.mb be x. The maximum number |
---|
| 51 | of records collected before the collection thread must block is equal |
---|
| 52 | to (r * x) / 4</description> |
---|
| 53 | </property> |
---|
| 54 | |
---|
| 55 | <property> |
---|
| 56 | <name>io.sort.spill.percent</name> |
---|
| 57 | <value>0.80</value> |
---|
| 58 | <description>The soft limit in either the buffer or record collection |
---|
| 59 | buffers. Once reached, a thread will begin to spill the contents to disk |
---|
| 60 | in the background. Note that this does not imply any chunking of data to |
---|
| 61 | the spill. A value less than 0.5 is not recommended.</description> |
---|
| 62 | </property> |
---|
| 63 | |
---|
| 64 | <property> |
---|
| 65 | <name>io.map.index.skip</name> |
---|
| 66 | <value>0</value> |
---|
| 67 | <description>Number of index entries to skip between each entry. |
---|
| 68 | Zero by default. Setting this to values larger than zero can |
---|
| 69 | facilitate opening large map files using less memory.</description> |
---|
| 70 | </property> |
---|
| 71 | |
---|
| 72 | <property> |
---|
| 73 | <name>mapred.job.tracker</name> |
---|
| 74 | <value>local</value> |
---|
| 75 | <description>The host and port that the MapReduce job tracker runs |
---|
| 76 | at. If "local", then jobs are run in-process as a single map |
---|
| 77 | and reduce task. |
---|
| 78 | </description> |
---|
| 79 | </property> |
---|
| 80 | |
---|
| 81 | <property> |
---|
| 82 | <name>mapred.job.tracker.http.address</name> |
---|
| 83 | <value>0.0.0.0:50030</value> |
---|
| 84 | <description> |
---|
| 85 | The job tracker http server address and port the server will listen on. |
---|
| 86 | If the port is 0 then the server will start on a free port. |
---|
| 87 | </description> |
---|
| 88 | </property> |
---|
| 89 | |
---|
| 90 | <property> |
---|
| 91 | <name>mapred.job.tracker.handler.count</name> |
---|
| 92 | <value>10</value> |
---|
| 93 | <description> |
---|
| 94 | The number of server threads for the JobTracker. This should be roughly |
---|
| 95 | 4% of the number of tasktracker nodes. |
---|
| 96 | </description> |
---|
| 97 | </property> |
---|
| 98 | |
---|
| 99 | <property> |
---|
| 100 | <name>mapred.task.tracker.report.address</name> |
---|
| 101 | <value>127.0.0.1:0</value> |
---|
| 102 | <description>The interface and port that task tracker server listens on. |
---|
| 103 | Since it is only connected to by the tasks, it uses the local interface. |
---|
| 104 | EXPERT ONLY. Should only be changed if your host does not have the loopback |
---|
| 105 | interface.</description> |
---|
| 106 | </property> |
---|
| 107 | |
---|
| 108 | <property> |
---|
| 109 | <name>mapred.local.dir</name> |
---|
| 110 | <value>${hadoop.tmp.dir}/mapred/local</value> |
---|
| 111 | <description>The local directory where MapReduce stores intermediate |
---|
| 112 | data files. May be a comma-separated list of |
---|
| 113 | directories on different devices in order to spread disk i/o. |
---|
| 114 | Directories that do not exist are ignored. |
---|
| 115 | </description> |
---|
| 116 | </property> |
---|
| 117 | |
---|
| 118 | <property> |
---|
| 119 | <name>mapred.system.dir</name> |
---|
| 120 | <value>${hadoop.tmp.dir}/mapred/system</value> |
---|
| 121 | <description>The shared directory where MapReduce stores control files. |
---|
| 122 | </description> |
---|
| 123 | </property> |
---|
| 124 | |
---|
| 125 | <property> |
---|
| 126 | <name>mapred.temp.dir</name> |
---|
| 127 | <value>${hadoop.tmp.dir}/mapred/temp</value> |
---|
| 128 | <description>A shared directory for temporary files. |
---|
| 129 | </description> |
---|
| 130 | </property> |
---|
| 131 | |
---|
| 132 | <property> |
---|
| 133 | <name>mapred.local.dir.minspacestart</name> |
---|
| 134 | <value>0</value> |
---|
| 135 | <description>If the space in mapred.local.dir drops under this, |
---|
| 136 | do not ask for more tasks. |
---|
| 137 | Value in bytes. |
---|
| 138 | </description> |
---|
| 139 | </property> |
---|
| 140 | |
---|
| 141 | <property> |
---|
| 142 | <name>mapred.local.dir.minspacekill</name> |
---|
| 143 | <value>0</value> |
---|
| 144 | <description>If the space in mapred.local.dir drops under this, |
---|
| 145 | do not ask more tasks until all the current ones have finished and |
---|
| 146 | cleaned up. Also, to save the rest of the tasks we have running, |
---|
| 147 | kill one of them, to clean up some space. Start with the reduce tasks, |
---|
| 148 | then go with the ones that have finished the least. |
---|
| 149 | Value in bytes. |
---|
| 150 | </description> |
---|
| 151 | </property> |
---|
| 152 | |
---|
| 153 | <property> |
---|
| 154 | <name>mapred.tasktracker.expiry.interval</name> |
---|
| 155 | <value>600000</value> |
---|
| 156 | <description>Expert: The time-interval, in miliseconds, after which |
---|
| 157 | a tasktracker is declared 'lost' if it doesn't send heartbeats. |
---|
| 158 | </description> |
---|
| 159 | </property> |
---|
| 160 | |
---|
| 161 | <property> |
---|
| 162 | <name>mapred.tasktracker.instrumentation</name> |
---|
| 163 | <value>org.apache.hadoop.mapred.TaskTrackerMetricsInst</value> |
---|
| 164 | <description>Expert: The instrumentation class to associate with each TaskTracker. |
---|
| 165 | </description> |
---|
| 166 | </property> |
---|
| 167 | |
---|
| 168 | <property> |
---|
| 169 | <name>mapred.tasktracker.memory_calculator_plugin</name> |
---|
| 170 | <value></value> |
---|
| 171 | <description> |
---|
| 172 | Name of the class whose instance will be used to query memory information |
---|
| 173 | on the tasktracker. |
---|
| 174 | |
---|
| 175 | The class must be an instance of |
---|
| 176 | org.apache.hadoop.util.MemoryCalculatorPlugin. If the value is null, the |
---|
| 177 | tasktracker attempts to use a class appropriate to the platform. |
---|
| 178 | Currently, the only platform supported is Linux. |
---|
| 179 | </description> |
---|
| 180 | </property> |
---|
| 181 | |
---|
| 182 | <property> |
---|
| 183 | <name>mapred.tasktracker.taskmemorymanager.monitoring-interval</name> |
---|
| 184 | <value>5000</value> |
---|
| 185 | <description>The interval, in milliseconds, for which the tasktracker waits |
---|
| 186 | between two cycles of monitoring its tasks' memory usage. Used only if |
---|
| 187 | tasks' memory management is enabled via mapred.tasktracker.tasks.maxmemory. |
---|
| 188 | </description> |
---|
| 189 | </property> |
---|
| 190 | |
---|
| 191 | <property> |
---|
| 192 | <name>mapred.tasktracker.procfsbasedprocesstree.sleeptime-before-sigkill</name> |
---|
| 193 | <value>5000</value> |
---|
| 194 | <description>The time, in milliseconds, the tasktracker waits for sending a |
---|
| 195 | SIGKILL to a process that has overrun memory limits, after it has been sent |
---|
| 196 | a SIGTERM. Used only if tasks' memory management is enabled via |
---|
| 197 | mapred.tasktracker.tasks.maxmemory.</description> |
---|
| 198 | </property> |
---|
| 199 | |
---|
| 200 | <property> |
---|
| 201 | <name>mapred.map.tasks</name> |
---|
| 202 | <value>2</value> |
---|
| 203 | <description>The default number of map tasks per job. |
---|
| 204 | Ignored when mapred.job.tracker is "local". |
---|
| 205 | </description> |
---|
| 206 | </property> |
---|
| 207 | |
---|
| 208 | <property> |
---|
| 209 | <name>mapred.reduce.tasks</name> |
---|
| 210 | <value>1</value> |
---|
| 211 | <description>The default number of reduce tasks per job. Typically set to 99% |
---|
| 212 | of the cluster's reduce capacity, so that if a node fails the reduces can |
---|
| 213 | still be executed in a single wave. |
---|
| 214 | Ignored when mapred.job.tracker is "local". |
---|
| 215 | </description> |
---|
| 216 | </property> |
---|
| 217 | |
---|
| 218 | <property> |
---|
| 219 | <name>mapred.jobtracker.restart.recover</name> |
---|
| 220 | <value>false</value> |
---|
| 221 | <description>"true" to enable (job) recovery upon restart, |
---|
| 222 | "false" to start afresh |
---|
| 223 | </description> |
---|
| 224 | </property> |
---|
| 225 | |
---|
| 226 | <property> |
---|
| 227 | <name>mapred.jobtracker.job.history.block.size</name> |
---|
| 228 | <value>3145728</value> |
---|
| 229 | <description>The block size of the job history file. Since the job recovery |
---|
| 230 | uses job history, its important to dump job history to disk as |
---|
| 231 | soon as possible. Note that this is an expert level parameter. |
---|
| 232 | The default value is set to 3 MB. |
---|
| 233 | </description> |
---|
| 234 | </property> |
---|
| 235 | |
---|
| 236 | <property> |
---|
| 237 | <name>mapred.jobtracker.taskScheduler</name> |
---|
| 238 | <value>org.apache.hadoop.mapred.JobQueueTaskScheduler</value> |
---|
| 239 | <description>The class responsible for scheduling the tasks.</description> |
---|
| 240 | </property> |
---|
| 241 | |
---|
| 242 | <property> |
---|
| 243 | <name>mapred.jobtracker.taskScheduler.maxRunningTasksPerJob</name> |
---|
| 244 | <value></value> |
---|
| 245 | <description>The maximum number of running tasks for a job before |
---|
| 246 | it gets preempted. No limits if undefined. |
---|
| 247 | </description> |
---|
| 248 | </property> |
---|
| 249 | |
---|
| 250 | <property> |
---|
| 251 | <name>mapred.map.max.attempts</name> |
---|
| 252 | <value>4</value> |
---|
| 253 | <description>Expert: The maximum number of attempts per map task. |
---|
| 254 | In other words, framework will try to execute a map task these many number |
---|
| 255 | of times before giving up on it. |
---|
| 256 | </description> |
---|
| 257 | </property> |
---|
| 258 | |
---|
| 259 | <property> |
---|
| 260 | <name>mapred.reduce.max.attempts</name> |
---|
| 261 | <value>4</value> |
---|
| 262 | <description>Expert: The maximum number of attempts per reduce task. |
---|
| 263 | In other words, framework will try to execute a reduce task these many number |
---|
| 264 | of times before giving up on it. |
---|
| 265 | </description> |
---|
| 266 | </property> |
---|
| 267 | |
---|
| 268 | <property> |
---|
| 269 | <name>mapred.reduce.parallel.copies</name> |
---|
| 270 | <value>5</value> |
---|
| 271 | <description>The default number of parallel transfers run by reduce |
---|
| 272 | during the copy(shuffle) phase. |
---|
| 273 | </description> |
---|
| 274 | </property> |
---|
| 275 | |
---|
| 276 | <property> |
---|
| 277 | <name>mapred.reduce.copy.backoff</name> |
---|
| 278 | <value>300</value> |
---|
| 279 | <description>The maximum amount of time (in seconds) a reducer spends on |
---|
| 280 | fetching one map output before declaring it as failed. |
---|
| 281 | </description> |
---|
| 282 | </property> |
---|
| 283 | |
---|
| 284 | <property> |
---|
| 285 | <name>mapred.task.timeout</name> |
---|
| 286 | <value>600000</value> |
---|
| 287 | <description>The number of milliseconds before a task will be |
---|
| 288 | terminated if it neither reads an input, writes an output, nor |
---|
| 289 | updates its status string. |
---|
| 290 | </description> |
---|
| 291 | </property> |
---|
| 292 | |
---|
| 293 | <property> |
---|
| 294 | <name>mapred.tasktracker.map.tasks.maximum</name> |
---|
| 295 | <value>2</value> |
---|
| 296 | <description>The maximum number of map tasks that will be run |
---|
| 297 | simultaneously by a task tracker. |
---|
| 298 | </description> |
---|
| 299 | </property> |
---|
| 300 | |
---|
| 301 | <property> |
---|
| 302 | <name>mapred.tasktracker.reduce.tasks.maximum</name> |
---|
| 303 | <value>2</value> |
---|
| 304 | <description>The maximum number of reduce tasks that will be run |
---|
| 305 | simultaneously by a task tracker. |
---|
| 306 | </description> |
---|
| 307 | </property> |
---|
| 308 | |
---|
| 309 | <property> |
---|
| 310 | <name>mapred.jobtracker.completeuserjobs.maximum</name> |
---|
| 311 | <value>100</value> |
---|
| 312 | <description>The maximum number of complete jobs per user to keep around |
---|
| 313 | before delegating them to the job history.</description> |
---|
| 314 | </property> |
---|
| 315 | |
---|
| 316 | <property> |
---|
| 317 | <name>mapred.jobtracker.instrumentation</name> |
---|
| 318 | <value>org.apache.hadoop.mapred.JobTrackerMetricsInst</value> |
---|
| 319 | <description>Expert: The instrumentation class to associate with each JobTracker. |
---|
| 320 | </description> |
---|
| 321 | </property> |
---|
| 322 | |
---|
| 323 | <property> |
---|
| 324 | <name>mapred.child.java.opts</name> |
---|
| 325 | <value>-Xmx200m</value> |
---|
| 326 | <description>Java opts for the task tracker child processes. |
---|
| 327 | The following symbol, if present, will be interpolated: @taskid@ is replaced |
---|
| 328 | by current TaskID. Any other occurrences of '@' will go unchanged. |
---|
| 329 | For example, to enable verbose gc logging to a file named for the taskid in |
---|
| 330 | /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: |
---|
| 331 | -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc |
---|
| 332 | |
---|
| 333 | The configuration variable mapred.child.ulimit can be used to control the |
---|
| 334 | maximum virtual memory of the child processes. |
---|
| 335 | </description> |
---|
| 336 | </property> |
---|
| 337 | |
---|
| 338 | <property> |
---|
| 339 | <name>mapred.child.ulimit</name> |
---|
| 340 | <value></value> |
---|
| 341 | <description>The maximum virtual memory, in KB, of a process launched by the |
---|
| 342 | Map-Reduce framework. This can be used to control both the Mapper/Reducer |
---|
| 343 | tasks and applications using Hadoop Pipes, Hadoop Streaming etc. |
---|
| 344 | By default it is left unspecified to let cluster admins control it via |
---|
| 345 | limits.conf and other such relevant mechanisms. |
---|
| 346 | |
---|
| 347 | Note: mapred.child.ulimit must be greater than or equal to the -Xmx passed to |
---|
| 348 | JavaVM, else the VM might not start. |
---|
| 349 | </description> |
---|
| 350 | </property> |
---|
| 351 | |
---|
| 352 | <property> |
---|
| 353 | <name>mapred.child.tmp</name> |
---|
| 354 | <value>./tmp</value> |
---|
| 355 | <description> To set the value of tmp directory for map and reduce tasks. |
---|
| 356 | If the value is an absolute path, it is directly assigned. Otherwise, it is |
---|
| 357 | prepended with task's working directory. The java tasks are executed with |
---|
| 358 | option -Djava.io.tmpdir='the absolute path of the tmp dir'. Pipes and |
---|
| 359 | streaming are set with environment variable, |
---|
| 360 | TMPDIR='the absolute path of the tmp dir' |
---|
| 361 | </description> |
---|
| 362 | </property> |
---|
| 363 | |
---|
| 364 | <property> |
---|
| 365 | <name>mapred.inmem.merge.threshold</name> |
---|
| 366 | <value>1000</value> |
---|
| 367 | <description>The threshold, in terms of the number of files |
---|
| 368 | for the in-memory merge process. When we accumulate threshold number of files |
---|
| 369 | we initiate the in-memory merge and spill to disk. A value of 0 or less than |
---|
| 370 | 0 indicates we want to DON'T have any threshold and instead depend only on |
---|
| 371 | the ramfs's memory consumption to trigger the merge. |
---|
| 372 | </description> |
---|
| 373 | </property> |
---|
| 374 | |
---|
| 375 | <property> |
---|
| 376 | <name>mapred.job.shuffle.merge.percent</name> |
---|
| 377 | <value>0.66</value> |
---|
| 378 | <description>The usage threshold at which an in-memory merge will be |
---|
| 379 | initiated, expressed as a percentage of the total memory allocated to |
---|
| 380 | storing in-memory map outputs, as defined by |
---|
| 381 | mapred.job.shuffle.input.buffer.percent. |
---|
| 382 | </description> |
---|
| 383 | </property> |
---|
| 384 | |
---|
| 385 | <property> |
---|
| 386 | <name>mapred.job.shuffle.input.buffer.percent</name> |
---|
| 387 | <value>0.70</value> |
---|
| 388 | <description>The percentage of memory to be allocated from the maximum heap |
---|
| 389 | size to storing map outputs during the shuffle. |
---|
| 390 | </description> |
---|
| 391 | </property> |
---|
| 392 | |
---|
| 393 | <property> |
---|
| 394 | <name>mapred.job.reduce.input.buffer.percent</name> |
---|
| 395 | <value>0.0</value> |
---|
| 396 | <description>The percentage of memory- relative to the maximum heap size- to |
---|
| 397 | retain map outputs during the reduce. When the shuffle is concluded, any |
---|
| 398 | remaining map outputs in memory must consume less than this threshold before |
---|
| 399 | the reduce can begin. |
---|
| 400 | </description> |
---|
| 401 | </property> |
---|
| 402 | |
---|
| 403 | <property> |
---|
| 404 | <name>mapred.map.tasks.speculative.execution</name> |
---|
| 405 | <value>true</value> |
---|
| 406 | <description>If true, then multiple instances of some map tasks |
---|
| 407 | may be executed in parallel.</description> |
---|
| 408 | </property> |
---|
| 409 | |
---|
| 410 | <property> |
---|
| 411 | <name>mapred.reduce.tasks.speculative.execution</name> |
---|
| 412 | <value>true</value> |
---|
| 413 | <description>If true, then multiple instances of some reduce tasks |
---|
| 414 | may be executed in parallel.</description> |
---|
| 415 | </property> |
---|
| 416 | |
---|
| 417 | <property> |
---|
| 418 | <name>mapred.job.reuse.jvm.num.tasks</name> |
---|
| 419 | <value>1</value> |
---|
| 420 | <description>How many tasks to run per jvm. If set to -1, there is |
---|
| 421 | no limit. |
---|
| 422 | </description> |
---|
| 423 | </property> |
---|
| 424 | |
---|
| 425 | <property> |
---|
| 426 | <name>mapred.min.split.size</name> |
---|
| 427 | <value>0</value> |
---|
| 428 | <description>The minimum size chunk that map input should be split |
---|
| 429 | into. Note that some file formats may have minimum split sizes that |
---|
| 430 | take priority over this setting.</description> |
---|
| 431 | </property> |
---|
| 432 | |
---|
| 433 | <property> |
---|
| 434 | <name>mapred.jobtracker.maxtasks.per.job</name> |
---|
| 435 | <value>-1</value> |
---|
| 436 | <description>The maximum number of tasks for a single job. |
---|
| 437 | A value of -1 indicates that there is no maximum. </description> |
---|
| 438 | </property> |
---|
| 439 | |
---|
| 440 | <property> |
---|
| 441 | <name>mapred.submit.replication</name> |
---|
| 442 | <value>10</value> |
---|
| 443 | <description>The replication level for submitted job files. This |
---|
| 444 | should be around the square root of the number of nodes. |
---|
| 445 | </description> |
---|
| 446 | </property> |
---|
| 447 | |
---|
| 448 | |
---|
| 449 | <property> |
---|
| 450 | <name>mapred.tasktracker.dns.interface</name> |
---|
| 451 | <value>default</value> |
---|
| 452 | <description>The name of the Network Interface from which a task |
---|
| 453 | tracker should report its IP address. |
---|
| 454 | </description> |
---|
| 455 | </property> |
---|
| 456 | |
---|
| 457 | <property> |
---|
| 458 | <name>mapred.tasktracker.dns.nameserver</name> |
---|
| 459 | <value>default</value> |
---|
| 460 | <description>The host name or IP address of the name server (DNS) |
---|
| 461 | which a TaskTracker should use to determine the host name used by |
---|
| 462 | the JobTracker for communication and display purposes. |
---|
| 463 | </description> |
---|
| 464 | </property> |
---|
| 465 | |
---|
| 466 | <property> |
---|
| 467 | <name>tasktracker.http.threads</name> |
---|
| 468 | <value>40</value> |
---|
| 469 | <description>The number of worker threads that for the http server. This is |
---|
| 470 | used for map output fetching |
---|
| 471 | </description> |
---|
| 472 | </property> |
---|
| 473 | |
---|
| 474 | <property> |
---|
| 475 | <name>mapred.task.tracker.http.address</name> |
---|
| 476 | <value>0.0.0.0:50060</value> |
---|
| 477 | <description> |
---|
| 478 | The task tracker http server address and port. |
---|
| 479 | If the port is 0 then the server will start on a free port. |
---|
| 480 | </description> |
---|
| 481 | </property> |
---|
| 482 | |
---|
| 483 | <property> |
---|
| 484 | <name>keep.failed.task.files</name> |
---|
| 485 | <value>false</value> |
---|
| 486 | <description>Should the files for failed tasks be kept. This should only be |
---|
| 487 | used on jobs that are failing, because the storage is never |
---|
| 488 | reclaimed. It also prevents the map outputs from being erased |
---|
| 489 | from the reduce directory as they are consumed.</description> |
---|
| 490 | </property> |
---|
| 491 | |
---|
| 492 | |
---|
| 493 | <!-- |
---|
| 494 | <property> |
---|
| 495 | <name>keep.task.files.pattern</name> |
---|
| 496 | <value>.*_m_123456_0</value> |
---|
| 497 | <description>Keep all files from tasks whose task names match the given |
---|
| 498 | regular expression. Defaults to none.</description> |
---|
| 499 | </property> |
---|
| 500 | --> |
---|
| 501 | |
---|
| 502 | <property> |
---|
| 503 | <name>mapred.output.compress</name> |
---|
| 504 | <value>false</value> |
---|
| 505 | <description>Should the job outputs be compressed? |
---|
| 506 | </description> |
---|
| 507 | </property> |
---|
| 508 | |
---|
| 509 | <property> |
---|
| 510 | <name>mapred.output.compression.type</name> |
---|
| 511 | <value>RECORD</value> |
---|
| 512 | <description>If the job outputs are to compressed as SequenceFiles, how should |
---|
| 513 | they be compressed? Should be one of NONE, RECORD or BLOCK. |
---|
| 514 | </description> |
---|
| 515 | </property> |
---|
| 516 | |
---|
| 517 | <property> |
---|
| 518 | <name>mapred.output.compression.codec</name> |
---|
| 519 | <value>org.apache.hadoop.io.compress.DefaultCodec</value> |
---|
| 520 | <description>If the job outputs are compressed, how should they be compressed? |
---|
| 521 | </description> |
---|
| 522 | </property> |
---|
| 523 | |
---|
| 524 | <property> |
---|
| 525 | <name>mapred.compress.map.output</name> |
---|
| 526 | <value>false</value> |
---|
| 527 | <description>Should the outputs of the maps be compressed before being |
---|
| 528 | sent across the network. Uses SequenceFile compression. |
---|
| 529 | </description> |
---|
| 530 | </property> |
---|
| 531 | |
---|
| 532 | <property> |
---|
| 533 | <name>mapred.map.output.compression.codec</name> |
---|
| 534 | <value>org.apache.hadoop.io.compress.DefaultCodec</value> |
---|
| 535 | <description>If the map outputs are compressed, how should they be |
---|
| 536 | compressed? |
---|
| 537 | </description> |
---|
| 538 | </property> |
---|
| 539 | |
---|
| 540 | <property> |
---|
| 541 | <name>map.sort.class</name> |
---|
| 542 | <value>org.apache.hadoop.util.QuickSort</value> |
---|
| 543 | <description>The default sort class for sorting keys. |
---|
| 544 | </description> |
---|
| 545 | </property> |
---|
| 546 | |
---|
| 547 | <property> |
---|
| 548 | <name>mapred.userlog.limit.kb</name> |
---|
| 549 | <value>0</value> |
---|
| 550 | <description>The maximum size of user-logs of each task in KB. 0 disables the cap. |
---|
| 551 | </description> |
---|
| 552 | </property> |
---|
| 553 | |
---|
| 554 | <property> |
---|
| 555 | <name>mapred.userlog.retain.hours</name> |
---|
| 556 | <value>24</value> |
---|
| 557 | <description>The maximum time, in hours, for which the user-logs are to be |
---|
| 558 | retained. |
---|
| 559 | </description> |
---|
| 560 | </property> |
---|
| 561 | |
---|
| 562 | <property> |
---|
| 563 | <name>mapred.hosts</name> |
---|
| 564 | <value></value> |
---|
| 565 | <description>Names a file that contains the list of nodes that may |
---|
| 566 | connect to the jobtracker. If the value is empty, all hosts are |
---|
| 567 | permitted.</description> |
---|
| 568 | </property> |
---|
| 569 | |
---|
| 570 | <property> |
---|
| 571 | <name>mapred.hosts.exclude</name> |
---|
| 572 | <value></value> |
---|
| 573 | <description>Names a file that contains the list of hosts that |
---|
| 574 | should be excluded by the jobtracker. If the value is empty, no |
---|
| 575 | hosts are excluded.</description> |
---|
| 576 | </property> |
---|
| 577 | |
---|
| 578 | <property> |
---|
| 579 | <name>mapred.max.tracker.blacklists</name> |
---|
| 580 | <value>4</value> |
---|
| 581 | <description>The number of blacklists for a taskTracker by various jobs |
---|
| 582 | after which the task tracker could be blacklisted across |
---|
| 583 | all jobs. The tracker will be given a tasks later |
---|
| 584 | (after a day). The tracker will become a healthy |
---|
| 585 | tracker after a restart. |
---|
| 586 | </description> |
---|
| 587 | </property> |
---|
| 588 | |
---|
| 589 | <property> |
---|
| 590 | <name>mapred.max.tracker.failures</name> |
---|
| 591 | <value>4</value> |
---|
| 592 | <description>The number of task-failures on a tasktracker of a given job |
---|
| 593 | after which new tasks of that job aren't assigned to it. |
---|
| 594 | </description> |
---|
| 595 | </property> |
---|
| 596 | |
---|
| 597 | <property> |
---|
| 598 | <name>jobclient.output.filter</name> |
---|
| 599 | <value>FAILED</value> |
---|
| 600 | <description>The filter for controlling the output of the task's userlogs sent |
---|
| 601 | to the console of the JobClient. |
---|
| 602 | The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and |
---|
| 603 | ALL. |
---|
| 604 | </description> |
---|
| 605 | </property> |
---|
| 606 | |
---|
| 607 | <property> |
---|
| 608 | <name>mapred.job.tracker.persist.jobstatus.active</name> |
---|
| 609 | <value>false</value> |
---|
| 610 | <description>Indicates if persistency of job status information is |
---|
| 611 | active or not. |
---|
| 612 | </description> |
---|
| 613 | </property> |
---|
| 614 | |
---|
| 615 | <property> |
---|
| 616 | <name>mapred.job.tracker.persist.jobstatus.hours</name> |
---|
| 617 | <value>0</value> |
---|
| 618 | <description>The number of hours job status information is persisted in DFS. |
---|
| 619 | The job status information will be available after it drops of the memory |
---|
| 620 | queue and between jobtracker restarts. With a zero value the job status |
---|
| 621 | information is not persisted at all in DFS. |
---|
| 622 | </description> |
---|
| 623 | </property> |
---|
| 624 | |
---|
| 625 | <property> |
---|
| 626 | <name>mapred.job.tracker.persist.jobstatus.dir</name> |
---|
| 627 | <value>/jobtracker/jobsInfo</value> |
---|
| 628 | <description>The directory where the job status information is persisted |
---|
| 629 | in a file system to be available after it drops of the memory queue and |
---|
| 630 | between jobtracker restarts. |
---|
| 631 | </description> |
---|
| 632 | </property> |
---|
| 633 | |
---|
| 634 | <property> |
---|
| 635 | <name>mapred.task.profile</name> |
---|
| 636 | <value>false</value> |
---|
| 637 | <description>To set whether the system should collect profiler |
---|
| 638 | information for some of the tasks in this job? The information is stored |
---|
| 639 | in the user log directory. The value is "true" if task profiling |
---|
| 640 | is enabled.</description> |
---|
| 641 | </property> |
---|
| 642 | |
---|
| 643 | <property> |
---|
| 644 | <name>mapred.task.profile.maps</name> |
---|
| 645 | <value>0-2</value> |
---|
| 646 | <description> To set the ranges of map tasks to profile. |
---|
| 647 | mapred.task.profile has to be set to true for the value to be accounted. |
---|
| 648 | </description> |
---|
| 649 | </property> |
---|
| 650 | |
---|
| 651 | <property> |
---|
| 652 | <name>mapred.task.profile.reduces</name> |
---|
| 653 | <value>0-2</value> |
---|
| 654 | <description> To set the ranges of reduce tasks to profile. |
---|
| 655 | mapred.task.profile has to be set to true for the value to be accounted. |
---|
| 656 | </description> |
---|
| 657 | </property> |
---|
| 658 | |
---|
| 659 | <property> |
---|
| 660 | <name>mapred.line.input.format.linespermap</name> |
---|
| 661 | <value>1</value> |
---|
| 662 | <description> Number of lines per split in NLineInputFormat. |
---|
| 663 | </description> |
---|
| 664 | </property> |
---|
| 665 | |
---|
| 666 | <property> |
---|
| 667 | <name>mapred.skip.attempts.to.start.skipping</name> |
---|
| 668 | <value>2</value> |
---|
| 669 | <description> The number of Task attempts AFTER which skip mode |
---|
| 670 | will be kicked off. When skip mode is kicked off, the |
---|
| 671 | tasks reports the range of records which it will process |
---|
| 672 | next, to the TaskTracker. So that on failures, TT knows which |
---|
| 673 | ones are possibly the bad records. On further executions, |
---|
| 674 | those are skipped. |
---|
| 675 | </description> |
---|
| 676 | </property> |
---|
| 677 | |
---|
| 678 | <property> |
---|
| 679 | <name>mapred.skip.map.auto.incr.proc.count</name> |
---|
| 680 | <value>true</value> |
---|
| 681 | <description> The flag which if set to true, |
---|
| 682 | SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented |
---|
| 683 | by MapRunner after invoking the map function. This value must be set to |
---|
| 684 | false for applications which process the records asynchronously |
---|
| 685 | or buffer the input records. For example streaming. |
---|
| 686 | In such cases applications should increment this counter on their own. |
---|
| 687 | </description> |
---|
| 688 | </property> |
---|
| 689 | |
---|
| 690 | <property> |
---|
| 691 | <name>mapred.skip.reduce.auto.incr.proc.count</name> |
---|
| 692 | <value>true</value> |
---|
| 693 | <description> The flag which if set to true, |
---|
| 694 | SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented |
---|
| 695 | by framework after invoking the reduce function. This value must be set to |
---|
| 696 | false for applications which process the records asynchronously |
---|
| 697 | or buffer the input records. For example streaming. |
---|
| 698 | In such cases applications should increment this counter on their own. |
---|
| 699 | </description> |
---|
| 700 | </property> |
---|
| 701 | |
---|
| 702 | <property> |
---|
| 703 | <name>mapred.skip.out.dir</name> |
---|
| 704 | <value></value> |
---|
| 705 | <description> If no value is specified here, the skipped records are |
---|
| 706 | written to the output directory at _logs/skip. |
---|
| 707 | User can stop writing skipped records by giving the value "none". |
---|
| 708 | </description> |
---|
| 709 | </property> |
---|
| 710 | |
---|
| 711 | <property> |
---|
| 712 | <name>mapred.skip.map.max.skip.records</name> |
---|
| 713 | <value>0</value> |
---|
| 714 | <description> The number of acceptable skip records surrounding the bad |
---|
| 715 | record PER bad record in mapper. The number includes the bad record as well. |
---|
| 716 | To turn the feature of detection/skipping of bad records off, set the |
---|
| 717 | value to 0. |
---|
| 718 | The framework tries to narrow down the skipped range by retrying |
---|
| 719 | until this threshold is met OR all attempts get exhausted for this task. |
---|
| 720 | Set the value to Long.MAX_VALUE to indicate that framework need not try to |
---|
| 721 | narrow down. Whatever records(depends on application) get skipped are |
---|
| 722 | acceptable. |
---|
| 723 | </description> |
---|
| 724 | </property> |
---|
| 725 | |
---|
| 726 | <property> |
---|
| 727 | <name>mapred.skip.reduce.max.skip.groups</name> |
---|
| 728 | <value>0</value> |
---|
| 729 | <description> The number of acceptable skip groups surrounding the bad |
---|
| 730 | group PER bad group in reducer. The number includes the bad group as well. |
---|
| 731 | To turn the feature of detection/skipping of bad groups off, set the |
---|
| 732 | value to 0. |
---|
| 733 | The framework tries to narrow down the skipped range by retrying |
---|
| 734 | until this threshold is met OR all attempts get exhausted for this task. |
---|
| 735 | Set the value to Long.MAX_VALUE to indicate that framework need not try to |
---|
| 736 | narrow down. Whatever groups(depends on application) get skipped are |
---|
| 737 | acceptable. |
---|
| 738 | </description> |
---|
| 739 | </property> |
---|
| 740 | |
---|
| 741 | <!-- Job Notification Configuration --> |
---|
| 742 | |
---|
| 743 | <!-- |
---|
| 744 | <property> |
---|
| 745 | <name>job.end.notification.url</name> |
---|
| 746 | <value>http://localhost:8080/jobstatus.php?jobId=$jobId&jobStatus=$jobStatus</value> |
---|
| 747 | <description>Indicates url which will be called on completion of job to inform |
---|
| 748 | end status of job. |
---|
| 749 | User can give at most 2 variables with URI : $jobId and $jobStatus. |
---|
| 750 | If they are present in URI, then they will be replaced by their |
---|
| 751 | respective values. |
---|
| 752 | </description> |
---|
| 753 | </property> |
---|
| 754 | --> |
---|
| 755 | |
---|
| 756 | <property> |
---|
| 757 | <name>job.end.retry.attempts</name> |
---|
| 758 | <value>0</value> |
---|
| 759 | <description>Indicates how many times hadoop should attempt to contact the |
---|
| 760 | notification URL </description> |
---|
| 761 | </property> |
---|
| 762 | |
---|
| 763 | <property> |
---|
| 764 | <name>job.end.retry.interval</name> |
---|
| 765 | <value>30000</value> |
---|
| 766 | <description>Indicates time in milliseconds between notification URL retry |
---|
| 767 | calls</description> |
---|
| 768 | </property> |
---|
| 769 | |
---|
| 770 | <!-- Proxy Configuration --> |
---|
| 771 | <property> |
---|
| 772 | <name>hadoop.rpc.socket.factory.class.JobSubmissionProtocol</name> |
---|
| 773 | <value></value> |
---|
| 774 | <description> SocketFactory to use to connect to a Map/Reduce master |
---|
| 775 | (JobTracker). If null or empty, then use hadoop.rpc.socket.class.default. |
---|
| 776 | </description> |
---|
| 777 | </property> |
---|
| 778 | |
---|
| 779 | <property> |
---|
| 780 | <name>mapred.task.cache.levels</name> |
---|
| 781 | <value>2</value> |
---|
| 782 | <description> This is the max level of the task cache. For example, if |
---|
| 783 | the level is 2, the tasks cached are at the host level and at the rack |
---|
| 784 | level. |
---|
| 785 | </description> |
---|
| 786 | </property> |
---|
| 787 | |
---|
| 788 | <property> |
---|
| 789 | <name>mapred.queue.names</name> |
---|
| 790 | <value>default</value> |
---|
| 791 | <description> Comma separated list of queues configured for this jobtracker. |
---|
| 792 | Jobs are added to queues and schedulers can configure different |
---|
| 793 | scheduling properties for the various queues. To configure a property |
---|
| 794 | for a queue, the name of the queue must match the name specified in this |
---|
| 795 | value. Queue properties that are common to all schedulers are configured |
---|
| 796 | here with the naming convention, mapred.queue.$QUEUE-NAME.$PROPERTY-NAME, |
---|
| 797 | for e.g. mapred.queue.default.submit-job-acl. |
---|
| 798 | The number of queues configured in this parameter could depend on the |
---|
| 799 | type of scheduler being used, as specified in |
---|
| 800 | mapred.jobtracker.taskScheduler. For example, the JobQueueTaskScheduler |
---|
| 801 | supports only a single queue, which is the default configured here. |
---|
| 802 | Before adding more queues, ensure that the scheduler you've configured |
---|
| 803 | supports multiple queues. |
---|
| 804 | </description> |
---|
| 805 | </property> |
---|
| 806 | |
---|
| 807 | <property> |
---|
| 808 | <name>mapred.acls.enabled</name> |
---|
| 809 | <value>false</value> |
---|
| 810 | <description> Specifies whether ACLs are enabled, and should be checked |
---|
| 811 | for various operations. |
---|
| 812 | </description> |
---|
| 813 | </property> |
---|
| 814 | |
---|
| 815 | <property> |
---|
| 816 | <name>mapred.queue.default.acl-submit-job</name> |
---|
| 817 | <value>*</value> |
---|
| 818 | <description> Comma separated list of user and group names that are allowed |
---|
| 819 | to submit jobs to the 'default' queue. The user list and the group list |
---|
| 820 | are separated by a blank. For e.g. alice,bob group1,group2. |
---|
| 821 | If set to the special value '*', it means all users are allowed to |
---|
| 822 | submit jobs. |
---|
| 823 | </description> |
---|
| 824 | </property> |
---|
| 825 | |
---|
| 826 | <property> |
---|
| 827 | <name>mapred.queue.default.acl-administer-jobs</name> |
---|
| 828 | <value>*</value> |
---|
| 829 | <description> Comma separated list of user and group names that are allowed |
---|
| 830 | to delete jobs or modify job's priority for jobs not owned by the current |
---|
| 831 | user in the 'default' queue. The user list and the group list |
---|
| 832 | are separated by a blank. For e.g. alice,bob group1,group2. |
---|
| 833 | If set to the special value '*', it means all users are allowed to do |
---|
| 834 | this operation. |
---|
| 835 | </description> |
---|
| 836 | </property> |
---|
| 837 | |
---|
| 838 | <property> |
---|
| 839 | <name>mapred.job.queue.name</name> |
---|
| 840 | <value>default</value> |
---|
| 841 | <description> Queue to which a job is submitted. This must match one of the |
---|
| 842 | queues defined in mapred.queue.names for the system. Also, the ACL setup |
---|
| 843 | for the queue must allow the current user to submit a job to the queue. |
---|
| 844 | Before specifying a queue, ensure that the system is configured with |
---|
| 845 | the queue, and access is allowed for submitting jobs to the queue. |
---|
| 846 | </description> |
---|
| 847 | </property> |
---|
| 848 | |
---|
| 849 | <property> |
---|
| 850 | <name>mapred.tasktracker.indexcache.mb</name> |
---|
| 851 | <value>10</value> |
---|
| 852 | <description> The maximum memory that a task tracker allows for the |
---|
| 853 | index cache that is used when serving map outputs to reducers. |
---|
| 854 | </description> |
---|
| 855 | </property> |
---|
| 856 | |
---|
| 857 | <property> |
---|
| 858 | <name>mapred.merge.recordsBeforeProgress</name> |
---|
| 859 | <value>10000</value> |
---|
| 860 | <description> The number of records to process during merge before |
---|
| 861 | sending a progress notification to the TaskTracker. |
---|
| 862 | </description> |
---|
| 863 | </property> |
---|
| 864 | |
---|
| 865 | <property> |
---|
| 866 | <name>mapred.reduce.slowstart.completed.maps</name> |
---|
| 867 | <value>0.05</value> |
---|
| 868 | <description>Fraction of the number of maps in the job which should be |
---|
| 869 | complete before reduces are scheduled for the job. |
---|
| 870 | </description> |
---|
| 871 | </property> |
---|
| 872 | |
---|
| 873 | </configuration> |
---|