final. @param name resource to be added, the classpath is examined for a file with that name.]]> final. @param url url of the resource to be added, the local filesystem is examined directly to find the resource, without referring to the classpath.]]> final. @param file file-path of resource to be added, the local filesystem is examined directly to find the resource, without referring to the classpath.]]> name property, null if no such property exists. Values are processed for variable expansion before being returned. @param name the property name. @return the value of the name property, or null if no such property exists.]]> name property, without doing variable expansion. @param name the property name. @return the value of the name property, or null if no such property exists.]]> value of the name property. @param name property name. @param value property value.]]> name property. If no such property exists, then defaultValue is returned. @param name property name. @param defaultValue default value. @return property value, or defaultValue if the property doesn't exist.]]> name property as an int. If no such property exists, or if the specified value is not a valid int, then defaultValue is returned. @param name property name. @param defaultValue default value. @return property value as an int, or defaultValue.]]> name property to an int. @param name property name. @param value int value of the property.]]> name property as a long. If no such property is specified, or if the specified value is not a valid long, then defaultValue is returned. @param name property name. @param defaultValue default value. @return property value as a long, or defaultValue.]]> name property to a long. @param name property name. @param value long value of the property.]]> name property as a float. If no such property is specified, or if the specified value is not a valid float, then defaultValue is returned. @param name property name. @param defaultValue default value. @return property value as a float, or defaultValue.]]> name property as a boolean. If no such property is specified, or if the specified value is not a valid boolean, then defaultValue is returned. @param name property name. @param defaultValue default value. @return property value as a boolean, or defaultValue.]]> name property to a boolean. @param name property name. @param value boolean value of the property.]]> name property as a collection of Strings. If no such property is specified then empty collection is returned.

This is an optimized version of {@link #getStrings(String)} @param name property name. @return property value as a collection of Strings.]]> name property as an array of Strings. If no such property is specified then null is returned. @param name property name. @return property value as an array of Strings, or null.]]> name property as an array of Strings. If no such property is specified then default value is returned. @param name property name. @param defaultValue The default value @return property value as an array of Strings, or default value.]]> name property as as comma delimited values. @param name property name. @param values The values]]> name property as a Class. If no such property is specified, then defaultValue is returned. @param name the class name. @param defaultValue default value. @return property value as a Class, or defaultValue.]]> name property as a Class implementing the interface specified by xface. If no such property is specified, then defaultValue is returned. An exception is thrown if the returned class does not implement the named interface. @param name the class name. @param defaultValue default value. @param xface the interface implemented by the named class. @return property value as a Class, or defaultValue.]]> name property to the name of a theClass implementing the given interface xface. An exception is thrown if theClass does not implement the interface xface. @param name property name. @param theClass property value. @param xface the interface implemented by the named class.]]> dirsProp with the given path. If dirsProp contains multiple directories, then one is chosen based on path's hash code. If the selected directory does not exist, an attempt is made to create it. @param dirsProp directory in which to locate the file. @param path file-path. @return local file under the directory with the given path.]]> dirsProp with the given path. If dirsProp contains multiple directories, then one is chosen based on path's hash code. If the selected directory does not exist, an attempt is made to create it. @param dirsProp directory in which to locate the file. @param path file-path. @return local file under the directory with the given path.]]> name. @param name configuration resource name. @return an input stream attached to the resource.]]> name. @param name configuration resource name. @return a reader attached to the resource.]]> String key-value pairs in the configuration. @return an iterator over the entries.]]> true to set quiet-mode on, false to turn it off.]]> Resources

Configurations are specified by resources. A resource contains a set of name/value pairs as XML data. Each resource is named by either a String or by a {@link Path}. If named by a String, then the classpath is examined for a file with that name. If named by a Path, then the local filesystem is examined directly, without referring to the classpath.

Hadoop by default specifies two resources, loaded in-order from the classpath:

  1. hadoop-default.xml : Read-only defaults for hadoop.
  2. hadoop-site.xml: Site-specific configuration for a given hadoop installation.
Applications may add additional resources, which are loaded subsequent to these resources in the order they are added.

Final Parameters

Configuration parameters may be declared final. Once a resource declares a value final, no subsequently-loaded resource can alter that value. For example, one might define a final parameter with:

  <property>
    <name>dfs.client.buffer.dir</name>
    <value>/tmp/hadoop/dfs/client</value>
    <final>true</final>
  </property>
Administrators typically define parameters as final in hadoop-site.xml for values that user applications may not alter.

Variable Expansion

Value strings are first processed for variable expansion. The available properties are:

  1. Other properties defined in this Configuration; and, if a name is undefined here,
  2. Properties in {@link System#getProperties()}.

For example, if a configuration resource contains the following property definitions:

  <property>
    <name>basedir</name>
    <value>/user/${user.name}</value>
  </property>
  
  <property>
    <name>tempdir</name>
    <value>${basedir}/tmp</value>
  </property>
When conf.get("tempdir") is called, then ${basedir} will be resolved to another property in this Configuration, while ${user.name} would then ordinarily be resolved to the value of the System property with that name.]]>
The balancer is a tool that balances disk space usage on an HDFS cluster when some datanodes become full or when new empty nodes join the cluster. The tool is deployed as an application program that can be run by the cluster administrator on a live HDFS cluster while applications adding and deleting files.

SYNOPSIS

 To start:
      bin/start-balancer.sh [-threshold ]
      Example: bin/ start-balancer.sh 
                     start the balancer with a default threshold of 10%
               bin/ start-balancer.sh -threshold 5
                     start the balancer with a threshold of 5%
 To stop:
      bin/ stop-balancer.sh
 

DESCRIPTION

The threshold parameter is a fraction in the range of (0%, 100%) with a default value of 10%. The threshold sets a target for whether the cluster is balanced. A cluster is balanced if for each datanode, the utilization of the node (ratio of used space at the node to total capacity of the node) differs from the utilization of the (ratio of used space in the cluster to total capacity of the cluster) by no more than the threshold value. The smaller the threshold, the more balanced a cluster will become. It takes more time to run the balancer for small threshold values. Also for a very small threshold the cluster may not be able to reach the balanced state when applications write and delete files concurrently.

The tool moves blocks from highly utilized datanodes to poorly utilized datanodes iteratively. In each iteration a datanode moves or receives no more than the lesser of 10G bytes or the threshold fraction of its capacity. Each iteration runs no more than 20 minutes. At the end of each iteration, the balancer obtains updated datanodes information from the namenode.

A system property that limits the balancer's use of bandwidth is defined in the default configuration file:

 
   dfs.balance.bandwidthPerSec
   1048576
   Specifies the maximum bandwidth that each datanode 
 can utilize for the balancing purpose in term of the number of bytes 
 per second. 
 
 

This property determines the maximum speed at which a block will be moved from one datanode to another. The default value is 1MB/s. The higher the bandwidth, the faster a cluster can reach the balanced state, but with greater competition with application processes. If an administrator changes the value of this property in the configuration file, the change is observed when HDFS is next restarted.

MONITERING BALANCER PROGRESS

After the balancer is started, an output file name where the balancer progress will be recorded is printed on the screen. The administrator can monitor the running of the balancer by reading the output file. The output shows the balancer's status iteration by iteration. In each iteration it prints the starting time, the iteration number, the total number of bytes that have been moved in the previous iterations, the total number of bytes that are left to move in order for the cluster to be balanced, and the number of bytes that are being moved in this iteration. Normally "Bytes Already Moved" is increasing while "Bytes Left To Move" is decreasing.

Running multiple instances of the balancer in an HDFS cluster is prohibited by the tool.

The balancer automatically exits when any of the following five conditions is satisfied:

  1. The cluster is balanced;
  2. No block can be moved;
  3. No block has been moved for five consecutive iterations;
  4. An IOException occurs while communicating with the namenode;
  5. Another balancer is running.

Upon exit, a balancer returns an exit code and prints one of the following messages to the output file in corresponding to the above exit reasons:

  1. The cluster is balanced. Exiting
  2. No block can be moved. Exiting...
  3. No block has been moved for 3 iterations. Exiting...
  4. Received an IO exception: failure reason. Exiting...
  5. Another balancer is running. Exiting...

The administrator can interrupt the execution of the balancer at any time by running the command "stop-balancer.sh" on the machine where the balancer is running.]]> in]]> out.]]> reset is true, then resets the checksum. @return number of bytes written. Will be equal to getChecksumSize();]]> reset is true, then resets the checksum. @return number of bytes written. Will be equal to getChecksumSize();]]> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.]]> The tool scans all files and directories, starting from an indicated root path. The following abnormal conditions are detected and handled:

  • files with blocks that are completely missing from all datanodes.
    In this case the tool can perform one of the following actions:
    • none ({@link NamenodeFsck#FIXING_NONE})
    • move corrupted files to /lost+found directory on DFS ({@link NamenodeFsck#FIXING_MOVE}). Remaining data blocks are saved as a block chains, representing longest consecutive series of valid blocks.
    • delete corrupted files ({@link NamenodeFsck#FIXING_DELETE})
  • detect files with under-replicated or over-replicated blocks
Additionally, the tool collects a detailed overall DFS statistics, and optionally can print detailed statistics on block locations and replication factors of each file. The tool also provides and option to filter open files during the scan.]]>
:/data[/] HTTP/1.1 }]]> This class has a number of metrics variables that are publicly accessible; these variables (objects) have methods to update their values; for example:

{@link #filesTotal}.set()]]> :/listPaths[/][[&option]*] HTTP/1.1 } Where option (default) in: recursive ("no") filter (".*") exclude ("\..*\.crc") Response: A flat list of files/directories in the following format: {@code

}]]>
The name-node can be started with one of the following startup options:
  • {@link FSConstants.StartupOption#REGULAR REGULAR} - normal startup
  • {@link FSConstants.StartupOption#FORMAT FORMAT} - format name node
  • {@link FSConstants.StartupOption#UPGRADE UPGRADE} - start the cluster upgrade and create a snapshot of the current file system state
  • {@link FSConstants.StartupOption#ROLLBACK ROLLBACK} - roll the cluster back to the previous state
The option is passed via configuration field: dfs.namenode.startup The conf will be modified to reflect the actual ports on which the NameNode is up and running if the user passes the port as zero in the conf. @param conf confirguration @throws IOException]]>
zero.]]> datanode whose total size is size @param datanode on which blocks are located @param size total size of blocks]]> blocksequence (namespace) 2) block->machinelist ("inodes") The first table is stored on disk and is very precious. The second table is rebuilt every time the NameNode comes up. 'NameNode' refers to both this class as well as the 'NameNode server'. The 'FSNamesystem' class actually performs most of the filesystem management. The majority of the 'NameNode' class itself is concerned with exposing the IPC interface to the outside world, plus some configuration management. NameNode implements the ClientProtocol interface, which allows clients to ask for DFS services. ClientProtocol is not designed for direct use by authors of DFS client code. End-users should instead use the org.apache.nutch.hadoop.fs.FileSystem class. NameNode also implements the DatanodeProtocol interface, used by DataNode programs that actually store DFS data blocks. These methods are invoked repeatedly and automatically by all the DataNodes in a DFS deployment. NameNode also implements the NamenodeProtocol interface, used by secondary namenodes or rebalancing processes to get partial namenode's state, for example partial blocksMap etc.]]>
The tool scans all files and directories, starting from an indicated root path. The following abnormal conditions are detected and handled:

  • files with blocks that are completely missing from all datanodes.
    In this case the tool can perform one of the following actions:
    • none ({@link #FIXING_NONE})
    • move corrupted files to /lost+found directory on DFS ({@link #FIXING_MOVE}). Remaining data blocks are saved as a block chains, representing longest consecutive series of valid blocks.
    • delete corrupted files ({@link #FIXING_DELETE})
  • detect files with under-replicated or over-replicated blocks
Additionally, the tool collects a detailed overall DFS statistics, and optionally can print detailed statistics on block locations and replication factors of each file.]]>
This class has a number of metrics variables that are publicly accessible; these variables (objects) have methods to update their values; for example:

{@link #syncs}.inc()]]> This class has a number of metrics variables that are publicly accessible; these variables (objects) have methods to update their values; for example:

{@link #blocksRead}.inc()]]> For the statistics that are sampled and averaged, one must specify a metrics context that does periodic update calls. Most do. The default Null metrics context however does NOT. So if you aren't using any other metrics context then you can turn on the viewing and averaging of sampled metrics by specifying the following two lines in the hadoop-meterics.properties file:

        dfs.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
        dfs.period=10
  

Note that the metrics are collected regardless of the context used. The context with the update thread is used to average the data periodically.

Name Node Status info is reported in another MBean @see org.apache.hadoop.dfs.datanode.metrics.FSDatasetMBean]]> Data Node runtime statistic info is report in another MBean @see org.apache.hadoop.dfs.datanode.metrics.DataNodeStatisticsMBean]]> Name Node runtime statistic info is report in another MBean @see org.apache.hadoop.dfs.namenode.metrics.NameNodeStatisticsMBean]]> For the statistics that are sampled and averaged, one must specify a metrics context that does periodic update calls. Most do. The default Null metrics context however does NOT. So if you aren't using any other metrics context then you can turn on the viewing and averaging of sampled metrics by specifying the following two lines in the hadoop-meterics.properties file:

        dfs.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
        dfs.period=10
  

Note that the metrics are collected regardless of the context used. The context with the update thread is used to average the data periodically.

Name Node Status info is report in another MBean @see org.apache.hadoop.dfs.namenode.metrics.FSNamesystemMBean]]> DistributedCache is a facility provided by the Map-Reduce framework to cache files (text, archives, jars etc.) needed by applications.

Applications specify the files, via urls (hdfs:// or http://) to be cached via the {@link JobConf}. The DistributedCache assumes that the files specified via hdfs:// urls are already present on the {@link FileSystem} at the path specified by the url.

The framework will copy the necessary files on to the slave node before any tasks for the job are executed on that node. Its efficiency stems from the fact that the files are only copied once per job and the ability to cache archives which are un-archived on the slaves.

DistributedCache can be used to distribute simple, read-only data/text files and/or more complex types such as archives, jars etc. Archives (zip, tar and tgz/tar.gz files) are un-archived at the slave nodes. Jars may be optionally added to the classpath of the tasks, a rudimentary software distribution mechanism. Files have execution permissions. Optionally users can also direct it to symlink the distributed cache file(s) into the working directory of the task.

DistributedCache tracks modification timestamps of the cache files. Clearly the cache files should not be modified by the application or externally while the job is executing.

Here is an illustrative example on how to use the DistributedCache:

     // Setting up the cache for the application
     
     1. Copy the requisite files to the FileSystem:
     
     $ bin/hadoop fs -copyFromLocal lookup.dat /myapp/lookup.dat  
     $ bin/hadoop fs -copyFromLocal map.zip /myapp/map.zip  
     $ bin/hadoop fs -copyFromLocal mylib.jar /myapp/mylib.jar
     $ bin/hadoop fs -copyFromLocal mytar.tar /myapp/mytar.tar
     $ bin/hadoop fs -copyFromLocal mytgz.tgz /myapp/mytgz.tgz
     $ bin/hadoop fs -copyFromLocal mytargz.tar.gz /myapp/mytargz.tar.gz
     
     2. Setup the application's JobConf:
     
     JobConf job = new JobConf();
     DistributedCache.addCacheFile(new URI("/myapp/lookup.dat#lookup.dat"), 
                                   job);
     DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job);
     DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job);
     
     3. Use the cached files in the {@link Mapper} or {@link Reducer}:
     
     public static class MapClass extends MapReduceBase  
     implements Mapper<K, V, K, V> {
     
       private Path[] localArchives;
       private Path[] localFiles;
       
       public void configure(JobConf job) {
         // Get the cached archives/files
         localArchives = DistributedCache.getLocalCacheArchives(job);
         localFiles = DistributedCache.getLocalCacheFiles(job);
       }
       
       public void map(K key, V value, 
                       OutputCollector<K, V> output, Reporter reporter) 
       throws IOException {
         // Use data from the cached archives/files here
         // ...
         // ...
         output.collect(k, v);
       }
     }
     
 

@see JobConf @see JobClient]]>
BufferedFSInputStream with the specified buffer size, and saves its argument, the input stream in, for later use. An internal buffer array of length size is created and stored in buf. @param in the underlying input stream. @param size the buffer size. @exception IllegalArgumentException if size <= 0.]]> setReplication of FileSystem @param src file name @param replication new replication @throws IOException @return true if successful; false if file does not exist or is a directory]]> ']]> fs.scheme.class whose value names the FileSystem class. The entire URI is passed to the FileSystem instance's initialize method.]]> Return all the files that match filePattern and are not checksum files. Results are sorted by their names.

A filename pattern is composed of regular characters and special pattern matching characters, which are:

?
Matches any single character.

*
Matches zero or more characters.

[abc]
Matches a single character from character set {a,b,c}.

[a-b]
Matches a single character from the character range {a...b}. Note that character a must be lexicographically less than or equal to character b.

[^a]
Matches a single character that is not from character set or range {a}. Note that the ^ character must occur immediately to the right of the opening bracket.

\c
Removes (escapes) any special meaning of character c.

{ab,cd}
Matches a string from the string set {ab, cd}

{ab,c{de,fh}}
Matches a string from the string set {ab, cde, cfh}
@param pathPattern a regular expression specifying a pth pattern @return an array of paths that match the path pattern @throws IOException]]>
All user code that may potentially use the Hadoop Distributed File System should be written to use a FileSystem object. The Hadoop DFS is a multi-machine system that appears as a single disk. It's useful because of its fault tolerance and potentially very large capacity.

The local implementation is {@link LocalFileSystem} and distributed implementation is {@link DistributedFileSystem}.]]> FilterFileSystem contains some other file system, which it uses as its basic file system, possibly transforming the data along the way or providing additional functionality. The class FilterFileSystem itself simply overrides all methods of FileSystem with versions that pass all requests to the contained file system. Subclasses of FilterFileSystem may further override some of these methods and may also provide additional methods and fields.]]> buf at offset and checksum into checksum. The method is used for implementing read, therefore, it should be optimized for sequential reading @param pos chunkPos @param buf desitination buffer @param offset offset in buf at which to store data @param len maximun number of bytes to read @return number of bytes read]]> -1 if the end of the stream is reached. @exception IOException if an I/O error occurs.]]> This method implements the general contract of the corresponding {@link InputStream#read(byte[], int, int) read} method of the {@link InputStream} class. As an additional convenience, it attempts to read as many bytes as possible by repeatedly invoking the read method of the underlying stream. This iterated read continues until one of the following conditions becomes true:

  • The specified number of bytes have been read,
  • The read method of the underlying stream returns -1, indicating end-of-file.
If the first read on the underlying stream returns -1 to indicate end-of-file then this method returns -1. Otherwise this method returns the number of bytes actually read. @param b destination buffer. @param off offset at which to start storing bytes. @param len maximum number of bytes to read. @return the number of bytes read, or -1 if the end of the stream has been reached. @exception IOException if an I/O error occurs. ChecksumException if any checksum error occurs]]>
n bytes of data from the input stream.

This method may skip more bytes than are remaining in the backing file. This produces no exception and the number of bytes skipped may include some number of bytes that were beyond the EOF of the backing file. Attempting to read from the stream after skipping past the end will result in -1 indicating the end of the file.

If n is negative, no bytes are skipped. @param n the number of bytes to be skipped. @return the actual number of bytes skipped. @exception IOException if an I/O error occurs. ChecksumException if the chunk to skip to is corrupted]]> This method may seek past the end of the file. This produces no exception and an attempt to read from the stream will result in -1 indicating the end of the file. @param pos the postion to seek to. @exception IOException if an I/O error occurs. ChecksumException if the chunk to seek to is corrupted]]> len bytes from stm @param stm an input stream @param buf destiniation buffer @param offset offset at which to store data @param len number of bytes to read @return actual number of bytes read @throws IOException if there is any IO error]]> len bytes from the specified byte array starting at offset off and generate a checksum for each data chunk.

This method stores bytes from the given array into this stream's buffer before it gets checksumed. The buffer gets checksumed and flushed to the underlying output stream when all data in a checksum chunk are in the buffer. If the buffer is empty and requested length is at least as large as the size of next checksum chunk size, this method will checksum and write the chunk directly to the underlying output stream. Thus it avoids uneccessary data copy. @param b the data. @param off the start offset in the data. @param len the number of bytes to write. @exception IOException if an I/O error occurs.]]> true if and only if pathname should be included]]> trash feature. Files are moved to a user's trash directory, a subdirectory of their home directory named ".Trash". Files are initially moved to a current sub-directory of the trash directory. Within that sub-directory their original path is preserved. Periodically one may checkpoint the current trash and remove older checkpoints. (This design permits trash management without enumeration of the full trash content, without date support in the filesystem, and without clock synchronization.)]]> A {@link FileSystem} backed by an FTP client provided by Apache Commons Net.

]]>
This class is a tool for migrating data from an older to a newer version of an S3 filesystem.

All files in the filesystem are migrated by re-writing the block metadata - no datafiles are touched.

]]>
Extracts AWS credentials from the filesystem URI or configuration.

]]>
A block-based {@link FileSystem} backed by Amazon S3.

@see NativeS3FileSystem]]>
If f is a file, this method will make a single call to S3. If f is a directory, this method will make a maximum of (n / 1000) + 2 calls to S3, where n is the total number of files and directories contained directly in f.

]]>
A {@link FileSystem} for reading and writing files stored on Amazon S3. Unlike {@link org.apache.hadoop.fs.s3.S3FileSystem} this implementation stores files on S3 in their native form so they can be read by other S3 tools.

@see org.apache.hadoop.fs.s3.S3FileSystem]]>
nth value.]]> nth value in the file.]]> public class IntArrayWritable extends ArrayWritable { public IntArrayWritable() { super(IntWritable.class); } } ]]> o is a ByteWritable with the same value.]]> This saves memory over creating a new DataInputStream and ByteArrayInputStream each time data is read.

Typical usage is something like the following:


 DataInputBuffer buffer = new DataInputBuffer();
 while (... loop condition ...) {
   byte[] data = ... get data ...;
   int dataLength = ... get data length ...;
   buffer.reset(data, dataLength);
   ... read buffer using DataInput methods ...
 }
 
]]>
This saves memory over creating a new DataOutputStream and ByteArrayOutputStream each time data is written.

Typical usage is something like the following:


 DataOutputBuffer buffer = new DataOutputBuffer();
 while (... loop condition ...) {
   buffer.reset();
   ... write buffer using DataOutput methods ...
   byte[] data = buffer.getData();
   int dataLength = buffer.getLength();
   ... write data to its ultimate destination ...
 }
 
]]>
the class of the item @param conf the configuration to store @param item the object to be stored @param keyName the name of the key to use @throws IOException : forwards Exceptions from the underlying {@link Serialization} classes.]]> the class of the item @param conf the configuration to use @param keyName the name of the key to use @param itemClass the class of the item @return restored object @throws IOException : forwards Exceptions from the underlying {@link Serialization} classes.]]> the class of the item @param conf the configuration to use @param items the objects to be stored @param keyName the name of the key to use @throws IndexOutOfBoundsException if the items array is empty @throws IOException : forwards Exceptions from the underlying {@link Serialization} classes.]]> the class of the item @param conf the configuration to use @param keyName the name of the key to use @param itemClass the class of the item @return restored object @throws IOException : forwards Exceptions from the underlying {@link Serialization} classes.]]> DefaultStringifier offers convenience methods to store/load objects to/from the configuration. @param the class of the objects to stringify]]> o is a DoubleWritable with the same value.]]> o is a FloatWritable with the same value.]]> When two sequence files, which have same Key type but different Value types, are mapped out to reduce, multiple Value types is not allowed. In this case, this class can help you wrap instances with different types.

Compared with ObjectWritable, this class is much more effective, because ObjectWritable will append the class declaration as a String into the output file in every Key-Value pair.

Generic Writable implements {@link Configurable} interface, so that it will be configured by the framework. The configuration is passed to the wrapped objects implementing {@link Configurable} interface before deserialization.

how to use it:
1. Write your own class, such as GenericObject, which extends GenericWritable.
2. Implements the abstract method getTypes(), defines the classes which will be wrapped in GenericObject in application. Attention: this classes defined in getTypes() method, must implement Writable interface.

The code looks like this:
 public class GenericObject extends GenericWritable {
 
   private static Class[] CLASSES = {
               ClassType1.class, 
               ClassType2.class,
               ClassType3.class,
               };

   protected Class[] getTypes() {
       return CLASSES;
   }

 }
 
@since Nov 8, 2006]]>
This saves memory over creating a new InputStream and ByteArrayInputStream each time data is read.

Typical usage is something like the following:


 InputBuffer buffer = new InputBuffer();
 while (... loop condition ...) {
   byte[] data = ... get data ...;
   int dataLength = ... get data length ...;
   buffer.reset(data, dataLength);
   ... read buffer using InputStream methods ...
 }
 
@see DataInputBuffer @see DataOutput]]>
o is a IntWritable with the same value.]]> closes the input and output streams at the end. @param in InputStrem to read from @param out OutputStream to write to @param conf the Configuration object]]> ignore any {@link IOException} or null pointers. Must only be used for cleanup in exception handlers. @param log the log to record problems to at debug level. Can be null. @param closeables the objects to close]]> o is a LongWritable with the same value.]]> A map is a directory containing two files, the data file, containing all keys and values in the map, and a smaller index file, containing a fraction of the keys. The fraction is determined by {@link Writer#getIndexInterval()}.

The index file is read entirely into memory. Thus key implementations should try to keep themselves small.

Map files are created by adding entries in-order. To maintain a large database, perform updates by copying the previous version of a database and merging in a sorted change list, to create a new version of the database in a new file. Sorting large change lists can be done with {@link SequenceFile.Sorter}.]]> key and val. Returns true if such a pair exists and false when at the end of the map]]> key or if it does not exist, at the first entry after the named key. - * @param key - key that we're trying to find - * @param val - data value if key is found - * @return - the key that was the closest match or null if eof.]]> key does not exist, return the first entry that falls just before the key. Otherwise, return the record that sorts just after. @return - the key that was the closest match or null if eof.]]> o is an MD5Hash whose digest contains the same values.]]> This saves memory over creating a new OutputStream and ByteArrayOutputStream each time data is written.

Typical usage is something like the following:


 OutputBuffer buffer = new OutputBuffer();
 while (... loop condition ...) {
   buffer.reset();
   ... write buffer using OutputStream methods ...
   byte[] data = buffer.getData();
   int dataLength = buffer.getLength();
   ... write data to its ultimate destination ...
 }
 
@see DataOutputBuffer @see InputBuffer]]>
A {@link Comparator} that operates directly on byte representations of objects.

@param @see DeserializerComparator]]>
SequenceFiles are flat files consisting of binary key/value pairs.

SequenceFile provides {@link Writer}, {@link Reader} and {@link Sorter} classes for writing, reading and sorting respectively.

There are three SequenceFile Writers based on the {@link CompressionType} used to compress key/value pairs:
  1. Writer : Uncompressed records.
  2. RecordCompressWriter : Record-compressed files, only compress values.
  3. BlockCompressWriter : Block-compressed files, both keys & values are collected in 'blocks' separately and compressed. The size of the 'block' is configurable.

The actual compression algorithm used to compress key and/or values can be specified by using the appropriate {@link CompressionCodec}.

The recommended way is to use the static createWriter methods provided by the SequenceFile to chose the preferred format.

The {@link Reader} acts as the bridge and can read any of the above SequenceFile formats.

SequenceFile Formats

Essentially there are 3 different formats for SequenceFiles depending on the CompressionType specified. All of them share a common header described below.

  • version - 3 bytes of magic header SEQ, followed by 1 byte of actual version number (e.g. SEQ4 or SEQ6)
  • keyClassName -key class
  • valueClassName - value class
  • compression - A boolean which specifies if compression is turned on for keys/values in this file.
  • blockCompression - A boolean which specifies if block-compression is turned on for keys/values in this file.
  • compression codec - CompressionCodec class which is used for compression of keys and/or values (if compression is enabled).
  • metadata - {@link Metadata} for this file.
  • sync - A sync marker to denote end of the header.
Uncompressed SequenceFile Format
  • Header
  • Record
    • Record length
    • Key length
    • Key
    • Value
  • A sync-marker every few 100 bytes or so.
Record-Compressed SequenceFile Format
  • Header
  • Record
    • Record length
    • Key length
    • Key
    • Compressed Value
  • A sync-marker every few 100 bytes or so.
Block-Compressed SequenceFile Format
  • Header
  • Record Block
    • Compressed key-lengths block-size
    • Compressed key-lengths block
    • Compressed keys block-size
    • Compressed keys block
    • Compressed value-lengths block-size
    • Compressed value-lengths block
    • Compressed values block-size
    • Compressed values block
  • A sync-marker every few 100 bytes or so.

The compressed blocks of key lengths and value lengths consist of the actual lengths of individual keys/values encoded in ZeroCompressedInteger format.

@see CompressionCodec]]>
key, skipping its value. True if another entry exists, and false at end of file.]]> key and val. Returns true if such a pair exists and false when at end of file]]> The position passed must be a position returned by {@link SequenceFile.Writer#getLength()} when writing this file. To seek to an arbitrary position, use {@link SequenceFile.Reader#sync(long)}.]]> SegmentDescriptor @param segments the list of SegmentDescriptors @param tmpDir the directory to write temporary files into @return RawKeyValueIterator @throws IOException]]> For best performance, applications should make sure that the {@link Writable#readFields(DataInput)} implementation of their keys is very efficient. In particular, it should avoid allocating memory.]]> This always returns a synchronized position. In other words, immediately after calling {@link SequenceFile.Reader#seek(long)} with a position returned by this method, {@link SequenceFile.Reader#next(Writable)} may be called. However the key may be earlier in the file than key last written when this method was called (e.g., with block-compression, it may be the first key in the block that was being written when this method was called).]]> key. Returns true if such a key exists and false when at the end of the set.]]> key. Returns key, or null if no match exists.]]> the class of the objects to stringify]]> position. Note that this method avoids using the converter or doing String instatiation @return the Unicode scalar value at position or -1 if the position is invalid or points to a trailing byte]]> what in the backing buffer, starting as position start. The starting position is measured in bytes and the return value is in terms of byte position in the buffer. The backing buffer is not converted to a string for this operation. @return byte position of the first occurence of the search string in the UTF-8 buffer or -1 if not found]]> o is a Text with the same contents.]]> replace is true, then malformed input is replaced with the substitution character, which is U+FFFD. Otherwise the method throws a MalformedInputException.]]> replace is true, then malformed input is replaced with the substitution character, which is U+FFFD. Otherwise the method throws a MalformedInputException. @return ByteBuffer: bytes stores at ByteBuffer.array() and length is ByteBuffer.limit()]]> In addition, it provides methods for string traversal without converting the byte array to a string.

Also includes utilities for serializing/deserialing a string, coding/decoding a string, checking if a byte array contains valid UTF8 code, calculating the length of an encoded string.]]> o is a UTF8 with the same contents.]]> Also includes utilities for efficiently reading and writing UTF-8. @deprecated replaced by Text]]> This is useful when a class may evolve, so that instances written by the old version of the class may still be processed by the new version. To handle this situation, {@link #readFields(DataInput)} implementations should catch {@link VersionMismatchException}.]]> o is a VIntWritable with the same value.]]> o is a VLongWritable with the same value.]]> out. @param out DataOuput to serialize this object into. @throws IOException]]> in.

For efficiency, implementations should attempt to re-use storage in the existing object where possible.

@param in DataInput to deseriablize this object from. @throws IOException]]>
Any key or value type in the Hadoop Map-Reduce framework implements this interface.

Implementations typically implement a static read(DataInput) method which constructs a new instance, calls {@link #readFields(DataInput)} and returns the instance.

Example:

     public class MyWritable implements Writable {
       // Some data     
       private int counter;
       private long timestamp;
       
       public void write(DataOutput out) throws IOException {
         out.writeInt(counter);
         out.writeLong(timestamp);
       }
       
       public void readFields(DataInput in) throws IOException {
         counter = in.readInt();
         timestamp = in.readLong();
       }
       
       public static MyWritable read(DataInput in) throws IOException {
         MyWritable w = new MyWritable();
         w.readFields(in);
         return w;
       }
     }
 

]]>
WritableComparables can be compared to each other, typically via Comparators. Any type which is to be used as a key in the Hadoop Map-Reduce framework should implement this interface.

Example:

     public class MyWritableComparable implements WritableComparable {
       // Some data
       private int counter;
       private long timestamp;
       
       public void write(DataOutput out) throws IOException {
         out.writeInt(counter);
         out.writeLong(timestamp);
       }
       
       public void readFields(DataInput in) throws IOException {
         counter = in.readInt();
         timestamp = in.readLong();
       }
       
       public int compareTo(MyWritableComparable w) {
         int thisValue = this.value;
         int thatValue = ((IntWritable)o).value;
         return (thisValue < thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
       }
     }
 

]]>
The default implementation reads the data into two {@link WritableComparable}s (using {@link Writable#readFields(DataInput)}, then calls {@link #compare(WritableComparable,WritableComparable)}.]]> The default implementation uses the natural ordering, calling {@link Comparable#compareTo(Object)}.]]> This base implemenation uses the natural ordering. To define alternate orderings, override {@link #compare(WritableComparable,WritableComparable)}.

One may optimize compare-intensive operations by overriding {@link #compare(byte[],int,int,byte[],int,int)}. Static utility methods are provided to assist in optimized implementations of this method.]]> Enum type @param in DataInput to read from @param enumType Class type of Enum @return Enum represented by String read from DataInput @throws IOException]]> len number of bytes in input streamin @param in input stream @param len number of bytes to skip @throws IOException when skipped less number of bytes]]> CompressionCodec for which to get the Compressor @return Compressor for the given CompressionCodec from the pool or a new one]]> CompressionCodec for which to get the Decompressor @return Decompressor for the given CompressionCodec the pool or a new one]]> Compressor to be returned to the pool]]> Decompressor to be returned to the pool]]> Implementations are assumed to be buffered. This permits clients to reposition the underlying input stream then call {@link #resetState()}, without having to also synchronize client buffers.]]> true indicating that more input data is required. @param b Input data @param off Start offset @param len Length]]> true if the input data buffer is empty and #setInput() should be called in order to provide more input.]]> true if the end of the compressed data output stream has been reached.]]> true indicating that more input data is required. @param b Input data @param off Start offset @param len Length]]> true if the input data buffer is empty and #setInput() should be called in order to provide more input.]]> true if a preset dictionary is needed for decompression. @return true if a preset dictionary is needed for decompression]]> true if the end of the compressed data output stream has been reached.]]> true if native-lzo library is loaded & initialized; else false]]> lzo compression/decompression pair. http://www.oberhumer.com/opensource/lzo/]]> true if lzo compressors are loaded & initialized, else false]]> true if lzo decompressors are loaded & initialized, else false]]> @return the total (non-negative) number of uncompressed bytes input so far]]> @return the total (non-negative) number of uncompressed bytes input so far]]> true if native-zlib is loaded & initialized and can be loaded for this job, else false]]> Keep trying a limited number of times, waiting a fixed time between attempts, and then fail by re-throwing the exception.

]]>
Keep trying for a maximum time, waiting a fixed time between attempts, and then fail by re-throwing the exception.

]]>
Keep trying a limited number of times, waiting a growing amount of time between attempts, and then fail by re-throwing the exception. The time between attempts is sleepTime mutliplied by the number of tries so far.

]]>
Keep trying a limited number of times, waiting a growing amount of time between attempts, and then fail by re-throwing the exception. The time between attempts is sleepTime mutliplied by a random number in the range of [0, 2 to the number of retries)

]]>
Set a default policy with some explicit handlers for specific exceptions.

]]>
A retry policy for RemoteException Set a default policy with some explicit handlers for specific exceptions.

]]>
Try once, and fail by re-throwing the exception. This corresponds to having no retry mechanism in place.

]]>
Try once, and fail silently for void methods, or by re-throwing the exception for non-void methods.

]]>
Keep trying forever.

]]>
A collection of useful implementations of {@link RetryPolicy}.

]]>
Determines whether the framework should retry a method for the given exception, and the number of retries that have been made for that operation so far.

@param e The exception that caused the method to fail. @param retries The number of times the method has been retried. @return true if the method should be retried, false if the method should not be retried but shouldn't fail with an exception (only for void methods). @throws Exception The re-thrown exception e indicating that the method failed and should not be retried further.]]>
Specifies a policy for retrying method failures. Implementations of this interface should be immutable.

]]>
Create a proxy for an interface of an implementation class using the same retry policy for each method in the interface.

@param iface the interface that the retry will implement @param implementation the instance whose methods should be retried @param retryPolicy the policy for retirying method call failures @return the retry proxy]]>
Create a proxy for an interface of an implementation class using the a set of retry policies specified by method name. If no retry policy is defined for a method then a default of {@link RetryPolicies#TRY_ONCE_THEN_FAIL} is used.

@param iface the interface that the retry will implement @param implementation the instance whose methods should be retried @param methodNameToPolicyMap a map of method names to retry policies @return the retry proxy]]>
A factory for creating retry proxies.

]]>
Prepare the deserializer for reading.

]]>
Deserialize the next object from the underlying input stream. If the object t is non-null then this deserializer may set its internal state to the next object read from the input stream. Otherwise, if the object t is null a new deserialized object will be created.

@return the deserialized object]]>
Close the underlying input stream and clear up any resources.

]]>
Provides a facility for deserializing objects of type from an {@link InputStream}.

Deserializers are stateful, but must not buffer the input since other producers may read from the input between calls to {@link #deserialize(Object)}.

@param ]]>
A {@link RawComparator} that uses a {@link Deserializer} to deserialize the objects to be compared so that the standard {@link Comparator} can be used to compare them.

One may optimize compare-intensive operations by using a custom implementation of {@link RawComparator} that operates directly on byte representations.

@param ]]>
An experimental {@link Serialization} for Java {@link Serializable} classes.

@see JavaSerializationComparator]]>
A {@link RawComparator} that uses a {@link JavaSerialization} {@link Deserializer} to deserialize objects that are then compared via their {@link Comparable} interfaces.

@param @see JavaSerialization]]>
Encapsulates a {@link Serializer}/{@link Deserializer} pair.

@param ]]>
Serializations are found by reading the io.serializations property from conf, which is a comma-delimited list of classnames.

]]>
A factory for {@link Serialization}s.

]]>
Prepare the serializer for writing.

]]>
Serialize t to the underlying output stream.

]]>
Close the underlying output stream and clear up any resources.

]]>
Provides a facility for serializing objects of type to an {@link OutputStream}.

Serializers are stateful, but must not buffer the output since other producers may write to the output between calls to {@link #serialize(Object)}.

@param ]]>
param, to the IPC server running at address, returning the value. Throws exceptions if there are network problems or if the remote code threw an exception.]]> Unwraps any IOException. @param lookupTypes the desired exception class. @return IOException, which is either the lookupClass exception or this.]]> This unwraps any Throwable that has a constructor taking a String as a parameter. Otherwise it returns this. @return Throwable]]> protocol is a Java interface. All parameters and return types must be one of:
  • a primitive type, boolean, byte, char, short, int, long, float, double, or void; or
  • a {@link String}; or
  • a {@link Writable}; or
  • an array of the above types
All methods in the protocol should throw only IOException. No field data of the protocol instance is transmitted.]]>
handlerCount determines the number of handler threads that will be used to process calls.]]>
This class has a number of metrics variables that are publicly accessible; these variables (objects) have methods to update their values; for example:

{@link #rpcQueueTime}.inc(time)]]> For the statistics that are sampled and averaged, one must specify a metrics context that does periodic update calls. Most do. The default Null metrics context however does NOT. So if you aren't using any other metrics context then you can turn on the viewing and averaging of sampled metrics by specifying the following two lines in the hadoop-meterics.properties file:

        rpc.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
        rpc.period=10
  

Note that the metrics are collected regardless of the context used. The context with the update thread is used to average the data periodically]]> JobTracker, as {@link JobTracker.State} @return the current state of the JobTracker.]]> ClusterStatus provides clients with information such as:

  1. Size of the cluster.
  2. Task capacity of the cluster.
  3. The number of currently running map & reduce tasks.
  4. State of the JobTracker.

Clients can query for the latest ClusterStatus, via {@link JobClient#getClusterStatus()}.

@see JobClient]]> Counters represent global counters, defined either by the Map-Reduce framework or applications. Each Counter can be of any {@link Enum} type.

Counters are bunched into {@link Group}s, each comprising of counters from a particular Enum class.]]> Group of counters, comprising of counters from a particular counter {@link Enum} class.

Grouphandles localization of the class name and the counter names.

]]>
FileInputFormat implementations can override this and return false to ensure that individual input files are never split-up so that {@link Mapper}s process entire files. @param fs the file system that the file is on @param filename the file name to check @return is this file splitable?]]> FileInputFormat is the base class for all file-based InputFormats. This provides a generic implementation of {@link #getSplits(JobConf, int)}. Subclasses of FileInputFormat can also override the {@link #isSplitable(FileSystem, Path)} method to ensure input-files are not split-up and are processed as a whole by {@link Mapper}s.]]> true if the job output should be compressed, false otherwise]]> Tasks' Side-Effect Files

Some applications need to create/write-to side-files, which differ from the actual job-outputs.

In such cases there could be issues with 2 instances of the same TIP (running simultaneously e.g. speculative tasks) trying to open/write-to the same file (path) on HDFS. Hence the application-writer will have to pick unique names per task-attempt (e.g. using the attemptid, say attempt_200709221812_0001_m_000000_0), not just per TIP.

To get around this the Map-Reduce framework helps the application-writer out by maintaining a special ${mapred.output.dir}/_temporary/_${taskid} sub-directory for each task-attempt on HDFS where the output of the task-attempt goes. On successful completion of the task-attempt the files in the ${mapred.output.dir}/_temporary/_${taskid} (only) are promoted to ${mapred.output.dir}. Of course, the framework discards the sub-directory of unsuccessful task-attempts. This is completely transparent to the application.

The application-writer can take advantage of this by creating any side-files required in ${mapred.work.output.dir} during execution of his reduce-task i.e. via {@link #getWorkOutputPath(JobConf)}, and the framework will move them out similarly - thus she doesn't have to pick unique paths per task-attempt.

Note: the value of ${mapred.work.output.dir} during execution of a particular task-attempt is actually ${mapred.output.dir}/_temporary/_{$taskid}, and this value is set by the map-reduce framework. So, just create any side-files in the path returned by {@link #getWorkOutputPath(JobConf)} from map/reduce task to take advantage of this feature.

The entire discussion holds true for maps of jobs with reducer=NONE (i.e. 0 reduces) since output of the map, in that case, goes directly to HDFS.

@return the {@link Path} to the task's temporary output directory for the map-reduce job.]]>
This method is used to validate the input directories when a job is submitted so that the {@link JobClient} can fail early, with an useful error message, in case of errors. For e.g. input directory does not exist.

@param job job configuration. @throws InvalidInputException if the job does not have valid input @deprecated getSplits is called in the client and can perform any necessary validation of the input]]>
Each {@link InputSplit} is then assigned to an individual {@link Mapper} for processing.

Note: The split is a logical split of the inputs and the input files are not physically split into chunks. For e.g. a split could be <input-file-path, start, offset> tuple. @param job job configuration. @param numSplits the desired number of splits, a hint. @return an array of {@link InputSplit}s for the job.]]> It is the responsibility of the RecordReader to respect record boundaries while processing the logical split to present a record-oriented view to the individual task.

@param split the {@link InputSplit} @param job the job that this split belongs to @return a {@link RecordReader}]]>
InputFormat describes the input-specification for a Map-Reduce job.

The Map-Reduce framework relies on the InputFormat of the job to:

  1. Validate the input-specification of the job.
  2. Split-up the input file(s) into logical {@link InputSplit}s, each of which is then assigned to an individual {@link Mapper}.
  3. Provide the {@link RecordReader} implementation to be used to glean input records from the logical InputSplit for processing by the {@link Mapper}.

The default behavior of file-based {@link InputFormat}s, typically sub-classes of {@link FileInputFormat}, is to split the input into logical {@link InputSplit}s based on the total size, in bytes, of the input files. However, the {@link FileSystem} blocksize of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size.

Clearly, logical splits based on input-size is insufficient for many applications since record boundaries are to respected. In such cases, the application has to also implement a {@link RecordReader} on whom lies the responsibilty to respect record-boundaries and present a record-oriented view of the logical InputSplit to the individual task. @see InputSplit @see RecordReader @see JobClient @see FileInputFormat]]> InputSplit. @return the number of bytes in the input split. @throws IOException]]> InputSplit is located as an array of Strings. @throws IOException]]> InputSplit represents the data to be processed by an individual {@link Mapper}.

Typically, it presents a byte-oriented view on the input and is the responsibility of {@link RecordReader} of the job to process this and present a record-oriented view. @see InputFormat @see RecordReader]]> JobClient.]]> jobid doesn't correspond to any known job. @throws IOException]]> JobClient is the primary interface for the user-job to interact with the {@link JobTracker}. JobClient provides facilities to submit jobs, track their progress, access component-tasks' reports/logs, get the Map-Reduce cluster status information etc.

The job submission process involves:

  1. Checking the input and output specifications of the job.
  2. Computing the {@link InputSplit}s for the job.
  3. Setup the requisite accounting information for the {@link DistributedCache} of the job, if necessary.
  4. Copying the job's jar and configuration to the map-reduce system directory on the distributed file-system.
  5. Submitting the job to the JobTracker and optionally monitoring it's status.

Normally the user creates the application, describes various facets of the job via {@link JobConf} and then uses the JobClient to submit the job and monitor its progress.

Here is an example on how to use JobClient:

     // Create a new JobConf
     JobConf job = new JobConf(new Configuration(), MyJob.class);
     
     // Specify various job-specific parameters     
     job.setJobName("myjob");
     
     job.setInputPath(new Path("in"));
     job.setOutputPath(new Path("out"));
     
     job.setMapperClass(MyJob.MyMapper.class);
     job.setReducerClass(MyJob.MyReducer.class);

     // Submit the job, then poll for progress until the job is complete
     JobClient.runJob(job);
 

Job Control

At times clients would chain map-reduce jobs to accomplish complex tasks which cannot be done via a single map-reduce job. This is fairly easy since the output of the job, typically, goes to distributed file-system and that can be used as the input for the next job.

However, this also means that the onus on ensuring jobs are complete (success/failure) lies squarely on the clients. In such situations the various job-control options are:

  1. {@link #runJob(JobConf)} : submits the job and returns only after the job has completed.
  2. {@link #submitJob(JobConf)} : only submits the job, then poll the returned handle to the {@link RunningJob} to query status and make scheduling decisions.
  3. {@link JobConf#setJobEndNotificationURI(String)} : setup a notification on job-completion, thus avoiding polling.

@see JobConf @see ClusterStatus @see Tool @see DistributedCache]]>
true if framework should keep the intermediate files for failed tasks, false otherwise.]]> Note:

@param dir the {@link Path} of the output directory for the map-reduce job.]]>
true if the outputs of the maps are to be compressed, false otherwise.]]> This comparator should be provided if the equivalence rules for keys for sorting the intermediates are different from those for grouping keys before each call to {@link Reducer#reduce(Object, java.util.Iterator, OutputCollector, Reporter)}.

For key-value pairs (K1,V1) and (K2,V2), the values (V1, V2) are passed in a single call to the reduce function if K1 and K2 compare as equal.

Since {@link #setOutputKeyComparatorClass(Class)} can be used to control how keys are sorted, this can be used in conjunction to simulate secondary sort on values.

Note: This is not a guarantee of the reduce sort being stable in any sense. (In any case, with the order of available map-outputs to the reduce being non-deterministic, it wouldn't make that much sense.)

@param theClass the comparator class to be used for grouping keys. It should implement RawComparator. @see #setOutputKeyComparatorClass(Class)]]>
combiner class used to combine map-outputs before being sent to the reducers. Typically the combiner is same as the the {@link Reducer} for the job i.e. {@link #getReducerClass()}. @return the user-defined combiner class used to combine map-outputs.]]> combiner class used to combine map-outputs before being sent to the reducers.

The combiner is a task-level aggregation operation which, in some cases, helps to cut down the amount of data transferred from the {@link Mapper} to the {@link Reducer}, leading to better performance.

Typically the combiner is same as the Reducer for the job i.e. {@link #setReducerClass(Class)}.

@param theClass the user-defined combiner class used to combine map-outputs.]]>
true. @return true if speculative execution be used for this job, false otherwise.]]> true if speculative execution should be turned on, else false.]]> true. @return true if speculative execution be used for this job for map tasks, false otherwise.]]> true if speculative execution should be turned on for map tasks, else false.]]> true. @return true if speculative execution be used for reduce tasks for this job, false otherwise.]]> true if speculative execution should be turned on for reduce tasks, else false.]]> 1. @return the number of reduce tasks for this job.]]> Note: This is only a hint to the framework. The actual number of spawned map tasks depends on the number of {@link InputSplit}s generated by the job's {@link InputFormat#getSplits(JobConf, int)}. A custom {@link InputFormat} is typically used to accurately control the number of map tasks for the job.

How many maps?

The number of maps is usually driven by the total size of the inputs i.e. total number of blocks of the input files.

The right level of parallelism for maps seems to be around 10-100 maps per-node, although it has been set up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.

The default behavior of file-based {@link InputFormat}s is to split the input into logical {@link InputSplit}s based on the total size, in bytes, of input files. However, the {@link FileSystem} blocksize of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size.

Thus, if you expect 10TB of input data and have a blocksize of 128MB, you'll end up with 82,000 maps, unless {@link #setNumMapTasks(int)} is used to set it even higher.

@param n the number of map tasks for this job. @see InputFormat#getSplits(JobConf, int) @see FileInputFormat @see FileSystem#getDefaultBlockSize() @see FileStatus#getBlockSize()]]>
1. @return the number of reduce tasks for this job.]]> How many reduces?

The right number of reduces seems to be 0.95 or 1.75 multiplied by (<no. of nodes> * mapred.tasktracker.reduce.tasks.maximum).

With 0.95 all of the reduces can launch immediately and start transfering map outputs as the maps finish. With 1.75 the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing.

Increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.

The scaling factors above are slightly less than whole numbers to reserve a few reduce slots in the framework for speculative-tasks, failures etc.

Reducer NONE

It is legal to set the number of reduce-tasks to zero.

In this case the output of the map-tasks directly go to distributed file-system, to the path set by {@link FileOutputFormat#setOutputPath(JobConf, Path)}. Also, the framework doesn't sort the map-outputs before writing it out to HDFS.

@param n the number of reduce tasks for this job.]]>
mapred.map.max.attempts property. If this property is not already set, the default is 4 attempts. @return the max number of attempts per map task.]]> mapred.reduce.max.attempts property. If this property is not already set, the default is 4 attempts. @return the max number of attempts per reduce task.]]> noFailures, the tasktracker is blacklisted for this job. @param noFailures maximum no. of failures of a given job per tasktracker.]]> blacklisted for this job. @return the maximum no. of failures of a given job per tasktracker.]]> failed. Defaults to zero, i.e. any failed map-task results in the job being declared as {@link JobStatus#FAILED}. @return the maximum percentage of map tasks that can fail without the job being aborted.]]> failed. @param percent the maximum percentage of map tasks that can fail without the job being aborted.]]> failed. Defaults to zero, i.e. any failed reduce-task results in the job being declared as {@link JobStatus#FAILED}. @return the maximum percentage of reduce tasks that can fail without the job being aborted.]]> failed. @param percent the maximum percentage of reduce tasks that can fail without the job being aborted.]]> The debug script can aid debugging of failed map tasks. The script is given task's stdout, stderr, syslog, jobconf files as arguments.

The debug command, run on the node where the map failed, is:

$script $stdout $stderr $syslog $jobconf.

The script file is distributed through {@link DistributedCache} APIs. The script needs to be symlinked.

Here is an example on how to submit a script

 job.setMapDebugScript("./myscript");
 DistributedCache.createSymlink(job);
 DistributedCache.addCacheFile("/debug/scripts/myscript#myscript");
 

@param mDbgScript the script name]]>
The debug script can aid debugging of failed reduce tasks. The script is given task's stdout, stderr, syslog, jobconf files as arguments.

The debug command, run on the node where the map failed, is:

$script $stdout $stderr $syslog $jobconf.

The script file is distributed through {@link DistributedCache} APIs. The script file needs to be symlinked

Here is an example on how to submit a script

 job.setReduceDebugScript("./myscript");
 DistributedCache.createSymlink(job);
 DistributedCache.addCacheFile("/debug/scripts/myscript#myscript");
 

@param rDbgScript the script name]]>
null if it hasn't been set. @see #setJobEndNotificationURI(String)]]> The uri can contain 2 special parameters: $jobId and $jobStatus. Those, if present, are replaced by the job's identifier and completion-status respectively.

This is typically used by application-writers to implement chaining of Map-Reduce jobs in an asynchronous manner.

@param uri the job end notification uri @see JobStatus @see Job Completion and Chaining]]>
When a job starts, a shared directory is created at location ${mapred.local.dir}/taskTracker/jobcache/$jobid/work/ . This directory is exposed to the users through job.local.dir . So, the tasks can use this space as scratch space and share files among them.

This value is available as System property also. @return The localized job specific shared directory]]>
JobConf is the primary interface for a user to describe a map-reduce job to the Hadoop framework for execution. The framework tries to faithfully execute the job as-is described by JobConf, however:
  1. Some configuration parameters might have been marked as final by administrators and hence cannot be altered.
  2. While some job parameters are straight-forward to set (e.g. {@link #setNumReduceTasks(int)}), some parameters interact subtly rest of the framework and/or job-configuration and is relatively more complex for the user to control finely (e.g. {@link #setNumMapTasks(int)}).

JobConf typically specifies the {@link Mapper}, combiner (if any), {@link Partitioner}, {@link Reducer}, {@link InputFormat} and {@link OutputFormat} implementations to be used etc.

Optionally JobConf is used to specify other advanced facets of the job such as Comparators to be used, files to be put in the {@link DistributedCache}, whether or not intermediate and/or job outputs are to be compressed (and how), debugability via user-provided scripts ( {@link #setMapDebugScript(String)}/{@link #setReduceDebugScript(String)}), for doing post-processing on task logs, task's stdout, stderr, syslog. and etc.

Here is an example on how to configure a job via JobConf:

     // Create a new JobConf
     JobConf job = new JobConf(new Configuration(), MyJob.class);
     
     // Specify various job-specific parameters     
     job.setJobName("myjob");
     
     FileInputFormat.setInputPaths(job, new Path("in"));
     FileOutputFormat.setOutputPath(job, new Path("out"));
     
     job.setMapperClass(MyJob.MyMapper.class);
     job.setCombinerClass(MyJob.MyReducer.class);
     job.setReducerClass(MyJob.MyReducer.class);
     
     job.setInputFormat(SequenceFileInputFormat.class);
     job.setOutputFormat(SequenceFileOutputFormat.class);
 

@see JobClient @see ClusterStatus @see Tool @see DistributedCache]]>
.]]> ]]> any job run on the jobtracker started at 200707121733, we would use :
 
 JobID.getTaskIDsPattern("200707121733", null);
 
which will return :
 "job_200707121733_[0-9]*" 
@param jtIdentifier jobTracker identifier, or null @param jobId job number, or null @return a regex pattern matching JobIDs]]>
An example JobID is : job_200707121733_0003 , which represents the third job running at the jobtracker started at 200707121733.

Applications should never construct or parse JobID strings, but rather use appropriate constructors or {@link #forName(String)} method. @see TaskID @see TaskAttemptID @see JobTracker#getNewJobId() @see JobTracker#getStartTime()]]> -archives -files inputjar args]]> zero. @param conf configuration for the JobTracker. @throws IOException]]> io.file.buffer.size specified in the given Configuration. @param in input stream @param conf configuration @throws IOException]]> Output pairs need not be of the same types as input pairs. A given input pair may map to zero or many output pairs. Output pairs are collected with calls to {@link OutputCollector#collect(Object,Object)}.

Applications can use the {@link Reporter} provided to report progress or just indicate that they are alive. In scenarios where the application takes an insignificant amount of time to process individual key/value pairs, this is crucial since the framework might assume that the task has timed-out and kill that task. The other way of avoiding this is to set mapred.task.timeout to a high-enough value (or even zero for no time-outs).

@param key the input key. @param value the input value. @param output collects mapped keys and values. @param reporter facility to report progress.]]>
Maps are the individual tasks which transform input records into a intermediate records. The transformed intermediate records need not be of the same type as the input records. A given input pair may map to zero or many output pairs.

The Hadoop Map-Reduce framework spawns one map task for each {@link InputSplit} generated by the {@link InputFormat} for the job. Mapper implementations can access the {@link JobConf} for the job via the {@link JobConfigurable#configure(JobConf)} and initialize themselves. Similarly they can use the {@link Closeable#close()} method for de-initialization.

The framework then calls {@link #map(Object, Object, OutputCollector, Reporter)} for each key/value pair in the InputSplit for that task.

All intermediate values associated with a given output key are subsequently grouped by the framework, and passed to a {@link Reducer} to determine the final output. Users can control the grouping by specifying a Comparator via {@link JobConf#setOutputKeyComparatorClass(Class)}.

The grouped Mapper outputs are partitioned per Reducer. Users can control which keys (and hence records) go to which Reducer by implementing a custom {@link Partitioner}.

Users can optionally specify a combiner, via {@link JobConf#setCombinerClass(Class)}, to perform local aggregation of the intermediate outputs, which helps to cut down the amount of data transferred from the Mapper to the Reducer.

The intermediate, grouped outputs are always stored in {@link SequenceFile}s. Applications can specify if and how the intermediate outputs are to be compressed and which {@link CompressionCodec}s are to be used via the JobConf.

If the job has zero reduces then the output of the Mapper is directly written to the {@link FileSystem} without grouping by keys.

Example:

     public class MyMapper<K extends WritableComparable, V extends Writable> 
     extends MapReduceBase implements Mapper<K, V, K, V> {
     
       static enum MyCounters { NUM_RECORDS }
       
       private String mapTaskId;
       private String inputFile;
       private int noRecords = 0;
       
       public void configure(JobConf job) {
         mapTaskId = job.get("mapred.task.id");
         inputFile = job.get("mapred.input.file");
       }
       
       public void map(K key, V val,
                       OutputCollector<K, V> output, Reporter reporter)
       throws IOException {
         // Process the <key, value> pair (assume this takes a while)
         // ...
         // ...
         
         // Let the framework know that we are alive, and kicking!
         // reporter.progress();
         
         // Process some more
         // ...
         // ...
         
         // Increment the no. of <key, value> pairs processed
         ++noRecords;

         // Increment counters
         reporter.incrCounter(NUM_RECORDS, 1);
        
         // Every 100 records update application-level status
         if ((noRecords%100) == 0) {
           reporter.setStatus(mapTaskId + " processed " + noRecords + 
                              " from input-file: " + inputFile); 
         }
         
         // Output the result
         output.collect(key, val);
       }
     }
 

Applications may write a custom {@link MapRunnable} to exert greater control on map processing e.g. multi-threaded Mappers etc.

@see JobConf @see InputFormat @see Partitioner @see Reducer @see MapReduceBase @see MapRunnable @see SequenceFile]]>
Provides default no-op implementations for a few methods, most non-trivial applications need to override some of them.

]]>
<key, value> pairs.

Mapping of input records to output records is complete when this method returns.

@param input the {@link RecordReader} to read the input records. @param output the {@link OutputCollector} to collect the outputrecords. @param reporter {@link Reporter} to report progress, status-updates etc. @throws IOException]]>
Custom implementations of MapRunnable can exert greater control on map processing e.g. multi-threaded, asynchronous mappers etc.

@see Mapper]]>
nearly equal content length.
Subclasses implement {@link #getRecordReader(InputSplit, JobConf, Reporter)} to construct RecordReader's for MultiFileSplit's. @see MultiFileSplit]]>
th Path]]> th Path]]> MultiFileSplit can be used to implement {@link RecordReader}'s, with reading one record per file. @see FileSplit @see MultiFileInputFormat]]> <key, value> pairs output by {@link Mapper}s and {@link Reducer}s.

OutputCollector is the generalization of the facility provided by the Map-Reduce framework to collect data output by either the Mapper or the Reducer i.e. intermediate outputs or the output of the job.

]]>
This is to validate the output specification for the job when it is a job is submitted. Typically checks that it does not already exist, throwing an exception when it already exists, so that output is not overwritten.

@param ignored @param job job configuration. @throws IOException when output should not be attempted]]>
OutputFormat describes the output-specification for a Map-Reduce job.

The Map-Reduce framework relies on the OutputFormat of the job to:

  1. Validate the output-specification of the job. For e.g. check that the output directory doesn't already exist.
  2. Provide the {@link RecordWriter} implementation to be used to write out the output files of the job. Output files are stored in a {@link FileSystem}.
@see RecordWriter @see JobConf]]>
true if the job output should be compressed, false otherwise]]> Typically a hash function on a all or a subset of the key.

@param key the key to be paritioned. @param value the entry value. @param numPartitions the total number of partitions. @return the partition number for the key.]]>
Partitioner controls the partitioning of the keys of the intermediate map-outputs. The key (or a subset of the key) is used to derive the partition, typically by a hash function. The total number of partitions is the same as the number of reduce tasks for the job. Hence this controls which of the m reduce tasks the intermediate key (and hence the record) is sent for reduction.

@see Reducer]]>
0.0 to 1.0. @throws IOException]]> RecordReader reads <key, value> pairs from an {@link InputSplit}.

RecordReader, typically, converts the byte-oriented view of the input, provided by the InputSplit, and presents a record-oriented view for the {@link Mapper} & {@link Reducer} tasks for processing. It thus assumes the responsibility of processing record boundaries and presenting the tasks with keys and values.

@see InputSplit @see InputFormat]]>
RecordWriter to future operations. @param reporter facility to report progress. @throws IOException]]> RecordWriter writes the output <key, value> pairs to an output file.

RecordWriter implementations write the job outputs to the {@link FileSystem}. @see OutputFormat]]> Reduces values for a given key.

The framework calls this method for each <key, (list of values)> pair in the grouped inputs. Output values must be of the same type as input values. Input keys must not be altered. The framework will reuse the key and value objects that are passed into the reduce, therefore the application should clone the objects they want to keep a copy of. In many cases, all values are combined into zero or one value.

Output pairs are collected with calls to {@link OutputCollector#collect(Object,Object)}.

Applications can use the {@link Reporter} provided to report progress or just indicate that they are alive. In scenarios where the application takes an insignificant amount of time to process individual key/value pairs, this is crucial since the framework might assume that the task has timed-out and kill that task. The other way of avoiding this is to set mapred.task.timeout to a high-enough value (or even zero for no time-outs).

@param key the key. @param values the list of values to reduce. @param output to collect keys and combined values. @param reporter facility to report progress.]]>
The number of Reducers for the job is set by the user via {@link JobConf#setNumReduceTasks(int)}. Reducer implementations can access the {@link JobConf} for the job via the {@link JobConfigurable#configure(JobConf)} method and initialize themselves. Similarly they can use the {@link Closeable#close()} method for de-initialization.

Reducer has 3 primary phases:

  1. Shuffle

    Reducer is input the grouped output of a {@link Mapper}. In the phase the framework, for each Reducer, fetches the relevant partition of the output of all the Mappers, via HTTP.

  2. Sort

    The framework groups Reducer inputs by keys (since different Mappers may have output the same key) in this stage.

    The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged.

    SecondarySort

    If equivalence rules for keys while grouping the intermediates are different from those for grouping keys before reduction, then one may specify a Comparator via {@link JobConf#setOutputValueGroupingComparator(Class)}.Since {@link JobConf#setOutputKeyComparatorClass(Class)} can be used to control how intermediate keys are grouped, these can be used in conjunction to simulate secondary sort on values.

    For example, say that you want to find duplicate web pages and tag them all with the url of the "best" known example. You would set up the job like:
    • Map Input Key: url
    • Map Input Value: document
    • Map Output Key: document checksum, url pagerank
    • Map Output Value: url
    • Partitioner: by checksum
    • OutputKeyComparator: by checksum and then decreasing pagerank
    • OutputValueGroupingComparator: by checksum
  3. Reduce

    In this phase the {@link #reduce(Object, Iterator, OutputCollector, Reporter)} method is called for each <key, (list of values)> pair in the grouped inputs.

    The output of the reduce task is typically written to the {@link FileSystem} via {@link OutputCollector#collect(Object, Object)}.

The output of the Reducer is not re-sorted.

Example:

     public class MyReducer<K extends WritableComparable, V extends Writable> 
     extends MapReduceBase implements Reducer<K, V, K, V> {
     
       static enum MyCounters { NUM_RECORDS }
        
       private String reduceTaskId;
       private int noKeys = 0;
       
       public void configure(JobConf job) {
         reduceTaskId = job.get("mapred.task.id");
       }
       
       public void reduce(K key, Iterator<V> values,
                          OutputCollector<K, V> output, 
                          Reporter reporter)
       throws IOException {
       
         // Process
         int noValues = 0;
         while (values.hasNext()) {
           V value = values.next();
           
           // Increment the no. of values for this key
           ++noValues;
           
           // Process the <key, value> pair (assume this takes a while)
           // ...
           // ...
           
           // Let the framework know that we are alive, and kicking!
           if ((noValues%10) == 0) {
             reporter.progress();
           }
         
           // Process some more
           // ...
           // ...
           
           // Output the <key, value> 
           output.collect(key, value);
         }
         
         // Increment the no. of <key, list of values> pairs processed
         ++noKeys;
         
         // Increment counters
         reporter.incrCounter(NUM_RECORDS, 1);
         
         // Every 100 keys update application-level status
         if ((noKeys%100) == 0) {
           reporter.setStatus(reduceTaskId + " processed " + noKeys);
         }
       }
     }
 

@see Mapper @see Partitioner @see Reporter @see MapReduceBase]]>
Enum. @param amount A non-negative amount by which the counter is to be incremented.]]> InputSplit that the map is reading from. @throws UnsupportedOperationException if called outside a mapper]]> {@link Mapper} and {@link Reducer} can use the Reporter provided to report progress or just indicate that they are alive. In scenarios where the application takes an insignificant amount of time to process individual key/value pairs, this is crucial since the framework might assume that the task has timed-out and kill that task.

Applications can also update {@link Counters} via the provided Reporter .

@see Progressable @see Counters]]>
progress of the job's map-tasks, as a float between 0.0 and 1.0. When all map tasks have completed, the function returns 1.0. @return the progress of the job's map-tasks. @throws IOException]]> progress of the job's reduce-tasks, as a float between 0.0 and 1.0. When all reduce tasks have completed, the function returns 1.0. @return the progress of the job's reduce-tasks. @throws IOException]]> true if the job is complete, else false. @throws IOException]]> true if the job succeeded, else false. @throws IOException]]> RunningJob is the user-interface to query for details on a running Map-Reduce job.

Clients can get hold of RunningJob via the {@link JobClient} and then query the running-job for details such as name, configuration, progress etc.

@see JobClient]]>
This allows the user to specify the key class to be different from the actual class ({@link BytesWritable}) used for writing

@param conf the {@link JobConf} to modify @param theClass the SequenceFile output key class.]]>
This allows the user to specify the value class to be different from the actual class ({@link BytesWritable}) used for writing

@param conf the {@link JobConf} to modify @param theClass the SequenceFile output key class.]]>
f. The filtering criteria is MD5(key) % f == 0.]]> f using the criteria record# % f == 0. For example, if the frequency is 10, one out of 10 records is returned.]]> . @param name The name of the server @param port The port to use on the server @param findPort whether the server should start at the given port and increment by 1 until it finds a free port.]]> points to the log directory "/static/" -> points to common static files (src/webapps/static) "/" -> the jsp server code from (src/webapps/)]]> all task attempt IDs of any jobtracker, in any job, of the first map task, we would use :
 
 TaskAttemptID.getTaskAttemptIDsPattern(null, null, true, 1, null);
 
which will return :
 "attempt_[^_]*_[0-9]*_m_000001_[0-9]*" 
@param jtIdentifier jobTracker identifier, or null @param jobId job number, or null @param isMap whether the tip is a map, or null @param taskId taskId number, or null @param attemptId the task attempt number, or null @return a regex pattern matching TaskAttemptIDs]]>
An example TaskAttemptID is : attempt_200707121733_0003_m_000005_0 , which represents the zeroth task attempt for the fifth map task in the third job running at the jobtracker started at 200707121733.

Applications should never construct or parse TaskAttemptID strings , but rather use appropriate constructors or {@link #forName(String)} method. @see JobID @see TaskID]]> the first map task of any jobtracker, of any job, we would use :

 
 TaskID.getTaskIDsPattern(null, null, true, 1);
 
which will return :
 "task_[^_]*_[0-9]*_m_000001*" 
@param jtIdentifier jobTracker identifier, or null @param jobId job number, or null @param isMap whether the tip is a map, or null @param taskId taskId number, or null @return a regex pattern matching TaskIDs]]>
An example TaskID is : task_200707121733_0003_m_000005 , which represents the fifth map task in the third job running at the jobtracker started at 200707121733.

Applications should never construct or parse TaskID strings , but rather use appropriate constructors or {@link #forName(String)} method. @see JobID @see TaskAttemptID]]> hadoop.log.dir.]]> true if the Job was added.]]> ([,]*) func ::= tbl(,"") class ::= @see java.lang.Class#forName(java.lang.String) path ::= @see org.apache.hadoop.fs.Path#Path(java.lang.String) } Reads expression from the mapred.join.expr property and user-supplied join types from mapred.join.define.<ident> types. Paths supplied to tbl are given as input paths to the InputFormat class listed. @see #compose(java.lang.String, java.lang.Class, java.lang.String...)]]> ,

) }]]> (tbl(,),tbl(,),...,tbl(,)) }]]> (tbl(,),tbl(,),...,tbl(,)) }]]> mapred.join.define.<ident> to a classname. In the expression mapred.join.expr, the identifier will be assumed to be a ComposableRecordReader. mapred.join.keycomparator can be a classname used to compare keys in the join. @see JoinRecordReader @see MultiFilterRecordReader]]> ...... }]]> capacity children to position id in the parent reader. The id of a root CompositeRecordReader is -1 by convention, but relying on this is not recommended.]]> override(S1,S2,S3) will prefer values from S3 over S2, and values from S2 over S1 for all keys emitted from all sources.]]> [,,...,]]]> out. TupleWritable format: {@code ...... }]]> It can be used instead of the default implementation, @link org.apache.hadoop.mapred.MapRunner, when the Map operation is not CPU bound in order to improve throughput.

Map implementations using this MapRunnable must be thread-safe.

The Map-Reduce job has to be configured to use this MapRunnable class (using the JobConf.setMapRunnerClass method) and the number of thread the thread-pool can use with the mapred.map.multithreadedrunner.threads property, its default value is 10 threads.

]]> pairs. Uses {@link StringTokenizer} to break text into tokens.]]> generateKeyValPairs(Object key, Object value); public void configure(JobConfjob); } The package also provides a base class, ValueAggregatorBaseDescriptor, implementing the above interface. The user can extend the base class and implement generateKeyValPairs accordingly. The primary work of generateKeyValPairs is to emit one or more key/value pairs based on the input key/value pair. The key in an output key/value pair encode two pieces of information: aggregation type and aggregation id. The value will be aggregated onto the aggregation id according the aggregation type. This class offers a function to generate a map/reduce job using Aggregate framework. The function takes the following parameters: input directory spec input format (text or sequence file) output directory a file specifying the user plugin class]]> When constructing the instance, if the factory property contextName.class exists, its value is taken to be the name of the class to instantiate. Otherwise, the default is to create an instance of org.apache.hadoop.metrics.spi.NullContext, which is a dummy "no-op" context which will cause all metric data to be discarded. @param contextName the name of the context @return the named MetricsContext]]> When the instance is constructed, this method checks if the file hadoop-metrics.properties exists on the class path. If it exists, it must be in the format defined by java.util.Properties, and all the properties in the file are set as attributes on the newly created ContextFactory instance. @return the singleton ContextFactory instance]]> getFactory() method.]]> startMonitoring() again after calling this. @see #close()]]> recordName. Throws an exception if the metrics implementation is configured with a fixed set of record names and recordName is not in that set. @param recordName the name of the record @throws MetricsException if recordName conflicts with configuration data]]> A record name identifies the kind of data to be reported. For example, a program reporting statistics relating to the disks on a computer might use a record name "diskStats".

A record has zero or more tags. A tag has a name and a value. To continue the example, the "diskStats" record might use a tag named "diskName" to identify a particular disk. Sometimes it is useful to have more than one tag, so there might also be a "diskType" with value "ide" or "scsi" or whatever.

A record also has zero or more metrics. These are the named values that are to be reported to the metrics system. In the "diskStats" example, possible metric names would be "diskPercentFull", "diskPercentBusy", "kbReadPerSecond", etc.

The general procedure for using a MetricsRecord is to fill in its tag and metric values, and then call update() to pass the record to the client library. Metric data is not immediately sent to the metrics system each time that update() is called. An internal table is maintained, identified by the record name. This table has columns corresponding to the tag and the metric names, and rows corresponding to each unique set of tag values. An update either modifies an existing row in the table, or adds a new row with a set of tag values that are different from all the other rows. Note that if there are no tags, then there can be at most one row in the table.

Once a row is added to the table, its data will be sent to the metrics system on every timer period, whether or not it has been updated since the previous timer period. If this is inappropriate, for example if metrics were being reported by some transient object in an application, the remove() method can be used to remove the row and thus stop the data from being sent.

Note that the update() method is atomic. This means that it is safe for different threads to be updating the same metric. More precisely, it is OK for different threads to call update() on MetricsRecord instances with the same set of tag names and tag values. Different threads should not use the same MetricsRecord instance at the same time.]]> MetricsContext.registerUpdater().]]> fileName attribute, if specified. Otherwise the data will be written to standard output.]]> This class is configured by setting ContextFactory attributes which in turn are usually configured through a properties file. All the attributes are prefixed by the contextName. For example, the properties file might contain:

 myContextName.fileName=/tmp/metrics.log
 myContextName.period=5
 
]]>
contextName.tableName. The returned map consists of those attributes with the contextName and tableName stripped off.]]> recordName. Throws an exception if the metrics implementation is configured with a fixed set of record names and recordName is not in that set. @param recordName the name of the record @throws MetricsException if recordName conflicts with configuration data]]> This class implements the internal table of metric data, and the timer on which data is to be sent to the metrics system. Subclasses must override the abstract emitRecord method in order to transmit the data.

]]> update and remove().]]> hostname or hostname:port. If the specs string is null, defaults to localhost:defaultPort. @return a list of InetSocketAddress objects.]]> ,name=" Where the and are the supplied parameters @param serviceName @param nameName @param theMbean - the MBean to register @return the named used to register the MBean]]> hadoop.rpc.socket.factory.class.<ClassName>. When no such parameter exists then fall back on the default socket factory as configured by hadoop.rpc.socket.factory.class.default. If this default socket factory is not configured, then fall back on the JVM default socket factory. @param conf the configuration @param clazz the class (usually a {@link VersionedProtocol}) @return a socket factory]]> hadoop.rpc.socket.factory.default @param conf the configuration @return the default socket factory as specified in the configuration or the JVM default socket factory if the configuration does not contain a default socket factory property.]]> : ://:/]]> : ://:/]]>
From documentation for {@link #getInputStream(Socket, long)}:
Returns InputStream for the socket. If the socket has an associated SocketChannel then it returns a {@link SocketInputStream} with the given timeout. If the socket does not have a channel, {@link Socket#getInputStream()} is returned. In the later case, the timeout argument is ignored and the timeout set with {@link Socket#setSoTimeout(int)} applies for reads.

Any socket created using socket factories returned by {@link #NetUtils}, must use this interface instead of {@link Socket#getInputStream()}. @see #getInputStream(Socket, long) @param socket @return InputStream for reading from the socket. @throws IOException]]>

Any socket created using socket factories returned by {@link #NetUtils}, must use this interface instead of {@link Socket#getInputStream()}. @see Socket#getChannel() @param socket @param timeout timeout in milliseconds. This may not always apply. zero for waiting as long as necessary. @return InputStream for reading from the socket. @throws IOException]]>

From documentation for {@link #getOutputStream(Socket, long)} :
Returns OutputStream for the socket. If the socket has an associated SocketChannel then it returns a {@link SocketOutputStream} with the given timeout. If the socket does not have a channel, {@link Socket#getOutputStream()} is returned. In the later case, the timeout argument is ignored and the write will wait until data is available.

Any socket created using socket factories returned by {@link #NetUtils}, must use this interface instead of {@link Socket#getOutputStream()}. @see #getOutputStream(Socket, long) @param socket @return OutputStream for writing to the socket. @throws IOException]]>

Any socket created using socket factories returned by {@link #NetUtils}, must use this interface instead of {@link Socket#getOutputStream()}. @see Socket#getChannel() @param socket @param timeout timeout in milliseconds. This may not always apply. zero for waiting as long as necessary. @return OutputStream for writing to the socket. @throws IOException]]>
node @param node a node @return true if node is already in the tree; false otherwise]]> scope if scope starts with ~, choose one from the all nodes except for the ones in scope; otherwise, choose one from scope @param scope range of nodes from which a node will be choosen @return the choosen node]]> scope but not in excludedNodes if scope starts with ~, return the number of nodes that are not in scope and excludedNodes; @param scope a path string that may start with ~ @param excludedNodes a list of nodes @return number of available nodes]]> reader It linearly scans the array, if a local node is found, swap it with the first element of the array. If a local rack node is found, swap it with the first element following the local node. If neither local node or local rack node is found, put a random replica location at postion 0. It leaves the rest nodes untouched.]]>
Create a new input stream with the given timeout. If the timeout is zero, it will be treated as infinite timeout. The socket's channel will be configured to be non-blocking. @see SocketInputStream#SocketInputStream(ReadableByteChannel, long) @param socket should have a channel associated with it. @param timeout timeout timeout in milliseconds. must not be negative. @throws IOException]]>

Create a new input stream with the given timeout. If the timeout is zero, it will be treated as infinite timeout. The socket's channel will be configured to be non-blocking. @see SocketInputStream#SocketInputStream(ReadableByteChannel, long) @param socket should have a channel associated with it. @throws IOException]]>

Create a new ouput stream with the given timeout. If the timeout is zero, it will be treated as infinite timeout. The socket's channel will be configured to be non-blocking. @see SocketOutputStream#SocketOutputStream(WritableByteChannel, long) @param socket should have a channel associated with it. @param timeout timeout timeout in milliseconds. must not be negative. @throws IOException]]>
= getCount(). @param newCapacity The new capacity in bytes.]]> Index idx = startVector(...); while (!idx.done()) { .... // read element of a vector idx.incr(); } ]]> This task takes the given record definition files and compiles them into java or c++ files. It is then up to the user to compile the generated files.

The task requires the file or the nested fileset element to be specified. Optional attributes are language (set the output language, default is "java"), destdir (name of the destination directory for generated java/c++ code, default is ".") and failonerror (specifies error handling behavior. default is true).

Usage

 <recordcc
       destdir="${basedir}/gensrc"
       language="java">
   <fileset include="**\/*.jr" />
 </recordcc>
 
]]>
]]> ugi as a comma separated string in conf as a property attr The String starts with the user name followed by the default group names, and other group names. @param conf configuration @param attr property name @param ugi a UnixUserGroupInformation]]> conf The object is expected to store with the property name attr as a comma separated string that starts with the user name followed by group names. If the property name is not defined, return null. It's assumed that there is only one UGI per user. If this user already has a UGI in the ugi map, return the ugi in the map. Otherwise, construct a UGI from the configuration, store it in the ugi map and return it. @param conf configuration @param attr property name @return a UnixUGI @throws LoginException if the stored string is ill-formatted.]]> This tool supports archiving and anaylzing (sort/grep) of log-files. It takes as input a) Input uri which will serve uris of the logs to be archived. b) Output directory (not mandatory). b) Directory on dfs to archive the logs. c) The sort/grep patterns for analyzing the files and separator for boundaries. Usage: Logalyzer -archive -archiveDir -analysis -logs -grep -sort -separator

]]> GenericOptionsParser to parse only the generic Hadoop arguments. The array of string arguments other than the generic arguments can be obtained by {@link #getRemainingArgs()}. @param conf the Configuration to modify. @param args command-line arguments.]]> GenericOptionsParser to parse given options as well as generic Hadoop options. The resulting CommandLine object can be obtained by {@link #getCommandLine()}. @param conf the configuration to modify @param options options built by the caller @param args User-specified arguments]]> Strings containing the un-parsed arguments or empty array if commandLine was not defined.]]> CommandLine object to process the parsed arguments. Note: If the object is created with {@link #GenericOptionsParser(Configuration, String[])}, then returned object will only contain parsed generic options. @return CommandLine representing list of arguments parsed against Options descriptor.]]> GenericOptionsParser is a utility to parse command line arguments generic to the Hadoop framework. GenericOptionsParser recognizes several standarad command line arguments, enabling applications to easily specify a namenode, a jobtracker, additional configuration resources etc.

Generic Options

The supported generic options are:

     -conf <configuration file>     specify a configuration file
     -D <property=value>            use value for given property
     -fs <local|namenode:port>      specify a namenode
     -jt <local|jobtracker:port>    specify a job tracker
     -files <comma separated list of files>    specify comma separated
                            files to be copied to the map reduce cluster
     -libjars <comma separated list of jars>   specify comma separated
                            jar files to include in the classpath.
     -archives <comma separated list of archives>    specify comma
             separated archives to be unarchived on the compute machines.

 

The general command line syntax is:

 bin/hadoop command [genericOptions] [commandOptions]
 

Generic command line arguments might modify Configuration objects, given to constructors.

The functionality is implemented using Commons CLI.

Examples:

 $ bin/hadoop dfs -fs darwin:8020 -ls /data
 list /data directory in dfs with namenode darwin:8020
 
 $ bin/hadoop dfs -D fs.default.name=darwin:8020 -ls /data
 list /data directory in dfs with namenode darwin:8020
     
 $ bin/hadoop dfs -conf hadoop-site.xml -ls /data
 list /data directory in dfs with conf specified in hadoop-site.xml
     
 $ bin/hadoop job -D mapred.job.tracker=darwin:50020 -submit job.xml
 submit a job to job tracker darwin:50020
     
 $ bin/hadoop job -jt darwin:50020 -submit job.xml
 submit a job to job tracker darwin:50020
     
 $ bin/hadoop job -jt local -submit job.xml
 submit a job to local runner
 
 $ bin/hadoop jar -libjars testlib.jar 
 -archives test.tgz -files file.txt inputjar args
 job submission with libjars, files and archives
 

@see Tool @see ToolRunner]]>
Class<T>) of the argument of type T. @param The type of the argument @param t the object to get it class @return Class<T>]]> List<T> to a an array of T[]. @param c the Class object of the items in the list @param list the list to convert]]> List<T> to a an array of T[]. @param list the list to convert @throws ArrayIndexOutOfBoundsException if the list is empty. Use {@link #toArray(Class, List)} if the list may be empty.]]> true if native-hadoop is loaded, else false]]> true if native hadoop libraries, if present, can be used for this job; false otherwise.]]> { pq.top().change(); pq.adjustTop(); } instead of
  { o = pq.pop(); o.change(); pq.push(o); }
 
]]>
Clients and/or applications can use the provided Progressable to explicitly report progress to the Hadoop framework. This is especially important for operations which take an insignificant amount of time since, in-lieu of the reported progress, the framework has to assume that an error has occured and time-out the operation.

]]>
Class is to be obtained @return the correctly typed Class of the given object.]]> Hadoop Pipes or Hadoop Streaming. It also checks to ensure that we are running on a *nix platform else (e.g. in Cygwin/Windows) it returns null. @param job job configuration @return a String[] with the ulimit command arguments or null if we are running on a non *nix platform or if the limit is unspecified.]]> Shell interface. @param cmd shell command to execute. @return the output of the executed command.]]> Shell can be used to run unix commands like du or df. It also offers facilities to gate commands by time-intervals.]]> ShellCommandExecutorshould be used in cases where the output of the command needs no explicit parsing and where the command, working directory and the environment remains unchanged. The output of the command is stored as-is and is expected to be small.]]> ArrayList of string values]]> charToEscape in the string with the escape char escapeChar @param str string @param escapeChar escape char @param charToEscape the char to be escaped @return an escaped string]]> charToEscape in the string with the escape char escapeChar @param str string @param escapeChar escape char @param charToEscape the escaped char @return an unescaped string]]> Tool, is the standard for any Map-Reduce tool/application. The tool/application should delegate the handling of standard command-line options to {@link ToolRunner#run(Tool, String[])} and only handle its custom arguments.

Here is how a typical Tool is implemented:

     public class MyApp extends Configured implements Tool {
     
       public int run(String[] args) throws Exception {
         // Configuration processed by ToolRunner
         Configuration conf = getConf();
         
         // Create a JobConf using the processed conf
         JobConf job = new JobConf(conf, MyApp.class);
         
         // Process custom command-line options
         Path in = new Path(args[1]);
         Path out = new Path(args[2]);
         
         // Specify various job-specific parameters     
         job.setJobName("my-app");
         job.setInputPath(in);
         job.setOutputPath(out);
         job.setMapperClass(MyApp.MyMapper.class);
         job.setReducerClass(MyApp.MyReducer.class);

         // Submit the job, then poll for progress until the job is complete
         JobClient.runJob(job);
       }
       
       public static void main(String[] args) throws Exception {
         // Let ToolRunner handle generic command-line options 
         int res = ToolRunner.run(new Configuration(), new Sort(), args);
         
         System.exit(res);
       }
     }
 

@see GenericOptionsParser @see ToolRunner]]>
Tool by {@link Tool#run(String[])}, after parsing with the given generic arguments. Uses the given Configuration, or builds one if null. Sets the Tool's configuration with the possibly modified version of the conf. @param conf Configuration for the Tool. @param tool Tool to run. @param args command-line arguments to the tool. @return exit code of the {@link Tool#run(String[])} method.]]> Tool with its Configuration. Equivalent to run(tool.getConf(), tool, args). @param tool Tool to run. @param args command-line arguments to the tool. @return exit code of the {@link Tool#run(String[])} method.]]> ToolRunner can be used to run classes implementing Tool interface. It works in conjunction with {@link GenericOptionsParser} to parse the generic hadoop command line arguments and modifies the Configuration of the Tool. The application-specific options are passed along without being modified.

@see Tool @see GenericOptionsParser]]>