Hadoop 0.20.1 Release Notes

These release notes include new developer and user-facing incompatibilities, features, and major improvements. The table below is sorted by Component.

Changes Since Hadoop 0.20.0

Common

Sub-task

[HADOOP-6213] - Remove commons dependency on commons-cli2

Bug

[HADOOP-4626] - API link in forrest doc should point to the same version of hadoop.
[HADOOP-4674] - hadoop fs -help should list detailed help info for the following commands: test, text, tail, stat & touchz
[HADOOP-4856] - Document JobInitializationPoller configuration in capacity scheduler forrest documentation.
[HADOOP-4931] - Document TaskTracker's memory management functionality and CapacityScheduler's memory based scheduling.
[HADOOP-5210] - Reduce Task Progress shows > 100% when the total size of map outputs (for a single reducer) is high
[HADOOP-5213] - BZip2CompressionOutputStream NullPointerException
[HADOOP-5349] - When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
[HADOOP-5533] - Recovery duration shown on the jobtracker webpage is inaccurate
[HADOOP-5539] - o.a.h.mapred.Merger not maintaining map out compression on intermediate files
[HADOOP-5636] - Job is left in Running state after a killJob
[HADOOP-5641] - Possible NPE in CapacityScheduler's MemoryMatcher
[HADOOP-5646] - TestQueueCapacities is failing Hudson tests for the last few builds
[HADOOP-5648] - Not able to generate gridmix.jar on already compiled version of hadoop
[HADOOP-5654] - TestReplicationPolicy.<init> fails on java.net.BindException
[HADOOP-5655] - TestMRServerPorts fails on java.net.BindException
[HADOOP-5688] - HftpFileSystem.getChecksum(..) does not work for the paths with scheme and authority
[HADOOP-5691] - org.apache.hadoop.mapreduce.Reducer should not be abstract.
[HADOOP-5711] - Change Namenode file close log to info
[HADOOP-5718] - Capacity Scheduler should not check for presence of default queue while starting up.
[HADOOP-5719] - Jobs failed during job initalization are never removed from Capacity Schedulers waiting list
[HADOOP-5736] - Update CapacityScheduler documentation to reflect latest changes
[HADOOP-5746] - Errors encountered in MROutputThread after the last map/reduce call can go undetected
[HADOOP-5796] - DFS Write pipeline does not detect defective datanode correctly in some cases (HADOOP-3339)
[HADOOP-5828] - Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
[HADOOP-5850] - map/reduce doesn't run jobs with 0 maps
[HADOOP-5863] - mapred metrics shows negative count of waiting maps and reduces
[HADOOP-5869] - TestQueueCapacitisues.apache.org/jjira/browse/HADOOP-OP-6017] - NameNode and SecondaryNameNode fail to restart because of abnormal filenames.
[HADOOP-6097] - Multiple bugs w/ Hadoop archives
[HADOOP-6139] - Incomplete help message is displayed for rm and rmr options.
[HADOOP-6141] - hadoop 0.20 branch "test-patch" is broken
[HADOOP-6145] - No error message for deleting non-existant file or directory.
[HADOOP-6215] - fix GenericOptionParser to deal with -D with '=' in the value

Improvement

[HADOOP-5726] - Remove pre-emption from the capacity scheduler code base

New Feature

[HADOOP-3315] - New binary file format
[HADOOP-5714] - Metric to show number of fs.exists (or number of getFileInfo) calls
[HADOOP-6080] - Handling of Trash with quota

HDFS

Bug

[HDFS-26] - HADOOP-5862 for version .20 (Namespace quota exceeded message unclear)
[HDFS-167] - DFSClient continues to retry indefinitely
[HDFS-438] - Improve help message for quotas
[HDFS-442] - dfsthroughput in test.jar throws NPE
[HDFS-485] - error : too many fetch failures
[HDFS-495] - Hadoop FSNamesystem startFileInternal() getLease() has bug
[HDFS-525] - ListPathsServlet.java uses static SimpleDateFormat that has threading issues

Improvement

[HDFS-504] - HDFS updates the modification time of a file when the file is closed.
[HDFS-527] - Refactor DFSClient constructors

Map/Reduce

Bug

[MAPREDUCE-2] - ArrayOutOfIndex error in KeyFieldBasedPartitioner on empty key
[MAPREDUCE-18] - Under load the shuffle sometimes gets incorrect data
[MAPREDUCE-40] - Memory management variables need a backwards compatibility option after HADOOP-5881
[MAPREDUCE-112] - Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API
[MAPREDUCE-124] - When abortTask of OutputCommitter fails with an Exception for a map-only job, the task is marked as success
[MAPREDUCE-130] - Delete the jobconf copy from the log directory of the JobTracker when the job is retired
[MAPREDUCE-179] - setProgress not called for new RecordReaders
[MAPREDUCE-383] - pipes combiner does not reset properly after a spill
[MAPREDUCE-421] - mapred pipes might return exit code 0 even when failing
[MAPREDUCE-430] - Task stuck in cleanup with OutOfMemoryErrors
[MAPREDUCE-565] - Partitioner does not work with new API
[MAPREDUCE-657] - CompletedJobStatusStore hardcodes filesystem to hdfs
[MAPREDUCE-687] - TestMiniMRMapRedDebugScript fails sometimes
[MAPREDUCE-735] - ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
[MAPREDUCE-745] - TestRecoveryManager fails sometimes
[MAPREDUCE-796] - Encountered "ClassCastException" on tasktracker while running wordcount with MultithreadedMapRunner
[MAPREDUCE-805] - Deadlock in Jobtracker
[MAPREDUCE-806] - WordCount example does not compile given the current instructions
[MAPREDUCE-807] - Stray user files in mapred.system.dir with permissions other than 777 can prevent the jobtracker from starting up.
[MAPREDUCE-818] - org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name doesnt exist.
[MAPREDUCE-827] - "hadoop job -status <jobid>" command should display job's completion status also.
[MAPREDUCE-832] - Too many WARN messages about deprecated memorty config variables in JobTacker log
[MAPREDUCE-834] - When TaskTracker config use old memory management values its memory monitoring is diabled.
[MAPREDUCE-838] - Task succeeds even when committer.commitTask fails with IOException
[MAPREDUCE-911] - TestTaskFail fail sometimes
[MAPREDUCE-924] - TestPipes crashes on trunk

Improvement

[MAPREDUCE-465] - Deprecate org.apache.hadoop.mapred.lib.MultithreadedMapRunner
[MAPREDUCE-487] - DBInputFormat support for Oracle
[MAPREDUCE-767] - to remove mapreduce dependency on commons-cli2

Changes Since Hadoop 0.19.1

Issue	Component	Notes
HADOOP-3344	build	Changed build procedure for libhdfs to build correctly for different platforms. Build instructions are in the Jira item.
HADOOP-4253	conf	Removed from class org.apache.hadoop.fs.RawLocalFileSystem deprecated methods public String getName(), public void lock(Path p, boolean shared) and public void release(Path p).
HADOOP-4454	conf	Changed processing of conf/slaves file to allow # to begin a comment.
HADOOP-4631	conf	Split hadoop-default.xml into core-default.xml, hdfs-default.xml and mapreduce-default.xml.
HADOOP-4035	contrib/capacity-sched	Changed capacity scheduler policy to take note of task memory requirements and task tracker memory availability.
HADOOP-4445	contrib/capacity-sched	Changed JobTracker UI to better present the number of active tasks.
HADOOP-4576	contrib/capacity-sched	Changed capacity scheduler UI to better present number of running and pending tasks.
HADOOP-4179	contrib/chukwa	Introduced Vaidya rule based performance diagnostic tool for Map/Reduce jobs.
HADOOP-4827	contrib/chukwa	Improved framework for data aggregation in Chuckwa.
HADOOP-4843	contrib/chukwa	Introduced Chuckwa collection of job history.
HADOOP-5030	contrib/chukwa	Changed RPM install location to the value specified by build.properties file.
HADOOP-5531	contrib/chukwa	Disabled Chukwa unit tests for 0.20 branch only.
HADOOP-4789	contrib/fair-share	Changed fair scheduler to divide resources equally between pools, not jobs.
HADOOP-4873	contrib/fair-share	Changed fair scheduler UI to display minMaps and minReduces variables.
HADOOP-3750	dfs	Removed deprecated method parseArgs from org.apache.hadoop.fs.FileSystem.
HADOOP-4029	dfs	Added name node storage information to the dfshealth page, and moved data node information to a separated page.
HADOOP-4103	dfs	Modified dfsadmin -report to report under replicated blocks. blocks with corrupt replicas, and missing blocks".
HADOOP-4567	dfs	Changed GetFileBlockLocations to return topology information for nodes that host the block replicas.
HADOOP-4572	dfs	Moved org.apache.hadoop.hdfs.{CreateEditsLog, NNThroughputBenchmark} to org.apache.hadoop.hdfs.server.namenode.
HADOOP-4618	dfs	Moved HTTP server from FSNameSystem to NameNode. Removed FSNamesystem.getNameNodeInfoPort(). Replaced FSNamesystem.getDFSNameNodeMachine() and FSNamesystem.getDFSNameNodePort() with new method FSNamesystem.getDFSNameNodeAddress(). Removed constructor NameNode(bindAddress, conf).
HADOOP-4826	dfs	Introduced new dfsadmin command saveNamespace to command the name service to do an immediate save of the file system image.
HADOOP-4970	dfs	Changed trash facility to use absolute path of the deleted file.
HADOOP-5468	documentation	Reformatted HTML documentation for Hadoop to use submenus at the left column.
HADOOP-3497	fs	Changed the semantics of file globbing with a PathFilter (using the globStatus method of FileSystem). Previously, the filtering was too restrictive, so that a glob of // and a filter that only accepts /a/b would not have matched /a/b. With this change /a/b does match.
HADOOP-4234	fs	Changed KFS glue layer to allow applications to interface with multiple KFS metaservers.
HADOOP-4422	fs/s3	Modified Hadoop file system to no longer create S3 buckets. Applications can create buckets for their S3 file systems by other means, for example, using the JetS3t API.
HADOOP-3063	io	Introduced BloomMapFile subclass of MapFile that creates a Bloom filter from all keys.
HADOOP-1230	mapred	Replaced parameters with context obejcts in Mapper, Reducer, Partitioner, InputFormat, and OutputFormat classes.
HADOOP-1650	mapred	Upgraded all core servers to use Jetty 6
HADOOP-3923	mapred	Moved class org.apache.hadoop.mapred.StatusHttpServer to org.apache.hadoop.http.HttpServer.
HADOOP-3986	mapred	Removed classes org.apache.hadoop.mapred.JobShell and org.apache.hadoop.mapred.TestJobShell. Removed from JobClient methods static void setCommandLineConfig(Configuration conf) and public static Configuration getCommandLineConfig().
HADOOP-4188	mapred	Removed Task's dependency on concrete file systems by taking list from FileSystem class. Added statistics table to FileSystem class. Deprecated FileSystem method getStatistics(Class<? extends FileSystem> cls).
HADOOP-4210	mapred	Changed public class org.apache.hadoop.mapreduce.ID to be an abstract class. Removed from class org.apache.hadoop.mapreduce.ID the methods public static ID read(DataInput in) and public static ID forName(String str).
HADOOP-4305	mapred	Improved TaskTracker blacklisting strategy to better exclude faulty tracker from executing tasks.
HADOOP-4435	mapred	Changed JobTracker web status page to display the amount of heap memory in use. This changes the JobSubmissionProtocol.
HADOOP-4565	mapred	Improved MultiFileInputFormat so that multiple blocks from the same node or same rack can be combined into a single split.
HADOOP-4749	mapred	Added a new counter REDUCE_INPUT_BYTES.
HADOOP-4783	mapred	Changed history directory permissions to 750 and history file permissions to 740.
HADOOP-3422	metrics	Changed names of ganglia metrics to avoid conflicts and to better identify source function.
HADOOP-4284	security	Introduced HttpServer method to support global filters.
HADOOP-4575	security	Introduced independent HSFTP proxy server for authenticated access to clusters.
HADOOP-4661	tools/distcp	Introduced distch tool for parallel ch{mod, own, grp}.