org.apache.hadoop.mapred
Class JobTracker

java.lang.Object
  org.apache.hadoop.mapred.JobTracker

All Implemented Interfaces:

VersionedProtocol, RefreshAuthorizationPolicyProtocol

public class JobTracker
extends Object
implements RefreshAuthorizationPolicyProtocol

JobTracker is the central location for submitting and tracking MR jobs in a network environment.

Nested Class Summary

static class	JobTracker.IllegalStateException A client tried to submit a job before the Job Tracker was ready.
static class	JobTracker.State

Field Summary

static int	CLUSTER_INCREMENT
static long	COUNTER_UPDATE_INTERVAL
static int	FILE_NOT_FOUND
static String	FOR_REDUCE_TASK The reduce task number for which this map output is being transferred
static String	FROM_MAP_TASK The map task from which the map output data is being transferred
static int	HEARTBEAT_INTERVAL_MIN
static org.apache.commons.logging.Log	LOG
static String	MAP_OUTPUT_LENGTH The custom http header used for the map output length.
static String	RAW_MAP_OUTPUT_LENGTH The custom http header used for the "raw" map output length.
static int	SUCCESS
static int	TRACKERS_OK
static int	UNKNOWN_TASKTRACKER
static long	versionID version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549.
static long	versionID
static String	WORKDIR

Fields inherited from interface org.apache.hadoop.security.authorize.RefreshAuthorizationPolicyProtocol

versionID

Method Summary

Collection<org.apache.hadoop.mapred.TaskTrackerStatus>	activeTaskTrackers() Get the active task tracker statuses in the cluster
void	addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener) Registers a JobInProgressListener for updates from this TaskTrackerManager.
Collection<org.apache.hadoop.mapred.TaskTrackerStatus>	blacklistedTaskTrackers() Get the blacklisted task tracker statuses in the cluster
Vector<org.apache.hadoop.mapred.JobInProgress>	completedJobs()
Vector<org.apache.hadoop.mapred.JobInProgress>	failedJobs()
void	failJob(org.apache.hadoop.mapred.JobInProgress job) Fail a job and inform the listeners.
static InetSocketAddress	getAddress(Configuration conf)
JobStatus[]	getAllJobs() Get all the jobs submitted.
String	getAssignedTracker(TaskAttemptID taskId) Get tracker name for a given task id.
String	getBuildVersion() Returns the buildVersion of the JobTracker
TaskReport[]	getCleanupTaskReports(JobID jobid) Grab a bunch of info on the cleanup tasks that make up the job
ClusterStatus	getClusterStatus() Deprecated. use getClusterStatus(boolean)
ClusterStatus	getClusterStatus(boolean detailed) Get the current status of the cluster
String	getFilesystemName() Grab the local fs name
int	getInfoPort()
static Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation>	getInstrumentationClass(Configuration conf)
org.apache.hadoop.mapred.JobInProgress	getJob(JobID jobid) Obtain the job object identified by jobid
Counters	getJobCounters(JobID jobid) Grab the current job counters
JobProfile	getJobProfile(JobID jobid) Grab a handle to a job that is already known to the JobTracker.
JobStatus[]	getJobsFromQueue(String queue) Gets all the jobs submitted to the particular Queue
JobStatus	getJobStatus(JobID jobid) Grab a handle to a job that is already known to the JobTracker.
String	getJobTrackerMachine()
static String	getLocalJobFilePath(JobID jobId) Get the localized job file path on the job trackers local file system
TaskReport[]	getMapTaskReports(JobID jobid) Grab a bunch of info on the map tasks that make up the job
JobID	getNewJobId() Allocates a new JobId string.
int	getNextHeartbeatInterval() Calculates next heartbeat interval using cluster size.
Node	getNode(String name) Return the Node in the network topology that corresponds to the hostname
Collection<Node>	getNodesAtMaxLevel() Returns a collection of nodes at the max level
int	getNumberOfUniqueHosts()
int	getNumResolvedTaskTrackers()
int	getNumTaskCacheLevels()
static Node	getParentNode(Node node, int level)
long	getProtocolVersion(String protocol, long clientVersion) Return protocol version corresponding to protocol interface.
JobQueueInfo	getQueueInfo(String queue) Gets scheduling information associated with the particular Job queue
org.apache.hadoop.mapred.QueueManager	getQueueManager() Return the QueueManager associated with the JobTracker.
JobQueueInfo[]	getQueues() Gets set of Job Queues associated with the Job Tracker
long	getRecoveryDuration() How long the jobtracker took to recover from restart.
TaskReport[]	getReduceTaskReports(JobID jobid) Grab a bunch of info on the reduce tasks that make up the job
List<org.apache.hadoop.mapred.JobInProgress>	getRunningJobs() Version that is called from a timer thread, and therefore needs to be careful to synchronize.
TaskReport[]	getSetupTaskReports(JobID jobid) Grab a bunch of info on the setup tasks that make up the job
long	getStartTime()
String	getSystemDir() Grab the jobtracker system directory path where job-specific files are to be placed.
TaskCompletionEvent[]	getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents) Get task completion events for the jobid, starting from fromEventId.
String[]	getTaskDiagnostics(TaskAttemptID taskId) Get the diagnostics for a given task
org.apache.hadoop.mapred.TaskTrackerStatus	getTaskTracker(String trackerID)
org.apache.hadoop.mapred.TaskInProgress	getTip(TaskID tipid) Returns specified TaskInProgress, or null.
int	getTotalSubmissions()
String	getTrackerIdentifier() Get the unique identifier (ie.
int	getTrackerPort()
boolean	hasRecovered() Whether the JT has recovered upon restart
boolean	hasRestarted() Whether the JT has restarted
org.apache.hadoop.mapred.HeartbeatResponse	heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus status, boolean restarted, boolean initialContact, boolean acceptNewTasks, short responseId) The periodic heartbeat mechanism between the TaskTracker and the JobTracker.
void	initJob(org.apache.hadoop.mapred.JobInProgress job) Initialize the Job
boolean	isBlacklisted(String trackerID) Whether the tracker is blacklisted or not
JobStatus[]	jobsToComplete() Get the jobs that are not completed and not failed
void	killJob(JobID jobid) Kill the indicated job
boolean	killTask(TaskAttemptID taskid, boolean shouldFail) Mark a Task to be killed
static void	main(String[] argv) Start the JobTracker process.
void	offerService() Run forever
void	refreshServiceAcl() Refresh the service-level authorization policy in-effect.
void	removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener) Unregisters a JobInProgressListener from this TaskTrackerManager.
void	reportTaskTrackerError(String taskTracker, String errorClass, String errorMessage) Report a problem to the job tracker.
Node	resolveAndAddToTopology(String name)
Vector<org.apache.hadoop.mapred.JobInProgress>	runningJobs()
static void	setInstrumentationClass(Configuration conf, Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> t)
void	setJobPriority(JobID jobid, String priority) Set the priority of a job
static JobTracker	startTracker(JobConf conf) Start the JobTracker with given configuration.
static JobTracker	startTracker(JobConf conf, String identifier)
void	stopTracker()
JobStatus	submitJob(JobID jobId) JobTracker.submitJob() kicks off a new job.
List<List<String>>	taskTrackerNames() Get the active and blacklisted task tracker names in the cluster.
Collection<org.apache.hadoop.mapred.TaskTrackerStatus>	taskTrackers() Get all the task trackers in the cluster

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

HEARTBEAT_INTERVAL_MIN

public static final int HEARTBEAT_INTERVAL_MIN

See Also:

Constant Field Values

CLUSTER_INCREMENT

public static final int CLUSTER_INCREMENT

See Also:

Constant Field Values

COUNTER_UPDATE_INTERVAL

public static final long COUNTER_UPDATE_INTERVAL

See Also:

Constant Field Values

SUCCESS

public static final int SUCCESS

See Also:

Constant Field Values

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND

See Also:

Constant Field Values

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH

The custom http header used for the map output length.

See Also:

Constant Field Values

RAW_MAP_OUTPUT_LENGTH

public static final String RAW_MAP_OUTPUT_LENGTH

The custom http header used for the "raw" map output length.

See Also:

Constant Field Values

FROM_MAP_TASK

public static final String FROM_MAP_TASK

The map task from which the map output data is being transferred

See Also:

Constant Field Values

FOR_REDUCE_TASK

public static final String FOR_REDUCE_TASK

The reduce task number for which this map output is being transferred

See Also:

Constant Field Values

WORKDIR

public static final String WORKDIR

See Also:

Constant Field Values

versionID

public static final long versionID

version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549. version 5 introduced that removes locateMapOutputs and instead uses getTaskCompletionEvents to figure finished maps and fetch the outputs version 6 adds maxTasks to TaskTrackerStatus for HADOOP-1245 version 7 replaces maxTasks by maxMapTasks and maxReduceTasks in TaskTrackerStatus for HADOOP-1274 Version 8: HeartbeatResponse is added with the next heartbeat interval. version 9 changes the counter representation for HADOOP-2248 version 10 changes the TaskStatus representation for HADOOP-2208 version 11 changes string to JobID in getTaskCompletionEvents(). version 12 changes the counters representation for HADOOP-1915 version 13 added call getBuildVersion() for HADOOP-236 Version 14: replaced getFilesystemName with getSystemDir for HADOOP-3135 Version 15: Changed format of Task and TaskStatus for HADOOP-153 Version 16: adds ResourceStatus to TaskTrackerStatus for HADOOP-3759 Version 17: Changed format of Task and TaskStatus for HADOOP-3150 Version 18: Changed status message due to changes in TaskStatus Version 19: Changed heartbeat to piggyback JobTracker restart information so that the TaskTracker can synchronize itself. Version 20: Changed status message due to changes in TaskStatus (HADOOP-4232) Version 21: Changed information reported in TaskTrackerStatus' ResourceStatus and the corresponding accessor methods (HADOOP-4035) Version 22: Replaced parameter 'initialContact' with 'restarted' in heartbeat method (HADOOP-4305) Version 23: Added parameter 'initialContact' again in heartbeat method (HADOOP-4869) Version 24: Changed format of Task and TaskStatus for HADOOP-4759 Version 25: JobIDs are passed in response to JobTracker restart

See Also:

Constant Field Values

TRACKERS_OK

public static final int TRACKERS_OK

See Also:

Constant Field Values

UNKNOWN_TASKTRACKER

public static final int UNKNOWN_TASKTRACKER

See Also:

Constant Field Values

versionID

public static final long versionID

See Also:

Constant Field Values

Method Detail

startTracker

public static JobTracker startTracker(JobConf conf)
                               throws IOException,
                                      InterruptedException

Start the JobTracker with given configuration. The conf will be modified to reflect the actual ports on which the JobTracker is up and running if the user passes the port as zero.

Parameters:

conf - configuration for the JobTracker.

Throws:

IOException

InterruptedException

startTracker

public static JobTracker startTracker(JobConf conf,
                                      String identifier)
                               throws IOException,
                                      InterruptedException

Throws:

IOException

InterruptedException

stopTracker

public void stopTracker()
                 throws IOException

Throws:

IOException

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolVersion in interface VersionedProtocol

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

Returns:

the version that the server will speak

Throws:

IOException

hasRestarted

public boolean hasRestarted()

Whether the JT has restarted

hasRecovered

public boolean hasRecovered()

Whether the JT has recovered upon restart

getRecoveryDuration

public long getRecoveryDuration()

How long the jobtracker took to recover from restart.

getInstrumentationClass

public static Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> getInstrumentationClass(Configuration conf)

setInstrumentationClass

public static void setInstrumentationClass(Configuration conf,
                                           Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> t)

getAddress

public static InetSocketAddress getAddress(Configuration conf)

offerService

public void offerService()
                  throws InterruptedException,
                         IOException

Run forever

Throws:

InterruptedException

IOException

getTotalSubmissions

public int getTotalSubmissions()

getJobTrackerMachine

public String getJobTrackerMachine()

getTrackerIdentifier

public String getTrackerIdentifier()

Get the unique identifier (ie. timestamp) of this job tracker start.

Returns:

a string with a unique identifier

getTrackerPort

public int getTrackerPort()

getInfoPort

public int getInfoPort()

getStartTime

public long getStartTime()

runningJobs

public Vector<org.apache.hadoop.mapred.JobInProgress> runningJobs()

getRunningJobs

public List<org.apache.hadoop.mapred.JobInProgress> getRunningJobs()

Version that is called from a timer thread, and therefore needs to be careful to synchronize.

failedJobs

public Vector<org.apache.hadoop.mapred.JobInProgress> failedJobs()

completedJobs

public Vector<org.apache.hadoop.mapred.JobInProgress> completedJobs()

taskTrackers

public Collection<org.apache.hadoop.mapred.TaskTrackerStatus> taskTrackers()

Get all the task trackers in the cluster

Returns:

Collection of TaskTrackerStatus

activeTaskTrackers

public Collection<org.apache.hadoop.mapred.TaskTrackerStatus> activeTaskTrackers()

Get the active task tracker statuses in the cluster

Returns:

Collection of active TaskTrackerStatus

taskTrackerNames

public List<List<String>> taskTrackerNames()

Get the active and blacklisted task tracker names in the cluster. The first element in the returned list contains the list of active tracker names. The second element in the returned list contains the list of blacklisted tracker names.

blacklistedTaskTrackers

public Collection<org.apache.hadoop.mapred.TaskTrackerStatus> blacklistedTaskTrackers()

Get the blacklisted task tracker statuses in the cluster

Returns:

Collection of blacklisted TaskTrackerStatus

isBlacklisted

public boolean isBlacklisted(String trackerID)

Whether the tracker is blacklisted or not

Parameters:

trackerID -

Returns:

true if blacklisted, false otherwise

getTaskTracker

public org.apache.hadoop.mapred.TaskTrackerStatus getTaskTracker(String trackerID)

resolveAndAddToTopology

public Node resolveAndAddToTopology(String name)

getNodesAtMaxLevel

public Collection<Node> getNodesAtMaxLevel()

Returns a collection of nodes at the max level

getParentNode

public static Node getParentNode(Node node,
                                 int level)

getNode

public Node getNode(String name)

Return the Node in the network topology that corresponds to the hostname

getNumTaskCacheLevels

public int getNumTaskCacheLevels()

getNumResolvedTaskTrackers

public int getNumResolvedTaskTrackers()

getNumberOfUniqueHosts

public int getNumberOfUniqueHosts()

Returns:

The number of unique hosts running tasktrackers.

addJobInProgressListener

public void addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)

Registers a JobInProgressListener for updates from this TaskTrackerManager.

removeJobInProgressListener

public void removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)

Unregisters a JobInProgressListener from this TaskTrackerManager.

getQueueManager

public org.apache.hadoop.mapred.QueueManager getQueueManager()

Return the QueueManager associated with the JobTracker.

Returns:

the QueueManager

getBuildVersion

public String getBuildVersion()
                       throws IOException

Returns the buildVersion of the JobTracker

Throws:

IOException

heartbeat

public org.apache.hadoop.mapred.HeartbeatResponse heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus status,
                                                            boolean restarted,
                                                            boolean initialContact,
                                                            boolean acceptNewTasks,
                                                            short responseId)
                                                     throws IOException

The periodic heartbeat mechanism between the TaskTracker and the JobTracker. The JobTracker processes the status information sent by the TaskTracker and responds with instructions to start/stop tasks or jobs, and also 'reset' instructions during contingencies.

Parameters:

status - the status update

restarted - true if the process has just started or restarted, false otherwise

initialContact - true if this is first interaction since 'refresh', false otherwise.

acceptNewTasks - true if the TaskTracker is ready to accept new tasks to run.

responseId - the last responseId successfully acted upon by the TaskTracker.

Returns:

a HeartbeatResponse with fresh instructions.

Throws:

IOException

getNextHeartbeatInterval

public int getNextHeartbeatInterval()

Calculates next heartbeat interval using cluster size. Heartbeat interval is incremented 1second for every 50 nodes.

Returns:

next heartbeat interval.

getFilesystemName

public String getFilesystemName()
                         throws IOException

Grab the local fs name

Throws:

IOException

reportTaskTrackerError

public void reportTaskTrackerError(String taskTracker,
                                   String errorClass,
                                   String errorMessage)
                            throws IOException

Report a problem to the job tracker.

Parameters:

taskTracker - the name of the task tracker

errorClass - the kind of error (eg. the class that was thrown)

errorMessage - the human readable error message

Throws:

IOException - if there was a problem in communication or on the remote side

getNewJobId

public JobID getNewJobId()
                  throws IOException

Allocates a new JobId string.

Returns:

a unique job name for submitting jobs.

Throws:

IOException

submitJob

public JobStatus submitJob(JobID jobId)
                    throws IOException

JobTracker.submitJob() kicks off a new job. Create a 'JobInProgress' object, which contains both JobProfile and JobStatus. Those two sub-objects are sometimes shipped outside of the JobTracker. But JobInProgress adds info that's useful for the JobTracker alone.

Throws:

IOException

getClusterStatus

@Deprecated
public ClusterStatus getClusterStatus()

Deprecated. use getClusterStatus(boolean)

Returns:

a summary of the cluster's status.

getClusterStatus

public ClusterStatus getClusterStatus(boolean detailed)

Get the current status of the cluster

Parameters:

detailed - if true then report tracker names as well

Returns:

summary of the state of the cluster

killJob

public void killJob(JobID jobid)
             throws IOException

Kill the indicated job

Throws:

IOException

initJob

public void initJob(org.apache.hadoop.mapred.JobInProgress job)

Initialize the Job

Parameters:

job - JobInProgress object

failJob

public void failJob(org.apache.hadoop.mapred.JobInProgress job)

Fail a job and inform the listeners. Other components in the framework should use this to fail a job.

Parameters:

job - JobInProgress object

setJobPriority

public void setJobPriority(JobID jobid,
                           String priority)
                    throws IOException

Set the priority of a job

Parameters:

jobid - id of the job

priority - new priority of the job

Throws:

IOException

getJobProfile

public JobProfile getJobProfile(JobID jobid)

Grab a handle to a job that is already known to the JobTracker.

Returns:

Profile of the job, or null if not found.

getJobStatus

public JobStatus getJobStatus(JobID jobid)

Grab a handle to a job that is already known to the JobTracker.

Returns:

Status of the job, or null if not found.

getJobCounters

public Counters getJobCounters(JobID jobid)

Grab the current job counters

getMapTaskReports

public TaskReport[] getMapTaskReports(JobID jobid)

Grab a bunch of info on the map tasks that make up the job

getReduceTaskReports

public TaskReport[] getReduceTaskReports(JobID jobid)

Grab a bunch of info on the reduce tasks that make up the job

getCleanupTaskReports

public TaskReport[] getCleanupTaskReports(JobID jobid)

Grab a bunch of info on the cleanup tasks that make up the job

getSetupTaskReports

public TaskReport[] getSetupTaskReports(JobID jobid)

Grab a bunch of info on the setup tasks that make up the job

getTaskCompletionEvents

public TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                                     int fromEventId,
                                                     int maxEvents)
                                              throws IOException

Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Parameters:

jobid - job id

fromEventId - event id to start from.

maxEvents - the max number of events we want to look at

Returns:

array of task completion events.

Throws:

IOException

getTaskDiagnostics

public String[] getTaskDiagnostics(TaskAttemptID taskId)
                            throws IOException

Get the diagnostics for a given task

Parameters:

taskId - the id of the task

Returns:

an array of the diagnostic messages

Throws:

IOException

getTip

public org.apache.hadoop.mapred.TaskInProgress getTip(TaskID tipid)

Returns specified TaskInProgress, or null.

killTask

public boolean killTask(TaskAttemptID taskid,
                        boolean shouldFail)
                 throws IOException

Mark a Task to be killed

Parameters:

taskid - the id of the task to kill.

shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.

Throws:

IOException

getAssignedTracker

public String getAssignedTracker(TaskAttemptID taskId)

Get tracker name for a given task id.

Parameters:

taskId - the name of the task

Returns:

The name of the task tracker

jobsToComplete

public JobStatus[] jobsToComplete()

Get the jobs that are not completed and not failed

Returns:

array of JobStatus for the running/to-be-run jobs.

getAllJobs

public JobStatus[] getAllJobs()

Get all the jobs submitted.

Returns:

array of JobStatus for the submitted jobs

getSystemDir

public String getSystemDir()

Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:

the system directory where job-specific files are to be placed.

See Also:

JobSubmissionProtocol.getSystemDir()

getJob

public org.apache.hadoop.mapred.JobInProgress getJob(JobID jobid)

Obtain the job object identified by jobid

Returns:

jobInProgress object

getLocalJobFilePath

public static String getLocalJobFilePath(JobID jobId)

Get the localized job file path on the job trackers local file system

Parameters:

jobId - id of the job

Returns:

the path of the job conf file on the local file system

main

public static void main(String[] argv)
                 throws IOException,
                        InterruptedException

Start the JobTracker process. This is used only for debugging. As a rule, JobTracker should be run as part of the DFS Namenode process.

Throws:

IOException

InterruptedException

getQueues

public JobQueueInfo[] getQueues()
                         throws IOException

Gets set of Job Queues associated with the Job Tracker

Returns:

Array of the Job Queue Information Object

Throws:

IOException

getQueueInfo

public JobQueueInfo getQueueInfo(String queue)
                          throws IOException

Gets scheduling information associated with the particular Job queue

Parameters:

queue - Queue Name

Returns:

Scheduling Information of the Queue

Throws:

IOException

getJobsFromQueue

public JobStatus[] getJobsFromQueue(String queue)
                             throws IOException

Gets all the jobs submitted to the particular Queue

Parameters:

queue - Queue name

Returns:

array of JobStatus for the submitted jobs

Throws:

IOException

refreshServiceAcl

public void refreshServiceAcl()
                       throws IOException

Description copied from interface: RefreshAuthorizationPolicyProtocol

Refresh the service-level authorization policy in-effect.

Specified by:

refreshServiceAcl in interface RefreshAuthorizationPolicyProtocol

Throws:

IOException