source: proiecte/HadoopJUnit/hadoop-0.20.1/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml @ 120

Last change on this file since 120 was 120, checked in by (none), 14 years ago

Added the mail files for the Hadoop JUNit Project
  • Property svn:executable set to *
File size: 20.0 KB
Line 
1<?xml version="1.0"?>
2<!--
3  Copyright 2002-2004 The Apache Software Foundation
4
5  Licensed under the Apache License, Version 2.0 (the "License");
6  you may not use this file except in compliance with the License.
7  You may obtain a copy of the License at
8
9      http://www.apache.org/licenses/LICENSE-2.0
10
11  Unless required by applicable law or agreed to in writing, software
12  distributed under the License is distributed on an "AS IS" BASIS,
13  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14  See the License for the specific language governing permissions and
15  limitations under the License.
16-->
17
18<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
19          "http://forrest.apache.org/dtd/document-v20.dtd">
20
21
22<document>
23
24  <header>
25    <title>
26      HDFS User Guide
27    </title>
28  </header>
29
30  <body>
31    <section> <title>Purpose</title>
32      <p>
33 This document is a starting point for users working with
34 Hadoop Distributed File System (HDFS) either as a part of a
35 <a href="http://hadoop.apache.org/">Hadoop</a>
36 cluster or as a stand-alone general purpose distributed file system.
37 While HDFS is designed to "just work" in many environments, a working
38 knowledge of HDFS helps greatly with configuration improvements and
39 diagnostics on a specific cluster.
40      </p>
41    </section>
42
43    <section> <title> Overview </title>
44      <p>
45 HDFS is the primary distributed storage used by Hadoop applications. A
46 HDFS cluster primarily consists of a NameNode that manages the
47 file system metadata and DataNodes that store the actual data. The
48 <a href="hdfs_design.html">HDFS Architecture</a> describes HDFS in detail. This user guide primarily deals with
49 the interaction of users and administrators with HDFS clusters.
50 The <a href="images/hdfsarchitecture.gif">HDFS architecture diagram</a> depicts
51 basic interactions among NameNode, the DataNodes, and the clients.
52 Clients contact NameNode for file metadata or file modifications and perform
53 actual file I/O directly with the DataNodes.
54      </p>
55      <p>
56 The following are some of the salient features that could be of
57 interest to many users.
58      </p> 
59    <ul>
60    <li>
61        Hadoop, including HDFS, is well suited for distributed storage
62        and distributed processing using commodity hardware. It is fault
63        tolerant, scalable, and extremely simple to expand.
64        <a href="mapred_tutorial.html">Map/Reduce</a>,
65        well known for its simplicity and applicability for large set of
66        distributed applications, is an integral part of Hadoop.
67    </li>
68    <li>
69        HDFS is highly configurable with a default configuration well
70        suited for many installations. Most of the time, configuration
71        needs to be tuned only for very large clusters.
72    </li>
73    <li>
74        Hadoop is written in Java and is supported on all major platforms.
75    </li>
76    <li>
77        Hadoop supports shell-like commands to interact with HDFS directly.
78    </li>
79    <li>
80        The NameNode and Datanodes have built in web servers that makes it
81        easy to check current status of the cluster.
82    </li>
83    <li>
84        New features and improvements are regularly implemented in HDFS.
85        The following is a subset of useful features in HDFS:
86      <ul>
87        <li>
88                File permissions and authentication.
89        </li>
90        <li>
91                <em>Rack awareness</em>: to take a node's physical location into
92                account while scheduling tasks and allocating storage.
93        </li>
94        <li>
95                Safemode: an administrative mode for maintenance.
96        </li>
97        <li>
98                <code>fsck</code>: a utility to diagnose health of the file system, to
99                find missing files or blocks.
100        </li>
101        <li>
102                Rebalancer: tool to balance the cluster when the data is
103                unevenly distributed among DataNodes.
104        </li>
105        <li>
106                Upgrade and rollback: after a software upgrade,
107            it is possible to
108                rollback to HDFS' state before the upgrade in case of unexpected
109                problems.
110        </li>
111        <li>
112                Secondary NameNode: performs periodic checkpoints of the
113                namespace and helps keep the size of file containing log of HDFS
114                modifications within certain limits at the NameNode.
115        </li>
116      </ul>
117    </li>
118    </ul>
119   
120    </section> <section> <title> Pre-requisites </title>
121    <p>
122        The following documents describe installation and set up of a
123        Hadoop cluster :
124    </p>
125        <ul>
126        <li>
127                <a href="quickstart.html">Hadoop Quick Start</a>
128                for first-time users.
129        </li>
130        <li>
131                <a href="cluster_setup.html">Hadoop Cluster Setup</a>
132                for large, distributed clusters.
133        </li>
134    </ul>
135    <p>
136        The rest of this document assumes the user is able to set up and run a
137        HDFS with at least one DataNode. For the purpose of this document,
138        both the NameNode and DataNode could be running on the same physical
139        machine.       
140    </p>
141     
142    </section> <section> <title> Web Interface </title>
143 <p>
144        NameNode and DataNode each run an internal web server in order to
145        display basic information about the current status of the cluster.
146        With the default configuration, the NameNode front page is at
147        <code>http://namenode-name:50070/</code>.
148        It lists the DataNodes in the cluster and basic statistics of the
149        cluster. The web interface can also be used to browse the file
150        system (using "Browse the file system" link on the NameNode front
151        page).
152 </p> 
153       
154    </section> <section> <title>Shell Commands</title>
155        <p>
156      Hadoop includes various shell-like commands that directly
157      interact with HDFS and other file systems that Hadoop supports.
158      The command
159      <code>bin/hadoop fs -help</code>
160      lists the commands supported by Hadoop
161      shell. Furthermore, the command
162      <code>bin/hadoop fs -help command-name</code>
163      displays more detailed help for a command. These commands support
164      most of the normal files ystem operations like copying files,
165      changing file permissions, etc. It also supports a few HDFS
166      specific operations like changing replication of files.
167     </p>
168
169   <section> <title> DFSAdmin Command </title>
170   <p>
171        The <code>bin/hadoop dfsadmin</code>
172        command supports a few HDFS administration related operations.
173        The <code>bin/hadoop dfsadmin -help</code> command
174        lists all the commands currently supported. For e.g.:
175   </p> 
176        <ul>
177        <li>
178            <code>-report</code>
179            : reports basic statistics of HDFS. Some of this information is
180            also available on the NameNode front page.
181        </li>
182        <li>
183                <code>-safemode</code>
184                : though usually not required, an administrator can manually enter
185                or leave Safemode.
186        </li>
187        <li>
188                <code>-finalizeUpgrade</code>
189                : removes previous backup of the cluster made during last upgrade.
190        </li>
191    <li>
192      <code>-refreshNodes</code>
193      : Updates the set of hosts allowed to connect to namenode.
194      Re-reads the config file to update values defined by dfs.hosts and
195      dfs.host.exclude and reads the entires (hostnames) in those files.
196      Each entry not defined in dfs.hosts but in dfs.hosts.exclude
197      is decommissioned. Each entry defined in dfs.hosts and also in
198      dfs.host.exclude is stopped from decommissioning if it has aleady
199      been marked for decommission. Entires not present in both the lists
200      are decommissioned.
201    </li>
202        </ul>
203        <p>
204          For command usage, see <a href="commands_manual.html#dfsadmin">dfsadmin command</a>.
205        </p> 
206   </section>
207   
208   </section> <section> <title> Secondary NameNode </title>
209   <p>
210     The NameNode stores modifications to the file system as a log
211     appended to a native file system file (<code>edits</code>).
212        When a NameNode starts up, it reads HDFS state from an image
213        file (<code>fsimage</code>) and then applies edits from the
214    edits log file. It then writes new HDFS state to the <code>fsimage</code>
215    and starts normal
216        operation with an empty edits file. Since NameNode merges
217        <code>fsimage</code> and <code>edits</code> files only during start up,
218    the edits log file could get very large over time on a busy cluster.
219    Another side effect of a larger edits file is that next
220    restart of NameNode takes longer.
221   </p>
222   <p>
223     The secondary NameNode merges the fsimage and the edits log files periodically
224     and keeps edits log size within a limit. It is usually run on a
225     different machine than the primary NameNode since its memory requirements
226     are on the same order as the primary NameNode. The secondary
227     NameNode is started by <code>bin/start-dfs.sh</code> on the nodes
228     specified in <code>conf/masters</code> file.
229   </p>
230   <p>
231     The start of the checkpoint process on the secondary NameNode is
232     controlled by two configuration parameters.
233   </p>
234   <ul>
235      <li>
236        <code>fs.checkpoint.period</code>, set to 1 hour by default, specifies
237        the maximum delay between two consecutive checkpoints, and
238      </li>
239      <li>
240        <code>fs.checkpoint.size</code>, set to 64MB by default, defines the
241        size of the edits log file that forces an urgent checkpoint even if
242        the maximum checkpoint delay is not reached.
243      </li>
244   </ul>
245   <p>
246     The secondary NameNode stores the latest checkpoint in a 
247     directory which is structured the same way as the primary NameNode's
248     directory. So that the check pointed image is always ready to be
249     read by the primary NameNode if necessary.
250   </p>
251   <p>
252     The latest checkpoint can be imported to the primary NameNode if
253     all other copies of the image and the edits files are lost.
254     In order to do that one should:
255   </p>
256   <ul>
257      <li>
258        Create an empty directory specified in the
259        <code>dfs.name.dir</code> configuration variable;
260      </li>
261      <li>
262        Specify the location of the checkpoint directory in the
263        configuration variable <code>fs.checkpoint.dir</code>;
264      </li>
265      <li>
266        and start the NameNode with <code>-importCheckpoint</code> option.
267      </li>
268   </ul>
269   <p>
270     The NameNode will upload the checkpoint from the
271     <code>fs.checkpoint.dir</code> directory and then save it to the NameNode
272     directory(s) set in <code>dfs.name.dir</code>.
273     The NameNode will fail if a legal image is contained in
274     <code>dfs.name.dir</code>.
275     The NameNode verifies that the image in <code>fs.checkpoint.dir</code> is
276     consistent, but does not modify it in any way.
277   </p>
278   <p>
279     For command usage, see <a href="commands_manual.html#secondarynamenode"><code>secondarynamenode</code> command</a>.
280   </p>
281   
282   </section> <section> <title> Rebalancer </title>
283    <p>
284      HDFS data might not always be be placed uniformly across the
285      DataNode. One common reason is addition of new DataNodes to an
286      existing cluster. While placing new blocks (data for a file is
287      stored as a series of blocks), NameNode considers various
288      parameters before choosing the DataNodes to receive these blocks.
289      Some of the considerations are:
290    </p>
291      <ul>
292      <li>
293        Policy to keep one of the replicas of a block on the same node
294        as the node that is writing the block.
295      </li>
296      <li>
297        Need to spread different replicas of a block across the racks so
298        that cluster can survive loss of whole rack.
299      </li>
300      <li>
301        One of the replicas is usually placed on the same rack as the
302        node writing to the file so that cross-rack network I/O is
303        reduced.
304      </li>
305      <li>
306        Spread HDFS data uniformly across the DataNodes in the cluster.
307      </li>
308      </ul>
309    <p>
310      Due to multiple competing considerations, data might not be
311      uniformly placed across the DataNodes.
312      HDFS provides a tool for administrators that analyzes block
313      placement and rebalanaces data across the DataNode. A brief
314      administrator's guide for rebalancer as a
315      <a href="http://issues.apache.org/jira/secure/attachment/12368261/RebalanceDesign6.pdf">PDF</a>
316      is attached to
317      <a href="http://issues.apache.org/jira/browse/HADOOP-1652">HADOOP-1652</a>.
318    </p>
319    <p>
320     For command usage, see <a href="commands_manual.html#balancer">balancer command</a>.
321   </p>
322   
323   </section> <section> <title> Rack Awareness </title>
324    <p>
325      Typically large Hadoop clusters are arranged in racks and
326      network traffic between different nodes with in the same rack is
327      much more desirable than network traffic across the racks. In
328      addition NameNode tries to place replicas of block on
329      multiple racks for improved fault tolerance. Hadoop lets the
330      cluster administrators decide which rack a node belongs to
331      through configuration variable <code>dfs.network.script</code>. When this
332      script is configured, each node runs the script to determine its
333      rack id. A default installation assumes all the nodes belong to
334      the same rack. This feature and configuration is further described
335      in <a href="http://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf">PDF</a>
336      attached to
337      <a href="http://issues.apache.org/jira/browse/HADOOP-692">HADOOP-692</a>.
338    </p>
339
340   </section> <section> <title> Safemode </title>
341    <p>
342      During start up the NameNode loads the file system state from the
343      fsimage  and the edits log file. It then waits for DataNodes
344      to report their blocks so that it does not prematurely start
345      replicating the blocks though enough replicas already exist in the
346      cluster. During this time NameNode stays in Safemode.
347      Safemode
348      for the NameNode is essentially a read-only mode for the HDFS cluster,
349      where it does not allow any modifications to file system or blocks.
350      Normally the NameNode leaves Safemode automatically after the DataNodes
351      have reported that most file system blocks are available.
352      If required, HDFS could be placed in Safemode explicitly
353      using <code>'bin/hadoop dfsadmin -safemode'</code> command. NameNode front
354      page shows whether Safemode is on or off. A more detailed
355      description and configuration is maintained as JavaDoc for
356      <a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/dfs/NameNode.html#setSafeMode(org.apache.hadoop.dfs.FSConstants.SafeModeAction)"><code>setSafeMode()</code></a>.
357    </p>
358   
359   </section> <section> <title> fsck </title>
360     <p>   
361      HDFS supports the <code>fsck</code> command to check for various
362      inconsistencies.
363      It it is designed for reporting problems with various
364      files, for example, missing blocks for a file or under-replicated
365      blocks. Unlike a traditional <code>fsck</code> utility for native file systems,
366      this command does not correct the errors it detects. Normally NameNode
367      automatically corrects most of the recoverable failures. By default
368      <code>fsck</code> ignores open files but provides an option to select all files during reporting.
369      The HDFS <code>fsck</code> command is not a
370      Hadoop shell command. It can be run as '<code>bin/hadoop fsck</code>'.
371      For command usage, see <a href="commands_manual.html#fsck"><code>fsck</code> command</a>.
372      <code>fsck</code> can be run on the whole file system or on a subset of files.
373     </p>
374     
375   </section> <section> <title> Upgrade and Rollback </title>
376     <p>
377      When Hadoop is upgraded on an existing cluster, as with any
378      software upgrade, it is possible there are new bugs or
379      incompatible changes that affect existing applications and were
380      not discovered earlier. In any non-trivial HDFS installation, it
381      is not an option to loose any data, let alone to restart HDFS from
382      scratch. HDFS allows administrators to go back to earlier version
383      of Hadoop and rollback the cluster to the state it was in
384      before
385      the upgrade. HDFS upgrade is described in more detail in
386      <a href="http://wiki.apache.org/hadoop/Hadoop%20Upgrade">upgrade wiki</a>.
387      HDFS can have one such backup at a time. Before upgrading,
388      administrators need to remove existing backup using <code>bin/hadoop
389      dfsadmin -finalizeUpgrade</code> command. The following
390      briefly describes the typical upgrade procedure:
391     </p>
392      <ul>
393      <li>
394        Before upgrading Hadoop software,
395        <em>finalize</em> if there an existing backup.
396        <code>dfsadmin -upgradeProgress status</code>
397        can tell if the cluster needs to be <em>finalized</em>.
398      </li>
399      <li>Stop the cluster and distribute new version of Hadoop.</li>
400      <li>
401        Run the new version with <code>-upgrade</code> option
402        (<code>bin/start-dfs.sh -upgrade</code>).
403      </li>
404      <li>
405        Most of the time, cluster works just fine. Once the new HDFS is
406        considered working well (may be after a few days of operation),
407        finalize the upgrade. Note that until the cluster is finalized,
408        deleting the files that existed before the upgrade does not free
409        up real disk space on the DataNodes.
410      </li>
411      <li>
412        If there is a need to move back to the old version,
413        <ul>
414          <li> stop the cluster and distribute earlier version of Hadoop. </li>
415          <li> start the cluster with rollback option.
416            (<code>bin/start-dfs.h -rollback</code>).
417          </li>
418        </ul>
419      </li>
420      </ul>
421   
422   </section> <section> <title> File Permissions and Security </title>
423     <p>           
424      The file permissions are designed to be similar to file permissions on
425      other familiar platforms like Linux. Currently, security is limited
426      to simple file permissions. The user that starts NameNode is
427      treated as the superuser for HDFS. Future versions of HDFS will
428      support network authentication protocols like Kerberos for user
429      authentication and encryption of data transfers. The details are discussed in the
430      <a href="hdfs_permissions_guide.html">HDFS Admin Guide: Permissions</a>.
431     </p>
432     
433   </section> <section> <title> Scalability </title>
434     <p>
435      Hadoop currently runs on clusters with thousands of nodes.
436      <a href="http://wiki.apache.org/hadoop/PoweredBy">Powered By Hadoop</a>
437      lists some of the organizations that deploy Hadoop on large
438      clusters. HDFS has one NameNode for each cluster. Currently
439      the total memory available on NameNode is the primary scalability
440      limitation. On very large clusters, increasing average size of
441      files stored in HDFS helps with increasing cluster size without
442      increasing memory requirements on NameNode.
443   
444      The default configuration may not suite very large clustes.
445      <a href="http://wiki.apache.org/hadoop/FAQ">Hadoop FAQ</a> page lists
446      suggested configuration improvements for large Hadoop clusters.
447     </p>
448     
449   </section> <section> <title> Related Documentation </title>         
450      <p>
451      This user guide is a good starting point for
452      working with HDFS. While the user guide continues to improve,
453      there is a large wealth of documentation about Hadoop and HDFS.
454      The following list is a starting point for further exploration:
455      </p>
456      <ul>
457      <li>
458        <a href="http://hadoop.apache.org/">Hadoop Home Page</a>: The start page for everything Hadoop.
459      </li>
460      <li>
461        <a href="http://wiki.apache.org/hadoop/FrontPage">Hadoop Wiki</a>
462        : Front page for Hadoop Wiki documentation. Unlike this
463        guide which is part of Hadoop source tree, Hadoop Wiki is
464        regularly edited by Hadoop Community.
465      </li>
466      <li> <a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a> from Hadoop Wiki.
467      </li>
468      <li>
469        Hadoop <a href="ext:api">JavaDoc API</a>.
470      </li>
471      <li>
472        Hadoop User Mailing List :
473        <a href="mailto:core-user@hadoop.apache.org">core-user[at]hadoop.apache.org</a>.
474      </li>
475      <li>
476         Explore <code>src/hdfs/hdfs-default.xml</code>.
477         It includes brief
478         description of most of the configuration variables available.
479      </li>
480      <li>
481        <a href="commands_manual.html">Hadoop Command Guide</a>: commands usage.
482      </li>
483      </ul>
484     </section>
485     
486  </body>
487</document>
488       
489
Note: See TracBrowser for help on using the repository browser.