1 | .\" |
---|
2 | .\" Copyright (c) 2008 Los Alamos National Security, LLC All rights reserved. |
---|
3 | .\" Copyright (c) 2008-2009 Sun Microsystems, Inc. All rights reserved. |
---|
4 | .\" |
---|
5 | .\" Man page for ORTE's Hostfile functionality |
---|
6 | .\" |
---|
7 | .\" .TH name section center-footer left-footer center-header |
---|
8 | .TH ORTE_HOSTS 7 "Dec 08, 2009" "1.4" "Open MPI" |
---|
9 | .\" ************************** |
---|
10 | .\" Name Section |
---|
11 | .\" ************************** |
---|
12 | .SH NAME |
---|
13 | . |
---|
14 | OpenRTE Hostfile and HOST Behavior \- Overview of OpenRTE's support for user-supplied hostfiles and comma-delimited lists of hosts |
---|
15 | . |
---|
16 | .\" ************************** |
---|
17 | .\" Description Section |
---|
18 | .\" ************************** |
---|
19 | .SH DESCRIPTION |
---|
20 | . |
---|
21 | .PP |
---|
22 | OpenRTE supports several levels of user-specified host lists based on an established |
---|
23 | precedence order. Users can specify a \fIdefault hostfile\fP that contains a list of |
---|
24 | nodes available to all app_contexts given on the command line. Only \fIone\fP default |
---|
25 | hostfile can be provided for any job. In addition, users |
---|
26 | can specify a \fIhostfile\fP that contains a list of nodes to be used for a specific |
---|
27 | app_context, or can provide a comma-delimited list of nodes to be used for that |
---|
28 | app_context via the \fI-host\fP command line option. |
---|
29 | .sp |
---|
30 | The precedence order applied to these various options depends to some extent on |
---|
31 | the local environment. The following table illustrates how host and hostfile directives |
---|
32 | work together to define the set of hosts upon which a job will execute |
---|
33 | in the absence of a resource manager (RM): |
---|
34 | .sp |
---|
35 | .nf |
---|
36 | default |
---|
37 | hostfile host hostfile Result |
---|
38 | ---------- ------ ---------- ----------------------------------------- |
---|
39 | unset unset unset Job is co-located with mpirun |
---|
40 | unset set unset Host defines resource list for the job |
---|
41 | unset unset set Hostfile defines resource list for the job |
---|
42 | unset set set Hostfile defines resource list for the job, |
---|
43 | then host filters the list to define the final |
---|
44 | set of nodes available to each application |
---|
45 | within the job |
---|
46 | set unset unset Default hostfile defines resource list for the job |
---|
47 | set set unset Default hostfile defines resource list for the job, |
---|
48 | then host filters the list to define the final |
---|
49 | set of nodes available to each application |
---|
50 | within the job |
---|
51 | set set set Default hostfile defines resource list for the job, |
---|
52 | then hostfile filters the list, and then host filters |
---|
53 | the list to define the final set of nodes available |
---|
54 | to each application within the job |
---|
55 | .fi |
---|
56 | .sp |
---|
57 | This changes somewhat in the presence of a RM as that entity specifies the |
---|
58 | initial allocation of nodes. In this case, the default hostfile, hostfile and host |
---|
59 | directives are all used to filter the RM's specification so that a user can utilize different |
---|
60 | portions of the allocation for different jobs. This is done according to the same precedence |
---|
61 | order as in the prior table, with the RM providing the initial pool of nodes. |
---|
62 | .sp |
---|
63 | . |
---|
64 | .\" ************************** |
---|
65 | .\" Relative Indexing |
---|
66 | .\" ************************** |
---|
67 | .SH RELATIVE INDEXING |
---|
68 | . |
---|
69 | .PP |
---|
70 | Once an initial allocation has been specified (whether by an RM, default hostfile, or hostfile), |
---|
71 | subsequent hostfile and -host specifications can be made using relative indexing. This allows a |
---|
72 | user to stipulate which hosts are to be used for a given app_context without specifying the |
---|
73 | particular host name, but rather its relative position in the allocation. |
---|
74 | .sp |
---|
75 | This can probably best be understood through consideration of a few examples. Consider the case |
---|
76 | where an RM has allocated a set of nodes to the user named "foo1, foo2, foo3, foo4". The user |
---|
77 | wants the first app_context to have exclusive use of the first two nodes, and a second app_context |
---|
78 | to use the last two nodes. Of course, the user could printout the allocation to find the names |
---|
79 | of the nodes allocated to them and then use -host to specify this layout, but this is cumbersome |
---|
80 | and would require hand-manipulation for every invocation. |
---|
81 | .sp |
---|
82 | A simpler method is to utilize OpenRTE's relative indexing capability to specify the desired |
---|
83 | layout. In this case, a command line of: |
---|
84 | .sp |
---|
85 | mpirun -pernode -host +n1,+n2 ./app1 : -host +n3,+n4 ./app2 |
---|
86 | .sp |
---|
87 | .PP |
---|
88 | would provide the desired pattern. The "+" syntax indicates that the information is being |
---|
89 | provided as a relative index to the existing allocation. Two methods of relative indexing |
---|
90 | are supported: |
---|
91 | .sp |
---|
92 | .TP |
---|
93 | .B +n<#> |
---|
94 | A relative index into the allocation referencing the <#> node. OpenRTE will substitute |
---|
95 | the <#> node in the allocation |
---|
96 | . |
---|
97 | . |
---|
98 | .TP |
---|
99 | .B +e[:<#>] |
---|
100 | A request for <#> empty nodes - i.e., OpenRTE is to substitute this reference with |
---|
101 | <#> nodes that have not yet been used by any other app_context. If the ":<#>" is not |
---|
102 | provided, OpenRTE will substitute the reference with all empty nodes. Note that OpenRTE |
---|
103 | does track the empty nodes that have been assigned in this manner, so multiple |
---|
104 | uses of this option will result in assignment of unique nodes up to the limit of the |
---|
105 | available empty nodes. Requests for more empty nodes than are available will generate |
---|
106 | an error. |
---|
107 | .sp |
---|
108 | .PP |
---|
109 | Relative indexing can be combined with absolute naming of hosts in any arbitrary manner, |
---|
110 | and can be used in hostfiles as well as with the -host command line option. In addition, |
---|
111 | any slot specification provided in hostfiles will be respected - thus, a user can specify |
---|
112 | that only a certain number of slots from a relative indexed host are to be used for a |
---|
113 | given app_context. |
---|
114 | .sp |
---|
115 | Another example may help illustrate this point. Consider the case where a user has a default |
---|
116 | hostfile containing: |
---|
117 | .sp |
---|
118 | .nf |
---|
119 | dummy1 slots=4 |
---|
120 | dummy2 slots=4 |
---|
121 | dummy3 slots=4 |
---|
122 | dummy4 slots=4 |
---|
123 | dummy5 slots=4 |
---|
124 | .fi |
---|
125 | .sp |
---|
126 | .PP |
---|
127 | This may, for example, be a hostfile that describes a set of commonly-used resources that |
---|
128 | the user wishes to execute applications against. For this particular application, the user |
---|
129 | plans to map byslot, and wants the first two ranks to be on the second node of any allocation, |
---|
130 | the next ranks to land on an empty node, have one rank specifically on dummy4, the next rank |
---|
131 | to be on the second node of the allocation again, and finally any remaining ranks to be on |
---|
132 | whatever empty nodes are left. To accomplish this, the user provides a hostfile of: |
---|
133 | .sp |
---|
134 | .nf |
---|
135 | +n2 slots=2 |
---|
136 | +e:1 |
---|
137 | dummy4 slots=1 |
---|
138 | +n2 |
---|
139 | +e |
---|
140 | .fi |
---|
141 | .sp |
---|
142 | .PP |
---|
143 | The user can now use this information in combination with OpenRTE's sequential mapper to |
---|
144 | obtain their specific layout: |
---|
145 | .sp |
---|
146 | .nf |
---|
147 | mpirun --default-hostfile dummyhosts -hostfile mylayout -mca rmaps seq ./my_app |
---|
148 | .fi |
---|
149 | .sp |
---|
150 | .PP |
---|
151 | which will result in: |
---|
152 | .nf |
---|
153 | .sp |
---|
154 | rank0 being mapped to dummy3 |
---|
155 | .br |
---|
156 | rank1 to dummy1 as the first empty node |
---|
157 | .br |
---|
158 | rank2 to dummy4 |
---|
159 | .br |
---|
160 | rank3 to dummy3 |
---|
161 | .br |
---|
162 | rank4 to dummy2 and rank5 to dummy5 as the last remaining unused nodes |
---|
163 | .sp |
---|
164 | .fi |
---|
165 | Note that the sequential mapper ignores the number of slots arguments as it only |
---|
166 | maps one rank at a time to each node in the list. |
---|
167 | .sp |
---|
168 | If the default round-robin mapper had been used, then the mapping would have resulted in: |
---|
169 | .sp |
---|
170 | .nf |
---|
171 | ranks 0 and 1 being mapped to dummy3 since two slots were specified |
---|
172 | .br |
---|
173 | ranks 2-5 on dummy1 as the first empty node, which has four slots |
---|
174 | .br |
---|
175 | rank6 on dummy4 since the hostfile specifies only a single slot from that node is to be used |
---|
176 | .br |
---|
177 | ranks 7 and 8 on dummy3 since only two slots remain available |
---|
178 | .br |
---|
179 | ranks 9-12 on dummy2 since it is the next available empty node and has four slots |
---|
180 | .br |
---|
181 | ranks 13-16 on dummy5 since it is the last remaining unused node and has four slots |
---|
182 | .fi |
---|
183 | .sp |
---|
184 | .PP |
---|
185 | Thus, the use of relative indexing can allow for complex mappings to be ported across |
---|
186 | allocations, including those obtained from automated resource managers, without the need |
---|
187 | for manual manipulation of scripts and/or command lines. |
---|
188 | . |
---|
189 | . |
---|
190 | .\" ************************** |
---|
191 | .\" See Also Section |
---|
192 | .\" ************************** |
---|
193 | . |
---|
194 | .SH SEE ALSO |
---|
195 | orterun(1) |
---|
196 | . |
---|