[120] | 1 | ### "Gridmix" Benchmark ### |
---|
| 2 | |
---|
| 3 | Contents: |
---|
| 4 | |
---|
| 5 | 0 Overview |
---|
| 6 | 1 Getting Started |
---|
| 7 | 1.0 Build |
---|
| 8 | 1.1 Configure |
---|
| 9 | 1.2 Generate test data |
---|
| 10 | 2 Running |
---|
| 11 | 2.0 General |
---|
| 12 | 2.1 Non-Hod cluster |
---|
| 13 | 2.2 Hod |
---|
| 14 | 2.2.0 Static cluster |
---|
| 15 | 2.2.1 Hod cluster |
---|
| 16 | |
---|
| 17 | |
---|
| 18 | * 0 Overview |
---|
| 19 | |
---|
| 20 | The scripts in this package model a cluster workload. The workload is |
---|
| 21 | simulated by generating random data and submitting map/reduce jobs that |
---|
| 22 | mimic observed data-access patterns in user jobs. The full benchmark |
---|
| 23 | generates approximately 2.5TB of (often compressed) input data operated on |
---|
| 24 | by the following simulated jobs: |
---|
| 25 | |
---|
| 26 | 1) Three stage map/reduce job |
---|
| 27 | Input: 500GB compressed (2TB uncompressed) SequenceFile |
---|
| 28 | (k,v) = (5 words, 100 words) |
---|
| 29 | hadoop-env: FIXCOMPSEQ |
---|
| 30 | Compute1: keep 10% map, 40% reduce |
---|
| 31 | Compute2: keep 100% map, 77% reduce |
---|
| 32 | Input from Compute1 |
---|
| 33 | Compute3: keep 116% map, 91% reduce |
---|
| 34 | Input from Compute2 |
---|
| 35 | Motivation: Many user workloads are implemented as pipelined map/reduce |
---|
| 36 | jobs, including Pig workloads |
---|
| 37 | |
---|
| 38 | 2) Large sort of variable key/value size |
---|
| 39 | Input: 500GB compressed (2TB uncompressed) SequenceFile |
---|
| 40 | (k,v) = (5-10 words, 100-10000 words) |
---|
| 41 | hadoop-env: VARCOMPSEQ |
---|
| 42 | Compute: keep 100% map, 100% reduce |
---|
| 43 | Motivation: Processing large, compressed datsets is common. |
---|
| 44 | |
---|
| 45 | 3) Reference select |
---|
| 46 | Input: 500GB compressed (2TB uncompressed) SequenceFile |
---|
| 47 | (k,v) = (5-10 words, 100-10000 words) |
---|
| 48 | hadoop-env: VARCOMPSEQ |
---|
| 49 | Compute: keep 0.2% map, 5% reduce |
---|
| 50 | 1 Reducer |
---|
| 51 | Motivation: Sampling from a large, reference dataset is common. |
---|
| 52 | |
---|
| 53 | 4) Indirect Read |
---|
| 54 | Input: 500GB compressed (2TB uncompressed) Text |
---|
| 55 | (k,v) = (5 words, 20 words) |
---|
| 56 | hadoop-env: FIXCOMPTEXT |
---|
| 57 | Compute: keep 50% map, 100% reduce Each map reads 1 input file, |
---|
| 58 | adding additional input files from the output of the |
---|
| 59 | previous iteration for 10 iterations |
---|
| 60 | Motivation: User jobs in the wild will often take input data without |
---|
| 61 | consulting the framework. This simulates an iterative job |
---|
| 62 | whose input data is all "indirect," i.e. given to the |
---|
| 63 | framework sans locality metadata. |
---|
| 64 | |
---|
| 65 | 5) API text sort (java, pipes, streaming) |
---|
| 66 | Input: 500GB uncompressed Text |
---|
| 67 | (k,v) = (1-10 words, 0-200 words) |
---|
| 68 | hadoop-env: VARINFLTEXT |
---|
| 69 | Compute: keep 100% map, 100% reduce |
---|
| 70 | Motivation: This benchmark should exercise each of the APIs to |
---|
| 71 | map/reduce |
---|
| 72 | |
---|
| 73 | Each of these jobs may be run individually or- using the scripts provided- |
---|
| 74 | as a simulation of user activity sized to run in approximately 4 hours on a |
---|
| 75 | 480-500 node cluster using Hadoop 0.15.0. The benchmark runs a mix of small, |
---|
| 76 | medium, and large jobs simultaneously, submitting each at fixed intervals. |
---|
| 77 | |
---|
| 78 | Notes(1-4): Since input data are compressed, this means that each mapper |
---|
| 79 | outputs a lot more bytes than it reads in, typically causing map output |
---|
| 80 | spills. |
---|
| 81 | |
---|
| 82 | |
---|
| 83 | |
---|
| 84 | * 1 Getting Started |
---|
| 85 | |
---|
| 86 | 1.0 Build |
---|
| 87 | |
---|
| 88 | 1) Compile the examples, including the C++ sources: |
---|
| 89 | > ant -Dcompile.c++=yes examples |
---|
| 90 | 2) Copy the pipe sort example to a location in the default filesystem |
---|
| 91 | (usually HDFS, default /gridmix/programs) |
---|
| 92 | > $HADOOP_HOME/hadoop dfs -mkdir $GRID_MIX_PROG |
---|
| 93 | > $HADOOP_HOME/hadoop dfs -put build/c++-examples/$PLATFORM_STR/bin/pipes-sort $GRID_MIX_PROG |
---|
| 94 | |
---|
| 95 | 1.1 Configure |
---|
| 96 | |
---|
| 97 | One must modify hadoop-env to supply the following information: |
---|
| 98 | |
---|
| 99 | HADOOP_HOME The hadoop install location |
---|
| 100 | GRID_MIX_HOME The location of these scripts |
---|
| 101 | APP_JAR The location of the hadoop example |
---|
| 102 | GRID_MIX_DATA The location of the datsets for these benchmarks |
---|
| 103 | GRID_MIX_PROG The location of the pipe-sort example |
---|
| 104 | |
---|
| 105 | Reasonable defaults are provided for all but HADOOP_HOME. The datasets used |
---|
| 106 | by each of the respective benchmarks are recorded in the Input::hadoop-env |
---|
| 107 | comment in section 0 and their location may be changed in hadoop-env. Note |
---|
| 108 | that each job expects particular input data and the parameters given to it |
---|
| 109 | must be changed in each script if a different InputFormat, keytype, or |
---|
| 110 | valuetype is desired. |
---|
| 111 | |
---|
| 112 | Note that NUM_OF_REDUCERS_FOR_*_JOB properties should be sized to the |
---|
| 113 | cluster on which the benchmarks will be run. The default assumes a large |
---|
| 114 | (450-500 node) cluster. |
---|
| 115 | |
---|
| 116 | 1.2 Generate test data |
---|
| 117 | |
---|
| 118 | Test data is generated using the generateData.sh script. While one may |
---|
| 119 | modify the structure and size of the data generated here, note that many of |
---|
| 120 | the scripts- particularly for medium and small sized jobs- rely not only on |
---|
| 121 | specific InputFormats and key/value types, but also on a particular |
---|
| 122 | structure to the input data. Changing these values will likely be necessary |
---|
| 123 | to run on small and medium-sized clusters, but any modifications must be |
---|
| 124 | informed by an explicit familiarity with the underlying scripts. |
---|
| 125 | |
---|
| 126 | It is sufficient to run the script without modification, though it may |
---|
| 127 | require up to 4TB of free space in the default filesystem. Changing the size |
---|
| 128 | of the input data (COMPRESSED_DATA_BYTES, UNCOMPRESSED_DATA_BYTES, |
---|
| 129 | INDIRECT_DATA_BYTES) is safe. A 4x compression ratio for generated, block |
---|
| 130 | compressed data is typical. |
---|
| 131 | |
---|
| 132 | * 2 Running |
---|
| 133 | |
---|
| 134 | 2.0 General |
---|
| 135 | |
---|
| 136 | The submissionScripts directory contains the high-level scripts submitting |
---|
| 137 | sized jobs for the gridmix benchmark. Each submits $NUM_OF_*_JOBS_PER_CLASS |
---|
| 138 | instances as specified in the gridmix-env script, where an instance is an |
---|
| 139 | invocation of a script as in $JOBTYPE/$JOBTYPE.$CLASS (e.g. |
---|
| 140 | javasort/text-sort.large). Each instance may submit one or more map/reduce |
---|
| 141 | jobs. |
---|
| 142 | |
---|
| 143 | There is a backoff script, submissionScripts/sleep_if_too_busy that can be |
---|
| 144 | modified to define throttling criteria. By default, it simply counts running |
---|
| 145 | java processes. |
---|
| 146 | |
---|
| 147 | 2.1 Non-Hod cluster |
---|
| 148 | |
---|
| 149 | The submissionScripts/allToSameCluster script will invoke each of the other |
---|
| 150 | submission scripts for the gridmix benchmark. Depending on how your cluster |
---|
| 151 | manages job submission, these scripts may require modification. The details |
---|
| 152 | are very context-dependent. |
---|
| 153 | |
---|
| 154 | 2.2 Hod |
---|
| 155 | |
---|
| 156 | Note that there are options in hadoop-env that control jobs sumitted thruogh |
---|
| 157 | Hod. One may specify the location of a config (HOD_CONFIG), the number of |
---|
| 158 | nodes to allocate for classes of jobs, and any additional options one wants |
---|
| 159 | to apply. The default includes an example for supplying a Hadoop tarball for |
---|
| 160 | testing platform changes (see Hod documentation). |
---|
| 161 | |
---|
| 162 | 2.2.0 Static Cluster |
---|
| 163 | |
---|
| 164 | > hod --hod.script=submissionScripts/allToSameCluster -m 500 |
---|
| 165 | |
---|
| 166 | 2.2.1 Hod-allocated cluster |
---|
| 167 | |
---|
| 168 | > ./submissionScripts/allThroughHod |
---|