Changes between Version 14 and Version 15 of Hadoop

Jan 19, 2010, 11:20:14 AM (14 years ago)



  • Hadoop

    v14 v15  
    1414 * 3 January - Add extended error reporting: the entire stack trace
    16 === Motivation ===
     16== Motivation ==
    1717The purpose of this project is to run a large number of tests in a distributed environment in order to get faster results and to take advantage of the available distributed infrastructures available nowadays.
    1818We chose to use Hadoop, an open source framework that comes with its own distributed filesystem and a MapReduce implementation. We used the pseudo-distributed configuration, that creates clusters locally, on the host machine. As testing tool, we used JUnit, a popular testing framework.
    1919We aim to offer the end used developer the same experience when running JUnit tests on a normal host machine as when running it with Hadoop, but having the advantages of running in a distributed environment: higher speed.
    21 === Project Architecture ===
     21== Project Architecture ==
    2222[[Image(HadoopRunnerArchitecture.PNG,align=right, hspace=20, vspace=20, width=40%, height=40%)]]
    2323In order to explain the architecture of the project, lets follow a normal execution. A user adnotates a JUnit test class with "@RunWith(HadoopRunner.class)". At runtime, the JUnit will then use our own test runner. Our runner checks the test class for the descriptions of the tests we want to run, writes then into a file and send this file to the Hadoop core. In the Map phase, we read the description of the tests and then try to run the tests. If the tests requires some additional classes, the HadoopRunner will send a request to a resource server for that class. The resource server will load the class into the HDFS and the HadoopRunner will launch a classic JUnit runner to run the test on the cluster. In the Reduce phase, we gather the results of the tests. We send back a serialized version of the running result and then deserialize it and report the results.
    26 === Development Problems ===
     26== Development Problems ==
    2727During the development phase of the project we encountered several problems. We describe them bellow and also the solution we have chosen.
    29 ==== Sending tests to clusters ====
     29=== Sending tests to clusters ===
    3030We had to design a protocol for sending the tests we want to run into the clusters, running them and finally, gathering the results. Do accomplish this, we decided to get the Description of each test we want to run (Description is a JUnit class, containing information about a single test), serialize it and send it into the cluster. Also, we would put on the HDFS a jar containing the JUnit test classes. In the Map phase we would deserialize the class Description, run the test and finally, in the Reduce phase, we would serialize the result and send back to the centralized management console a pair <Description Result>.
    32 ==== Loading resources into the cluster ====
     32=== Loading resources into the cluster ===
    3333At runtime, a test may require custom user defined classes. It would be impossible for a developer to know in advanced all the classed it may need to run the tests in all situation. So we needed a way of getting the test required classes into the cluster. Hadoop has its own ClassLoader that searches classes in the Hadoop Java environment, but this wasn't enough for us. So using the Hadoop API, we could get a hold of the situations when the Hadoop class loader needed classes that weren't already loaded into the HDFS and send a request for the class to a class server (a new component) and copy the bytecode of the class on the HDFS.
    35 ==== Reporting results ====
     35=== Reporting results ===
    3636After running the tests, in the reduce phase, we needed a way of sending the results to the centralized management console. The problem was that the classic JUnit runner returner a Result class, that wasn't serializable. So we transformed this class into a byte array, encoded into a Base64 format and send the result in this format and upon receiving it, we would decode the bytes and recreate the object.