Processing Big Data Project

Project Description

Investigates sequential and parallel techniques to help reduce the processing time for your new analysis techniques. Data from the parallel processing techniques can output a graph which can be used to extrapolate the results to give an estimate on how many processors will be required to stay under the time limit of 2 hours, the results from running the project will be used in an equation to calculate the number of processors required. the results would also be used to calculate the amount of ram will be required to hold all the memory. the code for the project is stored in separate files and in these separate files they are functions rather than scripts. the purpose of having the code split into functions is that they can be reused in different area of the project, for example the code to test the data will need to test the parallel processing section of the project as well as sequential section, and if the testing code was stored as a script, the code would need to be repeated but as it is a function it is therefore reusable, so in the parallel processing and sequential code they could simply call the function to test the code. another benefit of storing the code into separate functions and files is that they can be easily changed and adapted. since the code for any function is stored in a single location, changes made to the function would only need to be made in the function.