5011MODULECODE

You have been approached by a client who analysis atmospheric science and climate model data. They have developed a new analysis technique, but it takes too long to run for them to use it. They have asked you to investigate the use of big data techniques to reduce the processing time. They have a large volume of data to process, and the analysis needs to be repeated frequently. They have the following basic requirements:

Current analysis time is approximately 2.5 hours to analyse the climate model output data for a 1-hour time period.

The data for a single day of model output is approximately 250MB. However, they have over 100 years’ worth of data to analyse making a total of over 9TB.

Each day, they need to analyse the new data set for that day, so they wish to complete the analysis of the data for a 24-hour period (25 data sets) in under 2 hours.

It is not possible to hold on this in memory at one time, so the new process should load only 1 hour of data for processing at a time. If parallel processing is to occur, then 1 hour of data per worker can be loaded as needed.

You have been tasked with investigating the use of parallel processing to achieve the analysis speed required, with the following expectations: Test and compare the processing speed of sequential and parallel processing.

Extrapolate your findings to indicate the number of processors required to achieve the target processing time.

Test how your code responds to common errors, e.g. data that is text instead of numeric, use of NaN in the data as an error code.

Run automated tests that allow your client to set the tests running and return later to see the results, without user intervention. The data has been provided by the European Centre for Medium Range Weather Forecasts (ECMWF)

5011CEM-2021JANMAY/Big-Data-Programming

About

Resources

Stars

Watchers

Forks

Releases

Languages