Implementing big data solutions

Adrian Waddy

In this chapter we look at the topic of big data. In particular, we consider the rise of the phenomenon and talk about how an organisation can identify whether a big data approach to their data assets is likely to be beneficial. We shall consider the closely related but separate consideration of cloud-based computing and the options that encounter a decision maker at the outset of a big data project. We also discuss the benefits of standardising approaches to similar types of data assets and how we might store and model them depending on differing use cases, and examine the advantages and disadvantages of a monolithic big data platform approach compared with a multi-cluster one. And finally, we shall briefly note the importance of organising and staffing for a successful big data implementation.

In a few short years big data has gone from being a fringe subject, of interest to only international scale physics experiments¹¹See https://home.cern/science/computing/storage. and Internet giants, to one that affects many businesses and organisations across nearly all industries. This has been precipitated by the rapid digitisation of data, the receding of digital storage as a