Big Data Big Science

Big Data is an enormous amount of unstructured data, an amorphous behemoth, usually when people talk about big data, its petabytes and exabytes of data, which is beyond the processing capacity of traditional relational database systems, because its too huge, its fast, and the main point, it has no structure, but still, its important. Big data can reveal insights and critical acumen. It has its uses in analytics, predictive analytics, data visualization and getting real-time relevant content. For successfully acknowledging the values in the data will require a lot of scrutiny and exploration.

As we start, we know the data is towering up and its unstructured and unsymmetrical and it needs to be stored in a way, where it can be operated and scrutinize even in real time. This is where the database and data-ware house solutions like Hadoop, Storm and Greenplum come in.

Nathan Marz, Twitter Engineer and Author of Storm, states, “Storm makes it easy to write and scale complex real-time computations on a cluster of computers, doing for real-time processing what Hadoop did for batch processing. Storm guarantees that every message will be processed. And it’s fast — you can process millions of messages per second with a small cluster.” You can get into more details about the Storm at the Twitter Engineering Blog.

Hadoop which is already quite popular among engineers. The servers do not  share any memory or disk space, these are multiple servers with Hadoop installed, when the data is loaded, Hadoop slices the data and distributes it among the different servers and it even keeps the track of where the data is placed. The most well-known Hadoop user is Facebook.

Once the data is in place and operatable, then starts the procedures of curation, mining, analysis, algorithms and mathematics that needs to be performed on the data. This where the Data scientists and Statisticians come in, the ability to derive the insights from the data, that makes the data relevant. This is someone who can look into the raw potential of the data perform the science and make it pragmatic and advantageous for enterprises, organizations, entrepreneurs and professionals of various different streams and industries.

Finally, understand that big data will reveal a lot of insights, but before that you need to decide, what are you trying to innovate on ? what problem are you trying to solve ? This will make the data relevant and the insights valuable.

Image source: Aviat Networks