Welcome!

SDN Journal Authors: Yeshim Deniz, Liz McMillan, Elizabeth White, Pat Romanski, TJ Randall

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, Linux Containers, SDN Journal

@DXWorldExpo: Article

Big Data – How We Got Here and Is It For Everyone? | @BigDataExpo

An organization needs to be data driven in order to understand the data’s value and lament its absence

Big Data is an umbrella term for a multitude of new capabilities that are being used for storage and computing operations at scale. These capabilities allow organizations to store massive amounts of data, in disparate formats, and perform both batch and real-time analyses upon them.

The forces driving Big Data into the mainstream are the ever-decreasing cost of storage and processing, coupled with the open source enhancements of distributed systems techniques and software. Companies have realized that data storage is on the verge of being limitless, and they no longer need to be as judicious about what kinds of data they store. This realization has led to the storage of all manner of data, in addition to the traditional structured data found in relational databases.

Sometimes unstructured or semi-structured, this type of data encompasses emails, social media feeds, clickstreams, sensor data, videos and more. Further, the questions companies can ask of their data to realize value have become more complex. But the time window for analysis completion has remained the same or shrunk due to the massively parallel computation Big Data systems provide.

To organize all this limitless data - structured and unstructured - new tools have emerged. We no longer have just one hammer in our toolkit - the relational database - with which to fashion data. There are now a myriad of systems, thanks to big organizations, many of them Internet giants (Google, Yahoo!, Amazon, Facebook, and LinkedIn to name a few). Out of a need to scale storage and compute tasks confronting them, they made new data systems to satisfy these specific use cases. The most common characteristics of these systems are that they are not row based or relational databases. The systems have horizontal linear scalability (just add more nodes), and they expect components and nodes to fail so they are fault tolerant.

Most of these tools have been open sourced and, years later, are being adopted by other businesses that recognize their value. Examples of these non-relational databases are key-value pair databases (e.g., Riak), document databases (e.g., CouchDB, MongoDB), columnar databases (e.g., HBase, Cassandra), graph databases (e.g., Titan, Neo4J), distributed queues (e.g., Kafka, Kestrel) and spatial databases. There are also new computing tools such as Hadoop and Spark that allow immense amounts of data to be processed, and Storm, Samza and Spark Streaming which analyze data in near real-time, something that was previously only possible with supercomputers.

The ability to store unlimited amounts of disparate data in order to perform endless analysis in batch and real time is the allure of Big Data. But is it for everyone?

Are you ready for the Big Data bandwagon?
The promise of Big Data is very real - more data + new ways to analyze data = better business intelligence. So every business should be rushing out to see what Big Data can do for them, right? Not so fast.

Although Big Data has the capability to provide unprecedented insight into business operations, it is not for everyone. At least not yet. Big Data tools are still immature enough, even eight years after we've started down this road, that companies really have to be highly motivated to take on the task.

In my experience, this motivation typically comes from pain. For example, it is the pain of dropping valuable data on the floor because long-term storage isn't feasible within the current infrastructure. Or possibly it's the pain of not being able to monetize collected data due to technical hurdles with existing data systems. This type of pain - and the knowledge that the only way to escape it is to embark on Big Data - provides the fortitude necessary to push through on what can be quite a challenging project. In fact, it was this kind of pain, as experienced by Internet giants who needed to find a way to store, analyze and, ultimately, monetize the massive amounts of customer data to which they had access, that birthed the Big Data movement in the first place.

In addition to being highly motivated to take on Big Data, an organization must also be data driven. Organizations that treat data as currency are better equipped to tackle the challenges of Big Data, because they inherently understand its value and will nurture the initiative, no matter how unruly it becomes. Without this level of buy-in from top-level management, the obstacles of Big Data may prove to be too large to overcome.

The combination of the pain of unexploited data with a data-driven culture goes hand in hand. You typically won't find one without the other. It is when these two factors come together that organizations can then successfully tackle the challenge of Big Data. Without the right motivation or executive support, Big Data endeavors often die before they even begin. Ultimately, an organization needs to be data driven in order to understand the data's value and lament its absence.

More Stories By Brad Anderson

As vice president of Big Data informatics, Brad Anderson is responsible for Liaison's Big Data solution implementation, leveraging the company's world-class cloud infrastructure. He is a 20-year data management veteran with expertise in enterprise data warehouses and building and using non-relational Big Data tools.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


CloudEXPO Stories
The dream is universal: heuristic driven, global business operations without interruption so that nobody has to wake up at 4am to solve a problem. Building upon Nutanix Acropolis software defined storage, virtualization, and networking platform, Mark will demonstrate business lifecycle automation with freedom of choice and consumption models. Hybrid cloud applications and operations are controllable by the Nutanix Prism control plane with Calm automation, which can weave together the following: database as a service with Era, micro segmentation with Flow, event driven lifecycle operations with Epoch monitoring, and both financial and cloud governance with Beam. Combined together, the Nutanix Enterprise Cloud OS democratizes and accelerates every aspect of your business with simplicity, security, and scalability.
CloudEXPO has been the M&A capital for Cloud companies for more than a decade with memorable acquisition news stories which came out of CloudEXPO expo floor. DevOpsSUMMIT New York faculty member Greg Bledsoe shared his views on IBM's Red Hat acquisition live from NASDAQ floor. Acquisition news was announced during CloudEXPO New York which took place November 12-13, 2019 in New York City. Our Silicon Valley 2019 schedule will showcase 200 keynotes, sessions, general sessions, power panels, and hands on tutorials presented by 150 rockstar speakers in 10 hottest conference tracks of 2019:
Every organization is facing their own Digital Transformation as they attempt to stay ahead of the competition, or worse, just keep up. Each new opportunity, whether embracing machine learning, IoT, or a cloud migration, seems to bring new development, deployment, and management models. The results are more diverse and federated computing models than any time in our history.
Andrew Keys is co-founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereum.
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.