Welcome!

SDN Journal Authors: Pat Romanski, Patrick Hubbard, Elizabeth White, Sven Olav Lund, Liz McMillan

Related Topics: @CloudExpo, Microservices Expo, @BigDataExpo, SDN Journal

@CloudExpo: Blog Post

Big Compute Gives Life to New Data By @Nimbix | @CloudExpo [#BigData]

Deriving value through computation

What's Big Data without Big Compute? Basically just a large collection of unstructured information with little purpose and value.  It's not enough for data just to exist, we must derive value from it through computation - something commonly referred to as analytics.

The Quantum Nature of Big Data
With traditional data, we simply query it to derive results; all we need is currently stored within the data set itself.  For instance, for a customer database with dates of birth, we may just fetch the list of customers who were born after a certain date.  This is a simple query, not a computation, and therefore cannot be considered analytics.

However, with Big Data, we can distribute the information so that we can run complex analytics on it at scale.  Unlike traditional data, the information itself has little meaning until we process it.  The reason we distribute the data sets is not because they are large, but because we want to leverage clusters of computers to run more than just simple queries.  That is why in the Big Data model, the data itself doesn't hold the answer - to unleash it we must compute it.  Think of this as a virtual "Schrödinger's Cat"... it can mean anything until we actually look "in the box."  The difference is that we're not asking a simple question such as, "is it dead or alive," but rather more complex inquiries such as, "assuming it's alive, what might its future behavior be?"  Analytics, especially predictive ones, rely on patterns and their associations.  Because the data sets tend to change (or grow) over time, then we have to understand that the results of these complex computations will most definitely vary as well.

Knowing this, it's perhaps a major understatement to associate the term "Big Data" with data itself since it cannot really exist without Big Compute to make sense of all the information.

What's So Special About Big Compute?
Big Compute implies one of two things:

  1. Ordinary computing scaled across a massive parallel cluster
  2. High-Performance Computing (HPC)

The problem with the former is that it can only scale so far before performance drops off.  Furthermore, for it to really succeed, the data itself must be scaled just as wide.  This also brings with it practical challenges of systems management, complexity and infrastructure constraints such as networking and power.

High-Performance Computing is a more natural form of "Big Compute" because it scales and packs a powerful per-unit punch.  What does this mean? Simply that we can realize higher computation density with far fewer "moving parts."  A good example is using Graphics Processing Unit (GPUs) for vector calculations.  Sure, you can do this with Central Processing Unit (CPU) cores alone, but you've only got around 8-16 in each typical server node.  Each GPU can have hundreds or even thousands of cores.  If you vectorize your calculations to take advantage of this, you can do far more work with far less power and management complexity (at scale) than if you had to spread it across dozens or even hundreds of CPUs (and the servers they live in).

So this begs the question: Why does Big Compute really matter?  Can't simple algorithms on commodity compute already do predictive analytics?

The answer is of course yes, but there are two problems - one immediate and one future.

The immediate problem is that in many cases, the speed at which you get results matters just as much as the results themselves.  For example, if you are planning on using analytics to improve e-commerce, the best time to do this is while the customer is engaged in a transaction. Sure, there's still value in following up with the customer later, after you've crunched the data, but why not take advantage of the moment while he or she has credit card in hand to provide facts that may increase spend?

When you combine this with the fact that there may be thousands of concurrent transactions at any given time, over-subscribing commodity compute to perform complex analytics won't get you the results you need in time to maximize the value of Big Data.

This is where Big Compute can perform the same operations thousands of times faster.  In many cases, the value of the data is sensitive to the amount of time needed to compute it.  There are many examples of this - e-commerce is just a popular one.  In other cases, the data set itself is changing (generally growing) rapidly.  If analytics take too long, the results may already be obsolete or irrelevant once delivered.

Simply put, Big Compute powered by HPC is the fastest, most efficient way to derive value from data at scale - at the exact moment needed.

This brings us to the future problem with commodity compute.

Innovation in Algorithms
How do you derive future value from the same data you have (or are collecting) today?  If we look at Big Data as a two-part problem - storing the data and analyzing the data - we quickly realize where the greatest potential for innovation is.  It's not in storage because, although challenging, we've seen densities increase dramatically since the dawn of computing.  As a (crude) point of reference, a consumer could buy a three-terabyte hard disk in 2014 for less than the cost of a 200 gigabyte one just 10 years prior.  Higher storage densities mean less infrastructure to manage, and thus make storing large data sets more practical over time as well (not just cheaper.)  So we can rest easy knowing that all things being equal, as the data sets grow, so will the storage to hold it all in a relatively cost-effective way.

Obviously the most room for innovation is in the analytics algorithms themselves.  We will see both the speed and the quality of the computations increase dramatically over time.  Thanks to Big Compute, there's no need to compromise.  Commodity compute is a non-starter for algorithms that are too complex to run quickly against large data sets.

Just imagine the opportunities we'd miss if we avoided problems that are seemingly too hard to solve.  Big Compute makes it possible to run the most complex algorithms quickly, and the sky's the limit when it comes to the types of analytics we'll see as a result.

Big Compute will help Big Data evolve to not just be "bigger," but to be far more meaningful than we can ever imagine.

More Stories By Leo Reiter

Leo Reiter is CTO of Nimbix, providers of cloud-based High Performance Computing and Big Data platforms and applications to help organizations solve their most complex problems faster and easier.

@CloudExpo Stories
The dynamic nature of the cloud means that change is a constant when it comes to modern cloud-based infrastructure. Delivering modern applications to end users, therefore, is a constantly shifting challenge. Delivery automation helps IT Ops teams ensure that apps are providing an optimal end user experience over hybrid-cloud and multi-cloud environments, no matter what the current state of the infrastructure is. To employ a delivery automation strategy that reflects your business rules, making r...
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software. They hope to capture value from emerging technologies such as IoT, SDN, and AI. Ultimately, irrespective of the vertical, it is about deriving value from independent software applications participating in an ecosystem as one comprehensive solution. In his session at @ThingsExpo, Kausik Sridhar, founder and CTO of Pulzze Systems, will discuss how given the magnitude of today's applicati...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...
SYS-CON Events announced today that SkyScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SkyScale is a world-class provider of cloud-based, ultra-fast multi-GPU hardware platforms for lease to customers desiring the fastest performance available as a service anywhere in the world. SkyScale builds, configures, and manages dedicated systems strategically located in maximum-security...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant th...
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, will go over the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, applicatio...
Enterprises are adopting Kubernetes to accelerate the development and the delivery of cloud-native applications. However, sharing a Kubernetes cluster between members of the same team can be challenging. And, sharing clusters across multiple teams is even harder. Kubernetes offers several constructs to help implement segmentation and isolation. However, these primitives can be complex to understand and apply. As a result, it’s becoming common for enterprises to end up with several clusters. Thi...
Microsoft Azure Container Services can be used for container deployment in a variety of ways including support for Orchestrators like Kubernetes, Docker Swarm and Mesos. However, the abstraction for app development that support application self-healing, scaling and so on may not be at the right level. Helm and Draft makes this a lot easier. In this primarily demo-driven session at @DevOpsSummit at 21st Cloud Expo, Raghavan "Rags" Srinivas, a Cloud Solutions Architect/Evangelist at Microsoft, wi...
Containers are rapidly finding their way into enterprise data centers, but change is difficult. How do enterprises transform their architecture with technologies like containers without losing the reliable components of their current solutions? In his session at @DevOpsSummit at 21st Cloud Expo, Tony Campbell, Director, Educational Services at CoreOS, will explore the challenges organizations are facing today as they move to containers and go over how Kubernetes applications can deploy with lega...
SYS-CON Events announced today that Avere Systems, a leading provider of hybrid cloud enablement solutions, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Avere Systems was created by file systems experts determined to reinvent storage by changing the way enterprises thought about and bought storage resources. With decades of experience behind the company’s founders, Avere got its ...
SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. ANSeeN are the measurement electronics maker for X-ray and Gamma-ray and Neutron measurement equipment such as spectrometers, pulse shape analyzer, and CdTe-FPD. For more information, visit http://anseen.com/.
Today most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes significant work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reducti...
As you move to the cloud, your network should be efficient, secure, and easy to manage. An enterprise adopting a hybrid or public cloud needs systems and tools that provide: Agility: ability to deliver applications and services faster, even in complex hybrid environments Easier manageability: enable reliable connectivity with complete oversight as the data center network evolves Greater efficiency: eliminate wasted effort while reducing errors and optimize asset utilization Security: imple...
High-velocity engineering teams are applying not only continuous delivery processes, but also lessons in experimentation from established leaders like Amazon, Netflix, and Facebook. These companies have made experimentation a foundation for their release processes, allowing them to try out major feature releases and redesigns within smaller groups before making them broadly available. In his session at 21st Cloud Expo, Brian Lucas, Senior Staff Engineer at Optimizely, will discuss how by using...
In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lead...
SYS-CON Events announced today that Yuasa System will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Yuasa System is introducing a multi-purpose endurance testing system for flexible displays, OLED devices, flexible substrates, flat cables, and films in smartphones, wearables, automobiles, and healthcare.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that CAST Software will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CAST was founded more than 25 years ago to make the invisible visible. Built around the idea that even the best analytics on the market still leave blind spots for technical teams looking to deliver better software and prevent outages, CAST provides the software intelligence that matter ...