Welcome!

SDN Journal Authors: Pat Romanski, Destiny Bertucci, Liz McMillan, Elizabeth White, Amitabh Sinha

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, Open Source Cloud, @CloudExpo, Apache, SDN Journal

@DXWorldExpo: Article

The Fallacies of Big Data

No software, not even Hadoop, can make sense out of anything

The biggest problem with software is that it doesn’t do us any good at all unless our wetware is working properly – and unfortunately, the wetware which resides between our ears is limited, fallible, and insists on a good Chianti every now and then.

Improving our information technology, alas, only exacerbates this problem. Case in point: Big Data. As we’re able to collect, store, and analyze data sets of ever increasing size, our ability to understand and process the results of such analysis putters along, occasionally falling into hidden traps that we never even see coming.

I’m talking about fallacies: widely held beliefs that are nevertheless quite false. While we like to think of ourselves as creatures of logic and reason, we all fall victim to misperceptions, misjudgments, and miscalculations far more often than we care to admit, often without even realizing we’ve lost touch with reality. Such is the human condition.

Combine our natural proclivity to succumb to popular fallacies with the challenge of getting our wetware around just how big Big Data can be, and you have a recipe for disaster. But the good news is that there is hope. The best way to avoid an unseen trap in your path is to know it’s there. Fallacies are easy to avoid if you recognize them for what they are before they mislead you.

The Lottery Paradox
The first fallacy to recognize – and thus, to avoid – is the lottery paradox. The lottery paradox states that people place an inordinate emphasis on improbable events. Nobody would ever buy a lottery ticket if they based their decision to purchase on the odds of winning. As the probability of winning drops to extraordinarily low numbers (for example, the chance of winning the Powerball is less than 175,000,000 to 1), people simply lose touch with the reality of the odds.

Furthermore, it’s important to note that the chance someone will win the jackpot is relatively high, simply because so many tickets are sold. People erroneously correlate these two probabilities as though they were somehow comparable: “someone has to win, so why not me?” we all like to say, as we shell out our $2 per ticket. Assuming tens of millions of people were to read this article (I should be so lucky!) then it would be somewhat likely that some member of this impressive audience will win the lottery. But sorry to say, it won’t be you.

The same fallacy can crop up with Big Data. As the size of Big Data sets explode, the chance of finding a particular analytical result, in other words, a “nugget of wisdom,” becomes increasingly small. However, the chance of finding some interesting result is quite high. Our natural tendency to conflate these two probabilities can lead to excess investment in the expectation of a particular result. And then when we don’t get the result we’re looking for, we wonder if we’ve just wasted all the money we just sunk into all our Big Data tools.

Another way of looking at the lottery paradox goes under the name the law of truly large numbers. Essentially, this law states that if your sample size is very large, then any outrageous thing is likely to happen. And with Big Data, our sample sizes can be truly enormous. With the lottery example, we have a single outrageous event (I win the lottery!) but in a broader context, any outrageous result will occur as long as your data sets are large enough. But just because we’re dealing with Big Data doesn’t mean that outrageous events are any more likely than before.

The Fallacy of Statistical Significance
Anybody who’s ever wondered how political pollsters can draw broad conclusions of popular opinion based upon a small handful of people knows that statistical sampling can lead to plenty of monkey business. Small sampling sizes lead to large margins of uncertainty, which in turn can lead to statistically insignificant results. For example, if candidate A is leading candidate B by 2%, but the margin of error is 5%, then the 2% is insignificant – there’s a very good chance the 2% is the result of sampling error rather than reflecting the population at large. For a lead to be significant, it has to be a bit more than the margin of error. So if candidate A is leading by, say, 7%, we can be reasonably sure that lead reflects the true opinion of the population.

So far so good, but if we add Big Data to the mix, we have a different problem. Let’s say we up the sample size from a few hundred to a few million. Now our margins of error are a fraction of a percent. Candidate A may have a statistically significant lead even if it’s 50.1% vs. 49.9%. But while a 7% lead might be difficult to overcome in the few weeks leading up to an election, a 0.2% lead could easily be reversed in a single day. Our outsized sample size has led us to place too much stock in the notion of statistical significance, because it no longer relates to how we define significance in a broader sense.

The way to avoid this fallacy is to make proper use of sampling theory: even when you have immense Big Data sets, you may want to take random samples of a manageable size in order to obtain useful results. In other words, fewer data can actually be better than more data. Note that this sampling approach flies in the face of exhaustive processing algorithms like the ones that Hadoop is particularly good at, which are likely to lead you directly into the fallacy of statistical significance.

Playing with Numbers
Just as people struggle to grok astronomically small probabilities, people also struggle to get their heads around very large numbers as well. Inevitably, they end up resorting so some wacky metaphor that inevitably contains an astronomical comparison involving stacks of pancakes to the moon or some such. Such metaphors can help people understand large numbers – or they can simply confuse or mislead people about large numbers. Add Big Data to the mix and you suddenly have the power to sow misinformation far and wide.

Take, for example, the NSA. In a document released August 9th, the NSA explained that:

According to the figures published by a major tech provider, the Internet carries 1,826 Petabytes of information per day. In its foreign intelligence mission, NSA touches about 1.6% of that. However, of the 1.6% of the data, only 0.025% is actually selected for review. The net effect is that NSA analysts look at 0.00004% of the world’s traffic in conducting their mission – that’s less than one part in a million. Put another way, if a standard basketball court represented the global communications environment, NSA’s total collection would be represented by an area smaller than a dime on that basketball court.

Confused yet? Let’s pick apart what this paragraph is actually saying and you be the judge. The NSA claims to be analyzing 1.6% of 1,826 Petabytes per day, which works out to about 29 Petabytes per day, or 30,000 terabytes. (29 petabytes per day also works out to over 10 exabytes per year. Talk about Big Data!)

When they say they select 0.025% (one fortieth of a percent) of this 30,000 terabytes per day for review, what they’re saying is that their automated Big Data crunching analysis algorithms give them 7.5 terabytes of results to process manually, every day. To place this number into context, assume that those 7.5 terabytes consisted entirely of telephone call detail records, or CDRs. Now, we know that the NSA is analyzing far more than CDRs, but we can use CDRs to do a little counter-spin of our own. Since a rule of thumb is that an average CDR is 200 bytes long, 7.5 terabytes represents records of 37 quadrillion (37,000,000,000,000,000) phone calls, or about 5 million phone calls per day for each person on earth.

So, which is a more accurate way of looking at the NSA data analysis: a dime in a basketball court or 5 million phone calls per day for each man, woman, and child on the planet? The answer is that both comparisons are skewed to prove a point. You should take any such explanation of Big Data with a Big Data-sized grain of salt.

The ZapThink Take
Perhaps the most pernicious fallacy to target Big Data is the “more is better” paradox: the false assumption that if a certain quantity of data is good, then more data are necessarily better. In reality, more data can actually be a bad thing. You may be encouraging the creation of duplicate or incorrect data. The chance your data are redundant goes way up. And worst of all, you may be collecting increasing quantities of irrelevant data.

In our old, “small data” world, we were careful what data we collected in the first place, because we knew we were using tools that could only deal with so much data. So if you wanted, say, to understand the pitching stats for the Boston Red Sox, you’d start with only Red Sox data, not data from all of baseball. But now it’s all about Big Data! Let’s collect everything and anything, and let Hadoop make sense of it all!

But no software, not even Hadoop, can make sense out of anything. Only our wetware can do that. As our Big Data sets grow and our tools improve, we must never lose sight of the fact that our ability to understand what the technology tells us is a skill set we must continue to hone. Otherwise, not only are the data fooling us, but we’re actually fooling ourselves.

Image credit: _rockinfree

 

More Stories By Jason Bloomberg

Jason Bloomberg is the leading expert on architecting agility for the enterprise. As president of Intellyx, Mr. Bloomberg brings his years of thought leadership in the areas of Cloud Computing, Enterprise Architecture, and Service-Oriented Architecture to a global clientele of business executives, architects, software vendors, and Cloud service providers looking to achieve technology-enabled business agility across their organizations and for their customers. His latest book, The Agile Architecture Revolution (John Wiley & Sons, 2013), sets the stage for Mr. Bloomberg’s groundbreaking Agile Architecture vision.

Mr. Bloomberg is perhaps best known for his twelve years at ZapThink, where he created and delivered the Licensed ZapThink Architect (LZA) SOA course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, the leading SOA advisory and analysis firm, which was acquired by Dovel Technologies in 2011. He now runs the successor to the LZA program, the Bloomberg Agile Architecture Course, around the world.

Mr. Bloomberg is a frequent conference speaker and prolific writer. He has published over 500 articles, spoken at over 300 conferences, Webinars, and other events, and has been quoted in the press over 1,400 times as the leading expert on agile approaches to architecture in the enterprise.

Mr. Bloomberg’s previous book, Service Orient or Be Doomed! How Service Orientation Will Change Your Business (John Wiley & Sons, 2006, coauthored with Ron Schmelzer), is recognized as the leading business book on Service Orientation. He also co-authored the books XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996).

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting).

@CloudExpo Stories
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
Continuous Delivery makes it possible to exploit findings of cognitive psychology and neuroscience to increase the productivity and happiness of our teams. In his session at 22nd Cloud Expo | DXWorld Expo, Daniel Jones, CTO of EngineerBetter, will answer: How can we improve willpower and decrease technical debt? Is the present bias real? How can we turn it to our advantage? Can you increase a team’s effective IQ? How do DevOps & Product Teams increase empathy, and what impact does empath...
DevOps promotes continuous improvement through a culture of collaboration. But in real terms, how do you: Integrate activities across diverse teams and services? Make objective decisions with system-wide visibility? Use feedback loops to enable learning and improvement? With technology insights and real-world examples, in his general session at @DevOpsSummit, at 21st Cloud Expo, Andi Mann, Chief Technology Advocate at Splunk, explored how leading organizations use data-driven DevOps to close th...
As many know, the first generation of Cloud Management Platform (CMP) solutions were designed for managing virtual infrastructure (IaaS) and traditional applications. But that's no longer enough to satisfy evolving and complex business requirements. In his session at 21st Cloud Expo, Scott Davis, Embotics CTO, explored how next-generation CMPs ensure organizations can manage cloud-native and microservice-based application architectures, while also facilitating agile DevOps methodology. He expla...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software. They hope to capture value from emerging technologies such as IoT, SDN, and AI. Ultimately, irrespective of the vertical, it is about deriving value from independent software applications participating in an ecosystem as one comprehensive solution. In his session at @ThingsExpo, Kausik Sridhar, founder and CTO of Pulzze Systems, discussed how given the magnitude of today's application ...
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, discussed the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, application p...
Digital transformation is about embracing digital technologies into a company's culture to better connect with its customers, automate processes, create better tools, enter new markets, etc. Such a transformation requires continuous orchestration across teams and an environment based on open collaboration and daily experiments. In his session at 21st Cloud Expo, Alex Casalboni, Technical (Cloud) Evangelist at Cloud Academy, explored and discussed the most urgent unsolved challenges to achieve f...
"Digital transformation - what we knew about it in the past has been redefined. Automation is going to play such a huge role in that because the culture, the technology, and the business operations are being shifted now," stated Brian Boeggeman, VP of Alliances & Partnerships at Ayehu, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
You know you need the cloud, but you're hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You're looking at private cloud solutions based on hyperconverged infrastructure, but you're concerned with the limits inherent in those technologies. What do you do?
"I focus on what we are calling CAST Highlight, which is our SaaS application portfolio analysis tool. It is an extremely lightweight tool that can integrate with pretty much any build process right now," explained Andrew Siegmund, Application Migration Specialist for CAST, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone inn...
"We started a Master of Science in business analytics - that's the hot topic. We serve the business community around San Francisco so we educate the working professionals and this is where they all want to be," explained Judy Lee, Associate Professor and Department Chair at Golden Gate University, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Evatronix provides design services to companies that need to integrate the IoT technology in their products but they don't necessarily have the expertise, knowledge and design team to do so," explained Adam Morawiec, VP of Business Development at Evatronix, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.