Welcome!

SDN Journal Authors: Yeshim Deniz, Liz McMillan, Elizabeth White, Pat Romanski, TJ Randall

Related Topics: @DXWorldExpo, Microservices Expo, SDN Journal

@DXWorldExpo: Blog Feed Post

Fast Data vs. Big Data By @JnanDash | @BigDataExpo [#BigData]

The term “Fast Data” is applied to data that is in motion. This component is getting more and more significant

Fast Data vs. Big Data – How to Combine?

Today, all the discussion on Big Data centers around “static data” in a data lake (old Data Warehouse) accessed by BI tools or SQL on Hadoop (Hawk, Impala) or Map/Reduce algorithms (MapR) for analysis. This is looking at historical data and finding trends. Some new tools are trying to provide predictive analysis based on past trends. This area deals with mostly the volume and variety aspect of Big Data, but not the velocity or for “data in motion.”

The term “Fast Data” is applied to data that is in motion. This component is getting more and more significant as there is a constant streaming of data coming from edge devices such as sensors, smart phones and connected devices. As these devices explode (10 Billion now going towards 50B in a few years, according to market analysis), there will be a data explosion and that is not going to be addressed by current Big Data products and tools. What is needed is capture of this data at ingestion points, efficient storage plus management and doing real time analytics for faster decisions. Streaming data has been around for a while, but we are talking about two-way sensors where constant feedback and aggregation is needed. For example, smart meters in the utilities industry can provide readings from individual homes, but aggregating it at the transformer level is important to predict seasonality and other trends. With fast data, things that were not possible before become achievable: instant decisions can be made on realtime data to drive sales, connect with customers, inform business processes, and create value.

Fast data is the payoff for big data. While much can be accomplished by mining data to derive insights that enable a business to grow and change, looking into the past provides only hints about the future. Simply collecting vast amounts of data for exploration and analysis will not prepare a business to act in real time, as data flows into the organization from millions of endpoints. The IoT (Internet of Things) makes this much more significant.

Enterprises have to figure out a combined architecture for Fast Data as well as Big Data. Streams of data from edge devices will eventually migrate to the data lake, but much realtime analysis have to happen before. Technologies such as in-memory databases and complex event processing are needed to handle the performance. This space is still new and much more work is needed in the area of analytics that is real-time. The older OLTP systems will be inadequate to handle the demands of ingestion and analysis at an affordable cost.

It is time to look at the world of data with a wider lens than just Hadoop-centric Big Data!

Read the original blog entry...

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

CloudEXPO Stories
Steadfast specializes in flexible cloud environments, infrastructure hosting, and a full suite of reliable managed services and security. Complemented by expert consultation at all stages of design and deployment to maintenance and expansion planning, Steadfast delivers high-quality, cost-effective IT infrastructure solutions, personalized to customer needs.
Daniel Jones is CTO of EngineerBetter, helping enterprises deliver value faster. Previously he was an IT consultant, indie video games developer, head of web development in the finance sector, and an award-winning martial artist. Continuous Delivery makes it possible to exploit findings of cognitive psychology and neuroscience to increase the productivity and happiness of our teams.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science" is responsible for guiding the technology strategy within Hitachi Vantara for IoT and Analytics. Bill brings a balanced business-technology approach that focuses on business outcomes to drive data, analytics and technology decisions that underpin an organization's digital transformation strategy.
Most modern computer languages embed a lot of metadata in their application. We show how this goldmine of data from a runtime environment like production or staging can be used to increase profits. Adi conceptualized the Crosscode platform after spending over 25 years working for large enterprise companies like HP, Cisco, IBM, UHG and personally experiencing the challenges that prevent companies from quickly making changes to their technology, due to the complexity of their enterprise. An accomplished expert in Enterprise Architecture, Adi has also served as CxO advisor to numerous Fortune executives.
Eric Taylor, a former hacker, reveals what he's learned about cybersecurity. Taylor's life as a hacker began when he was just 12 years old and playing video games at home. Russian hackers are notorious for their hacking skills, but one American says he hacked a Russian cyber gang at just 15 years old. The government eventually caught up with Taylor and he pleaded guilty to posting the personal information on the internet, among other charges. Eric Taylor, who went by the nickname Cosmo the God, also posted personal information of celebrities and government officials, including Michelle Obama, former CIA director John Brennan, Kim Kardashian and Tiger Woods. Taylor recently became an advisor to cybersecurity start-up Path which helps companies make sure their websites are properly loading around the globe.