Welcome!

SDN Journal Authors: Yeshim Deniz, Liz McMillan, Elizabeth White, Pat Romanski, TJ Randall

Related Topics: @DXWorldExpo, Microservices Expo, SDN Journal

@DXWorldExpo: Blog Feed Post

Fast Data vs. Big Data By @JnanDash | @BigDataExpo [#BigData]

The term “Fast Data” is applied to data that is in motion. This component is getting more and more significant

Fast Data vs. Big Data – How to Combine?

Today, all the discussion on Big Data centers around “static data” in a data lake (old Data Warehouse) accessed by BI tools or SQL on Hadoop (Hawk, Impala) or Map/Reduce algorithms (MapR) for analysis. This is looking at historical data and finding trends. Some new tools are trying to provide predictive analysis based on past trends. This area deals with mostly the volume and variety aspect of Big Data, but not the velocity or for “data in motion.”

The term “Fast Data” is applied to data that is in motion. This component is getting more and more significant as there is a constant streaming of data coming from edge devices such as sensors, smart phones and connected devices. As these devices explode (10 Billion now going towards 50B in a few years, according to market analysis), there will be a data explosion and that is not going to be addressed by current Big Data products and tools. What is needed is capture of this data at ingestion points, efficient storage plus management and doing real time analytics for faster decisions. Streaming data has been around for a while, but we are talking about two-way sensors where constant feedback and aggregation is needed. For example, smart meters in the utilities industry can provide readings from individual homes, but aggregating it at the transformer level is important to predict seasonality and other trends. With fast data, things that were not possible before become achievable: instant decisions can be made on realtime data to drive sales, connect with customers, inform business processes, and create value.

Fast data is the payoff for big data. While much can be accomplished by mining data to derive insights that enable a business to grow and change, looking into the past provides only hints about the future. Simply collecting vast amounts of data for exploration and analysis will not prepare a business to act in real time, as data flows into the organization from millions of endpoints. The IoT (Internet of Things) makes this much more significant.

Enterprises have to figure out a combined architecture for Fast Data as well as Big Data. Streams of data from edge devices will eventually migrate to the data lake, but much realtime analysis have to happen before. Technologies such as in-memory databases and complex event processing are needed to handle the performance. This space is still new and much more work is needed in the area of analytics that is real-time. The older OLTP systems will be inadequate to handle the demands of ingestion and analysis at an affordable cost.

It is time to look at the world of data with a wider lens than just Hadoop-centric Big Data!

Read the original blog entry...

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

CloudEXPO Stories
While a hybrid cloud can ease that transition, designing and deploy that hybrid cloud still offers challenges for organizations concerned about lack of available cloud skillsets within their organization. Managed service providers offer a unique opportunity to fill those gaps and get organizations of all sizes on a hybrid cloud that meets their comfort level, while delivering enhanced benefits for cost, efficiency, agility, mobility, and elasticity.
Isomorphic Software is the global leader in high-end, web-based business applications. We develop, market, and support the SmartClient & Smart GWT HTML5/Ajax platform, combining the productivity and performance of traditional desktop software with the simplicity and reach of the open web. With staff in 10 timezones, Isomorphic provides a global network of services related to our technology, with offerings ranging from turnkey application development to SLA-backed enterprise support. Leading global enterprises use Isomorphic technology to reduce costs and improve productivity, developing & deploying sophisticated business applications with unprecedented ease and simplicity.
DevOps has long focused on reinventing the SDLC (e.g. with CI/CD, ARA, pipeline automation etc.), while reinvention of IT Ops has lagged. However, new approaches like Site Reliability Engineering, Observability, Containerization, Operations Analytics, and ML/AI are driving a resurgence of IT Ops. In this session our expert panel will focus on how these new ideas are [putting the Ops back in DevOps orbringing modern IT Ops to DevOps].
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understanding as the environment changes.
Enterprises are striving to become digital businesses for differentiated innovation and customer-centricity. Traditionally, they focused on digitizing processes and paper workflow. To be a disruptor and compete against new players, they need to gain insight into business data and innovate at scale. Cloud and cognitive technologies can help them leverage hidden data in SAP/ERP systems to fuel their businesses to accelerate digital transformation success.