Welcome!

SDN Journal Authors: Elizabeth White, Yeshim Deniz, Liz McMillan, Pat Romanski, TJ Randall

Related Topics: @DXWorldExpo, Open Source Cloud, @CloudExpo, Apache

@DXWorldExpo: Blog Feed Post

The Future of Data Management According to Doug Cutting (Father of Hadoop)

The pedigree of Hadoop came from Google’s GFS (Google File System, now HDFS) and Map-Reduce programming

I joined 600 people last night at a session sponsored by Hive to listen to Doug Cutting, the creator of Hadoop. Currently he is the chief architect at Cloudera and a director at Apache  Software Foundation. The hall at NetApp facility was overflowing with an eager audience. Doug spoke about the future of data management.

He narrated a brief history of Hadoop, how it was founded and how far it has come. As everyone knows, the pedigree of Hadoop came from Google’s GFS (Google File System, now HDFS) and Map-Reduce programming. Here are the key predictions he made:

  • Hadoop has grown to become the de-facto standard for Big Data. He had anticipated IBM and Microsoft to come up with alternative designs to compete with Hadoop, but that never happened. Both companies plus Oracle, HP and other players have endorsed Hadoop as the platform.
  • Hadoop will become the center of data management in future. It will not be the original HDFS+MR layers, but a whole new ecosystem called “The Enterprise Data Hub”. There will be an explosion of products surrounding Hadoop (all open systems). He cited examples of Pig, Hive, Sqoop, etc. Currently many SQL implementations over HDFS are coming up.
  • Will there be OLTP (Transactional systems) on Hadoop? He said yes. Current implementation of Impala (from Cloudera) has SQL on HDFS with Map-Reduce on top is proving quite efficient in ETL workloads. Several customers have started migrating from legacy world to Impala.
  • The new project at Google called Spanner is also leading the way to a future OLTP system distributed across the globe. This work will propel future additions to the Hadoop ecosystem.
  • He explained the big advantage of Open systems architecture and why that will become the norm over proprietary systems.
  • The future Hadoop ecosystem (Enterprise Data Hub) will be a threat to the current incumbents like Oracle, MySQL, SQL server, DB2, and Vertica. Current challenges of weak security and lack of standardization will be addressed eventually.

Doug is an engaging speaker and clearly showed he knows his subject well. I have my doubts on his future predictions, as DBMS’s take a long time to mature and provide all the critical functions for mission-critical applications. We have learnt that over the last 4 decades. Hadoop is still primarily a batch system doing offline analytics. Moving from there to do real-time production workload is quite a jump and will take many years to accomplish.

Then there are the new breed of highly efficient NoSQL databases like MongoDB that are being deployed to create “systems of engagement” at large enterprises. Also, the incumbents are not sitting idle either with a total market size of $30 Billion dollars. It is funny to remember that our tax records are still managed by Model 204 at IRS, a DBMS created during the 1960s. Switching databases is extremely cumbersome and not for the faint-hearted. Doug did say that future spending will steer more towards Hadoop.

Given the challenges of Big Data and the rapid adoption of Hadoop, we will watch this space as it unfolds over next couple of years.

Read the original blog entry...

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

CloudEXPO Stories
DXWorldEXPO LLC announced today that Nutanix has been named "Platinum Sponsor" of CloudEXPO | DevOpsSUMMIT | DXWorldEXPO New York, which will take place November 12-13, 2018 in New York City. Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. The Nutanix Enterprise Cloud Platform blends web-scale engineering and consumer-grade design to natively converge server, storage, virtualization and networking into a resilient, software-defined solution with rich machine intelligence.
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app security and encryption-related solutions. She holds a B.S. in Information and Computing Science from the University of Wisconsin at Green Bay, and an M.S. in Computer Science from Nova Southeastern University, and is an O'Reilly author.
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed by some of the world's largest financial institutions. The company develops and applies innovative machine-learning technologies to big data to predict financial, economic, and world events. The team is a group of passionate technologists, mathematicians, data scientists and programmers in Silicon Valley with over 100 patents to their names. Big Data Federation was incorporated in 2015 and is ...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like "How is my application doing" but no idea how to get a proper answer.
Having been in the web hosting industry since 2002, dhosting has gained a great deal of experience while working on a wide range of projects. This experience has enabled the company to develop our amazing new product, which they are now excited to present! Among dHosting's greatest achievements, they can include the development of their own hosting panel, the building of their fully redundant server system, and the creation of dhHosting's unique product, Dynamic Edge.