Welcome!

SDN Journal Authors: Elizabeth White, Yeshim Deniz, Liz McMillan, Pat Romanski, TJ Randall

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, IBM Cloud, @CloudExpo, SDN Journal

@DXWorldExpo: Article

Big Data OLTP with IBM DB2 BLU and DB2 pureScale

Big Data and analytical processing

Big Data as we know it today is more aligned to the analytical processing of large quantities of data. All the predominant use cases identified by the Big Data product vendors are more aligned with analytical processing. For example one of the major use cases of Big Data is about utilizing social media data to get into advertisement targeting. Naturally this kind of processing analyzing lots of unstructured data and come up with predictions on customer preferences and this use case is aligned with analytical processing. To support these kinds of analytical processing Columnar databases have emerged as a natural extension to Big Data processing. Columnar databases only reads columns involved in the query and not the entire row, making it a perfect fit for analytical processing.

Online Transaction Processing
On the other side of the Enterprise Data Access pattern we have the most important OLTP pattern. Most of the real life events that are important for the survival of an enterprise like an online ecommerce transaction need to follow ACID property, where by the Atomicity, Consistency, Isolation and Durability of the transaction needs to be maintained. Due to this need OLTP applications and their natural allies, the relational databases, continue to be popular and an essential part of enterprise data access patterns.

OLTP + Big Data
Advent of Big Data and the associated analytics initially looked like a separate stream from OLTP , mainly because of the data integration challenges. For example the tools meant for big data processing were from the new era like Hadoop, where as invariably OLTP applications like banking were built out of legacy platforms, mainly platforms like Mainframe. Also the response time needs for OLTP applications are very critical and most of the batch oriented big data processing platforms cannot cater to them.

However for use cases like fraud detection of Online transactions, we needed to combine the ACID nature of the transactions with the analytical capabilities enabled by the Big Data Platform, so far we did not have very many unified platforms to achieve this combination.

However the traditional RDBMS platforms which all along cater to the OLTP needs have started to release native Big Data Integration features as part of their offering this has facilitated easier integration of Big Data Analytics with OLTP.

We find that recently IBM DB2 which is one of the popular relational databases catering to the OLTP needs in Mainframe as well as Unix, Linux, Windows has announced the BLU Acceleration on top of the traditional database offering. The rest of the sections covers about DB2 BLU Acceleration and how it can fit a Big Data OLTP processing need.

DB2 with BLU Acceleration
Recently IBM Announced the offering information about DB2 10.5 for Linux, Unix and Windows. There are some major components within the DB2 10.5, which caters to the Big data OLTP Needs.

  • DB2 With BLU Acceleration
  • DB2 PureScale Clustering Technologies

DB2 with BLU Acceleration is dynamic in-memory technology that yields faster analytics without the costs or limits of in-memory only systems. It speeds analytics and reporting with a combination of in-memory and column store data retrieval, maximized CPU processing, and data skipping that allows for faster input/output.

IBM DB2 pureScale® database clustering technology. Helps to ensure transaction processing is highly available and extremely scalable.

The following are the important Analytical Aspects of DB2 With BLU Acceleration.

  • Fully Integrated Solution as part of the base DB2 database which enables the combination of OLTP with Big Data Analytics
  • Utilizes the Column Organized Table Architecture so that the I/O Operations on the Analytical queries are fully optimized
  • Advanced Compression Technologies further optimize the columnar storage
  • In Memory Database storage further optimizes already compressed columnar data
  • A new concept of Data Skipping further skips the data that is not of interest resulting in further efficiencies
  • All the above optimizations work in compliment with the Parallel Processing nature of the solution

Another important aspect is that BLU Acceleration works with the familiar regular DB2 environment such that all the commands like LOAD, IMPORT and other DML commands work seamlessly with it. The regular DDL commands have been extended to support BLU Acceleration.

On the other hand, IBM DB2 PureScale is a multi instance shared served model similar to PARALLEL SYSPLEX environment in Mainframe and can be an ideal candidate for large scale OLTP processing. A DB2 pureScale environment is ideal for short transactions where there is little need to parallelize each query. Queries are automatically routed to different members, based on member workload. The ideal scenario for a DB2 pureScale environment includes workloads that handle online transaction processing (OLTP) or enterprise resource planning (ERP).

With the possibility to combine operations, especially JOINS between traditional OLTP work load row organized tables and Analytical work load column organized tables within the same DB2 environment powered by PureScale & BLU Acceleration technologies, the below architecture provides a blue print for a Big Data OLTP environment.

There is not a lot of documentation available on DB2 BLU Acceleration at this time; however, from my understanding it looks like at present DB2 BLU Scale Up Vertically within a powerful server with multiple cores rather than Scale Out using multiple horizontal servers, while this understanding may be wrong, but availability of both Scale UP and Scale Out would be bigger boost in enterprise scenarios. In other words the ability to use the DB2 BLU in conjunction with DB2 Data Partitioning Feature will further enhance the scalability of the Analytical component.

Summary
Combining the Big Data features with traditional RDBMS makes enterprises easily embrace Hybrid Big Data much like the success of Hybrid Cloud we see a increasing adoption here. The above mentioned combination of BLU Acceleration In DB2 together with traditional features like PureScale is a good example of Big Data OLTP. As I write this article I find Sql Server 2014 announcements are flashing on various sites which talks about features like In Memory OLTP which will provide further choices for enterprises.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).

CloudEXPO Stories
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to advisory roles at startups. He has worked extensively on monetization, SAAS, IoT, ecosystems, partnerships and accelerating growth in new business initiatives.
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments that frequently get lost in the hype. The panel will discuss their perspective on what they see as they key challenges and/or impediments to adoption, and how they see those issues could be resolved or mitigated.
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app security and encryption-related solutions. She holds a B.S. in Information and Computing Science from the University of Wisconsin at Green Bay, and an M.S. in Computer Science from Nova Southeastern University, and is an O'Reilly author.
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like "How is my application doing" but no idea how to get a proper answer.
Having been in the web hosting industry since 2002, dhosting has gained a great deal of experience while working on a wide range of projects. This experience has enabled the company to develop our amazing new product, which they are now excited to present! Among dHosting's greatest achievements, they can include the development of their own hosting panel, the building of their fully redundant server system, and the creation of dhHosting's unique product, Dynamic Edge.