This report provides analysis and guidance about the benefits of adopting a real-time analytics strategy, and the essential components to execute on such a strategy.
Executive Summary This report provides analysis and guidance about the benefits of adopting a real-time analytics strategy, and the essential components to execute on such a strategy. The main conclusions and key takeaways are as follows:
- If a company has deployed a Big Data and analytics system, accessing, managing, and distributing analytic insights to the organization, that is laudable. It is also not enough. The game has (already) changed. Companies must deliver analytic insights when needed; and the answer to the question “when?” is becoming: “in real time.” - Seventy-five percent of the respondents to Stratecast’s 2014 Big Data and Analytics Survey either have already deployed a real-time analytics solution ( %), or are planning to do so ( %).1 Many respondents experienced smooth deployments and are obtaining positive ROI. A sizable group of respondents, however, are disappointed in the results thus far from their real-time analytics deployments. This group clearly feels it was poorly served by the vendors it worked with on its deployments: respondents cited poor training and a lack of maturity on the part of both the product and the vendor as factors in their disappointment. - Clear definitions and a blueprint for a successful real-time analytics deployment are needed—and this report offers both. Stratecast’s definition of real-time analytics centers on providing users with immediate insights before placing data into storage. The ingredients to achieve this include stream processing of data; in-memory computing; Big Data-supporting infrastructure, including shared-nothing processing and fault tolerance; and a number of ongoing open source initiatives at the Apache Software Foundation. - Real-time analytics can benefit any organization in any sector. It is also essential to artificial intelligence (AI). Empowering machines to learn and to exhibit other human-like behaviors means enabling a multitude of intricate, ultra-high-speed maneuvers; and real-time analytics is needed to provide data feeds every robotic step of the way. - Deploying real-time analytics adds cost and complexity to existing IT processes, and can overwhelm an organization with a tidal wave of new data. Yet, companies must adapt to the new real-time paradigm. Stratecast believes that sometime in 2015, real-time analytics will become a standard requirement of all data management systems.
Table Of Contents
Real-time Analytics: NOW Would Be Good Table of Contents Executive Summary .. 5 Introduction ... 6 Defining Real-time Analytics ..... 7 Three-fourths of Survey Respondents Use or Plan to Use Real-time Analytics ... 7 Many Reported Smooth Deployment, Positive ROIâbut Sizable Group was Dissatisfied ..... 8 A Blueprint for Real-time Analytics ..... 10 Stream Processing Overcomes the Limitations of Batch 11 In-Memory Processing Overcomes the Limitations of Secondary Storage . 11 Open Source Software Speeds Solutions to Market and Insights to Users . 12 Apache Hadoop Manages All Data Types 14 Apache YARN Powers Hadoop Cluster Management ... 15 Apache Spark Integrates with YARN and Other ASF Components to Manage Big Data ... 15 Spark Streaming Lets Programmers Write Streaming Data Processing Applications as Easily as Batch Jobs 16 Apache Storm Does for Real-time Processing what Hadoop Does for Batch Processing ... 16 Apache Drill Processes Data from a Wide Range of Sources in Seconds . 16 Apache Shark Enables SQL-like Queries Against Big Data. 16 Optimized Data Handling Architecture Accelerates the Effect. 17 Real-time Analytics Can Deliver Big-time Usability and Business Value .... 17 Real-time Analytics Can Accelerate Artificial Intelligence ..... 19 The Case for NOT Implementing Real-time Analytics ... 20 The Last Word ... 22
List of Figures
Exhibit 1: 37% of Stratecast Big Data Survey Respondents Use Real-time Analytics . 8 Exhibit 2: Most Mentions Indicated Real-time Analytics Projects Went Smoothly .... 9 Exhibit 3: 38% of Stratecast Big Data Survey Respondents Plan to Use Real-time Analytics . 9 Exhibit 4: Most Mentions: Expect to Implement Real-time Analytics in 1-2 Years .. 10 Exhibit 5: In-Memory Computing and Stream Processing Enable Real-time Analytics . 12 Exhibit 6: The ASF's Hadoop Ecosystem for Data Governance and Operations 13 Exhibit 7: The ASF's Hadoop Ecosystem for Data Processing Execution Engines . 14 Exhibit 8: Apache Spark Includes Other Useful Tools to Manage Big Data in Real Time 15 Exhibit 9: Apache Shark Queries an EDW More Than 80x Faster than Hive Itself .. 17