Cloud Data Warehouses and Data Warehouse Modernization

I can still remember my first time working with a data warehouse. The year was 2013, and I was working on my first post-school data engagement with an IT software organization. We were going to build a sales-centric enterprise data warehouse that would extract, cleanse and integrate a variety of data into a single, large repository, transforming the way the business managed their sales lifecycle. This enabled their team to leverage data in all facets of the sales lifecycle and optimize their ability to close deals.

I was fascinated by this concept, and its value proposition was clear and powerful. I was hooked at that point and have subsequently spent the majority of my career working on data engineering, data warehousing, and business intelligence solutions.

Shortly after this initial engagement, I began learning about cloud data warehouses. Today, organizations use tools like Azure Synapse and Snowflake to manage massive volumes of data every day. But it took a while to get to this point. Here’s a look at how data warehousing solutions have evolved, and what to consider as you modernize your approach to data storage.

Why Data Warehouse Modernization Is So Important

For many years, regardless of the industry, company size, or the BI platform, data warehouse structure was essentially the same. At the core, there would be a separate relational database to house the data, typically leveraging dimensional design schemas. A nightly data integration process would be developed to extract data from the line of business applications to load the data. These two components would make up the backend of the data warehouse and take the most time and effort to implement.

On the front end, there would be any number of business intelligence tools to give users direct access to slice and dice the data. This solution supported the operational and management reporting with respect to “what happened” types of business questions.

This was the typical data warehouse for many years, and it has served us well. However, new trends are causing it to break in several different ways, including but not limited to data growth, fast query expectations from users, non-relational/unstructured data and cloud-sourced data. Organizations are unable to meet the growing need to integrate and analyze a wide variety of data being generated from social, mobile and sensor data. Seventy-seven percent say that data intelligence is a major challenge. More importantly, these data warehouses struggle to answer the forward looking predictive questions necessary to run the business at the required levels of granularity or in a timely manner.

However, modern solutions, like cloud data warehouses can be designed to handle these new trends.

The Modern Data Warehouse Structure: What to Consider

Data warehouse modernization can have a different meaning depending on the organization’s level of Business Intelligence (BI) maturity. Modernization is relative to the organization’s current capabilities and needs. Some organizations today are still struggling with basic reporting and often export data into Excel to organize, filter and analyze their data. Because Excel offers benefits in reporting, some organizations often fail to see the value of investing in BI. Others have very mature data warehouse capabilities with multiple data platforms, advanced reporting tools and sophisticated power users.

From Kenway’s experience, many organizations are expected to upgrade their data warehouses and some of their analytical tools over the next several years. This may require a multi-platform environment to handle both the traditional data warehouse reporting needs and to handle big data analytics. It also may require a transition to a cloud data warehouse solution.

When thinking about modernizing your existing data warehouse, start by evaluating your existing reporting capabilities and revisit the original business drivers and assumptions. Start by asking the following questions to determine if you have a need to modernize your data warehouse:

Cloud Data Warehouses and Other Modern Storage Solutions

Cloud data warehouses are now widely touted as the future of data warehousing. They enable organizations to keep up with ever-expanding amounts of data. Data professionals say that data volumes grow by 63% every month at their companies. Many organizations are already short on IT talent, and managing on-premise solutions becomes unwieldy when data volumes are growing that rapidly. With a cloud data warehouse, you can rely on a third-party to maintain the hardware and system updates for your database needs and allocate IT resources to other, business-critical tasks.

Along with a cloud data warehouse, there are other new tools, techniques and data platforms available today that can be used to achieve data warehouse modernization:

Modernize Your Database

In conclusion, traditional data warehouses were never designed to handle the volume, variety and velocity of today’s data centric applications. Therefore, many organizations will need a more modern data warehouse platform to address many emerging business and technology requirements.

Are you interested in learning more about how Kenway can help you modernize your organization’s data warehouse? Kenway’s experts can help. Connect with us today at [email protected] for a consultation.

Cloud Data Warehouse FAQs

What is a cloud data warehouse?

A cloud data warehouse is a cloud platform acting as a centralized data store and serving data for analytical use-cases. Cloud data warehouses sit adjacent to a broad toolbox of public cloud data services and enables integration and use of these services to deliver applied data use-cases.

What is the difference between a cloud warehouse and a data warehouse?

Whereas traditional data warehouses require organizations to deploy and maintain on-premise hardware and software, cloud warehouses don’t require any physical hardware.

How does cloud data warehousing work?

With cloud data warehouses, third-party vendors manage all hardware and software updates. Data is stored in the cloud, and can be accessed from anywhere. When an organization needs to increase its storage capacity, it can simply upgrade its account with the vendor — there’s no need to add more on-premise hardware.

Is AWS a data warehouse?

AWS provides a wide variety of managed services, including data warehousing solutions.

 

Uncovering Measurable Value from Business Intelligence with Qlik Sense

When it comes to analytics projects, it’s sometimes difficult to quantify the return on investment (ROI) at the end of the engagement. Much of the value is derived from additional opportunities discovered, time saved and pain points highlighted. While these do have measurable values (potential revenue, hours, and issues, respectively), what is generally uncovered beyond the qualitative value is quantitative ROI that can only be recognized by taking further actions. Utilizing Qlik Sense, a data visualization and analytics tool, Kenway was able to go the extra mile for our client to deliver accurate and measurable value from their data.

Before we get into what we did, here is some background on Qlik Sense. Qlik Sense is a data visualization tool by Qlik. Qlik Sense builds upon the successes of Qlik’s initial product, QlikView, to combine a powerful data processing engine with a user-friendly user interface. Qlik Sense has powerful features including:

Now, back to our engagement—one of Kenway’s clients expressed concerns about the prices at which suppliers were selling them products. Our client believed that they were being sold products at prices above the negotiated rate. Businesses often find themselves in this situation, that is, they are aware that a problem exists, but they don’t have a way to confirm their suspicions confidently. In order to prove that our client was being charged incorrect rates, the client would need to combine data from multiple data sources, apply business rules to the data, and display calculations based on that data in a manner that allowed them to highlight deviations from their expectations. This is where Kenway utilized Qlik Sense to help. Using the tool’s aforementioned features, our team:

Kenway was able to build the analytics framework within Qlik Sense, foregoing a large ETL / data modeling effort for short, iterative analyses. The delivered application revealed that between fifteen and twenty thousand dollars were lost each week due to incorrect cost structures being applied. It also enabled the client to see exactly which POs this revenue could be recovered from. After training the client to refresh and explore the application, Kenway handed off the tool, allowing the client to pursue this otherwise lost opportunity moving forward. In terms of measurable ROI, the client invested about $30,000 in Kenway’s services and has the ability to recover about $18,000 per week in revenue. This enables a 100% ROI in less than two weeks of using this tool. After those two weeks, any additional recovered revenue is net gain! And it will be substantial!

Want to learn more about Qlik Sense to see if you can find your measurable value? Kenway is providing a Qlik Sense training workshop at Northwestern University’s Escape Big Data Analytics Conference on May 7th. The event is being held at Robert Morris University’s downtown Chicago campus. In our session, Data Visualization: Making Sense with Qlik Sense, we will provide a hands-on workshop that will allow you to leave understanding how to load data, build visualizations, and gain insights into your organization. To learn more about the conference, visit the conference’s website.

Download the full presentation here.

We’re also happy to field any questions! E-mail us at [email protected]–we’re always happy to help!

 

 

Big Data and the Canary in the Coal Mine

I have never been a big follower of fads. While I’m from Indiana, I might as well be from Missouri. True to the “Show Me State,” you have to show me the evidence before I’m going to believe it. This approach has saved me from following many of the greatest passing fads: Crocs, the Atkins Diet, myspace, Flash Mobs, Beanie Babies, the Macarena, and Koosh Balls. Therefore, I hope you appreciate my perspective and evaluation criteria when I tell you this: Big Data is not a passing fad.

The biggest challenge organizations face with regards to Big Data is that it is not well understood. When most people hear of Big Data, they think of Facebook, Twitter, and Yelp. They think of mining unstructured social media data in order to gauge customer sentiment. While these things are an aspect of Big Data, using a maritime metaphor, these would only be the minor swells that precede the tsunami. Big Data is evolving beyond these modest roots. The concept of Big Data goes beyond the use of unstructured data. It also refers to the rapid growth of data and how that data is then used to improve processes, create new opportunities, manage issues, and mitigate risks. Big Data is not new. Prior iterations of the evolution of data included: mainframe computing, client server applications, and the Internet. Each of these exponentially increased the amount of data in existence along with the methods in which it could be captured, stored, and utilized.

“Big Data is like teen sex. Everybody is talking about it, everyone thinks everyone else is doing it, so everyone claims they are doing it.” - Dan Ariely, Center for Advanced Hindsight at Duke University

The majority of the data that exists today was manually produced by humans. News stories, research papers, blogs, Facebook posts, and tweets were all manually written by a human being. Even the majority of the supposedly automated transactional data that exists today had heavy human involvement. A significant proportion of the data was entered manually. Point of sale data captured at a cash register or stock trades entered through a brokerage website are good examples. However, in the very near future the majority of data in existence will not be created by humans. This evolution will be highly disruptive, and the exponential growth of data will be like nothing we’ve ever seen before due to the adaptation of some new technologies. This is the coming Big Data tsunami.

The Internet of Things. If you haven’t heard this term before, I strongly suggest you head over to Google and do some reading. To summarize, the Internet of Things is the unique identification and connectivity of all the objects in the world. Technologies such as radio-frequency identification (RFID), near field communication, and advanced sensors have enabled what was first discussed as science fiction in the early 1990s to become reality. One of the simpler examples of the Internet of Things is smart warehouses. Smart warehouses contain products tagged with RFID tags. The inventory and location of all products is always known because of sensors that locate the objects by RFID tag. Additional products can be automatically ordered to replenish inventory, all without any human involvement. This innovation has led to discussions around smart homes where refrigerators could automatically build grocery lists and tell you what recipes you could cook for dinner based on having all the necessary ingredients. The ability to create the data and exchange it between tens of billions of new devices spanning the Internet, all without direct human involvement, is the tsunami. Don’t believe in Big Data yet? Consider General Electric’s new jet engines that analyze 5,000 data points every second. A single six-hour transcontinental flight will produce 240 terabytes of data that detail every aspect of the jet engine’s performance and health. Then consider that there are approximately 20,000 commercial aircrafts in operation today (CIO Magazine, June 13, 2013). Wake me up when you’ve completed your analysis on all flights in 2013.

The arrival of these initial swells to shore is the canary-in-the-coal-mine moment for your business, they are early warning signs that something big is on the way. While preparing your business for Big Data doesn’t mean that you need to start mining Facebook and Twitter data this moment, it does mean that you need to start preparing your organization by considering the following:

So, how is your canary doing? For an assessment of your company’s data strategy, contact us at [email protected].

 

Thinking About Big Data? Start by Thinking Small

Big Data, Predictive Analytics, Machine Learning Technology, the Internet of Things, Data Science, Unstructured Data, Regression Models, Hadoop Clusters, Data Lakes…

Have I successfully overwhelmed you?

The topic of Big Data can be intimidating. At times, Big Data Platforms can seem like magic boxes, environments that you feed all the data on which you can get your hands and in return, they will explain every aspect of your business. Other times, Big Data is presented as requiring resources highly skilled in applied mathematics or computer science to implement exceptionally complex algorithms. While it is true that having these tools available to you could bring a wealth of knowledge to your organization (two great examples are explored by the New York Times and Bloomberg), how many organizations have self-learning technologies and math PhDs on hand? How can your organization start harnessing the power of Big Data without making the immense investment these technologies require?

As industries have matured, competition has increased. Technological advances have lessened barriers to entry and costs of production, improved communication vehicles have allowed more companies exposure to potential clients, and, especially recently, substantial amounts of information have allowed companies to operate more optimally. In order to keep up with the changing marketplace, companies have turned to Big Data to provide more insightful information about their clients and adapting their decision making to the demographics of their customer base.

This brings us to your organization. In response to the new environment, you have begun to capture new information such as customer information (age, gender, location, etc.), but you are not sure how to utilize this information to help drive the strategic vision of your company.

One way for organizations to tap into the potential of new, diverse data sources and infrastructures capable of processing them is to leverage data to accomplish a familiar objective—to identify opportunities. Instead of looking at Big Data from a top-down approach, try going from the bottom up. Identify a target market that, based on your own business acumen and understanding of your industry, you believe is of high value. This can be traditional customer bases (mothers to be, like in the Target example by the New York Times) or more unconventional, internal groups (employees likely to leave, as in the VMware Case Study by Bloomberg). By first identifying your target group, you can begin to segment your historical data to see what the differentiating qualities are:

After you’ve identified some telling factors, you can then broaden the analysis to the entire data set:

If it looks like you have discovered a data anomaly, you are well on your way to having actionable findings! Presuming you have a large enough dataset, your initial findings could be directionally correct enough to be worth additional investment in the group. Perhaps you concentrate more of your marketing efforts for an offer in certain areas, or you structure a new subscription package for customers under 30 whose contracts are ending.

If you want to refine your target market even more, you can bring in more customer qualities and see if there is significant information gained by adding additional criteria to your data segmentation. You can get a better look at the math here, but the cornerstone of this idea is adding additional criteria (home owners under 40 vs. simply home owners) will give you an even better sense as to which groups should be targeted.

While this method will not provide the pinpoint precision that self-learning technologies or mathematicians have the potential to provide, it does provide valuable insights and has the potential to unearth substantial opportunities while minimizing barriers to entry. Perhaps more importantly, this exercise will begin to train your organization in how to better navigate and understand your data. This will allow you to start taking strides on your path to Big Data maturity, potentially leading to the more advanced practices more prevalent in popular case studies. Until then, start small, build some momentum, and start reaping the benefits of Big Data!