Data Warehouse Modernization
I can still remember when I first learned about the concept ‘data warehousing.’ The year was 1995, and I was working on my first job out of school for a healthcare organization. We had a department meeting to announce an exciting new project that would transform the way we managed healthcare. We were going to build a patient-centric enterprise data warehouse that would extract, cleanse and integrate a variety of data into a single, large repository. This would give providers a more complete picture of a patient’s medical history in order to provide the best possible care. I was fascinated by this concept, and its value proposition was clear and powerful. I was hooked at that point and have spent my entire career working on data warehousing and business intelligence solutions.
For many years, regardless of the industry, company size or the BI platform, data warehouse architecture was essentially the same. At the core, there would be a separate relational database to house the data, typically leveraging dimensional design schemas. A nightly data integration process would be developed to extract data from the line of business applications to load the data. These two components would make up the backend of the data warehouse and take the most time and effort to implement. On the front end, there would be any number of analytical and reporting tools to give business users direct access to slice and dice the data. This solution supported the operational and management reporting with respect to “what happened” types of business questions.
This was the typical data warehouse for many years, and it has served us well. However, new trends are causing it to break in several different ways including but not limited to data growth, fast query expectations from users, non-relational / unstructured data and cloud-sourced data. Organizations are unable to meet the growing need to integrate and analyze a wide variety of data being generated from social, mobile and sensor data. More importantly, these data warehouses struggle to answer the forward looking predictive questions necessary to run the business at the required levels of granularity or in a timely manner.
However, modern data warehouses can be designed to handle these new trends.
Data warehouse modernization can have a different meaning depending on the organization’s level of Business Intelligence (BI) maturity. Modernization is relative to the organization’s current capabilities and needs. Some organizations today are still struggling with basic reporting and often export data into Excel to organize, filter and analyze their data. While Excel offers benefits in reporting, some organizations often fail to see the value of investing in BI. Others have very mature data warehouse capabilities with multiple data platforms, advanced reporting tools and sophisticated power users.
From Kenway’s experience, many organizations are expected to upgrade their data warehouses and some of their analytical tools over the next several years. This may require a multi-platform environment to handle both the traditional data warehouse reporting needs and to handle big data analytics.
When thinking about modernizing your existing data warehouse, start by evaluating your existing reporting capabilities and revisit the original business drivers and assumptions. Start by asking the following questions to determine if you have a need to modernize your data warehouse:
- Does your environment quickly handle diverse data sources and a variety of subject areas?
- Can we handle excessive volumes of data (social, sensor, transaction, operational and analytical?)
- Do we support a multi-platform architecture to maximize scalability and performance?
- Have we leveraged new capabilities like data virtualization (cloud services) in addition to data integration?
- Are we using structures such as data lakes, Hadoop and NoSQL databases, or are we running only relational data mart / data warehouse structures?
Below are a few of the many new tools, techniques and data platforms available today that can be used to modernize your data warehouse environment:
- In-Database Analytics and Massive Parallel Processing (MPP) can allow organizations to query very large data volumes that traditional relational databases can’t handle well.
- Adding unstructured metrics to your data warehouse allows you to leverage data such as social, mobile, and emails to identify new consumer insights.
- Leveraging the Internet of Things (IOT) by integrating sensor-generated data into your manufacturing, supply chain or operations models helps provide real-time analytics.
- Adopting a cloud platform (private or public) lets you quickly scale for big data analytics and augment the traditional on-premises platforms.
- Implementing a data lake can store vast amounts of raw data that can be used to feed the traditional data warehouse, as well as support a “Sand Box” for data exploration.
- Extending your data warehouse to allow for Data Federation allows users to access data that is not stored in your actual data warehouse environment.
In conclusion, traditional data warehouses were never designed to handle the volume, variety and velocity of today’s data centric applications. Therefore, many organizations will need a more modern data warehouse platform to address many emerging business and technology requirements.
Are you interested in learning more about how Kenway can help you modernize your organization’s data warehouse? Contact us at firstname.lastname@example.org today.