The Problem
A large financial services organization, serving a global network of high-net-worth clients, faced challenges in efficiently delivering data products across the enterprise. The existing setup hindered rapid data product delivery and limited the ability to effectively scale data initiatives across the business. The organization needed a scalable solution that would:
- Centralize data from diverse sources while minimizing the effort required for data engineering
- Standardize data into a unified layer, while fueling innovation in a sandbox environment
- Empower various teams, such as data scientists and BI dashboard developers, to independently prepare and model data without interfering with each other's workflows
How Kenway Helped
Kenway implemented a Delta Lakehouse data platform on Azure Synapse, utilizing a medallion architecture (Bronze, Silver, and Gold layers). This platform enabled the organization to centralize, standardize, and govern data, while keeping data engineering complexity low and facilitating data access for multiple teams.
- Metadata-Driven Ingestion - We developed a metadata-driven ingestion framework processes using data contracts for different source types (flat files, databases, and application events). This minimized the complexity of onboarding new data sources, which now only required configuration updates to integrate additional data rather than code changes.
- Data Exploration Layer - A Silver Data Lake layer, accessible via serverless cloud technologies, enabled easy data exploration through familiar tools like SQL Server Management Studio and Power BI. This ensured teams could quickly analyze data without needing highly-specialized skills.
- Project Centric Databases - Serverless databases were provisioned for specific projects, on top of the Silver Data Lake Layer. Teams could tailor and use data as needed while ensuring governance through Source Control backed data contracts. These data contracts enabled change control with familiar practices like branching, versioning, and code reviews.
Results
This solution significantly reduced the effort needed to onboard new data sources and simplified data engineering. Teams across the organization could independently access and model data, speeding up data product delivery without disrupting each other's workflows. Governance and control over shared data assets were enhanced through established version control processes, ensuring consistency and compliance across the data lifecycle.
