Blog

Implementing data governance, and keeping it relevant as your data needs change, is an exercise in vigilance. Data is ingested from so many places — one-fifth of enterprises draw data from 1,000 or more sources — and implementing governance is an important investment that impacts your entire enterprise. 

Despite attempts to streamline your data governance workflows, reactive processes late in an analytical dataflow still involve manual effort, are highly complex and costly, and often deliver less than desirable results. It becomes even more difficult to scale as your data volume grows and your data needs evolve. The growth need is usually non-linear as complexity increases. These are common challenges for many companies, as data sprawls and becomes difficult to manage. 

To keep up with the pace of data ingestion and growth, seeking out automated data governance approaches and tools makes sense. Automation is a proven tactic in multiple areas of a modern enterprise and is very applicable here. 

When applied correctly, automated elements of data governance will help you reduce the time spent on the manual tasks of managing your data as well as the complexity of your initiatives. However, it’s still important to identify and address the root causes of many data governance challenges. Here’s what to consider as you explore governance automation, and why data contracts are key to streamlining your processes. 

The Approach to Automation

Data governance isn’t just about creating policies and implementing them through technology. Automation can and should be an important part of a data governance plan. A truly comprehensive data governance strategy considers a wide range of factors inclusive of people, process, and technology:

• Based on your business needs, how do you define data quality?
• Does everyone who engages with data (whether they work in IT or other parts of the business) understand the role they play in data governance? 
• Do you have policies in place that determine how data is acquired, transformed, and connected?
• What are your procedures for data cleansing, updating systems, and deploying operational processes to adhere to policies?

Being able to answer these questions allows you to begin to address the root causes of common data governance issues. When everyone is aligned about the definition of data quality and the role they play in maintaining it, you can eliminate many of the quality issues that occur after data is created or ingested. Achieving this requires upfront engagement between IT and the business to establish the data governance strategy and implement it as early as possible within the data lifecycle.

Moving your data governance activities upstream within the data lifecycle is referred to as “shifting left on data governance”. It’s similar to making a recipe. When you’re cooking, you don’t combine ingredients, cook them, and then determine what you’ve made after you eat it. But this is how data is often approached. Organizations ingest large amounts of data and then define how they plan to use it afterwards. 

The correct approach to making a recipe is to decide what you want to make, assemble the proper ingredients, and then follow the instructions for preparation. The same approach can be applied to your data. Determine what your data products are and which elements they’re made up of. This is your recipe. With that information, you can ensure that you capture and define your data properly from the beginning. This process requires collaboration between IT and the business. Once people are in alignment, you can implement processes and technology to support your decisions.

Implementing an effective shift-left approach to data governance doesn’t just reduce the strain on your resources. It also helps companies realize the benefits of better analytics capabilities, data quality, and collaboration while reducing the risk of non-compliance. Technology can help you move governance upstream and achieve these outcomes, but it’s important to consider people and processes as well before pursuing automation. 

Data Contracts and Data Governance

There’s no tool or combination of platforms that will advocate for business rules that facilitate processes, understand the long-term hazard of compromising data decisions, or create alignment within your business. That’s why it’s important to take a key step before diving into tools.

That step is the creation of data contracts. A data contract is an agreement between a data producer and data consumer that defines the data at a high level. They’re like cheat sheets that outline what your data should look like, which allows you to proactively manage the inflow of data into various platforms. 

An example data contract may include:

• The type of data being exchanged
• The structure of that data
• How the data will be used
• Service level agreements
• Ownership details

Creating data contracts allows you to shift left on data governance. They require data stakeholders to work together to define the structure (metadata) and types of data that are ingested and how its quality is determined. This allows you to implement a metadata-driven ingestion framework. When the ingestion mechanics of your platforms rely on metadata captured in data contracts, data never lands in the data environment without being documented or understood. The contract establishes a baseline of structure and quality from the time data is ingested.

The process of developing and implementing data contracts has numerous benefits:

• Ensures all parties understand their role in data governance processes and have shared expectations for data quality and reliability. Data contracts also include some standardization of data consumers’ activities, so data governance is applied throughout the data lifecycle.
• Allows you to gain insights into dependencies and coupling between domains and applications.
• Eliminates many reactive tasks, since metadata is critical to ingestion, not an afterthought.
• Serves as living, active documentation.
• Creates repeatable processes. Once you have data contracts in place, you can use them as a scalable framework as you begin to ingest new data types.

Data Contracts Set the Foundation for Automated Data Governance

Data contracts can also be used to program automation platforms that monitor data as it’s ingested and perform automated audits on an ongoing basis to ensure data meets the terms of the contract. This approach can also help you operationalize security and compliance requirements. You can flag regulated data so that it’s handled properly as it flows through your systems. 

This allows for more efficient and seamless workflows as data moves through its lifecycle. It also frees up the data governance team for more strategic activities and initiatives rather than working on a neverending data cleanup project.

Data Contracts Are for Every Company

Who can benefit from data contracts? Everyone. Whether you have streaming data, batch data, or data that’s manually entered, governance will be a constant, uphill battle. Every business needs  a clearly defined idea of what makes for good data, and what is required to execute within the company’s rules. Breaking down silos between technology and the business is also essential for every business. Many stakeholders play a role in governance, so they all need to be engaged and working towards the same outcomes.

Data contracts don’t require you to hire new resources or learn a new programming language. If you already have in-house development capabilities, then you can create data contracts, which can live inside your existing tech stack.

Implement Automated Data Governance Effectively with Kenway

While there’s no quick and easy solution to improving data quality, there are ways to automate data governance tasks. Data contracts help you set the foundation for using automation platforms and require that you break down silos between the business and IT department. If you don’t have the in-house resources to create your own data contracts, or if you need guidance on creating your framework, Kenway can help. 

We can coordinate between the various players in the data lifecycle and guide you through the processes and technology solutions needed to help you shift left on data governance. Our data experts work across industries, including highly regulated sectors like healthcare and finance. We also can take on platform-specific projects such as Salesforce data governance. 

For example, we helped a financial services firm move their data governance processes upstream. Using YAML, which can be read by both humans and machines, we streamlined their data architecture. Instead of 100 different data integrations between 100 different files, we created one integration point with 100 different data contracts. 

Now, business users can bring in new files with minimal developer support. The developer is engaged for a matter of hours, instead of a couple of weeks. Because of the simplicity of data contracts, the client already had the right infrastructure and skills in place, and we were there to walk them through the process.

For guidance on implementing data contracts and automating data governance, reach out to our experts: info@kenwayconsulting.com.

Automated Data Governance FAQs

What is automated data governance?

Automated data governance is the process of automating some of your data policies, business rules, and procedures. It’s not a replacement for a comprehensive data governance strategy, but it can help operationalize some aspects of your plan.

What is a data contract?

A data contract is an agreement between a data producer and data consumer that defines the data at a high level. It includes key metadata, such as how a piece of data is structured, when it’s available, and who owns it.