Agile Data Governance Month – A Data Governance Primer
It’s Agile Data Governance Month! During the month of September, Passerelle will release a weekly serial discussing Agile Data Governance. By looking at Data Governance as an iterative process, you can work within a common project delivery mindsight, starting small and building Data Governance into the foundation of your data management practices. Can’t wait to read the next installment? You can download the entire series as our Agile Data Governance Guide here.
Why focus on Data Governance?
Nearly 4 out of 5 businesses globally are looking to implement big data management, and for good reason. Data creation is expected to grow to 181 zettabytes in 2025 – a 150% increase over data creation in 2023.
Growth in data volume is driven by a combination of factors, including:
- Cloud Computing and Storage: As businesses and individuals migrate services to the cloud, the volume of data stored and processed grows.
- Increased Internet Usage: The International Telecommunication Union (ITU) reported that 5.3 billion people were using the Internet in 2022 – about 66% of the world’s population. This figure represents an increase of 24% since 2019. As more people get online, there will be a corresponding increase in the amount of data being created and shared.
- IoT (Internet of Things) Devices: Smart home and IoT devices are driving immense data growth – with more than 14 billion active endpoints in 2022.
- Regulatory Compliance: Compliance initiatives require the creation and retention of extensive records, such as transaction logs, customer information, and audit trails, to show adherence to legal standards and industry regulations. Documentation requirements lead to significant increases in data volume, creating both a repository of historical information and data management challenges.
With exponential data growth comes an imperative to keep data secure and ensure it can be used to fuel better marketing, forecasting and operational efficiency. For these reasons, Data Governance has never been more essential to data-intensive industries.
A Data Governance Primer
In the first article, we deliver a Data Governance primer that defines what Data Governance is (and what it is not). In Part Two, we will explore how to introduce the concept of Agile Data Governance into your organization. Part Three will identify challenger questions to help get your Agile Data Governance program off the ground. Finally, we’ll provide an overview of tools you should include in your Data Governance toolkit. If you can’t wait to read the serial, download it now as a complete reference guide.
Data Governance is typically applied across five areas of focus – Data Availability, Data Quality, Data Security, Data Usability and Data Auditability. While many useful tools help build a Data Governance program, most solutions aim to address one or two of these areas. Data Governance is not a silver bullet – Data Governance can’t be solved with one tool or through one initiative. It should be viewed as a holistic concept, and programs should be organically developed based on the needs and structure of an organization.
Data Governance Guiding Principles
At a surface level, these Guiding Principles for Data Governance might look simple – it would be hard to find anyone who would argue that data shouldn’t be trusted or accessible. Applying these principles takes intention, discipline and tools that add scalability, observability, and ultimately, sustainability to your Data Governance program.
- Data should be ingested, stored and processed to promote accessibility.
- Data can be trusted, known and measured.
- Data is relevant to business use cases.
- Data is secure, observable, auditable and compliant throughout its lifecycle.
Data should be managed to promote accessibility.
Making data available to the right user at the right time is a critical driver of digital transformation and a central tenet of Data Governance, and to do that, more and more organizations are turning to the cloud. According to a report by Gartner, most enterprise IT spending will shift to the cloud by 2025. Cloud adoption will only increase as organizations try to keep pace with data availability, cost savings, and scalability.
While cloud adoption will continue to increase, adopters of a cloud-based data management strategy have quickly come to understand that cloud utilization is not enough, especially when it comes to establishing Data Governance programs. To support data accessibility, data-minded organizations should focus on the structure of their data ecosystem, with an eye toward business use cases.
Create a governed Data Lake
A data lake is a central repository for all data – unlocking data from siloed systems and creating a landing zone for other data applications. Creating a data lake is often seen as the primary goal of a digital transformation initiative; in reality, creating a data lake is just the starting point. Data Governance should address how a data lake is administered and maintained.
For data to be usable within a data lake, care should be taken during the ingestion, integration, and transformation process. A data lake can become a swamp if improperly managed and organized. To prevent muddying the waters of your data lake, a metadata management system should be used to keep track of data. Regular data quality checks and monitoring should ensure data is accurate and up-to-date. Finally, data should be regularly cataloged, curated and endorsed, so it can be found easily, understood and trusted by anyone in the organization.
Ensure Data is Timely
Functional Data Governance ensures data is as timely as it needs to be. Most organizations don’t need real-time data ingestion, but that doesn’t mean they don’t need to ensure data is timely and current.
Change Data Capture (CDC) helps ensure data in the data lake warehouse is updated as changes are made to the source system. CDC tools track and capture changes made to a database, helping organizations identify and capture only the changes made to data, instead of processing an entire data set. With CDC, organizations don’t have to wait for a full data refresh to identify and capture data changes.
Provide data & analytics for business use
Building robust business analytics for getting key data insights with properly defined KPIs (Key Performance Indicators) is a straightforward focal point for emerging Data Governance programs. Analytics highlight the importance of access in Data Governance programs – with robust analytics, data end-users don’t have to wait on reports from IT and sift through erroneous or duplicate data to get exactly what they need. Relevant analytics removes human error from data decision-making.
Data is Known
Building a data literacy program can help drive Data Governance and will help identify data owners and stewards, if they are currently unknown. During the establishment of a data literacy program, you will want to identify and engage with relevant stakeholders, including IT personnel, data management teams, business analysts and data users.
Identify what (Metadata) you want to capture with your stakeholders in your business terms/glossary, data dictionary, including data element names, business terms and definitions, data types, accepted ranges and values and data sources. Your data dictionary will help define how data is maintained and who is responsible for maintaining it. Once your data dictionary is in place, every relevant stakeholder should be educated and have access to the data dictionary. Data literacy builds scalability into your data management by eliminating tribal knowledge and building a common understanding of basic data stewardship.
Data is Trusted
In the last decade, data-centric industries have invested massively in point solutions for online application platforms, customer relationship management (CRM) and fraud detection. While each serves a functional purpose, they also create multiple versions of the truth, which can have far-reaching implications across business lines and operational teams.
These sprawling data ecosystems leave business users with unpalatable options, resulting in incomplete customer pictures, and shadow data marts/data warehouses. There is a better way. To fully leverage the benefits of their data, without sacrificing the agility of self-service reporting, organizations can create a Single Version of the Truth with a well-governed enterprise lake warehouse. With an enterprise lake warehouse, business users across an organization can confidently access trusted data, knowing they are pulling the most relevant and complete data, and getting a complete, 360-degree view of customers and their enterprise operations.
Data is Relevant to Business Use Cases.
Moving data to an enterprise data lake warehouse is just the start of a data estate modernization initiative. To make data usable to different use cases and business users, your data architecture must support integration into various applications.
Data transformation is THE essential bridge from data lake to business use cases, ensuring your data can be trusted, observed, and acted upon as it moves through your organization. Data availability and accessibility, enabled by data transformation, helps support the most critical indicator of a successful Data Governance program – a Data Governance Culture. To promote a Data Governance Culture, champions must exist throughout your organization – from IT to customer service, sales, and senior leadership.
The easiest way to show the value of Data Governance is to make data available for any use case – which often means making it available to third-party data science applications and software platforms. But that’s not as easy as it sounds – data applications are diverse, including ML/AI models, analytics dashboards, and marketing platforms. The data required to drive customer support use cases will be completely different from the data required for laser-focused marketing, or KPI-driven C-suites.
Data integration and transformation are essential to ensuring data consumers have access to relevant and ready-to-use data.
Data is secure, observable, auditable and compliant throughout its lifecycle.
It has always been imperative for organizations to keep data secure. But putting data behind a lock and key doesn’t make it more valuable. As data ecosystems grow and applications become more diverse, data security, observability and compliance are foundational in data management.
An important first step is to define Data Governance policies and procedures to be applied across your organization. When putting those processes in place, the right technology tools can help automate and scale Data Governance – reducing the burden on your technical team.
How to Build Your Data Governance Posture
Data Governance might be seen as a heavy lift, and even viewed as competition for other valuable resources in IT. It’s time to abandon that mindset. Regardless of where you are in your data estate modernization journey, Data Governance should be top of mind.
If you are just getting started, the first step should be establishing a clear understanding of data assets and how they are used throughout the organization. This includes identifying all data sources, classifying data based on sensitivity and importance, and defining ownership. Once you’ve established a glossary of data in the organization, you can determine how you want to control data – establishing who can access data, how they will access it, and the security, auditability, and visibility measures in place to enforce your Data Governance strategy.
If you have a floundering Data Governance program, take a step back and envision what you could improve. Tools that can automate manual processes help you troubleshoot faster, and ultimately free up resources for higher-value work that pays for itself and helps create a scalable Data Governance framework. Integrated Data Governance keeps data secure while making it accessible and observable throughout the data lifecycle. By integrating your Data Governance program throughout your data catalog, ETL/ELT and data stack, you can build practical workflows and formalized Data Governance roles, setting a strong, scalable foundation for your organization.
Need help getting started? Contact us for a complimentary consultation.
