The Exchange of Mass Amounts of Data: Ensuring Well-Governed Data Distribution
The exchange of mass amounts of data is critical for the majority of business processes today, enabling innovative customer experiences at scale. But quickly getting pristinely-clean, high-quality data where it needs to be—whether to an in-house system or to external partners—is a big challenge for data teams. And to do so in real time is even more complex. Moving data securely, reliably, and quickly requires good data governance—but what kind of frameworks are required to ensure data is well-governed through real-time distribution within the organization?
Build a Central, Self-Service Portal
To ensure data remains well-governed throughout its lifecycle, start by building a central hub where data from all your separate repositories can be accessed in one place. From here, you can set up multiple pipelines with rules, restrictions, and policies dictating data accessibility, data velocity (e.g., whether data is streamed or not), schema enforcement, data quality, and more. This self-service portal should allow your organization to virtualize all data sources into a single, unified data layer. This provides a bird’s-eye view of your data landscape, making it easier for users to access and use while implementing governance controls around data access, privacy, security, and more. Having this centralized self-service portal is key to federating data out across the company.
Establish Quality-of-Service Governance
Whether data will be shared in real-time or asynchronously, it’s important to ensure that all data adheres to the governance defined based on its sensitivity and value. Even data that may not seem necessary to access in real-time today could become critical in the future. From the onset, you should apply varying levels of governance and controls around access and security depending on the data. This means applying rigor around governance at the beginning of the data lifecycle, which might include robust data quality monitoring, lineage tracking, and security controls, depending on value and sensitivity of the data. That way, any dataset can easily be surfaced and shared as requirements evolve, without costly refactoring later on.
Publish Once, Publish Right
When data moves in milliseconds, strong governance ensures that it flows to the right places through the right rules at the right time. Make sure to establish rules about when and where data is published, and to which applications it becomes available, but also to establish monitoring and observability. Teams need confidence their data will be available for specific critical use cases exactly when they need it, whether that’s in real time or asynchronous. At Capital One, the use of real-time data helps detect fraud and enable fast, secure transactions—but batch data is still needed to power use cases and drive AI/ML at scale.
Make Data Traceable and Auditable
Transparency is critical when setting up a data governance structure. Teams need to be able to monitor and audit all data flows to ensure compliance with governance frameworks, identify potential issues, ensure data security, and improve overall efficiency.
Invest in the Right Storage
To make wide-scale data sharing possible, companies need to invest heavily in the right storage and infrastructure. Most data lakes and warehouses also allow users to toggle levels of access and monitoring for specific datasets. Make sure to check on the level of controls and monitoring offered by your vendors of choice. Not all data needs to be stored in the highest performance (and highest cost) warehouses all the time — some data can be stored more economically in data lakes if it doesn’t need to be accessed and shared in real-time. Even within the context of real-time data, there are mechanisms to trade off cost and performance. The key is to establish smart governance mechanisms to intelligently move data across storage tiers based on access requirements and use cases through the establishment of quality of service and SLAs that define latency, retention, and cost tolerance.
Conclusion
By taking a strategic approach to data governance upfront, an enterprise can unlock the full potential of their data at scale. Users can find, access, and use data quickly, securely, and reliably to power real-time applications and critical decision-making. While implementing robust data governance is a significant investment—and tight cooperation between data, business, and leadership teams—the competitive advantages of being a truly data-driven organization make the effort worthwhile.
Frequently Asked Questions
Q: What is the main challenge in implementing data governance?
A: The main challenge is ensuring that data is well-governed throughout its lifecycle, while also providing users with fast and secure access to data.
Q: What are the key components of a data governance framework?
A: The key components are a central self-service portal, quality-of-service governance, publish once, publish right, make data traceable and auditable, and invest in the right storage.
Q: How can organizations ensure data quality and security?
A: Organizations can ensure data quality and security by implementing robust data quality monitoring, lineage tracking, and security controls, and by applying varying levels of governance and controls around access and security depending on the data.
Q: What is the importance of transparency in data governance?
A: Transparency is critical in data governance, as it allows teams to monitor and audit all data flows, ensure compliance with governance frameworks, identify potential issues, and improve overall efficiency.
