Augmented data management: Data fabric versus data mesh
Data fabric and data mesh are emerging data management concepts that are meant to address the organizational change and complexities of understanding, governing and working with enterprise data in a hybrid multicloud ecosystem. The good news is that both data architecture concepts are complimentary. But what exactly are a data fabric and data mesh, and how can you use these data management solutions to take advantage of your enterprise data for better decision-making?
What’s a data fabric?
Gartner defines a data fabric as “a design concept that serves as an integrated layer of data and connecting processes. A data fabric utilizes continuous analytics over existing, discoverable and inferenced metadata to support the design, deployment and utilization of integrated and reusable datasets across all environments, including hybrid and multicloud platforms.” [1]
The data fabric architectural approach can simplify data access in an organization and facilitate self-service data consumption at scale. This approach breaks down data silos, allowing for new opportunities to shape data governance, data integration, single customer views and trustworthy AI implementations among other common industry use cases.
Since its uniquely metadata-driven, the abstraction layer of a data fabric makes it easier to model, integrate and query any data sources, build data pipelines, and integrate data in real-time. A data fabric also streamlines deriving insights from data through better data observability and data quality by automating manual tasks across data platforms using machine learning. This improves data engineering productivity and time-to-value for data consumers.
What’s a data mesh?
According to Forrester, “A data mesh is a decentralized sociotechnical approach to share, access and manage analytical data in complex and large-scale environments—within or across organizations using.” [2]
The data mesh architecture is an approach that aligns data sources by business domains, or functions, with data owners. With data ownership decentralization, data owners can create data products for their respective domains, meaning data consumers, both data scientist and business users, can use a combination of these data products for data analytics and data science.
The value of the data mesh approach is that it shifts the creation of data products to subject matter experts upstream who know the business domains best compared to relying on data engineers to cleanse and integrate data products downstream.
Furthermore, the data mesh accelerates re-use of data products by enabling a publish-and-subscribe model and leveraging APIs, which makes it easier for data consumers to get the data products they need including reliable updates.
Data fabric vs data mesh: How does a data fabric relate to a data mesh?
A data fabric and data mesh can co-exist. In fact, there’s three ways a data fabric enables the implementation of a data mesh:
- Provides data owners data products creation capabilities like cataloging data assets, transforming assets into products and following federated governance policies
- Enable data owners and data consumers to use the data products in various ways such as publishing data products to the catalog, searching and find data products, and querying or visualizing data products leveraging data virtualization or using APIs.
- Use insights from data fabric metadata to automate tasks by learning from patterns as part of the data product creation process or as part of the process of monitoring data product
A data fabric gives you the flexibility to start with a use case allowing you to get quick-time-to-value regardless of where your data is.
When it comes to data management, a data fabric provides the capabilities needed to implement and take full advantage of a data mesh by automating many of the tasks required to create data products and manage the lifecycle of data products. By using the flexibility of a data fabric foundation, you can implement a data mesh, continuing to take advantage of a use case centric data architecture regardless if your data resides on premises or in the cloud.
1 “Data Fabric Architecture is Key to Modernizing Data Management and Data Integration” Gartner. 11 May 2021.
2 “Exposing The Data Mesh Blind Side” Forrester. 3 March 2022
The post Augmented data management: Data fabric versus data mesh appeared first on Journey to AI Blog.