Data Mesh – What You Need to Know?
What is Data Mesh?
This term was introduced to the data world by Zhamak Dehghani in 2019 and is based on four principles that govern this data architecture model:
Domain ownership:
This principle requires domain teams to take responsibility for their data. According to this principle, analytical data should be organized by domains, bounded by team boundaries and system context, following domain architecture. In this way, the central data team no longer owns this data.
Data as a product:
Considering that there are consumers for the data beyond the domain, the domain team responsible for the data is responsible for meeting the needs of other domains by providing data with quality assurance, with the understanding that this data should be treated like any service or microservice in the organization.
Infrastructure platform for self-serve data:
The data platform will provide domain-independent capabilities, tools, or systems so that the Data Product can be created, executed, or maintained for all domains.
Federated governance:
To ensure that organizational rules and industry regulations are met, the governance area must provide standardization to be followed throughout the Data Mesh to ensure adherence. In this way, LGPD principles, for example, must be applied according to these standardized rules.
Why should you use Data Mesh?
Let’s imagine that your company plans to run a special event for Black Friday. A few weeks before it, you ask the analytics team for a report with information related to a specific product or customer preference, for example, based on a region.
Organizations invested in a centralized data lake precisely to drive their business based on the information it contains.
The data team always wants to answer every question quickly; however, there are demands already being developed, in addition to fixes in data pipelines broken after a change in an operational database, for example.
Because of scenarios like these, the data team cannot keep up with the speed required to serve all domains in the company, and it can often become a bottleneck.
At the same time, to gain more business agility, organizations have also invested in decentralized, service-oriented architectures oriented around domains. Even with the knowledge held by these domains, the central data team is overloaded when responding to these data-driven insight requests.
As the organization grows, the situation tends to get worse. Therefore, the best way out of this problem is to hand responsibility for the data to the domain teams. This is the idea behind Data Mesh: giving the domain responsibility for analytical data, with an architecture that allows teams to perform cross-domain data analysis in a way similar to a microservice or API.
How to Design a Data Mesh?
To create a Data Mesh architecture, it is important to emphasize that domain teams will perform cross-domain data analysis on their own. Each domain team is responsible for its operational and analytical data. Each domain creates its Data Product from its analytical data based on the needs of other domains.
Domain teams must be governed by global policies, security, and documentation standards from federated governance. So that domain teams can discover, understand, and use Data Products, the data team must ensure that the platform is domain-agnostic, allowing domain teams to analyze efficiently and easily create their own Data Products.
An enabling team should guide how the process should work and direct domain teams on how to create, analyze, model, and maintain Data Products in an interoperable way.
What are the components of a Data Mesh architecture?
Data Product
A Data Product is usually a data set that can be accessed by other domains, similar to an API. For example, a sales report in a Synapse Analytics table or a JSON file in Azure BLOB storage. A Data Product can also be a PDF report or a Machine Learning model that predicts next month’s sales estimate.
For the Data Product to be visible, it must be described with metadata, including information such as ownership, contact, data location and access, update frequency, as well as a data model specification.
Throughout the life cycle, the domain team must be responsible for ensuring the quality of this data, as well as its availability and the handling of duplicates or missing data.
Microsoft Purview can be used to design a Data Product.
Federated governance
This team is made up of members from all areas that compose the Data Mesh, agreeing on the global policies that define how Data Products should be created and maintained.
Global policies provide standards that can define how a given data set should be made available, such as a CSV or JSON in a Blob repository or in a Synapse table, for example, in addition to documenting predefined metadata sets for each Data Product in a wiki.
Among the global policies, they must ensure that rules such as LGPD are met to guarantee legal compliance for each sector.
Analytical Data
Data is usually ingested as raw and unstructured data. In earlier stages, this data needs to be cleaned and structured into small events oriented to domains and entities that represent business objects, such as items whose state changes over time.
This data can often be imported manually through a CSV file or even by email. Data from other teams is integrated as external data.
The published Data Product is the result of aggregating events, entities, manual data, and external data.
Ingestion
For domain teams to ingest their data, their software system must be designed according to a domain-oriented architecture.
If the organization has a messaging system, such as Kafka or Event Hub, for example, data can be collected and sent to the data platform by ingestion tools such as Azure Data Factory, Databricks, or Azure Synapse, including in real time or NRT.
With these well-defined events, ingestion processes will ensure data cleansing and quality, excluding duplicates and anonymizing sensitive data.
When it is not feasible to change the software to guarantee an ingestion process, it is possible to use CDC to transmit this information directly to the data platform, as well as use an ETL or ELT process when data that is not available in real time or NRT is needed for the Data Product.
Data Cleansing
Data cleansing processes are the responsibility of domain teams and consist of ensuring that data quality is maintained; for this, no one is better than the domain team that knows its data well to handle it.
To achieve an efficient data cleansing process, a few points must be ensured:
Structuring: transform NoSQL or semi-structured data into analytical models, such as extracting fields from a JSON.
Mitigation: provide information consistency when data changes, for example, filling null values with sensible defaults.
Deduplication: remove all duplicate entries.
Completeness: Make sure the data contains agreed periods, even when there are technical problems during ingestion.
Fix outliers: identify and correct invalid data generated by bugs.
Usually in the Azure environment, processes after mapping by Purview are processed by Azure Data Factory, Azure Synapse, or Databricks, running routines that ensure data cleansing and quality.
Analytics
Domain teams aggregate their analytical data as Data Products that are relevant to other domains.
Through notebooks such as Databricks or Synapse Analytics, domain teams perform analytical queries to investigate and build their models.
Data visualization can be done through tools such as Power BI, where we can visually see any trends or anomalies.
More advanced analytical models can be created with tools such as Synapse or Databricks, where data scientists can generate their statistical models through languages such as Python, Scala, or R and specific libraries.
Data Platform
The data platform functions as a hub that supports self-service for the organization so that domain teams can create their Data Products.
Each domain team receives its own isolated area, with preparation for data ingestion, querying, storage, and visualization.
The data catalog must be created, along with git repositories and any tools that ensure the process of creating a data pipeline within the organization. One of the responsibilities of the data platform is to efficiently combine all the tools that support these pipelines and ensure that the processes are executed.
Data Mesh Architecture on Microsoft Azure
Conclusion
The concept of Data Mesh is relatively new, as mentioned at the beginning and based on the source I used to produce this content: Zhamak Dehghani developed the concept in 2019.
To reach this level of Big Data in a company, it is very important for the company to have a high degree of data maturity, because governance, security, and quality processes must be implemented and efficient for the organization.
All organizations seek not only quality but also speed in data response for strategic decision-making.
To do this, it is essential for these companies to become data driven, evangelize data across all domain units of the organization, and mature these processes with structured analytics, DataOps, and efficient data governance. The path is not short, but with these points addressed, the successful implementation of Data Mesh in the company is inevitable.
Source: O’Reilly – Data Mesh, Zhamak Dehghani, 2019.
