To leverage the full potential that lies in their data and build a new base for data democratization, data-driven leaders currently rethink their data strategies and data governance approaches. One of the most-discussed concepts is the data mesh paradigm as it was brought up by Zhamak Dehghani in 2019.
In a keynote from September 2020 (here at Youtube) she describes it as a “paradigm shift in analytical data management architecture”. It was developed from the observation that many leaders in big data management still failed to see the benefits they were hoping for – despite their undisputed proficiency (and budgets spent) in the area.
In short, the data mesh paradigm works to bring trends in data management together with insights and ideas from domain-driven architectures for distributed environments and product thinking.
Zhamak Dehghani describes the building blocks as follows:
“Data mesh as a platform; distributed data products oriented around domains and owned by independent cross-functional teams who have embedded data engineers and data product owners, using common data infrastructure as a platform to host, prep and serve their data assets.”
Cited from this article by Zhamak Dehghani, worth the read for more details.
Data Mesh as Bridge towards Digitization
In our latest projects, we introduced customers to the new paradigm, from strategy to conception to implementation. Data Mesh can be utilized as a bridge to more data democratization and scaling. The focus was, among others, classic enterprises whose business model is not yet based on digitization.
These companies deal with very specific challenges like
- monolithic systems that have grown over decades in some cases,
- heterogeneous operational and analytical environments with hundreds of data transfer routes between them,
- silo-like organizational structures,
- central competence centers,
- long lead times for data to insight,
- and more.
The lamented symptoms are always similar in organizations: data management and analytics are too complex, too lengthy and not adding value. Unfortunately, this led to even more siloed solutions driven by success-incentivized departments that needed access to their data quickly. As a result, business units demand more data democratization or, at worst, implement their own solutions, which in turn reinforces the undesirable silos and contrasts with the meaningful single point of truth efforts by data warehouse initiatives from the past few decades.
Another trend this provoked was the demand for more data democratization. This is often answered with self-service BI and explorative analytics solutions. In this regard, companies are unsure how far they should go to offer an effective balance between centralized governance and decentralized degrees of freedom.
Advantages of Democratized Data as a Product
In this situation, the principles of the data mesh approach call for data to be understood as a product and to be autonomously created and managed by domains. Federated governance should make central policies more flexible and infrastructure accessible to domains in self-service mode. The concept holds out the prospect of many advantages:
- higher productivity through transfer of responsibility to domains
- associated independence allows more data products to be created and shared between the domains
- lamented bottlenecks in central functions become irrelevant.
In a figurative sense, the entity can breathe and scale again.
From Monoliths to Data Mesh with Company-Specific Approach
Working in companies with organizational silos and according to data structures, we vote for a company-specific specification and adaptation of the generic data mesh principles. By that we work with and not against the unique company situation towards a desired vision.
Usually, the change towards data mesh is part of a larger transformation that leads to a data-driven organization. This calls for a holistic way when introducing a data mesh, by applying proven methodologies of business process and data management and including the building blocks defined in the original concept.
In a very concrete and practical way, this approach interlocks the individually designed principles, via required capabilities, processes to be adapted, architecture components to be complemented, up to technical blueprints that the domains configure and use for their analytical data products.
CAMELOT provides holistic support by applying proven methods of business process and data management, product thinking, enterprise architecture, and cultural awareness to organizational and technological change. This article provides further insights and an example for a domain architecture in a pharmaceutical company.
