In 2025, roughly 463 ZB of data will be created every day and its value estimated on $229.4 billion. Although digitization is not a new topic, these numbers show: the velocity and value of data has reached unexpected dimensions. This enormous monetary value is made up of optimization of internal processes and decisions, monetization of data itself and the development of data-driven business models, among other things. However, to unleash its value effectively, an organization must become data-driven.
Becoming Data-Driven: What Does It Mean?
There is one crucial aspect that must be taken into account for a successful digital transformation: the high velocity at which data is generated in huge quantities on a daily, hourly, and even minute-by-minute basis. The sheer volume of information exceeds an unimaginable value and is of utmost importance for companies with regard to their potential when it comes to data-driven decisions. To effectively maximize the potential value of an organization’s data, data must be reliable, of high quality and easily accessible to those who need to use it. Still, in most companies, this is not the reality. In fact, the data is stored in silos and therefore cannot be found easily and is not available in a timely manner. Often, it is of low quality, poorly structured and formatted, decreasing its interoperability.
A well-suited approach to break down these silos and unleash the data’s full potential is establishing a modern data architecture. The following blog should serve as a guide on how to design the foundation of a sustainable modern data architecture — the governance framework.
Six Core Elements of a Data Architecture Governance Framework
When implementing and maintaining a modern data architecture, appropriate governance is critical. Having the right framework for data architecture governance is an essential foundation for establishing a data architecture organization. Based on the experience we have gained in our data architecture projects, six core elements have proven to be a suitable basis.
It starts with defining the processes needed to develop and maintain a strategy-matching data architecture, each on a strategic, tactical, and operative level (first element). Second, the roles required for a proper data architecture organization need to be defined. This should include specifying the relevant tasks, as well as accountabilities and responsibilities of each role. Further mapping of these roles to the relevant processes (third element) as well as positioning them within the organization results in an organizational framework (fourth element), that, together with the establishment of committees, ensures holistic and synergetic planning, approval, and control of the data architecture governance. The creation of standards and policies (fifth element) around metadata handling and governance processes, among others, enables consistent data standardization and integration across the enterprise. This helps building an enterprise data architecture that supports the operating model (sixth element) required to break down silos within the business and technical processes and remove barriers to unleash the full potential of data.
In our first blog we will focus on the first three elements, as they build the foundation for the other elements. The other elements will be described in more depth in one of the upcoming blogs. Therefore, we start with the processes, before moving on into an organizational framework, highlighting roles needed to fulfill these steps.
Processes
Our experience shows that it is useful to start the design of a data architecture governance framework with the definition of the processes at strategic, tactical and operational levels.
As the name indicates, the strategic processes focus on long-term planning of data architecture activities. Decisions and approvals made here serve as a polar star, providing orientation for the whole organization. The processes involved in this early stage start with agreeing and defining the overall data strategy, followed by assessments and evaluations of existing structures: What are the requirements needed by the business, and to what regards are they already being met by the IT infrastructure? Next to that, the definition of a change management and communication strategy as well as the management of data governance processes is crucial. Alongside that, it is important to assess and evaluate how the organizational framework is currently arranged and what roles and tasks are included.
On the tactical level, processes are generally characterized by modelling activities and proactive data quality and requirements management. They involve handling medium- and long-term decisions regarding general data architecture standards and guidelines as well as creation and maintenance of (logical) data models. In this stage, managing the general lifecycle concept of data products is in focus – from their creation to onboarding and updating up until their archival. It should be decided how to handle general data compliance, privacy, and security as well as data quality along that lifecycle. Next to that, processes around metadata management and data product requests need to be defined. At last, the evaluation of tooling and its implementation should be considered at this level.
Finally, the tasks and processes on the operational level are focusing on short-term proactive and reactive activities around data utilization and enablement. These include the general maintenance of data products along their lifecycle, the management of business and technical metadata as well as managing the data product requests. In addition to that, organizational change management activities should take place to ensure the organizational change needs of target groups are being identified and met by elaborating further education and managing the knowledge as well as providing continuous support on a technical and business level. Eventually, the initiatives related to data architecture governance need to be managed, executed, and supported according to the processes defined.It has to be kept in mind, that some of these processes can or should be carried out simultaneously, while others are co-dependent and need a successor process to be performed first.
Crucial roles for maintaining modern data architecture
After having gained an overview of the necessary process steps of modern data architecture, let us now address the essential roles that fulfill these steps. Figure 5 shows five roles according to the three planning levels.
The quality manager and the data product owners are involved on the tactical level. While the quality manager is concerned about overall data quality standards and policies, the data product owners manage data products along the lifecycle and data product requests within their domains. The transformation manager works flexibly across all three levels of planning as this position focuses on executing training and managing change, including a communication strategy.
The data architect has principal responsibility at the strategic level for managing strategy, governance processes, initiatives, and organization. Typical tasks for this position include defining the vision for data architecture governance, determining the main deliverables and timeline, and deriving strategic goals. The key requirements for this role are an excellent understanding of data architecture, business processes, relevant systems, and governance process implications. To be a successful leader, strong communication skills and flexible leadership are needed as this role is usually responsible for resolving conflicts within the data architecture team and driving harmonization with other ecosystems. Finally, and most importantly, the data architect must have a strong willingness to take the initiative and bear the appropriate responsibility, with the ability to see the big picture, as this role represents and is accountable for the entire data architecture organization.
The data stewards maintain technical & business metadata and data lineage as well as provide support at the operational level. As discussed in the previous article, metadata and data lineage are essential components of a data catalog that enables a data-driven enterprise. Thus, the data steward can be considered the engine of a data catalog, ensuring that metadata is available and searchable for other users. In daily routine, the data steward collects, tracks, and troubleshoots insufficient data product lineages and develops technical solutions to automate the extraction of technical metadata. Therefore, exceptional knowledge of tools and technical solutions for metadata extraction is required. Additionally, a good understanding of business processes and relevant systems is required to capture business requirements and provide business context to data products. Since for data products both technical and business metadata need to be managed, depending on the scale and complexity of the organization, the role of the data steward can be divided into technical and business data steward.
Combining these essential roles with the above-mentioned processes results in an organizational governance framework – the foundation of sustainable modern data architecture.
Most organizations already manage vast amounts of data but barely benefit from its potential due to their business and technical silos, low quality, poorly structured and formatted data, and decreased interoperability. A modern data architecture with a strong data architecture governance foundation helps to extract the advantage out of the existing data and ensures the long-term competitiveness of organizations. Ultimately, there is no one-size-fits-all organizational structure. At Camelot, we can help you plan and build your company-specific modern and sustainable data architecture governance framework.
We would like to thank Stefan Morgenweck for his valuable contribution to this article.
