Reading time: 7 min
The current judgment of a data warehouse is influenced by many factors, we will write about it in a broader perspective in another post. I will now examine in detail a factor that does not affect directly but affects the function and usability of a data warehouse in the long run: the change management processes of data warehouses.
There are two mistakes you can make in this field.
Technical control
One is lax change management. If neither the development policy nor the technology used in a data warehouse prohibits poor quality code from loading data, sooner or later poor quality code will inevitably load more and more data. This can lead to several problems. Incorrect or inconsistent data is more likely to get into the data warehouse.
Even if the data loading is correct, the auditability of the entered data may be corrupted.
- Incorrect or inconsistent data is more likely to enter the data warehouse.
- Even if the data loading is correct, the audibility of the entered data may be impaired.
- Downtime may become more frequent.
- Poor quality code is also difficult to maintain, so it will be more and more expensive and lengthy to find and fix bugs and develop new features.
Fortunately, we encounter rarely this type of problem at our clients. In the life of a company, the reporting technologies of the period before the development of a data warehouse are more characterized by such problems, which may also play a role in the management’s decision to build a data warehouse.
Out of the frying pan into the fire
The other mistake is if we go too far and make the framework and development processes that support the data warehouse too strict and complex, we have recently encountered this problem with one of our clients. Every step of the data warehouse development process was thoroughly documented and audited in a very high, abstraction level system. This may seem like an advantage at first, but unfortunately, it had reached the level where it reversed. The system made development at least as difficult as it helped, and the special expertise needed for development had to be gained from an external source. The processes around the outsourcing of work have made all changes very time consuming and expensive. Thus, on the one hand, data warehouse development projects have often failed, either due to an increase in budget or duration. On the other hand, the necessary correction of possible design errors was not always done due to high costs. There were clear signs of a data warehouse failure here, and one reason behind was probably the high level of technical change control.
Good practices
Of course, there is no such level of technical change management that is appropriate for every data warehouse. With the size and complexity of the data warehouse and the strictness of the industry regulation, the level of the necessary change control can change radically. A whole different kind of process is needed to develop a data warehouse with an attendance sheet or bank data, but here are some good examples.
On the business side, we see a very good practice at one of our clients. We implemented the data warehouse a few years ago, and we have been entrusting further developments since then. This could easily have led them to a situation similar to the previous bad example, where even the slightest improvement/bug fix is preceded or made impossible by a lengthy procurement and bargaining process. This could be avoided by defining a regular quarterly support framework from which (in addition to infrequent fixes) they could also order minor improvements with a low-level decision.
On the development side, I may sound biased saying this, but I think the Hiflylabs framework is a good compromise in this area. Different levels of audibility can be set at the project or even table level. Development tools and automated loaders greatly simplify development but are not so complex that a relatively experienced data warehousing expert would have trouble mastering them. When automating, we always make sure that tasks that are more complex than the capacity of the given device can be easily integrated into the system, so that the system does not become rigid. Once the data warehouse has reached the level of complexity that a framework is needed at all, it is worthwhile to apply it, and it will satisfy needs with a high level of complexity and auditable expectations.
Business control
For long-term success, it is not enough to balance the change management of the data warehouse from a technical point of view, it is important to design the change control processes from a business point of view. Here, too, you can make the same two mistakes.
Poor control
On the one hand, if there is no single BI professional oversight of the data warehouse, each business area is free to drag the data warehouse to its own perspective. Therefore, there is a need for a BI team responsible for the data warehouse that has adequate analytical resources to effectively monitor development needs and has sufficient authority within the company to be able to make a meaningful impact on needs and implementation. If this team is missing or not strong enough, the individual developments may not be built on each other but may be implemented independently or even against each other. If all the needs of each business area are met without someone first examining their impact on other features, replacing them with existing features, or negotiating with other business areas that are expected to need a similar feature, it will greatly degrade data warehouse quality in the long run. The same function is implemented multiple times, even with conflicting content, while lacking functions that benefit all business areas alike. One of the most important functions of the data warehouse, the concept of unification, is lost. Business areas are dissatisfied with the quality of the data, frustrated by the fact that their statements contradict each other while devoting far more resources than necessary to improve the data warehouse.
On the other side
On the other hand, of course, you can also go to the other extreme here. If the professional control of the BI team becomes so strong that it already becomes an obstacle to the improvements needed for the business areas, they will gain the data they need from other sources and the data warehouse will sooner or later become insignificant.
In a previous job, the head of the company’s global data warehouse team once used the term “sacrament” in a professional conversation about the data in the data warehouse. The positive and negative consequences of this attitude were evident in the data warehouse he supervised. All the data inside was very well reasoned, in a unified structure, with a very good user interface. It handled the problems of integrating data from different source systems smoothly, and their hosts were able to enter manually maintained metadata in a user-friendly intuitive interface. So overall, it was a pleasure to use the data in the data warehouse. On the downside, however, the data warehouse covered only two aspects of the operation of the 10,000+-person global company. Of course, this was not enough for any of the departments, and the company flourished with data warehouses with complementary functions (and varied quality) according to local, regional and business areas.
Good practice
Proper balance is also important in this area. There must be a strong BI team with adequate analytical capacity and sufficient authority over the data warehouse to enforce a unified data warehouse structure. But this team must also constantly pay attention to business needs, not to lose sight of user expectations by focusing on the internal operation of the data warehouse.
Lessons learned
I met with an internal BI team who, who, although not billed to internal “clients”, accepted change requests from business areas solely on the basis of business reasons that expressed the benefits of the change in cash. Such a model may not be feasible in all situations and in itself does not protect from the failure of too strong (or even too weak) change management. However, I think if a BI team has a reasonable vision of the data warehouse, a system like this can help make that vision come true.
Most of the time, when designing or implementing a data warehouse, we consider the data warehouse as a technical solution to a business problem. We believe that if our solution meets the client’s business needs, the data warehouse is expected to be successful, and this is often the case in the short term. However, we must not forget that the long-term success of data warehouses is determined much more by the business structures and processes developed than the level of implementation.
Author
Tamás Bárász – Senior BI Consultant