What is datawarehouse automation?

What is datawarehouse automation?


There are a few definitions

The Data Warehouse Institute (TDWI)

“…using technology to gain efficiencies and improve effectiveness in data warehousing processes. Data warehouse automation is much more than simply automating the development process. It encompasses all of the core processes of data warehousing including design, development, testing, deployment, operations, impact analysis, and change management.”


DWA is not a data warehouse appliance, nor data-warehouse-as-a-service (DWaaS) – it’s software that automaticly generates a data warehouse by analyzing the data itself and applying best practices for DW design embedded in the technology. Another name for this type of technology is “metadata-generated analytics”.
Source: Forrester: Data Warehouse Automation Helps Close The Data-To-Insight Gap by Boris Evelson and Nasry Angel.

Our view

Basicly these definitions are a way of saying that the predictability of datasets allows you to standardize your processes based on metadata.
The physical implementation only follows patterns that allow you to use generators for those patterns.
Based on the metadata that you store to do this, you can change your way of working in many ways .. as the definition of TDWI already indicates.


But you have to change your mindset

Change the mindset from procedural programming in ETL Tools to setbased operations.
Change the mindset from handcoding to work with templates and metadata.
This is what datawarehouse automation is about.
Use the built-in intelligence of a tool to get rid of repetitive errorprone handcoding of ETL Tasks.
Use the built-in intelligence of a tool to do those tasks in 20% of the time.
Implement changes to your datawarehouse by adding some metadata (for instance a new column) and pressing the “Generate” button. Prevent the need to manually implement changes throughout all the layers of your data warehouse.

A data warehouse architecture with multiple tiers

Classic ETL is slowly dying

The answer to growing data volumes and growing variety of sources is automation. So your team can deliver!