DEVELOPING A METHODOLOGY TO UNIFY DATA LOADING INTO THE DATA VAULT MODEL

Konakov Pavel Olegovich

In the process of working with the corporate data warehouse there is a question of standardization and unification of data loading streams from the sources. A unified approach is necessary to keep the principle of actualization and historicity within the data objects loaded by different streams. This paper proposes to create an additional layer of abstraction for metadata describing the loading principle using ETL-processes. An ETL process is divided into two parts: the control flow, which is based on a pre-designed template with the worked out algorithm for collecting parameters to start the work flow that implements the logic of data loading from the source into the target table. Based on the described meta-information, the corresponding DWH component collects flows for the ETL flow orchestrator.

Keywords:big data, data processing, corporate data warehouse, ETL-process, standardization of development