PepkorIT Migrates Oracle Enterprise Data Warehouse to Google BigQuery
Steinhoff International Holdings, Cape Town, South Africa, www.steinhoffinternational.com, is a global retailer with stores across multiple regions. Steinhoff’s IT division PepkorIT was responsible for maintaining an existing on-prem multi-tenant Oracle-based Enterprise Data Warehouse (EDW) with custom SQL script ETL pipelines, daily store transaction activity batch loads, and various monthly consumer information data feeds, and analytical BI dashboards.
Legacy Oracle EDW was running out of capacity and needed to improve time to insights from days to hours and minutes.
The decision was made to migrate EDW to Google Cloud Platform and needed planning, design, implementation assistance. Myers-Holum Inc. (MHI) led the project to define the future Google Cloud-Based solution architecture and implement a data ingestion framework using Data Flow that reused the same pipeline for both batch ingestion from Google Cloud Storage, and real-time ingestion from OLTP database binary logs streaming through Google Cloud PubSub into BigQuery.
The MHI solution centered around self-healing Data Flow pipelines that allowed for schema changes over time with minimum operational intervention and automatic data reprocessing. The solution included sensitive data masking, balance and control system tables, full data lineage for data landed into GCP, data quality rules implementation, BigQuery schema design based on Myers-Holum industry best practices, downstream data processing for BI and Analytics use cases, and job monitoring using Stackdriver and Datastudio dashboards.
Google Cloud BigQuery-based EDW allowed Steinhoff to reduce time to insights from days to minutes. Google Cloud serverless technologies such as Data Flow provided scalable infrastructure to ingest batch and real-time data quickly and reliably while reducing CapEx and Opex costs.