Objective: Migrating Complex Teradata Environment to Google Cloud
Datametica’s client, an American-based retailer, wanted to migrate its Teradata environment to Google Cloud Platform (GCP), in order to optimize performance, cost and remove overall infrastructure provisioning delays.
The client was looking for a partner to enable their migration from the on-premise Teradata DataWarehouse to the Google Cloud Platform, with upgraded versions of Cognos for enterprise reporting.
Challenges: Technical, Scalability, and Costs Issues
Key technical challenges faced by the client –
- Very complex legacy Cognos reporting
- Long running and complex analytics cube building
- Concern on how to integrate enterprise scheduling tasks from Control-M
- A desire to move orchestration and dependency management through composer
Legacy:
- Control-M Jobs
- Tables/Views/Macros
- Informatica Mappings
- Shell scripts
- SQLs, BTEQ’s and UDF’s
- .NET/C# Applications
Solutions: Lift and Shift Migration to GCP
- Datametica provided the client with a detailed assessment of the existing environment by using Eagle – an automated assessment & data migration planning technology – and provided the detailed migration plan, along with an understanding of the current data model, ETL logic, associated workloads, pipelines, and deliverables.
- Planned the jobs and tables based on the dependencies by leveraging strong technical analysis of the on-premise tools and understanding of the data model.
- Converted and repointed Informatica ingestion and transformation jobs to GCP by using Datametica’s Automated Code (SQL, Script) and ETL conversion service – Raven.
- Performed data validation using the Datametica Automated data validation tool – Pelican between Teradata and BigQuery.
- Implemented the GCS Fuse mechanism in order to establish communication between Control-M (on-premise workload automation tool) and GCP Cloud Composer.
- Cloud Data Fusion – a fully managed, cloud-native, enterprise data integration service – was leveraged for quickly building and managing data pipelines.
- Composer DAGs integrated with Wiki in order to maintain strong documentation for failure handling scenarios.
- C# Application Optimization done for uploading data into BigQuery.
- Optimization of complex and long-running Cognos reports
- Modernization incorporated into GCP
- Eliminated the old source system for sales subject area by implementing the new data pipelines that has eliminated the bugs in the old source.
- Reconciliation / validations of new source system for sales data.
- Cognos Compatible Query Mode (CQM) models converted into Dynamic Query Mode models (DQM).
- Cognos IQD converted into Cognos FrameWork (FM) models
- Changes from multiple database connections to single connection for Cognos packages.
Client Benefits: Improved Efficiency, Reduced Cost, and Optimized Infrastructure Provisioning Timelines
- Improvements in efficiency and performance due to the consolidation of different market places into a single unified data model on GCP.
- Using automation, Datametica migrated complex Teradata data warehouse, analytics, and reporting workloads to GCP.
- Datametica solutions delivered faster cloud migration with lower costs and lower business risk.
- Collated a number of Teradata objects were duplicated, reducing maintenance.
- Automated Pelican testing, gave confidence in the decommissioning of the Teradata Warehouse.
- Ability to eliminate compute restrictions as GCP we can run more reports at a same time
- Optimization of Cognos Reports : Three of the most critical reports had the most significant improvements, some were improved from ~75 minutes to ~30 seconds.