Data Validation Post Cloud Migration: Netezza to GCP – BigQuery

Netezza is a well-known and powerful data warehouse by IBM that enables you to store data and analyze huge quantities of data. A legacy system like Netezza, however, needs significant hardware, maintenance, and license expenses. Due to issues with node management, data volume per source, and archiving expenses, scaling this might be challenging. The storage and processing power of Netezza are limited by hardware appliances. Extending the appliance’s capacity after the maximum usage is achieved is a difficult task that is sometimes even impossible.

Majority of the businesses are executing data warehouse modernization and cloud migration initiatives, in order to unlock scalability and futuristic technologies. Google Cloud Platform – BigQuery is among the top cloud players in the market presently. BigQuery eliminates the need for a database administrator and the management of infrastructure. BigQuery is a serverless, petabyte-scale data warehouse that can scan billions of rows without an index in a matter of seconds and is fully managed. Since it utilizes Google’s infrastructure, BigQuery can replicate each query and perform it on tens of thousands of servers at once. Moreover, it has built-in capabilities for geospatial analysis, business analytics, and machine learning.

Migration of Netezza to Google Cloud Platform – BigQuery

Some of the common reasons why businesses are migrating from legacy Netezza to BigQuery are technical challenges, scalability issues, batch processing performance, and cost. As per our experience at Datametica, these are among the most common reasons why businesses choose GCP over Netezza. Over the years, we have successfully completed numerous migrations from Netezza to GCP with the help of our automated cloud migration suite.

Data warehouse modernization and cloud migration have many technical challenges in themselves, but to gain confidence in order to retire existing legacy data systems, it is important to have detailed, in-depth data validation. This process has all together different pitfalls, but data validation ultimately leads to successful cloud migration.

Data Validation Post Migration: Real-World Application

In one of the cases, a leading healthcare organization needed to carry out data validation while moving their data warehouse to GCP. Time was of the utmost importance to the company. They wanted to finish the data warehouse migration before Netezza’s life cycle ended. the client’s initial plan, the cloud engineers would validate the data using a conventionally manual process. The entire process was time-consuming and carried a significant commercial risk if it took longer than expected. 25 BigQuery engineers were going to be hired by the client, and their job would have been to manually validate data. They would not be able to run many instances of data validation and would only be able to evaluate a sample of the database rather than the complete database. The client had opted for a manual data validation process as they were struggling to find an automated data validation technology that could perform detailed validation, conclude the validation process beforehand, and give them confidence in decommissioning their legacy Netezza system.

When Datametica was approached by the client, we proposed our automated data validation and reconciliation technology, Pelican. The client was initially a bit hesitant as they wanted to perform validation without risking time extension and data security. In such cases, the best way to demonstrate the capability of your product is to start with POC. In this case, we started with the same approach. The successful POC enabled clients to understand the capabilities of Pelican and instilled confidence in them. Datametica’s Pelican has the capability to validate and reconcile petabyte-scale data down to the cell level and across heterogeneous systems to automate, accelerate, and streamline the data validation process during data transfer to the cloud without data movement.

Pelican performed data validation in parallel to the migration process to help the client cut down on time. Unlike its competitors and other tools, Pelican has the capability to perform parallel validation, whereas they only allow post-migration validation of data. In comparison to the manual validation process, this helped the client save over 60% of their time.

Following features of Pelican made this happen:

• Automation: Automated data validation & reconciliation
• Zero Factor: No coding requirement & data movement
• Accurate: Intelligent comparison at cell level
• Parallel Run: Validation during migration
• Insights: Detailed reports identifying discrepancies

Closing Lines

Datametica’s automated data validation technology – Pelican not only helped its client ‘one of the top healthcare companies’ finish the data validation before-time but also enabled them to confidently decommission their legacy data warehouse – Netezza. The client saved 60% of the time in the data validation process in comparison to the previously planned manual data validation method. The additional benefits of using automated data validator Pelican included validation of the entire dataset down to the cell level each and every time (as needed) rather than the traditional approach of sampling and one time-validation. This assured 100% accuracy. Pelican and other automated products of Datametica, Raven – automated code conversion technology, and Eagle – automated data warehouse assessment and cloud migration planning tool, have empowered many customers of Datametica by making their cloud migration journey seamless.