Streamlining Data Validation in Cloud Migration with Pelican’s Rules Engine and in-built Scheduler

Validating migrated data is a complex process that demands the utmost precision to successfully complete the Cloud migration process. Most migration projects involve migrating huge volumes of data (in petabytes!), which can affect the accuracy of the validation process in the long run. Such inaccuracies or missing data lead to significant losses for organizations, lowering their confidence to decommission their legacy warehouses. What they need is a tool that can help them seamlessly validate their migrated data with accuracy, while saving time and costs. Sounds too good to be true – but it isn’t anymore!

One-stop shop for data validation

Datametica’s Pelican automates and simplifies the validation process across heterogeneous source and target platforms with minimal network bandwidth consumption and no data movement. Check out how Pelican’s Zero Factor Features make it easier to validate data.

In this blog, we dive deeper into Pelican’s rules engine and automated in-built scheduler to see how it helps save time and money during the validation process.

⚠️ [NEWS]: AI-Powered Pelican is now available for users on the Google Cloud Marketplace, AWS Marketplace, and Azure Marketplace!

Build complex validations seamlessly with default and custom rules

In a typical migration project that uses the “lift and shift” method to improve the quality of migrated data, the format of migrated data changes, including Date to timestamp/Datetime, CHAR to STRING, differences in scale and precision of different data types such as Float and Double, and more. When “Date” is converted to Datetime, it is flagged as a data discrepancy, despite the fact that the new format was intended to improve data integrity. For example, see how the format changes for different data types in the target system during migration and how it might get flagged as a data mismatch during the validation process. 

For example, let’s see a couple of simple data format changes between the source and target system during migration. In case one, the data, let’s take date, was stored in the source was in the format of MM/DD/YYYY, which was 05/08/2023. During migration it was stored in the target table as YYYY-MM-DD with time that is hh:mm:ss, so in the target table the same value was stored as 2023-05-08 13:10:10. A general validation tool will mark this as a mismatch, but Pelican’s intelligence capabilities validate this comparison because of its rules engine.

Note: XXXXX is space

Table 1: Examples of various data type format changes during migration

Pelican’s rules engine resolves this issue by automatically applying default expressions to validate various post-migration data types. The rules engine features both automated and manual formats. In the automation format, Pelican automatically reads the Source and Target table DDLs and is intelligent enough to anticipate data mismatch issue and resolves them in advance by automatically applying default expressions, such as applying TRIM on Source (Case I) OR removing hh:mm:ss from the target (Case II). In the manual format, Pelican permits more complex custom expressions, such as REGEX, if the automated expression is insufficient to resolve the issue.

With Pelican’s rules engine, users can also define custom rules to apply to specific data fields or entire datasets, as well as combine multiple rules to construct more complex validations. 

Schedule validation of multiple tables with Pelican

In the world of data validation, automated scheduling plays a crucial role in ensuring the accuracy and quality of datasets. Let’s delve into three different modes of scheduling in Pelican: Inbuilt Scheduler, Group Scheduler, and Complex Scheduling. Each mode offers unique capabilities to handle data validation efficiently. Let’s explore them in detail!

Inbuilt Scheduler:

The Inbuilt Scheduler of Pelican is a powerful feature for handling validation of incremental data. By validating data incrementally, organizations can ensure ongoing accuracy and quality without the need to validate the entire dataset each time. This mode allows for efficient and targeted validation, saving time and resources. With the Pelican’s Inbuilt Scheduler, data can be validated in smaller subsets, ensuring that only the newly added or modified data is validated.

Group Scheduler:

Data lineage is the key to understanding the movement and transformation of data throughout its lifecycle. The Group Scheduler in Pelican leverages data lineage to accelerate the validation process and visualize the results for multiple tables simultaneously. Lineage data is uploaded into Pelican, and schedulers are created on all relevant tables. These schedulers are then grouped using the Group Scheduler functionality. When executed, the Group Scheduler triggers the entire data pipeline, validating the data and providing a comprehensive view of the results for all tables. This approach saves time and effort by streamlining the validation process.

Complex Scheduling:

In some cases, organizations require more advanced scheduling capabilities, such as event-based or trigger-based automated executions. Pelican provides APIs that allow users to leverage complex scheduling features. By consuming these APIs, organizations can automate data validation processes based on specific events or triggers. This enables seamless integration with other systems and applications, ensuring timely and accurate data validation.

Suggested read: Automating Test Management & Validation Process Management for a Successful Cloud Migration

Time to accelerate your validation process 

Through its rules engine and diverse scheduling options, Pelican can streamline the validation processes and enable you to make informed decisions based on reliable and validated data. With accurate validation of huge volumes of data, you can ensure the completion of your data migration successfully within the estimated time and budget while complying with industry guidelines. 

Pelican is an all-inclusive solution that assists you in standardizing your data, thereby increasing its reliability and consistency. Reach out to our experts to learn more about Datametica and its suite of automated data migration tools that can help you in data warehouse migration and modernization and cloud optimization for performance enhancement and cost saving. 

To learn more about Pelican’s data validation capabilities, join our experts as they discuss the “Challenges Faced in Data Validation in Large-Scale Modernization”. Join the webinar; register now!