How EXL helped a leading insurance provider migrate SAS to PySpark with 50% efficiency gains
The insurance industry is rapidly digitizing, with companies constantly innovating to leverage data for actionable insights. A prominent insurance provider embarked on the ambitious project of modernizing their legacy systems by transitioning their data warehouse to Databricks. However, the intricate task of migrating complex, legacy SAS codes embedded with Oracle SQL into an advanced PySpark environment presented significant challenges. That’s when they turned to EXL and their AI-driven Code Harbor™ technology.
Challenge
The insurance provider faced multiple roadblocks in their modernization initiative:
- Complex legacy environment: A significant portion of their SAS code contained embedded Oracle SQL queries, making the migration process highly intricate.
- Efficiency challenges: Manual efforts to refactor and transition tens of thousands of lines of SAS code to PySpark were inefficient and error-prone.
- Data governance and documentation: Limited documentation of the existing legacy codebase and lack of a structured data lineage created roadblocks for debugging and validation.
- Resource constraints: The migration required expertise in multiple coding languages, which placed additional pressure on their internal development teams.
- Testing overheads: Ensuring accuracy in data pipelines during user acceptance testing (UAT) and reconciliation required extensive manual effort.
Together, these factors posed a risk to the project's timeline and the business’s ability to achieve a seamless digital transformation.
Solution
EXL deployed their proprietary Code Harbor multi-agent solution to tackle the client’s challenges head-on. Code Harbor combines advanced AI technologies with structured methodologies to deliver an efficient and scalable migration experience.
Key steps in implementation:
1. Automated code migration
- EXL’s Code Harbor migrated complex SAS codes (including embedded Oracle SQL queries) to PySpark with over 64 data tables accurately converted. By leveraging large language models (LLMs) and optimized algorithms, EXL ensured all transformations followed best practices for scalability and performance.
2. Data cataloging and lineage generation
- The solution automatically generated a comprehensive data catalog along with schema and lineage mapping to reinforce governance and ensure compliance with BI center of excellence patterns.
3. Synthetic data testing
- Extensive testing was conducted in EXL’s environment using synthetic data to validate the migrated code for accuracy and efficiency. Code Harbor’s automated validation agents significantly reduced error rates and debugging time.
4. End-to-end support for UAT
- EXL provided continuous support to implement UAT in the client’s environment, ensuring smooth deployment and mitigation of potential risks. Collaboration between EXL’s experts and the client ensured precise reconciliation of legacy outputs with the newly implemented PySpark codebase.
5. Flexibility and documentation
- Code Harbor™ offered intuitive documentation in natural language, enabling easier understanding of the migrated code for future audits and maintenance.
Results
The collaboration between EXL and the insurance provider yielded exceptional outcomes:
- 50% efficiency improvement in code migration
Automated migration reduced manual effort and significantly decreased the time required to transition SAS code to PySpark.
- 50% reduction in UAT and reconciliation efforts
EXL’s automated validation agents streamlined testing processes, ensuring accuracy while cutting down on labor-intensive manual reconciliation.
- Enhanced data governance
The implementation adhered to BI center of excellence technical frameworks, providing standardized documentation and improved audit readiness.
- Productivity gains
By minimizing resource-intensive debugging and testing processes, the client’s teams could focus on higher-value tasks, realizing operational efficiency.