Executive Summary
A government agency engaged our consultancy to recover a stalled migration project. Their enterprise data platform, built on Hadoop, had become unstable and was suffering frequent outages, placing critical datasets at risk. The agency needed to secure their data as an immediate priority, but the project had lost focus and was making little progress.
We restructured the initiative into two parallel workstreams: an immediate as-is migration to Azure Synapse Analytics to eliminate the risk of data loss, and a concurrent requirements gathering exercise for the future state. The migration was delivered at pace, adopting a lake-first design pattern that secured existing data whilst positioning the agency for future platform evolution, including potential adoption of Microsoft Fabric.
Challenge
The government agency’s enterprise data platform, built on Hadoop, had never fully met their requirements. The platform suffered from excessive complexity, poorly designed data products, and an inability to support innovative data use cases. The situation had become critical: the platform had reached end of support, system outages were increasingly frequent, and the internal team lacked the specialist skills required to maintain it.
Despite the urgency, the project to address these issues had lost its way. The team had become focused on evaluating products and gathering future-state requirements rather than prioritising the immediate risk to data integrity. Meanwhile, each outage increased the likelihood of irrecoverable data loss.
A strategic intervention was required to refocus the project on immediate risk mitigation whilst establishing a foundation for future platform development.
Solution
The project was restructured into two parallel workstreams to address both immediate and longer-term requirements.
Immediate Migration Scope
The primary workstream focused on migrating existing functionalities and datasets from the Hadoop platform to Azure Synapse Analytics, with clear objectives:
- Historic Data Migration: Establish pipelines to migrate historic data from Hadoop and secure it in an Azure Data Lake
- Code Repository and Peer Review: Establish version control and peer review processes alongside release pipelines to enable parallel testing
- Future Dataset Collection: Develop pipelines to collect future datasets in parallel with Hadoop, bypassing the legacy platform for new submissions
- Transformation and Presentation Migration: Migrate transformation and presentation code to Azure Synapse Analytics
Design Principles
Data transformations adopted a lake-first design pattern to ensure maximum flexibility for future enhancements. By persisting data to the data lake, any subsequent platform transition would be significantly simplified.
Azure Synapse Analytics was selected for its broad, scalable data management capabilities. Data processing was migrated to Spark, with integration pipelines enabling continuity of upstream processes. The presentation layer was migrated to Serverless SQL Pool, maintaining operational continuity for downstream consumers.
Technical Implementation
The migration was executed through a structured, phased implementation process designed to minimise disruption whilst maintaining delivery momentum.
Planning and Preparation
The initial phase focused on understanding the existing data landscape within the Hadoop ecosystem, identifying datasets for migration, establishing connectivity between services, enabling peer review processes within Azure DevOps, and defining the sequence of activities required for a smooth transition.
Migration Strategy
A phased migration strategy was adopted to manage complexity effectively. Data was extracted to flat files using integration pipelines that could be run repeatedly to bring content up to date as required.
Execution
Upon successfully migrating data into the data lake, the focus shifted to reconciliation to ensure accuracy and completeness. Concurrently, development efforts addressed the transfer of transformation scripts and construction of the semantic layer in Serverless SQL Pool.
Views were deployed through a CI/CD process as development completed, making them immediately available for review. This enabled continual and early feedback on any issues identified.
Challenges Overcome
Several significant challenges were encountered and resolved during the migration.
Data Management Processes: Two critical data management issues required attention. The first concerned the unconventional practice of using source control as a repository for some data sources, which was rectified by realigning data submissions with a consistent and controlled approach.
The second issue was uncontrolled self-service access that allowed end users to modify business logic, leading to inconsistencies and a lack of governance. All business logic was formally brought under central management, complemented by a structured change control process. Dedicated views were created within Serverless SQL Pool to support each variation independently, allowing continuity of reporting whilst enabling future rationalisation.
SQL Syntax Incompatibility: The discrepancy between Hive SQL syntax and the ANSI standards used in Serverless SQL Pool required a thorough function mapping exercise to ensure accurate transposition.
Business Logic Inconsistencies: Significant inconsistencies in business logic presented reconciliation challenges. The team evaluated content for suspected inconsistencies, facilitating collaborative review with data consumers. Inconsistencies were corrected in both platforms to ensure data integrity and consistency in reconciliation testing.
Results and Benefits
The migration culminated in the decommissioning of the legacy Hadoop platform, eliminating the source of instability that had prompted the engagement. Azure Synapse Analytics now serves as a right-sized data platform aligned with the agency’s needs, data profile, and usage patterns.
Key Benefits
- Risk Mitigation: The migration delivered a reliable, scalable, and flexible data management platform at pace, eliminating the risk of data loss from platform failure
- Scalable Architecture: The new architecture adapts to fluctuating demands, enabling the agency to manage data more effectively with capacity to scale resources as required
- Lake-First Design: The adoption of lake-first design principles has streamlined data management practices and positioned the agency for easier integration of future products and innovations
- Embedded Best Practices: The implementation process, underscored by a phased approach and strategic use of CI/CD processes, has embedded strong development practices within the internal team
- Improved Governance: Centralised management of business logic and structured change control processes have improved data governance and consistency
Lessons Learned
- Phased Migration: Adopting a phased approach allowed for greater control over the process, enabling the team to address challenges incrementally
- Early Semantic Layer Development: Developing the semantic layer early enabled prompt evaluation of data across all layers, with issues identified and addressed swiftly
- Regular Feedback Cycles: Incorporating early and regular feedback was crucial for aligning the migration with project objectives and maintaining delivery momentum
- CI/CD Automation: The use of CI/CD processes for deploying views enabled faster testing, integration, and feedback, underscoring the value of automation in streamlining migration tasks
Wrapping Up
This migration demonstrated the value of clear prioritisation when facing multiple technical and strategic challenges. By separating immediate risk mitigation from future-state planning, the project delivered tangible benefits at pace whilst establishing a foundation for longer-term evolution.
The agency now benefits from a stable, scalable data platform that aligns with their operational requirements. The lake-first approach has positioned them advantageously for future platform decisions, including potential adoption of Microsoft Fabric.
Future Considerations
Microsoft Fabric was announced as a unified data platform shortly after this migration was commissioned. Several features on the Fabric roadmap would need to mature before the agency could seriously consider transitioning. However, the lake-first design approach has strategically positioned them to facilitate a smoother transition should the need arise, ensuring that any future platform evolution would be significantly less disruptive than it might otherwise have been.
