Data Migration Project (CN Group & Flatirons, 2021-2023)
Info
- Employer Flatirons (CN Group contractor)
- Project type Software development
- Period 2021-2023
- My Role Technical Lead
- Engineering team size 25+
- Technologies Java, AWS, Kubernetes, Postgres, Linux, Windows, Docker
- Architecture Client-server, Master-slave patterns
- Keywords High availability, Data migration, Data integrity
- Skills developed Technical leadership, AWS knowledge
Description
The primary goal of this project was to adapt and migrate the existing platform from an on-premise Linux environment to AWS. This entailed adapting the system source code to ensure cloud compliance while maintaining support for the on-premise architecture.
The team also executed a protocol to transfer large volumes of data and metadata, automating the analysis of differences between the old and new instances. This intense data integrity process presented extensive technical challenges, including:
- Migrating hundreds of billions of objects (database and filesystems) across various environments and architectures.
- Developing mechanisms to validate data integrity.
- Migrating thousands of users to the new system, dynamically adapting the permission matrix while preserving the same level of user access.
The project team consisted of over 25 engineers from an international team spread across more than eight time zones (map1, map2), working for over 24 months.
As the Technical Lead on the project, key responsibilities included:
- Solution design and architecture.
- Coordination of technical activities for developers, DevOps engineers, and QA engineers.
- Client-facing technical representation; acting as the client's representative in front of their clients (two of the top three largest airlines in the world).
This project significantly enhanced communication and leadership skills, improved timeline estimations for large-scale tasks, and fostered the adaptation of technical solutions to dynamic business needs.
Data Migration & Architecture Transformation Case Study
Tech Stack
- Java 8/11
- SpringBoot
- JPA and Hibernate
- Maven
- SQL (Oracle/MsSQL Server/Postgres)
- Apache Tomcat
- Linux shell scripting and Windows Powershell
- Apache Solr
- AWS
- Git
- Jenkins
- Docker
- Kubernetes
- Terraform
Last Update: 2023.10.11