History Events Database (Orange, 2013-2019)
Info
- Employer Orange Romania
- Project type Software development
- Period 2013-2019
- My Role Developer, promoted to Lead Developer
- Technologies C++, PL/SQL, Java, Perl, Shell
- Architecture Master-slave pattern
- Keywords Performance, Fine tunning, Scalability, Billions of events per day, Replication, Disaster recovery
- Skills developed Abstract algebra, Relational algebra, Object oriented programming, Oracle fine-tuning, 3G, 4G
Description
The main purpose of History Events Database (HEDB) is to collect and store all the (raw) network events in an optimal way so that they can be accessed by other clients later. The number of events to be stored is huge (tens of billions per day), set withing different time frames (sometimes even several years) and dependent on the type of the event and on business or legal requirements.
The consumers of these sets of data are diverse and each of them has different purposes and performance requirements:
- the rating, charging, billing, and invoicing platforms, which use data to ultimately charge users for their network traffic
- the CRM & online platforms, which use data to expose to users their events history
- internal analytics platforms, which measure the network activity and generate usage forecasts
- archiving and audit platforms
Initially, the HEDB ecosystem consisted of only 2 databases only, which were enough considering the requirements at that time. But with the advent of new technologies like 4G and ultimately 5G, the situation escalated quickly and we had to find new solutions to scale up.
In order to achieve this I’ve created the data model based on Oracle DBMS by using a dynamic algorithm of rotating tablespaces to decide the storage allocation for each category of input data. For example, the older or “less important” records, will be assigned or moved to slower storages. The tables are also carefully partitioned/sub-partitioned into ranges or time frames in order to avoid full table scans and reduce the cost of the queries.
The parameters of each table are individually customised keeping in mind the data input format when deciding the parameters of compression, extent size, indexes or other constraints. The data model was also designed to allow for the replication and data restoration in case of disasters.
Being a very large database (a collection of databases actually) with a dynamic data model in continuous development, the system requires a very well prepared and maintained documentation which was also listed as one of my responsibilities.
It was a very complex project, similar to a long journey, on which I worked on for almost 6 years in order to fully develop this data model, to progressively migrate data from the older model to this one and to ensure transparency for consumers.
I initially started my work on this project as a C++ & SQL developer and after approx 2 years I was promoted to lead developer. Following this, in order to accommodate new technical challenges in the 4G era, I started rewriting the logic of the ecosystem entirely using Java (Spring), Perl, Shell and PL/SQL.
During this experience I learned a lot about databases, I improved my knowledge on telecom networks and I developed and applied my abstract & relational algebra skills.
Team
While working on this project I collaborated with an amazingly creative and dedicated team. Please reach out to the following team members for further references:
- Daniela Niculae – Billing Enablers and Convergent Services Manager @Orange Romania
- Adrian Manole – Software Development Engineer @Orange Services
- Delia Ioana Lepadatu – Software Development Engineer @Orange Services
Work with me
I am an experienced engineer when it comes to designing large and scalable databases and data models for enterprise applications. During my career as a software engineer, working with databases was an important part of my activity. I have always enjoyed studying different technologies and paradigms and I am always looking forward to new challenges.
If your business needs a reliable database model or if you need a piece of advice on a complex database problem, please feel free to contact me anytime.