In an interview with Gaming Eminence, Rares Rusu, Principal Developer at Paddy Power, offers a behind-the-scenes look at the technological shift driving the company’s innovation. Rares discusses overcoming the limitations of traditional relational databases, adopting Couchbase’s cloud platform, and rethinking workflows to improve agility and scalability. He also explores how the team delivers real-time data processing for millions of customers, ensures global data consistency, and handles peak traffic during major events.
Gaming Eminence: Traditional databases often come with months-long development cycles that can hinder innovation. Can you discuss specific limitations you encountered with your previous infrastructure, and how these challenges prompted the shift to Couchbase's cloud database platform?
RR) In the entertainment industry, customer expectations for engaging, creative experiences have increased significantly in recent years. To keep up with this rapid pace, we need to be able to develop new customer experiences as quickly as possible – but traditional relational databases can greatly slow things down for our development teams. With our previous database structure, development times could reach up to three months – which was simply no longer tenable.
One reason for this slowdown is that traditional, relational database modelling eventually leads to large schemas. We accumulated a lot of software and systems that deliver experiences to customers over recent years, and as these systems have grown, we’ve faced challenges in managing various datasets and introducing new integrations.
Additionally, we found our traditional databases require specialised roles, like database administrators (DBAs). As these roles become essential and we rely on a few people to shoulder a lot of the workload, projects can suffer from delays. Now, with Couchbase, our teams have full control over how they model solutions and upgrade them. Developers become the owners of the entire application state, including storage. We don’t have specialised roles like database admins with Couchbase. For example, if we need a new document or index, it’s a development task, not a database task. This spreads the knowledge across the team, so we don’t have bottlenecks slowing things down. It’s a much more democratic approach and improves overall productivity.
Gaming Eminence: Processing over 500,000 events every three minutes for a user base of more than 7 million active customers is a significant feat. What architectural or technological strategies did you implement to handle such high-volume, real-time data processing, and how has this impacted the customer experience?
RR) The customer events and inputs we consume are quite varied – everything from registrations to deposits, bets and spins. All of these events require careful modelling as the attributes of a registration event differ from a deposit event. While we do have “customer” as a core entity, that variation in how different events look is significant. This is where NoSQL and document-based models have been particularly useful, as they allow for this variation without incurring the heavy costs typically associated with schema upgrades in relational databases.
This also helps us with real-time processing. We started with the goal that any activity a customer does should be reflected in real time. For example, when a customer registers, we process that event instantly and issue a welcome offer to their account. This is just one example of prioritizing real-time data processing, and how moving away from using scripts or daily batch processes achieves a much better user experience. The new Couchbase Capella solution provides consistent response times even when the load increases, allowing us to continue to meet customer needs in real time.
Gaming Eminence: The ability to quickly prototype and adapt to new requirements is crucial in the fast-paced gaming industry. How has the adoption of a cloud-based database transformed your development workflow, and what benefits have you observed in terms of speed and innovation?
RR) Switching to a cloud-based approach has made our teams more agile. Compare this with our experience using relational databases. In the past, database modelling had to be done upfront when developing projects, and this process was typically separate from application modelling.
However, when we adopted a cloud-based solution, database modelling became more of a byproduct of our design sessions. During these sessions, the team gathers to describe the solution and identify the entities, such as customers or promotions, that need to be included in the application. Storing these entities then becomes straightforward because the team models them as documents, which is a direct and simple translation from the application to storage. This approach doesn't require the kind of complex adaptation that is often necessary with relational databases.
Working with a document-based mindset shortens the feedback loop. We're able to quickly create and adapt prototypes based on business requirements, which allows us to respond more effectively to changes.
Gaming Eminence: Providing a seamless single sign-on experience across multiple brands requires meticulous coordination. What technical hurdles did you face in ensuring both security and user convenience, and how does your current database infrastructure support these goals?
RR) One of the primary technical challenges in providing a seamless single sign-on (SSO) experience across multiple brands was managing user sessions securely between different platforms. To address this, we developed a session management system that stores relevant session data in a document-oriented database. This system allows users to navigate between various brand consoles without the need to log in separately, effectively creating a custom SSO experience.
Our database infrastructure is pivotal in supporting this functionality by enabling easy retrieval and modification of session data as users switch platforms. This not only enhances user convenience but also maintains high security standards. The flexibility of a schema-less database allows our development team to adapt swiftly to evolving requirements without compromising on scalability or security.
Additionally, we implemented cross-data center replication to ensure that the same dataset is consistently available across multiple locations. This replication capability enables us to serve traffic from different regions without sacrificing performance or data consistency, which is crucial for maintaining the reliability and security of our SSO system.
Gaming Eminence: Consistency and accessibility of data across various locations are vital for uninterrupted customer service. How does your upgraded IT infrastructure ensure global data consistency and availability, and what measures are in place to maintain this reliability under the dynamic demands of the gaming industry?
RR) Ensuring data consistency and accessibility across multiple locations is critical for uninterrupted customer service in the gaming industry. In our upgraded IT infrastructure, we've implemented cross data center replication, which aligns seamlessly with our topology and enables real-time data synchronisation between multiple data centers.
During our transition from a single heritage data center to a dual data center setup, adopting bi-directional data replication was transformative. Operating two clusters that replicate data in real-time allowed us to maintain seamless accessibility of the same dataset from different locations. This capability is essential for providing uninterrupted service, especially under dynamic demands and heavy loads. Our infrastructure efficiently handles requests without noticeable latency or data conflicts, ensuring all applications access the most up-to-date information and delivering a smooth, consistent user experience. Without such a robust replication process, meeting our business requirements would have been significantly more challenging, potentially requiring compromises on service delivery.
Flexibility and scalability are also paramount in handling the unique spikes in load we regularly experience. For instance, major horse racing events like the Cheltenham Festival and the Grand National result in intense bursts of activity due to high volumes of betting traffic. Unlike events where traffic is more evenly distributed, these spikes demand an infrastructure capable of scaling rapidly. By distributing data across multiple nodes and clusters, we're able to maintain performance and responsiveness during these peak periods. In an industry characterised by rapid changes in demand and fluctuating traffic, such scalability is crucial. Our architectural approach ensures we can adapt in real-time, preserving a reliable and consistent customer experience even when faced with the most intense loads.