How to replatform Endeca rules to Solr
Apr 07, 2020 • 8 min read
Apr 07, 2020 • 8 min read
The convergence of in-store and online retail operations in recent years has put significant new demands on the management of pricing, offers, and promotions as compared to pre-omnichannel times. The outdated, simplistic approaches to determine the price of products and promotional offerings are still prevalent in the industry, leading to loss in revenue, market share, and margins.
Some of the issues may be traced to the use of old products like Oracle ATG as the core pricing system, suffering from a lack of critical capabilities such as predictive analytics, machine learning, real-time modeling and personalization, to mention a few. As a result, the business cannot accurately predict the results of any specific campaign, optimize discounts, or identify lost margins when discounts combine to dip the sales price below the cost.
We have had the privilege of working with a number of large retailers on the design and implementation of these next-generation pricing engines, which are built with open source technologies and run on the cloud. In this blog post, we will describe our approach to building microservices architecture for a modern pricing, offers, and promotion engine learned over several large-scale implementations with the leading omnichannel retailers.
Different retailers have different business models when it comes to pricing. Some may focus on luxury products, some may position themselves as having an everyday low price, and others may have an expensive list price, but provide a lot of discounts, promotions, and offers to entice shoppers. While the specific mechanisms of customer engagement for those types of retailers may be different, all of them have the goal to offer the right product to the right customer at the ideal price point. Matching the product with a customer via personalized search is something that we touched on in this article. Understanding customers by collecting and analyzing data is a topic for a different discussion and requires many systems, collectively called customer 360, to work well in concert. Matching the customer with price, and more specifically, managing various components of prices, is going to be our main focus in this post. The reason we are doing an examination about prices, offers, and promotions in a single discussion is that all of them are components that contribute to the price that a customer pays for products in a shopping cart. This can be seen in the infographic below:
During the shopping experience, a customer gets exposed to prices, offers, and promotions in different areas. Product discovery starts with product search and browse, and sometimes even earlier, in external search engines via SEO, or emails via marketing campaigns. Then, when building a shopping cart, a customer needs both up-to-date and correct information about the total cost of the cart, including base prices, discounts, offers, promotions, coupons, taxes, etc. In all cases, it is crucial to show accurate information and calculate the total cost in real-time, since latency in a shopping cart is especially important due to the consumer frustration it causes.
Keeping that in mind, let’s review the major components that make up the total value of a shopping cart:
From the customer perspective, the pricing, offers, and promotions engine should provide a consistent and fast experience by correctly calculating the total value of a shopping cart and an order. From the merchandising perspective, besides offering a rich set of offers and promotions, and being able to configure flexible business rules, it is very important to have a single authoring interface where all prices, offers, and promotions can be set for all channels and stores.
The non-functional requirements for the system are typically very strict, including:
Keeping in mind the requirements above, the conceptual architecture of the service becomes clear:
This architecture consists of several major components: the core platform, the analytics and machine learning to augment business decisions, and the clients of the system. We assume that the core platform is omnichannel and the client services are also omnichannel. These consist of the wallet that may require information about available offers, the search & browse services that show pricing information during product discovery and export it for external search engines, and most importantly, the cart & checkout services that use the core platform to calculate the total value of the order. We will get to the analytics and machine learning components later in the article. For now, let’s zoom into the core platform.
The core platform may consist of several microservices, and the scale of each microservice is often dictated by the specific retailer’s environment and the complexity of the business logic. Major components, however, stay the same. The core platform needs to provide an authoring web user interface to the merchandising managers, a transactional service for downstream components, and integration services, including batch and in-stream synchronization. The architecture of such a system can be seen in this infographic:
In order to fulfill strict throughput and latency requirements, we recommend implementing a caching layer with a proven and high-performance technology, such as Apache Ignite. The bonus of this approach is that it can be used as a side cache or run calculations inside. When the caching and computation fabric is implemented correctly, the choice of a database is less important. Cassandra may be the natural choice for cloud deployment, but to support transactional capabilities and convenient SQL semantics, MySQL or PostreSQL are better. In many cases, even databases provided and hosted by cloud vendors may be a good option to save on deployment and configuration effort, such as Amazon RDS, Google SQL, Google Spanner, Azure SQL, or Azure Cosmos.
One of the biggest challenges in implementing omnichannel pricing, offers, and promotions is ensuring that the in-store experience is consistent with the online experience, and that the merchandising team can use one user interface to configure information across channels and in all of their stores. In the best case scenario, when a retailer doesn’t have many physical stores, or when a stores’ connectivity to the Internet is very good, the simple method may work. In this case, the services can be deployed in the cloud, and every time a customer needs to check out in the store, the point of sale or kiosk will call the central cloud instance of the service.
Unfortunately, not all retailers can always provide a fast and reliable Internet connection from stores. Even if basic connectivity for payments may work, the bandwidth, latency, and stability may be too poor for advanced services, such as pricing, offers, promotions, and an updated catalog. In this case, we recommend deploying local instances of the system in stores. That way, availability of up-to-date data and low latency is guaranteed. Store deployment comes with its own set of challenges, but with the proper implementation, most of them can be successfully mitigated. A reference architecture of this system is shown below:
While it may seem unnecessary in practice to support the store-level deployment of pricing services, most of the infrastructure that needs to be implemented is required for deployment of store components anyways. These components include point of sale, inventory and order management, and other customer facing and back-office systems. So, the base infrastructure and deployment process will be reused across all applications and services that need to be deployed in the store. Frankly, deploying and operating store applications is a separate interesting topic, and we hope to discuss it in greater detail in a future blog post.
During our past projects with large retailers, we have accumulated a number of techniques, tools, and processes to help optimize pricing systems. These range from foundational approaches, to deployment and monitoring, to utilization of advanced analytics including AI and ML for troubleshooting, identification, and resolution of issues.
Promotion-driven retailers typically have a large number of offers and promotions configured in the system at any given time, and this information changes frequently. At some point in time, the amount of offers and promotions may become too large to efficiently manage for a few merchandising managers. As can be seen in the infographic below, this traditional setup also leads to a reactionary approach rather than one that can respond in real-time:
In the best case scenario, the traditional approach with a handful of merchandising managers leads to an increased effort. Unfortunately, in our experience, it may also lead to lost margins and revenue due to excessive promotions. The challenges typically come from two sources:
In both cases, analytics and machine learning may find overlapping and unintentionally deep discounts. Ideally, business users operate on the level of business objectives and constraints, and the machine learning is implemented to generate granular decisions, such as creating individual promotions and offers for specific customers, as seen below:
A full-fledged algorithmic platform can cover the majority of business decisions. We won’t cover all of the possible use cases in this article, but if you are interested in learning more, we recommend taking a look at our book on this subject. While such implementation on a full scale may take significant effort, the first step is to implement a safeguard that can analyze combinations of promotions and offers, and alert business users when discounts are too deep. The threshold for alerting can be set either manually for various channels, regions, or product categories, or analyzed based on historical demand. In our experience, even such a basic implementation may save millions of dollars in margins.
Another small step towards an algorithmic approach is implementation of real-time offers for specific use cases. One of these use cases is that of an abandoned shopping cart. Abandoned shopping carts are one of the biggest problems in retail, and reducing abandonment rates may increase revenues significantly. In our experience, even a simple implementation of real-time offers to customers with a high propensity to abandon their carts may decrease the cart abandonment rate by several percentage points. The exact strategies of what offers to give, how to give them, and when to give them may vary. If behavior analysis shows the customer has a high propensity to abandon the cart, the offers may be given while the customer is still on the site, to prevent this from occurring. Or, if analysis shows that there is a high chance that the customer returns to a previously abandoned cart when given a certain incentive, the offer may be sent via email after they’ve already abandoned it. While the results of such offers may be positive, we advise implementing machine learning models carefully to avoid giving offers to customers who already have a high probability of conversion, as doing so would reduce margins.
Implementation of a modern pricing, offers, and promotions service enables a rich, omnichannel experience for customers. Equally important, it provides omnichannel management experience for merchandising managers and business users, increasing their efficiency, and leading to higher revenue and better margins. Conversion rates should increase, and cart abandonment rates should decrease, giving proof of pricing being at a better place for consumers.
Initial implementation and replatforming of base services from ATG allows the closing of immediate gaps, such as omnichannel experience, richness, and flexibility of business rules required by a specific retailer. In the long run, such replatforming opens opportunities for further improvements, such as embedding advanced analytics, machine learning, and an algorithmic approach. These improvements may improve the quality of prices, offers, and promotions, and improve the bottom line by an even greater margin.
The solution we laid out should help with creating a more modern omnichannel pricing engine for any retailer, as it uses open source technologies and a microservices approach. Many retailers move off ATG because they want consistent and highly personalized prices and promotions, which are not otherwise provided.
If you’d like to learn more about ATG replatforming, be sure to check out our upcoming posts in this series. In future articles, we will discuss building other services of the digital customer experience platform, including a customer profile, cart, and checkout. We will also touch on how to implement a product information management system, content management, and customer 360 capability. If you want to learn more, give us a call or leave us a comment!