How to replatform Endeca rules to Solr
Apr 07, 2020 • 8 min read
Apr 07, 2020 • 8 min read
If you are an enterprise omni-channel retailer with a significant and growing online presence, it is likely that Oracle ATG has been your platform of choice for many years. You have probably invested thousands of hours and tens of millions of dollars in the customization, integration and tuning of your Oracle ATG implementations. No system is more mission-critical to you than your core e-commerce platform, which reliably generates billions of dollars in high-margin sales. The last thing you want to do is to scrap all this investment and move to another platform.
But you have no choice.For large companies like yours Oracle has lost its way. It has fallen behind the technology curve and failed to provide a compelling roadmap to address the new demands of digital retail. ATG was originally an innovative, open platform built by some of the brightest engineers in the industry, but over the years Oracle has turned ATG into a massive, rigid “black box” that’s limited in functionality, painfully hard to change, slow to deploy, expensive to maintain, and darn near impossible to innovate with.
As you know from first-hand experience, the problem with your current platform is three-fold:
Leading large retailers that have already started the move away from ATG cite the following common business reasons as the most pressing drivers of their efforts to build next-generation digital services outside of ATG. If you already know the reasons to get off of ATG and are interested in the best replatforming strategies you can check out this helpful guide here.
Catalog services in ATG don’t scale to omni-channel workloads.
The product catalog in ATG is designed to support an online store as a stand-alone channel, and is poorly-suited to act as a unified omni-channel catalog that manages product selection and availability across all of a retailer’s offline and online stores. ATG keeps its catalog in a SQL database, which results in increased hardware demands, especially at peak times — and limits its scalability. Implementing an omni-channel catalog requires investing in the technology that can serve a unified selection of products across physical stores ("infinite aisle"), web, mobile, and social channels. The optimal software stack to support such workloads consists of a combination of horizontally scalable, highly available NoSQL datastores, such as Apache Cassandra running a high-throughput messaging system like Apache Kafka.
Consumers continue to demand faster delivery and lower prices. The ability to sell an order via the web and fulfill it in the store has become a must-have feature for the omni-channel retailer. However, this requires near-real-time stock availability synchronization between the stores and the web platform. Implementing this synchronization on top of ATG puts excessive load on the underlying database, limiting the synchronization frequency and slowing down the online product navigation and search capabilities. By moving the catalog, navigation, and inventory functions out of ATG and onto a modern platform, retailers unlock access to (close to) real-time stock availability synchronization, as well as a scalable foundation to optimize sourcing and fulfillment logic. The optimal software stack for the inventory service is usually based on in-memory compute fabrics like Apache Ignite and Java.
Effective product pricing that consistently delivers the highest margins is a complex combination of base price, current promotions, personal offers, and other considerations such as inventory availability — all of which need to be applied in a specific order in real time while the customer is browsing the product or ready to check out. ATG tries to perform pricing calculations within a SQL database. This severely limits the complexity of the calculations that can be performed, which in turn either limits the ability to roll out compelling promotions or can lead to financial losses when the promotions combine in an unexpected way (the "coupon problem"). Moving the pricing calculation to in-memory compute fabric like Apache Ignite allows sophisticated pricing rules and optimizations to be performed in real-time.
No other single feature affects revenue and convergence for online shoppers more than search technology. Traditionally, ATG users would purchase Endeca, Oracle’s companion search product, which became widely popular for its outstanding merchandising features. However, over the past few years, Endeca has failed to keep up with the times. It doesn’t provide advanced search features such as understanding natural language, integrating search with dynamic facets, finding the best product match in the case of incomplete queries, manipulating relevance ranks based on personalized information about the user or utilizing machine learning to offer visual search. Open source search technologies, such as Apache Solr, combined with other big data technologies and machine learning frameworks like TensorFlow, have leapfrogged Endeca as a foundation for advanced search services.
On the technology management and cost efficiency side, there are also ample drivers to get off monolithic, closed, aged ATG — that must run on your premises in static, big iron infrastructure — and replace it with a collection of lightweight, composite, dynamically scalable services built with open source components running on self-optimizing hybrid clouds. Here are some of the most obvious problems with ATG deployments that are pushing retailers towards alternatives:
With ATG’s traditional on-premise deployment architecture, retailers are required to stockpile hardware in anticipation of the Black Friday peak season. Often, the capacity which is required to handle peak season traffic is five to ten times larger than the steady state requirements, leading to the infrastructure being 10% utilized 80% of the time. Offloading the most load-sensitive components, including front end and search, to a public cloud delivers major cost savings. A side benefit is a better user experience due to decreased response time from a myriad of possible deployment options and points of delivery. The technology stack used to deliver such improvements includes an infrastructure-as-a-service cloud provider such as AWS, Google cloud or Azure; Docker and open source, noSQL data stores such as Apache Cassandra, Apache Lucene / Solr or Apache Ignite.
Shortening development and test cycles is mandatory to achieve the time-to-market targets critical in digital retail. The sad state of affairs in ATG deployments is that development, testing speed, and agility are severely limited by the availability of properly configured and tested environments. Oracle does not support on-demand provisioning of ATG environments. The software stack and proprietary build process make it near-impossible to achieve fully-automated, safe one-click deployment of new changes. Proprietary configuration management also makes it nearly impossible to push the latest changes through a DevOps pipeline in an automated fashion. Customers have to move to a modern microservices architectures to regain the ability to innovate, experiment, and evolve rapidly with a fast rollout of new services.
It’s 2017. For most retailers, the “to cloud or not to cloud” debate is over. They have adopted a “cloud first” approach and are now working diligently to rationalize their migration to the cloud with all their major applications. Oracle simply doesn’t support any cloud option apart from Oracle cloud, and even that is not a real option for ATG/Endeca users. ATG cannot be deployed on AWS, Google Cloud or Azure. It must stay on-premises forever, and that alone is putting it firmly and decisively into a “legacy” bucket, right next to your mainframe systems.
We often talk to CTOs of companies like yours, that are running on ATG today. Nearly without exception, they share the struggles described above. As a group, they are looking for a compelling alternative to ATG. Yet replacing Oracle ATG in one giant replatforming effort is not practical. They are looking for specific blueprints that enable them to meet growing business demands incrementally while also moving in the right architectural direction.
Over the last two years, Grid Dynamics had the good fortune to work with a number of large retailer customers — both department stores and brands — that are successfully making the transition from Oracle ATG to the cloud, open source and microservices. In the process, we have developed an extensive portfolio of blueprints related to various facets of replatforming, particularly in the areas of catalog & search services, dynamic pricing, real time inventory, responsive/adaptive UI, personalization and machine learning, DevOps, microservices and the cloud.
We also have the depth and breadth of expertise to help with all facets of the process — from business consulting to rationalizing “why?” and “why now?”; to architecture and product management; to choosing the appropriate architecture stacks, and correctly estimating timelines and the resources needed to launch the new services; to engineering resources to help you scale the development, testing and DevOps activities, or execute any portion of the migration.