Full-text & faceted search over In-Memory Data Grids
Modern in-memory data grid (IMDG) solutions provide different facilities for execution of queries over whole stored data sets with different levels of sophistication. Oracle Coherence provides Query facilities (one time full scan and continuous querying with Cost-Based-Optimized). GigaSpaces has a JDBC Query interface with the ability to use hash and B-Tree indexes. Those solutions work quite well for many problem areas. However, for heavy loads and complex multi-criteria queries those facilities can quickly become a bottleneck.
There is a class of workloads that produce high loads with complex queries on IMDGs. Retail companies that use IMDGs for their item catalogs are a good example. Those catalogs are hit by diverse stream of multi-criteria queries. A typical query you may see there looks like:
Fortunately, the Compass Framework allows you to process such queries effectively. You can build inverse indexes with Apache Lucene and store them on a grid. This capability is based on the very modular design of the Lucene framework. All index I/O operations are well-hidden by the abstraction of FileDirectory.
For now Compass provides implementations for Coherence, GigaSpaces and Terracotta, introducing an unprecedented ability to build a vertical search solution on top of In-Memory Data Grids.
In addition, Compass has a sophisticated object-to-document mapping system that allows you to make stored objects searchable just by adding Java annotations or XML mapping files. Mappings can also be built in runtime.
However, despite its great codebase, Compass documentation is pretty sparse. It may take significant time to dive into the code and docs to get what you want. But the results will overcome all your expectations. Search engine performance on top of data grids easily overcomes any old-generation search technology.
There is a class of workloads that produce high loads with complex queries on IMDGs. Retail companies that use IMDGs for their item catalogs are a good example. Those catalogs are hit by diverse stream of multi-criteria queries. A typical query you may see there looks like:
give me cell phones with MP3 support, Java and in red color.
Fortunately, the Compass Framework allows you to process such queries effectively. You can build inverse indexes with Apache Lucene and store them on a grid. This capability is based on the very modular design of the Lucene framework. All index I/O operations are well-hidden by the abstraction of FileDirectory.
For now Compass provides implementations for Coherence, GigaSpaces and Terracotta, introducing an unprecedented ability to build a vertical search solution on top of In-Memory Data Grids.
In addition, Compass has a sophisticated object-to-document mapping system that allows you to make stored objects searchable just by adding Java annotations or XML mapping files. Mappings can also be built in runtime.
However, despite its great codebase, Compass documentation is pretty sparse. It may take significant time to dive into the code and docs to get what you want. But the results will overcome all your expectations. Search engine performance on top of data grids easily overcomes any old-generation search technology.
Enjoy!
Labels: data grid, distributed cache, gigaspaces, grid computing, ~Alexey Kharlamov
at
10:39 PM
2 Comments:
Great article...
Is there an code example available?
Yes, we have functioning search code. However, it is property of our customer. But I can answer specific questions about the use-case.
Post a Comment
Subscribe to Post Comments [Atom]
Links to this post:
Create a Link
<< Home