Grid Designer's Blog

Remember the legendary Fallout 2 game? One of its cool features was perks. You could increase the number of action points, strength, critical strike percentage, etc. One of the most useful perks for those who wanted to carry as much as possible was the perk called Pack Rat.



But how does it relate to Data Grids? The answer is very simple: it is possible to be more efficient at storing the data in the data grid by "packing" the objects. That's the goal of the PackRat project, released yesterday on OpenSpaces.org. The initial idea came out of a customer project with TuneWiki, a lyrics distribution site developed with GigaSpaces XAP 6.0 where it was important to pack as many lyrics as possible in a space without noticeable loss of search and retrieval performance.

Let's say we want to develop a hypothetical lyrics distribution service. First, we need a simple mechanism that packs together all entities related to a single lyric:

import org.openspaces.packrat.annotations.*

@CompressedClass(type = "entry")
public class Lyrics {
  @IndexField
  private Integer id;
  @IndexField
  private String title;
  @IndexField
  private String artist;
  @CompressedField
  private Integer version;
  @CompressedField
  private String language;
  @CompressedField
  private String comment;
  @CompressedField
  private String album;
  @CompressedField
  private String year;
  @CompressedField
  private String genre;
  @CompressedField
  private String lyrics;

  // other stuff
}

Let's say a song can be searched by artist and title (kind of a composite key). This means that all other non-searchable fields (like our lyrics field) can be stored in compressed form in a binary field and unpacked on demand. The definition for such entry looks like this:

public class LyricsPacked implement Entry{
  private Integer id;
  private String title;
  private String artist;
  private byte[] binary;

  // other stuff
}

To pack and unpack entries we need to create an instance of Packer. This class implements Entry interface and has a method pack, which packs a given object. This is how you may write a packed entry to a space:

Packer packer = new Packer();
IJSpace space = (IJSpace) SpaceFinder.find("jini://*/*/mySpace");

Lyrics lyric = new Lyrics();
// assign necessary fields

space.write(packer.pack(lyric), null, Lease.FOREVER);

At some point in time, you'll need to search the lyrics that are stored packed in a space. This code snippet shows how you'd do it using
Packer.packForTemplate(Object template), which packs the template before searching for it:

Lyrics templateLyrics = new Lyrics();
Entry[] entries = space.readMultiple(packer.packForTemplate(templateLyrics), null, 1000);

for (Entry entry:entries) {
  Lyrics lyrics = packer.unpack(entry);
  // process unpacked lyrics
}

For more examples, check the project documentation page.

Update: PackRat was submitted to the OpenSpaces Developer Challenge. It has won the Early Bird Draw.

Labels: data grid, gigaspaces, grid computing, grid consulting, openspaces.org, PackRat, ~Alexander Kusnetsov