Grid Designer's Blog

We are excited to announce that we have taken our Convergence project to the next step in the last few weeks. Last time we demonstrated how data aware routing can speed up the combination of compute grids and data grids. Since then, we have developed new grid adapters for our Convergence project and moved to the cloud.

New adapters: Sun Grid Engine, GemFire

Sun Grid Engine (SGE) is an open source batch-queuing system, developed and supported by Sun Microsystems. SGE is typically used in a server farm or high-performance computing (HPC) cluster and is responsible for accepting, scheduling, dispatching, and managing the remote and distributed execution of large numbers of standalone, parallel or interactive user jobs. It also manages and schedules the allocation of distributed resources such as processors, memory, disk space, and software licenses. We have developed an adapter that wraps SGE's Java API and enables data aware routing of tasks.

GemFire is an Enterprise Data Fabric (EDF) solution from GemStone Systems, Inc. It is a high performance, distributed in-memory-data-grid (IMDG) that offers very low latency, high resiliency, scalability and high throughput data sharing and event distribution features for high performance computing applications that need access to real-time data. We have developed a monitor component, which can query location of data from GemFire regions with partitioned schema.

Running on the cloud

Along with integrating new grids, we changed our demo application to use a cloud infrastructure. It is deployed now on Amazon EC2. Our demo now allocates servers, completes setup of cluster software (SGE and GemFire in this case) and starts the demo application server on the fly. Just a single click and few minutes later you will have a new cluster up and running.

The deployment process is straightforward and requires just a couple of scripts. First, we allocate one EC2 server and start the master node. When the master is started, it starts allocating worker nodes and sets up the grid software. When the cluster setup is complete, we start the demo control UI on the master node, and from that moment our interactive data aware routing demo is available. Each cluster has a time to live, and when its lifespan expires, all servers are returned to EC2. We are using OpenSolaris AMI, provided by Sun Microsystems, for all our servers. Deployment of GemFire is trivial --we just need to copy few jars. Installing SGE has its quirks, but after we have figured out how to do it correctly, it just works.

From a usability point of view, cloud hosting has huge benefits. Each developer can work with his own cluster, or even several clusters (e.g., comparing the effects of data aware routing between clusters of different size).

Data-aware routing demo

We have adapted our existing demo application to work with new grids and be scalable on a cloud. In our demo application we are simulating a financial application. We have a set of trade objects, which are loaded into a partitioned GemFire region. 50k objects are placed on each server (size of each object is about 2Kb). Those trades belong to different equal-sized books (5K trades in each book). The job here is to evaluate all books using Sun Grid Engine to distribute work across cluster nodes. This means that a job consists of 10*(number of servers) tasks and each task should fetch 5K trades (about 10MB of data) from the data grid and perform some calculation over them. For simplicity, we just sum the IDs of the trades and return the result.

Unlike DataSynapse GridServer, SGE is a process-oriented computational grid. Each task in SGE is an operating system process, and execution time includes all JVM startup and class loading overheads. Usually tasks are long enough, and process start up time does not make much difference, but in the case of an interactive demo we should find a balance. We cannot make the user wait half an hour to see the first result, but if we make tasks too short, the effect of data aware routing will not be visible due to JVM startup overheads.

The demo runs a constant flow of jobs and constantly measures job completion latency and task completion latency. To illustrate data aware routing advantages we introduced 3 different scheduling modes. “Data aware” mode is where we perform data aware routing, ensuring local space access for the engine. “Neutral” mode is where we do unguided DS scheduling as SGE sees fit. “Anti data aware” mode is where we deliberately violate data awareness and ensure network space access. The user can change the task scheduling mode on the fly and see the impact of the scheduling mode on performance.

The diagram on the left shows the completion time of the job and small gray bars show cumulative task execution time on each server in cluster. Gray bars do not include JVM start up time, so cumulative execution time on each server is considerably smaller then job execution time.

The diagram on the right shows average task completion latency (without JVM startup time), so you can see effect of data aware routing on task and job level.

Labels: cloud, convergence, data aware routing, gemfire, grid computing, Sun Grid Engine, ~Alexey Ragozin