Grid Designer's Blog

Grid Dynamics was one of the sponsors for Incisive Media's Waters Power 2009 event that was held in New York City. Main agenda of the conference was to showcase the most up-to date developments in HPC with in-depth analysis of cloud computing, virtualization and SOA solutions, bringing the latest strategies, techniques and technologies that give optimum performance and maximum efficiency for any data center. Major HPC vendors and many wall street firms were represented at the conference

Key note speech was presented by Jeffrey Birnbaum from Merrill Lynch. It was a well presented speech that highlighted the opportunities, challenges and approaches of cloud computing for HPC. Highlights from his speech are

• Main attraction of cloud computing is to drive the cost of computing down. Google and Amazon had done a good job getting the cost pretty low. How can the enterprises do the same, or come close?
• Most cloud providers like Amazon, Google etc. are GbE Based. Everyone is moving towards 10GbE and this serves as foundation for viable clouds
• Enterprise cloud infrastructure needs a global file system. All software is installed on that file system. This makes it simple for the end users, in order to run any compute environment - just mount a file system
• In order to get global scalability - use multiple copies of the files
• Replicate the files in real-time on any update. It might be better to wait on update than deal with eventual consistency
• For better performance cache the files in regional locations
• Do not provide node-level redundancy and increase the cost of hardware, buy commodity hardware and design for failure. Route the workload from a node that failed to some available node (like what Google and Amazon do)
• Design Data Centers around PODs connected by layer-2, not layer-3.

Technologies that will change the world,

• Multi-core/multi-socket commodity compute nodes (run more VMs, writing more parallel code)
• 10GbE with iWARP or RDMA (lower latency to storage)
• Flash-based storage at 200K IOPS (totally changes how you think about the problems)

Implications of the above changes are dramatic and will affect the way we design the Data Centers and Applications

There was a Panel Discussion about "A silver lining for cloud computing". Panel included Victoria Livschitz, Founder and CEO of Grid Dynamics. Questions asked by the moderator triggered insightful discussions and sometimes contradicting answers,

• Define cloud computing? This is probably most asked question in cloud computing forums and glad to hear we are closing in on a definition. This brought up another question about what stage of technology maturity is cloud computing in. Victoria's response to this was that it is in really early stages and it will take about 5 yrs to be mature enough for enterprise adoption
  
• What applications are not suitable for cloud computing? It was acknowledged by few panel members that low latency applications may be bad candidates before cloud infrastructure matures but one speaker thought that this is up to the design of the cloud and it is possible to host low latency applications if architected well
  
• An interesting conversation that came up is when cloud will get more adoption. This is same question that keeps coming up when any new technology comes into light. Old companies with huge investments in the existing infrastructure and approaches will need to spend more time adopting while newer players will benefit from them much sooner
  
• Victoria made an insightful argument about the way applications were designed and implemented, this was about the data and its ownership. Most of the applications now are designed with data ownership as the premise but the promise of consuming and providing data as a service has many opportunities. For e.g., cloud computing is making infrastructure, software and computation as a service and having data also as a consumable entity gives the applications unbound possibilities

The thoughts about the data ownership and promise of being able to consume this as a service were echoed by few other speakers in later discussions

Ken Michellini's (from CitiHub) speech about "Keeping your feet on the ground: Unveiling the truth about cloud computing" was educational. It talked about the ROI calculations in a public cloud, 3rd party hosted and internal cloud scenarios and gave some ideas on how you can make these decisions when building your next big application. He also talked about characteristics of a good cloud application and typical challenges in building cloud applications.

There were few more Panel discussions that we attended including "Building the 21st century data center", "Preparing for Next phase in grid computing: What it takes to build the perfect data grid" and "Virtual reality: Optimizing storage, application and network virtualization" and another presentation about real life lessons learned while administering grids "Nuts and Bolts: Practical issues in grid administration"

I would like to extend congratulations to Incisive Media for a well conducted conference that let many financial industry experts brain storm and discuss the opportunities of Cloud. Great job guys!

Labels: Conference, hpc, Microsoft HPC, Waters, ~Shravan Kumar