VMware and Grid Computing

[Vitaly91]

Guys,

I am curious to find out your thoughts on VMware and Grid Computing (Utility Computing). For example, I know a company that basically "rents" their systems to run some financial scenarios for their clients. What they basically do is setup a grid of servers to run the calculations for the clients. Now I was thinking about this and, to me, one of the things that can be done for this company using VMware is to speed up provisioning by moving all these servers onto several ESX boxes and creating a template to provision servers on a "need" basis.

However, is there anything else that escapes my mind that could be done using VMware for them? Is it possible to create a grid itself by pulling together into a big resource pool several ESX boxes?

Unfortunately, I am not very familiar with Grid/Utility computing area and maybe I just don't realize some of the intricate details of the technology.

Any thoughts on this?

Thanks in advance!

[mreferre · Accepted solution]

This is an interesting matter.

The way I see this is that Grid Computing has always tried to work through the introduction of API's so that you could develop and Grid-enabled application and being able to run it "on the Grid" (i.e. a bunch of servers that exposes these Grid API's and that can be rented / shared etc etc).

The generic idea is cool but it has two main drawbacks in my opinion:

\- you have to re-develop your applications against these Grid API's

\- the world should agree on these Grid API's standard (and you can imagine how difficult this is).

Virtualization, in my opinion brings in new alternative to implement this same concept but in a much easier way. You could have a "grid of VI3" (or whatever virtual infrustructure technology you want ... MS, Xen, KVM, etc etc) and be able to run your workloads (i.e. virtual machines... or better appliances) without worring too much about where it is running. You are just renting part of that infrastructure for the time required.

This solves the two issues above:

\- you can bundle your standard Win/linux/Solaris/etc etc application into a virtual machine without mofifying a single bit and run it anywhere.

\- although we would all like to see a single disk format among all the technologies at this point moving a workload from virtualization technology A to virtualization technology B is going to be as easy as doing a V2V (not a big issue).

I am going to touch (VERY BRIEFLY THOUGH) on this concept here:

http://it20.info/blogs/main/archive/2007/08/09/40.aspx

This single topic (Grid etc) would require an entire session in my opinion on its own.

I'd be interested to continue this discussion with you either on this thread or off-line.

Massimo.

[Vitaly91]

Ciao, Massimo!

Sorry for such a delayed reply. It has been a crazy week for me.

Here are my thoughts on using VMware in/for Grid Computing:

I think, the issue is that VMware could be an "overkill" for Grid Computing. It may sound odd, but the reason I say this is that it is cheaper for a company to just buy a bunch of low end servers and run their Grid on it instead of buying several big ESX boxes. If you buy a bunch of cheap old servers that have enough "juice" in them to run the app you need to run, why spend money on a bigger system? If one of these cheap boxes blows up, you replace it (again, very cheap). However, if one of the bigger ones blows up, now your cost is much higher. Plus, by having low end servers and "rotating" them out to introduce the "next cheap server", you are always running on decent hardware that is cheap and easily replaced. If you commit money up front by buying a high-end ESX box, in 3 years that box will not be high-end but only mediocre. So now you have to go though the pain of red tape and proving to your finance guys that you NEED to buy a new high-end box.

Now, you can argue that VMware also offers you a simplified management. Yes, but at a higher cost than the cost of a person who just images a bunch of disks ahead of time and when a disk goes bad on one of those cheap boxes, he just swaps it. Also, you can manage building all these cheap boxes with a variety of other inexpensive software packages that operate similar to Altiris or VissionApp.

So my question then would be, why VMware? How or where would it add value to a Grid Computing provider?

Thank you very much for reply, Massimo!

[AMcCreath]

Hi Guys,

This will undoubtedly be a future move for VMware, how to make a "service" run across boxes, not just across processors, 'LockStep' is one step nearer, in that it increases application availability across hosts, but the problem remains, how do you split/ slice process cycles across hosts? It just is not possible from an infrastructure perspective yet....

Well here's a funky idea, it's called mainframe.... now how do we get mainframe access in a low(er) cost x86 environment?

I have customers chomping on the bit to have this functionality, but i think it is at least 18months/2yrs away as things have to change at the hardware level first.

If you want to read more of my comments then follow Massimo's link it IT20, I'll beposting up there when the thread gets going

Ciao,

Andy

[mreferre]

I think that your argument is an oversimplification of a much more complex matter.

>......at a higher cost than the cost of a person who just images a bunch of

>disks ahead of time and when a disk goes bad on one of those cheap

>boxes, he just swaps it

Well this would be like trying to solve the global warming issue suggesting that all people on earth use a bicycle for local transfers and trains for long distances. Yes in theory this would work just fine but I guess it is an oversimplification of a much more complex issue ..... just like the x86 servers management can't be oversimplified like you tried to do.

First virtualization (and not necessarily only VI3) brings in a very powerful concept that is .... decoupling the application from the hw. Second you can implement this VI3 farm/grid using whatever servers you want .... from the cheapest white-boxes around all the way up to 16-socket monsters.

I wouldn't stick on the definition of virtualization = big boxes ...... this is an old legacy bound to the concept of consolidation (which is still important in my opinion). We have just scracthed the surface of what virtualization can do.

Massimo.

[mreferre]

>Well here's a funky idea, it's called mainframe.... now how do we get

>mainframe access in a low(er) cost x86 environment?

>I have customers chomping on the bit to have this functionality, but i

>think it is at least 18months/2yrs away as things have to change at the

>hardware level first.

Andrew I think we will get there ... the only difference is that in the mainframe space you can rely on a single server .... here you cannot (no matter what) so the x86 mainframe will most likely look like a "bunch" of x86 systems tied together to form a single resource pool that allows you for concurrent maintenance, fault tolerance etc etc etc

Massimo.