Does such a processing beast exist?

[rudder]

Disclaimer: Not looking to bypass EULA - curious about something though... plus it is evil microsoft 😉

So I am familiar with render farms and have been looking at Windows HPC server and renderpal V2. Here at our office we use a program from Microsoft called Ultramap. Basically it is for processing aerial photos. And we have a freaking a lot of them. I just put a 57Tb NAS device in production just for this task.

Currently we have a license that allows us to use 20 cores. I have not checked on the cost to add more cores so this is why this is just a feeler kind of post. They may be cheap.

I am hitting the 20 core limit. I have 4 Dell optiplex 745 (Pentium D crap) and 4 proliant Dl360g4 that I took from surplus.

Anyway, these jobs still take 1-2 days of cpu time to complete. I have access to a large number of surplus PC's and servers, in a freeze locker, and I don't pay the electric bill.

Is there a product that will create a virtual cpu which is actually a processing farm of multiple machines? Where the client sees it as one core, but the host farms out the processing behind the scenes. It would have to run on Windows because the Ultramap software is Microsoft's and they only make a Windows client.

Or should I just see how much licensing is to add more cores? And yes the license is based on cores not cpus. So if I threw in an 8 core cpu in the mix that would take nearly half my license. I am checking now with our sales rep on costs for bumping up the license.. but though I would see. Maybe we could use our brainpower to develop something.

EDIT: and I know the first thing I need to do is get rid of the Pentium D's. However due to Tennessee flooding, I can not take down these machines in the near future and they have vendor hardware in them atm. When I get some breathing room I will throw these in the trash.

 

[somethingsketchy]

Sounds like your only option would be to purchase more licenses for more cores, because no matter how many machines you have with whatever combination of cores to processors, you will still have a 20 core max limit.

You "could" try running the software in a virtual machine, however you'll take a performance hit trying to virtualize more cores than you have for physical cores. Either way, your best bet would be to try and upgrade the individual processors as much as you can, to try and utilize any additional instruction sets that may be useful for Ultramap and renderpal V2.

Short of having a powerful piece of computing equipment to virtualize cores, you're limited to upgrading your licensing to use more cores.

 

[mfenn]

Is there a product that will create a virtual cpu which is actually a processing farm of multiple machines? Where the client sees it as one core, but the host farms out the processing behind the scenes. It would have to run on Windows because the Ultramap software is Microsoft's and they only make a Windows client.

There are some software packages that take a cluster of distributed memory machines and make it look like one large shared memory machines. They don't do anything about the number of CPU's though. Honestly, if someone invented this, you would know, because they would be very rich and famous. 😛

Are you sure that your software is licensed per-core and not per-socket? Most MS products that I am familiar with are on a per-socket basis. Admittedly, I am a Linux guy though. It's kind of sad that anyone would be stuck with MS's solutions for HPC tasks. 🙁🙁🙁🙁

 

[Fox5]

Upgrade the license OR upgrade the pcs. A bunch of i5/i7/xeons/opterons would probably crunch through much faster.

Also, are you sure you're even cpu limited? You could be reaching a memory limit, or a streaming limit from the hard drive.

 

[rudder]

Are you sure that your software is licensed per-core and not per-socket? Most MS products that I am familiar with are on a per-socket basis. Admittedly, I am a Linux guy though. It's kind of sad that anyone would be stuck with MS's solutions for HPC tasks. 🙁🙁🙁🙁

Microsoft product...so they are not going to make a linux anything. I checked on the license and it is $40,000 for 20 more cores. And yes, it is cores. Throw in that quad and you have used up 4. Oracle is expensive as crap but at least they license per socket.

Luckily we have a microsoft guy on site so he is getting me in touch with the right people. I will trying to find out how Server 2008 HPC would handle this. Maybe it is cheaper to buy the license for that. I will probably be the first one asking this so they will probably have to rethink their pricing before I get an answer.

And Fox5, it could be other components slowing down the process, but mostly the CPU. On the pentium D's we use a dedicated SATA drive, 3gb ram, and the workstation is attached through a gig switch which also contains the file server, same VLAN, no firewall. Each workstation takes a chunk, processes it, and puts it back. The new storage array now has 4 1gb links to it. The old storage was scsi direct attached to a proliant Dl380. The peak output of that server through the NIC was something like 500mbits/s.

I am in the process of upgrading that part of the network to 10gbe.

 

[Fox5]

Microsoft product...so they are not going to make a linux anything. I checked on the license and it is $40,000 for 20 more cores. And yes, it is cores. Throw in that quad and you have used up 4. Oracle is expensive as crap but at least they license per socket.

Luckily we have a microsoft guy on site so he is getting me in touch with the right people. I will trying to find out how Server 2008 HPC would handle this. Maybe it is cheaper to buy the license for that. I will probably be the first one asking this so they will probably have to rethink their pricing before I get an answer.

And Fox5, it could be other components slowing down the process, but mostly the CPU. On the pentium D's we use a dedicated SATA drive, 3gb ram, and the workstation is attached through a gig switch which also contains the file server, same VLAN, no firewall. Each workstation takes a chunk, processes it, and puts it back. The new storage array now has 4 1gb links to it. The old storage was scsi direct attached to a proliant Dl380. The peak output of that server through the NIC was something like 500mbits/s.

I am in the process of upgrading that part of the network to 10gbe.

Easiest way to check would just be to monitor cpu utilization. If it stays around 100% all the time, then you're cpu bound, or the app is using busy waits until it receives data.

$40k is a lot of money. If you were going to spend anywhere near that money, you'd probably just be better off buying new systems. You could easily double (possibly triple) the per core performance with newer cpus. Perhaps get a dual socket 6 core system (or single socket twelve core) and a dual socket 4 core system. Or a dual socket 12 core system and just find something else for the other cores to do. (provided you have a windows license that will let you have that many cores in a system)