image processing

[drwoo123]

I was wondering if anyone knows of any useful application or API that can do distributed image processing. For example I have an image with AxB dimensions. Lets say the image is 500MB. I want to resize - clearly on a single processor machine this will have limited speed, and I will only be able to scale up for so much by adding processors to one machine. So i was thinking of having about 10 machines doing the processing. I have approximately 10,000 images like this.

Sort of like a render farm in animation - but for still images.

Thanks in advance

 

[HN]

put 1000 pictures on each machine and run the same batch process (irfanview?) on each machine.

 

[itachi]

well, if you have a program to resize the image.. you can use clustering software to distribute the task among different nodes.
http://www.cmbi.kun.nl/models/
this'll work with both windows and linux, and has a relatively simple interface and small footprint.

http://www.rocksclusters.org/Rocks/
more efficient and powerful.. but it's more for a dedicated linux cluster.

 

[drwoo123]

well I am hoping for something that would be a little different - in the with the idea of the following - as I had more servers - the system speeds up - I know of course I could manually divide the files and do exactly what you are saying - but im hoping that lets say that when there are only 1000 images left - each machine would then have 100 images that it can process - or more accurately - the load is distributed across the servers. I know I can write my own queue management application and then just use some shared storage - but looking for something more elegant than that. I can't be the first person to have run across this issue.


Don't know if I am being clear.


thanks again

 

[itachi]

Originally posted by: drwoo123
well I am hoping for something that would be a little different - in the with the idea of the following - as I had more servers - the system speeds up - I know of course I could manually divide the files and do exactly what you are saying - but im hoping that lets say that when there are only 1000 images left - each machine would then have 100 images that it can process - or more accurately - the load is distributed across the servers. I know I can write my own queue management application and then just use some shared storage - but looking for something more elegant than that. I can't be the first person to have run across this issue.


Don't know if I am being clear.


thanks again

you don't need to manually divide it.. you just set up a job and send it to the main node. lets say you want to resize an image that's 32,768 x 16,384 to 256x128.. you'd need a command-line program that could resize the image.. the cluster software will send all the files needed to run independently on all the nodes.. the main node then processes the program and splits it up into smaller tasks, and tells each node what to do.. once the node completes the task, it uploads it back to the main node.. where its reassembled.

if you have 1000 images left.. with 10 nodes.. then with a perfect distribution, each node will process 1/10th of a file, 1000 times.

check out www.sourceforge.net for image processing programs.
for your application.. models@home (the first one i mentioned) would be more ideal for your application. if you want to set it up, i'll try to help you out as much as i can.

 

[drwoo123]

Wow Excellent - I will check it out - its exactly what I was talking about - great info - ill try and do some research and then ask some meaningful questions.

 

[Childs]

There are lots of apps that do this. Typically, as itachi said, you need a command line app on each node, they you just take the frame range and use some simple logic to divide the frames into segments, and send them off to each machine in your cluster. At work we use Qmaster for this to do Shake, Maya, After Effects, etc. Others use rush. But the last time I checked there are lots of apps to do this, most shops just write their own.

 

[drwoo123]

Originally posted by: Childs
There are lots of apps that do this. Typically, as itachi said, you need a command line app on each node, they you just take the frame range and use some simple logic to divide the frames into segments, and send them off to each machine in your cluster. At work we use Qmaster for this to do Shake, Maya, After Effects, etc. Others use rush. But the last time I checked there are lots of apps to do this, most shops just write their own.

Any personal recommendations on a really "good one" - or perhaps some comparison info?

Also links for rush and qmaster - google search turned up too much junk.

thanks

 

[delroy]

here's one more you maye have missed
http://icl.cs.utk.edu/netsolvedev/applications/dips.html

.... At its core, DIPS uses NetSolve as its metacomputing resource to provide unprecedented computing power by aggregating distributed resources on the Internet to a single system.