I've been the recipient of some really priceless advice on these forums in the past, and so I thought I'd make a very small contribution back by sharing some of the research I did in the past couple days on duplicate image finders.
The one I've used for years is Unique Filer and I'm included a long review of it at the start. I was (and still am) looking for a better program to use, and so at the end of the review, I've included my thoughts on a few alternatives. If anyone else has suggestions, I'd be very grateful! (I found this thread [http://forums.anandtech.com/showthread.php?t=94935&highlight=duplicate+image] from a couple years ago which I used as a starting point in finding new targets).
* Unique Filer [http://www.uniquefiler.com/
This very old program (I used v1.4, not the 2.0 beta) is one I used a few years back and became very familiar with, so for full disclosure I have some bias here. I started this project looking to find a better program but really liking this one, and after reviewing the programs listed below, I'm not switching yet. The features I'm describing here are a baseline for what I expect to be in the other programs I reviewed.
Program installs very simply using a ZIP file, no bloat at all. It's shareware/nagware, but the nagging is very minor (about a 5 second delay when starting the program -- although it claims to nag you after every 10 deletions, it doesn't actually do this). It allows you to compare one folder of images (or a set of folders) either against themselves or against another set of folders. You can set it to find only identical images, or to find similar images based on an analysis of the actual image content, and in my experience its extremely good at identifying images that have been slightly cropped/distorted, have altered contrast, or have different resolutions, formats or sizes. You can set a "fuzz" factor that determines how sensitive the algorithm is, and after processing the results, a list of potential matches is produced. By looking at thumbnail images of each pair, you can manually flag images for deletion. There is also a setting to automatically flag (or automatically delete) images that are extremely similar or identical. Processing of images is not fast, but to speed things up you can save a datafile containing the comparison data on a set of images, which makes comparing that set against new images/sets much faster.
The pros of this program are that it has all of the features above (which I consider to be the most important), and does its job very well. Its speed was approximately the same as the other programs, and from experience I know it to be very accurate, the only exception being that it often gets wildly false positives on black and white line art (but its smart enough to never flag these for auto deletion unless they're genuinely identical). As an extra whistle/bell, you can use it to do simple filename/file size searches that don't actually "look" at images, which is a much faster scan, and you can use the same function on non-image files too.
There are a lot of cons, however, which made me start looking for a replacement. First, being almost 10 years old, the GUI is very dated and the icons are not intuitive at all. Until you become familiar with it, you really have to rely on the (very useful) hovering tooltips. The preview window for image pairs is resizable (which is a plus), but it does not auto scale the images into comparable thumbnails, so if you have a batch of images of mixed resolution (and with a very large spread) its not great for visually comparing them unless you have a large display. It does not have a function allowing you to move flagged images to a designated folder, so your only option for flagged images is to send them to the recycling bin.
Lastly and the biggest feature that is missing, is that the auto-flag criteria is very limited, based just on file size discrepancies. This is very useful for auto flagging identical images, but for images that are similar but not identical (e.g. the same image but different resolution, file type, or very minor difference) you have to manually go through and flag which copy you want to keep and which to delete. The biggest improvement would be a system that would at least initially pre-flag for you based on criteria (e.g. flag the lowest resolution image, or the greatest/smallest file size).
Here are some alternative programs I tried out:
* Image Comparer 3.0 [http://www.bolidesoft.com/imagecomparer.html]
I only gave this program a cursory look, since its commercial and the demo lacks one of the key functions most people would want (batch deletion/moving of duplicate files). I'd be willing to pay up to $20 for a program that really outshines the freeware alternatives, but at $35.00 I didn't really consider this utility.
* DupDetector [http://www.photo-freeware.net/dupdetector.php]
Freeware. It has a simple but usable interface and includes all the key features of Unique Filer, but also an auto delete setting where you can specify criteria for choosing between similar but not identical images (e.g. largest/smallest file size, modification date, resolution). However, the auto delete function didn't seem to work perfectly; at one point it was chugging along beautifully, but then after I tweaked my criteria, it stopped working all together. A further (but very minor) issue is that it appears not to recognize certain image types. It gets GIFs and JPEGs but it didn't appear to recognize BMPs or PNGs (possibly both).
* Photo Sort [http://www.asmdev.net/]
Another commercial program; the full version costs $25, but I had heard a lot of praise for this so I checked it out. The demo is fully functional except that it block batch deletes, which is fair in that it lets you see clearly how good a job it does without actually doing the job until you register. It has a lovely GUI, but overall this program was one of the worst I tested. First, it claims to be the fastest program of this type out there, in part because its written in assembly language, but although I didn't quantify this, it subjectively felt like it took substantially longer to run comparison searches. The much bigger issue is its complete lack of options/functionality: you give it a folder name, it searches that folder for similar images, and then its done. No ability to compare one set against another, to tweak the threshold at which similar images are flagged, or to save your comparison data to speed up future searches. Might be useful purely for eliminating 100% identical images, but the free programs already do that extremely well.
* Similar Images [http://fileforum.betanews.com/detail/SimilarImages/1143805292/1]
I liked this program (freeware) a lot; very clean user interface, with a prominent tooltip box explaining each function. It has all of the features of Unique Filer, as well as choosable criteria for autodeletion, very similar to DupDetector. However, the program repeatedly crashed out on me each time I actually tried to run a comparison! I run W7. I might come back to this one and try to get it to work by tweaking compatibility settings, and I'd definitely advise checking it out if you have an older OS or think you can make it work.
I know this is a pretty niche problem (duplicate images) overall, at least judging by the how old all the programs I found are, but like I said at the start, I'd be really grateful if anyone has suggestions for better utilities to try out.
The one I've used for years is Unique Filer and I'm included a long review of it at the start. I was (and still am) looking for a better program to use, and so at the end of the review, I've included my thoughts on a few alternatives. If anyone else has suggestions, I'd be very grateful! (I found this thread [http://forums.anandtech.com/showthread.php?t=94935&highlight=duplicate+image] from a couple years ago which I used as a starting point in finding new targets).
* Unique Filer [http://www.uniquefiler.com/
This very old program (I used v1.4, not the 2.0 beta) is one I used a few years back and became very familiar with, so for full disclosure I have some bias here. I started this project looking to find a better program but really liking this one, and after reviewing the programs listed below, I'm not switching yet. The features I'm describing here are a baseline for what I expect to be in the other programs I reviewed.
Program installs very simply using a ZIP file, no bloat at all. It's shareware/nagware, but the nagging is very minor (about a 5 second delay when starting the program -- although it claims to nag you after every 10 deletions, it doesn't actually do this). It allows you to compare one folder of images (or a set of folders) either against themselves or against another set of folders. You can set it to find only identical images, or to find similar images based on an analysis of the actual image content, and in my experience its extremely good at identifying images that have been slightly cropped/distorted, have altered contrast, or have different resolutions, formats or sizes. You can set a "fuzz" factor that determines how sensitive the algorithm is, and after processing the results, a list of potential matches is produced. By looking at thumbnail images of each pair, you can manually flag images for deletion. There is also a setting to automatically flag (or automatically delete) images that are extremely similar or identical. Processing of images is not fast, but to speed things up you can save a datafile containing the comparison data on a set of images, which makes comparing that set against new images/sets much faster.
The pros of this program are that it has all of the features above (which I consider to be the most important), and does its job very well. Its speed was approximately the same as the other programs, and from experience I know it to be very accurate, the only exception being that it often gets wildly false positives on black and white line art (but its smart enough to never flag these for auto deletion unless they're genuinely identical). As an extra whistle/bell, you can use it to do simple filename/file size searches that don't actually "look" at images, which is a much faster scan, and you can use the same function on non-image files too.
There are a lot of cons, however, which made me start looking for a replacement. First, being almost 10 years old, the GUI is very dated and the icons are not intuitive at all. Until you become familiar with it, you really have to rely on the (very useful) hovering tooltips. The preview window for image pairs is resizable (which is a plus), but it does not auto scale the images into comparable thumbnails, so if you have a batch of images of mixed resolution (and with a very large spread) its not great for visually comparing them unless you have a large display. It does not have a function allowing you to move flagged images to a designated folder, so your only option for flagged images is to send them to the recycling bin.
Lastly and the biggest feature that is missing, is that the auto-flag criteria is very limited, based just on file size discrepancies. This is very useful for auto flagging identical images, but for images that are similar but not identical (e.g. the same image but different resolution, file type, or very minor difference) you have to manually go through and flag which copy you want to keep and which to delete. The biggest improvement would be a system that would at least initially pre-flag for you based on criteria (e.g. flag the lowest resolution image, or the greatest/smallest file size).
Here are some alternative programs I tried out:
* Image Comparer 3.0 [http://www.bolidesoft.com/imagecomparer.html]
I only gave this program a cursory look, since its commercial and the demo lacks one of the key functions most people would want (batch deletion/moving of duplicate files). I'd be willing to pay up to $20 for a program that really outshines the freeware alternatives, but at $35.00 I didn't really consider this utility.
* DupDetector [http://www.photo-freeware.net/dupdetector.php]
Freeware. It has a simple but usable interface and includes all the key features of Unique Filer, but also an auto delete setting where you can specify criteria for choosing between similar but not identical images (e.g. largest/smallest file size, modification date, resolution). However, the auto delete function didn't seem to work perfectly; at one point it was chugging along beautifully, but then after I tweaked my criteria, it stopped working all together. A further (but very minor) issue is that it appears not to recognize certain image types. It gets GIFs and JPEGs but it didn't appear to recognize BMPs or PNGs (possibly both).
* Photo Sort [http://www.asmdev.net/]
Another commercial program; the full version costs $25, but I had heard a lot of praise for this so I checked it out. The demo is fully functional except that it block batch deletes, which is fair in that it lets you see clearly how good a job it does without actually doing the job until you register. It has a lovely GUI, but overall this program was one of the worst I tested. First, it claims to be the fastest program of this type out there, in part because its written in assembly language, but although I didn't quantify this, it subjectively felt like it took substantially longer to run comparison searches. The much bigger issue is its complete lack of options/functionality: you give it a folder name, it searches that folder for similar images, and then its done. No ability to compare one set against another, to tweak the threshold at which similar images are flagged, or to save your comparison data to speed up future searches. Might be useful purely for eliminating 100% identical images, but the free programs already do that extremely well.
* Similar Images [http://fileforum.betanews.com/detail/SimilarImages/1143805292/1]
I liked this program (freeware) a lot; very clean user interface, with a prominent tooltip box explaining each function. It has all of the features of Unique Filer, as well as choosable criteria for autodeletion, very similar to DupDetector. However, the program repeatedly crashed out on me each time I actually tried to run a comparison! I run W7. I might come back to this one and try to get it to work by tweaking compatibility settings, and I'd definitely advise checking it out if you have an older OS or think you can make it work.
I know this is a pretty niche problem (duplicate images) overall, at least judging by the how old all the programs I found are, but like I said at the start, I'd be really grateful if anyone has suggestions for better utilities to try out.
Last edited:
Facebook
Twitter