Question Favorite Windows program to find/remove duplicate files

[Lil'John]

Title basically is it.

I'm creating some master backups before replicating them to a large file server(movies, music, and other files) Currently, the content is going to be spread out on three 4TB drives almost completely full. I'm 90% certain there are some duplicate files.

I can hook the drives up to a Windows 8 or 11 computers or Windows 2012 server machine. If needed, I can setup an Ubuntu machine if it is going to make things much easier.

I'm hoping to find duplicates by name or by actual file... for example a movie named John.mp4 and John.nf.mp4 are the same but I want to find/delete the .nf one. The files could be in multiple nested directories.

Any good cheap/free utilities to help me?

 

[TheELF]

Everything search engine.
After it indexes all of the drives you can search for blank (everything) and it will allow you to sort by size, name, or date while showing all the files on all the drives all at once.
You can also only look for pictures or videos or whatever file type one at a time.
Sorting by size is a good way to find files that are the exact same.
It also accepts wildcards, so searching for John*.mp4 would find you your duplicates.

 

[Lil'John]

@TheELF Thank you for the reply. I'm not sure how Everything is any better than just plain Windows search (still looking)

For those looking for it:

https://www.voidtools.com/support/everything/

The problem is I've got at least 100k files if not much more to parse through. Movies was at least 1k, TV series was at 5k files, music was pretty small, but misc files/backups was that literal 100k++ files.

I could manually go through the media but no chance of misc files/backups that are nested deep.

 

[igor_kavinski]

dupeGuru

dupeGuru is a cross-platform (Linux, OS X, Windows) GUI tool to find duplicate files in a system. It’s written mostly in Python 3 and has the peculiarity of using multiple GUI toolkits, all using the same core Python code. On OS X, the UI layer is written in Objective-C and uses Cocoa. On Linux...

dupeguru.voltaicideas.net

Even has a fuzzy photo search to find similar photos.

 

[Lil'John]

dupeGuru

dupeGuru is a cross-platform (Linux, OS X, Windows) GUI tool to find duplicate files in a system. It’s written mostly in Python 3 and has the peculiarity of using multiple GUI toolkits, all using the same core Python code. On OS X, the UI layer is written in Objective-C and uses Cocoa. On Linux...

dupeguru.voltaicideas.net

Even has a fuzzy photo search to find similar photos.

I just gave this a whirl and it was hugely helpful. LOTS of fat trimmed. I didn't try to delete 100k's of file at once but I was able to track directories and delete via file explorer.

 

[Timur Born]

Usually I use "AllDup", which is free and works well for duplicate files and images. If you need this very regularly then I would suggest looking into a paid product that offers multi-core support for much faster results.

 

[Timur Born]

This prompted me to look for free multi-core duplicate file finder again and I did find one.

GitHub - qarmin/czkawka: Multi functional app to find duplicates, empty folders, similar images etc.

Multi functional app to find duplicates, empty folders, similar images etc. - GitHub - qarmin/czkawka: Multi functional app to find duplicates, empty folders, similar images etc.

github.com

 

[james1701]

Delete.