Question Removing duplicated entries in a very large single csv file

[Chattanooga_Chuck]

I have a MASSIVE csv file. I generate one of these a week from 12mb up 50mb in size. Some lines do get duplicated. I have to go in and search the duplicates out. I am looking for some way to automate this task. I have looked a while but have yet to find anything.
Has anyone here heard/seen anything that does this?