I'd say that, barring extensive research, Phelps is the greatest swimmer ever. It's not the records that make him so. It's not the golds that do it either. It's the versatility. Name a swimmer, any swimmer, and they're pretty much a 1-trick pony.
Alexander Popov was a 50 and 100 guy. The 'Thorpedo' was a sprinter. Kitajima can only do breaststroke. Piersol can only do backstroke. Matt Biondi was almost exclusively a freestyler. Spitz was a freestyler and flyer. Janet Evans was a middle distance swimmer. Jeff Rouse was a backstroker.
IMO, only 1 person can hold a candle to him: Summer Sanders. She was the only athlete with the same versatility. Her career was just much shorter.
Too many swimmers get too specialized. Backstrokers, as a rule, can hardly float when it comes to breaststroke, and vice versa. Sprinters can barely complete 2 laps. Being a flyer I think gives Phelps an advantage. 16 years or amateur swimming experience taught me that butterflyers tend to be the most versatile swimmers in the pool.