<< So whose tougher >>
Define "tougher". Out of all listed, the Rangers have the least difficult qualification process and physical requirements, not to imply that its 'easy'. Although the Rangers fall under the same Special Operations Command as the Navy Seals and Green Berets, they are not considered "special forces". Rangers are advanced light infantry and fast attack forces who are "special forces capable", which means they are qualified to operate in joint missions with true special forces (usually in close supportive roles).
In terms of stamina and physical endurance, the Navy Seals would get my vote. Believe it or not, NONE of the U.S. special forces receive much hand-to-hand and martial arts training. Navy Seals receive a 25-hour (approx.) crash course in martial arts based hand-to-hand combat, and that is a vast improvement from what they received a decade ago. It is very common for special forces operators from all branches to seek additional hand-to-hand training on their own time because many feel as though their military hand-to-hand training wasn't as comprehensive as they would like.
As one poster stated, there is no answer to a question like this. It comes down to the individual, not a particular branch of service or unit. You're going to find operators from all branches who are as tough as they come, including the British and Australian SAS, and you can find the occasional guy right off the street who is tougher than most.