Gene Hackman in the French Connection fits your description precisely; the French Connection II is also worth a watch, but not as good as the first. Heat features an ensemble of criminals that may fit the bill as well. Then you have L.A. Confidential, Goodfellas, and Casino with the tough guy characters in prominent supporting roles. Lastly, The Duellists has Harvey Keitel playing a great tough guy, though in an antagonistic role.
I'd also highly recommend Drive and Taxi Driver as others brought them up, but I don't personally think of the characters as tough guys exactly. So if you watch them, it might be best not to go in with that expectation.