One of the most absurd things I remember from the series was in the episode rascals where a few ferengi manage to take over the ship. If ~5 enemies with no special abilities manage to take over the federation flagship that has a 1,000 member crew something is wrong.
It's also hard to believe that random ships from societies no one has never heard of before have no problem going up against the Enterprise in a fight. It would be like a Nimitz class aircraft carrier being taken out by some Somali pirates.