You know what sucks, living on the second floor of a building with three floors. Then you get the nice noise from the people below and above you. Not usually too much of a problem though because I live in a college apartment, so no old people complaining that we're "making noise too late" at 11. Plus it cuts down on people getting up super early on the weekend, which kicks ass.
But man, those guys we had above us last year always sounded like they were moving their furniture around. Kind of annoying, especially at 6 AM (on a Sunday morning). Eh, that's apartments for you though.
As for your situation, you got to be quiet if you have anal people around, but it sounds like you were. I'd just talk to the manager and tell them the people are full of it.