Sounds like you did the right thing, to me.
Man, back when I was in HS, we had a bunch of "mentally developmentally disabled" kids or whatever they call them now. Some of them weren't bad, but some of them were downright awful. There was this one kid, he looked like he must have been 21, and he was there the entire time I was, 7th grade through 12th, and I know he didn't graduate when I did... anyway, he had a habit of whipping it out. He'd always get yelled at by whatever teacher or "liaison" was taking them around, but he never learned.
Another kid, about 6'8" and 300 lbs... he had a really high-pitched voice and talked really slow (he couldn't help those, I'm sure), but he always had this weird haircut. Like, inverse sideburns. They actually shaved upwards into normal hair growth. Think of Billy-Bob Thornton in Slingblade. And he would always ask for people's pictures, or pinch girls' a$$es... and he'd be nice and polite when talking to you normally but would keep muttering things like "i love you... you're a i love you..." At lunch, he'd always eat yogurt and spill half the cup on his shirt, and then lick it off his shirt.
Combine that with the really fat girl who galloped and skipped everywhere, the wheelchair-bound kid whose head was about as thick as large textbook (not even joking... it was about 3" thick and flat as a board), the 2 or 3 who had Down's, and the invasive smell of zero hygeine they all had, and they weren't a very fun bunch to be around.