I'm a bit guilty of that as well... I will not invite some of my friends to stuff because I don't want them showing up with their kid (made the mistake of having a couple people over for pizza/beer when I was moving into my apartment and 80% of it was under construction... next think you know, my friends' toddler is running around in an area where I know there are nails and staples on the floor)
on the other hand, if the same friends send out a text seeing if anyone wants to hang out at their place for beer and poker on a Friday night after the kids are put to bed, that's a fine idea once in awhile.
Right, I go out to relax. I can't relax around children. They break stuff, they get into stuff, and their parents get upset when I act like myself around them. (Can you keep the cursing down, there are KIDS here!).
If they invite me, I expect children and I can decide if I want to deal with children. When I invite them I don't want children. If I did my wife and I would have children. In fact, if I don't explicitly say "bring the kids" that means no kids. Just like if I don't say "bring your wife" that means no wife! I hate guys who want to bring their girlfriend/wife to man outings almost as much.
When I call you and say "I've got 4 tickets to six flags, I'm taking my niece, bring your kid and let's go." I'm prepared for children. When I call you and say "Man, this week was stressful...you want to grab a cup of coffee and catch up?" That means I want adult conversation that's not interrupted every 5 minutes by a crazy person (toddler).
I'm understanding when my friends say no. I get that friends with kids might need notice, or might not be able to go. Just like I know my poor hipster friends can't come to my night out at a Ruth Chris. That doesn't mean I won't email everyone saying "Drinks and dinner at Chris's tonight, who's coming?"
They can say no just like I say no when I'm invited to that horrible dive bar downtown.