Well, like everyone said, you have to take a stand and give her an ultimatium right back: that she can't keep giving *you* one.
If all you really did was not pay attention to her, she left, and you asked for her back, of course he'll always hold it against you. The problem here is how "big" you let that become, but behaving guilty every time she brings it up. If you're not like that anymore, you need to point that out and make sure she understands. If there are bigger problems... it is sad you have kids, but don't "stay together for the kids". You can still be a part of their life, while not married to her.
People have suggested seeing someone, and that might be the way to go. Don't let her keep saying that, and you not doing anything though.