I've taken Zoloft and Klonopin on and off for about six years. Whatever the negative side effects, for me at least, the benefit of taking them outweighs them. I have noticed some recent memory and concentration problems, but they're related to something else, not the meds (but the meds exaggerate them some days, I think).
Zoloft doesn't improve my mood. The best way I can describe it is that it takes some of the depth out of the valleys of my depression. The lows aren't as low anymore, but the highs aren't as high either. It just sort of makes my mood in general more even, not necessarily better. I still have days when I have panic attacks and get very depressed to the point where I think about harming myself - but I know if I wasn't taking the meds, I probably would go further than just thinking about it.
If you're concerned about the side effects, definitely talk to your prescribing doctor (I hope you're seeing a psychiatrist for anti-depressant scripts). He/she might change the dose or give you something else; you're right though - quitting on your own was foolish (I did it too, several times).
If you don't think your prescribing doc is listening to you or if you don't feel like he/she's addressing your concerns about side effects, switch doctors if you can. Finding the right doc for this kind of thing is really key to managing your symptoms and dosages.
Good luck!