Handbrake is a great program and I use it all the time. But it does not meet the need here. What I want to do is rip the dvd, compress it down and burn a new dvd on a single layer disk. Just like dvdshrink. Mac the Ripper solved the first part. However, I narrowed it down to a few options I'm exploring.
1. Rip with handbreak to the highest quality reasonable, use ffmpegX or Burn.app to create dvd. This works and I've tested it. I already rip all the dvd's I own to Xbox 360 compatable mp4 so turning those into dvd's should be just fine. However, for just doing a dvd backup, this is time consuming...very time consuming.
2. compile lxdvdrip and use that. - Provided I can get it to compile properly, this works great and I've used it on linux in the past.
3. Use a combo of transcode, myplayer, dvdauthor, etc via a script to do the same thing. - The hardest route, but could give the best results.