According to ESPN.com, on average, Brady has been the 2nd QB taken in their leagues while Palmer has been 3rd. It basically comes down to whether you think Brady will get it done despite Branch's holdout and Troy Brown being the #2 there or if Palmer will have fully recovered from his injury. It's a tough call. You could always take a guy like Hasselbeck or take a chance that McNabb or Culpepper will do surprisingly well.
-Tom