Depends on what you're trying to do that the exhaustion is impeding. Are you trying to do calculus? Being mentally exhausted is harder. Are you trying to run a marathon? Being physically exhausted is harder.
Since I'm not much of an athlete usually, I tend to mentally exhausted a lot more than physically. I eat like crap, don't sleep much, and more often than not I'm pushing myself pretty hard in my work.