Get the zooms out of the way first. You definitely want as wide a FL range as you can get. 24mm is pretty darn wide though, so if you find you don't swap to the 16-35 often then you can sell it and swap to a prime. But an ultrawide is vital for certain kinds of photography IMO. I would not discount the 17-40 f/4L. You could get that and the 85mm/1.8 for less than the price of the 16-35. Going from f/2.8 to f/4 is a decision you're apparently already comfortable with in your choice of walkaround and telephoto zooms, and wide-angles work with longer shutter speeds anyway, so I really don't see that you'd be giving up much by going with the 17-40. Anyway, based on your list, the first three are decided:
24-105
16-35
70-200 f/4L IS
Now you are faced with the choice of primes. Based on your list, I would actually propose a 3rd option: get the 85mm/1.8 and also pick up another crop body. 20D, 30D, whatever. (I would have just kept the 20D, personally.) That way you double your focal length options for every lens you own. The 85 becomes a 135 equivalent. The 70-200 becomes a 112-320, which dovetails nicely with the range of the 24-105. (You won't miss those 7mm between 105mm and 112mm.) Or you can put the 16-35 on the 5DII and the 24-105 on the crop, you have 16-35 covered and the crop becomes a 38-168.... again, a perfect dovetail between the two lenses on the two bodies. Except when I'm travelling and need to keep my kit as lightweight as possible, my 70-200 lives on my 20D and my 24-105 lives on my 5D. I want walkaround, I grab the 5D. I want long, I grab the 20D. It helps not to have to change lenses too.
Don't get me wrong, the 35L is awesome, as is the 135L. I covet both of them. I have actually considered selling my 50/1.4, 85/1.8 and 100Macro to get the 35L. That is the lens that I *want* and I think works beautifully on FF and also covers the handy "normal" range on crop bodies. But I can't quite bring myself to give up the awesomeness of the 85/1.8 and 100Macro. So I will just be saving up my pennies until I can afford the 35L.