Fleece will be warmer. It traps more air forming a thicker insulating layer. Cotton also doesn't wick very well.
Edit: I'm a natural fiber snob and better than 90% of my wardrobe is 100% cotton, but for some things, petrochemicals are just better.
I no longer buy sweater that is made from cotton. it is heavy and it doesn't keep me warm. wool is the sh1t!