in every session, meaning if you have firefox opened and never close it, firefox caches every single webpage you view for faster access. So, depending on how long you've had your browser open and how much it had to cache, it can pump up the memory usage.
there's an option to limit how much cache firefox uses, check for that. also, you didn't mention any extensions you use, which also contributes to crappy memory usage.
/end standard reply to firefox uses too much memory.
edit - unless you're gaming and notice an impact on your game, do people still seriously have a task monitor open and watching memory usage? Never understood why people get loads of ram and then be stingy with it. It's not going anywhere and easily clearable. lol.