Simple answer is yes....complicated answer is you'll be running at the default slowest memory speed installed. You also lose dual-channel if available as the channels have to match in size.
That said, more is usually better. I run three sticks and could care less as to what the default slowest stick is. For what that box does it works great with no downtime over hardware in 8 years. At the time I was building/upgrading I just bought the stuff that was on sale. Since I've gotten interested in performance I've resisted the urge to change things as this has worked so very well for so long.