Regardless of the dollar amounts, if you are borrowing, borrow at the least interest %. So borrowing $2000 extra from your mortgage at 6% is better than having $2000 on a 12% credit card.
If you can make more money on your investments then you should invest instead of pay off the mortgage. However, you do have to pay tax on interest that you earn,, so this would cancel out the tax credit that you get for paying mortgage interest. So, unless you can get an investment that pays more than your mortgage (such things don't exist unless you are willing to take a risk), you should just pay off your mortgage. However, lets say you have a 30-yr mortgage at 6%, and then in 5 years interest rates jump up to 10%. You are still paying only 6% on your mortgage, but now you can earn 10% with GICs or gov't bonds, this is when you should invest instead of paying off the mortgage.
Just like any investment, it all has to do with your risk tolerance. The more you are willing to risk (stocks, etc) to more you should be investing instead of paying off your mortgage.