The interest payment of 6.25% at this point in your loan is $536.46/month (dropping steadilly as you pay the principal). If you stay for exactly 3 years, you'd pay a total of $18908 in intererst. The three year point is when you either sell or refinance a 2nd time (although a second refinance would probably be unlikely if you have a stable relationship with your partner).
The interest payment of 4.625% on $106k starts at $408.54/month. Over three years, you'd pay $14361 in interest.
Thus, if you stayed exactly three years, you'd save $4546, but you'd have to fork over the $3000 in refinance costs. You'd end up GAINING $1546. The refinance would be a wise move in this case, or in any case where you stay longer.
If you stayed only 2 years, you'd net a whopping gain of $46 after all is said and done.
Finally, if you have to move after one year (lost job, divorce, unforseen incident, etc), this refinance would LOSE you $1470.
It all comes down to when you move.
Note: there will be minor tax differences that I didn't include (more points now vs less interest then). But at your situation, the tax consequences are probably not too big since you owe so little.