As everyone is noting, this is a common but not universal problem with pointers. It's due to a electronic bias which unfortunately is not user adjustable. I had one Toshiba that was almost constantly drifting. I had to disable the trackpoint. This was not a problem since it was almost unusable even if it worked OK. The next Toshiba only drifted if both the mouse and trackpoint was simultaneously active. After it got to be a year old or so, the drift pretty much went away but it still didn't work well. I now have a ThinkPad and this is the first pointer that is a reasonable substitute for a mouse. Also, it doesn't drift at all even with the mouse enabled.
I suggest trying disabling the mouse when you use the pointer (and vice versa) and see if that helps.