well, they're both objects that refer indirectly to another object.
the big difference is in the amount of control
you need to dereference a pointer with the * operator but you don't need to dereference & with any operator
it actually makes a bit of a difference when coding stuff like overloaded operators
of course then you get into null and const pointers.
In short, it doesn't really make a difference. Usually, it works out fine. Besides, use java and avoid all this crap
[edit]
just thought of something else. references can't be null. Pointers can be. That makes a difference. Say you need to pass an object to a function without allocating it any space. References have no null reference. This helps if you want to use an object in a function without testing if it exists. It saves a little hassle but these little things are easy to miss.