Well, your reference is always up when you're looking at the screen of the phone, while recording video and rotating it in your hand.
Fix the phone to your head and then rotate side-to-side, and you'll see how your phone sees the world.
Just figure out which way is "up" to your phone. If it's a smartphone, chances are it's meant to be held horizontally. This is the reason you see so many vertical framed cellphone videos on youtube.