That is actually one one of the reasons skeptics give as proof that time travel is impossible. Personally, I don't think it's possible to travel through time. You can view the past (by viewing old light rays, ie: looking at the sun is looking 8 minutes into the past), but you can't interact with it. I believe that time is an abstract concept invented by people, not a physical or metaphysical or tangible thing. Think about it, to a slow turtle, time may seem accelerated. When they look at us, perhaps they see us moving at impossible speeds because their brains perceive time at a different rate. To that turtle, it may seem like a few moments to move 100 ft across the ground when in actuality, 10 minutes have passed. Or maybe a fly views time differently too. Maybe we move in slow motion to a fly.
We just created a concept of time that corresponds to our own perception of how fast things move. You can't go back in time one year because there is no such thing as a year. It's an abstract concept designed by us to keep track of when things happen.
Just my oppinion...